A Step-by-Step Guide to Developing Sentiment Model with SentenceTransformers
A Deep Dive into Transformer-based Text Representations for Sentiment Analysis
Machine Learning is fantastic. We can achieve marvellous results on pattern recognition tasks using machine learning. The trouble is a machine learning model only understands the numbers. And when we have text data, they are not directly applicable. We need some way to translate the text into numbers.
Several techniques, such as bag-of-words, tf-idf, word2vec, etc., transform text into numbers. The traditional bag-of-words approach often fails to capture the details and context in the text. This leads to inaccurate predictions. However, recent advancements in NLP have introduced powerful techniques like transformers. They have revolutionised how to represent the text data numerically and preserve the meaning of the data.
In this guide, we will explore SentenceTransformers and their applications in sentiment identification. We’ll start by understanding the SentenceTransformers package. It can capture context and handle complex language. From there, we’ll cover the practical steps. We will cover data prep, training, and evaluation.
Understanding Sentence Transformers
SentenceTransformers is a Python framework based on PyTorch and Transformers. It offers a large collection of pre-trained models tuned for various tasks. It can generate dense vector representations, or embeddings, for entire sentences or passages. These embeddings capture the semantic and contextual information within the text. These embeddings enable more accurate sentiment classification compared to traditional word-level embeddings. SentenceTransformers can harness transfer learning and pre-trained models. They can be fine-tuned for sentiment tasks. This results in better performance and adaptability.
You can use this framework to calculate sentence and text embeddings. It works for over 100 languages. You can then compare them using cosine similarity. This lets you find sentences with a similar meaning.
You can install the SentenceTransformers package using the following command.
pip install -U sentence-transformers
Build a Sentiment Model
Import necessary libraries
First, let's import the required libraries.
We need pandas
and numpy
to load and manipulate data. For the model development, we will use scikit-learn
library. Here, we have imported the metrics
module to evaluate the mode, model_selection
to split the data into train and test sets, and the linear_model
to train a model.
We imported the SentenceTransformer
class from the sentence_transformers package. This will help us to create sentence embeddings.
Load data
We will use Amazon's customer review data to build a sentiment classification model.
This data has 20k rows with 76.2% positive and 23.8% negative reviews.
So, we have 20000 reviews where 15.2k(76%) are positive and 4.7k(24%) are negative.
Create samples
We will split the data into train (80%) and test (20%) sets. We will train the model on the train set and then will evaluate the model performance on the holdout set (test).
While splitting the dataset we used the stratify
parameter so that we have an equal proportion of positive and negative samples in both train and set sets.
After this step, we have the following shapes for the train and test sets respectively.
# train shape
(16000,)
# test shape
(4000,)
Create features
Now, it is time to use the magic of sentence-transformer
library to get the numerical representation of the customer reviews.
There is a lot going on in the above code. Let’s understand it piece by piece.
First, we instantiate an object of SentenceTransformer
class. For this, we used a pre-trained language model called all-MiniLM-L6-v2
. This model is trained on 2 billion word pairs and is well-equipped to generate numerical features from the text data. This model generates a feature vector of 384 dimensions for each row in the input data.
Then we applied this model to our training
and test data to get the numpy
arrays of numbers.
After this step, we have the following shapes for the train and test sets respectively.
# train shape
(16000, 384)
# test shape
(4000, 384)
Notice that we got 384 for each training example.
Train model
Now, we have all we need to train a classification model. We will train a simple Logistic Regression model for this demonstration.
This simple model achieves an accuracy of 90% on the train and test sets. Which is fantastic. We can further improve this accuracy by training a more complex model such as XGBoost or neural networks.
Conclusion
In the guide, we saw how sentence-transformer makes our life a bit easier. While traditional methods of transforming text to numbers do require a lot of data preprocessing. The sentence-transformer package works out of the box.