The customer support teams were drowning with the overwhelming volume of customer inquiries at every company I\u2019ve worked at. Have you had similar experiences?<\/p>\n
What if I told you that you could use AI to automatically identify<\/strong>, categorize,<\/strong> and even resolve<\/strong> the most common issues?<\/p>\n By fine-tuning a transformer model like BERT, you can build an automated system that tags tickets by issue type and routes them to the right team.<\/p>\n In this tutorial, I\u2019ll show you how to fine-tune a transformer model for emotion classification in five steps:<\/p>\n By the end, you\u2019ll have a fine-tuned model that classifies emotions from text inputs with high accuracy, and you\u2019ll also learn how to interpret these predictions using SHAP.<\/p>\n This same approach can be applied to real-world use cases beyond emotion classification, such as customer support automation, sentiment analysis, content moderation, and more.<\/p>\n Let\u2019s dive in!<\/p>\n When selecting a transformer model for Text Classification<\/a><\/strong>, here\u2019s a quick breakdown of the most common models:<\/p>\n For this tutorial, I chose to fine-tune DistilBERT because it offers the best balance between performance and efficiency.<\/p>\n Ensure you have the required libraries installed:<\/p>\n I used the Emotions dataset for NLP<\/a> by Praveen Govi, available on Kaggle and licensed for commercial use<\/a>. It contains text labeled with emotions. The data comes in three Each line contains a sentence and its corresponding emotion label, separated by a semicolon:<\/p>\n Let\u2019s load the dataset:<\/p>\n This dataset contains 16k training examples<\/strong> and 2k examples<\/strong> for the validation and testing. Here\u2019s the label distribution breakdown:<\/p>\n The bar chart above shows that the dataset is imbalanced,<\/strong> with the majority of samples labels as joy and sadness.<\/p>\n For a fine-tuning a production model, I would consider experimenting with different sampling techniques to overcome this class imbalance problem and improve the model\u2019s performance.<\/p>\n Next, I loaded in DistilBERT\u2019s tokenizer:<\/p>\n Then, I used it to tokenize text data and transform the labels into numerical IDs:<\/p>\n Next, I loaded a pre-trained DistilBERT model with a classification head for our text classification text. I also specified what the labels for this dataset looks like:<\/p>\n The pre-trained DistilBERT model for classification consists of five layers plus a classification head<\/strong>.<\/p>\n To prevent overfitting, I froze the first four layers<\/strong>, preserving the knowledge learned during pre-training. This allows the model to retain general language understanding while only fine-tuning the fifth layer and classification head to adapt to my dataset. Here\u2019s how I did this:<\/p>\n Given the label imbalance, I thought accuracy may not be the most appropriate metric, so I chose to include other metrics suited for classification problems like precision, recall, F1-score, and AUC score.<\/p>\n I also used \u201cweighted\u201d averaging for F1-score, precision, and recall to address the class imbalance problem. This parameter ensures that all classes contribute proportionally to the metric and prevent any single class from dominating the results:<\/p>\n Let\u2019s set up the training process:<\/p>\n After training, I evaluated the model\u2019s performance on the test set:<\/p>\n This is how the fine-tuned DistilBERT model did on the test set compared to the pre-trained base model:<\/p>\n Before fine-tuning, the pre-trained model performed poorly on our dataset, because it hasn\u2019t seen the specific emotion labels before. It was essentially guessing at random, as reflected in an AUC score of 0.5 that indicates no better than chance.<\/p>\n After fine-tuning, the model significantly improved across all metrics<\/strong>, achieving 83% accuracy in correctly identifying emotions. This demonstrates that the model has successfully learned meaningful patterns in the data, even with just 16k training samples.<\/p>\n That\u2019s amazing!<\/p>\n I tested the fine-tuned model on three sentences and here are the emotions that it predicted:<\/p>\n Impressive, right?!<\/p>\n I wanted to understand how the model made its predictions, I used using SHAP<\/a> (Shapley Additive exPlanations) to visualize feature importance.<\/p>\n I started by creating an explainer:<\/p>\n Then, I computed SHAP values using the explainer:<\/p>\n The plot below visualizes how each word in the input text contributes to the model\u2019s output using SHAP values:<\/p>\n In this case, the plot shows that \u201canxiety\u201d is the most important factor in predicting \u201cfear\u201d as the emotion.<\/p>\n The SHAP text plot is a nice, intuitive, and interactive way to understand predictions by breaking down how much each word influences the final prediction.<\/p>\n You\u2019ve successfully learned to fine-tune DistilBERT for emotion classification from text data! (You can check out the model on Hugging Face here<\/a>).<\/p>\n Transformer models can be fine-tuned for many real-world applications, including:<\/p>\n Fine-tuning is an effective and efficient way to adapt powerful pre-trained models to specific tasks with a relatively small dataset.<\/p>\n What will you fine-tune next?<\/p>\n Want to build your AI skills?<\/strong><\/p>\n\n
Choosing the Right Transformer Model<\/h2>\n
\n
Step 1: Setup and Installing Dependencies<\/strong><\/h3>\n
!pip install datasets transformers torch scikit-learn shap<\/code><\/pre>\nStep 2: Load and Preprocess Data<\/strong><\/h3>\n
.txt<\/code> files: train, validation,<\/strong> and test<\/strong>.<\/p>\ntext; emotion\n\"i didnt feel humiliated\"; \"sadness\"\n\"i am feeling grouchy\"; \"anger\"\n\"im updating my blog because i feel shitty\"; \"sadness\"<\/code><\/pre>\nParsing the Dataset into a Pandas DataFrame<\/strong><\/h3>\n
def parse_emotion_file(file_path):\n\"\"\"\n\u00a0 \u00a0 Parses a text file with each line in the format: {text; emotion}\n\u00a0 \u00a0 and returns a pandas DataFrame with 'text' and 'emotion' columns.\n\n\u00a0 \u00a0 Args:\n\u00a0 \u00a0 - file_path (str): Path to the .txt file to be parsed\n\n\u00a0 \u00a0 Returns:\n\u00a0 \u00a0 - df (pd.DataFrame): DataFrame containing 'text' and 'emotion' columns\n\u00a0 \u00a0 \"\"\"\n\u00a0 \u00a0 texts = []\n\u00a0 \u00a0 emotions = []\n\u00a0 \u00a0\n\u00a0 \u00a0 with open(file_path, 'r', encoding='utf-8') as file:\n\u00a0 \u00a0 \u00a0 \u00a0 for line in file:\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 try:\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # Split each line by the semicolon separator\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 text, emotion = line.strip().split(';')\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # append text and emotion to separate lists\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 texts.append(text)\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 emotions.append(emotion)\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 except ValueError:\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 continue\n\u00a0 \u00a0\n\u00a0 \u00a0 return pd.DataFrame({'text': texts, 'emotion': emotions})\n\n# Parse text files and store as Pandas DataFrames\ntrain_df = parse_emotion_file(\"train.txt\")\nval_df = parse_emotion_file(\"val.txt\")\ntest_df = parse_emotion_file(\"test.txt\")<\/code><\/pre>\nUnderstanding the Label Distribution<\/strong><\/h3>\n
Step 3: Tokenization and Data Preprocessing<\/strong><\/h3>\n
from transformers import AutoTokenizer\n\n# Define the model path for DistilBERT\nmodel_name = \"distilbert-base-uncased\"\n\n# Load the tokenizer\ntokenizer = AutoTokenizer.from_pretrained(model_name)<\/code><\/pre>\n# Tokenize data\ndef preprocess_function(df, label2id):\n\u00a0 \u00a0 \"\"\"\n\u00a0 \u00a0 Tokenizes text data and transforms labels into numerical IDs.\n\n\u00a0 \u00a0 Args:\n\u00a0 \u00a0 \u00a0 \u00a0 df (dict or pandas.Series): A dictionary-like object containing \"text\" and \"emotion\" fields.\n\u00a0 \u00a0 \u00a0 \u00a0 label2id (dict): A mapping from emotion labels to numerical IDs.\n\n\u00a0 \u00a0 Returns:\n\u00a0 \u00a0 \u00a0 \u00a0 dict: A dictionary containing:\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 - \"input_ids\": Encoded token sequences\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 - \"attention_mask\": Mask to indicate padding tokens\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 - \"label\": Numerical labels for classification\n\n\u00a0 \u00a0 Example usage:\n\u00a0 \u00a0 \u00a0 \u00a0 train_dataset = train_dataset.map(lambda x: preprocess_function(x, tokenizer, label2id), batched=True)\n\u00a0 \u00a0 \"\"\"\n\u00a0 \u00a0 tokenized_inputs = tokenizer(\n\u00a0 \u00a0 \u00a0 \u00a0 df[\"text\"],\n\u00a0 \u00a0 \u00a0 \u00a0 padding=\"longest\",\n\u00a0 \u00a0 \u00a0 \u00a0 truncation=True,\n\u00a0 \u00a0 \u00a0 \u00a0 max_length=512,\n\u00a0 \u00a0 \u00a0 \u00a0 return_tensors=\"pt\"\n\u00a0 \u00a0 )\n\n\u00a0 \u00a0 tokenized_inputs[\"label\"] = [label2id.get(emotion, -1) for emotion in df[\"emotion\"]]\n\u00a0 \u00a0 return tokenized_inputs\n\u00a0 \u00a0\n# Convert the DataFrames to HuggingFace Dataset format\ntrain_dataset = Dataset.from_pandas(train_df)\n\n# Apply the 'preprocess_function' to tokenize text data and transform labels\ntrain_dataset = train_dataset.map(lambda x: preprocess_function(x, label2id), batched=True)<\/code><\/pre>\nStep 4: Fine-Tuning Model<\/strong><\/h3>\n
# Get the unique emotion labels from the 'emotion' column in the training DataFrame\nlabels = train_df[\"emotion\"].unique()\n\n# Create label-to-id and id-to-label mappings\nlabel2id = {label: idx for idx, label in enumerate(labels)}\nid2label = {idx: label for idx, label in enumerate(labels)}\n\n# Initialize model\nmodel = AutoModelForSequenceClassification.from_pretrained(\n\u00a0 \u00a0 model_name,\n\u00a0 \u00a0 num_labels=len(labels),\n\u00a0 \u00a0 id2label=id2label,\n\u00a0 \u00a0 label2id=label2id\n)<\/code><\/pre>\n# freeze base model parameters\nfor name, param in model.base_model.named_parameters():\n\u00a0 \u00a0 param.requires_grad = False\n\n# keep classifier trainable\nfor name, param in model.base_model.named_parameters():\n\u00a0 \u00a0 if \"transformer.layer.5\" in name or \"classifier\" in name:\n\u00a0 \u00a0 \u00a0 \u00a0 param.requires_grad = True<\/code><\/pre>\nDefining Metrics<\/strong><\/h3>\n
def compute_metrics(p):\n\u00a0 \u00a0 \"\"\"\n\u00a0 \u00a0 Computes accuracy, F1 score, precision, and recall metrics for multiclass classification.\n\n\u00a0 \u00a0 Args:\n\u00a0 \u00a0 p (tuple): Tuple containing predictions and labels.\n\n\u00a0 \u00a0 Returns:\n\u00a0 \u00a0 dict: Dictionary with accuracy, F1 score, precision, and recall metrics, using weighted averaging\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 to account for class imbalance in multiclass classification tasks.\n\u00a0 \u00a0 \"\"\"\n\u00a0 \u00a0 logits, labels = p\n\u00a0 \u00a0\n\u00a0 \u00a0 # Convert logits to probabilities using softmax (PyTorch)\n\u00a0 \u00a0 softmax = torch.nn.Softmax(dim=1)\n\u00a0 \u00a0 probs = softmax(torch.tensor(logits))\n\u00a0 \u00a0\n\u00a0 \u00a0 # Convert logits to predicted class labels\n\u00a0 \u00a0 preds = probs.argmax(axis=1)\n\n\u00a0 \u00a0 return {\n\u00a0 \u00a0 \u00a0 \u00a0 \"accuracy\": accuracy_score(labels, preds),\u00a0 # Accuracy metric\n\u00a0 \u00a0 \u00a0 \u00a0 \"f1_score\": f1_score(labels, preds, average='weighted'),\u00a0 # F1 score with weighted average for imbalanced data\n\u00a0 \u00a0 \u00a0 \u00a0 \"precision\": precision_score(labels, preds, average='weighted'),\u00a0 # Precision score with weighted average\n\u00a0 \u00a0 \u00a0 \u00a0 \"recall\": recall_score(labels, preds, average='weighted'),\u00a0 # Recall score with weighted average\n\u00a0 \u00a0 \u00a0 \u00a0 \"auc_score\": roc_auc_score(labels, probs, average=\"macro\", multi_class=\"ovr\")\n\u00a0 \u00a0 }<\/code><\/pre>\n# Define hyperparameters\nlr = 2e-5\nbatch_size = 16\nnum_epochs = 3\nweight_decay = 0.01\n\n# Set up training arguments for fine-tuning models\ntraining_args = TrainingArguments(\n\u00a0 \u00a0 output_dir=\".\/results\",\n\u00a0 \u00a0 evaluation_strategy=\"steps\",\n\u00a0 \u00a0 eval_steps=500,\n\u00a0 \u00a0 learning_rate=lr,\n\u00a0 \u00a0 per_device_train_batch_size=batch_size,\n\u00a0 \u00a0 per_device_eval_batch_size=batch_size,\n\u00a0 \u00a0 num_train_epochs=num_epochs,\n\u00a0 \u00a0 weight_decay=weight_decay,\n\u00a0 \u00a0 logging_dir=\".\/logs\",\n\u00a0 \u00a0 logging_steps=500,\n\u00a0 \u00a0 load_best_model_at_end=True,\n\u00a0 \u00a0 metric_for_best_model=\"eval_f1_score\",\n\u00a0 \u00a0 greater_is_better=True,\n)\n\n# Initialize the Trainer with the model, arguments, and datasets\ntrainer = Trainer(\n\u00a0 \u00a0 model=model,\n\u00a0 \u00a0 args=training_args,\n\u00a0 \u00a0 train_dataset=train_dataset,\n\u00a0 \u00a0 eval_dataset=val_dataset,\n\u00a0 \u00a0 tokenizer=tokenizer,\n\u00a0 \u00a0 compute_metrics=compute_metrics,\n)\n\n# Train the model\nprint(f\"Training {model_name}...\")\ntrainer.train()<\/code><\/pre>\nStep 5: Evaluating Model Performance<\/strong><\/h3>\n
# Generate predictions on the test dataset with fine-tuned model\npredictions_finetuned_model = trainer.predict(test_dataset)\npreds_finetuned = predictions_finetuned_model.predictions.argmax(axis=1)\n\n# Compute evaluation metrics (accuracy, precision, recall, and F1 score)\neval_results_finetuned_model = compute_metrics((predictions_finetuned_model.predictions, test_dataset[\"label\"]))<\/code><\/pre>\nStep 6: Interpreting Predictions with SHAP<\/strong><\/h3>\n
\n
![\"\ud83d\ude31\"](\"https:\/\/i0.wp.com\/s.w.org\/images\/core\/emoji\/15.0.3\/72x72\/1f631.png?ssl=1\")
\n<\/li>\n![\"\ud83d\ude21\"](\"https:\/\/i0.wp.com\/s.w.org\/images\/core\/emoji\/15.0.3\/72x72\/1f621.png?ssl=1\")
\n<\/li>\n![\"\ud83d\ude00\"](\"https:\/\/i0.wp.com\/s.w.org\/images\/core\/emoji\/15.0.3\/72x72\/1f600.png?ssl=1\")
\n<\/li>\n<\/ol>\n# Build a pipeline object for predictions\npreds = pipeline(\n\u00a0 \u00a0 \"text-classification\",\n\u00a0 \u00a0 model=model_finetuned,\n\u00a0 \u00a0 tokenizer=tokenizer,\n\u00a0 \u00a0 return_all_scores=True,\n)\n\n# Create an explainer\nexplainer = shap.Explainer(preds)<\/code><\/pre>\n# Compute SHAP values using explainer\nshap_values = explainer(example_texts)\n\n# Make SHAP text plot\nshap.plots.text(shap_values)<\/code><\/pre>\nSummary<\/strong><\/h2>\n
\n
\n
![\"\"](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXfdpdGG2nGefo5cz3tmpBuw7-Uefo3uu9UqCAPO6z3QEvruHA0xd2r_7Tx7qJNTEJWsRA5r7f3AVgmMPVm5o3D_mNCEe3TEh6xwgqoa9kdqGTtsORFAV3sYJbZUXXrj4zcwtNytMg?key=EdH2R_lZ0C0rh-CBHFDARV_1\")
![\"\"](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXdmtFY_1j5ii7Ck3HYwJu5h5bOBtagiADiijo6ZizxSwdF0sd5-5dN48o0xM6vIZ4Jc8jg9_OtdDx99MpphTwdUyeBaadIWZ6RsV2Etd1IqmlTC-5eChfe3MazIZ45jw40CbPEIAA?key=EdH2R_lZ0C0rh-CBHFDARV_1\")
![\"\"](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXd_aMzQqh8lbjBUKZiI0cNtQ0ND1mklPP49k1vcaa3_jAcUKNIWWlHK8a0yRFJ-LecTxMcTgbaxoh57L45sEYeZcWwM1-KJ5UQt6_f95CO2y0vuCJ5UAA2SsmbH1reR25ffwgiZ?key=EdH2R_lZ0C0rh-CBHFDARV_1\")
![\"\ud83d\udc49\ud83c\udffb\"](\"https:\/\/i0.wp.com\/s.w.org\/images\/core\/emoji\/15.0.3\/72x72\/1f449-1f3fb.png?ssl=1\")
I run the