With the new age of problem-solving augmented by Large Language Models (LLMs), only a handful of problems remain that have subpar solutions. Most classification problems (at a PoC level) can be solved by leveraging LLMs at 70\u201390% Precision\/F1 with just good prompt engineering techniques, as well as adaptive in-context-learning (ICL) examples.<\/p>\n
What happens when you want to consistently achieve performance\u00a0higher<\/strong> than that \u2014 when prompt engineering no longer suffices?<\/p>\n Text classification is one of the oldest and most well-understood examples of supervised learning. Given this premise, it should\u00a0really\u00a0not be hard to build robust, well-performing classifiers that handle a large number of input classes, right\u2026?<\/p>\n Welp. It is.<\/p>\n It actually has to do a lot more with the \u2018constraints\u2019 that the algorithm is generally expected to work under:<\/p>\n Ever tried building a classifier beyond a few dozen classes under these conditions? (I mean, even GPT could probably do a great job up to ~30 text classes with just a few samples\u2026)<\/p>\n Considering you take the GPT route \u2014 If you have more than a couple dozen classes or a sizeable amount of data to be classified, you are gonna have to reach deep into your pockets with the system prompt, user prompt, few shot example tokens that you will need to classify\u00a0one sample.<\/strong><\/span>\u00a0<\/strong>That is after making peace with the throughput of the API, even if you are running async queries.<\/p>\n In applied ML, problems like these are generally tricky to solve since they don\u2019t fully satisfy the requirements of supervised learning or aren\u2019t cheap\/fast enough to be run via an LLM. This particular pain point is what the R.E.D algorithm addresses: semi-supervised learning, when the training data per class is not enough to build (quasi)traditional classifiers.<\/p>\n R.E.D: Recursive<\/a> Expert Delegation<\/strong>\u00a0is a novel framework that changes how we approach text classification. This is an applied ML paradigm \u2014 i.e., there is no\u00a0fundamentally different<\/em>\u00a0architecture to what exists, but its a highlight reel of ideas that work best to build something that is practical and scalable.<\/p>\n In this post, we will be working through a specific example where we have a large number of text classes (100\u20131000), each class only has few samples (30\u2013100), and there are a non-trivial number of samples to classify (10,000\u2013100,000). We approach this as a\u00a0semi-supervised learning<\/strong>\u00a0problem via R.E.D.<\/p>\n Let\u2019s dive in.<\/p>\n Instead of having a single classifier classify between a large number of classes, R.E.D. intelligently:<\/p>\n The intuition behind it is not very hard to grasp:\u00a0Active Learning<\/a>\u00a0employs humans as domain experts to consistently \u2018correct\u2019 or \u2018validate\u2019 the outputs from an ML model, with continuous training. This stops when the model achieves acceptable performance. We intuit and rebrand the same, with a few clever innovations that will be detailed in a research pre-print later.<\/p>\n Let\u2019s take a deeper look\u2026<\/p>\n When the number of input labels (classes) is high, the complexity of learning a linear decision boundary between classes increases. As such, the quality of the classifier deteriorates as the number of classes increases. This is especially true when the classifier does not have enough\u00a0samples<\/strong>\u00a0to learn from \u2014 i.e. each of the training classes has only a few samples.<\/p>\n This is very reflective of a real-world scenario, and the primary motivation behind the creation of R.E.D.<\/p>\n Some ways of improving a classifier\u2019s performance under these constraints:<\/p>\n Greedy Subset Selection does exactly this \u2014 since the scope of the problem is Text Classification<\/a>, we form embeddings of the training labels, reduce their dimensionality via UMAP, then form\u00a0S<\/em><\/strong>\u00a0subsets from them. Each of the\u00a0S\u00a0<\/em><\/strong>subsets has elements as\u00a0n\u00a0<\/em><\/strong>training labels. We pick training labels greedily, ensuring that every label we pick for the subset is the most dissimilar label w.r.t. the other labels that exist in the subset:<\/p>\n the result of this greedy subset sampling is all the training labels clearly boxed into subsets, where each subset has at most only\u00a0n\u00a0<\/em><\/strong>classes. This inherently makes the job of a classifier easier, compared to the original\u00a0S\u00a0<\/em><\/strong>classes it would have to classify between otherwise!<\/p>\n Cascade this after the initial label subset formation \u2014 i.e., this classifier is only classifying between a given\u00a0subset\u00a0<\/strong>of classes.<\/p>\n Picture this: when you have low amounts of training data, you absolutely cannot create a hold-out set that is meaningful for evaluation. Should you do it at all? How do you know if your classifier is working well?<\/p>\n We approached this problem slightly differently \u2014 we defined the fundamental job of a semi-supervised classifier to be\u00a0pre-emptive<\/strong> classification of a sample. This means that regardless of what a sample gets classified as it will be \u2018verified\u2019 and \u2018corrected\u2019 at a later stage: this classifier only needs to identify what needs to be verified.<\/p>\n As such, we created a design for how it would treat its data:<\/p>\n Oversampling on noise is a faux-safety measure, to ensure that adjacent data that belongs to another class is most likely predicted as noise instead of slipping through for verification.<\/p>\n How do you check if this classifier is working well \u2014 in our experiments, we define this as the number of \u2018uncertain\u2019 samples in a classifier\u2019s prediction. Using uncertainty sampling and information gain principles, we were effectively able to gauge if a classifier is \u2018learning\u2019 or not, which acts as a pointer towards classification performance. This classifier is consistently retrained unless there is an inflection point in the number of uncertain samples predicted, or there is only a delta of information being added iteratively by new samples.<\/p>\n This is the heart of the approach \u2014 using an LLM as a proxy for a human validator. The human validator approach we are talking about is Active Labelling<\/p>\n Let\u2019s get an intuitive understanding of Active Labelling:<\/p>\n For Active Labelling to work, there are expectations involved for an SME:<\/p>\n Given these expectations and intuitions, we can \u2018mimic\u2019 these using an LLM:<\/p>\n The result? A cost-effective framework that relies on a fast, cheap classifier to make pre-emptive classifications, and an LLM that verifies these using (meaning of the label + dynamically sourced training samples that are similar to the current classification):<\/p>\n By feeding the valid samples (voted_data) to our classifier under controlled parameters, we achieve the \u2018recursive\u2019 part of our algorithm:<\/p>\n By doing this, we were able to achieve close-to-human-expert validation numbers on controlled multi-class datasets. Experimentally, R.E.D. scales up to\u00a01,000 classes while maintaining a competent degree of accuracy<\/strong>\u00a0almost on par with human experts (90%+ agreement).<\/p>\n I believe this is a significant achievement in applied ML, and has real-world uses for production-grade expectations of cost, speed, scale, and adaptability. The technical report, publishing later this year, highlights relevant code samples as well as experimental setups used to achieve given results.<\/p>\n All images, unless otherwise noted, are by the author<\/em><\/p>\n Interested in more details? Reach out to me over Medium<\/a> or email for a chat!<\/p>\n \n The post R.E.D.: Scaling Text Classification with Expert Delegation<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\nThe classification conundrum<\/h2>\n
\n
The R.E.D. algorithm<\/h2>\n
How it works<\/h2>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/03\/RED-1.png?ssl=1\")
\n
Greedy subset selection with least similar elements<\/h3>\n
\n
import numpy as np\nfrom sklearn.metrics.pairwise import cosine_similarity\n\n\ndef avg_embedding(candidate_embeddings):\n return np.mean(candidate_embeddings, axis=0)\n\ndef get_least_similar_embedding(target_embedding, candidate_embeddings):\n similarities = cosine_similarity(target_embedding, candidate_embeddings)\n least_similar_index = np.argmin(similarities) # Use argmin to find the index of the minimum\n least_similar_element = candidate_embeddings[least_similar_index]\n return least_similar_element\n\n\ndef get_embedding_class(embedding, embedding_map):\n reverse_embedding_map = {value: key for key, value in embedding_map.items()}\n return reverse_embedding_map.get(embedding) # Use .get() to handle missing keys gracefully\n\n\ndef select_subsets(embeddings, n):\n visited = {cls: False for cls in embeddings.keys()}\n subsets = []\n current_subset = []\n\n while any(not visited[cls] for cls in visited):\n for cls, average_embedding in embeddings.items():\n if not current_subset:\n current_subset.append(average_embedding)\n visited[cls] = True\n elif len(current_subset) >= n:\n subsets.append(current_subset.copy())\n current_subset = []\n else:\n subset_average = avg_embedding(current_subset)\n remaining_embeddings = [emb for cls_, emb in embeddings.items() if not visited[cls_]]\n if not remaining_embeddings:\n break # handle edge case\n \n least_similar = get_least_similar_embedding(target_embedding=subset_average, candidate_embeddings=remaining_embeddings)\n\n visited_class = get_embedding_class(least_similar, embeddings)\n\n \n if visited_class is not None:\n visited[visited_class] = True\n\n\n current_subset.append(least_similar)\n \n if current_subset: # Add any remaining elements in current_subset\n subsets.append(current_subset)\n \n\n return subsets<\/code><\/pre>\nSemi-supervised classification with noise oversampling<\/h3>\n
\n
Proxy active learning via an LLM agent<\/h3>\n
\n
\n
\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/03\/RED-2-1024x232.webp?resize=1024%2C232&ssl=1\")
\n
import math\n\ndef calculate_uncertainty(clf, sample):\n predicted_probabilities = clf.predict_proba(sample.reshape(1, -1))[0] # Reshape sample for predict_proba\n uncertainty = -sum(p * math.log(p, 2) for p in predicted_probabilities)\n return uncertainty\n\n\ndef select_informative_samples(clf, data, k):\n informative_samples = []\n uncertainties = [calculate_uncertainty(clf, sample) for sample in data]\n\n # Sort data by descending order of uncertainty\n sorted_data = sorted(zip(data, uncertainties), key=lambda x: x[1], reverse=True)\n\n # Get top k samples with highest uncertainty\n for sample, uncertainty in sorted_data[:k]:\n informative_samples.append(sample)\n\n return informative_samples\n\n\ndef proxy_label(clf, llm_judge, k, testing_data):\n #llm_judge - any LLM with a system prompt tuned for verifying if a sample belongs to a class. Expected output is a bool : True or False. True verifies the original classification, False refutes it\n predicted_classes = clf.predict(testing_data)\n\n # Select k most informative samples using uncertainty sampling\n informative_samples = select_informative_samples(clf, testing_data, k)\n\n # List to store correct samples\n voted_data = []\n\n # Evaluate informative samples with the LLM judge\n for sample in informative_samples:\n sample_index = testing_data.tolist().index(sample.tolist()) # changed from testing_data.index(sample) because of numpy array type issue\n predicted_class = predicted_classes[sample_index]\n\n # Check if LLM judge agrees with the prediction\n if llm_judge(sample, predicted_class):\n # If correct, add the sample to voted data\n voted_data.append(sample)\n\n # Return the list of correct samples with proxy labels\n return voted_data<\/code><\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/03\/RED-3.png?ssl=1\")