A launch pad for projects with small\u00a0datasets<\/h4>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2Az8V20Ww4ZGw4NpytI7DhSA.jpeg?ssl=1\")
Photo by Google DeepMind: https:\/\/www.pexels.com\/photo\/an-artist-s-illustration-of-artificial-intelligence-ai-this-image-depicts-how-ai-can-help-humans-to-understand-the-complexity-of-biology-it-was-created-by-artist-khyati-trehan-as-part-17484975\/<\/a><\/figcaption><\/figure>\nMachine Learning (ML) has driven remarkable breakthroughs in computer vision, natural language processing, and speech recognition, largely due to the abundance of data in these fields. However, many challenges\u200a\u2014\u200aespecially those tied to specific product features or scientific research\u200a\u2014\u200asuffer from limited data quality and quantity. This guide provides a roadmap for tackling small data problems based on your data constraints, and offers potential solutions, guiding your decision making early\u00a0on.<\/p>\n
1. Difficulty with Gathering more\u00a0Data<\/h3>\n
Raw data is rarely a blocker for ML projects. High-quality labels on the other hand, are often prohibitively expensive and laborious to collect. Where obtaining an expert-labelled \u201cground truth\u201d requires domain expertise, intensive fieldwork, or specialised knowledge. For instance, your problem might focus on rare events, maybe, endangered species monitoring, extreme climate events, or unusual manufacturing defects. Other times, business specific or scientific questions might be too specialised for off-the-shelf large-scale datasets. Ultimately this means many projects fail because label acquisition is too expensive.<\/p>\n
2. Core Challenges of Small\u00a0Datasets<\/h3>\n
With only a small dataset, any new project starts off with inherent risks. How much of the true variability does your dataset capture? In many ways this question is unanswerable the smaller your dataset gets. Making testing and validation increasingly difficult, and leaving a great deal of uncertainty about how well your model actually generalises. Your model doesn\u2019t know what your data doesn\u2019t capture. This means, with potentially only a few hundred samples, both the richness of the features you can extract, and the number of features you can use decreases, without significant risk of overfitting (that in many cases you can\u2019t measure). This often leaves you limited to classical ML algorithms (Random Forest, SVM etc\u2026), or heavily regularised deep learning methods. The presence of class imbalance will only exacerbate your problems. Making small datasets even more sensitive to noise, where only a few incorrect labels or faulty measurements will cause havoc and headaches.<\/p>\n
3. Questions to Get you\u00a0Started<\/h3>\n
For me, working the problem starts with asking a few simple questions about the data, labelling process, and end goals. By framing your problem with a \u201cchecklist\u201d, we can clarify the constraints of your data. Have a go at answering the questions below:<\/p>\n
Is your dataset fully, partially, or mostly unlabelled?<\/strong><\/p>\n\n- \nFully labeled<\/strong>: You have labels for (nearly) all samples in your\u00a0dataset.<\/li>\n
- \nPartially labelled<\/strong>: A portion of the dataset has labels, but there\u2019s a large portion of unlabelled data.<\/li>\n
- \nMostly unlabelled<\/strong>: You have very few (or no) labeled data\u00a0points.<\/li>\n<\/ul>\n
How reliable are the labels you do\u00a0have?<\/strong><\/p>\n\n- \nHighly reliable<\/strong>: If multiple annotates agree on labels, or they are confirmed by trusted experts or well-established protocols.<\/li>\n
- \nNoisy or weak<\/strong>: Labels may be crowd-sourced, generated automatically, or prone to human or sensor\u00a0error.<\/li>\n<\/ul>\n
Are you solving one problem, or do you have multiple (related) tasks?<\/strong><\/p>\n\n- \nSingle-task<\/strong>: A singular objective, such as a binary classification or a single regression target.<\/li>\n
- \nMulti-task<\/strong>: Multiple outputs or multiple objectives.<\/li>\n<\/ul>\n
Are you dealing with rare events or heavily imbalanced classes?<\/strong><\/p>\n\n- \nYes<\/strong>: Positive examples are very scarce (e.g., \u201cequipment failure,\u201d \u201cadverse drug reactions,\u201d or \u201cfinancial fraud\u201d).<\/li>\n
- \nNo<\/strong>: Classes are somewhat balanced, or your task doesn\u2019t involve highly skewed distributions.<\/li>\n<\/ul>\n
Do you have expert knowledge available, and if so, in what\u00a0form?<\/strong><\/p>\n\n- \nHuman experts<\/strong>: You can periodically query domain experts to label new data or verify predictions.<\/li>\n
- \nModel-based experts<\/strong>: You have access to well-established simulation or physical models (e.g., fluid dynamics, chemical kinetics) that can inform or constrain your ML\u00a0model.<\/li>\n
- \nNo<\/strong>: No relevant domain expertise available to guide or correct the\u00a0model.<\/li>\n<\/ul>\n
Is labelling new data possible, and at what\u00a0cost?<\/strong><\/p>\n\n- \nFeasible and affordable<\/strong>: You can acquire more labeled examples if necessary.<\/li>\n
- \nDifficult or expensive<\/strong>: Labelling is time-intensive, costly, or requires specialised domain knowledge (e.g., medical diagnosis, advanced scientific measurements).<\/li>\n<\/ul>\n
Do you have prior knowledge or access to pre-trained models relevant to your\u00a0data?<\/strong><\/p>\n\n- \nYes<\/strong>: There exist large-scale models or datasets in your domain (e.g., ImageNet for images, BERT for\u00a0text).<\/li>\n
- \nNo<\/strong>: Your domain is niche or specialised, and there aren\u2019t obvious pre-trained resources.<\/li>\n<\/ul>\n
4. Matching Questions to Techniques<\/h3>\n
With your answers to the questions above ready, we can move towards establishing a list of potential techniques for tackling your problem. In practice, small dataset problems require hyper-nuanced experimentation, and so before implementing the techniques below give yourself a solid foundation by starting with a simple model, get a full pipeline working as quickly as possible and always cross-validate. This gives you a baseline to iteratively apply new techniques based on your error analysis, while focusing on conducting small scale experiments. This also helps avoid building an overly complicated pipeline that\u2019s never properly validated. With a baseline in place, chances are your dataset will evolve rapidly. Tools like DVC or MLflow help track dataset versions and ensure reproducibility. In a small-data scenario, even a handful of new labeled examples can significantly change model performance\u200a\u2014\u200aversion control helps systematically manage\u00a0that.<\/p>\n
With that in mind, here\u2019s how your answers to the questions above point towards specific strategies described later in this\u00a0post:<\/p>\n
Fully Labeled + Single Task + Sufficiently Reliable\u00a0Labels:<\/strong><\/p>\n\n- \nData Augmentation<\/strong> (Section 5.7) to increase effective sample\u00a0size.<\/li>\n
- \nEnsemble Methods<\/strong> (Section 5.9) if you can afford multiple model training\u00a0cycles.<\/li>\n
- \nTransfer Learning<\/strong> (Section 5.1) if a pre-trained model in your domain (or a related domain) is available.<\/li>\n<\/ul>\n
Partially Labeled + Labelling is Reliable or Achievable:<\/strong><\/p>\n\n- \nSemi-Supervised Learning<\/strong> (Section 5) to leverage a larger pool of unlabelled data.<\/li>\n
- \nActive Learning<\/strong> (Section 5.6) if you have a human expert who can label the most informative samples.<\/li>\n
- \nData Augmentation<\/strong> (Section 5.7) where possible.<\/li>\n<\/ul>\n
Rarely Labeled or Mostly Unlabelled + Expert Knowledge Available:<\/strong><\/p>\n\n- \nActive Learning<\/strong> (Section 5.6) to selectively query an expert (especially if the expert is a\u00a0person).<\/li>\n
- \nProcess-Aware (Hybrid) Models<\/strong> (Section 5.10) if your \u201cexpert\u201d is a well-established simulation or\u00a0model.<\/li>\n<\/ul>\n
Rarely Labeled or Mostly Unlabelled + No Expert \/ No Additional Labels:<\/strong><\/p>\n\n- \nSelf-Supervised Learning<\/strong> (Section 5.2) to exploit inherent structure in unlabelled data.<\/li>\n
- \nFew-Shot or Zero-Shot Learning<\/strong> (Section 5.4) if you can rely on meta-learning or textual descriptions to handle novel\u00a0classes.<\/li>\n
- \nWeakly Supervised Learning<\/strong> (Section 5.5) if your labels exist but are imprecise or high-level.<\/li>\n<\/ul>\n
Multiple Related\u00a0Tasks:<\/strong><\/p>\n\n- \nMultitask Learning<\/strong> (Section 5.8) to share representations between tasks, effectively pooling \u201csignal\u201d across the entire\u00a0dataset.<\/li>\n<\/ul>\n
Dealing with Noisy or Weak\u00a0Labels:<\/strong><\/p>\n\n- \nWeakly Supervised Learning<\/strong> (Section 5.5) which explicitly handles label\u00a0noise.<\/li>\n
- Combine with Active Learning<\/strong> or a small \u201cgold standard\u201d subset to clean up the worst labelling errors.<\/li>\n<\/ul>\n
Highly Imbalanced \/ Rare\u00a0Events:<\/strong><\/p>\n\n- \nData Augmentation<\/strong> (Section 5.7) targeting minority classes (e.g., synthetic minority oversampling).<\/li>\n
- \nActive Learning<\/strong> (Section 5.6) to specifically label more of the rare\u00a0cases.<\/li>\n
- \nProcess-Aware Models<\/strong> (Section 5.10) or domain expertise to confirm rare cases, if possible.<\/li>\n<\/ul>\n
Have a Pre-Trained Model or Domain-Specific Knowledge:<\/strong><\/p>\n\n- \nTransfer Learning<\/strong> (Section 5.1) is often the quickest\u00a0win.<\/li>\n
- \nProcess-Aware Models<\/strong> (Section 5.10) if combining your domain knowledge with ML can reduce data requirements.<\/li>\n<\/ul>\n
5. Strategies for Tackling Small\u00a0Data<\/h3>\n
Hopefully, the above has provided a starting point for solving your small data problem. It\u2019s worth noting that many of the techniques discussed are complex and resource intensive. So keep in mind you\u2019ll likely need to get buy-in from your team and project managers before starting. This is best done through clear, concise communication of the potential value they might provide. Frame experiments as strategic, foundational work that can be reused, refined, and leveraged for future projects. Focus on demonstrating clear, measurable impact from a short, tightly-scoped pilot.<\/p>\n
Despite the relatively simple picture painted of each technique below, it\u2019s important to keep in mind there\u2019s no one-size-fits-all solution, and applying these techniques isn\u2019t like stacking lego bricks, nor do they work out-of-the-box. To get you started I\u2019ve provided a brief overview of each technique, this is by no means exhaustive, but looks to offer a starting point for your own research.<\/p>\n
5.1 Transfer\u00a0Learning<\/h3>\n
Transfer learning is about reusing existing models to solve new related problems. By starting with pre-trained weights, you leverage representations learned from large, diverse datasets and fine-tune the model on your smaller, target\u00a0dataset.<\/p>\n
Why it\u00a0helps:<\/strong><\/p>\n\n- Leverages powerful features learnt from larger, often diverse datasets.<\/li>\n
- Fine-tuning pre-trained models typically leads to higher accuracy, even with limited samples, while reducing training\u00a0time.<\/li>\n
- Ideal when compute resources or project timelines prevent training a model from\u00a0scratch.<\/li>\n<\/ul>\n
Tips:<\/strong><\/p>\n\n- Select a model aligned with your problem domain or a large general-purpose \u201cfoundation model\u201d like Mistral (language) or CLIP\/SAM (vision), accessible on platforms like Hugging Face. These models often outperform domain-specific pre-trained models due to their general-purpose capabilities.<\/li>\n
- Freeze layers that capture general features while fine-tuning only a few layers on\u00a0top.<\/li>\n
- To counter the risk of overfitting to your small datasets try pruning. Here, less important weights or connections are removed reducing the number of trainable parameters and increasing inference speed.<\/li>\n
- If interpretability is required, large black-box models may not be\u00a0ideal.<\/li>\n
- Without access to the pre-trained models source dataset, you risk reinforcing sampling biases during fine-tuning.<\/li>\n<\/ul>\n
A nice example of transfer learning is described in the following paper<\/a>. Where leveraging a pre-trained ResNet model enabled better classification of chest X-ray images and detecting COVID-19. Supported by the use of dropout and batch normalisation, the researchers froze the initial layers of the ResNet base model, while fine-tuning later layers, capturing task-specific, high-level features. This proved to be a cost effective method for achieving high accuracy with a small\u00a0dataset.<\/p>\n5.2 Self-Supervised Learning<\/h3>\n
Self-supervised learning is a pre-training technique where artificial tasks (\u201cpretext tasks\u201d) are created to learn representations from broad unlabelled data. Examples include predicting masked tokens for text or rotation prediction, colorisation for images. The result is general-purpose representations you can later pair with transfer-learning (section 5.1) or semi-supervised (section 5) and fine-tune with your smaller\u00a0dataset.<\/p>\n
Why it\u00a0helps:<\/strong><\/p>\n\n- Pre-trained models serve as a strong initialisation point, reducing the risk of future overfitting.<\/li>\n
- Learns to represent data in a way that captures intrinsic patterns and structures (e.g., spatial, temporal, or semantic relationships), making them more effective for downstream tasks.<\/li>\n<\/ul>\n
Tips:<\/strong><\/p>\n\n- Pre-tasks like cropping, rotation, colour jitter, or noise injection are excellent for visual tasks. However it\u2019s a balance, as excessive augmentation can distort the distribution of small\u00a0data.<\/li>\n
- Ensure unlabelled data is representative of the small dataset\u2019s distribution to help the model learn features that generalise well.<\/li>\n
- Self-supervised methods can be compute-intensive; often requiring enough unlabelled data to truly benefit and a large computation budget.<\/li>\n<\/ul>\n
LEGAL-BERT<\/a> is a prominent example of self-supervised learning. Legal-BERT is a domain-specific variant of the BERT language model, pre-trained on a large dataset of legal documents to improve its understanding of legal language, terminology, and context. The key, is the use of unlabelled data, where techniques such as masked language modelling (the model learns to predict masked words) and next sentence prediction (learning the relationships between sentences, and determining if one follows another) removes the requirement for labelling. This text embedding model can then be used for more specific legal based ML\u00a0tasks.<\/p>\n5.3 Semi-Supervised Learning<\/h3>\n
Leverages a small labeled dataset in addition to a larger unlabelled set. The model iteratively refines predictions on unlabelled data, to generate task specific predictions that can be used as \u201cpseudo-labels\u201d for further iterations.<\/p>\n
Why it\u00a0helps:<\/strong><\/p>\n\n- Labeled data guides the task-specific objective, while the unlabelled data is used to improve generalisation (e.g., through pseudo-labelling, consistency regularisation, or other techniques).<\/li>\n
- Improves decision boundaries and can boost generalisation.<\/li>\n<\/ul>\n
Tips:<\/strong><\/p>\n\n- Consistency regularisation is a method that assumes model predictions should be consistent across small perturbations (noise, augmentations) made to unlabelled data. The idea is to \u201csmooth\u201d the decision boundary of sparsely populated high-dimensional space.<\/li>\n
- Pseudo-labelling allows you to train an initial model with a small dataset and use future predictions on unlabelled data as \u201cpseudo\u201d labels for future training. With the aim of generalising better and reducing overfitting.<\/li>\n<\/ul>\n
Financial fraud detection is a problem that naturally lends itself to semi-supervised learning, with very little real labelled data (confirmed fraud cases) and a large set of unlabelled transaction data. The following paper<\/a> proposes a neat solution, by modelling transactions, users, and devices as nodes in a graph, where edges represent relationships, such as shared accounts or devices. The small set of labeled fraudulent data is then used to train the model by propagating fraud signals across the graph to the unlabelled nodes. For example, if a fraudulent transaction (labeled node) is linked to multiple unlabelled nodes (e.g., related users or devices), the model learns patterns and connections that might indicate\u00a0fraud.<\/p>\n5.4 Few-Shot and Zero-Shot Learning<\/h3>\n
Few and zero-shot learning refers to a broad collection of techniques designed to tackle very small datasets head on. Generally these methods train a model to identify \u201cnovel\u201d classes unseen during training, with a small labelled dataset used primarily for\u00a0testing.<\/p>\n
Why it\u00a0helps:<\/strong><\/p>\n\n- These approaches enable models to quickly adapt to new tasks or classes without extensive retraining.<\/li>\n
- Useful for domains with rare or unique categories, such as rare diseases or niche object detection.<\/li>\n<\/ul>\n
Tips:<\/strong><\/p>\n\n- Probably the most common technique, known as similarity-based learning, trains a model to compare pairs of items and decide if they belong to the same class. By learning a similarity or distance measure the model can generalise to unseen classes by comparing new instances to class prototypes (your small set of labelled data during testing) during testing. This approach requires a good way to represent different types of input (embedding), often created using Siamese neural networks or similar\u00a0models.<\/li>\n
- Optimisation-based meta-learning, aims to train a model to quickly adapt to new tasks or classes using only a small amount of training data. A popular example is model-agnostic meta-learning (MAML). Where a \u201cmeta-learner\u201d is trained on many small tasks, each with its own training and testing examples. The goal is to teach the model to start from a good initial state, so when it encounters a new task, it can quickly learn and adjust with minimal additional training. These are not simple methods to implement.<\/li>\n
- A more classical technique, one-class classification, is where a binary classifier (like one class SVM) is trained on data from only one class, and learns to detect outliers during\u00a0testing.<\/li>\n
- Zero-shot approaches, such as CLIP or large language models with prompt engineering, enable classification or detection of unseen categories using textual cues (e.g., \u201ca photo of a new product\u00a0type\u201d).<\/li>\n
- In zero-shot cases, combine with active learning (human in the loop) to label the most informative examples.<\/li>\n<\/ul>\n
It\u2019s important to maintain realistic expectations when implementing few-shot and zero-shot techniques. Often, the aim is to achieve usable or \u201cgood enough\u201d performance. As a direct comparison of traditional deep-learning (DL) methods, the following study<\/a> compares both DL and few-shot learning (FSL) for classifying 20 coral reef fish species from underwater images with applications for detecting rare species with limited available data. It should come as no surprise that the best model tested was a DL model based on ResNet. With ~3500 examples for each species the model achieved an accuracy of 78%. However, collecting this volume of data for rare species is beyond practical. Subsequently, the number of samples was reduced to 315 per species, and the accuracy dropped to 42%. In contrast, the FSL model, achieved comparable results with as few as 5 labeled images per species, and better performance beyond 10 shots. Here, the Reptile algorithm was used, which is a meta-learning-based FSL approach. This was trained by repeatedly solving small classification problems (e.g., distinguishing a few classes) drawn from the MiniImageNet dataset (a useful benchmark dataset for FSL). During fine-tuning, the model was then trained using a few labeled examples (1 to 30 shots per species).<\/p>\n5.5 Weakly Supervised Learning<\/h3>\n
Weakly supervised learning describes a set of techniques for building models with noisy, inaccurate or restricted sources to label large quantities of data. We can split the topic into three: incomplete, inexact, and inaccurate supervision, distinguished by the confidence in the labels. Incomplete supervision occurs when only a subset of examples has ground-truth labels. Inexact supervision involves coarsely-grained labels, like labelling an MRI image as \u201clung cancer\u201d without specifying detailed attributes. Inaccurate supervision arises when labels are biased or incorrect due to\u00a0human.<\/p>\n
Why it\u00a0helps:<\/strong><\/p>\n\n- Partial or inaccurate data is often simpler and cheaper to get hold\u00a0of.<\/li>\n
- Enables models to learn from a larger pool of information without the need for extensive manual labelling.<\/li>\n
- Focuses on extracting meaningful patterns or features from data, that can amplify the value of any existing well labeled examples.<\/li>\n<\/ul>\n
Tips:<\/strong><\/p>\n\n- Use a small subset of high-quality labels (or an ensemble) to correct systematic labelling errors.<\/li>\n
- For scenarios where coarse-grained labels are available (e.g., image-level labels but not detailed instance-level labels), Multi-instance learning can be employed. Focusing on bag-level classification since instance-level inaccuracies are less impactful.<\/li>\n
- Label filtering, correction, and inference techniques can mitigate label noise and minimise reliance on expensive manual\u00a0labels.<\/li>\n<\/ul>\n
The primary goal of this technique is to estimate more informative or higher dimensional data with limited information. As an example, this paper<\/a> presents a weakly supervised learning approach to estimating a 3D human poses. The method relies on 2D pose annotations, avoiding the need for expensive 3D ground-truth data. Using an adversarial reprojection network (RepNet), the model predicts 3D poses and reprojects them into 2D views to compare with 2D annotations, minimising reprojection error. This approach leverages adversarial training to enforce plausibility of 3D poses and showcases the potential of weakly supervised methods for complex tasks like 3D pose estimation with limited labeled\u00a0data.<\/p>\n5.6 Active\u00a0Learning<\/h3>\n
Active learning seeks to optimise labelling efforts by identifying unlabelled samples that, once labeled, will provide the model with the most informative data. A common approach is uncertainty sampling, which selects samples where the model\u2019s predictions are least certain. This uncertainty is often quantified using measures such as entropy or margin sampling. This is highly iterative; each round influences the model\u2019s next set of predictions.<\/p>\n
Why it\u00a0helps:<\/strong><\/p>\n\n- Optimises expert time; you label fewer samples\u00a0overall.<\/li>\n
- Quickly identifies edge cases that improve model robustness.<\/li>\n<\/ul>\n
Tips:<\/strong><\/p>\n\n- Diversity sampling is an alternative selection approach that focuses on diverse area of the feature space. For instance, clustering can be used to select a few representative samples from each\u00a0cluster.<\/li>\n
- Try to use multiple selection methods to avoid introducing bias.<\/li>\n
- Introducing an expert human in the loop can be logistically difficult, managing availability with a labelling workflow that can be slow\/expensive.<\/li>\n<\/ul>\n
This technique has been extensively used in chemical analysis and materials research. Where, large databases of real and simulated molecular structures and their properties have been collected over decades. These databases are particularly useful for drug discovery, where simulations like docking are used to predict how small molecules (e.g., potential drugs) interact with targets such as proteins or enzymes. However, the computational cost of performing these types of calculations over millions of molecules makes brute force studies impractical. This is where active learning comes in. One such study<\/a> showed that by training a predictive model on an initial subset of docking results and iteratively selecting the most uncertain molecules for further simulations, researchers were able to drastically reduce the number of molecules tested while still identifying the best candidates.<\/p>\n5.7 Data Augmentation<\/h3>\n
Artificially increase your dataset by applying transformations to existing examples\u200a\u2014\u200asuch as flipping or cropping images, translation or synonym replacement for text and time shifts or random cropping for time-series. Alternatively, upsample underrepresented data with ADASYN (Adaptive Synthetic Sampling) and SMOTE (Synthetic Minority Over-sampling Technique).<\/p>\n
Why it\u00a0helps:<\/strong><\/p>\n\n- The model focuses on more general and meaningful features rather than specific details tied to the training\u00a0set.<\/li>\n
- Instead of collecting and labelling more data, augmentation provides a cost-effective alternative.<\/li>\n
- Improves generalisation by increasing the diversity of training data, helping learn robust and invariant features rather than overfitting to specific patterns.<\/li>\n<\/ul>\n
Tips:<\/strong><\/p>\n\n- Keep transformations domain-relevant (e.g., flipping images vertically might make sense for flower images, less so for medical\u00a0X-rays).<\/li>\n
- Pay attention that any augmentations don\u2019t distort the original data distribution, preserving the underlying patterns.<\/li>\n
- Explore GANs, VAEs, or diffusion models to produce synthetic data\u200a\u2014\u200abut this often requires careful tuning, domain-aware constraints, and enough initial\u00a0data.<\/li>\n
- Synthetic oversampling (like SMOTE) can introduce noise or spurious correlations if the classes or feature space are complex and not well understood.<\/li>\n<\/ul>\n
Data augmentation is an incredibly broad topic, with numerous surveys exploring the current state-of-the-art across various fields, including computer vision (review paper<\/a>), natural language processing (review paper<\/a>), and time-series data (review paper<\/a>). It has become an integral component of most machine learning pipelines due to its ability to enhance model generalisation. This is particularly critical for small datasets, where augmenting input data by introducing variations, such as transformations or noise, and removing redundant or irrelevant features can significantly improve a model\u2019s robustness and performance.<\/p>\n5.8 Multitask Learning<\/h3>\n
Here we train one model to solve several tasks simultaneously. This improves how well models perform by encouraging them to find patterns or solutions that work well for multiple goals at the same time. Lower layers capture general features that benefit all tasks, even if you have limited data for\u00a0some.<\/p>\n
Why it\u00a0helps:<\/strong><\/p>\n\n- Shared representations are learned across tasks, effectively increasing sample\u00a0size.<\/li>\n
- The model is less likely to overfit, as it must account for patterns relevant to all tasks, not just\u00a0one.<\/li>\n
- Knowledge learned from one task can provide insights that improve performance on\u00a0another.<\/li>\n<\/ul>\n
Tips:<\/strong><\/p>\n\n- Tasks need some overlap or synergy to meaningfully share representations; otherwise this method will hurt performance.<\/li>\n
- Adjust per-task weights carefully to avoid letting one task dominate training.<\/li>\n<\/ul>\n
The scarcity of data for many practical applications of ML makes sharing both data and models across tasks an attractive proposition. This is enabled by Multitask learning, where tasks benefit from shared knowledge and correlations in overlapping domains. However, it requires a large, diverse dataset that integrates multiple related properties. Polymer design is one example<\/a> where this has been successful. Here, a hybrid dataset of 36 properties across 13,000 polymers, covering a mix of mechanical, thermal, and chemical characteristics, was used to train a deep-learning-based MTL architecture. The multitask model outperformed single-task models for every polymer property. Particularly, for underrepresented properties.<\/p>\n5.9 Ensemble\u00a0Learning<\/h3>\n
Ensembles aggregate predictions from several base models to improve robustness. Generally, ML algorithms can be limited in a variety of ways: high variance, high bias, and low accuracy. This manifests as different uncertainty distributions for different models across predictions. Ensemble methods limit the variance and bias errors associated with a single model; for example, bagging reduces variance without increasing the bias, while boosting reduces\u00a0bias.<\/p>\n
Why it\u00a0helps:<\/strong><\/p>\n\n- Diversifies \u201copinions\u201d across different model architectures.<\/li>\n
- Reduces variance, mitigating overfitting risk.<\/li>\n<\/ul>\n
Tips:<\/strong><\/p>\n\n- Avoid complex base models which can easily overfit small datasets. Instead, use regularised models such as shallow trees or linear models with added constraints to control complexity.<\/li>\n
- Bootstrap aggregating (bagging) methods like Random Forest can be particularly useful for small datasets. By training multiple models on bootstrapped subsets of the data, you can reduce overfitting while increasing robustness. This is effective for algorithms prone to high variance, such as decision\u00a0trees.<\/li>\n
- Combine different base models types (e.g., SVM, tree-based models, and logistic regression) with a simple meta-model like logistic regression to combine predictions.<\/li>\n<\/ul>\n
As an example, the following paper<\/a> highlights ensemble learning as a method to improve the classification of cervical cytology images. In this case, three pre-trained neural networks\u200a\u2014\u200aInception v3, Xception, and DenseNet-169\u200a\u2014\u200awere used. The diversity of these base models ensured the ensemble benefits from each models unique strengths and feature extraction capabilities. This combined with the fusion of model confidences, via a method that rewards confident, accurate predictions while penalising uncertain ones, maximised the utility of the limited data. Combined with transfer learning, the final predictions were robust to the errors of any particular model, despite the small dataset\u00a0used.<\/p>\n5.10 Process-Aware (Hybrid)\u00a0Models<\/h3>\n
Integrate domain-specific knowledge or physics-based constraints into ML models. This embeds prior knowledge, reducing the model\u2019s reliance on large data to infer patterns. For example, using partial differential equations alongside neural networks for fluid dynamics.<\/p>\n
Why it\u00a0helps:<\/strong><\/p>\n\n- Reduces the data needed to learn patterns that are already well understood.<\/li>\n
- Acts as a form of regularisation, guiding the model to plausible solutions even when the data is sparse or\u00a0noisy.<\/li>\n
- Improves interpretability and trust in domain-critical contexts.<\/li>\n<\/ul>\n
Tips:<\/strong><\/p>\n\n- Continually verify that model outputs make physical\/biological sense, not just numerical sense.<\/li>\n
- Keep domain constraints separate but feed them as inputs or constraints in your model\u2019s loss function.<\/li>\n
- Be careful to balance domain-based constraints with your models ability to learn new phenomena.<\/li>\n
- In practice, bridging domain-specific knowledge with data-driven methods often involves serious collaboration, specialised code, or hardware.<\/li>\n<\/ul>\n
Constraining a model, in this way requires a deep understanding of your problem domain, and is often applied to problems where the environment the model operates in is well understood, such as physical systems. An example of this is lithium-ion battery modelling<\/a>, where domain knowledge of battery dynamics is integrated into the ML process. This allows the model to capture complex behaviours and uncertainties missed by traditional physical models, ensuring physically consistent predictions and improved performance under real-world conditions like battery\u00a0aging.<\/p>\n6. Conclusion<\/h3>\n
For me, projects constrained by limited data are some of the most interesting projects to work on\u200a\u2014\u200adespite the higher risk of failure, they offer an opportunity to explore the state-of-the-art and experiment. These are tough problems! However, systematically applying the strategies covered in this post can greatly improve your odds of delivering a robust, effective model. Embrace the iterative nature of these problems: refine labels, employ augmentations, and analyse errors in quick cycles. Short pilot experiments help validate each technique\u2019s impact before you invest\u00a0further.<\/p>\n
![\"\"](\"https:\/\/medium.com\/_\/stat?event=post.clientViewed&referrerSource=full_rss&postId=194492e7a6f8\")
<\/p>\n
\nWhere to Start When Data is Limited<\/a> was originally published in Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\n
Machine Learning (ML) has driven remarkable breakthroughs in computer vision, natural language processing, and speech recognition, largely due to the abundance of data in these fields. However, many challenges\u200a\u2014\u200aespecially those tied to specific product features or scientific research\u200a\u2014\u200asuffer from limited data quality and quantity. This guide provides a roadmap for tackling small data problems based on your data constraints, and offers potential solutions, guiding your decision making early\u00a0on.<\/p>\n
1. Difficulty with Gathering more\u00a0Data<\/h3>\n
Raw data is rarely a blocker for ML projects. High-quality labels on the other hand, are often prohibitively expensive and laborious to collect. Where obtaining an expert-labelled \u201cground truth\u201d requires domain expertise, intensive fieldwork, or specialised knowledge. For instance, your problem might focus on rare events, maybe, endangered species monitoring, extreme climate events, or unusual manufacturing defects. Other times, business specific or scientific questions might be too specialised for off-the-shelf large-scale datasets. Ultimately this means many projects fail because label acquisition is too expensive.<\/p>\n
2. Core Challenges of Small\u00a0Datasets<\/h3>\n
With only a small dataset, any new project starts off with inherent risks. How much of the true variability does your dataset capture? In many ways this question is unanswerable the smaller your dataset gets. Making testing and validation increasingly difficult, and leaving a great deal of uncertainty about how well your model actually generalises. Your model doesn\u2019t know what your data doesn\u2019t capture. This means, with potentially only a few hundred samples, both the richness of the features you can extract, and the number of features you can use decreases, without significant risk of overfitting (that in many cases you can\u2019t measure). This often leaves you limited to classical ML algorithms (Random Forest, SVM etc\u2026), or heavily regularised deep learning methods. The presence of class imbalance will only exacerbate your problems. Making small datasets even more sensitive to noise, where only a few incorrect labels or faulty measurements will cause havoc and headaches.<\/p>\n
3. Questions to Get you\u00a0Started<\/h3>\n
For me, working the problem starts with asking a few simple questions about the data, labelling process, and end goals. By framing your problem with a \u201cchecklist\u201d, we can clarify the constraints of your data. Have a go at answering the questions below:<\/p>\n
Is your dataset fully, partially, or mostly unlabelled?<\/strong><\/p>\n How reliable are the labels you do\u00a0have?<\/strong><\/p>\n Are you solving one problem, or do you have multiple (related) tasks?<\/strong><\/p>\n Are you dealing with rare events or heavily imbalanced classes?<\/strong><\/p>\n Do you have expert knowledge available, and if so, in what\u00a0form?<\/strong><\/p>\n Is labelling new data possible, and at what\u00a0cost?<\/strong><\/p>\n Do you have prior knowledge or access to pre-trained models relevant to your\u00a0data?<\/strong><\/p>\n With your answers to the questions above ready, we can move towards establishing a list of potential techniques for tackling your problem. In practice, small dataset problems require hyper-nuanced experimentation, and so before implementing the techniques below give yourself a solid foundation by starting with a simple model, get a full pipeline working as quickly as possible and always cross-validate. This gives you a baseline to iteratively apply new techniques based on your error analysis, while focusing on conducting small scale experiments. This also helps avoid building an overly complicated pipeline that\u2019s never properly validated. With a baseline in place, chances are your dataset will evolve rapidly. Tools like DVC or MLflow help track dataset versions and ensure reproducibility. In a small-data scenario, even a handful of new labeled examples can significantly change model performance\u200a\u2014\u200aversion control helps systematically manage\u00a0that.<\/p>\n With that in mind, here\u2019s how your answers to the questions above point towards specific strategies described later in this\u00a0post:<\/p>\n Fully Labeled + Single Task + Sufficiently Reliable\u00a0Labels:<\/strong><\/p>\n Partially Labeled + Labelling is Reliable or Achievable:<\/strong><\/p>\n Rarely Labeled or Mostly Unlabelled + Expert Knowledge Available:<\/strong><\/p>\n Rarely Labeled or Mostly Unlabelled + No Expert \/ No Additional Labels:<\/strong><\/p>\n Multiple Related\u00a0Tasks:<\/strong><\/p>\n Dealing with Noisy or Weak\u00a0Labels:<\/strong><\/p>\n Highly Imbalanced \/ Rare\u00a0Events:<\/strong><\/p>\n Have a Pre-Trained Model or Domain-Specific Knowledge:<\/strong><\/p>\n Hopefully, the above has provided a starting point for solving your small data problem. It\u2019s worth noting that many of the techniques discussed are complex and resource intensive. So keep in mind you\u2019ll likely need to get buy-in from your team and project managers before starting. This is best done through clear, concise communication of the potential value they might provide. Frame experiments as strategic, foundational work that can be reused, refined, and leveraged for future projects. Focus on demonstrating clear, measurable impact from a short, tightly-scoped pilot.<\/p>\n Despite the relatively simple picture painted of each technique below, it\u2019s important to keep in mind there\u2019s no one-size-fits-all solution, and applying these techniques isn\u2019t like stacking lego bricks, nor do they work out-of-the-box. To get you started I\u2019ve provided a brief overview of each technique, this is by no means exhaustive, but looks to offer a starting point for your own research.<\/p>\n Transfer learning is about reusing existing models to solve new related problems. By starting with pre-trained weights, you leverage representations learned from large, diverse datasets and fine-tune the model on your smaller, target\u00a0dataset.<\/p>\n Why it\u00a0helps:<\/strong><\/p>\n Tips:<\/strong><\/p>\n A nice example of transfer learning is described in the following paper<\/a>. Where leveraging a pre-trained ResNet model enabled better classification of chest X-ray images and detecting COVID-19. Supported by the use of dropout and batch normalisation, the researchers froze the initial layers of the ResNet base model, while fine-tuning later layers, capturing task-specific, high-level features. This proved to be a cost effective method for achieving high accuracy with a small\u00a0dataset.<\/p>\n Self-supervised learning is a pre-training technique where artificial tasks (\u201cpretext tasks\u201d) are created to learn representations from broad unlabelled data. Examples include predicting masked tokens for text or rotation prediction, colorisation for images. The result is general-purpose representations you can later pair with transfer-learning (section 5.1) or semi-supervised (section 5) and fine-tune with your smaller\u00a0dataset.<\/p>\n Why it\u00a0helps:<\/strong><\/p>\n Tips:<\/strong><\/p>\n LEGAL-BERT<\/a> is a prominent example of self-supervised learning. Legal-BERT is a domain-specific variant of the BERT language model, pre-trained on a large dataset of legal documents to improve its understanding of legal language, terminology, and context. The key, is the use of unlabelled data, where techniques such as masked language modelling (the model learns to predict masked words) and next sentence prediction (learning the relationships between sentences, and determining if one follows another) removes the requirement for labelling. This text embedding model can then be used for more specific legal based ML\u00a0tasks.<\/p>\n Leverages a small labeled dataset in addition to a larger unlabelled set. The model iteratively refines predictions on unlabelled data, to generate task specific predictions that can be used as \u201cpseudo-labels\u201d for further iterations.<\/p>\n Why it\u00a0helps:<\/strong><\/p>\n Tips:<\/strong><\/p>\n Financial fraud detection is a problem that naturally lends itself to semi-supervised learning, with very little real labelled data (confirmed fraud cases) and a large set of unlabelled transaction data. The following paper<\/a> proposes a neat solution, by modelling transactions, users, and devices as nodes in a graph, where edges represent relationships, such as shared accounts or devices. The small set of labeled fraudulent data is then used to train the model by propagating fraud signals across the graph to the unlabelled nodes. For example, if a fraudulent transaction (labeled node) is linked to multiple unlabelled nodes (e.g., related users or devices), the model learns patterns and connections that might indicate\u00a0fraud.<\/p>\n Few and zero-shot learning refers to a broad collection of techniques designed to tackle very small datasets head on. Generally these methods train a model to identify \u201cnovel\u201d classes unseen during training, with a small labelled dataset used primarily for\u00a0testing.<\/p>\n Why it\u00a0helps:<\/strong><\/p>\n Tips:<\/strong><\/p>\n It\u2019s important to maintain realistic expectations when implementing few-shot and zero-shot techniques. Often, the aim is to achieve usable or \u201cgood enough\u201d performance. As a direct comparison of traditional deep-learning (DL) methods, the following study<\/a> compares both DL and few-shot learning (FSL) for classifying 20 coral reef fish species from underwater images with applications for detecting rare species with limited available data. It should come as no surprise that the best model tested was a DL model based on ResNet. With ~3500 examples for each species the model achieved an accuracy of 78%. However, collecting this volume of data for rare species is beyond practical. Subsequently, the number of samples was reduced to 315 per species, and the accuracy dropped to 42%. In contrast, the FSL model, achieved comparable results with as few as 5 labeled images per species, and better performance beyond 10 shots. Here, the Reptile algorithm was used, which is a meta-learning-based FSL approach. This was trained by repeatedly solving small classification problems (e.g., distinguishing a few classes) drawn from the MiniImageNet dataset (a useful benchmark dataset for FSL). During fine-tuning, the model was then trained using a few labeled examples (1 to 30 shots per species).<\/p>\n Weakly supervised learning describes a set of techniques for building models with noisy, inaccurate or restricted sources to label large quantities of data. We can split the topic into three: incomplete, inexact, and inaccurate supervision, distinguished by the confidence in the labels. Incomplete supervision occurs when only a subset of examples has ground-truth labels. Inexact supervision involves coarsely-grained labels, like labelling an MRI image as \u201clung cancer\u201d without specifying detailed attributes. Inaccurate supervision arises when labels are biased or incorrect due to\u00a0human.<\/p>\n Why it\u00a0helps:<\/strong><\/p>\n Tips:<\/strong><\/p>\n The primary goal of this technique is to estimate more informative or higher dimensional data with limited information. As an example, this paper<\/a> presents a weakly supervised learning approach to estimating a 3D human poses. The method relies on 2D pose annotations, avoiding the need for expensive 3D ground-truth data. Using an adversarial reprojection network (RepNet), the model predicts 3D poses and reprojects them into 2D views to compare with 2D annotations, minimising reprojection error. This approach leverages adversarial training to enforce plausibility of 3D poses and showcases the potential of weakly supervised methods for complex tasks like 3D pose estimation with limited labeled\u00a0data.<\/p>\n Active learning seeks to optimise labelling efforts by identifying unlabelled samples that, once labeled, will provide the model with the most informative data. A common approach is uncertainty sampling, which selects samples where the model\u2019s predictions are least certain. This uncertainty is often quantified using measures such as entropy or margin sampling. This is highly iterative; each round influences the model\u2019s next set of predictions.<\/p>\n Why it\u00a0helps:<\/strong><\/p>\n Tips:<\/strong><\/p>\n This technique has been extensively used in chemical analysis and materials research. Where, large databases of real and simulated molecular structures and their properties have been collected over decades. These databases are particularly useful for drug discovery, where simulations like docking are used to predict how small molecules (e.g., potential drugs) interact with targets such as proteins or enzymes. However, the computational cost of performing these types of calculations over millions of molecules makes brute force studies impractical. This is where active learning comes in. One such study<\/a> showed that by training a predictive model on an initial subset of docking results and iteratively selecting the most uncertain molecules for further simulations, researchers were able to drastically reduce the number of molecules tested while still identifying the best candidates.<\/p>\n Artificially increase your dataset by applying transformations to existing examples\u200a\u2014\u200asuch as flipping or cropping images, translation or synonym replacement for text and time shifts or random cropping for time-series. Alternatively, upsample underrepresented data with ADASYN (Adaptive Synthetic Sampling) and SMOTE (Synthetic Minority Over-sampling Technique).<\/p>\n Why it\u00a0helps:<\/strong><\/p>\n Tips:<\/strong><\/p>\n Data augmentation is an incredibly broad topic, with numerous surveys exploring the current state-of-the-art across various fields, including computer vision (review paper<\/a>), natural language processing (review paper<\/a>), and time-series data (review paper<\/a>). It has become an integral component of most machine learning pipelines due to its ability to enhance model generalisation. This is particularly critical for small datasets, where augmenting input data by introducing variations, such as transformations or noise, and removing redundant or irrelevant features can significantly improve a model\u2019s robustness and performance.<\/p>\n Here we train one model to solve several tasks simultaneously. This improves how well models perform by encouraging them to find patterns or solutions that work well for multiple goals at the same time. Lower layers capture general features that benefit all tasks, even if you have limited data for\u00a0some.<\/p>\n Why it\u00a0helps:<\/strong><\/p>\n Tips:<\/strong><\/p>\n The scarcity of data for many practical applications of ML makes sharing both data and models across tasks an attractive proposition. This is enabled by Multitask learning, where tasks benefit from shared knowledge and correlations in overlapping domains. However, it requires a large, diverse dataset that integrates multiple related properties. Polymer design is one example<\/a> where this has been successful. Here, a hybrid dataset of 36 properties across 13,000 polymers, covering a mix of mechanical, thermal, and chemical characteristics, was used to train a deep-learning-based MTL architecture. The multitask model outperformed single-task models for every polymer property. Particularly, for underrepresented properties.<\/p>\n Ensembles aggregate predictions from several base models to improve robustness. Generally, ML algorithms can be limited in a variety of ways: high variance, high bias, and low accuracy. This manifests as different uncertainty distributions for different models across predictions. Ensemble methods limit the variance and bias errors associated with a single model; for example, bagging reduces variance without increasing the bias, while boosting reduces\u00a0bias.<\/p>\n Why it\u00a0helps:<\/strong><\/p>\n Tips:<\/strong><\/p>\n As an example, the following paper<\/a> highlights ensemble learning as a method to improve the classification of cervical cytology images. In this case, three pre-trained neural networks\u200a\u2014\u200aInception v3, Xception, and DenseNet-169\u200a\u2014\u200awere used. The diversity of these base models ensured the ensemble benefits from each models unique strengths and feature extraction capabilities. This combined with the fusion of model confidences, via a method that rewards confident, accurate predictions while penalising uncertain ones, maximised the utility of the limited data. Combined with transfer learning, the final predictions were robust to the errors of any particular model, despite the small dataset\u00a0used.<\/p>\n Integrate domain-specific knowledge or physics-based constraints into ML models. This embeds prior knowledge, reducing the model\u2019s reliance on large data to infer patterns. For example, using partial differential equations alongside neural networks for fluid dynamics.<\/p>\n Why it\u00a0helps:<\/strong><\/p>\n Tips:<\/strong><\/p>\n Constraining a model, in this way requires a deep understanding of your problem domain, and is often applied to problems where the environment the model operates in is well understood, such as physical systems. An example of this is lithium-ion battery modelling<\/a>, where domain knowledge of battery dynamics is integrated into the ML process. This allows the model to capture complex behaviours and uncertainties missed by traditional physical models, ensuring physically consistent predictions and improved performance under real-world conditions like battery\u00a0aging.<\/p>\n For me, projects constrained by limited data are some of the most interesting projects to work on\u200a\u2014\u200adespite the higher risk of failure, they offer an opportunity to explore the state-of-the-art and experiment. These are tough problems! However, systematically applying the strategies covered in this post can greatly improve your odds of delivering a robust, effective model. Embrace the iterative nature of these problems: refine labels, employ augmentations, and analyse errors in quick cycles. Short pilot experiments help validate each technique\u2019s impact before you invest\u00a0further.<\/p>\n Where to Start When Data is Limited<\/a> was originally published in Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\n\n
\n
\n
\n
\n
\n
\n
4. Matching Questions to Techniques<\/h3>\n
\n
\n
\n
\n
\n
\n
\n
\n
5. Strategies for Tackling Small\u00a0Data<\/h3>\n
5.1 Transfer\u00a0Learning<\/h3>\n
\n
\n
5.2 Self-Supervised Learning<\/h3>\n
\n
\n
5.3 Semi-Supervised Learning<\/h3>\n
\n
\n
5.4 Few-Shot and Zero-Shot Learning<\/h3>\n
\n
\n
5.5 Weakly Supervised Learning<\/h3>\n
\n
\n
5.6 Active\u00a0Learning<\/h3>\n
\n
\n
5.7 Data Augmentation<\/h3>\n
\n
\n
5.8 Multitask Learning<\/h3>\n
\n
\n
5.9 Ensemble\u00a0Learning<\/h3>\n
\n
\n
5.10 Process-Aware (Hybrid)\u00a0Models<\/h3>\n
\n
\n
6. Conclusion<\/h3>\n
<\/p>\n
\n