Delve into an end-to-end Machine Learning project to improve the quality of the Open Food Facts\u00a0database<\/h4>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2A-6WhpiGzUAb__bj8tmFwUQ.png?ssl=1\")
Image generated with\u00a0Flux1<\/figcaption><\/figure>\nOpen Food Facts\u2019 purpose is to create the largest open-source food database in the world<\/strong>. To this day, it has collected over 3 millions products and their information thanks to its contributors.<\/p>\nNutritional value, eco-score, product origins,\u2026 Various data that define each product and give consumers and researchers insights about what they put in their\u00a0plates.<\/p>\n
This information is provided by the community of users and contributors, who actively add products data, take pictures, and fill any missing data into the database through the mobile\u00a0app<\/a>.<\/p>\nUsing the product picture, Open Food Facts extracts the ingredients list, typically located on the back of the packaging, through Optical Character Recognition (OCR). The product composition is then parsed and added to the database.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/400\/0%2A095wbwP-o4N6ykFG.jpg?ssl=1\")
List of ingredients on the product packaging<\/figcaption><\/figure>\nHowever, it often appears that the text extraction doesn\u2019t go\u00a0well\u2026<\/p>\n
Ingr\u00e9dients: Jambon do porc, sel, dextrose, ar\u00f4me naturels, antioxydant: E316, conservateur: E250
^<\/pre>\nThese typos may seem minimal, but when the list is parsed to extract individual ingredients, such errors create unrecognized ingredients,<\/strong> which harm the quality of the database. Light reflections, folded packaging, low-quality pictures, and other factors all complicate the ingredient parsing\u00a0process.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/400\/0%2Az0OscAwbw4e7tzPM.jpg?ssl=1\")
<\/figure>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/300\/0%2AkcE1i9RReIsLpNDO.jpg?ssl=1\")
Examples of packaging pictures where the OCR fails (from the Open Food Facts database)<\/figcaption><\/figure>\nOpen Food Facts has tried to solve this issue for years using Regular Expressions and existing solutions such as Elasticsearch\u2019s corrector, without success. Until recently.<\/strong><\/p>\nThanks to the latest advancements in artificial intelligence, we now have access to powerful Large Language Models<\/strong>, also called\u00a0LLMs<\/strong>.<\/p>\nBy training our own model, we created the Ingredients Spellcheck <\/strong>and managed to not only outperform proprietary LLMs such as GPT-4o<\/strong> or Claude 3.5 Sonnet <\/strong>on this task, but also to reduce the number of unrecognized ingredients in the database by\u00a011%<\/strong>.<\/p>\nThis article walks you through the different stages of the project and shows you how we managed to improve the quality of the database using Machine Learning.<\/strong><\/p>\nEnjoy the\u00a0reading!<\/p>\n
Define the\u00a0problem<\/h3>\n
When a product is added by a contributor, its pictures go through a series of processes to extract all relevant information. One crucial step is the extraction of the list of ingredients<\/strong>.<\/p>\nWhen a word is identified as an ingredient, it is cross-referenced with a taxonomy <\/strong>that contains a predefined list of recognized ingredients. If the word matches an entry in the taxonomy, it is tagged as an ingredient and added to the product\u2019s information.<\/p>\nThis tagging process ensures that ingredients are standardized and easily searchable, providing accurate data for consumers and analysis\u00a0tools.<\/p>\n
But if an ingredient is not recognized, the process\u00a0fails.<\/strong><\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2Alke6K2JoJHOEP9p4f-LurQ.png?ssl=1\")
The ingredient \u201cJambon do porc\u201d (Pork ham) was not recognized by the parser (from the Product Edition\u00a0page)<\/figcaption><\/figure>\nFor this reason, we introduced an additional layer to the process: the Ingredients Spellcheck<\/strong>, designed to correct ingredient lists before they are processed by the ingredient parser.<\/p>\nA simpler approach would be the Peter Norvig algorithm<\/a>, which processes each word by applying a series of character deletions, additions, and replacements to identify potential corrections.<\/p>\nHowever, this method proved to be insufficient for our use case, for several\u00a0reasons:<\/p>\n
\n- \nSpecial Characters and Formatting<\/strong>: Elements like commas, brackets, and percentage signs hold critical importance in ingredient lists, influencing product composition and allergen labeling (e.g., \u201csalt (1.2%)\u201d).<\/em>\n<\/li>\n
- \nMultilingual Challenges<\/strong>: the database contains products from all over the word with a wide variety of languages. This further complicates a basic character-based approach like Norvig\u2019s, which is language-agnostic.<\/li>\n<\/ul>\n
Instead, we turned to the latest advancements in Machine Learning, particularly Large Language Models (LLMs)<\/strong>, which excel in a wide variety of Natural Language Processing (NLP)<\/strong> tasks, including spelling correction.<\/p>\nThis is the path we decided to\u00a0take.<\/p>\n
Evaluate<\/h3>\nYou can\u2019t improve what you don\u2019t\u00a0measure.<\/p><\/blockquote>\n
What is a good correction? And how to measure the performance of the corrector, LLM or\u00a0non-LLM?<\/strong><\/p>\nOur first step is to understand and catalog the diversity of errors the Ingredient Parser encounters.<\/p>\n
Additionally, it\u2019s essential to assess whether an error should even be corrected in the first place. Sometimes, trying to correct mistakes could do more harm than\u00a0good:<\/p>\n
flour, salt (1!2%)
# Is it 1.2% or 12%?...<\/pre>\nFor these reasons, we created the Spellcheck Guidelines<\/strong>, a set of rules that limits the corrections. These guidelines will serve us in many ways throughout the project, from the dataset generation to the model evaluation.<\/em><\/p>\nThe guidelines was notably used to create the Spellcheck Benchmark<\/strong><\/a>, a curated dataset containing approximately 300 lists of ingredients manually corrected.<\/p>\nThis benchmark is the cornerstone of the project<\/strong>. It enables us to evaluate any solution, Machine Learning or simple heuristic, on our use\u00a0case.<\/p>\nIt goes along the Evaluation algorithm<\/strong>, a custom solution we developed that transform a set of corrections into measurable metrics.<\/p>\nThe Evaluation Algorithm<\/h4>\n
Most of the existing metrics and evaluation algorithms for text-relative tasks compute the similarity between a reference and a prediction, such as BLEU<\/a> or ROUGE <\/a>scores for language translation or summarization.<\/p>\nHowever, in our case, these metrics fail\u00a0short.<\/p>\n
We want to evaluate how well the Spellcheck algorithm recognizes and fixes the right words in a list of ingredients. Therefore, we adapt the Precision<\/strong> and Recall<\/strong> metrics for our\u00a0task:<\/p>\n\nPrecision <\/strong>= Right corrections by the model \/ \u200bTotal corrections made by the\u00a0model<\/p><\/blockquote>\n\nRecall <\/strong>= Right corrections by the model \/ \u200bTotal number of\u00a0errors<\/p><\/blockquote>\nHowever, we don\u2019t have the fine-grained view of which words were supposed to be corrected\u2026 We only have access\u00a0to:<\/p>\n
\n- The original<\/em>: the list of ingredients as present in the database;<\/strong><\/li>\n
- The reference<\/em>: how we expect this list to be corrected;<\/strong><\/li>\n
- The prediction<\/em>: the correction from the\u00a0model.<\/strong><\/li>\n<\/ul>\n
Is there any way to calculate the number of errors that were correctly corrected, the ones that were missed by the Spellcheck, and finally the errors that were wrongly corrected?<\/p><\/blockquote>\n
The answer is\u00a0yes!<\/p>\n
Original: \"Th cat si on the fride,\"
Reference: \"The cat is on the fridge.\"
Prediction: \"Th big cat is in the fridge.\"<\/pre>\nWith the example above, we can easily spot which words were supposed to be corrected: The\u00a0, is and fridge\u00a0; and which words were wrongly corrected: on into in. Finally, we see that an additional word was added: big\u00a0.<\/p>\n
If we align these 3 sequences in pairs, original-reference and original-prediction\u00a0, we can detect which words were supposed to be corrected, and those that weren\u2019t. This alignment problem is typical in bio-informatic, called Sequence Alignment<\/a>, whose purpose is to identify regions of similarity.<\/p>\nThis is a perfect analogy for our spellcheck evaluation task.<\/p>\n
Original: \"Th - cat si on the fride,\"
Reference: \"The - cat is on the fridge.\"
1 0 0 1 0 0 1
Original: \"Th - cat si on the fride,\"
Prediction: \"Th big cat is in the fridge.\"
0 1 0 1 1 0 1
FN FP TP FP TP<\/pre>\nBy labeling each pair with a 0 or 1 whether the word changed or not, we can calculate how often the model correctly fixes mistakes (True Positives\u200a\u2014\u200aTP)<\/strong>, incorrectly changes correct words (False Positives\u200a\u2014\u200aFP)<\/strong>, and misses errors that should have been corrected (False Negatives\u200a\u2014\u200aFN).<\/strong><\/p>\nIn other words, we can calculate the Precision <\/strong>and Recall <\/strong>of the Spellcheck!<\/p>\nWe now have a robust algorithm that is capable of evaluating any Spellcheck solution!<\/p>\n
You can find the algorithm in the project repository<\/a>.<\/p>\nLarge Language\u00a0Models<\/h3>\n
Large Language Models (LLMs) have proved being great help in tackling Natural Language task in various industries.<\/p>\n
They constitute a path we have to explore for our use\u00a0case.<\/p>\n
Many LLM providers brag about the performance of their model on leaderboards, but how do they perform on correcting error in lists of ingredients? Thus, we evaluated them!<\/p><\/blockquote>\n
We evaluated GPT-3.5 <\/strong>and GPT-4o <\/strong>from OpenAI<\/strong>, Claude-Sonnet-3.5 <\/strong>from Anthropic<\/strong>, and Gemini-1.5-Flash<\/strong> from Google <\/strong>using our custom benchmark and evaluation algorithm.<\/p>\nWe prompted detailed instructions to orient the corrections towards our custom guidelines.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AgS97vKb5scseuWGdu4k57Q.png?ssl=1\")
LLMs evaluation on our benchmark (image from\u00a0author)<\/figcaption><\/figure>\nGPT-3.5-Turbo<\/strong> delivered the best performance compared to other models, both in terms of metrics and manual review. Special mention goes to Claude-Sonnet-3.5<\/strong>, which showed impressive error corrections (high Recall), but often provided additional irrelevant explanations, lowering its Precision.<\/p>\nGreat! We have an LLM that works! Time to create the feature in the\u00a0app!<\/p>\n
Well,<\/em><\/strong> not so\u00a0fast\u2026<\/p>\nUsing private LLMs reveals many challenges:<\/p>\n
\n- \nLack of Ownership<\/strong>: We become dependent on the providers and their models. New model versions are released frequently, altering the model\u2019s behavior. This instability, primarily because the model is designed for general purposes rather than our specific task, complicates long-term maintenance.<\/li>\n
- \nModel Deletion Risk<\/strong>: We have no safeguards against providers removing older models. For instance, GPT-3.5 is slowly being replace by more performant models, despite being the best model for this\u00a0task!<\/li>\n
- \nPerformance Limitations<\/strong>: The performance of a private LLM is constrained by its prompts. In other words, our only way of improving outputs is through better prompts since we cannot modify the core weights of the model by training it on our own\u00a0data.<\/li>\n<\/ol>\n
For these reasons, we chose to focus our efforts on open-source solutions that would provide us with complete control and outperform general\u00a0LLMs.<\/em><\/strong><\/p>\nTrain our own\u00a0model<\/h3>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2A01OI_L5U0UkasdviWDOPxg.png?ssl=1\")
The model training workflow: from dataset extraction to model training (image from\u00a0author)<\/figcaption><\/figure>\nAny machine learning solution starts with data. In our case, data is the corrected lists of ingredients.<\/p>\n
However, not all lists of ingredients are equal. Some are free of unrecognized ingredients, some are just so unreadable they would be no point correcting them.<\/p>\n
Therefore, we find a perfect balance by choosing lists of ingredients having between 10 and 40 percent of unrecognized ingredients<\/strong>. We also ensured there\u2019s no duplicate within the dataset, but also with the benchmark to prevent any data leakage during the evaluation stage.<\/p>\nWe extracted 6000 uncorrected lists from the Open Food Facts database using DuckDB<\/a>, a fast in-process SQL tool capable of processing millions of rows under the\u00a0second.<\/p>\nHowever, those extracted lists are not corrected yet, and manually annotating them would take too much time and resources\u2026<\/p>\n
However, we have access to LLMs we already evaluated on the exact task. Therefore, we prompted GPT-3.5-Turbo, the best model on our benchmark, to correct every list in respect of our guidelines.<\/strong><\/p>\nThe process took less than an hour and cost nearly\u00a02$<\/strong>.<\/p>\nWe then manually reviewed the dataset using Argilla<\/a>, an open-source annotation tool specialized in Natural Language Processing tasks. This process ensures the dataset is of sufficient quality to train a reliable\u00a0model.<\/p>\nWe now have at our disposal a <\/strong>training dataset<\/strong><\/a> and an <\/strong>evaluation benchmark<\/strong><\/a> to train our own model on the Spellcheck task.<\/strong><\/p>\nTraining<\/h4>\n
For this stage, we decided to go with Sequence-to-Sequence Language Models<\/strong>. In other words, these models take a text as input and returns a text as output, which suits the spellcheck process.<\/p>\nSeveral models fit this role, such as the T5 family<\/strong> developed by Google in 2020, or the current open-source LLMs such as Llama<\/strong> or Mistral<\/strong>, which are designed for text generation and following instructions.<\/p>\nThe model training consists in a succession of steps, each one requiring different resources allocations, such as cloud GPUs, data validation and logging. For this reason, we decided to orchestrate the training using Metaflow<\/a>, a pipeline orchestrator designed for Data science and Machine Learning projects.<\/p>\nThe training pipeline is composed as\u00a0follow:<\/p>\n
\n- Configurations and hyperparameters are imported to the pipeline from config yaml\u00a0files;<\/li>\n
- The training job is launched in the cloud using AWS Sagemaker<\/a>, along the set of model hyperparameters and the custom modules such as the evaluation algorithm. Once the job is done, the model artifact is stored in an AWS S3 bucket. All training details are tracked using Comet\u00a0ML<\/a>;<\/li>\n
- The fine-tuned model is then evaluated on the benchmark <\/strong>using the evaluation algorithm. Depending on the model sizem this process can be extremely long. Therefore, we used vLLM<\/a>, a Python library designed to accelerates LLM inferences;<\/li>\n
- The predictions against the benchmark, also stored in AWS S3, are sent to Argilla<\/a> for human-evaluation.<\/li>\n<\/ul>\n
After iterating over and over between refining the data and the model training, we achieved performance comparable to proprietary LLMs <\/strong>on the Spellcheck task, scoring an F1-Score of\u00a00.65<\/strong>.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/935\/1%2AfdGAj-K2IiQVgPP6zMYzCg.png?ssl=1\")
<\/figure>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2Am_EdG5UeBREYmKrVSr14pQ.png?ssl=1\")
LLMs evaluation on our benchmark (image from\u00a0author)<\/figcaption><\/figure>\nThe model, a fine-tuned Mistral-7B-Base-v0.3<\/a>, is available on the Hugging Face platform and is publicly available, along its dataset<\/a> and evaluation benchmark<\/a>.<\/p>\nopenfoodfacts\/spellcheck-mistral-7b \u00b7 Hugging Face<\/a><\/p>\nFurthermore, we estimated the Spellcheck reduced the number of unrecognized ingredients by 11%, which is promising!<\/strong><\/p>\nNow comes the final phase of the project: integrating the model into Open Food\u00a0Facts.<\/p>\n
Deployment & Integration<\/h3>\n
Our model is\u00a0big!<\/strong><\/p>\n7 billions parameters, which means 14 GB <\/strong>of memory required to run it in float16<\/strong>, without considering the 20% overhead\u00a0factor.<\/strong><\/p>\nAdditionally, large models often mean low throughput during inference<\/strong>, which can make them inappropriate for real-time serving. We need GPUs with large memory to run this model in production, such as the Nvidia L4<\/a>, which is equipped with 24GB of\u00a0VRAM.<\/p>\nBut the price of running these instances in the cloud is quite expensive\u2026<\/strong><\/p>\nHowever, a possibility to provide a real-time experience for our users, without requiring GPU instances running 24\/7, is batch inference<\/strong>.<\/p>\nLists of ingredients are processed in batches by the model on a regular basis, then stored in the database. This way, we pay only for the resources used during the batch processing!<\/strong><\/p>\nBatch Job<\/h4>\n
We developed a batch processing system to handle large-scale text processing using LLMs efficiently with Google Batch\u00a0Job<\/a>.<\/p>\n![\"\"](\"https:\/\/cdn-images-1.medium.com\/max\/729\/0*nevKLqwBo6wmAOQz\")
Batch processing system (image from\u00a0author)<\/figcaption><\/figure>\nThe process begins by extracting data from the Open Food Facts database using DuckDB, which processes 43 GB of data in under 2\u00a0minutes!<\/strong><\/p>\nThe extracted data is then sent to a Google Bucket, triggering a Google Batch\u00a0Job.<\/p>\n
This job uses a pre-prepared Docker image containing all necessary dependencies and algorithms. To optimize the resource-intensive LLM processing, we reuse vLLM achieving impressive performances, correcting 10,000 lists of ingredients in 20 minutes only with a GPU\u00a0L4<\/strong>!<\/p>\nAfter successful processing, the corrected data is saved in a intermediate database containing the predictions of all models in Open Food Facts, served by Robotoff<\/a>.<\/p>\nWhen a contributor modifies a product details, they\u2019re presented with the spellcheck corrections, ensuring users remain the key decision-makers in Open Food Facts\u2019 data quality\u00a0process.<\/strong><\/p>\nThis system allows Open Food Facts to leverage advanced AI capabilities for improving data quality while preserving its community-driven approach.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2A5o9gFYzHlgBTnutt1MnE1g.png?ssl=1\")
Batch jobs in GCP (image from\u00a0author)<\/figcaption><\/figure>\nConlusion<\/h3>\n
In this article, we walked you through the development and the integration of the Ingredients Spellcheck<\/strong>, an LLM-powered up feature to correct OCR-extracted lists of ingredients.<\/p>\nWe first developed a set of rules, the Spellcheck Guidelines<\/strong>, to restrict the corrections\u00a0. We created a benchmark of corrected lists of ingredients that, along a custom evaluation algorithm, to evaluate any solution to the\u00a0problem.<\/p>\nWith this setup, we evaluated various private LLMs and determined that GPT-3.5-Turbo<\/strong> was the most suitable model for our specific use case. However, we also demonstrated that relying on a private LLM imposes significant limitations, including lack of ownership and restricted opportunities to improve or fine-tune such a large model (175 billion parameters).<\/p>\nTo address these challenges, we decided to develop our own model<\/strong>, fine-tuning it on synthetically corrected texts extracted from the database. After several iterations and experiments, we successfully achieved good performances with an open-source model<\/strong>. Not only did we match private LLMs performance, it also solved the ownership problem we were facing, giving us full control over our\u00a0model.<\/p>\nWe then integrated this model into the Open Food Facts <\/strong>using batch inference deployment, <\/strong>enabling us to process thousands of lists on a regular basis. The predictions are stored in Robotoff database as Insights <\/strong>before being validated by contributors, leaving OFF data quality ownership to contributors<\/strong>.<\/p>\nNext step<\/h3>\n
The Spellcheck integration is still a work in progress<\/strong>. We are working on designing the user interface to propose ML generated corrections and let contributors accept, deny, or modify corrections. We expect fully integrating the feature by the end of the\u00a0year.<\/strong><\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AdCG8KVAH_jLmapejAprfRQ.png?ssl=1\")
Spellcheck corrections validated by users built in Hugging Face\u00a0Space<\/a><\/figcaption><\/figure>\nAdditionally, we plan to continue refining the model through iterative improvements. Its performance can be significantly enhanced by improving the quality of the training data and incorporating user feedback. This approach will allow us to fine-tune the model continuously, ensuring it remains highly effective and aligned with real-world use\u00a0cases.<\/p>\n
The model, along its datasets, can be find in the official Hugging Face repository<\/a>. The code used to developped this model is available in the OpenFoodFacts-ai\/spellcheck<\/a> Github repository.<\/p>\nThank you for reading that far! We hope you enjoyed the\u00a0reading.<\/p>\n
If you too, you want to contribute to Open Food Facts, you\u00a0can:<\/p>\n
\n- Contribute to the Open Food Facts GitHub<\/a>: explore open issues that align with your\u00a0skills,<\/li>\n
- Download the Open Food Facts mobile app<\/a>: add new products to the database or improve existing ones by simply scanning their barcodes,<\/li>\n
- Join the Open Food Facts Slack<\/a> and start discussing with other contributors in the OFF community.<\/li>\n<\/ul>\n
We can\u2019t wait to see you join the community!<\/p>\n
Don\u2019t hesitate to check our other articles:<\/p>\n
DuckDB & Open Food Facts: the largest open food database in the palm of your hand \ud83e\udd86\ud83c\udf4a<\/a><\/p>\n![\"\"](\"https:\/\/medium.com\/_\/stat?event=post.clientViewed&referrerSource=full_rss&postId=d74dfe02e0e4\")
<\/p>\n
\nHow Did Open Food Facts Fix OCR-Extracted Ingredients Using Open-Source LLMs?<\/a> was originally published in Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\n
Open Food Facts\u2019 purpose is to create the largest open-source food database in the world<\/strong>. To this day, it has collected over 3 millions products and their information thanks to its contributors.<\/p>\n Nutritional value, eco-score, product origins,\u2026 Various data that define each product and give consumers and researchers insights about what they put in their\u00a0plates.<\/p>\n This information is provided by the community of users and contributors, who actively add products data, take pictures, and fill any missing data into the database through the mobile\u00a0app<\/a>.<\/p>\n Using the product picture, Open Food Facts extracts the ingredients list, typically located on the back of the packaging, through Optical Character Recognition (OCR). The product composition is then parsed and added to the database.<\/p>\n However, it often appears that the text extraction doesn\u2019t go\u00a0well\u2026<\/p>\n These typos may seem minimal, but when the list is parsed to extract individual ingredients, such errors create unrecognized ingredients,<\/strong> which harm the quality of the database. Light reflections, folded packaging, low-quality pictures, and other factors all complicate the ingredient parsing\u00a0process.<\/p>\n Open Food Facts has tried to solve this issue for years using Regular Expressions and existing solutions such as Elasticsearch\u2019s corrector, without success. Until recently.<\/strong><\/p>\n Thanks to the latest advancements in artificial intelligence, we now have access to powerful Large Language Models<\/strong>, also called\u00a0LLMs<\/strong>.<\/p>\n By training our own model, we created the Ingredients Spellcheck <\/strong>and managed to not only outperform proprietary LLMs such as GPT-4o<\/strong> or Claude 3.5 Sonnet <\/strong>on this task, but also to reduce the number of unrecognized ingredients in the database by\u00a011%<\/strong>.<\/p>\n This article walks you through the different stages of the project and shows you how we managed to improve the quality of the database using Machine Learning.<\/strong><\/p>\n Enjoy the\u00a0reading!<\/p>\n When a product is added by a contributor, its pictures go through a series of processes to extract all relevant information. One crucial step is the extraction of the list of ingredients<\/strong>.<\/p>\n When a word is identified as an ingredient, it is cross-referenced with a taxonomy <\/strong>that contains a predefined list of recognized ingredients. If the word matches an entry in the taxonomy, it is tagged as an ingredient and added to the product\u2019s information.<\/p>\n This tagging process ensures that ingredients are standardized and easily searchable, providing accurate data for consumers and analysis\u00a0tools.<\/p>\n But if an ingredient is not recognized, the process\u00a0fails.<\/strong><\/p>\n For this reason, we introduced an additional layer to the process: the Ingredients Spellcheck<\/strong>, designed to correct ingredient lists before they are processed by the ingredient parser.<\/p>\n A simpler approach would be the Peter Norvig algorithm<\/a>, which processes each word by applying a series of character deletions, additions, and replacements to identify potential corrections.<\/p>\n However, this method proved to be insufficient for our use case, for several\u00a0reasons:<\/p>\n Instead, we turned to the latest advancements in Machine Learning, particularly Large Language Models (LLMs)<\/strong>, which excel in a wide variety of Natural Language Processing (NLP)<\/strong> tasks, including spelling correction.<\/p>\n This is the path we decided to\u00a0take.<\/p>\n You can\u2019t improve what you don\u2019t\u00a0measure.<\/p><\/blockquote>\n What is a good correction? And how to measure the performance of the corrector, LLM or\u00a0non-LLM?<\/strong><\/p>\n Our first step is to understand and catalog the diversity of errors the Ingredient Parser encounters.<\/p>\n Additionally, it\u2019s essential to assess whether an error should even be corrected in the first place. Sometimes, trying to correct mistakes could do more harm than\u00a0good:<\/p>\n For these reasons, we created the Spellcheck Guidelines<\/strong>, a set of rules that limits the corrections. These guidelines will serve us in many ways throughout the project, from the dataset generation to the model evaluation.<\/em><\/p>\n The guidelines was notably used to create the Spellcheck Benchmark<\/strong><\/a>, a curated dataset containing approximately 300 lists of ingredients manually corrected.<\/p>\n This benchmark is the cornerstone of the project<\/strong>. It enables us to evaluate any solution, Machine Learning or simple heuristic, on our use\u00a0case.<\/p>\n It goes along the Evaluation algorithm<\/strong>, a custom solution we developed that transform a set of corrections into measurable metrics.<\/p>\n Most of the existing metrics and evaluation algorithms for text-relative tasks compute the similarity between a reference and a prediction, such as BLEU<\/a> or ROUGE <\/a>scores for language translation or summarization.<\/p>\n However, in our case, these metrics fail\u00a0short.<\/p>\n We want to evaluate how well the Spellcheck algorithm recognizes and fixes the right words in a list of ingredients. Therefore, we adapt the Precision<\/strong> and Recall<\/strong> metrics for our\u00a0task:<\/p>\n \nPrecision <\/strong>= Right corrections by the model \/ \u200bTotal corrections made by the\u00a0model<\/p><\/blockquote>\n \nRecall <\/strong>= Right corrections by the model \/ \u200bTotal number of\u00a0errors<\/p><\/blockquote>\n However, we don\u2019t have the fine-grained view of which words were supposed to be corrected\u2026 We only have access\u00a0to:<\/p>\n Is there any way to calculate the number of errors that were correctly corrected, the ones that were missed by the Spellcheck, and finally the errors that were wrongly corrected?<\/p><\/blockquote>\n The answer is\u00a0yes!<\/p>\n With the example above, we can easily spot which words were supposed to be corrected: The\u00a0, is and fridge\u00a0; and which words were wrongly corrected: on into in. Finally, we see that an additional word was added: big\u00a0.<\/p>\n If we align these 3 sequences in pairs, original-reference and original-prediction\u00a0, we can detect which words were supposed to be corrected, and those that weren\u2019t. This alignment problem is typical in bio-informatic, called Sequence Alignment<\/a>, whose purpose is to identify regions of similarity.<\/p>\n This is a perfect analogy for our spellcheck evaluation task.<\/p>\n By labeling each pair with a 0 or 1 whether the word changed or not, we can calculate how often the model correctly fixes mistakes (True Positives\u200a\u2014\u200aTP)<\/strong>, incorrectly changes correct words (False Positives\u200a\u2014\u200aFP)<\/strong>, and misses errors that should have been corrected (False Negatives\u200a\u2014\u200aFN).<\/strong><\/p>\n In other words, we can calculate the Precision <\/strong>and Recall <\/strong>of the Spellcheck!<\/p>\n We now have a robust algorithm that is capable of evaluating any Spellcheck solution!<\/p>\n You can find the algorithm in the project repository<\/a>.<\/p>\n Large Language Models (LLMs) have proved being great help in tackling Natural Language task in various industries.<\/p>\n They constitute a path we have to explore for our use\u00a0case.<\/p>\n Many LLM providers brag about the performance of their model on leaderboards, but how do they perform on correcting error in lists of ingredients? Thus, we evaluated them!<\/p><\/blockquote>\n We evaluated GPT-3.5 <\/strong>and GPT-4o <\/strong>from OpenAI<\/strong>, Claude-Sonnet-3.5 <\/strong>from Anthropic<\/strong>, and Gemini-1.5-Flash<\/strong> from Google <\/strong>using our custom benchmark and evaluation algorithm.<\/p>\n We prompted detailed instructions to orient the corrections towards our custom guidelines.<\/p>\n GPT-3.5-Turbo<\/strong> delivered the best performance compared to other models, both in terms of metrics and manual review. Special mention goes to Claude-Sonnet-3.5<\/strong>, which showed impressive error corrections (high Recall), but often provided additional irrelevant explanations, lowering its Precision.<\/p>\n Great! We have an LLM that works! Time to create the feature in the\u00a0app!<\/p>\n Well,<\/em><\/strong> not so\u00a0fast\u2026<\/p>\n Using private LLMs reveals many challenges:<\/p>\n For these reasons, we chose to focus our efforts on open-source solutions that would provide us with complete control and outperform general\u00a0LLMs.<\/em><\/strong><\/p>\n Any machine learning solution starts with data. In our case, data is the corrected lists of ingredients.<\/p>\n However, not all lists of ingredients are equal. Some are free of unrecognized ingredients, some are just so unreadable they would be no point correcting them.<\/p>\n Therefore, we find a perfect balance by choosing lists of ingredients having between 10 and 40 percent of unrecognized ingredients<\/strong>. We also ensured there\u2019s no duplicate within the dataset, but also with the benchmark to prevent any data leakage during the evaluation stage.<\/p>\n We extracted 6000 uncorrected lists from the Open Food Facts database using DuckDB<\/a>, a fast in-process SQL tool capable of processing millions of rows under the\u00a0second.<\/p>\n However, those extracted lists are not corrected yet, and manually annotating them would take too much time and resources\u2026<\/p>\n However, we have access to LLMs we already evaluated on the exact task. Therefore, we prompted GPT-3.5-Turbo, the best model on our benchmark, to correct every list in respect of our guidelines.<\/strong><\/p>\n The process took less than an hour and cost nearly\u00a02$<\/strong>.<\/p>\n We then manually reviewed the dataset using Argilla<\/a>, an open-source annotation tool specialized in Natural Language Processing tasks. This process ensures the dataset is of sufficient quality to train a reliable\u00a0model.<\/p>\n We now have at our disposal a <\/strong>training dataset<\/strong><\/a> and an <\/strong>evaluation benchmark<\/strong><\/a> to train our own model on the Spellcheck task.<\/strong><\/p>\n For this stage, we decided to go with Sequence-to-Sequence Language Models<\/strong>. In other words, these models take a text as input and returns a text as output, which suits the spellcheck process.<\/p>\n Several models fit this role, such as the T5 family<\/strong> developed by Google in 2020, or the current open-source LLMs such as Llama<\/strong> or Mistral<\/strong>, which are designed for text generation and following instructions.<\/p>\n The model training consists in a succession of steps, each one requiring different resources allocations, such as cloud GPUs, data validation and logging. For this reason, we decided to orchestrate the training using Metaflow<\/a>, a pipeline orchestrator designed for Data science and Machine Learning projects.<\/p>\n The training pipeline is composed as\u00a0follow:<\/p>\n After iterating over and over between refining the data and the model training, we achieved performance comparable to proprietary LLMs <\/strong>on the Spellcheck task, scoring an F1-Score of\u00a00.65<\/strong>.<\/p>\n The model, a fine-tuned Mistral-7B-Base-v0.3<\/a>, is available on the Hugging Face platform and is publicly available, along its dataset<\/a> and evaluation benchmark<\/a>.<\/p>\n openfoodfacts\/spellcheck-mistral-7b \u00b7 Hugging Face<\/a><\/p>\n Furthermore, we estimated the Spellcheck reduced the number of unrecognized ingredients by 11%, which is promising!<\/strong><\/p>\n Now comes the final phase of the project: integrating the model into Open Food\u00a0Facts.<\/p>\n Our model is\u00a0big!<\/strong><\/p>\n 7 billions parameters, which means 14 GB <\/strong>of memory required to run it in float16<\/strong>, without considering the 20% overhead\u00a0factor.<\/strong><\/p>\n Additionally, large models often mean low throughput during inference<\/strong>, which can make them inappropriate for real-time serving. We need GPUs with large memory to run this model in production, such as the Nvidia L4<\/a>, which is equipped with 24GB of\u00a0VRAM.<\/p>\n But the price of running these instances in the cloud is quite expensive\u2026<\/strong><\/p>\n However, a possibility to provide a real-time experience for our users, without requiring GPU instances running 24\/7, is batch inference<\/strong>.<\/p>\n Lists of ingredients are processed in batches by the model on a regular basis, then stored in the database. This way, we pay only for the resources used during the batch processing!<\/strong><\/p>\n We developed a batch processing system to handle large-scale text processing using LLMs efficiently with Google Batch\u00a0Job<\/a>.<\/p>\n The process begins by extracting data from the Open Food Facts database using DuckDB, which processes 43 GB of data in under 2\u00a0minutes!<\/strong><\/p>\n The extracted data is then sent to a Google Bucket, triggering a Google Batch\u00a0Job.<\/p>\n This job uses a pre-prepared Docker image containing all necessary dependencies and algorithms. To optimize the resource-intensive LLM processing, we reuse vLLM achieving impressive performances, correcting 10,000 lists of ingredients in 20 minutes only with a GPU\u00a0L4<\/strong>!<\/p>\n After successful processing, the corrected data is saved in a intermediate database containing the predictions of all models in Open Food Facts, served by Robotoff<\/a>.<\/p>\n When a contributor modifies a product details, they\u2019re presented with the spellcheck corrections, ensuring users remain the key decision-makers in Open Food Facts\u2019 data quality\u00a0process.<\/strong><\/p>\n This system allows Open Food Facts to leverage advanced AI capabilities for improving data quality while preserving its community-driven approach.<\/p>\n In this article, we walked you through the development and the integration of the Ingredients Spellcheck<\/strong>, an LLM-powered up feature to correct OCR-extracted lists of ingredients.<\/p>\n We first developed a set of rules, the Spellcheck Guidelines<\/strong>, to restrict the corrections\u00a0. We created a benchmark of corrected lists of ingredients that, along a custom evaluation algorithm, to evaluate any solution to the\u00a0problem.<\/p>\n With this setup, we evaluated various private LLMs and determined that GPT-3.5-Turbo<\/strong> was the most suitable model for our specific use case. However, we also demonstrated that relying on a private LLM imposes significant limitations, including lack of ownership and restricted opportunities to improve or fine-tune such a large model (175 billion parameters).<\/p>\n To address these challenges, we decided to develop our own model<\/strong>, fine-tuning it on synthetically corrected texts extracted from the database. After several iterations and experiments, we successfully achieved good performances with an open-source model<\/strong>. Not only did we match private LLMs performance, it also solved the ownership problem we were facing, giving us full control over our\u00a0model.<\/p>\n We then integrated this model into the Open Food Facts <\/strong>using batch inference deployment, <\/strong>enabling us to process thousands of lists on a regular basis. The predictions are stored in Robotoff database as Insights <\/strong>before being validated by contributors, leaving OFF data quality ownership to contributors<\/strong>.<\/p>\n The Spellcheck integration is still a work in progress<\/strong>. We are working on designing the user interface to propose ML generated corrections and let contributors accept, deny, or modify corrections. We expect fully integrating the feature by the end of the\u00a0year.<\/strong><\/p>\n Additionally, we plan to continue refining the model through iterative improvements. Its performance can be significantly enhanced by improving the quality of the training data and incorporating user feedback. This approach will allow us to fine-tune the model continuously, ensuring it remains highly effective and aligned with real-world use\u00a0cases.<\/p>\n The model, along its datasets, can be find in the official Hugging Face repository<\/a>. The code used to developped this model is available in the OpenFoodFacts-ai\/spellcheck<\/a> Github repository.<\/p>\n Thank you for reading that far! We hope you enjoyed the\u00a0reading.<\/p>\n If you too, you want to contribute to Open Food Facts, you\u00a0can:<\/p>\n We can\u2019t wait to see you join the community!<\/p>\n Don\u2019t hesitate to check our other articles:<\/p>\n DuckDB & Open Food Facts: the largest open food database in the palm of your hand \ud83e\udd86\ud83c\udf4a<\/a><\/p>\n How Did Open Food Facts Fix OCR-Extracted Ingredients Using Open-Source LLMs?<\/a> was originally published in Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\nIngr\u00e9dients: Jambon do porc, sel, dextrose, ar\u00f4me naturels, antioxydant: E316, conservateur: E250
^<\/pre>\n<\/figure>\nDefine the\u00a0problem<\/h3>\n
\n
Evaluate<\/h3>\n
flour, salt (1!2%)
# Is it 1.2% or 12%?...<\/pre>\nThe Evaluation Algorithm<\/h4>\n
\n
Original: \"Th cat si on the fride,\"
Reference: \"The cat is on the fridge.\"
Prediction: \"Th big cat is in the fridge.\"<\/pre>\nOriginal: \"Th - cat si on the fride,\"
Reference: \"The - cat is on the fridge.\"
1 0 0 1 0 0 1
Original: \"Th - cat si on the fride,\"
Prediction: \"Th big cat is in the fridge.\"
0 1 0 1 1 0 1
FN FP TP FP TP<\/pre>\nLarge Language\u00a0Models<\/h3>\n
\n
Train our own\u00a0model<\/h3>\n
Training<\/h4>\n
\n
<\/figure>\nDeployment & Integration<\/h3>\n
Batch Job<\/h4>\n
Conlusion<\/h3>\n
Next step<\/h3>\n
\n
<\/p>\n
\n