LLMs are great\u2026 if they can fit all of your\u00a0data<\/h4>\n![\"\"](\"https:\/\/cdn-images-1.medium.com\/max\/1024\/0*LLApz7tkaqL9eQFl\")
Photo by Christopher Burns<\/a> on\u00a0Unsplash<\/a><\/figcaption><\/figure>\nOriginally published at <\/em>https:\/\/blog.developer.bazaarvoice.com<\/em><\/a> on October 28,\u00a02024.<\/em><\/p>\nIntroduction<\/h3>\n
Large language models are fantastic tools for unstructured text, but what if your text doesn\u2019t fit in the context window? Bazaarvoice faced exactly this challenge when building our AI Review Summaries feature: millions of user reviews simply won\u2019t fit into the context window of even newer LLMs and, even if they did, it would be prohibitively expensive.<\/p>\n
In this post, I share how Bazaarvoice tackled this problem by compressing the input text without loss of semantics. Specifically, we use a multi-pass hierarchical clustering approach that lets us explicitly adjust the level of detail we want to lose in exchange for compression, regardless of the embedding model chosen. The final technique made our Review Summaries feature financially feasible and set us up to continue to scale our business in the\u00a0future.<\/p>\n
The Problem<\/h3>\n
Bazaarvoice has been collecting user-generated product reviews for nearly 20 years so we have a lot<\/em> of data. These product reviews are completely unstructured, varying in length and content. Large language models are excellent tools for unstructured text: they can handle unstructured data and identify relevant pieces of information amongst distractors.<\/p>\nLLMs have their limitations, however, and one such limitation is the context window: how many tokens (roughly the number of words) can be put into the network at once. State-of-the-art large language models, such as Athropic\u2019s Claude version 3, have extremely large context windows of up to 200,000 tokens. This means you can fit small novels into them, but the internet is still a vast, every-growing collection of data, and our user-generated product reviews are no different.<\/p>\n
We hit the context window limit while building our Review Summaries feature that summarizes all of the reviews of a specific product on our clients website. Over the past 20 years, however, many products have garnered thousands of reviews that quickly overloaded the LLM context window. In fact, we even have products with millions of reviews that would require immense re-engineering of LLMs to be able to process in one\u00a0prompt.<\/p>\n
Even if it was technically feasible, the costs would be quite prohibitive. All LLM providers charge based on the number of input and output tokens. As you approach the context window limits for each product, of which we have millions, we can quickly run up cloud hosting bills in excess of six\u00a0figures.<\/p>\n
Our Approach<\/h3>\n
To ship Review Summaries despite these technical, and financial, limitations, we focused on a rather simple insight into our data: Many reviews say the same thing. In fact, the whole idea of a summary relies on this: review summaries capture the recurring insights, themes, and sentiments of the reviewers. We realized that we can capitalize on this data duplication to reduce the amount of text we need to send to the LLM, saving us from hitting the context window limit and<\/em> reducing the operating cost of our\u00a0system.<\/p>\nTo achieve this, we needed to identify segments of text that say the same thing. Such a task is easier said than done: often people use different words or phrases to express the same\u00a0thing.<\/p>\n
Fortunately, the task of identifying if text is semantically similar has been an active area of research in the natural language processing field. The work by Agirre et. al. 2013 (SEM 2013 shared task: Semantic Textual Similarity. In Second Joint Conference on Lexical and Computational Semantics<\/em>) even published a human-labeled data of semantically similar sentences known as the STS Benchmark. In it, they ask humans to indicate if textual sentences are semantically similar or dissimilar on a scale of 1\u20135, as illustrated in the table below (from Cer et. al., SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation<\/em><\/a>):<\/p>\n![\"\"](\"https:\/\/cdn-images-1.medium.com\/max\/616\/0*JMgNQbYheovGnVwl\")
<\/figure>\nThe STSBenchmark dataset is often used to evaluate how well a text embedding model can associate semantically similar sentences in its high-dimensional space. Specifically, Pearson\u2019s correlation is used to measure how well the embedding model represents the human judgements.<\/p>\n
Thus, we can use such an embedding model to identify semantically similar phrases from product reviews, and then remove repeated phrases before sending them to the\u00a0LLM.<\/p>\n
Our approach is as\u00a0follows:<\/p>\n
\n- First, product reviews are segmented the into sentences.<\/li>\n
- An embedding vector is computed for each sentence using a network that performs well on the STS benchmark<\/li>\n
- Agglomerative clustering is used on all embedding vectors for each\u00a0product.<\/li>\n
- An example sentence\u200a\u2014\u200athe one closest to the cluster centroid\u200a\u2014\u200ais retained from each cluster to send to the LLM, and other sentences within each cluster are\u00a0dropped.<\/li>\n
- Any small clusters are considered outliers, and those are randomly sampled for inclusion in the\u00a0LLM.<\/li>\n
- The number of sentences each cluster represents is included in the LLM prompt to ensure the weight of each sentiment is considered.<\/li>\n<\/ul>\n
This may seem straightforward when written in a bulleted list, but there were some devils in the details we had to sort out before we could trust this approach.<\/p>\n
Embedding Model Evaluation<\/h3>\n
First, we had to ensure the model we used effectively embedded text in a space where semantically similar sentences are close, and semantically dissimilar ones are far away. To do this, we simply used the STS benchmark dataset and computed the Pearson correlation for the models we desired to consider. We use AWS as a cloud provider, so naturally we wanted to evaluate their Titan Text Embedding<\/a> models.<\/p>\nBelow is a table showing the Pearson\u2019s correlation on the STS Benchmark for different Titan Embedding models:<\/p>\n
Originally published at <\/em>https:\/\/blog.developer.bazaarvoice.com<\/em><\/a> on October 28,\u00a02024.<\/em><\/p>\n Large language models are fantastic tools for unstructured text, but what if your text doesn\u2019t fit in the context window? Bazaarvoice faced exactly this challenge when building our AI Review Summaries feature: millions of user reviews simply won\u2019t fit into the context window of even newer LLMs and, even if they did, it would be prohibitively expensive.<\/p>\n In this post, I share how Bazaarvoice tackled this problem by compressing the input text without loss of semantics. Specifically, we use a multi-pass hierarchical clustering approach that lets us explicitly adjust the level of detail we want to lose in exchange for compression, regardless of the embedding model chosen. The final technique made our Review Summaries feature financially feasible and set us up to continue to scale our business in the\u00a0future.<\/p>\n Bazaarvoice has been collecting user-generated product reviews for nearly 20 years so we have a lot<\/em> of data. These product reviews are completely unstructured, varying in length and content. Large language models are excellent tools for unstructured text: they can handle unstructured data and identify relevant pieces of information amongst distractors.<\/p>\n LLMs have their limitations, however, and one such limitation is the context window: how many tokens (roughly the number of words) can be put into the network at once. State-of-the-art large language models, such as Athropic\u2019s Claude version 3, have extremely large context windows of up to 200,000 tokens. This means you can fit small novels into them, but the internet is still a vast, every-growing collection of data, and our user-generated product reviews are no different.<\/p>\n We hit the context window limit while building our Review Summaries feature that summarizes all of the reviews of a specific product on our clients website. Over the past 20 years, however, many products have garnered thousands of reviews that quickly overloaded the LLM context window. In fact, we even have products with millions of reviews that would require immense re-engineering of LLMs to be able to process in one\u00a0prompt.<\/p>\n Even if it was technically feasible, the costs would be quite prohibitive. All LLM providers charge based on the number of input and output tokens. As you approach the context window limits for each product, of which we have millions, we can quickly run up cloud hosting bills in excess of six\u00a0figures.<\/p>\n To ship Review Summaries despite these technical, and financial, limitations, we focused on a rather simple insight into our data: Many reviews say the same thing. In fact, the whole idea of a summary relies on this: review summaries capture the recurring insights, themes, and sentiments of the reviewers. We realized that we can capitalize on this data duplication to reduce the amount of text we need to send to the LLM, saving us from hitting the context window limit and<\/em> reducing the operating cost of our\u00a0system.<\/p>\n To achieve this, we needed to identify segments of text that say the same thing. Such a task is easier said than done: often people use different words or phrases to express the same\u00a0thing.<\/p>\n Fortunately, the task of identifying if text is semantically similar has been an active area of research in the natural language processing field. The work by Agirre et. al. 2013 (SEM 2013 shared task: Semantic Textual Similarity. In Second Joint Conference on Lexical and Computational Semantics<\/em>) even published a human-labeled data of semantically similar sentences known as the STS Benchmark. In it, they ask humans to indicate if textual sentences are semantically similar or dissimilar on a scale of 1\u20135, as illustrated in the table below (from Cer et. al., SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation<\/em><\/a>):<\/p>\n The STSBenchmark dataset is often used to evaluate how well a text embedding model can associate semantically similar sentences in its high-dimensional space. Specifically, Pearson\u2019s correlation is used to measure how well the embedding model represents the human judgements.<\/p>\n Thus, we can use such an embedding model to identify semantically similar phrases from product reviews, and then remove repeated phrases before sending them to the\u00a0LLM.<\/p>\n Our approach is as\u00a0follows:<\/p>\n This may seem straightforward when written in a bulleted list, but there were some devils in the details we had to sort out before we could trust this approach.<\/p>\n First, we had to ensure the model we used effectively embedded text in a space where semantically similar sentences are close, and semantically dissimilar ones are far away. To do this, we simply used the STS benchmark dataset and computed the Pearson correlation for the models we desired to consider. We use AWS as a cloud provider, so naturally we wanted to evaluate their Titan Text Embedding<\/a> models.<\/p>\n Below is a table showing the Pearson\u2019s correlation on the STS Benchmark for different Titan Embedding models:<\/p>\nIntroduction<\/h3>\n
The Problem<\/h3>\n
Our Approach<\/h3>\n
<\/figure>\n
\n
Embedding Model Evaluation<\/h3>\n