Exploring the Power of Lamini Memory\u00a0Tuning<\/h4>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2A8T5JV9SpEDjyLaKQPLTngw.jpeg?ssl=1\")
Image generated by DALL-E\u00a03<\/figcaption><\/figure>\nAccuracy is often critical for LLM applications, especially in cases such as API calling or summarisation of financial reports. Fortunately, there are ways to enhance precision. The best practices to improve accuracy include the following steps:<\/p>\n
\n- You can start simply with prompt engineering techniques<\/strong>\u200a\u2014\u200aadding more detailed instructions, using few-shot prompting, or asking the model to think step-by-step.<\/li>\n
- If accuracy is still insufficient, you can incorporate a self-reflection step<\/strong>, for example, to return errors from the API calls and ask the LLM to correct mistakes.<\/li>\n
- The next option is to provide the most relevant context to the LLM using RAG (Retrieval-Augmented Generation)<\/strong> to boost precision further.<\/li>\n<\/ul>\n
We\u2019ve explored this approach in my previous TDS article, \u201cFrom Prototype to Production: Enhancing LLM Accuracy\u201d<\/em><\/a>.<\/em> In that project, we built an SQL Agent and went from 0% valid SQL queries to 70% accuracy. However, there are limits to what we can achieve with prompt. To break through this barrier and reach the next frontier of accuracy, we need to adopt more advanced techniques.<\/p>\nThe most promising option is fine-tuning. With fine-tuning, we can move from relying solely on information in prompts to embedding additional information directly into the model\u2019s\u00a0weights.<\/p>\n
Fine-tuning<\/h3>\n
Let\u2019s start by understanding what fine-tuning is. Fine-tuning is the process of refining pre-trained models by training them on smaller, task-specific datasets to enhance their performance in particular applications. Basic models are initially trained on vast amounts of data, which allows them to develop a broad understanding of language. Fine-tuning, however, tailors these models to specialized tasks, transforming them from general-purpose systems into highly targeted tools. For example, instruction fine-tuning taught GPT-2 to chat and follow instructions, and that\u2019s how ChatGPT\u00a0emerged.<\/p>\n
Basic LLMs are initially trained to predict the next token based on vast text corpora. Fine-tuning typically adopts a supervised approach, where the model is presented with specific questions and corresponding answers, allowing it to adjust its weights to improve accuracy.<\/p>\n
Historically, fine-tuning required updating all model weights, a method known as full fine-tuning. This process was computationally expensive since it required storing all the model weights, states, gradients and forward activations in memory. To address these challenges, parameter-efficient fine-tuning techniques were introduced. PEFT methods update only the small set of the model parameters while keeping the rest frozen. Among these methods, one of the most widely adopted is LoRA<\/a> (Low-Rank Adaptation), which significantly reduces the computational cost without compromising performance.<\/p>\nPros &\u00a0cons<\/h4>\n
Before considering fine-tuning, it\u2019s essential to weigh its advantages and limitations.<\/p>\n
Advantages:<\/strong><\/p>\n\n- Fine-tuning enables the model to learn and retain significantly more information than can be provided through prompts\u00a0alone.<\/li>\n
- It usually gives higher accuracy, often exceeding 90%.<\/li>\n
- During inference, it can reduce costs by enabling the use of smaller, task-specific models instead of larger, general-purpose ones.<\/li>\n
- Fine-tuned small models can often be deployed on-premises, eliminating reliance on cloud providers such as OpenAI or Anthropic. This approach reduces costs, enhances privacy, and minimizes dependency on external infrastructure.<\/li>\n<\/ul>\n
Disadvantages:<\/strong><\/p>\n\n- Fine-tuning requires upfront investments for model training and data preparation.<\/li>\n
- It requires specific technical knowledge and may involve a steep learning\u00a0curve.<\/li>\n
- The quality of results depends heavily on the availability of high-quality training\u00a0data.<\/li>\n<\/ul>\n
Since this project is focused on gaining knowledge, we will proceed with fine-tuning. However, in real-world scenarios, it\u2019s important to evaluate whether the benefits of fine-tuning justify all the associated costs and\u00a0efforts.<\/p>\n
Execution<\/h4>\n
The next step is to plan how we will approach fine-tuning. After listening to the \u201cImproving Accuracy of LLM Applications\u201d<\/a> course, I\u2019ve decided to try the Lamini platform<\/a> for the following reasons:<\/p>\n\n- It offers a simple one-line API call to fine-tune the model. It\u2019s especially convenient since we\u2019re just starting to learn a new technique.<\/li>\n
- Although it\u2019s not free and can be quite expensive for toy projects (at $1 per tuning step), they offer free credits upon registration, which are sufficient for initial\u00a0testing.<\/li>\n
- Lamini has implemented a new approach, Lamini Memory Tuning, which promises zero loss of factual accuracy while preserving general capabilities. This is a significant claim, and it\u2019s worth testing out. We will discuss this approach in more detail\u00a0shortly.<\/li>\n<\/ul>\n
Of course, there are lots of other fine-tuning options you can consider:<\/p>\n
\n- \nThe Llama documentation<\/a> provides numerous recipes for fine-tuning, which can be executed on a cloud server or even locally for smaller\u00a0models.<\/li>\n
- There are many step-by-step guides available online, including the tutorial on how to fine-tune Llama on Kaggle from DataCamp<\/a>.<\/li>\n
- You can fine-tune not only open-sourced models. OpenAI also offers<\/a> the capability to fine-tune their\u00a0models.<\/li>\n<\/ul>\n
Lamini Memory\u00a0Tuning<\/h4>\n
As I mentioned earlier, Lamini released a new approach to fine-tuning, and I believe it\u2019s worth discussing it in more\u00a0detail.<\/p>\n
Lamini introduced the Mixture of Memory Experts (MoME) approach, which enables LLMs to learn a vast amount of factual information with almost zero loss, all while maintaining generalization capabilities and requiring a feasible amount of computational resources.<\/p>\n
To achieve this, Lamini extended a pre-trained LLM by adding a large number (on the order of 1 million) of LoRA adapters along with a cross-attention layer. Each LoRA adapter is a memory expert, functioning as a type of memory for the model. These memory experts specialize in different aspects, ensuring that the model retains faithful and accurate information from the data it was tuned on. Inspired by information retrieval, these million memory experts are equivalent to indices from which the model intelligently retrieves and\u00a0routes.<\/p>\n
At inference time, the model retrieves a subset of the most relevant experts at each layer and merges back into the base model to generate a response to the user\u00a0query.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2A34DX-ZD4n-w_kJY0rpc47A.png?ssl=1\")
Figure from the paper by Li et al. 2024 |\u00a0source<\/a><\/figcaption><\/figure>\nLamini Memory Tuning is said<\/a> to be capable of achieving 95% accuracy. The key difference from traditional instruction fine-tuning is that instead of optimizing for average error across all tasks, this approach focuses on achieving zero error for the facts the model is specifically trained to remember.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AXcTs2BwiKVu0CDnY093_oA.png?ssl=1\")
Figure from the paper by Li et al. 2024 |\u00a0source<\/a><\/figcaption><\/figure>\nSo, this approach allows an LLM to preserve its ability to generalize with average error on everything else while recalling the important facts nearly perfectly.<\/p>\n
For further details, you can refer to the research paper \u201cBanishing LLM Hallucinations Requires Rethinking Generalization\u201d<\/a> by Li et al.\u00a0(2024)<\/p><\/blockquote>\nLamini Memory Tuning holds great promise\u200a\u2014\u200alet\u2019s see if it delivers on its potential in practice.<\/p>\n
Setup<\/h3>\n
As always, let\u2019s begin by setting everything up. As we discussed, we\u2019ll be using Lamini<\/a> to fine-tune Llama, so the first step is to install the Lamini\u00a0package.<\/p>\npip install lamini<\/pre>\nAdditionally, we need to set up the Lamini API Key on their website<\/a> and specify it as an environment variable.<\/p>\nexport LAMINI_API_KEY=\"<YOUR-LAMINI-API-KEY>\"
<\/pre>\nAs I mentioned above, we will be improving the SQL Agent, so we need a database. For this example, we\u2019ll continue using ClickHouse, but feel free to choose any database that suits your needs. You can find more details on the ClickHouse setup and the database schema in the previous\u00a0article<\/a>.<\/p>\nCreating a training\u00a0dataset<\/h3>\n
To fine-tune an LLM, we first need a dataset\u200a\u2014\u200ain our case, a set of pairs of questions and answers (SQL queries). The task of putting together a dataset might seem daunting, but luckily, we can leverage LLMs to do\u00a0it.<\/p>\n
The key factors to consider while preparing the\u00a0dataset:<\/p>\n
\n- \nThe quality of the data<\/strong> is crucial, as we will ask the model to remember these\u00a0facts.<\/li>\n
- \nDiversity<\/strong> in the examples is important so that a model can learn how to handle different cases.<\/li>\n
- It\u2019s preferable to use real data<\/strong> rather than synthetically generated data since it better represents real-life questions.<\/li>\n
- The usual minimum size for a fine-tuning dataset is around 1,000 examples<\/strong>, but the more high-quality data, the\u00a0better.<\/li>\n<\/ul>\n
Generating examples<\/h4>\n
All the information required to create question-and-answer pairs is present in the database schema, so it will be a feasible task for an LLM to generate examples. Additionally, I have a representative set<\/a> of Q&A pairs that I used for RAG approach, which we can present to the LLM as examples of valid queries (using the few-shot prompting technique). Let\u2019s load the RAG\u00a0dataset.<\/p>\n# loading a set of examples
with open('rag_set.json', 'r') as f:
rag_set = json.loads(f.read())
rag_set_df = pd.DataFrame(rag_set)
rag_set_df['qa_fmt'] = list(map(
lambda x, y: \"question: %s, sql_query: %s\" % (x, y),
rag_set_df.question,
rag_set_df.sql_query
))<\/pre>\nThe idea is to iteratively provide the LLM with the schema information and a set of random examples (to ensure diversity in the questions) and ask it to generate a new, similar, but different Q&A\u00a0pair.<\/p>\n
Let\u2019s create a system prompt that includes all the necessary details about the database\u00a0schema.<\/p>\n
generate_dataset_system_prompt = '''
You are a senior data analyst with more than 10 years of experience writing complex SQL queries.
There are two tables in the database you're working with with the following schemas.
Table: ecommerce.users
Description: customers of the online shop
Fields:
- user_id (integer) - unique identifier of customer, for example, 1000004 or 3000004
- country (string) - country of residence, for example, \"Netherlands\" or \"United Kingdom\"
- is_active (integer) - 1 if customer is still active and 0 otherwise
- age (integer) - customer age in full years, for example, 31 or 72
Table: ecommerce.sessions
Description: sessions for online shop
Fields:
- user_id (integer) - unique identifier of customer, for example, 1000004 or 3000004
- session_id (integer) - unique identifier of session, for example, 106 or 1023
- action_date (date) - session start date, for example, \"2021-01-03\" or \"2024-12-02\"
- session_duration (integer) - duration of session in seconds, for example, 125 or 49
- os (string) - operation system that customer used, for example, \"Windows\" or \"Android\"
- browser (string) - browser that customer used, for example, \"Chrome\" or \"Safari\"
- is_fraud (integer) - 1 if session is marked as fraud and 0 otherwise
- revenue (float) - income in USD (the sum of purchased items), for example, 0.0 or 1506.7
Write a query in ClickHouse SQL to answer the following question.
Add \"format TabSeparatedWithNames\" at the end of the query to get data from ClickHouse database in the right format.
'''<\/pre>\nThe next step is to create a template for the user\u00a0query.<\/p>\n
generate_dataset_qa_tmpl = '''
Considering the following examples, please, write question
and SQL query to answer it, that is similar but different to provided below.
Examples of questions and SQL queries to answer them:
{examples}
'''<\/pre>\nSince we need a high-quality dataset, I prefer using a more advanced model\u200a\u2014\u200aGPT-4o\u2014 rather than Llama. As usual, I\u2019ll initialize the model and create a dummy tool for structured output.<\/p>\n
from langchain_core.tools import tool
@tool
def generate_question_and_answer(comments: str, question: str, sql_query: str) -> str:
\"\"\"Returns the new question and SQL query
Args:
comments (str): 1-2 sentences about the new question and answer pair,
question (str): new question
sql_query (str): SQL query in ClickHouse syntax to answer the question
\"\"\"
pass
from langchain_openai import ChatOpenAI
generate_qa_llm = ChatOpenAI(model=\"gpt-4o\", temperature = 0.5)
.bind_tools([generate_question_and_answer])<\/pre>\nNow, let\u2019s combine everything into a function that will generate a Q&A pair and create a set of examples.<\/p>\n
# helper function to combine system + user prompts
def get_openai_prompt(question, system):
messages = [
(\"system\", system),
(\"human\", question)
]
return messages
def generate_qa():
# selecting 3 random examples
sample_set_df = rag_set_df.sample(3)
examples = 'nn'.join(sample_set_df.qa_fmt.values)
# constructing prompt
prompt = get_openai_prompt(
generate_dataset_qa_tmpl.format(examples = examples),
generate_dataset_system_prompt)
# calling LLM
qa_res = generate_qa_llm.invoke(prompt)
try:
rec = qa_res.tool_calls[0]['args']
rec['examples'] = examples
return rec
except:
pass
# executing function
qa_tmp = []
for i in tqdm.tqdm(range(2000)):
qa_tmp.append(generate_qa())
new_qa_df = pd.DataFrame(qa_tmp)<\/pre>\nI generated 2,000 examples, but in reality, I used a much smaller dataset for this toy project. Therefore, I recommend limiting the number of examples to\u00a0200\u2013300.<\/p>\n
Cleaning the\u00a0dataset<\/h4>\n
As we know, \u201cgarbage in, garbage out<\/em>\u201d, so an essential step before fine-tuning is cleaning the data generated by the\u00a0LLM.<\/p>\nThe first\u200a\u2014\u200aand most obvious\u200a\u2014\u200acheck is to ensure that each SQL query is\u00a0valid.<\/p>\n
def is_valid_output(s):
if s.startswith('Database returned the following error:'):
return 'error'
if len(s.strip().split('n')) >= 1000:
return 'too many rows'
return 'ok'
new_qa_df['output'] = new_qa_df.sql_query.map(get_clickhouse_data)
new_qa_df['is_valid_output'] = new_qa_df.output.map(is_valid_output)<\/pre>\nThere are no invalid SQL queries, but some questions return over 1,000\u00a0rows.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AHXCZDXwbv4QMf223K-SikA.png?ssl=1\")
<\/figure>\nAlthough these cases are valid, we\u2019re focusing on an OLAP<\/a> scenario with aggregated stats, so I\u2019ve retained only queries that return 100 or fewer\u00a0rows.<\/p>\nnew_qa_df['output_rows'] = new_qa_df.output.map(
lambda x: len(x.strip().split('n')))
filt_new_qa_df = new_qa_df[new_qa_df.output_rows <= 100]<\/pre>\nI also eliminated cases with empty output\u200a\u2014\u200aqueries that return no rows or only the\u00a0header.<\/p>\n
filt_new_qa_df = filt_new_qa_df[filt_new_qa_df.output_rows > 1]<\/pre>\nAnother important check is for duplicate questions. The same question with different answers could confuse the model, as it won\u2019t be able to tune to both solutions simultaneously. And in fact, we have such\u00a0cases.<\/p>\n
filt_new_qa_df = filt_new_qa_df[['question', 'sql_query']].drop_duplicates()
filt_new_qa_df['question'].value_counts().head(10)<\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AffXDz59QnB8IcNOf8VduIg.png?ssl=1\")
<\/figure>\nTo resolve these duplicates, I\u2019ve kept only one answer for each question.<\/p>\n
filt_new_qa_df = filt_new_qa_df.drop_duplicates('question') <\/pre>\nAlthough I generated around 2,000 examples, I\u2019ve decided to use a smaller dataset of 200 question-and-answer pairs. Fine-tuning with a larger dataset would require more tuning steps and be more expensive.<\/p>\n
sample_dataset_df = pd.read_csv('small_sample_for_finetuning.csv', sep = 't')<\/pre>\nYou can find the final training dataset on\u00a0GitHub<\/a>.<\/p><\/blockquote>\nNow that our training dataset is ready, we can move on to the most exciting part\u200a\u2014\u200afine-tuning.<\/p>\n
Fine-tuning<\/h3>\nThe first iteration<\/h4>\n
The next step is to generate the sets of requests and responses for the LLM that we will use to fine-tune the\u00a0model.<\/p>\n
Since we\u2019ll be working with the Llama model, let\u2019s create a helper function to construct a prompt for\u00a0it.<\/p>\n
def get_llama_prompt(user_message, system_message=\"\"):
system_prompt = \"\"
if system_message != \"\":
system_prompt = (
f\"<|start_header_id|>system<|end_header_id|>nn{system_message}\"
f\"<|eot_id|>\"
)
prompt = (f\"<|begin_of_text|>{system_prompt}\"
f\"<|start_header_id|>user<|end_header_id|>nn\"
f\"{user_message}\"
f\"<|eot_id|>\"
f\"<|start_header_id|>assistant<|end_header_id|>nn\"
)
return prompt <\/pre>\nFor requests, we will use the following system prompt, which includes all the necessary information about the data\u00a0schema.<\/p>\n
generate_query_system_prompt = '''
You are a senior data analyst with more than 10 years of experience writing complex SQL queries.
There are two tables in the database you're working with with the following schemas.
Table: ecommerce.users
Description: customers of the online shop
Fields:
- user_id (integer) - unique identifier of customer, for example, 1000004 or 3000004
- country (string) - country of residence, for example, \"Netherlands\" or \"United Kingdom\"
- is_active (integer) - 1 if customer is still active and 0 otherwise
- age (integer) - customer age in full years, for example, 31 or 72
Table: ecommerce.sessions
Description: sessions of usage the online shop
Fields:
- user_id (integer) - unique identifier of customer, for example, 1000004 or 3000004
- session_id (integer) - unique identifier of session, for example, 106 or 1023
- action_date (date) - session start date, for example, \"2021-01-03\" or \"2024-12-02\"
- session_duration (integer) - duration of session in seconds, for example, 125 or 49
- os (string) - operation system that customer used, for example, \"Windows\" or \"Android\"
- browser (string) - browser that customer used, for example, \"Chrome\" or \"Safari\"
- is_fraud (integer) - 1 if session is marked as fraud and 0 otherwise
- revenue (float) - income in USD (the sum of purchased items), for example, 0.0 or 1506.7
Write a query in ClickHouse SQL to answer the following question.
Add \"format TabSeparatedWithNames\" at the end of the query to get data from ClickHouse database in the right format.
Answer questions following the instructions and providing all the needed information and sharing your reasoning.
'''<\/pre>\nLet\u2019s create the responses in the format suitable for Lamini fine-tuning. We need to prepare a list of dictionaries with input and output\u00a0keys.<\/p>\n
formatted_responses = []
for rec in sample_dataset_df.to_dict('records'):
formatted_responses.append(
{
'input': get_llama_prompt(rec['question'],
generate_query_system_prompt),
'output': rec['sql_query']
}
)<\/pre>\nNow, we are fully prepared for fine-tuning. We just need to select a model and initiate the process. We will be fine-tuning the Llama 3.1 8B\u00a0model.<\/p>\n
from lamini import Lamini
llm = Lamini(model_name=\"meta-llama\/Meta-Llama-3.1-8B-Instruct\")
finetune_args = {
\"max_steps\": 50,
\"learning_rate\": 0.0001
}
llm.train(
data_or_dataset_id=formatted_responses,
finetune_args=finetune_args,
) <\/pre>\nWe can specify several hyperparameters, and you can find all the details in the Lamini documentation<\/a>. For now, I\u2019ve passed only the most essential ones to the function:<\/p>\n\n- max_steps: This determines the number of tuning steps. The documentation recommends using 50 steps for experimentation to get initial results without spending too much\u00a0money.<\/li>\n
- learning_rate: This parameter determines the step size of each iteration while moving toward a minimum of a loss function (Wikipedia<\/a>). The default is 0.0009, but based on the guidance<\/a>, I\u2019ve decided to use a smaller\u00a0value.<\/li>\n<\/ul>\n
Now, we just need to wait for 10\u201315 minutes while the model trains, and then we can test\u00a0it.<\/p>\n
finetuned_llm = Lamini(model_name='<model_id>')
# you can find Model ID in the Lamini interface
question = '''How many customers made purchase in December 2024?'''
prompt = get_llama_prompt(question, generate_query_system_prompt)
finetuned_llm.generate(prompt, max_new_tokens=200)
# select uniqExact(s.user_id) as customers
# from ecommerce.sessions s join ecommerce.users u
# on s.user_id = u.user_id
# where (toStartOfMonth(action_date) = '2024-12-01') and (revenue > 0)
# format TabSeparatedWithNames<\/pre>\nIt\u2019s worth noting that we\u2019re using Lamini for inference as well and will have to pay for it. You can find up-to-date information about the costs\u00a0here<\/a>.<\/p>\nAt first glance, the result looks promising, but we need a more robust accuracy evaluation to confirm\u00a0it.<\/p>\n
Additionally, it\u2019s worth noting that since we\u2019ve fine-tuned the model for our specific task, it now consistently returns SQL queries, meaning we may no longer need to use tool calls for structured output.<\/p>\n
Evaluating the\u00a0quality<\/h4>\n
We\u2019ve discussed LLM accuracy evaluation in detail in my previous article<\/a>, so here I\u2019ll provide a brief\u00a0recap.<\/p>\nWe use a golden set of question-and-answer pairs to evaluate the model\u2019s quality. Since this is a toy example, I\u2019ve limited the set to just 10 pairs, which you can review on\u00a0GitHub<\/a>.<\/p>\nThe evaluation process consists of two\u00a0parts:<\/p>\n
\n- \nSQL Query Validity<\/strong>: First, we check that the SQL query is valid, meaning ClickHouse doesn\u2019t return errors during execution.<\/li>\n
- \nQuery Correctness<\/strong>: Next, we ensure that the generated query is correct. We compare the outputs of the generated and true queries using LLMs to verify that they provide semantically identical results.<\/li>\n<\/ul>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2Atk4l1_dm-xo7zGNwsIUEhw.png?ssl=1\")
<\/figure>\nThe initial results are far from ideal, but they are significantly better than the base Llama model (which produced zero valid SQL queries). Here\u2019s what we\u00a0found:<\/p>\n
\n- ClickHouse returned errors for two\u00a0queries.<\/li>\n
- Three queries were executed, but the results were incorrect.<\/li>\n
- Five queries were\u00a0correct.<\/li>\n<\/ul>\n
No surprises\u200a\u2014\u200athere\u2019s no silver bullet, and it\u2019s always an iterative process. Let\u2019s investigate what went\u00a0wrong.<\/p>\n
Diving into the\u00a0errors<\/h4>\n
The approach is straightforward. Let\u2019s examine the errors one by one to understand why we got these results and how we can fix them. We\u2019ll start with the first unsuccessful example.<\/p>\n
Question: <\/strong>Which country had the highest number of first-time users in\u00a02024?<\/p>\nGolden query:<\/strong><\/p>\nselect
country,
count(distinct user_id) as users
from
(
select user_id, min(action_date) as first_date
from ecommerce.sessions
group by user_id
having toStartOfYear(first_date) = '2024-01-01'
) as t
inner join ecommerce.users as u
on t.user_id = u.user_id
group by country
order by users desc
limit 1
format TabSeparatedWithNames <\/pre>\nGenerated query:<\/strong><\/p>\nselect
country,
count(distinct u.user_id) as first_time_users
from ecommerce.sessions s
join ecommerce.users u
on s.user_id = u.user_id
where (toStartOfYear(action_date) = '2024-01-01')
and (s.session_id = 1)
group by country
order by first_time_users desc
limit 1
format TabSeparatedWithNames <\/pre>\nThe query is valid, but it returns an incorrect result. The issue lies in the model\u2019s assumption that the first session for each user will always have session_id = 1. Since Lamini Memory Tuning allows the model to learn facts from the training data, let\u2019s investigate why the model made this assumption. Likely, it\u2019s in our\u00a0data.<\/p>\n
Let\u2019s review all the examples that mention first. I\u2019ll use broad and simple search criteria to get a high-level view.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2ANaLd2eOIyQARpDIVduMHbg.png?ssl=1\")
<\/figure>\nAs we can see, there are no examples mentioning first-time users\u200a\u2014\u200aonly references to the first quarter. It\u2019s no surprise that the model wasn\u2019t able to capture this concept. The solution is straightforward: we just need to add a set of examples with questions and answers specifically about first-time users.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2Akir1l5lIGb59hwSucwh9zw.png?ssl=1\")
<\/figure>\nLet\u2019s move on to the next problematic case.<\/p>\n
Question: <\/strong>What was the fraud rate in 2023, expressed as a percentage?<\/p>\nGolden query:<\/strong><\/p>\nselect
100*uniqExactIf(user_id, is_fraud = 1)\/uniqExact(user_id) as fraud_rate
from ecommerce.sessions
where (toStartOfYear(action_date) = '2023-01-01')
format TabSeparatedWithNames <\/pre>\nGenerated query:<\/strong><\/p>\nselect
100*countIf(is_fraud = 1)\/count() as fraud_rate
from ecommerce.sessions
where (toStartOfYear(action_date) = '2023-01-01')
format TabSeparatedWithNames <\/pre>\nHere\u2019s another misconception: we assumed that the fraud rate is based on the share of users, while the model calculated it based on the share of sessions.<\/p>\n
Let\u2019s check the examples related to the fraud rate in the data. There are two cases: one calculates the share of users, while the other calculates the share of sessions.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AuK5OtHQ_1JdqxDUzpEx9Hg.png?ssl=1\")
<\/figure>\nTo fix this issue, I corrected the incorrect answer and added more accurate examples involving fraud rate calculations.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2Ae9Lwq0X9ChU0DZ3cJcIBqA.png?ssl=1\")
<\/figure>\nI\u2019d like to discuss another incorrect case, as it will highlight an important aspect of the process of resolving these\u00a0issues.<\/p>\n
Question: <\/strong>What are the median and interquartile range (IQR) of purchase revenue for each\u00a0country?<\/p>\nGolden query:<\/strong><\/p>\nselect
country,
median(revenue) as median_revenue,
quantile(0.25)(revenue) as percentile_25_revenue,
quantile(0.75)(revenue) as percentile_75_revenue
from ecommerce.sessions AS s
inner join ecommerce.users AS u
on u.user_id = s.user_id
where (revenue > 0)
group by country
format TabSeparatedWithNames <\/pre>\nGenerated query:<\/strong><\/p>\nselect
country,
median(revenue) as median_revenue,
quantile(0.25)(revenue) as percentile_25_revenue,
quantile(0.75)(revenue) as percentile_75_revenue
from ecommerce.sessions s join ecommerce.users u
on s.user_id = u.user_id
group by country
format TabSeparatedWithNames <\/pre>\nWhen inspecting the problem, it\u2019s crucial to focus on the model\u2019s misconceptions or incorrect assumptions. For example, in this case, there may be a temptation to add examples similar to those in the golden dataset, but that would be too specific. Instead, we should address the actual root cause of the model\u2019s misconception:<\/p>\n
\n- It understood the concepts of median and IQR quite\u00a0well.<\/li>\n
- The split by country is also\u00a0correct.<\/li>\n
- However, it misinterpreted the concept of \u201cpurchase revenue\u201d, including sessions where there was no purchase at all (revenue =\u00a00).<\/li>\n<\/ul>\n
So, we need to ensure that our datasets contain enough information related to purchase revenue. Let\u2019s take a look at what we have now. There\u2019s only one example, and it\u2019s incorrect.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AG2egFUDVmXVIn-c028mhFA.png?ssl=1\")
<\/figure>\nLet\u2019s fix this example and add more cases of purchase revenue calculations.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2Ar5U3h_osGk0t-bVadGclSg.png?ssl=1\")
<\/figure>\nUsing a similar approach, I\u2019ve added more examples for the two remaining incorrect queries and compiled an updated, cleaned version of the training dataset. You can find it on GitHub<\/a>. With this, our data is ready to the next iteration.<\/p>\nAnother iteration of fine-tuning<\/h4>\n
Before diving into fine-tuning, it\u2019s essential to double-check the quality of the training dataset by ensuring that all SQL queries are\u00a0valid.<\/p>\n
clean_sample_dataset_df = pd.read_csv(
'small_sample_for_finetuning_cleaned.csv', sep = 't',
on_bad_lines = 'warn')
clean_sample_dataset_df['output'] = clean_sample_dataset_df.sql_query
.map(lambda x: get_clickhouse_data(str(x)))
clean_sample_dataset_df['is_valid_output'] = clean_sample_dataset_df['output']
.map(is_valid_output)
print(clean_sample_dataset_df.is_valid_output.value_counts())
# is_valid_output
# ok 241
clean_formatted_responses = []
for rec in clean_sample_dataset_df.to_dict('records'):
clean_formatted_responses.append(
{
'input': get_llama_prompt(
rec['question'],
generate_query_system_prompt),
'output': rec['sql_query']
}
)<\/pre>\nNow that we\u2019re confident in the data, we can proceed with fine-tuning. This time, I\u2019ve decided to train it for 150 steps to achieve better accuracy.<\/p>\n
finetune_args = {
\"max_steps\": 150,
\"learning_rate\": 0.0001
}
llm = Lamini(model_name=\"meta-llama\/Meta-Llama-3.1-8B-Instruct\")
llm.train(
data_or_dataset_id=clean_formatted_responses,
finetune_args=finetune_args
)<\/pre>\nAfter waiting a bit longer than last time, we now have a new fine-tuned model with nearly zero loss after 150 tuning\u00a0steps.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AxlS07_G_1hBgpAbz0qSNPA.png?ssl=1\")
<\/figure>\nWe can run the evaluation again and see much better results. So, our approach is\u00a0working.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2A0ebDI1HlKcgoiOMMKVMpLQ.png?ssl=1\")
<\/figure>\nThe result is astonishing, but it\u2019s still worth examining the incorrect example to understand what went wrong. We got an incorrect result for the question we discussed earlier: \u201cWhat are the median and interquartile range (IQR) of purchase revenue for each country?\u201d<\/em> However, this time, the model generated a query that is exactly identical to the one in the golden\u00a0set.<\/p>\nselect
country,
median(revenue) as median_revenue,
quantile(0.25)(revenue) as percentile_25_revenue,
quantile(0.75)(revenue) as percentile_75_revenue
from ecommerce.sessions AS s
inner join ecommerce.users AS u
on u.user_id = s.user_id
where (s.revenue > 0)
group by country
format TabSeparatedWithNames<\/pre>\nSo, the issue actually lies in our evaluation. In fact, if you try to execute this query multiple times, you\u2019ll notice that the results are slightly different each time. The root cause is that the quantile function in ClickHouse<\/a> computes approximate values using reservoir sampling, which is why we\u2019re seeing varying results. We could have used quantileExact instead to get more consistent numbers.<\/p>\nThat said, this means that fine-tuning has allowed us to achieve 100% accuracy. Even though our toy golden dataset consists of just 10 questions, this is a tremendous achievement. We\u2019ve progressed all the way from zero valid queries with vanilla Llama to 70% accuracy with RAG and self-reflection, and now, thanks to Lamini Memory Tuning, we\u2019ve reached 100% accuracy.<\/p>\n
You can find the full code on\u00a0GitHub<\/a>.<\/p><\/blockquote>\nSummary<\/h3>\n
In this article, we continued exploring techniques to improve LLM accuracy:<\/p>\n
\n- After trying RAG and self-reflection in the previous article<\/a>, we moved on to a more advanced technique\u200a\u2014\u200afine-tuning.<\/li>\n
- We experimented with Memory Tuning developed by Lamini, which enables a model to remember a large volume of facts with near-zero errors.<\/li>\n
- In our example, Memory Tuning performed exceptionally well, and we achieved 100% accuracy on our evaluation set of 10 questions.<\/li>\n<\/ul>\n
Thank you a lot for reading this article. I hope this article was insightful for you. If you have any follow-up questions or comments, please leave them in the comments\u00a0section.<\/p><\/blockquote>\n
Reference<\/h3>\nAll the images are produced by the author unless otherwise stated.<\/p><\/blockquote>\n
This article is inspired by the \u201cImproving Accuracy of LLM Applications\u201d<\/a> short course from DeepLearning.AI.<\/p>\nDisclaimer: <\/strong>I am not affiliated with Lamini in any way. The views expressed in this article are solely my own, based on independent testing and evaluation of the Lamini platform. This post is intended for educational purposes and does not constitute an endorsement of any specific tool or\u00a0service.<\/em><\/p>\n![\"\"](\"https:\/\/medium.com\/_\/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb2491a740d4\")
<\/p>\n
\nThe Next Frontier in LLM Accuracy<\/a> was originally published in Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\n
Accuracy is often critical for LLM applications, especially in cases such as API calling or summarisation of financial reports. Fortunately, there are ways to enhance precision. The best practices to improve accuracy include the following steps:<\/p>\n
- \n
- You can start simply with prompt engineering techniques<\/strong>\u200a\u2014\u200aadding more detailed instructions, using few-shot prompting, or asking the model to think step-by-step.<\/li>\n
- If accuracy is still insufficient, you can incorporate a self-reflection step<\/strong>, for example, to return errors from the API calls and ask the LLM to correct mistakes.<\/li>\n
- The next option is to provide the most relevant context to the LLM using RAG (Retrieval-Augmented Generation)<\/strong> to boost precision further.<\/li>\n<\/ul>\n
We\u2019ve explored this approach in my previous TDS article, \u201cFrom Prototype to Production: Enhancing LLM Accuracy\u201d<\/em><\/a>.<\/em> In that project, we built an SQL Agent and went from 0% valid SQL queries to 70% accuracy. However, there are limits to what we can achieve with prompt. To break through this barrier and reach the next frontier of accuracy, we need to adopt more advanced techniques.<\/p>\n
The most promising option is fine-tuning. With fine-tuning, we can move from relying solely on information in prompts to embedding additional information directly into the model\u2019s\u00a0weights.<\/p>\n
Fine-tuning<\/h3>\n
Let\u2019s start by understanding what fine-tuning is. Fine-tuning is the process of refining pre-trained models by training them on smaller, task-specific datasets to enhance their performance in particular applications. Basic models are initially trained on vast amounts of data, which allows them to develop a broad understanding of language. Fine-tuning, however, tailors these models to specialized tasks, transforming them from general-purpose systems into highly targeted tools. For example, instruction fine-tuning taught GPT-2 to chat and follow instructions, and that\u2019s how ChatGPT\u00a0emerged.<\/p>\n
Basic LLMs are initially trained to predict the next token based on vast text corpora. Fine-tuning typically adopts a supervised approach, where the model is presented with specific questions and corresponding answers, allowing it to adjust its weights to improve accuracy.<\/p>\n
Historically, fine-tuning required updating all model weights, a method known as full fine-tuning. This process was computationally expensive since it required storing all the model weights, states, gradients and forward activations in memory. To address these challenges, parameter-efficient fine-tuning techniques were introduced. PEFT methods update only the small set of the model parameters while keeping the rest frozen. Among these methods, one of the most widely adopted is LoRA<\/a> (Low-Rank Adaptation), which significantly reduces the computational cost without compromising performance.<\/p>\n
Pros &\u00a0cons<\/h4>\n
Before considering fine-tuning, it\u2019s essential to weigh its advantages and limitations.<\/p>\n
Advantages:<\/strong><\/p>\n
- \n
- Fine-tuning enables the model to learn and retain significantly more information than can be provided through prompts\u00a0alone.<\/li>\n
- It usually gives higher accuracy, often exceeding 90%.<\/li>\n
- During inference, it can reduce costs by enabling the use of smaller, task-specific models instead of larger, general-purpose ones.<\/li>\n
- Fine-tuned small models can often be deployed on-premises, eliminating reliance on cloud providers such as OpenAI or Anthropic. This approach reduces costs, enhances privacy, and minimizes dependency on external infrastructure.<\/li>\n<\/ul>\n
Disadvantages:<\/strong><\/p>\n
- \n
- Fine-tuning requires upfront investments for model training and data preparation.<\/li>\n
- It requires specific technical knowledge and may involve a steep learning\u00a0curve.<\/li>\n
- The quality of results depends heavily on the availability of high-quality training\u00a0data.<\/li>\n<\/ul>\n
Since this project is focused on gaining knowledge, we will proceed with fine-tuning. However, in real-world scenarios, it\u2019s important to evaluate whether the benefits of fine-tuning justify all the associated costs and\u00a0efforts.<\/p>\n
Execution<\/h4>\n
The next step is to plan how we will approach fine-tuning. After listening to the \u201cImproving Accuracy of LLM Applications\u201d<\/a> course, I\u2019ve decided to try the Lamini platform<\/a> for the following reasons:<\/p>\n
- \n
- It offers a simple one-line API call to fine-tune the model. It\u2019s especially convenient since we\u2019re just starting to learn a new technique.<\/li>\n
- Although it\u2019s not free and can be quite expensive for toy projects (at $1 per tuning step), they offer free credits upon registration, which are sufficient for initial\u00a0testing.<\/li>\n
- Lamini has implemented a new approach, Lamini Memory Tuning, which promises zero loss of factual accuracy while preserving general capabilities. This is a significant claim, and it\u2019s worth testing out. We will discuss this approach in more detail\u00a0shortly.<\/li>\n<\/ul>\n
Of course, there are lots of other fine-tuning options you can consider:<\/p>\n
- \n
- \nThe Llama documentation<\/a> provides numerous recipes for fine-tuning, which can be executed on a cloud server or even locally for smaller\u00a0models.<\/li>\n
- There are many step-by-step guides available online, including the tutorial on how to fine-tune Llama on Kaggle from DataCamp<\/a>.<\/li>\n
- You can fine-tune not only open-sourced models. OpenAI also offers<\/a> the capability to fine-tune their\u00a0models.<\/li>\n<\/ul>\n
Lamini Memory\u00a0Tuning<\/h4>\n
As I mentioned earlier, Lamini released a new approach to fine-tuning, and I believe it\u2019s worth discussing it in more\u00a0detail.<\/p>\n
Lamini introduced the Mixture of Memory Experts (MoME) approach, which enables LLMs to learn a vast amount of factual information with almost zero loss, all while maintaining generalization capabilities and requiring a feasible amount of computational resources.<\/p>\n
To achieve this, Lamini extended a pre-trained LLM by adding a large number (on the order of 1 million) of LoRA adapters along with a cross-attention layer. Each LoRA adapter is a memory expert, functioning as a type of memory for the model. These memory experts specialize in different aspects, ensuring that the model retains faithful and accurate information from the data it was tuned on. Inspired by information retrieval, these million memory experts are equivalent to indices from which the model intelligently retrieves and\u00a0routes.<\/p>\n
At inference time, the model retrieves a subset of the most relevant experts at each layer and merges back into the base model to generate a response to the user\u00a0query.<\/p>\n
Figure from the paper by Li et al. 2024 |\u00a0source<\/a><\/figcaption><\/figure>\n Lamini Memory Tuning is said<\/a> to be capable of achieving 95% accuracy. The key difference from traditional instruction fine-tuning is that instead of optimizing for average error across all tasks, this approach focuses on achieving zero error for the facts the model is specifically trained to remember.<\/p>\n
Figure from the paper by Li et al. 2024 |\u00a0source<\/a><\/figcaption><\/figure>\n So, this approach allows an LLM to preserve its ability to generalize with average error on everything else while recalling the important facts nearly perfectly.<\/p>\n
For further details, you can refer to the research paper \u201cBanishing LLM Hallucinations Requires Rethinking Generalization\u201d<\/a> by Li et al.\u00a0(2024)<\/p><\/blockquote>\n
Lamini Memory Tuning holds great promise\u200a\u2014\u200alet\u2019s see if it delivers on its potential in practice.<\/p>\n
Setup<\/h3>\n
As always, let\u2019s begin by setting everything up. As we discussed, we\u2019ll be using Lamini<\/a> to fine-tune Llama, so the first step is to install the Lamini\u00a0package.<\/p>\n
pip install lamini<\/pre>\n
Additionally, we need to set up the Lamini API Key on their website<\/a> and specify it as an environment variable.<\/p>\n
export LAMINI_API_KEY=\"<YOUR-LAMINI-API-KEY>\"
<\/pre>\nAs I mentioned above, we will be improving the SQL Agent, so we need a database. For this example, we\u2019ll continue using ClickHouse, but feel free to choose any database that suits your needs. You can find more details on the ClickHouse setup and the database schema in the previous\u00a0article<\/a>.<\/p>\n
Creating a training\u00a0dataset<\/h3>\n
To fine-tune an LLM, we first need a dataset\u200a\u2014\u200ain our case, a set of pairs of questions and answers (SQL queries). The task of putting together a dataset might seem daunting, but luckily, we can leverage LLMs to do\u00a0it.<\/p>\n
The key factors to consider while preparing the\u00a0dataset:<\/p>\n
- \n
- \nThe quality of the data<\/strong> is crucial, as we will ask the model to remember these\u00a0facts.<\/li>\n
- \nDiversity<\/strong> in the examples is important so that a model can learn how to handle different cases.<\/li>\n
- It\u2019s preferable to use real data<\/strong> rather than synthetically generated data since it better represents real-life questions.<\/li>\n
- The usual minimum size for a fine-tuning dataset is around 1,000 examples<\/strong>, but the more high-quality data, the\u00a0better.<\/li>\n<\/ul>\n
Generating examples<\/h4>\n
All the information required to create question-and-answer pairs is present in the database schema, so it will be a feasible task for an LLM to generate examples. Additionally, I have a representative set<\/a> of Q&A pairs that I used for RAG approach, which we can present to the LLM as examples of valid queries (using the few-shot prompting technique). Let\u2019s load the RAG\u00a0dataset.<\/p>\n
# loading a set of examples
with open('rag_set.json', 'r') as f:
rag_set = json.loads(f.read())
rag_set_df = pd.DataFrame(rag_set)
rag_set_df['qa_fmt'] = list(map(
lambda x, y: \"question: %s, sql_query: %s\" % (x, y),
rag_set_df.question,
rag_set_df.sql_query
))<\/pre>\nThe idea is to iteratively provide the LLM with the schema information and a set of random examples (to ensure diversity in the questions) and ask it to generate a new, similar, but different Q&A\u00a0pair.<\/p>\n
Let\u2019s create a system prompt that includes all the necessary details about the database\u00a0schema.<\/p>\n
generate_dataset_system_prompt = '''
You are a senior data analyst with more than 10 years of experience writing complex SQL queries.
There are two tables in the database you're working with with the following schemas.
Table: ecommerce.users
Description: customers of the online shop
Fields:
- user_id (integer) - unique identifier of customer, for example, 1000004 or 3000004
- country (string) - country of residence, for example, \"Netherlands\" or \"United Kingdom\"
- is_active (integer) - 1 if customer is still active and 0 otherwise
- age (integer) - customer age in full years, for example, 31 or 72
Table: ecommerce.sessions
Description: sessions for online shop
Fields:
- user_id (integer) - unique identifier of customer, for example, 1000004 or 3000004
- session_id (integer) - unique identifier of session, for example, 106 or 1023
- action_date (date) - session start date, for example, \"2021-01-03\" or \"2024-12-02\"
- session_duration (integer) - duration of session in seconds, for example, 125 or 49
- os (string) - operation system that customer used, for example, \"Windows\" or \"Android\"
- browser (string) - browser that customer used, for example, \"Chrome\" or \"Safari\"
- is_fraud (integer) - 1 if session is marked as fraud and 0 otherwise
- revenue (float) - income in USD (the sum of purchased items), for example, 0.0 or 1506.7
Write a query in ClickHouse SQL to answer the following question.
Add \"format TabSeparatedWithNames\" at the end of the query to get data from ClickHouse database in the right format.
'''<\/pre>\nThe next step is to create a template for the user\u00a0query.<\/p>\n
generate_dataset_qa_tmpl = '''
Considering the following examples, please, write question
and SQL query to answer it, that is similar but different to provided below.
Examples of questions and SQL queries to answer them:
{examples}
'''<\/pre>\nSince we need a high-quality dataset, I prefer using a more advanced model\u200a\u2014\u200aGPT-4o\u2014 rather than Llama. As usual, I\u2019ll initialize the model and create a dummy tool for structured output.<\/p>\n
from langchain_core.tools import tool
@tool
def generate_question_and_answer(comments: str, question: str, sql_query: str) -> str:
\"\"\"Returns the new question and SQL query
Args:
comments (str): 1-2 sentences about the new question and answer pair,
question (str): new question
sql_query (str): SQL query in ClickHouse syntax to answer the question
\"\"\"
pass
from langchain_openai import ChatOpenAI
generate_qa_llm = ChatOpenAI(model=\"gpt-4o\", temperature = 0.5)
.bind_tools([generate_question_and_answer])<\/pre>\nNow, let\u2019s combine everything into a function that will generate a Q&A pair and create a set of examples.<\/p>\n
# helper function to combine system + user prompts
def get_openai_prompt(question, system):
messages = [
(\"system\", system),
(\"human\", question)
]
return messages
def generate_qa():
# selecting 3 random examples
sample_set_df = rag_set_df.sample(3)
examples = 'nn'.join(sample_set_df.qa_fmt.values)
# constructing prompt
prompt = get_openai_prompt(
generate_dataset_qa_tmpl.format(examples = examples),
generate_dataset_system_prompt)
# calling LLM
qa_res = generate_qa_llm.invoke(prompt)
try:
rec = qa_res.tool_calls[0]['args']
rec['examples'] = examples
return rec
except:
pass
# executing function
qa_tmp = []
for i in tqdm.tqdm(range(2000)):
qa_tmp.append(generate_qa())
new_qa_df = pd.DataFrame(qa_tmp)<\/pre>\nI generated 2,000 examples, but in reality, I used a much smaller dataset for this toy project. Therefore, I recommend limiting the number of examples to\u00a0200\u2013300.<\/p>\n
Cleaning the\u00a0dataset<\/h4>\n
As we know, \u201cgarbage in, garbage out<\/em>\u201d, so an essential step before fine-tuning is cleaning the data generated by the\u00a0LLM.<\/p>\n
The first\u200a\u2014\u200aand most obvious\u200a\u2014\u200acheck is to ensure that each SQL query is\u00a0valid.<\/p>\n
def is_valid_output(s):
if s.startswith('Database returned the following error:'):
return 'error'
if len(s.strip().split('n')) >= 1000:
return 'too many rows'
return 'ok'
new_qa_df['output'] = new_qa_df.sql_query.map(get_clickhouse_data)
new_qa_df['is_valid_output'] = new_qa_df.output.map(is_valid_output)<\/pre>\nThere are no invalid SQL queries, but some questions return over 1,000\u00a0rows.<\/p>\n
<\/figure>\nAlthough these cases are valid, we\u2019re focusing on an OLAP<\/a> scenario with aggregated stats, so I\u2019ve retained only queries that return 100 or fewer\u00a0rows.<\/p>\n
new_qa_df['output_rows'] = new_qa_df.output.map(
lambda x: len(x.strip().split('n')))
filt_new_qa_df = new_qa_df[new_qa_df.output_rows <= 100]<\/pre>\nI also eliminated cases with empty output\u200a\u2014\u200aqueries that return no rows or only the\u00a0header.<\/p>\n
filt_new_qa_df = filt_new_qa_df[filt_new_qa_df.output_rows > 1]<\/pre>\n
Another important check is for duplicate questions. The same question with different answers could confuse the model, as it won\u2019t be able to tune to both solutions simultaneously. And in fact, we have such\u00a0cases.<\/p>\n
filt_new_qa_df = filt_new_qa_df[['question', 'sql_query']].drop_duplicates()
filt_new_qa_df['question'].value_counts().head(10)<\/pre>\n<\/figure>\nTo resolve these duplicates, I\u2019ve kept only one answer for each question.<\/p>\n
filt_new_qa_df = filt_new_qa_df.drop_duplicates('question') <\/pre>\nAlthough I generated around 2,000 examples, I\u2019ve decided to use a smaller dataset of 200 question-and-answer pairs. Fine-tuning with a larger dataset would require more tuning steps and be more expensive.<\/p>\n
sample_dataset_df = pd.read_csv('small_sample_for_finetuning.csv', sep = 't')<\/pre>\nYou can find the final training dataset on\u00a0GitHub<\/a>.<\/p><\/blockquote>\n
Now that our training dataset is ready, we can move on to the most exciting part\u200a\u2014\u200afine-tuning.<\/p>\n
Fine-tuning<\/h3>\n
The first iteration<\/h4>\n
The next step is to generate the sets of requests and responses for the LLM that we will use to fine-tune the\u00a0model.<\/p>\n
Since we\u2019ll be working with the Llama model, let\u2019s create a helper function to construct a prompt for\u00a0it.<\/p>\n
def get_llama_prompt(user_message, system_message=\"\"):
system_prompt = \"\"
if system_message != \"\":
system_prompt = (
f\"<|start_header_id|>system<|end_header_id|>nn{system_message}\"
f\"<|eot_id|>\"
)
prompt = (f\"<|begin_of_text|>{system_prompt}\"
f\"<|start_header_id|>user<|end_header_id|>nn\"
f\"{user_message}\"
f\"<|eot_id|>\"
f\"<|start_header_id|>assistant<|end_header_id|>nn\"
)
return prompt <\/pre>\nFor requests, we will use the following system prompt, which includes all the necessary information about the data\u00a0schema.<\/p>\n
generate_query_system_prompt = '''
You are a senior data analyst with more than 10 years of experience writing complex SQL queries.
There are two tables in the database you're working with with the following schemas.
Table: ecommerce.users
Description: customers of the online shop
Fields:
- user_id (integer) - unique identifier of customer, for example, 1000004 or 3000004
- country (string) - country of residence, for example, \"Netherlands\" or \"United Kingdom\"
- is_active (integer) - 1 if customer is still active and 0 otherwise
- age (integer) - customer age in full years, for example, 31 or 72
Table: ecommerce.sessions
Description: sessions of usage the online shop
Fields:
- user_id (integer) - unique identifier of customer, for example, 1000004 or 3000004
- session_id (integer) - unique identifier of session, for example, 106 or 1023
- action_date (date) - session start date, for example, \"2021-01-03\" or \"2024-12-02\"
- session_duration (integer) - duration of session in seconds, for example, 125 or 49
- os (string) - operation system that customer used, for example, \"Windows\" or \"Android\"
- browser (string) - browser that customer used, for example, \"Chrome\" or \"Safari\"
- is_fraud (integer) - 1 if session is marked as fraud and 0 otherwise
- revenue (float) - income in USD (the sum of purchased items), for example, 0.0 or 1506.7
Write a query in ClickHouse SQL to answer the following question.
Add \"format TabSeparatedWithNames\" at the end of the query to get data from ClickHouse database in the right format.
Answer questions following the instructions and providing all the needed information and sharing your reasoning.
'''<\/pre>\nLet\u2019s create the responses in the format suitable for Lamini fine-tuning. We need to prepare a list of dictionaries with input and output\u00a0keys.<\/p>\n
formatted_responses = []
for rec in sample_dataset_df.to_dict('records'):
formatted_responses.append(
{
'input': get_llama_prompt(rec['question'],
generate_query_system_prompt),
'output': rec['sql_query']
}
)<\/pre>\nNow, we are fully prepared for fine-tuning. We just need to select a model and initiate the process. We will be fine-tuning the Llama 3.1 8B\u00a0model.<\/p>\n
from lamini import Lamini
llm = Lamini(model_name=\"meta-llama\/Meta-Llama-3.1-8B-Instruct\")
finetune_args = {
\"max_steps\": 50,
\"learning_rate\": 0.0001
}
llm.train(
data_or_dataset_id=formatted_responses,
finetune_args=finetune_args,
) <\/pre>\nWe can specify several hyperparameters, and you can find all the details in the Lamini documentation<\/a>. For now, I\u2019ve passed only the most essential ones to the function:<\/p>\n
- \n
- max_steps: This determines the number of tuning steps. The documentation recommends using 50 steps for experimentation to get initial results without spending too much\u00a0money.<\/li>\n
- learning_rate: This parameter determines the step size of each iteration while moving toward a minimum of a loss function (Wikipedia<\/a>). The default is 0.0009, but based on the guidance<\/a>, I\u2019ve decided to use a smaller\u00a0value.<\/li>\n<\/ul>\n
Now, we just need to wait for 10\u201315 minutes while the model trains, and then we can test\u00a0it.<\/p>\n
finetuned_llm = Lamini(model_name='<model_id>')
# you can find Model ID in the Lamini interface
question = '''How many customers made purchase in December 2024?'''
prompt = get_llama_prompt(question, generate_query_system_prompt)
finetuned_llm.generate(prompt, max_new_tokens=200)
# select uniqExact(s.user_id) as customers
# from ecommerce.sessions s join ecommerce.users u
# on s.user_id = u.user_id
# where (toStartOfMonth(action_date) = '2024-12-01') and (revenue > 0)
# format TabSeparatedWithNames<\/pre>\nIt\u2019s worth noting that we\u2019re using Lamini for inference as well and will have to pay for it. You can find up-to-date information about the costs\u00a0here<\/a>.<\/p>\n
At first glance, the result looks promising, but we need a more robust accuracy evaluation to confirm\u00a0it.<\/p>\n
Additionally, it\u2019s worth noting that since we\u2019ve fine-tuned the model for our specific task, it now consistently returns SQL queries, meaning we may no longer need to use tool calls for structured output.<\/p>\n
Evaluating the\u00a0quality<\/h4>\n
We\u2019ve discussed LLM accuracy evaluation in detail in my previous article<\/a>, so here I\u2019ll provide a brief\u00a0recap.<\/p>\n
We use a golden set of question-and-answer pairs to evaluate the model\u2019s quality. Since this is a toy example, I\u2019ve limited the set to just 10 pairs, which you can review on\u00a0GitHub<\/a>.<\/p>\n
The evaluation process consists of two\u00a0parts:<\/p>\n
- \n
- \nSQL Query Validity<\/strong>: First, we check that the SQL query is valid, meaning ClickHouse doesn\u2019t return errors during execution.<\/li>\n
- \nQuery Correctness<\/strong>: Next, we ensure that the generated query is correct. We compare the outputs of the generated and true queries using LLMs to verify that they provide semantically identical results.<\/li>\n<\/ul>\n
<\/figure>\nThe initial results are far from ideal, but they are significantly better than the base Llama model (which produced zero valid SQL queries). Here\u2019s what we\u00a0found:<\/p>\n
- \n
- ClickHouse returned errors for two\u00a0queries.<\/li>\n
- Three queries were executed, but the results were incorrect.<\/li>\n
- Five queries were\u00a0correct.<\/li>\n<\/ul>\n
No surprises\u200a\u2014\u200athere\u2019s no silver bullet, and it\u2019s always an iterative process. Let\u2019s investigate what went\u00a0wrong.<\/p>\n
Diving into the\u00a0errors<\/h4>\n
The approach is straightforward. Let\u2019s examine the errors one by one to understand why we got these results and how we can fix them. We\u2019ll start with the first unsuccessful example.<\/p>\n
Question: <\/strong>Which country had the highest number of first-time users in\u00a02024?<\/p>\n
Golden query:<\/strong><\/p>\n
select
country,
count(distinct user_id) as users
from
(
select user_id, min(action_date) as first_date
from ecommerce.sessions
group by user_id
having toStartOfYear(first_date) = '2024-01-01'
) as t
inner join ecommerce.users as u
on t.user_id = u.user_id
group by country
order by users desc
limit 1
format TabSeparatedWithNames <\/pre>\nGenerated query:<\/strong><\/p>\n
select
country,
count(distinct u.user_id) as first_time_users
from ecommerce.sessions s
join ecommerce.users u
on s.user_id = u.user_id
where (toStartOfYear(action_date) = '2024-01-01')
and (s.session_id = 1)
group by country
order by first_time_users desc
limit 1
format TabSeparatedWithNames <\/pre>\nThe query is valid, but it returns an incorrect result. The issue lies in the model\u2019s assumption that the first session for each user will always have session_id = 1. Since Lamini Memory Tuning allows the model to learn facts from the training data, let\u2019s investigate why the model made this assumption. Likely, it\u2019s in our\u00a0data.<\/p>\n
Let\u2019s review all the examples that mention first. I\u2019ll use broad and simple search criteria to get a high-level view.<\/p>\n
<\/figure>\nAs we can see, there are no examples mentioning first-time users\u200a\u2014\u200aonly references to the first quarter. It\u2019s no surprise that the model wasn\u2019t able to capture this concept. The solution is straightforward: we just need to add a set of examples with questions and answers specifically about first-time users.<\/p>\n
<\/figure>\nLet\u2019s move on to the next problematic case.<\/p>\n
Question: <\/strong>What was the fraud rate in 2023, expressed as a percentage?<\/p>\n
Golden query:<\/strong><\/p>\n
select
100*uniqExactIf(user_id, is_fraud = 1)\/uniqExact(user_id) as fraud_rate
from ecommerce.sessions
where (toStartOfYear(action_date) = '2023-01-01')
format TabSeparatedWithNames <\/pre>\nGenerated query:<\/strong><\/p>\n
select
100*countIf(is_fraud = 1)\/count() as fraud_rate
from ecommerce.sessions
where (toStartOfYear(action_date) = '2023-01-01')
format TabSeparatedWithNames <\/pre>\nHere\u2019s another misconception: we assumed that the fraud rate is based on the share of users, while the model calculated it based on the share of sessions.<\/p>\n
Let\u2019s check the examples related to the fraud rate in the data. There are two cases: one calculates the share of users, while the other calculates the share of sessions.<\/p>\n
<\/figure>\nTo fix this issue, I corrected the incorrect answer and added more accurate examples involving fraud rate calculations.<\/p>\n
<\/figure>\nI\u2019d like to discuss another incorrect case, as it will highlight an important aspect of the process of resolving these\u00a0issues.<\/p>\n
Question: <\/strong>What are the median and interquartile range (IQR) of purchase revenue for each\u00a0country?<\/p>\n
Golden query:<\/strong><\/p>\n
select
country,
median(revenue) as median_revenue,
quantile(0.25)(revenue) as percentile_25_revenue,
quantile(0.75)(revenue) as percentile_75_revenue
from ecommerce.sessions AS s
inner join ecommerce.users AS u
on u.user_id = s.user_id
where (revenue > 0)
group by country
format TabSeparatedWithNames <\/pre>\nGenerated query:<\/strong><\/p>\n
select
country,
median(revenue) as median_revenue,
quantile(0.25)(revenue) as percentile_25_revenue,
quantile(0.75)(revenue) as percentile_75_revenue
from ecommerce.sessions s join ecommerce.users u
on s.user_id = u.user_id
group by country
format TabSeparatedWithNames <\/pre>\nWhen inspecting the problem, it\u2019s crucial to focus on the model\u2019s misconceptions or incorrect assumptions. For example, in this case, there may be a temptation to add examples similar to those in the golden dataset, but that would be too specific. Instead, we should address the actual root cause of the model\u2019s misconception:<\/p>\n
- \n
- It understood the concepts of median and IQR quite\u00a0well.<\/li>\n
- The split by country is also\u00a0correct.<\/li>\n
- However, it misinterpreted the concept of \u201cpurchase revenue\u201d, including sessions where there was no purchase at all (revenue =\u00a00).<\/li>\n<\/ul>\n
So, we need to ensure that our datasets contain enough information related to purchase revenue. Let\u2019s take a look at what we have now. There\u2019s only one example, and it\u2019s incorrect.<\/p>\n
<\/figure>\nLet\u2019s fix this example and add more cases of purchase revenue calculations.<\/p>\n
<\/figure>\nUsing a similar approach, I\u2019ve added more examples for the two remaining incorrect queries and compiled an updated, cleaned version of the training dataset. You can find it on GitHub<\/a>. With this, our data is ready to the next iteration.<\/p>\n
Another iteration of fine-tuning<\/h4>\n
Before diving into fine-tuning, it\u2019s essential to double-check the quality of the training dataset by ensuring that all SQL queries are\u00a0valid.<\/p>\n
clean_sample_dataset_df = pd.read_csv(
'small_sample_for_finetuning_cleaned.csv', sep = 't',
on_bad_lines = 'warn')
clean_sample_dataset_df['output'] = clean_sample_dataset_df.sql_query
.map(lambda x: get_clickhouse_data(str(x)))
clean_sample_dataset_df['is_valid_output'] = clean_sample_dataset_df['output']
.map(is_valid_output)
print(clean_sample_dataset_df.is_valid_output.value_counts())
# is_valid_output
# ok 241
clean_formatted_responses = []
for rec in clean_sample_dataset_df.to_dict('records'):
clean_formatted_responses.append(
{
'input': get_llama_prompt(
rec['question'],
generate_query_system_prompt),
'output': rec['sql_query']
}
)<\/pre>\nNow that we\u2019re confident in the data, we can proceed with fine-tuning. This time, I\u2019ve decided to train it for 150 steps to achieve better accuracy.<\/p>\n
finetune_args = {
\"max_steps\": 150,
\"learning_rate\": 0.0001
}
llm = Lamini(model_name=\"meta-llama\/Meta-Llama-3.1-8B-Instruct\")
llm.train(
data_or_dataset_id=clean_formatted_responses,
finetune_args=finetune_args
)<\/pre>\nAfter waiting a bit longer than last time, we now have a new fine-tuned model with nearly zero loss after 150 tuning\u00a0steps.<\/p>\n
<\/figure>\nWe can run the evaluation again and see much better results. So, our approach is\u00a0working.<\/p>\n
<\/figure>\nThe result is astonishing, but it\u2019s still worth examining the incorrect example to understand what went wrong. We got an incorrect result for the question we discussed earlier: \u201cWhat are the median and interquartile range (IQR) of purchase revenue for each country?\u201d<\/em> However, this time, the model generated a query that is exactly identical to the one in the golden\u00a0set.<\/p>\n
select
country,
median(revenue) as median_revenue,
quantile(0.25)(revenue) as percentile_25_revenue,
quantile(0.75)(revenue) as percentile_75_revenue
from ecommerce.sessions AS s
inner join ecommerce.users AS u
on u.user_id = s.user_id
where (s.revenue > 0)
group by country
format TabSeparatedWithNames<\/pre>\nSo, the issue actually lies in our evaluation. In fact, if you try to execute this query multiple times, you\u2019ll notice that the results are slightly different each time. The root cause is that the quantile function in ClickHouse<\/a> computes approximate values using reservoir sampling, which is why we\u2019re seeing varying results. We could have used quantileExact instead to get more consistent numbers.<\/p>\n
That said, this means that fine-tuning has allowed us to achieve 100% accuracy. Even though our toy golden dataset consists of just 10 questions, this is a tremendous achievement. We\u2019ve progressed all the way from zero valid queries with vanilla Llama to 70% accuracy with RAG and self-reflection, and now, thanks to Lamini Memory Tuning, we\u2019ve reached 100% accuracy.<\/p>\n
You can find the full code on\u00a0GitHub<\/a>.<\/p><\/blockquote>\n
Summary<\/h3>\n
In this article, we continued exploring techniques to improve LLM accuracy:<\/p>\n
- \n
- After trying RAG and self-reflection in the previous article<\/a>, we moved on to a more advanced technique\u200a\u2014\u200afine-tuning.<\/li>\n
- We experimented with Memory Tuning developed by Lamini, which enables a model to remember a large volume of facts with near-zero errors.<\/li>\n
- In our example, Memory Tuning performed exceptionally well, and we achieved 100% accuracy on our evaluation set of 10 questions.<\/li>\n<\/ul>\n
Thank you a lot for reading this article. I hope this article was insightful for you. If you have any follow-up questions or comments, please leave them in the comments\u00a0section.<\/p><\/blockquote>\n
Reference<\/h3>\n
All the images are produced by the author unless otherwise stated.<\/p><\/blockquote>\n
This article is inspired by the \u201cImproving Accuracy of LLM Applications\u201d<\/a> short course from DeepLearning.AI.<\/p>\n
Disclaimer: <\/strong>I am not affiliated with Lamini in any way. The views expressed in this article are solely my own, based on independent testing and evaluation of the Lamini platform. This post is intended for educational purposes and does not constitute an endorsement of any specific tool or\u00a0service.<\/em><\/p>\n
\nThe Next Frontier in LLM Accuracy<\/a> was originally published in Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\n
- After trying RAG and self-reflection in the previous article<\/a>, we moved on to a more advanced technique\u200a\u2014\u200afine-tuning.<\/li>\n
- \nQuery Correctness<\/strong>: Next, we ensure that the generated query is correct. We compare the outputs of the generated and true queries using LLMs to verify that they provide semantically identical results.<\/li>\n<\/ul>\n
- \nSQL Query Validity<\/strong>: First, we check that the SQL query is valid, meaning ClickHouse doesn\u2019t return errors during execution.<\/li>\n
- \nDiversity<\/strong> in the examples is important so that a model can learn how to handle different cases.<\/li>\n
- There are many step-by-step guides available online, including the tutorial on how to fine-tune Llama on Kaggle from DataCamp<\/a>.<\/li>\n
- \nThe Llama documentation<\/a> provides numerous recipes for fine-tuning, which can be executed on a cloud server or even locally for smaller\u00a0models.<\/li>\n
- If accuracy is still insufficient, you can incorporate a self-reflection step<\/strong>, for example, to return errors from the API calls and ask the LLM to correct mistakes.<\/li>\n