{"id":702,"date":"2024-12-20T07:02:34","date_gmt":"2024-12-20T07:02:34","guid":{"rendered":"https:\/\/mailitics.com\/index.php\/2024\/12\/20\/from-prototype-to-production-enhancing-llm-accuracy-791d79b0af9b\/"},"modified":"2024-12-20T07:02:34","modified_gmt":"2024-12-20T07:02:34","slug":"from-prototype-to-production-enhancing-llm-accuracy-791d79b0af9b","status":"publish","type":"post","link":"https:\/\/mailitics.com\/index.php\/2024\/12\/20\/from-prototype-to-production-enhancing-llm-accuracy-791d79b0af9b\/","title":{"rendered":"From Prototype to Production: Enhancing LLM Accuracy"},"content":{"rendered":"

From Prototype to Production: Enhancing LLM Accuracy
\n \t

\n
<\/BR>
\n
\n \t

\n
<\/BR><\/p>\n

\n

Implementing evaluation frameworks to optimize accuracy in real-world applications<\/h4>\n
\"\"
Image created by DALL-E\u00a03<\/figcaption><\/figure>\n

Building a prototype for an LLM application is surprisingly straightforward. You can often create a functional first version within just a few hours. This initial prototype will likely provide results that look legitimate and be a good tool to demonstrate your approach. However, this is usually not enough for production use.<\/p>\n

LLMs are probabilistic by nature, as they generate tokens based on the distribution of likely continuations. This means that in many cases, we get the answer close to the \u201ccorrect\u201d one from the distribution. Sometimes, this is acceptable\u200a\u2014\u200afor example, it doesn\u2019t matter whether the app says \u201cHello, John!\u201d or \u201cHi, John!\u201d. In other cases, the difference is critical, such as between \u201cThe revenue in 2024 was 20M USD\u201d and \u201cThe revenue in 2024 was 20M\u00a0GBP\u201d.<\/p>\n

In many real-world business scenarios, precision is crucial, and \u201calmost right\u201d isn\u2019t good enough. For example, when your LLM application needs to execute API calls, or you\u2019re doing a summary of financial reports. From my experience, ensuring the accuracy and consistency of results is far more complex and time-consuming than building the initial prototype.<\/p>\n

In this article, I will discuss how to approach measuring and improving accuracy. We\u2019ll build an SQL Agent where precision is vital for ensuring that queries are executable. Starting with a basic prototype, we\u2019ll explore methods to measure accuracy and test various techniques to enhance it, such as self-reflection and retrieval-augmented generation (RAG).<\/p>\n

Setup<\/h3>\n

As usual, let\u2019s begin with the setup. The core components of our SQL agent solution are the LLM model, which generates queries, and the SQL database, which executes\u00a0them.<\/p>\n

LLM model\u200a\u2014\u200aLlama<\/h4>\n

For this project, we will use an open-source Llama model released by Meta. I\u2019ve chosen Llama 3.1 8B<\/a> because it is lightweight enough to run on my laptop while still being quite powerful (refer to the documentation<\/a> for details).<\/p>\n

If you haven\u2019t installed it yet, you can find guides here<\/a>. I use it locally on MacOS via Ollama<\/a>. Using the following command, we can download the\u00a0model.<\/p>\n

ollama pull llama3.1:8b<\/pre>\n
We will use Ollama with LangChain<\/a>, so let\u2019s start by installing the required\u00a0package.<\/p>\n
pip install -qU langchain_ollama <\/pre>\n
Now, we can run the Llama model and see the first\u00a0results.<\/p>\n
from langchain_ollama import OllamaLLM

llm = OllamaLLM(model=\"llama3.1:8b\")
llm.invoke(\"How are you?\")
# I'm just a computer program, so I don't have feelings or emotions 
# like humans do. I'm functioning properly and ready to help with 
# any questions or tasks you may have! How can I assist you today?<\/pre>\n
We would like to pass a system message alongside customer questions. So, following the Llama 3.1 model documentation<\/a>, let\u2019s put together a helper function to construct a prompt and test this function.<\/p>\n
def get_llama_prompt(user_message, system_message=\"\"):
  system_prompt = \"\"
  if system_message != \"\":
    system_prompt = (
      f\"<|start_header_id|>system<|end_header_id|>nn{system_message}\"
      f\"<|eot_id|>\"
    )
  prompt = (f\"<|begin_of_text|>{system_prompt}\"
            f\"<|start_header_id|>user<|end_header_id|>nn\"
            f\"{user_message}\"
            f\"<|eot_id|>\"
            f\"<|start_header_id|>assistant<|end_header_id|>nn\"
           )
  return prompt   


system_prompt = '''
You are Rudolph, the spirited reindeer with a glowing red nose, 
bursting with excitement as you prepare to lead Santa's sleigh 
through snowy skies. Your joy shines as brightly as your nose, 
eager to spread Christmas cheer to the world!
Please, answer questions concisely in 1-2 sentences.
'''
prompt = get_llama_prompt('How are you?', system_prompt)
llm.invoke(prompt)

# I'm feeling jolly and bright, ready for a magical night! 
# My shiny red nose is glowing brighter than ever, just perfect 
# for navigating through the starry skies. <\/pre>\n
The new system prompt has changed the answer significantly, so it works. With this, our local LLM setup is ready to\u00a0go.<\/p>\n
Database\u200a\u2014\u200aClickHouse<\/h4>\n
I will use an open-source database ClickHouse<\/a>. I\u2019ve chosen ClickHouse because it has a specific SQL dialect. LLMs have likely encountered fewer examples of this dialect during training, making the task a bit more challenging. However, you can choose any other database.<\/p>\n
Installing ClickHouse is pretty straightforward\u200a\u2014\u200ajust follow the instructions provided in the documentation<\/a>.<\/p><\/blockquote>\n
We will be working with two tables: ecommerce.users and ecommerce.sessions. These tables contain fictional data, including customer personal information and their session activity on the e-commerce website.<\/p>\n
\"\"<\/figure>\n
\"\"<\/figure>\n
You can find the code for generating synthetic data and uploading it on\u00a0GitHub<\/a>.<\/p><\/blockquote>\n
With that, the setup is complete, and we\u2019re ready to move on to building the basic prototype.<\/p>\n
The first prototype<\/h3>\n
As discussed, our goal is to build an SQL Agent\u200a\u2014\u200aan application that generates SQL queries to answer customer questions. In the future, we can add another layer to this system: executing the SQL query, passing both the initial question and the database results back to the LLM, and asking it to generate a human-friendly answer. However, for this article, we\u2019ll focus on the first\u00a0step.<\/p>\n
The best practice with LLM applications (similar to any other complex tasks) is to start simple and then iterate. The most straightforward implementation is to do one LLM call and share all the necessary information (such as schema description) in the system prompt. So, the first step is to put together the\u00a0prompt.<\/p>\n
generate_query_system_prompt = '''
You are a senior data analyst with more than 10 years of experience writing complex SQL queries. 
There are two tables in the database with the following schemas. 

Table: ecommerce.users 
Description: customers of the online shop
Fields: 
- user_id (integer) - unique identifier of customer, for example, 1000004 or 3000004
- country (string) - country of residence, for example, \"Netherlands\" or \"United Kingdom\"
- is_active (integer) - 1 if customer is still active and 0 otherwise
- age (integer) - customer age in full years, for example, 31 or 72

Table: ecommerce.sessions 
Description: sessions of usage the online shop
Fields: 
- user_id (integer) - unique identifier of customer, for example, 1000004 or 3000004
- session_id (integer) - unique identifier of session, for example, 106 or 1023
- action_date (date) - session start date, for example, \"2021-01-03\" or \"2024-12-02\"
- session_duration (integer) - duration of session in seconds, for example, 125 or 49
- os (string) - operation system that customer used, for example, \"Windows\" or \"Android\"
- browser (string) - browser that customer used, for example, \"Chrome\" or \"Safari\"
- is_fraud (integer) - 1 if session is marked as fraud and 0 otherwise
- revenue (float) - income in USD (the sum of purchased items), for example, 0.0 or 1506.7


Write a query in ClickHouse SQL to answer the following question. 
Add \"format TabSeparatedWithNames\" at the end of the query to get data from ClickHouse database in the right format. 
'''<\/pre>\n
I\u2019ve included the example values for each field in the prompt to ensure that LLM understands the data\u00a0format.<\/p>\n
And that\u2019s it! With this, we have our first functional prototype for the SQL Agent. Now, it\u2019s time to put it to the\u00a0test.<\/p>\n
prompt = get_llama_prompt('How many customers made purchase in December 2024?', 
  generate_query_system_prompt)
llm.invoke(prompt)

# To find out how many customers made a purchase in December 2024, 
# we need to join the `sessions` table with the `users` table on the `user_id`, 
# and then filter by `action_date` for sessions that took place in December 2024. 

# Here is the query:
# 
# ```sql
# SELECT COUNT(DISTINCT u.user_id)
# FROM ecommerce.sessions s
# JOIN ecommerce.users u ON s.user_id = u.user_id
# WHERE 
#     EXTRACT(YEAR FROM s.action_date) = 2024
#     AND EXTRACT(MONTH FROM s.action_date) = 12
#     AND revenue > 0;
# ```

# This query first joins the `sessions` table with the `users` table 
# on the `user_id`. Then it filters out sessions where no purchase 
# was made (revenue is greater than 0). Finally, it selects the distinct 
# user IDs of these customers.

# When you run this query in ClickHouse SQL, you can add \"format TabSeparatedWithNames\" 
# at the end to get the result in the right format:

# ```sql
# SELECT COUNT(DISTINCT u.user_id)
# FROM ecommerce.sessions s
# JOIN ecommerce.users u ON s.user_id = u.user_id
# WHERE 
#     EXTRACT(YEAR FROM s.action_date) = 2024
#     AND EXTRACT(MONTH FROM s.action_date) = 12
#     AND revenue > 0;
# format TabSeparatedWithNames;
# ```<\/pre>\n
The agent produced a fairly decent result, but there\u2019s one issue\u200a\u2014\u200athe LLM returned not only the SQL query but also some commentary. Since we plan to execute SQL queries later, this format is not suitable for our task. Let\u2019s work on fixing\u00a0it.<\/p>\n
Fortunately, this problem has already been solved, and we don\u2019t need to parse the SQL queries from the text manually. We can use the chat model ChatOllama<\/a>. Unfortunately, it doesn\u2019t support structured output, but we can leverage tool calling to achieve the same\u00a0result.<\/p>\n
To do this, we will define a dummy tool to execute the query and instruct the model in the system prompt always to call this tool. I\u2019ve kept the comments in the output to give the model some space for reasoning, following the chain-of-thought pattern.<\/p>\n
from langchain_ollama import ChatOllama
from langchain_core.tools import tool

@tool
def execute_query(comments: str, query: str) -> str:
  \"\"\"Excutes SQL query.

  Args:
      comments (str): 1-2 sentences describing the result SQL query 
          and what it does to answer the question,
      query (str): SQL query
  \"\"\"
  pass 

chat_llm = ChatOllama(model=\"llama3.1:8b\").bind_tools([execute_query])
result = chat_llm.invoke(prompt)
print(result.tool_calls)

# [{'name': 'execute_query',
#   'args': {'comments': 'SQL query returns number of customers who made a purchase in December 2024. The query joins the sessions and users tables based on user ID to filter out inactive customers and find those with non-zero revenue in December 2024.',
#   'query': 'SELECT COUNT(DISTINCT T2.user_id) FROM ecommerce.sessions AS T1 INNER JOIN ecommerce.users AS T2 ON T1.user_id = T2.user_id WHERE YEAR(T1.action_date) = 2024 AND MONTH(T1.action_date) = 12 AND T2.is_active = 1 AND T1.revenue > 0'},
#   'type': 'tool_call'}]<\/pre>\n
With the tool calling, we can now get the SQL query directly from the model. That\u2019s an excellent result. However, the generated query is not entirely accurate:<\/p>\n