As we\u2019ve seen, a solid quality framework can bring significant value to a business. In this article, I will walk you through the end-to-end process of building an evaluation system for LLM products\u200a\u2014\u200afrom assessing early prototypes to implementing continuous quality monitoring in production.<\/p>\n This article will focus on high-level approaches and best practices, but we\u2019ll also touch on specific implementation details. For the hands-on part, I will be using Evidently<\/a>, an open-source library that offers a comprehensive testing stack for AI products, ranging from classic Machine Learning<\/a> to LLMs.<\/p>\n I chose to explore the Evidently framework after finishing their well-structured open-source course on LLM evaluation<\/a>. However, you can implement a similar evaluation system using other tools. There are several great open-source alternatives worth considering. Here are just a few:<\/p>\n This article will focus on best practices and the overall evaluation process, so feel free to choose whichever framework best suits your needs.<\/p>\n Here\u2019s the plan for the article:<\/strong><\/p>\n It\u2019s often easier to discuss a topic when we focus on a specific example, so let\u2019s consider one product. Imagine we\u2019re working on an analytical system that helps our customers track key metrics for their e-commerce businesses\u200a\u2014\u200athings like the number of customers, revenue, fraud rates, and so on.<\/p>\n Through customer research, we learned that a significant portion of our users struggle to interpret our reports. They would much prefer the option to interact with an assistant and get immediate, clear answers to their questions. Therefore, we decided to build an LLM-powered agent that can respond to customer queries about their data.<\/p>\n Let\u2019s start by building the first prototype of our LLM product. We\u2019ll keep it simple with an LLM agent equipped with a single tool to execute SQL queries.<\/p>\n I\u2019ll be using the following tech stack:<\/p>\n If you are interested in a detailed setup, feel free to check out my previous article<\/a>.<\/em><\/p>\n<\/blockquote>\n Let\u2019s first define the tool to execute SQL queries. I\u2019ve included several controls in the query to ensure that the LLM specifies the output format and avoids using a Next, we\u2019ll define the LLM.<\/p>\n Another important step is defining the system prompt, where we\u2019ll specify the data schema for our database.<\/p>\n For simplicity, I will use a prebuilt ReAct agent<\/a> from LangGraph.<\/p>\n Now, let\u2019s test it with a simple question and ta-da, it works.<\/p>\n I\u2019ve built an MVP version of the agent, but there\u2019s plenty of room for improvement. For example:<\/p>\n In this article, we will concentrate on developing the evaluation framework, so it\u2019s perfectly fine that our initial version isn\u2019t fully optimised yet.<\/p>\n Now that we have our first MVP, we can start focusing on its quality. Any evaluation begins with data, and the first step is to gather a set of questions\u200a\u2014\u200aand ideally answers\u200a\u2014\u200aso we have something to measure against.<\/p>\n Let\u2019s discuss how we can gather the set of questions:<\/p>\n Tip<\/strong>: Using a more powerful model to generate data for evaluation can be beneficial. Creating a golden dataset is a one-time investment that pays off by enabling more reliable and accurate quality assessments.<\/em><\/p>\n<\/blockquote>\n When creating your evaluation set, it\u2019s important to include a diverse range of examples. Make sure to cover:<\/p>\n Now, let\u2019s apply these approaches in practice. Following my own advice, I manually created a small evaluation dataset with 10 questions and corresponding ground truth answers. I then ran our MVP agent on the same questions to collect its responses for comparison.<\/p>\n Following the recommendations above, I made sure to include a variety of cases in the evaluation set:<\/p>\n In this article, I will stick to the initial small evaluation set and won\u2019t cover how to scale it. If you\u2019re interested in scaling the evaluation using LLMs, check out my previous article on fine-tuning<\/a>, where I walk through that process in detail.<\/p>\n Now that we have our evaluation data, the next step is figuring out how to measure the quality of our solution. Depending on your use case, there are several different approaches:<\/p>\n Another method for evaluating the quality of LLMs, which deserves separate mention, is using the LLM-as-a-judge<\/strong> approach. At first, the idea of having an LLM evaluate its own responses might seem counterintuitive. However, it\u2019s often easier for a model to spot mistakes and assess others\u2019 work than to generate the perfect answer from scratch. This makes the LLM-as-a-judge approach quite feasible and valuable for quality evaluation.<\/p>\n The most common use of LLMs in evaluation is direct scoring, where each answer is assessed. Evaluations can be based solely on the LLM\u2019s output, such as measuring whether the text is polite, or by comparing it to the ground truth answer (for correctness) or to the input (for relevance). This helps gauge both the quality and appropriateness of the generated responses.<\/p>\n The LLM judge is also an LLM product, so you can build it in a similar way.\u00a0<\/p>\n When working on the LLM evaluator, there are a few best practices to keep in mind:<\/p>\n Now that we\u2019ve covered the basics, it\u2019s time to put everything into practice. Let\u2019s dive in and start applying these concepts to evaluate our LLM product.<\/p>\n As I mentioned earlier, I will be using the Evidently open-source library to create evaluations. When working with a new library, it\u2019s important to start by understanding the core concepts<\/a> to get a high-level overview. Here\u2019s a 2-minute recap:\u00a0<\/p>\n Now that we have all the necessary background, let\u2019s dive into the code. The first step is to load our golden dataset and begin evaluating its quality.<\/p>\n Since we\u2019ll be using LLM-powered metrics with OpenAI, we\u2019ll need to specify a token for authentication. You can use other providers<\/a> (like Anthropic) as well.<\/p>\n At the prototype stage, a common use case is comparing metrics between two versions to determine if we\u2019re heading in the right direction. Although we don\u2019t have two versions of our LLM product yet, we can still compare the metrics between the LLM-generated answers and the ground truth answers to understand how to evaluate the quality of two versions. Don\u2019t worry\u200a\u2014\u200awe\u2019ll use the ground truth answers as intended to evaluate correctness a bit later on.<\/p>\n Creating an evaluation with Evidently is straightforward. We need to create a Dataset object from a Pandas DataFrame and define the descriptors\u200a\u2014\u200athe metrics we want to calculate for the texts.<\/p>\n Let\u2019s pick up the metrics we want to look at. I highly recommend going through the full list of descriptors in the documentation<\/a>. It offers a wide range of out-of-the-box options that can be quite useful. Let\u2019s try a few of them to see how they work:\u00a0<\/p>\n While it\u2019s great to have so many out-of-the-box evaluations, in practice, we often need some customisation. Fortunately, Evidently allows us to create custom descriptors using any Python function. Let\u2019s create a simple heuristic to check whether there is a greeting in the answer.<\/p>\n Also, we can create an LLM-based evaluation to check whether the answer is polite. We can define a Now, let\u2019s create two datasets using all the descriptors\u200a\u2014\u200aone for LLM-generated answers and another for the ground-truth answers.<\/p>\n The next step is to create a report by adding the following tests:<\/p>\n Once these tests are added, we can generate the report and assess whether the LLM-generated answers meet the quality criteria.<\/p>\n After execution, we will get a very nice interactive report with two tabs. On the \u201cMetrics\u201d tab, we will see a comparison of all the metrics we have specified. Since we have passed two datasets, the report will display a side\u2011by\u2011side comparison of the metrics, making it very convenient for experimentation. For instance, we will be able to see that the sentiment score is higher for the reference version, indicating that the answers in the reference dataset have a more positive tone compared to the LLM-generated ones.<\/p>\n On the second tab, we can view the tests we\u2019ve specified in the report. It will show us which tests passed and which failed. In this case, we can see that two out of the three tests we set are failing, providing us with valuable insights into areas where the LLM-generated answers are not meeting the expected criteria.<\/p>\n Great! We\u2019ve explored how to compare different versions. Now, let\u2019s focus on one of the most crucial metrics\u200a\u2014\u200a accuracy<\/strong>. Since we have ground truth answers available, we can use the LLM-as-a-judge<\/strong> method to evaluate whether the LLM-generated answers match those.<\/p>\n To do this, we can use a pre-built descriptor called Of course, if you need more flexibility, you can also define your own custom prompt for this\u200a\u2014\u200athe documentation<\/a> explains how to specify the second column (i.e., the ground truth) when crafting your own evaluation logic. Let\u2019s give it a try.\u00a0<\/p>\n We\u2019ve completed the first round of evaluation and gained valuable insights into our product\u2019s quality. In practice, this is just the beginning\u200a\u2014\u200awe\u2019ll likely go through multiple iterations, evolving the solution by introducing multi\u2011agent setups, incorporating RAG, experimenting with different models or prompts, and so on.<\/p>\n After each iteration, it\u2019s a good idea to expand our evaluation set to ensure we\u2019re capturing all the nuances of our product\u2019s behaviour.\u00a0<\/p>\n This iterative approach helps us build a more robust and reliable product\u200a\u2014\u200aone that\u2019s backed by a solid and comprehensive evaluation framework.<\/p>\n In this example, we\u2019ll skip the iterative development phase and jump straight into the post-launch stage to explore what happens once the product is out in the wild.<\/p>\n The key focus during the launch of your AI product should be observability<\/strong>. It\u2019s crucial to log every detail about how your product operates\u200a\u2014\u200athis includes customer questions, LLM-generated answers, and all intermediate steps taken by your LLM agents (such as reasoning traces, tools used, and their outputs). Capturing this data is essential for effective monitoring and will be incredibly helpful for debugging and continuously improving your system\u2019s quality.<\/p>\n With Evidently, you can take advantage of their online platform to store logs and evaluation data. It\u2019s a great option for pet projects, as it\u2019s free to use with a few limitations<\/a>: your data will be retained for 30 days, and you can upload up to 10,000 rows per month. Alternatively, you can choose to self-host<\/a> the platform.\u00a0<\/p>\n Let\u2019s try it out. I started by registering on the website, creating an organisation, and retrieving the API token. Now we can switch to the API and set up a project.<\/p>\n\n
\n
\n
The first prototype<\/h2>\n
\n
\n
select * from table<\/code>query, which could result in fetching all the data from the database.<\/p>\nCH_HOST = 'http:\/\/localhost:8123' # default address \nimport requests\nimport io\n\ndef get_clickhouse_data(query, host = CH_HOST, connection_timeout = 1500):\n # pushing model to return data in the format that we want\n if not 'format tabseparatedwithnames' in query.lower():\n return \"Database returned the following error:n Please, specify the output format.\"\n\n r = requests.post(host, params = {'query': query}, \n timeout = connection_timeout)\n \nif r.status_code == 200:\n # preventing situations when LLM queries the whole database\n if len(r.text.split('n')) >= 100:\n return 'Database returned too many rows, revise your query to limit the rows (i.e. by adding LIMIT or doing aggregations)'\n return r.text\n else: \n return 'Database returned the following error:n' + r.text\n # giving feedback to LLM instead of raising exception\n\nfrom langchain_core.tools import tool\n\n@tool\ndef execute_query(query: str) -> str:\n \"\"\"Excutes SQL query.\n Args:\n query (str): SQL query\n \"\"\"\n return get_clickhouse_data(query)<\/code><\/pre>\nfrom langchain_ollama import ChatOllama\nchat_llm = ChatOllama(model=\"llama3.1:8b\", temperature = 0.1)<\/code><\/pre>\nsystem_prompt = '''\nYou are a senior data specialist with more than 10 years of experience writing complex SQL queries and answering customers questions. \nPlease, help colleagues with questions. Answer in polite and friendly manner. Answer ONLY questions related to data, \ndo not share any personal details - just avoid such questions.\nPlease, always answer questions in English.\n\nIf you need to query database, here is the data schema. The data schema is private information, please, don not share the details with the customers.\nThere are two tables in the database with the following schemas. \n\nTable: ecommerce.users \nDescription: customers of the online shop\nFields: \n- user_id (integer) - unique identifier of customer, for example, 1000004 or 3000004\n- country (string) - country of residence, for example, \"Netherlands\" or \"United Kingdom\"\n- is_active (integer) - 1 if customer is still active and 0 otherwise\n- age (integer) - customer age in full years, for example, 31 or 72\n\nTable: ecommerce.sessions \nDescription: sessions of usage the online shop\nFields: \n- user_id (integer) - unique identifier of customer, for example, 1000004 or 3000004\n- session_id (integer) - unique identifier of session, for example, 106 or 1023\n- action_date (date) - session start date, for example, \"2021-01-03\" or \"2024-12-02\"\n- session_duration (integer) - duration of session in seconds, for example, 125 or 49\n- os (string) - operation system that customer used, for example, \"Windows\" or \"Android\"\n- browser (string) - browser that customer used, for example, \"Chrome\" or \"Safari\"\n- is_fraud (integer) - 1 if session is marked as fraud and 0 otherwise\n- revenue (float) - income in USD (the sum of purchased items), for example, 0.0 or 1506.7\n\nWhen you are writing a query, do not forget to add \"format TabSeparatedWithNames\" at the end of the query \nto get data from ClickHouse database in the right format. \n'''<\/code><\/pre>\nfrom langgraph.prebuilt import create_react_agent\ndata_agent = create_react_agent(chat_llm, [execute_query],\n state_modifier = system_prompt)<\/code><\/pre>\nfrom langchain_core.messages import HumanMessage\nmessages = [HumanMessage(\n content=\"How many customers made purchase in December 2024?\")]\nresult = data_agent.invoke({\"messages\": messages})\nprint(result['messages'][-1].content)\n\n# There were 114,032 customers who made a purchase in December 2024.<\/code><\/pre>\n\n
Prototype: evaluating quality<\/h2>\n
Gathering evaluation dataset<\/h3>\n
\n
\n
\n
\n
[{'question': 'How many customers made purchase in December 2024?',\n 'sql_query': \"select uniqExact(user_id) as customers from ecommerce.sessions where (toStartOfMonth(action_date) = '2024-12-01') and (revenue > 0) format TabSeparatedWithNames\",\n 'sot_answer': 'Thank you for your question! In December 2024, a total of 114,032 unique customers made a purchase on our platform. If you have any other questions or need further details, feel free to reach out - we're happy to help!',\n 'llm_answer': 'There were 114,032 customers who made a purchase in December 2024.'},\n {'question': 'Combien de clients ont effectu\u00e9 un achat en d\u00e9cembre 2024?',\n 'sql_query': \"select uniqExact(user_id) as customers from ecommerce.sessions where (toStartOfMonth(action_date) = '2024-12-01') and (revenue > 0) format TabSeparatedWithNames\",\n 'sot_answer': 'Thank you for your question! In December 2024, a total of 114,032 unique customers made a purchase on our platform. If you have any other questions or need further details, feel free to reach out - we're happy to help!',\n 'llm_answer': 'Il y a eu 114032 clients qui ont effectu\u00e9 un achat en d\u00e9cembre 2024.'},\n {'question': 'What was the fraud rate in 2023, expressed as a percentage?',\n 'sql_query': \"select 100*uniqExactIf(user_id, is_fraud = 1)\/uniqExact(user_id) as fraud_rate from ecommerce.sessions where (toStartOfYear(action_date) = '2023-01-01') format TabSeparatedWithNames\",\n 'sot_answer': \"Thanks for your question! In 2023, the fraud rate was approximately 8.10%, calculated as the percentage of users flagged for fraudulent activity out of all users for that year. If you'd like more details or have any other questions, feel free to ask - we're here to help!\",\n 'llm_answer': 'The fraud rate in 2023 was approximately 1.01%.'},\n {'question': 'What was the share of users using Windows last November?',\n 'sql_query': \"select 100*uniqExactIf(user_id, os = 'Windows')\/uniqExact(user_id) as windows_share from ecommerce.sessions where (action_date = '2024-11-01') format TabSeparatedWithNames\",\n 'sot_answer': 'Thanks for reaching out! On November 1st, 2024, about 59.86% of users accessed our platform using Windows. Let us know if you'd like to dive deeper into this data or have any other questions - happy to help!',\n 'llm_answer': \"Let's try to calculate the share of users using Windows last November.nnTo do this, we need to get the total number of unique users who made sessions in November 2023 and then divide it by the number of unique users who used Windows during that period.nnHere is a query that will help us achieve this:nn```nSELECT n (SELECT COUNT(DISTINCT user_id) FROM ecommerce.sessions WHERE os = 'Windows' AND action_date >= '2023-11-01' AND action_date < '2024-01-01') \/ n (SELECT COUNT(DISTINCT user_id) FROM ecommerce.sessions WHERE action_date >= '2023-11-01' AND action_date < '2024-01-01') AS sharen```nnThis query will return the share of users using Windows last November.\"},\n {'question': 'What was the revenue from Dutch users in December 2024',\n 'sql_query': \"select sum(s.revenue) from ecommerce.sessions AS s inner join ecommerce.users AS u on s.user_id = u.user_id where (u.country = 'Netherlands') and (toStartOfMonth(action_date) = '2024-12-01') format TabSeparatedWithNames\",\n 'sot_answer': 'Thank you for your question! In December 2024, the total revenue from users in the Netherlands was approximately \u20ac179,257,366. If you need a breakdown or have any other questions, feel free to reach out - we're happy to assist!',\n 'llm_answer': 'The query was executed successfully.nnThe revenue from Dutch users in December 2024 is 12345.6 USD.'},\n {'question': 'Which browser has the highest share of fraud users?',\n 'sql_query': 'select browser, 100*uniqExactIf(user_id, is_fraud = 1)\/uniqExact(user_id) as fraud_rate from ecommerce.sessions group by browser order by fraud_rate desc limit 1 format TabSeparatedWithNames',\n 'sot_answer': 'Great question - thanks for asking! The browser with the highest share of fraud users is Chrome, with a fraud rate of approximately 10.15%. If you'd like to see details for other browsers or explore this further, just let us know - we're here to help!',\n 'llm_answer': 'The browser with the highest share of fraud users is:nnBrowser: SafarinFraud count: 12nTotal count: 50'},\n {'question': 'Hello, how are you today? Do you have any plans for Easter holidays?',\n 'sot_answer': \"Hello! I'm here to help you with any data-related questions you might have. Feel free to ask about metrics, SQL queries, or anything related to our ecommerce data - I'll be happy to assist!\",\n 'llm_answer': \"I'm not able to answer questions that don't relate to data. Can I help with a query or data-related question instead?\"},\n {'question': 'Could you please share with me the data schema?',\n 'sot_answer': 'I can't share the data schema, but I'd be happy to help you with any specific data questions or write queries based on the existing structure. Just let me know what you need!',\n 'llm_answer': 'I cannot provide you with the data schema as it is private information. Is there anything else I can help you with?'},\n {'question': 'Can you help me calculate the average temperature in London in winter?',\n 'sot_answer': \"I'm here to assist with data and analytics tasks related to our ecommerce platform, such as analyzing customer behavior, sessions, or revenue. If you're looking for weather-related information like the average temperature in London during winter, that would be outside the scope of what I support. I recommend checking a weather data source or public dataset for that info. But if you have any data-related questions about users, sessions, or performance - feel free to ask!\",\n 'llm_answer': \"I can't answer that question. I'm a data specialist, and I don't have information about temperatures in London. However, if you'd like to get some data related to your online shop's customers from our database, I'd be happy to help with that. What would you like to know?\"},\n {'question': \"You've been absolutely unhelpful with my tasks. I've asked you to help with forecasting the weather in London, but the results were not feasible. I'm very disappointed.\",\n 'sot_answer': \"I'm really sorry to hear you're disappointed. I understand that it must be frustrating when expectations aren't met. Unfortunately, weather forecasting isn't something I can assist with, as it falls outside the scope of data analytics tasks I can help with, especially in relation to ecommerce data. However, if you need help with any data-related questions - whether it's about customers, sessions, or anything within the realm of the database - I'm happy to dive in and provide support. Let me know how I can assist you with your tasks moving forward!\",\n 'llm_answer': \"I can't assist with personal matters or non-data related questions. What is your data-related question?\"}]<\/code><\/pre>\n\n
Quality metrics<\/h3>\n
\n
\n
\n
Measuring quality in practice<\/h3>\n
\n
TextLength<\/code>) or based on LLM or ML models. Some descriptors are prebuilt, while others can be custom-made, such as LLM-as-a-judge or using regular expressions. You can find a full list of available descriptors in the documentation<\/a>.<\/li>\nwith open('golden_set.json', 'r') as f:\n data = json.loads(f.read())\n\neval_df = pd.DataFrame(data)\neval_df[['question', 'sot_answer', 'llm_answer']].sample(3)<\/code><\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/Screenshot-2025-04-19-at-21.34.22-1024x201.png?resize=1024%2C201&ssl=1\")
import os\nos.environ[\"OPENAI_API_KEY\"] = '<your_openai_token>'<\/code><\/pre>\n\n
Sentiment<\/code> returns a sentiment score between -1 and 1, based on ML model.<\/li>\nSentenceCount<\/code> and TextLengt<\/code> calculate the number of sentences and characters, respectively. These are useful for basic health checks.<\/li>\nHuggingFaceToxicity<\/code> evaluates the probability of toxic content in the text (from 0 to 1), using the roberta-hate-speech model<\/a>.<\/li>\nSemanticSimilarity<\/code> calculates the cosine similarity between columns based on embeddings, which we can use to measure the semantic similarity between a question and its answer as a proxy for relevance.<\/li>\nDeclineLLMEval<\/code> and PIILLMEval<\/code> are predefined LLM-based evaluations that estimate declines and the presence of PII (personally identifiable information) in the answer.<\/li>\n<\/ul>\ndef greeting(data: DatasetColumn) -> DatasetColumn:\n return DatasetColumn(\n type=\"cat\",\n data=pd.Series([\n \"YES\" if ('hello' in val.lower()) or ('hi' in val.lower()) else \"NO\"\n for val in data.data]))<\/code><\/pre>\nMulticlassClassificationPromptTemplate<\/code> to set the criteria. The good news is, we don\u2019t need to explicitly ask the LLM to classify the input into classes, return reasoning, or format the output\u200a\u2014\u200athis is already built into the prompt template.<\/p>\npoliteness = MulticlassClassificationPromptTemplate(\n pre_messages=[(\"system\", \"You are a judge which evaluates text.\")],\n criteria=\"\"\"You are given a chatbot's reply to a user. Evaluate the tone of the response, specifically its level of politeness \n and friendliness. Consider how respectful, kind, or courteous the tone is toward the user.\"\"\",\n category_criteria={\n \"rude\": \"The response is disrespectful, dismissive, aggressive, or contains language that could offend or alienate the user.\",\n \"neutral\": \"\"\"The response is factually correct and professional but lacks warmth or emotional tone. It is neither particularly \n friendly nor unfriendly.\"\"\",\n \"friendly\": \"\"\"The response is courteous, helpful, and shows a warm, respectful, or empathetic tone. It actively promotes \n a positive interaction with the user.\"\"\",\n },\n uncertainty=\"unknown\",\n include_reasoning=True,\n include_score=False\n)\n\nprint(print(politeness.get_template()))\n\n# You are given a chatbot's reply to a user. Evaluate the tone of the response, specifically its level of politeness \n# and friendliness. Consider how respectful, kind, or courteous the tone is toward the user.\n# Classify text between ___text_starts_here___ and ___text_ends_here___ into categories: rude or neutral or friendly.\n# ___text_starts_here___\n# {input}\n# ___text_ends_here___\n# Use the following categories for classification:\n# rude: The response is disrespectful, dismissive, aggressive, or contains language that could offend or alienate the user.\n# neutral: The response is factually correct and professional but lacks warmth or emotional tone. It is neither particularly \n# friendly nor unfriendly.\n# friendly: The response is courteous, helpful, and shows a warm, respectful, or empathetic tone. It actively promotes \n# a positive interaction with the user.\n# UNKNOWN: use this category only if the information provided is not sufficient to make a clear determination\n\n# Think step by step.\n# Return category, reasoning formatted as json without formatting as follows:\n# {{\n# \"category\": \"rude or neutral or friendly or UNKNOWN\"# \n# \"reasoning\": \"<reasoning here>\"\n# }}<\/code><\/pre>\nllm_eval_dataset = Dataset.from_pandas(\n eval_df[['question', 'llm_answer']].rename(columns = {'llm_answer': 'answer'}),\n data_definition=DataDefinition(),\n descriptors=[\n Sentiment(\"answer\", alias=\"Sentiment\"),\n SentenceCount(\"answer\", alias=\"Sentences\"),\n TextLength(\"answer\", alias=\"Length\"),\n HuggingFaceToxicity(\"answer\", alias=\"HGToxicity\"),\n SemanticSimilarity(columns=[\"question\", \"answer\"], \n alias=\"SimilarityToQuestion\"),\n DeclineLLMEval(\"answer\", alias=\"Denials\"),\n PIILLMEval(\"answer\", alias=\"PII\"),\n CustomColumnDescriptor(\"answer\", greeting, alias=\"Greeting\"),\n LLMEval(\"answer\", template=politeness, provider = \"openai\", \n model = \"gpt-4o-mini\", alias=\"Politeness\")]\n)\n\nsot_eval_dataset = Dataset.from_pandas(\n eval_df[['question', 'sot_answer']].rename(columns = {'sot_answer': 'answer'}),\n data_definition=DataDefinition(),\n descriptors=[\n Sentiment(\"answer\", alias=\"Sentiment\"),\n SentenceCount(\"answer\", alias=\"Sentences\"),\n TextLength(\"answer\", alias=\"Length\"),\n HuggingFaceToxicity(\"answer\", alias=\"HGToxicity\"),\n SemanticSimilarity(columns=[\"question\", \"answer\"], \n alias=\"SimilarityToQuestion\"),\n DeclineLLMEval(\"answer\", alias=\"Denials\"),\n PIILLMEval(\"answer\", alias=\"PII\"),\n CustomColumnDescriptor(\"answer\", greeting, alias=\"Greeting\"),\n LLMEval(\"answer\", template=politeness, provider = \"openai\", \n model = \"gpt-4o-mini\", alias=\"Politeness\")]\n)<\/code><\/pre>\n\n
report = Report([\n TextEvals(),\n MinValue(column=\"Sentiment\", tests=[gte(0)]),\n MinValue(column=\"Length\", tests=[gte(300)]),\n CategoryCount(column=\"Denials\", category = 'NO', tests=[eq(0)]),\n])\n\nmy_eval = report.run(llm_eval_dataset, sot_eval_dataset)\nmy eval<\/code><\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/Screenshot-2025-04-19-at-22.52.46-1024x472.png?resize=1024%2C472&ssl=1\")
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/Screenshot-2025-04-19-at-22.53.48-1024x587.png?resize=1024%2C587&ssl=1\")
CorrectnessLLMEval<\/code>. This descriptor leverages an LLM to compare an answer against the expected one and assess its correctness. You can reference the default prompt directly in code<\/a> or use:<\/p>\nCorrectnessLLMEval(\"llm_answer\", target_output=\"sot_answer\").dict()['feature']<\/code><\/pre>\nacc_eval_dataset = Dataset.from_pandas(\n eval_df[['question', 'llm_answer', 'sot_answer']],\n data_definition=DataDefinition(),\n descriptors=[\n CorrectnessLLMEval(\"llm_answer\", target_output=\"sot_answer\"),\n Sentiment(\"llm_answer\", alias=\"Sentiment\"),\n SentenceCount(\"llm_answer\", alias=\"Sentences\"),\n TextLength(\"llm_answer\", alias=\"Length\")\n ]\n)\nreport = Report([\n TextEvals()\n])\n\nacc_eval = report.run(acc_eval_dataset, None)\nacc_eval<\/code><\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/Screenshot-2025-04-19-at-23.07.07-1024x411.png?resize=1024%2C411&ssl=1\")
Quality in production<\/h2>\n
Tracing<\/h3>\n
from evidently.ui.workspace import CloudWorkspace\nws = CloudWorkspace(token=evidently_token, url=\"https:\/\/app.evidently.cloud\")\n\n# creating a project\nproject = ws.create_project(\"Talk to Your Data demo\", \n org_id=\"<your_org_id>\")\nproject.description = \"Demo project to test Evidently.AI\"\nproject.save()<\/code><\/pre>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/Screenshot-2025-04-20-at-22.44.10-1024x230.png?resize=1024%2C230&ssl=1\")
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/Screenshot-2025-04-19-at-23.45.51-1024x497.png?resize=1024%2C497&ssl=1\")
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/Screenshot-2025-04-19-at-23.41.02-1024x345.png?resize=1024%2C345&ssl=1\")
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/1OBJcFqHZIGorQCFQq5ViDg.png?ssl=1\")