Beyond the headlines and online buzz, how can we assess the model\u2019s reasoning abilities using recognized benchmarks?\u00a0<\/p>\n Deepseek<\/a>\u2019s user interface<\/a> makes it easy to explore its capabilities, but using it programmatically offers deeper insights and more seamless integration into real-world applications. Understanding how to run such models locally also provides enhanced control and offline access.<\/p>\n In this article, we explore how to use Ollama <\/strong>and OpenAI\u2019s simple-evals<\/strong> to evaluate the reasoning capabilities of DeepSeek-R1\u2019s distilled models based on the famous GPQA-Diamond <\/strong>benchmark.<\/p>\n (1) <\/strong>What are Reasoning Models?<\/a> Here is the link to the accompanying GitHub repo<\/a> for this article.<\/p>\n Reasoning models, such as DeepSeek-R1 and OpenAI\u2019s o-series models (e.g., o1, o3), are large language models (LLMs) trained using reinforcement learning to perform reasoning.\u00a0<\/p>\n Reasoning models think before they answer, producing a long internal chain of thought before responding. They excel in complex problem-solving, coding, scientific reasoning, and multi-step planning for agentic workflows.<\/p>\n DeepSeek-R1 is a state-of-the-art open-source LLM designed for advanced reasoning<\/strong>, introduced in January 2025 in the paper \u201c<\/em>DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning<\/em><\/a>\u201d<\/em>.<\/p>\n The model is a 671-billion-parameter LLM trained with extensive use of reinforcement learning (RL), based on this pipeline:<\/p>\n To be precise, DeepSeek trained two models:<\/p>\n The outcome is that the model achieved performance comparable to leading models like OpenAI\u2019s o1 model<\/a> across tasks such as mathematics, coding, and complex reasoning.<\/p>\n Alongside the full model, they also open-sourced six smaller dense models (also named DeepSeek-R1) of different sizes (1.5B, 7B, 8B, 14B, 32B, 70B), distilled from DeepSeek-R1 based on Qwen<\/a> or Llama<\/a> as the base model.<\/p>\n Distillation <\/strong>is a technique where a smaller model (the \u201cstudent\u201d) is trained to replicate the performance of a larger, more powerful pre-trained model (the \u201cteacher\u201d).\u00a0<\/p>\n In this case, the teacher is the 671B DeepSeek-R1 model, and the students are the six models distilled using these open-source base models:<\/p>\n DeepSeek-R1 was used as the teacher model to generate 800,000 training samples, a mix of reasoning and non-reasoning samples, for distillation via supervised fine-tuning<\/strong> of the base models (1.5B, 7B, 8B, 14B, 32B, and 70B).<\/p>\n So why do we do distillation in the first place?\u00a0<\/strong><\/p>\n The goal is to transfer the reasoning abilities of larger models, such as DeepSeek-R1 671B, into smaller, more efficient models. This empowers the smaller models to handle complex reasoning tasks while being faster and more resource-efficient.<\/p>\n Furthermore, DeepSeek-R1 has a massive number of parameters (671 billion),<\/strong> making it challenging to run on most consumer-grade machines.\u00a0<\/p>\n Even the most powerful MacBook Pro, with a maximum of 128GB of unified memory, is inadequate to run a 671-billion-parameter model.<\/p>\n As such, distilled models open up the possibility of being deployed on devices with limited computational resources.<\/p>\n Unsloth<\/a> achieved an impressive feat by quantizing the original 671B-parameter DeepSeek-R1 model down to just 131GB\u200a\u2014\u200aa remarkable 80% reduction in size. However, a 131GB VRAM requirement remains a significant hurdle.<\/em><\/p>\n<\/blockquote>\n With six distilled model sizes to choose from, selecting the right one largely depends on the capabilities of the local device hardware.\u00a0<\/p>\n For those with high-performance GPUs or CPUs and a need for maximum performance, the larger DeepSeek-R1 models (32B and up) are ideal\u200a\u2014\u200aeven the quantized 671B version is viable.<\/p>\n However, if one has limited resources or prefers quicker generation times (as I do), the smaller distilled variants, such as 8B or 14B, are a better fit.<\/p>\n For this project, I will be using the DeepSeek-R1 distilled <\/strong>Qwen-14B<\/strong><\/a> model, which aligns with the hardware constraints I faced.<\/strong><\/p>\n LLMs are typically evaluated using standardized benchmarks that assess their performance across various tasks, including language understanding, code generation, instruction following, and question answering. Common examples include MMLU<\/a>, HumanEval<\/a>, and MGSM<\/a>.<\/p>\n To measure an LLM\u2019s capacity for reasoning, we need more challenging, reasoning-heavy benchmarks that go beyond surface-level tasks. Here are some popular examples focused on evaluating advanced reasoning capabilities:<\/p>\nContents<\/h2>\n
\n
(2) <\/strong>What is DeepSeek-R1?<\/a>
(3) <\/strong>Understanding Distillation and DeepSeek-R1 Distilled Models<\/a>
(4) <\/strong>Selection of Distilled Model<\/a>
(5) <\/strong>Benchmarks for Evaluating Reasoning<\/a>
(6) <\/strong>Tools Used<\/a>
(7) <\/strong>Results of Evaluation<\/a>
(8) <\/strong>Step-by-Step Walkthrough<\/a><\/p>\n<\/blockquote>\n
\n(1) What are Reasoning Models?<\/h2>\n
\n(2) What is DeepSeek-R1?<\/h2>\n
\n
\n
\n(3) Understanding Distillation and DeepSeek-R1\u2019s Distilled Models<\/h2>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/Untitled-1024x510.png?resize=1024%2C510&ssl=1\")
\n
\n
\n(4) Selection of Distilled Model<\/h2>\n
\n(5) Benchmarks for Evaluating Reasoning<\/h2>\n
(i) AIME 2024\u200a\u2014\u200aCompetition Math<\/h3>\n
\n
(ii) Codeforces\u200a\u2014\u200aCompetition Code<\/h3>\n
\n
(iii) GPQA Diamond\u200a\u2014\u200aPhD-Level Science Questions<\/h3>\n
\n