{"id":2121,"date":"2025-02-28T07:05:19","date_gmt":"2025-02-28T07:05:19","guid":{"rendered":"https:\/\/mailitics.com\/index.php\/2025\/02\/28\/how-llms-work-reinforcement-learning-rlhf-deepseek-r1-openai-o1-alphago\/"},"modified":"2025-02-28T07:05:19","modified_gmt":"2025-02-28T07:05:19","slug":"how-llms-work-reinforcement-learning-rlhf-deepseek-r1-openai-o1-alphago","status":"publish","type":"post","link":"https:\/\/mailitics.com\/index.php\/2025\/02\/28\/how-llms-work-reinforcement-learning-rlhf-deepseek-r1-openai-o1-alphago\/","title":{"rendered":"How LLMs Work: Reinforcement Learning, RLHF, DeepSeek R1, OpenAI o1, AlphaGo"},"content":{"rendered":"

How LLMs Work: Reinforcement Learning, RLHF, DeepSeek R1, OpenAI o1, AlphaGo
\n \t

\n
<\/BR>
\n
\n \t

\n
<\/BR><\/p>\n

\n

Welcome to part 2 of my LLM deep dive. If you\u2019ve not read Part 1, I highly encourage you to check it out first<\/a>.\u00a0<\/p>\n

Previously, we covered the first two major stages of training an LLM:<\/p>\n

    \n
  1. Pre-training\u200a\u2014\u200aLearning from massive datasets to form a base model.<\/li>\n
  2. Supervised fine-tuning (SFT)\u200a\u2014\u200aRefining the model with curated examples to make it useful.<\/li>\n<\/ol>\n

    Now, we\u2019re diving into the next major stage: Reinforcement Learning (RL)<\/strong>. While pre-training and SFT are well-established, RL is still evolving but has become a critical part of the training pipeline.<\/p>\n

    I\u2019ve taken reference from Andrej Karpathy\u2019s widely popular 3.5-hour YouTube<\/a>. Andrej is a founding member of OpenAI, his insights are gold\u200a\u2014\u200ayou get the idea.<\/p>\n

    Let\u2019s go \"\ud83d\ude80\"<\/p>\n

    What\u2019s the purpose of reinforcement learning (RL)?<\/h2>\n

    Humans and LLMs process information differently. What\u2019s intuitive for us\u200a\u2014\u200alike basic arithmetic\u200a\u2014\u200amay not be for an LLM, which only sees text as sequences of tokens. Conversely, an LLM can generate expert-level responses on complex topics simply because it has seen enough examples during training.<\/p>\n

    This difference in cognition makes it challenging for human annotators to provide the \u201cperfect\u201d set of labels that consistently guide an LLM toward the right answer.<\/p>\n

    RL bridges this gap by allowing the model to <\/em>learn from its own experience<\/em><\/strong>.<\/em><\/p>\n

    Instead of relying solely on explicit labels, the model explores different token sequences and receives feedback\u200a\u2014\u200areward signals\u200a\u2014\u200aon which outputs are most useful. Over time, it learns to align better with human intent.<\/p>\n

    Intuition behind RL<\/h2>\n

    LLMs are stochastic\u200a\u2014\u200ameaning their responses aren\u2019t fixed. Even with the same prompt, the output varies because it\u2019s sampled from a probability distribution.<\/p>\n

    We can harness this randomness by generating thousands or even millions of possible responses in parallel<\/strong>. Think of it as the model exploring different paths\u200a\u2014\u200asome good, some bad. Our goal is to encourage it to take the better paths more often.<\/strong><\/p>\n

    To do this, we train the model on the sequences of tokens that lead to better outcomes. Unlike supervised fine-tuning, where human experts provide labeled data, reinforcement learning allows the model to <\/strong>learn from itself.<\/em><\/strong><\/p>\n

    The model discovers which responses work best, and after each training step, we update its parameters. Over time, this makes the model more likely to produce high-quality answers when given similar prompts in the future.<\/p>\n

    But how do we determine which responses are best? And how much RL should we do? The details are tricky, and getting them right is not trivial.<\/p>\n

    RL is not \u201cnew\u201d\u200a\u2014\u200aIt can surpass human expertise (AlphaGo, 2016)<\/h2>\n

    A great example of RL\u2019s power is DeepMind\u2019s AlphaGo, the first AI to defeat a professional Go player and later surpass human-level play.<\/p>\n

    In the 2016 Nature paper<\/a> (graph below), when a model was trained purely by SFT (giving the model tons of good examples to imitate from), the model was able to reach human-level performance, but never surpass it<\/strong>.<\/p>\n

    The dotted line represents Lee Sedol\u2019s performance\u200a\u2014\u200athe best Go player in the world.<\/p>\n

    This is because SFT is about replication, not innovation\u200a\u2014\u200ait doesn\u2019t allow the model to discover new strategies beyond human knowledge.<\/em><\/strong><\/p>\n

    However, RL enabled AlphaGo to play against itself, refine its strategies, and ultimately exceed human expertise<\/strong> (blue line).<\/p>\n

    \"\"
    Image taken from AlphaGo 2016 paper<\/a><\/figcaption><\/figure>\n

    RL represents an exciting frontier in AI\u200a\u2014\u200awhere models can explore strategies beyond human imagination when we train it on a diverse and challenging pool of problems to refine it\u2019s thinking strategies.<\/p>\n

    RL foundations recap<\/h2>\n

    Let\u2019s quickly recap the key components of a typical RL setup:<\/p>\n

    \"\"
    Image by author<\/figcaption><\/figure>\n