Deepseek<\/a> has recently made quite a buzz <\/strong>in the AI community, thanks to its impressive performance at relatively low costs. I think this is a perfect opportunity to dive deeper into how Large Language Models (LLMs) are trained. In this article, we will focus on the Reinforcement Learning (RL) side of things: we will cover TRPO, PPO, and, more recently, GRPO (don\u2019t worry, I will explain all these terms soon!)\u00a0<\/p>\n I have aimed to keep this article relatively easy to read and accessible, by minimizing the math, so you won\u2019t need a deep Reinforcement Learning background to follow along. However, I will assume that you have some familiarity with Machine Learning, Deep Learning, and a basic understanding of how LLMs work.<\/p>\n I hope you enjoy the article!<\/p>\n Before diving into RL specifics, let\u2019s briefly recap the three main stages of training a Large Language Model:<\/p>\n Before diving deeper, let\u2019s briefly revisit the core ideas behind Reinforcement Learning.<\/p>\n RL is quite straightforward to understand at a high level: an agent<\/strong> interacts with an environment<\/strong>. The agent resides in a specific state<\/strong> within the environment and can take actions<\/strong> to transition to other states. Each action yields a reward<\/strong> from the environment: this is how the environment provides feedback that guides the agent\u2019s future actions.\u00a0<\/p>\n Consider the following example: a robot<\/strong> (the agent) navigates (and tries to exit) a maze<\/strong> (the environment).<\/p>\n Easy! Now, let\u2019s now make an analogy to how RL is used in the context of LLMs.<\/p>\n When used during LLM training, RL is defined by the following components:<\/p>\n Note: in some cases, rewards can actually get simpler. For example, in DeepSeekMath, rule-based approaches<\/strong> can be used because math responses tend to be more deterministic (correct or wrong answer)<\/p>\n<\/blockquote>\n Policy<\/strong> is the final concept we need for now. In RL terms, a policy is simply the strategy for deciding which action to take. In the case of an LLM, the policy outputs a probability distribution over possible tokens at each step: in short, this is what the model uses to sample the next token to generate. Concretely, the policy is determined by the model\u2019s parameters (weights). During RL training, we adjust these parameters so the LLM becomes more likely to produce \u201cbetter\u201d tokens\u2014 that is, tokens that produce higher reward scores.<\/p>\n We often write the policy as:<\/p>\n where a<\/em> is the action (a token to generate), s<\/em> the state (the query and tokens generated so far), and \u03b8 <\/em>(model\u2019s parameters).<\/p>\n This idea of finding the best policy is the whole point of RL! Since we don\u2019t have labeled data (like we do in supervised learning) we use rewards to adjust our policy to take better actions.<\/strong> (In LLM terms: we adjust the parameters of our LLM to generate better tokens.)<\/em><\/p>\n Let\u2019s take a quick step back to how supervised learning typically works. you have labeled data and use a loss function (like cross-entropy) to measure how close your model\u2019s predictions are to the true labels.<\/p>\n We can then use algorithms like backpropagation and gradient descent to minimize our loss function and update the weights \u03b8<\/em> of our model.<\/p>\n Recall that our policy also outputs probabilities! In that sense, it is analogous to the model\u2019s predictions in supervised learning\u2026 We are tempted to write something like<\/em>:<\/p>\n where s<\/em> is the current state and a<\/em> is a possible action.<\/p>\n A(s, a) <\/em>is called the advantage function<\/strong> and measures how good is the chosen action in the current state, compared to a baseline. This is very much like the notion of labels <\/strong>in supervised learning but derived from rewards<\/strong> instead of explicit labeling. To simplify<\/em>, we can write the advantage as:<\/p>\n In practice, the baseline is calculated using a value function<\/strong>. This is a common term in RL that I will explain later. What you need to know for now is that it measures the expected reward we would receive if we continue following the current policy from the state s<\/em>.<\/p>\n TRPO (Trust Region Policy Optimization) builds on this idea of using the advantage function but adds a critical ingredient for stability<\/strong>: it constrains<\/strong> how far the new policy can deviate from the old policy at each update step (similar to what we do with batch gradient descent for example).<\/p>\n Putting it all together, TRPO tries to maximize<\/strong> a surrogate objective (which involves the advantage and the policy ratio) subject to a KL divergence constraint<\/strong>.<\/p>\n While TRPO was a significant advancement, it\u2019s no longer used widely in practice, especially for training LLMs, due to its computationally intensive gradient calculations.<\/p>\n Instead, PPO is now the preferred approach in most LLMs architecture, including ChatGPT, Gemini, and more.<\/p>\n<\/blockquote>\n It is actually quite similar to TRPO, but instead of enforcing a hard constraint on the KL divergence<\/strong>, PPO introduces a \u201cclipped<\/strong> surrogate objective\u201d that implicitly restricts policy updates, and greatly simplifies the optimization process.<\/p>\n Here is a breakdown of the PPO objective function we maximize to tweak our model\u2019s parameters.<\/p>\n Let\u2019s first talk more about the advantage<\/strong> and the value functions<\/strong> I introduced earlier.<\/p>\n In typical setups (like PPO), a value model<\/strong> is trained alongside the policy. Its goal is to predict the value of each action we take (each token generated by the model), using the rewards we obtain (remember that the value should represent the expected cumulative reward).<\/p>\n Here is how it works in practice. Take the query \u201cWhat is 2+2?\u201d as an example. Our model outputs \u201c2+2 is 4\u201d and receives a reward of 0.8 for that response. We then go backward and attribute discounted rewards<\/strong> to each prefix:<\/p>\n where \u03b3<\/em> is the discount factor (0.9 for example). We then use these prefixes and associated values to train the value model.<\/p>\n Important note: the value model and the reward model are two different things. The reward model is trained before the RL process and uses pairs of (query, response) and human ranking. The value model is trained concurrently to the policy, and aims at predicting the future expected reward at each step of the generation process.<\/p>\n<\/blockquote>\n Even if in practice, the reward model is often derived from the policy (training only the \u201chead\u201d), we still end up maintaining many models and handling multiple training procedures (policy, reward, value model). GRPO<\/strong> streamlines this by introducing a more efficient method.<\/p>\n Remember what I said earlier?<\/p>\n In PPO, we decided to use our value function as the baseline. GRPO chooses something else: Here is what GRPO does: concretely, for each query<\/strong>, GRPO generates a group of responses (group of size G) and uses their rewards to calculate each response\u2019s advantage as a z-score<\/strong>:<\/p>\n where r\u1d62<\/em> is the reward of the i<\/em>-th response and \u03bc<\/em> and \u03c3<\/em> are the mean and standard deviation of rewards in that group.<\/p>\n This naturally eliminates the need for a separate value model. This idea makes a lot of sense when you think about it! It aligns with the value function we introduced before<\/strong> and also measures, in a sense, an \u201cexpected\u201d reward we can obtain. Also, this new method is well adapted to our problem because LLMs can easily generate multiple non-deterministic outputs<\/strong> by using a low temperature <\/em>(controls the randomness of tokens generation).<\/p>\n This is the main idea behind GRPO: getting rid of the value model.<\/p>\n<\/blockquote>\n Finally, GRPO adds a KL divergence<\/strong> term (to be exact, GRPO uses a simple approximation of the KL divergence to improve the algorithm further) directly into its objective, comparing the current policy to a reference policy<\/strong> (often the post-SFT model).<\/p>\n See the final formulation below:<\/p>\n And\u2026 that\u2019s mostly it for GRPO!<\/strong> I hope this gives you a clear overview of the process: it still relies on the same foundational ideas as TRPO and PPO but introduces additional improvements to make training more efficient, faster, and cheaper\u200a\u2014\u200akey factors behind DeepSeek\u2019s success<\/strong>.<\/p>\n Reinforcement Learning has become a cornerstone for training today\u2019s Large Language Models, particularly through PPO, and more recently GRPO. Each method rests on the same RL fundamentals\u200a\u2014\u200astates, actions, rewards, and policies\u200a\u2014\u200abut adds its own twist to balance stability, efficiency, and human alignment:<\/p>\n \u2022 TRPO<\/strong> introduced strict policy constraints via KL divergence<\/p>\n \u2022 PPO<\/strong> eased those constraints with a clipped objective<\/p>\n \u2022 GRPO<\/strong> took an extra step by removing the value model requirement and using group-based reward normalization. Of course, DeepSeek also benefits from other innovations, like high-quality data and other training strategies, but that is for another time!<\/p>\n I hope this article gave you a clearer picture of how these methods connect and evolve. I believe that Reinforcement Learning will become the main focus in training LLMs<\/strong> to improve their performance, surpassing pre-training and SFT in driving future innovations.\u00a0<\/p>\n If you\u2019re interested in diving deeper, feel free to check out the references below or explore my previous posts.<\/p>\n Thanks for reading, and feel free to leave a clap and a comment!<\/p>\n Want to learn more about Transformers or dive into the math behind the Curse of Dimensionality? Check out my previous articles:<\/p>\n Transformers: How Do They Transform Your Data?<\/strong> The Math Behind \u201cThe Curse of Dimensionality\u201d<\/strong> References:<\/p>\n The post Training Large Language Models: From TRPO to\u00a0GRPO<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\nThe 3 steps of LLM\u00a0training<\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1600\/1%2AgDF9kq-aLcUvG0XFBU2DGA.png?ssl=1\")
\n
Reinforcement Learning\u00a0Basics<\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1600\/0%2An2QZe4jUFk_OccaY.png?ssl=1\")
\n
RL in the context of\u00a0LLMs<\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1600\/1%2A8hJsPwyITye84NWqHAe2MA.png?ssl=1\")
\n
\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1600\/0%2Aoj50mh1PYdcv5CBf.png?ssl=1\")
<\/figure>\nTRPO (Trust Region Policy Optimization)<\/h3>\n
An analogy with supervised learning<\/h4>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1600\/0%2AYaLNgOyGdTJdnmzK.png?ssl=1\")
<\/figure>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1600\/0%2AmCxBSj_yWyLAHEEV.png?ssl=1\")
<\/figure>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1600\/0%2ABkfvw7w81TyeuKEs.png?ssl=1\")
<\/figure>\nWhat is\u00a0TRPO?<\/h4>\n
\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1600\/0%2A2pSGRrxLNHmL9Yns.png?ssl=1\")
<\/figure>\n\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1600\/0%2AO6IZum3z9DVSJTIS.png?ssl=1\")
<\/figure>\nPPO (Proximal Policy Optimization)<\/h3>\n
\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/2400\/1%2AoVi9NcQvD15nTp4G7rJueA.png?ssl=1\")
GRPO (Group Relative Policy Optimization)<\/h3>\n
How is the value function usually obtained?<\/h4>\n
\n
\n
What\u2019s new in\u00a0GRPO<\/h4>\n
<\/figure>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1600\/0%2A3cqWAmG6B5tNjU8A.png?ssl=1\")
<\/figure>\n\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/2400\/1%2Auw87yMbJbxD2RsV2WWQPrg.png?ssl=1\")
Conclusion<\/h3>\n
\n
Diving into the Transformers architecture and what makes them unbeatable at language tasks<\/em>towardsdatascience.com<\/a><\/a><\/p>\n
Dive into the \u201cCurse of Dimensionality\u201d concept and understand the math behind all the surprising phenomena that arise\u2026<\/em>towardsdatascience.com<\/a><\/a><\/p>\n
\n\n
\n\n