{"id":671,"date":"2024-12-19T07:01:23","date_gmt":"2024-12-19T07:01:23","guid":{"rendered":"https:\/\/mailitics.com\/index.php\/2024\/12\/19\/navigating-soft-actor-critic-reinforcement-learning-8e1a7406ce48\/"},"modified":"2024-12-19T07:01:23","modified_gmt":"2024-12-19T07:01:23","slug":"navigating-soft-actor-critic-reinforcement-learning-8e1a7406ce48","status":"publish","type":"post","link":"https:\/\/mailitics.com\/index.php\/2024\/12\/19\/navigating-soft-actor-critic-reinforcement-learning-8e1a7406ce48\/","title":{"rendered":"Navigating Soft Actor-Critic Reinforcement Learning"},"content":{"rendered":"

Navigating Soft Actor-Critic Reinforcement Learning
\n \t

\n
<\/BR>
\n
\n \t

\n
<\/BR><\/p>\n

\n

Understanding the theory and implementation of SAC RL in the context of Bioengineering<\/h4>\n
\"\"
Image generated by the author using ChatGPT-4o<\/figcaption><\/figure>\n

Introduction<\/h3>\n

The research domain of Reinforcement Learning (RL) has evolved greatly over the past years. The use of deep reinforcement learning methods such as Proximal Policy Optimisation (PPO) (Schulman, 2017) and Deep Deterministic Policy Gradient (DDPG) (Lillicrap, 2015) have enabled agents to solve tasks in high-dimensional environments. However, many of these model-free RL algorithms have struggled with stability during the training process. These challenges arise due to the brittle convergence properties, high variance in gradient estimation, very high sample complexity, and the sensitivity to hyperparameters in continuous action spaces. Given these problems, it is imperative to consider a newly devised RL algorithm that avoids such issues and expands applicability to complex, real-world problems. This new algorithm is the Soft Actor-Critic (SAC) deep RL network. (Haarnoja, 2018)<\/p>\n

\"\"
Model Architecture of Soft Actor-Critic Networks. Image taken from (Du,\u00a02023)<\/a><\/figcaption><\/figure>\n

SAC is an off-policy Actor-Critic deep RL algorithm which is designed to address the stability and efficiency constraints of its predecessors. The SAC algorithm is based on the maximum entropy RL framework which aims for the actor part of the network to maximise the expected reward, while maximising entropy. It combines off-policy updates with a more stable formulation of the stochastic Actor-Critic method. An off-policy algorithm enables faster learning and better sample efficiency using experience replay, unlike on-policy methods such as PPO, which require new samples for each gradient step. For on-policy methods such as PPO, for each gradient step in the learning process, new samples must be collected. The aim of using stochastic policies and maximising entropy comes to promote the robustness and exploration of the algorithm by encouraging more randomness in the actions. Additionally, unlike PPO and DDPG, SAC uses twin Q-networks with a separate Actor network and entropy tuning to improve the stability and convergence when combining off-policy learning with high dimensional, nonlinear function approximation.<\/p>\n

Off-policy RL methods have had a wide impact on bioengineering systems that improve patient lives. More specifically, RL has been applied to domains such as robotic arm control, drug delivery methods and most notably de novo drug design. (Svensson, 2024) Svensson et al. has used a number of on- and off-policy frameworks and different variants of replay buffers to learn a RNN-based molecule generation policy, to be active against DRD2 (a dopamine receptor). The paper realises that using experience replay across the board for high, intermediate and low scoring molecules has shown effects in improving the structural diversity and the number of active molecules generated. Replay buffers improve sample efficiency in training agents. They also reported that the use of off-policy methods and more specifically SAC, helps in promoting structural diversity by preventing mode collapse.<\/p>\n

Theoretical Explanation<\/h3>\n

SAC uses \u2018soft\u2019 value functions by introducing the objective function with an entropy term, \u0397(\u03c0(a|s))<\/strong>. Accordingly, the network seeks to maximise both the expected return of lifetime rewards and the entropy of the policy. The entropy of the policy is defined as the unpredictability of a random variable, which increases with the range of possible values. Thus, the new entropy regularised objective becomes:<\/p>\n

\"\"
Entropy Regularised Objective<\/figcaption><\/figure>\n

\u03b1 <\/strong>is the temperature parameter that balances between exploration and exploitation.<\/p>\n

In the implementation of soft value functions, we aim to maximise the entropy as the algorithm would assign equal probabilities to actions that have a similar Q-value. Maximising entropy also helps with preventing the agent from choosing actions that exploit inconsistencies in approximated Q-values. We can finally understand how SAC improves brittleness by allowing the network to explore more and not assign very high probabilities to one range of actions. This part is inspired by Vaishak V.Kumar<\/a>\u2019s explanation of the entropy maximisation in \u201cSoft Actor-Critic Demystified\u201d.<\/p>\n

The SAC paper authors discuss that since the state value function approximates the soft value, there is really no essential need to train separate function approximators for the policy, since they relate to the state value according to the following equation. However, training three separate approximators provided better convergence.<\/p>\n

\"\"
Soft State Value\u00a0Function<\/figcaption><\/figure>\n

The three function approximator networks are characterised as\u00a0follows:<\/p>\n