In July 1959, Arthur Samuel developed one of the first agents <\/em>to play the game of checkers. What constitutes an agent that plays checkers can be best described in Samuel\u2019s own words, \u201c\u2026a computer [that] can be programmed so that it will learn to play a better game of checkers than can be played by the person who wrote the program\u201d [1]. The checkers\u2019 agent tries to follow the idea of simulating every possible move given the current situation <\/strong>and selecting the most advantageous<\/em> one i.e. one that brings the player closer to winning. The move\u2019s \u201cadvantageousness\u201d <\/em>is determined by an evaluation function, which the agent improves through experience. Naturally, the concept of an agent is not restricted to the game of checkers, and many practitioners have sought to match or surpass human performance in popular games. Notable examples include IBM\u2019s Deep Blue <\/em>(which managed to defeat Garry Kasparov, a chess world champion at the time), and Tesauro\u2019s TD-Gammon, <\/em>a temporal-difference<\/strong> approach, where the evaluation function was modelled using a neural network. In fact, TD-Gammon<\/em>\u2019s playing style was so uncommon that some experts even adopted some strategies it conjured up [2].<\/p>\n Unsurprisingly, research into creating such \u2018agents\u2019 only skyrocketed, with novel approaches able to reach peak human performance in complex games. In this post, we explore one such approach: the DQN<\/strong> approach introduced in 2013 by Mnih et al, in which playing Atari games is approached through a synthesis of Deep Neural Networks<\/a> <\/strong>and TD-Learning <\/strong>(NB: <\/strong>the original paper came out in 2013, but we will focus on the 2015 version which comes with some technical improvements) [3, 4]. Before we continue, you should note that in the ever-expanding space of new approaches, DQN has been superseded by faster and more refined state-of-the-art methods. Yet, it remains an ideal stepping stone in the field of Deep Reinforcement Learning<\/strong>, widely recognized for combining deep learning with reinforcement learning. Hence, readers aiming to dive into Deep-RL are encouraged to begin with DQN.<\/p>\n This post is sectioned as follows: first, I define the problem with playing Atari games and explain why some traditional methods can be intractable. Finally, I present the specifics of the DQN approach and dive into the technical implementation.<\/p>\n For the remainder of the post, I\u2019ll assume that you know the basics of supervised learning, neural networks (basic <\/em>FFNs<\/em><\/a> and <\/em>CNNs<\/em><\/a>) and also basic reinforcement learning concepts (Bellman equations, TD-learning, Q-learning etc) If some of these RL concepts are foreign to you, then this <\/em>playlist<\/em><\/a> is a good introduction.\u00a0\u00a0<\/em><\/p>\n Atari<\/em> is a nostalgia-laden term, featuring iconic games such as Pong, Breakout, Asteroids <\/em>and many more. In this post, we restrict ourselves to Pong. Pong is a 2-player game, where each player controls a paddle and can use said paddle to hit the incoming ball. Points are scored when the opponent is unable to return the ball, in other words, the ball goes past them. A player wins when they reach 21 points.\u00a0<\/p>\n Considering the sequential nature of the game, it might be appropriate to frame the problem as an RL problem, and then apply one of the solution methods. We can frame the game as an MDP:<\/p>\n The states would represent the current game state (where the ball or player paddle is etc, analogous to the idea of a search state). The rewards encapsulate our idea of winning and the actions correspond to the buttons on the Atari 2600 console. Our goal now becomes finding a policy<\/p>\n also known as the optimal policy<\/em>. Let\u2019s see what might happen if we try to train an agent using some classical RL algorithms.\u00a0<\/p>\n A straightforward solution might entail solving the problem using a tabular approach. We could enumerate all states (and actions) and associate each state with a corresponding state or state-action value. We could then apply one of the classical RL methods (Monte-Carlo, TD-Learning, Value Iteration etc), taking a dynamic Programming<\/a> approach. However, employing this approach faces large pitfalls rather quickly. What do we consider as states? How many states do we have to enumerate?<\/p>\n It quickly becomes quite difficult to answer these questions. Defining a state becomes difficult as many elements are in play when considering the idea of a state (i.e. the states need to be Markovian, encapsulate a search state etc). What about visual output (frames) to represent a state? After all this is how we as humans interact with Atari games. We see frames, deduce information regarding the game state and then choose the appropriate action. However, there are impossibly many states when using this representation, which would make our tabular approach quite intractable, memory-wise.<\/p>\n Now for the sake of argument imagine that we have enough memory to hold a table of this size. Even then we would need to explore all the states a good number of times to get good approximations of the value function. We would need to explore <\/em>all possible states (or state-action) enough times to arrive at a useful value. Herein lies the runtime hurdle; it would be quite infeasible for the values to converge for all the states in the table in a reasonable amount of time as we have infinite states.<\/p>\n Perhaps instead of framing it as a reinforcement learning problem, can we instead rephrase it into a supervised learning problem? Perhaps a formulation in which the states are samples and the labels are the actions performed. Even this perspective brings forth new problems. Atari games are inherently sequential, each state is sampled based on the previous. This breaks the i.i.d assumptions applied in supervised learning, negatively affecting supervised learning-based solutions. Similarly, we would need to create a hand-labelled dataset, perhaps employing a human expert to hand label actions for each frame. This would be expensive and laborious, and still might yield insufficient results.<\/p>\n Solely relying on either supervised learning or RL may lead to inefficient learning, whether due to computational constraints or suboptimal policies. This calls for a more efficient approach to solving Atari games.<\/p>\n I assume you have some basic knowledge of PyTorch, Numpy and Python, though I\u2019ll try to be as articulate as possible. For those unfamiliar, I recommend consulting: <\/em>pytorch <\/em><\/a>& <\/em>numpy<\/em><\/a>.\u00a0<\/em><\/p>\n Deep-Q Networks aim to overcome the aforementioned barriers through a variety of techniques. Let\u2019s go through each of the problems step-by-step and address how DQN mitigates or solves these challenges.<\/p>\n It\u2019s quite hard to come up with a formal state definition for Atari games due to their diversity. DQN is designed to work for most Atari games, and as a result, we need a stated formalization that is compatible with said games. To this end, the visual representation (pixel values) of the games at any given moment are used to fashion a state. Naturally, this entails a continuous state space. This connects to our previous discussion on potential ways to represent states.<\/p>\n The challenge of continuous states is solved through function approximation. <\/em>Function approximation (FA) aims to approximate the state-action value function directly using a function approximation. Let\u2019s go through the steps to understand what the FA does.\u00a0<\/p>\n Imagine that we have a network that given a state outputs the value of being in said state and performing a certain action. We then select actions based on the highest reward. However, this network would be short-sighted, only taking into account one timestep. Can we incorporate possible rewards from further down the line? Yes we can! This is the idea of the expected return<\/em>. From this view, the FA becomes quite simple to understand; we aim to find a function: <\/a><\/p>\n In other words, a function which outputs the expected return of being in a given state after performing an action.\u00a0<\/em><\/p>\n This idea of approximation becomes crucial due to the continuous nature of the state space. By using a FA, we can exploit the idea of generalization. States close to each other (similar pixel values) will have similar Q-values, meaning that we don\u2019t need to cover the entire (infinite) state space, greatly lowering our computational overhead.\u00a0<\/p>\n DQN employs FA in tandem with Q-learning. <\/em>As a small refresher, Q-learning aims to find the expected return for being in a state and performing a certain action using bootstrapping<\/em>. Bootstrapping models the expected return that we mentioned using the current Q-function. This ensures that we don\u2019t need to wait till the end of an episode to update our Q-function. Q-learning is also 0ff-policy<\/em>, which means that the data we use to learn the Q-function is different from the actual policy being learned. The resulting Q-function <\/em>then corresponds to the optimal Q-function and can be used to find the optimal policy (just find the action that maximizes the Q-value in a given state). Moreover, Q-learning is a model-free <\/em>solution, meaning that we don\u2019t need to know the dynamics of the environment (transition functions etc) to learn an optimal policy, unlike in value iteration. Thus, DQN is also off-policy and model-free.<\/p>\n By using a neural network as our approximator, we need not construct a full table containing all the states and their respective Q-values. Our neural network will output the Q-value for being a given state and performing a certain action. From this point on, we refer to the approximator as the Q-network.<\/p>\n Since our states are defined by images, using a basic feed-forward network (FFN) would incur a large computational overhead. For this specific reason, we employ the use of a convolutional network, which is much better able to learn the distinct features of each state. The CNNs are able to distill the images down to a representation (this is the idea of representation learning), which is then fed to a FFN. The neural network architecture can be seen above. Instead of returning one value for:<\/p>\n we return an array with each value corresponding to a possible action in the given state (for Pong we can perform 6 actions, so we return 6 values).<\/p>\n Recall that to train a neural network we need to define a loss function that captures our goals. DQN uses the MSE loss function. For the predicted values we the output of our Q-network. For the true values, we use the bootstrapped values. Hence, our loss function becomes the following:<\/p>\n If we differentiate the loss function with respect to the weights we arrive at the following equation.<\/p>\n Plugging this into the stochastic gradient descent (SGD) equation, we arrive at Q-learning <\/em>[4].\u00a0<\/p>\n By performing SGD updates using the MSE loss function, we perform Q-learning. However, this is an approximation of Q-learning, as we don\u2019t update on a single move but instead on a batch of moves. The expectation is simplified for expedience, though the message remains the same.<\/p>\n From another perspective, you can also think of the MSE loss function as nudging the predicted Q-values as close to the bootstrapped Q-values (after all this is what the MSE loss intends). This inadvertently mimics Q-learning, and slowly converges to the optimal Q-function.<\/p>\n By employing a function approximator, we become subject to the conditions of supervised learning, namely that the data is i.i.d. But in the case of Atari games (or MDPs) this condition is often not upheld. Samples from the environment are sequential in nature, making them dependent on each other. Similarly, as the agent improves the value function and updates its policy, the distribution from which we sample also changes, violating the condition of sampling from an identical distribution.<\/p>\n To solve this the authors of DQN capitalize on the idea of an experience replay<\/em>. This concept is core to keep the training of DQN stable and convergent. An experience replay is a buffer which stores the tuple (s, a, r, s\u2019, d) <\/em>where s, a, r, s\u2019 <\/em>are returned after performing an action in an MDP, and d <\/em>is a boolean representing whether the episode has finished or not. The replay has a maximum capacity which is defined beforehand. It might be simpler to think of the replay as a queue or a FIFO data structure; old samples are removed to make room for new samples. The experience replay is used to sample a random batch of tuples which are then used for training.<\/p>\n The experience replay helps with the alleviation of two major challenges when using neural network function approximators with RL problems. The first deals with the independence of the samples. By randomly<\/strong> sampling a batch of moves and then using those for training we decouple the training process from the sequential nature of Atari games. Each batch may have actions from different timesteps (or even different episodes), giving a stronger semblance of independence.\u00a0<\/p>\n Secondly, the experience replay addresses the issue of non-stationarity. As the agent learns, changes in its behaviour are reflected in the data. This is the idea of non-stationarity; <\/em>the distribution of data changes over time. By reusing samples in the replay and using a FIFO structure, we limit the adverse effects of non-stationarity on training. The distribution of the data still changes, but slowly and its effects are less impactful. Since Q-learning is an off-policy algorithm, we still end up learning the optimal policy, making this a viable solution. These changes allow for a more stable training procedure.<\/p>\n As a serendipitous side effect, the experience replay also allows for better data efficiency. Before training examples were discarded after being used for a single update step. However, through the use of an experience replay, we can reuse moves that we have made in the past for updates.<\/p>\n A change made in the 2015 Nature version of DQN was the introduction of a target network. Neural networks are fickle; slight changes in the weights can introduce drastic changes in the output. This is unfavourable for us, as we use the outputs of the Q-network to bootstrap our targets. If the targets are prone to large changes, it will destabilize training, which naturally we want to avoid. To alleviate this issue, the authors introduce a target network<\/em>, which copies the weights of the Q-network every set amount of timesteps. By using the target network for bootstrapping, our bootstrapped targets are less unstable, making training more efficient.<\/p>\n Lastly, the DQN authors stack four consecutive frames after executing an action. This remark is made to ensure the Markovian property holds [9]. A singular frame omits many details of the game state such as the velocity and direction of the ball. A stacked representation is able to overcome these obstacles, providing a holistic view of the game at any given timestep.<\/p>\n With this, we have covered most of the major techniques used for training a DQN agent. Let\u2019s go over the training procedure. The procedure will be more of an overview, and we\u2019ll iron out the details in the implementation section.<\/p>\n One important clarification arises from step 2. In this step, we perform a process called \u03b5-greedy action selection. In \u03b5-greedy, we randomly choose an action with probability \u03b5, and otherwise choose the best possible action (according to our learned Q-network). Choosing an appropriate \u03b5 allows for the sufficient exploration of actions which is crucial to converge to a reliable Q-function. We often start with a high \u03b5 and slowly decay this value over time.<\/p>\n If you want to follow along with my implementation of DQN then you will need the following libraries (apart from Numpy and PyTorch). I provide a concise explanation of their use.<\/p>\n Let\u2019s import all of the necessary libraries.<\/p>\n First, we construct an environment, using the ALE framework. Since we are working with pong we create an environment with the name The With this, we are given states (we name them observations) with the shape (210, 160, 3). Hence the states are RGB images with the shape 210\u00d7160. An example can be seen in Figure 2. When training our DQN agent, an image of this size adds unnecessary computational overhead. A similar observation can be made about the fact that the frames are RGB (3 channels).<\/p>\n To solve this, we downsample the frame down to 84\u00d784 and transform it into grayscale. We can do this by employing a wrapper from SB3, which does this for us. Now every time we perform an action our output will be in grayscale (with 1 channel) and of size 84\u00d784.<\/p>\n The wrapper above does more than downsample and turn our frame into grayscale. Let\u2019s go over some other changes the wrapper introduces.<\/p>\n Apart from these we also introduce an additional frame stack wrapper ( Let\u2019s combine all of the wrappers into one function that returns an environment that does all of the above.<\/p>\n These changes are derived from the 2015 Nature paper and help to stabilize training [3]. The interfacing with Now that we have an appropriate environment let\u2019s move on to create the replay buffer.<\/p>\n The replay buffer works by allocating space in the memory for the given capacity. We maintain a pointer that keeps track of the number of objects added. Every time a new tuple is added we replace the oldest tuples with the new ones. To sample a minibatch, we first randomly sample a minibatch in Some of the aspects of the replay buffer are inspired by [8]. The replay buffer proved to be the biggest bottleneck in training the agent, and thus small speed-ups in the code proved to be monumentally important. An alternative strategy which uses an We can now use this to create a function that creates a buffer and preloads a given number of tuples with a random policy.<\/p>\n The function is quite straightforward, we create a buffer and environment object and then preload the buffer using a random policy. Note that we squeeze the observations to remove the redundant color channel. Let\u2019s move on to the next step and define the function approximator.<\/p>\n That covers the model architecture. I used a linear \u03b5-decay scheme, but feel free to try another. We can also create an auxiliary class that keeps track of important metrics. The class keeps track of rewards received for the last few episodes along with the respective lengths of said episodes.<\/p>\n Great! Now we have everything we need to start training our agent. Let\u2019s define the training function and go over how it works. Before that, we need to create the necessary objects to pass into our training function along with some hyperparameters. A small note: in the paper the authors use RMSProp, but instead we\u2019ll use Adam. Adam proved to work for me with the given parameters, but you are welcome to try RMSProp or other variations.<\/p>\n The training procedure closely follows Figure 6 and the algorithm described in the paper [4]. We first create the necessary objects such as the loss function etc and reset the environment. Then we can start the training loop, by using the Q-network to give us an action based on the \u03b5-greedy policy. We simulate the environment one step forward using the action and push the resultant tuple onto the replay. If the update frequency condition is met, we can proceed with a training step. The motivation behind the update frequency element is something I am not 100% confident in. Currently, the explanation I can provide revolves around computational efficiency: training every 4 steps instead of every step majorly speeds up the algorithm and seems to work relatively well. In the update step itself, we sample a minibatch of tuples and run the model forward to produce predicted Q-values. We then create the target values (the bootstrapped true labels) using the piecewise function in step 5 in Figure 6. Performing an SGD step becomes quite straightforward from this point, since we can rely on autograd<\/a> to compute the gradients and the optimizer to update the parameters.<\/p>\n If you followed along until now, you can use the following test function to test your saved model.<\/p>\n Here\u2019s how you can use it:<\/p>\n That\u2019s everything for the code! You can see a trained agent below in Figure 8. It behaves quite similar to a human might play Pong, and is able to (consistently) beat the AI on the easiest difficulty. This naturally invites the question, how well does it perform on higher difficulties? Try it out using your own agent or my trained one!\u00a0<\/p>\n An additional agent was trained on the game Breakout as well, the agent can be seen in Figure 9. Once again, I used the default mode and difficulty. It might be interesting to see how well it performs in different modes or difficulties.<\/p>\n DQN solves the issue of training agents to play Atari games. By using a FA, experience replay etc, we are able to train an agent that mimics or even surpasses human performance in Atari games [3]. Deep-RL agents can be finicky and you might have noticed that we use a lot <\/strong>of techniques to ensure that training is stable. If things are going wrong with your implementation it might not hurt to look at the details again.\u00a0<\/p>\n If you want to check out the code for my implementation you can use this link<\/a>. The repo also contains code to train your own model on the game of your choice (as long as it\u2019s in ALE), as well as the trained weights for both Pong and Breakout.<\/p>\n I hope this was a helpful introduction to training DQN agents. To take things to the next level maybe you can try to tweak details to beat the higher difficulties. If you want to look further, there are many extensions to DQN you can explore, such as Dueling DQNs, Prioritized Replay etc.\u00a0<\/p>\n [1] A. L. Samuel, \u201cSome Studies in Machine Learning Using the Game of Checkers,\u201d IBM Journal of Research and Development<\/em>, vol. 3, no. 3, pp. 210\u2013229, 1959. doi:10.1147\/rd.33.0210.<\/p>\n [2] Sammut, Claude; Webb, Geoffrey I., eds. (2010), \u201cTD-Gammon\u201d<\/a>, Encyclopedia of Machine Learning<\/em>, Boston, MA: Springer US, pp. 955\u2013956, doi<\/a>:10.1007\/978\u20130\u2013387\u201330164\u20138_813<\/a>, ISBN<\/a> 978\u20130\u2013387\u201330164\u20138<\/a>, retrieved 2023\u201312\u201325<\/p>\n [3] Mnih, Volodymyr, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, \u2026 and Demis Hassabis. \u201cHuman-Level Control through Deep Reinforcement Learning.\u201d Nature<\/em> 518, no. 7540 (2015): 529\u2013533. https:\/\/doi.org\/10.1038\/nature14236<\/a><\/p>\n [4] Mnih, Volodymyr, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, \u2026 and Demis Hassabis. \u201cPlaying Atari with Deep Reinforcement Learning.\u201d arXiv preprint arXiv:1312.5602<\/em> (2013). https:\/\/arxiv.org\/abs\/1312.5602<\/a><\/p>\n [5] Sutton, Richard S., and Andrew G. Barto. Reinforcement Learning: An Introduction<\/em>. 2nd ed., MIT Press, 2018.<\/p>\n [6] Russell, Stuart J., and Peter Norvig. Artificial Intelligence: A Modern Approach<\/em>. 4th ed., Pearson, 2020.<\/p>\n [7] Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning<\/em>. MIT Press.<\/p>\n [8] Bailey, Jay. Deep Q-Networks Explained. 13 Sept. 2022, www.lesswrong.com\/posts\/kyvCNgx9oAwJCuevo\/deep-q-networks-explained<\/a>.<\/p>\n [9] Hausknecht, M., & Stone, P. (2015). Deep recurrent Q-learning for partially observable MDPs. arXiv preprint arXiv:1507.06527<\/em>. https:\/\/arxiv.org\/abs\/1507.06527<\/a><\/p>\n The post Learning How to Play Atari Games Through Deep Neural Networks<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\nThe Problem At Hand<\/h2>\n
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/Screenshot-2025-02-18-at-11.14.28%25E2%2580%25AFAM.png?resize=350%2C66&ssl=1\")
<\/figure>\n![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/Screenshot-2025-02-18-at-11.15.41%25E2%2580%25AFAM.png?resize=232%2C56&ssl=1\")
<\/figure>\nDQN: Intuition & Implementation<\/h2>\n
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/Screenshot-2025-02-18-at-11.48.42%25E2%2580%25AFAM.png?resize=300%2C56&ssl=1\")
<\/figure>\n![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/Screenshot-2025-02-18-at-11.51.38%25E2%2580%25AFAM.png?resize=126%2C56&ssl=1\")
<\/figure>\n![\"\"](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXckqQre6ZJwjaUGVqApWU6jAPnWUvh5GiyIuEcN4iURH4tIbDgAhz0qptFAAvgxq8q6w4ZopmFxT8EOtxavN75_hX5MbZ3P2HH8FaLDr-hxOtmEKqcyPDP_vfPp0G__dZpkaKEbzg?key=p89efGvPxuY--eWFyJcptZjv\")
![\"\"](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXdJSiFiBi1YXQfGASOneFmbtQAupS4Zpq0t3Rv2T9xWto3xdxx4ymb1TB9tqlLya-DSUW7hFnLzkInS-UeU8OwKEz_XT1mn0Cp_EDslmJbfelcjPZnhVvo4Rnzt6YXKfn5e2eg4RQ?key=p89efGvPxuY--eWFyJcptZjv\")
Implementation<\/h2>\n
\n
import numpy as np\nimport time\nimport torch\nimport torch.nn as nn\nimport gymnasium as gym\nimport ale_py\n\nfrom collections import deque # FIFO queue data structurefrom tqdm import tqdm\u00a0 # progress barsfrom gymnasium.wrappers import FrameStack\nfrom gymnasium.wrappers.frame_stack import LazyFrames\nfrom stable_baselines3.common.atari_wrappers import (\n\u00a0 AtariWrapper,\n\u00a0 FireResetEnv,\n)\n\ngym.register_envs(ale_py) # we need to register ALE with gym\n\n# use cuda if you have it otherwise cpu\ndevice = 'cuda' if torch.cuda.is_available() else 'cpu'\ndevice<\/code><\/pre>\nPongNoFrameskip-v4<\/code>. With this, we can create an environment using the following code:<\/p>\nenv = gym.make('PongNoFrameskip-v4', render_mode='rgb_array')<\/code><\/pre>\nrgb_array<\/code> parameter tells ALE to return pixel values instead of RAM codes (which is the default). The code to interact with the Atari becomes extremely simple with gym<\/code>. The following excerpt encapsulates most of the utilities that we will need from gym<\/code>.<\/p>\n# this code restarts\/starts a environment to the beginning of an episode\nobservation, _ = env.reset()\nfor _ in range(100):\u00a0 # number of timesteps\n\u00a0 # randomly get an action from possible actions\n\u00a0 action = env.action_space.sample()\n\u00a0 # take a step using the given action\n\u00a0 # observation_prime refers to s', terminated and truncated refer to\n\u00a0 # whether an episode has finished or been cut short\n\u00a0 observation_prime, reward, terminated, truncated, _ = env.step(action)\n\u00a0 observation = observation_prime<\/code><\/pre>\nenv = AtariWrapper(env, terminal_on_life_loss=False, frame_skip=4)<\/code><\/pre>\n\n
FrameStack<\/code>). This performs what was discussed above, stacking 4 frames on top of each to keep the states Markovian. The ALE environment returns LazyFrames, which are designed to be more memory efficient, as the same frame might occur multiple times. However, they are not compatible with many of the operations that we perform throughout the training procedure. To convert LazyFrames into usable objects, we apply a custom wrapper which converts an observation to Numpy before returning it to us. The code is shown below.<\/p>\nclass LazyFramesToNumpyWrapper(gym.ObservationWrapper): # subclass obswrapper\n\u00a0 def __init__(self, env):\n\u00a0 \u00a0 \u00a0 super().__init__(env)\n\u00a0 \u00a0 \u00a0 self.env = env # the environment that we want to convert\n\n\u00a0 def observation(self, observation):\n\u00a0 \u00a0 \u00a0 # if its a LazyFrames object then turn it into a numpy array\n\u00a0 \u00a0 \u00a0 if isinstance(observation, LazyFrames):\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 return np.array(observation)\n\u00a0 \u00a0 \u00a0 return observation<\/code><\/pre>\ndef make_env(game, render='rgb_array'):\n\u00a0 env = gym.make(game, render_mode=render)\n\u00a0 env = AtariWrapper(env, terminal_on_life_loss=False, frame_skip=4)\n\u00a0 env = FrameStack(env, num_stack=4)\n\u00a0 env = LazyFramesToNumpyWrapper(env)\n\u00a0 # sometimes a environment needs that the fire button be\n\u00a0 # pressed to start the game, this makes sure that game is started when needed\n\u00a0 if \"FIRE\" in env.unwrapped.get_action_meanings():\n\u00a0 \u00a0 \u00a0 env = FireResetEnv(env)\n\u00a0 return env<\/code><\/pre>\ngym<\/code> remains the same as shown above. An example of the preprocessed states can be seen in Figure 7.<\/p>\nclass ReplayBuffer:\n\n\u00a0 def __init__(self, capacity, device):\n\u00a0 \u00a0 \u00a0 self.capacity = capacity\n\u00a0 \u00a0 \u00a0 self._buffer =\u00a0 np.zeros((capacity,), dtype=object) # stores the tuples\n\u00a0 \u00a0 \u00a0 self._position = 0 # keep track of where we are\n\u00a0 \u00a0 \u00a0 self._size = 0\n\u00a0 \u00a0 \u00a0 self.device = device\n\n\u00a0 def store(self, experience):\n\u00a0 \u00a0 \u00a0 \"\"\"Adds a new experience to the buffer,\n\u00a0 \u00a0 \u00a0 \u00a0 overwriting old entries when full.\"\"\"\n\u00a0 \u00a0 \u00a0 idx = self._position % self.capacity # get the index to replace\n\u00a0 \u00a0 \u00a0 self._buffer[idx] = experience\n\u00a0 \u00a0 \u00a0 self._position += 1\n\u00a0 \u00a0 \u00a0 self._size = min(self._size + 1, self.capacity) # max size is the capacity\n\n\u00a0 def sample(self, batch_size):\n\u00a0 \u00a0 \u00a0 \"\"\" Sample a batch of tuples and load it onto the device\n\u00a0 \u00a0 \u00a0 \"\"\"\n\u00a0 \u00a0 \u00a0 # if the buffer is not full capacity then return everything we have\n\u00a0 \u00a0 \u00a0 buffer = self._buffer[0:min(self._position-1, self.capacity-1)]\n\u00a0 \u00a0 \u00a0 # minibatch of tuples\n\u00a0 \u00a0 \u00a0 batch = np.random.choice(buffer, size=[batch_size], replace=True)\n\n\u00a0 \u00a0 \u00a0 # we need to return the objects as torch tensors, hence we delegate\n\u00a0 \u00a0 \u00a0 # this task to the transform function\n\u00a0 \u00a0 \u00a0 return (\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 self.transform(batch, 0, shape=(batch_size, 4, 84, 84), dtype=torch.float32),\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 self.transform(batch, 1, shape=(batch_size, 1), dtype=torch.int64),\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 self.transform(batch, 2, shape=(batch_size, 1), dtype=torch.float32),\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 self.transform(batch, 3, shape=(batch_size, 4, 84, 84), dtype=torch.float32),\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 self.transform(batch, 4, shape=(batch_size, 1), dtype=torch.bool)\n\u00a0 \u00a0 \u00a0 )\n\u00a0 \u00a0 \u00a0\n\u00a0 def transform(self, batch, index, shape, dtype):\n\u00a0 \u00a0 \u00a0 \"\"\" Transform a passed batch into a torch tensor for a given axis.\n\u00a0 \u00a0 \u00a0 E.g. if index 0 of a tuple means the state then we return all states\n\u00a0 \u00a0 \u00a0 as a torch tensor. We also return a specified shape.\n\u00a0 \u00a0 \u00a0 \"\"\"\n\u00a0 \u00a0 \u00a0 # reshape the tensors as needed\n\u00a0 \u00a0 \u00a0 batched_values = np.array([val[index] for val in batch]).reshape(shape)\n\u00a0 \u00a0 \u00a0 # convert to torch tensors\n\u00a0 \u00a0 \u00a0 batched_values = torch.as_tensor(batched_values, dtype=dtype, device=self.device)\n\u00a0 \u00a0 \u00a0 return batched_values\n\n\u00a0 # below are some magic methods I used for debugging, not very important\n\u00a0 # they just turn the object into an arraylike object\n\u00a0 def __len__(self):\n\u00a0 \u00a0 \u00a0 return self._size\n\n\u00a0 def __getitem__(self, index):\n\u00a0 \u00a0 \u00a0 return self._buffer[index]\n\n\u00a0 def __setitem__(self, index, value: tuple):\n\u00a0 \u00a0 \u00a0 self._buffer[index] = value<\/code><\/pre>\nnumpy<\/code> and then convert it into torch<\/code> tensors, also loading it to the appropriate device.<\/p>\ndeque<\/code> object to hold the tuples can also be used. If you are creating your own buffer, I would emphasize that you spend a little more time to ensure its efficiency.\u00a0<\/p>\ndef load_buffer(preload, capacity, game, *, device):\n\u00a0 # make the environment\n\u00a0 env = make_env(game)\n\u00a0 # create the buffer\n\u00a0 buffer = ReplayBuffer(capacity,device=device)\n\u00a0\n\u00a0 # start the environment\n\u00a0 observation, _ = env.reset()\n\u00a0 # run for as long as the specified preload\n\u00a0 for _ in tqdm(range(preload)):\n\u00a0 \u00a0 \u00a0 # sample random action -> random policy\u00a0\n\u00a0 \u00a0 \u00a0 action = env.action_space.sample()\n\u00a0 \u00a0\n\u00a0 \u00a0 \u00a0 observation_prime, reward, terminated, truncated, _ = env.step(action)\n\u00a0 \u00a0 \u00a0\n\u00a0 \u00a0 \u00a0 # store the results from the action as a python tuple object\n\u00a0 \u00a0 \u00a0 buffer.store((\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 observation.squeeze(), # squeeze will remove the unnecessary grayscale channel\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 action,\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 reward,\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 observation_prime.squeeze(),\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 terminated or truncated))\n\u00a0 \u00a0 \u00a0 # set old observation to be new observation_prime\n\u00a0 \u00a0 \u00a0 observation = observation_prime\n\u00a0 \u00a0 \u00a0\n\u00a0 \u00a0 \u00a0 # if the episode is done, then restart the environment\n\u00a0 \u00a0 \u00a0 done = terminated or truncated\n\u00a0 \u00a0 \u00a0 if done:\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 observation, _ = env.reset()\n\u00a0\n\u00a0 # return the env AND the loaded buffer\n\u00a0 return buffer, env<\/code><\/pre>\nclass DQN(nn.Module):\n\n\u00a0 def __init__(\n\u00a0 \u00a0 \u00a0 self,\n\u00a0 \u00a0 \u00a0 env,\n\u00a0 \u00a0 \u00a0 in_channels = 4, # number of stacked frames\n\u00a0 \u00a0 \u00a0 hidden_filters = [16, 32],\n\u00a0 \u00a0 \u00a0 start_epsilon = 0.99, # starting epsilon for epsilon-decay\n\u00a0 \u00a0 \u00a0 max_decay = 0.1, # end epsilon-decay\n\u00a0 \u00a0 \u00a0 decay_steps = 1000, # how long to reach max_decay\n\u00a0 \u00a0 \u00a0 *args,\n\u00a0 \u00a0 \u00a0 **kwargs\n\u00a0 ) -> None:\n\u00a0 \u00a0 \u00a0 super().__init__(*args, **kwargs)\n\u00a0 \u00a0 \u00a0\n\u00a0 \u00a0 \u00a0 # instantiate instance vars\n\u00a0 \u00a0 \u00a0 self.start_epsilon = start_epsilon\n\u00a0 \u00a0 \u00a0 self.epsilon = start_epsilon\n\u00a0 \u00a0 \u00a0 self.max_decay = max_decay\n\u00a0 \u00a0 \u00a0 self.decay_steps = decay_steps\n\u00a0 \u00a0 \u00a0 self.env = env\n\u00a0 \u00a0 \u00a0 self.num_actions = env.action_space.n\n\u00a0 \u00a0\n\u00a0 \u00a0 \u00a0 # Sequential is an arraylike object that allows us to\n\u00a0 \u00a0 \u00a0 # perform the forward pass in one line\n\u00a0 \u00a0 \u00a0 self.layers = nn.Sequential(\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 nn.Conv2d(in_channels, hidden_filters[0], kernel_size=8, stride=4),\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 nn.ReLU(),\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 nn.Conv2d(hidden_filters[0], hidden_filters[1], kernel_size=4, stride=2),\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 nn.ReLU(),\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 nn.Flatten(start_dim=1),\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 nn.Linear(hidden_filters[1] * 9 * 9, 512), # the final value is calculated by using the equation for CNNs\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 nn.ReLU(),\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 nn.Linear(512, self.num_actions)\n\u00a0 \u00a0 \u00a0 )\n\u00a0 \u00a0 \u00a0 \u00a0\n\u00a0 \u00a0 \u00a0 # initialize weights using he initialization\n\u00a0 \u00a0 \u00a0 # (pytorch already does this for conv layers but not linear layers)\n\u00a0 \u00a0 \u00a0 # this is not necessary and nothing you need to worry about\n\u00a0 \u00a0 \u00a0 self.apply(self._init)\n\n\u00a0 def forward(self, x):\n\u00a0 \u00a0 \u00a0 \"\"\" Forward pass. \"\"\"\n\u00a0 \u00a0 \u00a0 # the \/255.0 performs normalization of pixel values to be in [0.0, 1.0]\n\u00a0 \u00a0 \u00a0 return self.layers(x \/ 255.0)\n\n\u00a0 def epsilon_greedy(self, state, dim=1):\n\u00a0 \u00a0 \u00a0 \"\"\"Epsilon greedy. Randomly select value with prob e,\n\u00a0 \u00a0 \u00a0 \u00a0 else choose greedy action\"\"\"\n\n\u00a0 \u00a0 \u00a0 rng = np.random.random() # get random value between [0, 1]\n\u00a0 \u00a0 \u00a0\n\u00a0 \u00a0 \u00a0 if rng < self.epsilon: # for prob under e\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # random sample and return as torch tensor\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 action = self.env.action_space.sample()\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 action = torch.tensor(action)\n\u00a0 \u00a0 \u00a0 else:\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # use torch no grad to make sure no gradients are accumulated for this\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # forward pass\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 with torch.no_grad():\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 q_values = self(state)\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # choose best action\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 action = torch.argmax(q_values, dim=dim)\n\n\u00a0 \u00a0 \u00a0 return action\n\u00a0\n\u00a0 def epsilon_decay(self, step):\n\u00a0 \u00a0 \u00a0 # linearly decrease epsilon\n\u00a0 \u00a0 \u00a0 self.epsilon = self.max_decay + (self.start_epsilon - self.max_decay) * max(0, (self.decay_steps - step) \/ self.decay_steps)\n\u00a0\n\u00a0 def _init(self, m):\n\u00a0 \u00a0 # initialize layers using he init\n\u00a0 \u00a0 if isinstance(m, (nn.Linear, nn.Conv2d)):\n\u00a0 \u00a0 \u00a0 nn.init.kaiming_normal_(m.weight, nonlinearity='relu')\n\u00a0 \u00a0 \u00a0 if m.bias is not None:\n\u00a0 \u00a0 \u00a0 \u00a0 nn.init.zeros_(m.bias)<\/code><\/pre>\nclass MetricTracker:\n\u00a0 def __init__(self, window_size=100):\n\u00a0 \u00a0 \u00a0 # the size of the history we use to track stats\n\u00a0 \u00a0 \u00a0 self.window_size = window_size\n\u00a0 \u00a0 \u00a0 self.rewards = deque(maxlen=window_size)\n\u00a0 \u00a0 \u00a0 self.current_episode_reward = 0\n\u00a0 \u00a0 \u00a0\n\u00a0 def add_step_reward(self, reward):\n\u00a0 \u00a0 \u00a0 # add received reward to the current reward\n\u00a0 \u00a0 \u00a0 self.current_episode_reward += reward\n\u00a0 \u00a0 \u00a0\n\u00a0 def end_episode(self):\n\u00a0 \u00a0 \u00a0 # add reward for episode to history\n\u00a0 \u00a0 \u00a0 self.rewards.append(self.current_episode_reward)\n\u00a0 \u00a0 \u00a0 # reset metrics\n\u00a0 \u00a0 \u00a0 self.current_episode_reward = 0\n\u00a0\n\u00a0 # property just makes it so that we can return this value without\n\u00a0 # having to call it as a function\n\u00a0 @property\n\u00a0 def avg_reward(self):\n\u00a0 \u00a0 \u00a0 return np.mean(self.rewards) if self.rewards else 0<\/code><\/pre>\nTIMESTEPS = 6000000 # total number of timesteps for training\nLR = 2.5e-4 # learning rate\nBATCH_SIZE = 64 # batch size, change based on your hardware\nC = 10000 # the interval at which we update the target network\nGAMMA = 0.99 # the discount value\nTRAIN_FREQ = 4 # in the paper the SGD updates are made every 4 actions\nDECAY_START = 0 # when to start e-decay\nFINAL_ANNEAL = 1000000 # when to stop e-decay\n\n# load the buffer\nbuffer_pong, env_pong = load_buffer(50000, 150000, game='PongNoFrameskip-v4')\n\n# create the networks, push the weights of the q_network onto the target network\nq_network_pong = DQN(env_pong, decay_steps=FINAL_ANNEAL).to(device)\ntarget_network_pong = DQN(env_pong, decay_steps=FINAL_ANNEAL).to(device)\ntarget_network_pong.load_state_dict(q_network_pong.state_dict())\n\n# create the optimizer\noptimizer_pong = torch.optim.Adam(q_network_pong.parameters(), lr=LR)\n\n# metrics class instantiation\nmetrics = MetricTracker()<\/code><\/pre>\ndef train(\n\u00a0 env,\n\u00a0 name, # name of the agent, used to save the agent\n\u00a0 q_network,\n\u00a0 target_network,\n\u00a0 optimizer,\n\u00a0 timesteps,\n\u00a0 replay, # passed buffer\n\u00a0 metrics, # metrics class\n\u00a0 train_freq, # this parameter works complementary to frame skipping\n\u00a0 batch_size,\n\u00a0 gamma, # discount parameter\n\u00a0 decay_start,\n\u00a0 C,\n\u00a0 save_step=850000, # I recommend setting this one high or else a lot of models will be saved\n):\n\u00a0 loss_func = nn.MSELoss() # create the loss object\n\u00a0 start_time = time.time() # to check speed of the training procedure\n\u00a0 episode_count = 0\n\u00a0 best_avg_reward = -float('inf')\n\u00a0\n\u00a0 # reset the env\n\u00a0 obs, _ = env.reset()\n\u00a0\n\u00a0\n\u00a0 for step in range(1, timesteps+1): # start from 1 just for printing progress\n\n\u00a0 \u00a0 \u00a0 # we need to pass tensors of size (batch_size, ...) to torch\n\u00a0 \u00a0 \u00a0 # but the observation is just one so it doesn't have that dim\n\u00a0 \u00a0 \u00a0 # so we add it artificially (step 2 in procedure)\n\u00a0 \u00a0 \u00a0 batched_obs = np.expand_dims(obs.squeeze(), axis=0)\n\u00a0 \u00a0 \u00a0 # perform e-greedy on the observation and convert the tensor into numpy and send it to the cpu\n\u00a0 \u00a0 \u00a0 action = q_network.epsilon_greedy(torch.as_tensor(batched_obs, dtype=torch.float32, device=device)).cpu().item()\n\u00a0 \u00a0 \u00a0\n\u00a0 \u00a0 \u00a0 # take an action\n\u00a0 \u00a0 \u00a0 obs_prime, reward, terminated, truncated, _ = env.step(action)\n\n\u00a0 \u00a0 \u00a0 # store the tuple (step 3 in the procedure)\n\u00a0 \u00a0 \u00a0 replay.store((obs.squeeze(), action, reward, obs_prime.squeeze(), terminated or truncated))\n\u00a0 \u00a0 \u00a0 metrics.add_step_reward(reward)\n\u00a0 \u00a0 \u00a0 obs = obs_prime\n\u00a0 \u00a0 \u00a0\n\u00a0 \u00a0 \u00a0 # train every 4 steps as per the paper\n\u00a0 \u00a0 \u00a0 if step % train_freq == 0:\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # sample tuples from the replay (step 4 in the procedure)\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 observations, actions, rewards, observation_primes, dones = replay.sample(batch_size)\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # we don't want to accumulate gradients for this operation so use no_grad\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 with torch.no_grad():\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 q_values_minus = target_network(observation_primes)\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # get the max over the target network\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 boostrapped_values = torch.amax(q_values_minus, dim=1, keepdim=True)\n\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # this line basically makes so that for every sample in the minibatch which indicates\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # that the episode is done, we return the reward, else we return the\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # the bootstrapped reward (step 5 in the procedure)\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 y_trues = torch.where(dones, rewards, rewards + gamma * boostrapped_values)\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 y_preds = q_network(observations)\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # compute the loss\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # the gather gets the values of the q_network corresponding to the\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # action taken\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 loss = loss_func(y_preds.gather(1, actions), y_trues)\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # set the grads to 0, and perform the backward pass (step 6 in the procedure)\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 optimizer.zero_grad()\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 loss.backward()\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 optimizer.step()\n\u00a0 \u00a0 \u00a0\n\u00a0 \u00a0 \u00a0 # start the e-decay\n\u00a0 \u00a0 \u00a0 if step > decay_start:\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 q_network.epsilon_decay(step)\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 target_network.epsilon_decay(step)\n\u00a0 \u00a0 \u00a0\n\u00a0 \u00a0 \u00a0 # if the episode is finished then we print some metrics\n\u00a0 \u00a0 \u00a0 if terminated or truncated:\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # compute steps per sec\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 elapsed_time = time.time() - start_time\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 steps_per_sec = step \/ elapsed_time\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 metrics.end_episode()\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 episode_count += 1\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # reset the environment\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 obs, _ = env.reset()\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # save a model if above save_step and if the average reward has improved\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # this is kind of like early-stopping, but we don't stop we just save a model\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 if metrics.avg_reward > best_avg_reward and step > save_step:\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 best_avg_reward = metrics.avg_reward\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 torch.save({\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 'step': step,\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 'model_state_dict': q_network.state_dict(),\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 'optimizer_state_dict': optimizer.state_dict(),\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 'avg_reward': metrics.avg_reward,\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 }, f\"models\/{name}_dqn_best_{step}.pth\")\n\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # print some metrics\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 print(f\"rStep: {step:,}\/{timesteps:,} | \"\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 f\"Episodes: {episode_count} | \"\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 f\"Avg Reward: {metrics.avg_reward:.1f} | \"\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 f\"Epsilon: {q_network.epsilon:.3f} | \"\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 f\"Steps\/sec: {steps_per_sec:.1f}\", end=\"r\")\n\n\u00a0 \u00a0 \u00a0 # update the target network\n\u00a0 \u00a0 \u00a0 if step % C == 0:\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 target_network.load_state_dict(q_network.state_dict())<\/code><\/pre>\ndef test(game, model, num_eps=2):\n\u00a0 # render human opens an instance of the game so you can see it\n\u00a0 env_test = make_env(game, render='human')\n\u00a0\n\u00a0 # load the model\n\u00a0 q_network_trained = DQN(env_test)\n\u00a0 q_network_trained.load_state_dict(torch.load(model, weights_only=False)['model_state_dict'])\n\u00a0 q_network_trained.eval() # set the model to inference mode (no gradients etc)\n\u00a0 q_network_trained.epsilon = 0.05 # a small amount of stochasticity\n\u00a0\n\u00a0\n\u00a0 rewards_list = []\n\u00a0\n\u00a0 # run for set amount of episodes\n\u00a0 for episode in range(num_eps):\n\u00a0 \u00a0 \u00a0 print(f'Episode {episode}', end='r', flush=True)\n\u00a0 \u00a0 \u00a0\n\u00a0 \u00a0 \u00a0 # reset the env\n\u00a0 \u00a0 \u00a0 obs, _ = env_test.reset()\n\u00a0 \u00a0 \u00a0 done = False\n\u00a0 \u00a0 \u00a0 total_reward = 0\n\u00a0 \u00a0 \u00a0\n\u00a0 \u00a0 \u00a0 # until the episode is not done, perform the action from the q-network\n\u00a0 \u00a0 \u00a0 while not done:\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 batched_obs = np.expand_dims(obs.squeeze(), axis=0)\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 action = q_network_trained.epsilon_greedy(torch.as_tensor(batched_obs, dtype=torch.float32)).cpu().item()\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 next_observation, reward, terminated, truncated, _ = env_test.step(action)\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 total_reward += reward\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 obs = next_observation\n\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 done = terminated or truncated\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0\n\u00a0 \u00a0 \u00a0 rewards_list.append(total_reward)\n\u00a0\n\u00a0 # close the environment, since we use render human\n\u00a0 env_test.close()\n\u00a0 print(f'Average episode reward achieved: {np.mean(rewards_list)}')<\/code><\/pre>\n# make sure you use your latest model! I also renamed my model path so\n# take that into account\ntest('PongNoFrameskip-v4', 'models\/pong_dqn_best_6M.pth')<\/code><\/pre>\nSummary<\/strong><\/h2>\n
References<\/h2>\n
![\"\"](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXeUN9j0epxY9K5-EbKX6Azq4jJN2cEe3S3J4Y0wDq3fTgeHTwy2ojvhNWVe12JKOwWV2qx6inLfR2pcIIDChDd0yqy5TUe5EV7OOnCDfoxHo3CGn7HHmZaiTdBC_uQ0DmohIkT8?key=p89efGvPxuY--eWFyJcptZjv\")
![\"\"](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXc5d_NAzv5jcUQBhtshIVoXiKbVtrfNsxU3hYTbjFbqoGufe2X5B0Nr2iIB9ZFLLg8FN-Wil63B3PdusadBQJ-iRS0XNxo6auIVAkNRUzStnXr0e3bBx57N4G9NmktbK3f0okLwuw?key=p89efGvPxuY--eWFyJcptZjv\")
![\"\"](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXcyGd7mIa-5pJ0PQaWBJX81EUQdLRixmV8CKNONHzxbazLro9vkkChC64xt_0BTvgDl-Y0czAiqercGFX1DGxoM7BktqU9vV2J8OL_lRAY2K4iqZeX0UJ_kIIzE-ZC9_A-V3Vyo?key=p89efGvPxuY--eWFyJcptZjv\")
![\"\"](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXd8hStDVDvoVsofptH36QaNCt9rwzDvJVJOCqW_yPyUSHBsKCgf2iZUKjlUrOJ4SK5XKyoZiQq52b2erxnfM-kb-J-_-k3AhUXGCsC9_fla5zqEr1IznZ1oQQEw-gcVBq3DVlUlCA?key=p89efGvPxuY--eWFyJcptZjv\")
![\"\"](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXdSGgVBOisG0N-aa9hHhhYUd4kN43ghLAY628xEiGzzGO18fqggQrYnL5zMJTSFtNB3KP66qAOi8joVKb66cbKE1cxze1oARjxh5GyfMt3YeOAYk2Aue2rNiPb_mjxgQ2opd4LN?key=p89efGvPxuY--eWFyJcptZjv\")
![\"\"](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXfybYt-lQUE_WBhy8NbR0PH9BQ1yzG6RuWnymCtVOVwQpfUpGFfpuk6QVAj2vdzVYPze5FeR3jBbsxoG_jwr7LdR_qaxMlOGMdWv6fNokptaDxOH7tsXnhftD8bwAYoxskW9y3SBw?key=p89efGvPxuY--eWFyJcptZjv\")
![\"\"](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXckeNTYxMf8hFz2OEKv_Xu9G3CYsspIZyYLivg_AOKkb2Iq8mRP0JhQuRoBqgTewD8FR0E309OuiDFYU9ZMH8qa1IZYpnhEAh8v2cEZB7M_SsQc9UjWVfVHshABz4O2wlprqLL4?key=p89efGvPxuY--eWFyJcptZjv\")
![\"\"](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXdxazpKo_qvsXx4Aob9lmDwbzYiR0kr5qESK1nyzb9KmZBm0j_IUGXIgjdsqr_WD8LoPupC5b6u4AenZxK-FGn3fJ2vodUruCrtOTgGKB24Crt-4eCRK2wWQBUkoa7mWHjVX6cavA?key=p89efGvPxuY--eWFyJcptZjv\")