The paper in four sentences
The deep Q-network (DQN) learns to play Atari games from the screen pixels and the score, with no game-specific knowledge, using the same network, learning method and settings for all 49 games tested (each game trained separately).
It learns by guessing how much future score each joystick move will bring, and correcting each guess using its own guess about the next moment.
Two tricks keep that self-correcting process from spiralling: a big memory of past moments that it learns from in random order, and a frozen copy of itself that provides the "next moment" guesses and is only updated now and then.
It beats the best earlier learning methods on 43 of the 49 games and reaches more than 75% of a professional human tester's level on 29 of them, though games that need long-term exploration, like Montezuma's Revenge, remain out of reach.
Idea · Abstract · Fig. 3
One agent, 49 games
This is the grown-up version of the 2013 workshop paper. The idea is the same: an agent that watches an Atari game screen, presses joystick buttons and is told the score, and that has to work out everything else for itself. What changed is the scale and the care. Instead of seven games, there are 49. Instead of training for 10 million frames, each game gets 50 million frames, about 38 days of play at the console's normal speed.
"One agent" needs reading carefully. A fresh copy is trained from scratch on each game; there is not one network that plays all 49. What is shared is the recipe: the network design, the learning method and every setting are identical across games. That is the point being made. Nothing was tuned per game, so the success has to come from the general method, not from human insight into each game.
The games are wildly different: shooters like Space Invaders, sports like Boxing and Tennis, racing like Enduro, mazes like Ms. Pac-Man. Some reward you constantly, some rarely.
TakeawaySame design, same settings, separate training on each of 49 games: the test is whether a general learning recipe can reach human-level play without help.
Idea · Methods: preprocessing
What the agent actually sees
The Atari console could only draw a few moving objects (sprites) at once, so many games draw some objects only on even frames and others only on odd frames, relying on the eye to blur the flicker. A program that looks at single frames would see enemies blink in and out of existence. The paper fixes this by taking, for every pixel, the brighter of the current and previous frame, so a flickering object is always visible.
The picture is then turned grey and shrunk to 84 × 84 pixels, and the agent is given the last four of these images at once. One still frame cannot tell you which way a ball is moving; four can. The agent picks a move only every fourth frame and holds it in between.
Play with a made-up game screen below. Switch off the max-of-two-frames step and watch the enemy blink; switch to a single frame and try to say which way the ball is travelling.
TakeawaySmall preprocessing choices (remove flicker, stack four frames) give a still-image network the information it needs: where things are and which way they are moving.
Check yourself
- Why give the network four frames rather than one?
- A single frame shows position but not motion. Four frames let it infer speed and direction, such as which way the ball is going.
- What does taking the maximum over two frames fix?
- Sprites that the game draws only on alternate frames. The maximum keeps them visible in every processed frame.
Idea · Main text · Methods
Stop chasing your own tail: the target network
Like the 2013 version, DQN learns a Q-value for each move: its guess of the future score that move will bring. Each guess is corrected towards "reward just received + a discounted version of my own best guess for the next moment". The awkward part is that the answer key is written by the student. Every time the network updates, its guess for the next moment shifts too, so the target it is aiming at keeps moving. With a large neural network, where changing one guess nudges many others, this can make learning wobble or even blow up.
The fix is a target network: a frozen copy of the Q-network that is used only to compute those "next moment" guesses. The main network learns towards targets that stay still for 10,000 updates. Then the copy is refreshed, and the process repeats. Each stretch becomes an ordinary, stable learning problem with a fixed answer key.
The paper keeps the other stabiliser from 2013, experience replay: the last million moments are stored, and learning uses random handfuls of them. Below, watch what the target looks like during training with and without a frozen copy.
TakeawayA periodically refreshed frozen copy gives the network a fixed answer key for thousands of updates at a time, which keeps self-correcting learning stable.
Check yourself
- Why is the target "moving" in plain Q-learning with a network?
- The target is computed from the same network that is being trained, so every update changes the target as well as the estimate.
- What exactly is frozen in the target network, and for how long?
- A full copy of the network's weights, used only to compute targets. It is replaced by a fresh copy of the main network every 10,000 updates.
Idea · Fig. 3 · Extended Data Table 2
What "human level" means here
Scores in different games are in different units, so the paper puts them on one scale. A random button-masher scores 0%, and a professional games tester who practised each game for about two hours scores 100%. An agent at 75% has closed three quarters of the gap between random and that human.
On this scale DQN beat the best earlier learning methods on 43 of the 49 games, and reached 75% or more on 29. On some games it went far beyond the tester. In Breakout it discovered the classic trick of digging a tunnel up the side of the wall so the ball bounces around behind the bricks.
It failed badly on games where reward is rare and you must explore deliberately. In Montezuma's Revenge you need many precise moves before the first point, so random button pressing almost never finds a reward to learn from.
Takeaway"Human level" means a share of the gap between random play and a practised tester. DQN cleared 75% on 29 of 49 games, and was weakest where rewards are rare.
Idea · Fig. 4 · Discussion
What the network learned, and what came next
The authors looked inside the trained Space Invaders network. They took its last hidden layer for many game moments and laid them out on a 2-D map so that similar internal patterns sit close together. Screens that look alike landed near each other, as you would expect. More interestingly, some screens that look quite different but promise a similar future score also landed together. The network had organised what it sees around what matters for the score.
The paper became a landmark: it showed deep learning and reinforcement learning working together at scale, and the same team went on to AlphaGo.
TakeawayThe network learned an internal picture of the game organised by future reward. Its weaknesses (overestimation, poor exploration) set the agenda for the next five years.
Abstract, restated
Reinforcement learning agents have needed hand-crafted features or low-dimensional, fully observed state spaces. The deep Q-network combines Q-learning with a deep convolutional network to learn policies directly from high-dimensional pixels and the game score. Tested on 49 Atari 2600 games with one algorithm, architecture and set of hyperparameters, it surpasses all previous algorithms and reaches a level comparable to a professional human games tester. Nature 518, 529–533 (26 February 2015).
Section · Main text · Methods
The loss with a target network
The optimal action-value function obeys the Bellman equation Q*(s, a) = 𝔼s′[r + γ maxa′ Q*(s′, a′) | s, a]. DQN parameterises Q(s, a; θi) and, at iteration i, minimises over uniformly sampled minibatches from the replay memory D:
θ− are the target-network parameters, set equal to θ every C updates and held fixed in between. The paper's argument: an update that increases Q(st, at) often also increases Q(st+1, a) for all a, and so raises the target, which can lead to oscillation or divergence. Delaying the target's response adds a lag between an update and its effect on the targets, making divergence less likely.
A second change: the error term r + γ max Q(θ−) − Q(θ) is clipped to [−1, 1]. Because the gradient of the absolute value is ±1, this is equivalent to using a squared loss for errors inside (−1, 1) and an absolute-value loss outside, which is what is now called the Huber loss. Rewards are also clipped to {−1, 0, +1} as in 2013.
Section · Methods · Algorithm 1 · Extended Data Table 1
Architecture, preprocessing and settings
Preprocessing φ. Per-pixel maximum over the current and previous frame (removes flicker from sprites drawn on alternate frames), extract the luminance (Y) channel, rescale to 84 × 84, stack the m = 4 most recent frames (the paper notes 3 or 5 also work).
Network. Input 84 × 84 × 4 → conv 32 filters 8 × 8 stride 4 → conv 64 filters 4 × 4 stride 2 → conv 64 filters 3 × 3 stride 1 → fully connected 512 → linear output, one per valid action (4 to 18). All hidden layers are rectifiers. Compared with 2013 (16 and 32 filters, two conv layers, 256 hidden units) this is a substantially larger network.
| Setting | Value |
|---|---|
| Minibatch size | 32 |
| Replay memory | 1,000,000 most recent frames |
| Agent history length m | 4 |
| Target network update period C | 10,000 |
| Discount γ | 0.99 |
| Action repeat (frame skip) | 4 |
| Update frequency | 1 gradient update per 4 actions |
| Optimiser | RMSProp, learning rate 0.00025, gradient momentum 0.95 |
| Exploration ε | 1.0 → 0.1 linearly over first 1M frames |
| Replay start size | 50,000 (uniform random policy first) |
| Training length | 50 million frames per game |
Values from the Methods and Extended Data Table 1, cross-checked against the "Nature settings" reproduced in the Dopamine framework (Castro et al., 2018). The same values were used for every game; they were selected by an informal search on a handful of games, not per game.
Section · Fig. 3 · Fig. 4 · Methods: evaluation
Evaluation and results
Protocol. Each trained agent plays 30 episodes of up to 5 minutes with ε = 0.05, each episode beginning with a random number of no-op actions so that it starts from varied states. The professional tester played about 20 episodes of up to 5 minutes per game after about 2 hours of practice, with sound off and no pausing or reloading. The random baseline presses a random button at 10 Hz (every sixth frame).
Headline numbers. DQN outperforms the best existing reinforcement learning methods on 43 of 49 games and achieves more than 75% of the human score on 29 games (Fig. 3). It does well on games such as Boxing, Breakout, Enduro and River Raid, and learns long-horizon strategies in some (tunnelling in Breakout), but is close to random on games requiring extended exploration and planning such as Montezuma's Revenge.
Representations (Fig. 4). t-SNE on last-hidden-layer activations from 2 hours of Space Invaders play, coloured by predicted value, places perceptually similar states together and also groups some perceptually dissimilar states with similar expected reward.
Ablation (Extended Data Table 3). On a subset of games, trained for 10 million frames at three learning rates each, the authors compare DQN with and without replay and with and without the separate target network. The paper uses this comparison to argue that both components matter to performance; the individual numbers are not reproduced here.
Check yourself
- Why randomise the number of no-op actions at the start of evaluation episodes?
- Atari emulation is deterministic, so an agent could memorise one action sequence. Random starts make the evaluation test a policy rather than a memorised trajectory.
- Why does the target network help when an update to Q(s, a) generalises to Q(s′, ·)?
- Without it, raising Q(s, a) also raises the target r + γ max Q(s′, ·), creating a positive feedback loop. Freezing θ− breaks the loop for C updates.
Section · Discussion and lineage
Limitations and later refinements
- Sample cost. 50 million frames (about 38 days of play) per game, far beyond what a human needs.
- Exploration. ε-greedy dithering cannot find sparse rewards (Montezuma's Revenge).
- Overestimation. The max in the target uses the same noisy estimates to choose and evaluate actions, biasing values upward.
- Discrete actions only. The output layer enumerates actions, so continuous control needs other methods.
- Theory. The "deadly triad" (function approximation, bootstrapping, off-policy data) still has no convergence guarantee; the target network and replay are effective heuristics.
- Evaluation fragility. Later work showed results depend on start conditions (no-op versus human starts) and on sticky actions (Machado et al., 2018).
Reference
Glossary
- θ, θ−
- Online and target network parameters.
- C
- Number of updates between target-network refreshes.
- Replay memory D
- Ring buffer of recent transitions (s, a, r, s′) sampled uniformly.
- Error clipping
- Clipping the TD error to [−1, 1], equivalent to a Huber loss.
- Normalised score
- Percentage of the gap between random play and the human tester.
- No-op start
- Beginning an evaluation episode with a random number of do-nothing actions.
- t-SNE
- A method for laying out high-dimensional vectors in 2-D while keeping neighbours close.
- Deadly triad
- Sutton and Barto's name for the unstable combination of function approximation, bootstrapping and off-policy learning.