Study guide · Robotics & reinforcement learning · 2015

Human-level control through deep reinforcement learning

Volodymyr Mnih et al. · Nature

Can a single learning agent, with one fixed design, reach human-level play across dozens of different video games from pixels and the score alone? This Nature paper scales up the 2013 Atari agent with one key addition, a slowly updated "target network", and reaches more than 75% of a professional games tester's score (measured above random play) on 29 of 49 games.

The paper in four sentences

The deep Q-network (DQN) learns to play Atari games from the screen pixels and the score, with no game-specific knowledge, using the same network, learning method and settings for all 49 games tested (each game trained separately).

It learns by guessing how much future score each joystick move will bring, and correcting each guess using its own guess about the next moment.

Two tricks keep that self-correcting process from spiralling: a big memory of past moments that it learns from in random order, and a frozen copy of itself that provides the "next moment" guesses and is only updated now and then.

It beats the best earlier learning methods on 43 of the 49 games and reaches more than 75% of a professional human tester's level on 29 of them, though games that need long-term exploration, like Montezuma's Revenge, remain out of reach.

Idea · Abstract · Fig. 3

One agent, 49 games

This is the grown-up version of the 2013 workshop paper. The idea is the same: an agent that watches an Atari game screen, presses joystick buttons and is told the score, and that has to work out everything else for itself. What changed is the scale and the care. Instead of seven games, there are 49. Instead of training for 10 million frames, each game gets 50 million frames, about 38 days of play at the console's normal speed.

"One agent" needs reading carefully. A fresh copy is trained from scratch on each game; there is not one network that plays all 49. What is shared is the recipe: the network design, the learning method and every setting are identical across games. That is the point being made. Nothing was tuned per game, so the success has to come from the general method, not from human insight into each game.

The games are wildly different: shooters like Space Invaders, sports like Boxing and Tennis, racing like Enduro, mazes like Ms. Pac-Man. Some reward you constantly, some rarely.

Everyday pictureThink of a cookery school that teaches one method and then sends its graduates to 49 different kitchens. Each graduate learns their own kitchen on the job, but they all arrived with the same training and the same habits. The paper is testing the method, not any one cook.

TakeawaySame design, same settings, separate training on each of 49 games: the test is whether a general learning recipe can reach human-level play without help.

Idea · Methods: preprocessing

What the agent actually sees

The Atari console could only draw a few moving objects (sprites) at once, so many games draw some objects only on even frames and others only on odd frames, relying on the eye to blur the flicker. A program that looks at single frames would see enemies blink in and out of existence. The paper fixes this by taking, for every pixel, the brighter of the current and previous frame, so a flickering object is always visible.

The picture is then turned grey and shrunk to 84 × 84 pixels, and the agent is given the last four of these images at once. One still frame cannot tell you which way a ball is moving; four can. The agent picks a move only every fourth frame and holds it in between.

Play with a made-up game screen below. Switch off the max-of-two-frames step and watch the enemy blink; switch to a single frame and try to say which way the ball is travelling.

A toy screen, not a real Atari game. Left: the raw frame; the blue enemy is drawn only on even frames, as many Atari games did. Right: what the network receives. With "max of two frames" each pixel takes the brighter of this frame and the last one, so the enemy never vanishes. With 4 frames stacked, older frames appear fainter, so the ball's direction of travel is visible.

TakeawaySmall preprocessing choices (remove flicker, stack four frames) give a still-image network the information it needs: where things are and which way they are moving.

Check yourself
Why give the network four frames rather than one?
A single frame shows position but not motion. Four frames let it infer speed and direction, such as which way the ball is going.
What does taking the maximum over two frames fix?
Sprites that the game draws only on alternate frames. The maximum keeps them visible in every processed frame.

Idea · Main text · Methods

Stop chasing your own tail: the target network

Like the 2013 version, DQN learns a Q-value for each move: its guess of the future score that move will bring. Each guess is corrected towards "reward just received + a discounted version of my own best guess for the next moment". The awkward part is that the answer key is written by the student. Every time the network updates, its guess for the next moment shifts too, so the target it is aiming at keeps moving. With a large neural network, where changing one guess nudges many others, this can make learning wobble or even blow up.

The fix is a target network: a frozen copy of the Q-network that is used only to compute those "next moment" guesses. The main network learns towards targets that stay still for 10,000 updates. Then the copy is refreshed, and the process repeats. Each stretch becomes an ordinary, stable learning problem with a fixed answer key.

The paper keeps the other stabiliser from 2013, experience replay: the last million moments are stored, and learning uses random handfuls of them. Below, watch what the target looks like during training with and without a frozen copy.

An illustrative toy: one state that leads back to itself, noisy reward averaging 1, discount 0.9, so the true value is 10. Amber: the target the network is chasing. Blue: its current estimate. With refresh every 1 update the target moves every single step (the 2013 setting); with a frozen copy it stands still in flat steps and then jumps. Very long freezes make progress slow, one refresh at a time, so C is a trade-off. In this tiny linear toy every setting is stable; the paper's point is that in a large nonlinear network the fixed answer key prevents the feedback loops that cause oscillation and divergence.
Everyday pictureTrying to hit a dartboard that someone moves every time you throw is maddening. Agree that the board stays put for the next hundred throws, then gets moved to where you now think it should be, and you can actually improve.

TakeawayA periodically refreshed frozen copy gives the network a fixed answer key for thousands of updates at a time, which keeps self-correcting learning stable.

Check yourself
Why is the target "moving" in plain Q-learning with a network?
The target is computed from the same network that is being trained, so every update changes the target as well as the estimate.
What exactly is frozen in the target network, and for how long?
A full copy of the network's weights, used only to compute targets. It is replaced by a fresh copy of the main network every 10,000 updates.

Idea · Fig. 3 · Extended Data Table 2

What "human level" means here

Scores in different games are in different units, so the paper puts them on one scale. A random button-masher scores 0%, and a professional games tester who practised each game for about two hours scores 100%. An agent at 75% has closed three quarters of the gap between random and that human.

On this scale DQN beat the best earlier learning methods on 43 of the 49 games, and reached 75% or more on 29. On some games it went far beyond the tester. In Breakout it discovered the classic trick of digging a tunnel up the side of the wall so the ball bounces around behind the bricks.

It failed badly on games where reward is rare and you must explore deliberately. In Montezuma's Revenge you need many precise moves before the first point, so random button pressing almost never finds a reward to learn from.

The paper's normalised score: 100 × (agent − random) / (human − random). The slider values are made up; move them to see how the same raw score can be a strong or weak result depending on where random play and the human sit.

Takeaway"Human level" means a share of the gap between random play and a practised tester. DQN cleared 75% on 29 of 49 games, and was weakest where rewards are rare.

Idea · Fig. 4 · Discussion

What the network learned, and what came next

The authors looked inside the trained Space Invaders network. They took its last hidden layer for many game moments and laid them out on a 2-D map so that similar internal patterns sit close together. Screens that look alike landed near each other, as you would expect. More interestingly, some screens that look quite different but promise a similar future score also landed together. The network had organised what it sees around what matters for the score.

The paper became a landmark: it showed deep learning and reinforcement learning working together at scale, and the same team went on to AlphaGo.

Later refinementDQN tends to overestimate Q-values, because it uses the same maximum both to pick and to score the next move. Double DQN (2015) fixed this by letting the online network choose and the target network score. Prioritised replay (2015) replays surprising memories more often, and Rainbow (2017) combined six such improvements. Montezuma's Revenge was finally cracked by methods built for exploration, such as Random Network Distillation (2018) and Go-Explore (2019), and Agent57 (2020) beat the human baseline on all 57 Atari games.

TakeawayThe network learned an internal picture of the game organised by future reward. Its weaknesses (overestimation, poor exploration) set the agenda for the next five years.

Follow the idea

Built on: DQN (2013)

Led to: AlphaGo (2016)