The paper in four sentences
AlphaZero uses one neural network that looks at a position and gives two answers: which moves look promising, and how likely it is to win from here.
To choose a move it runs a lookahead search guided by those answers; the search's final choice is usually better than the network's first instinct.
It learns only from games against itself: the network is trained to predict what its search would play and who actually won, so each round of training makes the next round of search stronger.
Starting from random play and given only the rules, the same algorithm beat the strongest chess program after 4 hours of training, the strongest shogi program in under 2, and AlphaGo Lee in 8.
Idea · Introduction
No human games, no human tricks
For decades the best chess programs were built from human knowledge: carefully hand-tuned rules for judging positions ("a bishop is worth about three pawns"), opening books of known good starts, endgame tables, and a fast search that looks at tens of millions of positions per second. Stockfish, the 2016 world computer chess champion, is the pinnacle of that approach. Elmo, a champion shogi program (shogi is Japanese chess, played on a 9 × 9 board, where captured pieces can be dropped back in), works similarly.
AlphaGo had beaten top humans at Go using neural networks, but it first learned from 30 million human positions and used Go-specific inputs. Its successor AlphaGo Zero dropped the human games. AlphaZero goes further: the same program, with no game-specific knowledge beyond the rules, for three very different games.
TakeawayThe question: how far can pure self-play get, with no human games, no opening books and no hand-written evaluation?
Idea · Methods: self-play
The self-improvement loop
The network gives two outputs for any position: a policy (a probability for each legal move) and a value (expected result, from −1 for a loss through 0 for a draw to +1 for a win). On its own the network is a quick instinct. A search that uses it, imagining moves and replies and asking the network to judge where they lead, makes much better decisions than the instinct alone, because it checks the instinct against what actually happens a few moves later.
That gap is what drives learning. During self-play each move is chosen by search, and the network is then trained so that its instinct (policy) matches what the search preferred, and its value matches who actually won the game. A better network makes the search better, which produces better training targets, and so on.
Below, a miniature version of exactly this loop learns the stick game from the AlphaGo guide: take 1, 2 or 3 sticks, and whoever takes the last stick wins. The "network" is just a table with a policy and a value for each pile size, starting completely uninformed.
TakeawaySearch turns a weak instinct into a better decision; training turns that decision into a better instinct. Repeat, and the program teaches itself.
Check yourself
- Where do the training targets come from, if there are no human games?
- From the program's own search (for the policy) and the final results of its own games (for the value).
- Why does too little search stall learning in the toy?
- If the search hardly looks ahead, its choice is no better than the network's instinct, so training just copies the instinct back to itself and nothing improves.
Idea · Methods: differences from AlphaGo Zero
One algorithm, three games
AlphaZero is essentially AlphaGo Zero made general. Go cannot be drawn, but chess and shogi can, so the value predicts the expected result rather than a probability of winning. Go looks the same if you rotate or mirror the board, and AlphaGo Zero exploited that to multiply its data; chess and shogi do not, so AlphaZero does not use that trick.
AlphaGo Zero also kept a "champion" network and only replaced it when a new version won 55% of a test match. AlphaZero simply keeps one network and updates it continuously. And it uses the same settings for all three games, except for how much random noise is added to encourage exploring different openings, which is scaled to the typical number of legal moves.
The network sees the board as a stack of simple layers (where each piece type is, plus the last few positions), and outputs a probability for every conceivable move: 4,672 move types in chess, 11,259 in shogi and 362 in Go (every point plus "pass").
TakeawayRemove everything Go-specific (symmetry tricks, no-draw assumption, per-game tuning) and the same loop works for chess and shogi too.
Idea · Fig. 1 · Table 1
Results: champions within hours
Training used 5,000 first-generation TPUs (Google's AI chips) to play self-play games and 64 second-generation TPUs to train the network. Measured on a rating scale during training, AlphaZero overtook Stockfish after 4 hours, Elmo after less than 2 hours, and AlphaGo Lee after 8 hours. Full training took 9 hours for chess, 12 for shogi and 34 for Go.
In 100-game matches at one minute per move, it did not lose a single chess game to Stockfish (28 wins, 72 draws), won 90 of 100 against Elmo, and beat the 3-day-trained AlphaGo Zero 60 to 40.
TakeawaySelf-play alone, in hours of (massively parallel) training, produced the strongest player in all three games.
Idea · Table S4 · Fig. 2 · Discussion
Searching less, but better
The most surprising number: in chess AlphaZero looked at about 80 thousand positions per second, while Stockfish looked at about 70 million, nearly a thousand times more. In shogi it was 40 thousand against Elmo's 35 million. AlphaZero's network makes each position far more expensive to evaluate, but its judgement lets it focus the search on the lines that matter. Its strength also grew faster than its rivals' when given more thinking time.
Chess players noticed something else: with no opening book, AlphaZero independently discovered many of the best-known human openings and played them frequently during its training.
TakeawayA thousand times fewer positions, chosen by a learned judgement, beat brute force. The published version answered the main criticisms of the preprint.
Abstract, restated
Chess programs have relied on sophisticated search and hand-crafted evaluation refined by human experts over decades. AlphaGo Zero achieved superhuman Go through tabula rasa reinforcement learning from self-play. AlphaZero generalises this into a single algorithm that achieves superhuman performance in many challenging domains: starting from random play and given no domain knowledge except the rules, within 24 hours it reached a superhuman level in chess, shogi and Go and convincingly defeated a world-champion program in each. arXiv:1712.01815 (December 2017); expanded version in Science 362, 1140–1144 (2018).
Section · Methods · Eq. 1
Network, search and loss
A deep network (p, v) = fθ(s) maps a board position to a vector of move probabilities pa = Pr(a|s) and a scalar expected outcome v ≈ 𝔼[z|s]. Instead of alpha-beta search with domain-specific enhancements, AlphaZero uses general-purpose MCTS: each simulation descends the tree selecting moves with low visit count, high move probability and high value (the PUCT rule from AlphaGo), expands a leaf, evaluates it with fθ (no rollouts), and backs up v. The search returns π, the root visit-count distribution.
Self-play games use 800 simulations per move; moves are sampled in proportion to root visit counts during training and chosen greedily in evaluation. At the end of the game, z ∈ {−1, 0, +1}. Parameters are trained on positions from recent self-play games with
a sum of mean-squared value error, policy cross-entropy against the search distribution, and L2 regularisation. The policy target π is the result of "policy improvement" by search; the value target is "policy evaluation" by self-play outcome, so the loop is a form of approximate policy iteration with MCTS as the improvement operator.
| Change from AlphaGo Zero | Why |
|---|---|
| Value estimates the expected outcome, including draws | Chess and shogi can be drawn; Go (with komi) cannot. |
| No 8-fold rotation/reflection augmentation, no random transforms during MCTS | Chess and shogi boards are not symmetric. |
| One network updated continually; no 55% gating against a best player | Simpler; self-play always uses the latest parameters. |
| Same hyperparameters for all games, except Dirichlet noise α | Generality; α scaled to typical number of legal moves. |
Section · Methods: exploration
Exploration noise at the root
To diversify self-play, Dirichlet noise is mixed into the root prior, P(sroot, a) = (1 − ε)pa + ε ηa, η ∼ Dir(α), with ε = 0.25 in AlphaGo Zero. AlphaZero sets α = 0.3, 0.15 and 0.03 for chess, shogi and Go, "scaled in inverse proportion to the approximate number of legal moves in a typical position". Small α concentrates the noise on a few moves; large α spreads it evenly. Keeping α × (number of moves) roughly constant keeps the character of the noise similar across games.
Section · Tables S1–S2
Representations
| Go | Chess | Shogi | |
|---|---|---|---|
| Input planes | 17 | 119 | 362 |
| Board | 19 × 19 | 8 × 8 | 9 × 9 |
| Policy output | 19 × 19 + 1 = 362 | 8 × 8 × 73 = 4,672 | 9 × 9 × 139 = 11,259 |
Inputs are binary or constant planes: piece positions for each player over an 8-step history (chess, shogi), plus repetition counts, side to move, castling rights, move counters and, in shogi, pieces in hand. In chess, the 73 planes per square encode "queen-like" moves (8 directions × up to 7 squares = 56), 8 knight moves and 9 underpromotions; illegal moves are masked before the softmax. The only domain knowledge is the rules (to generate legal moves and detect the end of the game) and these input/output encodings; there is no opening book, endgame table or hand-crafted evaluation.
Section · Fig. 1 · Tables 1, 2, S4
Training and results
Training. 700,000 steps of minibatch size 4,096 per game, starting from random parameters; 5,000 first-generation TPUs for self-play, 64 second-generation TPUs for training. Learning rate 0.2, dropped to 0.02, 0.002 and 0.0002 over training. Self-play: 44 million games (chess, 9 hours), 24 million (shogi, 12 hours), 21 million (Go, 34 hours).
Elo during training (Fig. 1). AlphaZero exceeded Stockfish after 4 hours (300k steps), Elmo after under 2 hours (110k steps), and AlphaGo Lee after 8 hours (165k steps).
| Game | Opponent | Win | Draw | Loss |
|---|---|---|---|---|
| Chess | Stockfish (2016 TCEC champion) | 28 | 72 | 0 |
| Shogi | Elmo (2017 CSA champion) | 90 | 2 | 8 |
| Go | AlphaGo Zero (3 days) | 60 | – | 40 |
Table 1: 100 games per match, 1 minute per move; Stockfish and Elmo used 64 threads and a 1 GB hash; AlphaZero and AlphaGo Zero used a single machine with 4 TPUs. As White against Stockfish AlphaZero scored 25 wins and 25 draws; as Black, 3 wins and 47 draws.
Search speed (Table S4). AlphaZero searches 80 thousand positions per second in chess (Stockfish 70 million) and 40 thousand in shogi (Elmo 35 million); 16 thousand in Go. Fig. 2 shows AlphaZero's Elo scaling more steeply with thinking time than Stockfish's or Elmo's.
Openings (Table 2). On 12 popular human openings (each played over 100,000 times online), AlphaZero independently discovered and frequently played them during self-play; starting from these openings, it beat Stockfish overall, scoring 40.3% wins, 58.8% draws and 0.8% losses as White and 8.0%, 88.8% and 3.2% as Black, over 600 games per colour (12 openings × 50 games; aggregated from Table 2).
Check yourself
- Why does AlphaZero sample moves in proportion to visit counts during self-play but choose greedily in evaluation?
- Sampling diversifies the training data so the network sees a range of positions; in evaluation you want the strongest move.
- What role does MCTS play in the learning algorithm, as opposed to in play?
- It is the policy-improvement operator: its visit distribution π is a better policy than p, and training p towards π distils the improvement into the network.
Section · Science 2018 and lineage
Limitations, the published version, and lineage
- Match conditions (preprint). Fixed 1 minute per move (which prevents Stockfish's time management), a year-old Stockfish 8, a 1 GB hash said by its developers to be too small for 64 threads, and no opening book. These criticisms were raised by the Stockfish developers and others after the preprint.
- Compute. Thousands of TPUs for self-play; hard to reproduce outside a large lab. Community projects did so over months with distributed volunteers.
- Perfect-information, known-rules games only. The search needs an exact simulator of the rules.
Reference
Glossary
- Tabula rasa
- Learning from a blank slate: random initial parameters, no human data.
- fθ(s) = (p, v)
- Network with a policy head and a value head.
- π
- Normalised root visit counts from MCTS; the policy training target.
- z
- Game outcome from the current player's view: −1, 0 or +1.
- Alpha-beta
- The classical pruned minimax search used by Stockfish and Elmo.
- Dirichlet noise
- Random probability vector mixed into root priors to encourage exploration.
- TPU
- Tensor Processing Unit, Google's neural-network accelerator.
- Elmo, Stockfish
- Champion shogi and chess engines (2017 CSA, 2016 TCEC).