The paper in four sentences
The cost of training a language model is roughly proportional to its size times the amount of text it reads, so for a fixed budget, a bigger model must read less text and a smaller one can read more.
The authors trained more than 400 models of different sizes on different amounts of text and, using three independent methods, found the best split: every time you double the model, also double the text.
By that rule, the giant models of the day (GPT-3, Gopher, Megatron-Turing NLG) were far too big for the amount of text they had been trained on.
To test it, they trained Chinchilla with the same compute as their own 280-billion-parameter Gopher but with a quarter of the parameters and four times the data, and it was better on nearly every benchmark while being cheaper to use.
Idea · §1 · Eq. 1
A fixed budget, two ways to spend it
A language model has a number of adjustable parameters (its size, N) and is trained by reading a number of tokens (word pieces) of text (D). Every token read costs work proportional to the model's size, so the total computation is roughly 6 × N × D arithmetic operations. If your budget of computation is fixed, a bigger model means fewer tokens and vice versa.
Both help: bigger models can represent more, and more text teaches them more. The question the paper asks is what split gives the lowest error for a given budget. That question matters enormously, because these training runs cost millions of dollars and are done once.
TakeawayWith compute fixed, model size and data trade off, and there is a best size in the middle.
Check yourself
- If you double the model size with a fixed budget, what happens to the number of tokens it can read?
- It halves, because compute is roughly 6 × size × tokens.
Idea · §3 · Figs. 2–4 · Table 2
Three ways to find the sweet spot, one answer
The authors trained over 400 models, from 70 million to over 16 billion parameters, on 5 to 500 billion tokens, and analysed them three different ways:
- Approach 1: train each model size for several different lengths, then for every amount of compute, note which run reached the lowest loss.
- Approach 2: fix nine compute budgets, train many model sizes at each, and find the bottom of each U-shaped curve (an "IsoFLOP profile", a curve of equal compute, like the one above).
- Approach 3: fit a single formula for loss as a function of size and data to all the runs, and solve it for the best split.
All three agreed: as compute grows, the best model size and the best amount of data should grow at about the same rate. Doubling one means doubling the other. In round numbers, the best amount of training text is about 20 tokens per parameter. The earlier Kaplan et al. analysis had said model size should grow almost three times as fast as data (on log scales, 0.73 versus 0.27).
Why the difference? The Chinchilla authors blamed mainly the fact that Kaplan et al. used the same training schedule length for every run (later studies found other details, such as how parameters were counted, mattered more). A learning rate that is still high partway through a long schedule makes a model look worse than it would if its schedule had been set for that shorter run, which made long training on more data look less useful than it is.
TakeawayThree methods agree that model size and training tokens should scale equally with compute, roughly 20 tokens per parameter.
Idea · Tables 1, 3
The giants were undertrained
Apply the rule to the biggest models of 2020 and 2021 and they all look the same way: far too many parameters for the text they were trained on. Most were trained on roughly 300 billion tokens, whatever their size. By the paper's estimates, a 175-billion-parameter model should see several trillion tokens. Click a model below to see what the rule says.
TakeawayBy this analysis, every large model of its day could have been several times smaller and trained on several times more text for the same cost.
Idea · §4 · Tables 4–6
Chinchilla versus Gopher
The test: Gopher, DeepMind's own 280-billion-parameter model, trained on 300 billion tokens, against Chinchilla, 70 billion parameters trained on 1.4 trillion tokens of the same kind of text, with roughly the same training compute. Chinchilla also switched to AdamW for training, which the authors found helped.
Chinchilla won almost everywhere. On MMLU, a broad exam-style test covering 57 subjects, it averaged 67.6% against Gopher's 60.0%. It also beat Gopher on 58 of 62 BIG-bench tasks, on reading comprehension, on closed-book question answering, and on language modelling. And because it is four times smaller, it is about four times cheaper to run and to fine-tune afterwards.
| Benchmark | Gopher 280B | Chinchilla 70B |
|---|---|---|
| MMLU, 5-shot average (Table 6) | 60.0% | 67.6% |
| BIG-bench, average of 62 tasks | 54.4% | 65.1% |
| LAMBADA (zero-shot) | 74.5% | 77.4% |
| RACE-h (reading comprehension) | 71.6% | 82.3% |
| Natural Questions, 64-shot (closed book) | 28.2% | 35.5% |
TakeawayA model a quarter of Gopher's size, trained on four times the data for the same compute, was better on almost every benchmark and cheaper to run.
Check yourself
- Why might a company deliberately train a model that is smaller than Chinchilla-optimal?
- Because the model will be run many times after training. A smaller model trained on more tokens can reach the same quality at a higher training cost but a much lower cost per answer.
- What does the paper imply is the next bottleneck after compute?
- Data. Compute-optimal models need trillions of high-quality tokens; the authors note that scaling datasets is only useful if the data stays high quality.
Abstract, restated
The optimal model size and number of training tokens for a transformer language model under a given compute budget are investigated. Current large language models are significantly undertrained, a consequence of scaling model size while keeping data roughly constant. Training over 400 language models from 70M to over 16B parameters on 5 to 500B tokens shows that, for compute-optimal training, model size and training tokens should be scaled equally. This is tested with Chinchilla (70B parameters, 4× more data) using the same compute as Gopher (280B), which it uniformly and significantly outperforms, along with GPT-3 (175B), Jurassic-1 (178B) and Megatron-Turing NLG (530B), reaching 67.5% average accuracy on MMLU, more than 7 points above Gopher. Hoffmann et al. (DeepMind), NeurIPS 2022 (arXiv 2203.15556).
Section · §2–3 · Eq. 1
Problem statement and Approaches 1–2
All models are trained on MassiveText for less than one epoch, with a cosine learning-rate schedule decaying by 10×. FLOPs are approximated by C ≈ 6ND, which Appendix F checks against a full count (Table A4).
Why this differs from Kaplan et al. (§2). Kaplan et al. used a fixed number of training tokens and learning-rate schedule for all models, so intermediate losses of long-schedule runs stood in for shorter runs; the authors find that setting the cosine cycle much longer than the run (more than about 25% overshoot) noticeably degrades the loss (Appendix B). Kaplan et al.'s analysis also leaned mostly on smaller models, and the authors observe slight curvature in the frontier at larger scale, so fits dominated by small models can mislead.
Approach 1 (§3.1, Fig. 2). For each model size from 70M to over 10B, train with four cosine cycle lengths spanning a 16× range of token counts. Smooth each training curve and, at each of 1,500 log-spaced FLOP values, take the run achieving the lowest loss. Fit Nopt ∝ Ca, Dopt ∝ Cb: a = 0.50, b = 0.50.
Approach 2 (§3.2, Fig. 3). Nine fixed budgets from 6 × 1018 to 3 × 1021 FLOPs, varying model size up to 16B with tokens set by the budget. Fit a parabola to final loss against log N for each budget, take its minimum, and regress: a = 0.49, b = 0.51. The figure below re-runs this procedure on synthetic losses.
Section · §3.3 · Eqs. 2–4 · App. D.2
Approach 3: the parametric loss
E is the irreducible loss (entropy of natural text); the second term is the shortfall of a perfectly trained N-parameter model; the third is the shortfall from finite training on D tokens. The fit minimises a Huber loss (δ = 10−3) between predicted and observed log loss over all runs, with L-BFGS from a grid of initialisations (Eq. 3). The reported estimates are E = 1.69, A = 406.4, B = 410.7, α = 0.34, β = 0.28. Minimising Eq. 2 subject to C = 6ND gives the closed form
Table 2 gives a = 0.46, b = 0.54 for this approach. Approach 3 predicts somewhat smaller optimal models than Approaches 1–2 (Fig. 4 projects about 40B for Gopher's budget, versus 67B from Approach 1 in Table 3), so the authors conclude Gopher-optimal lies between about 40B and 70B. Explore Eq. 4 below with either set of constants.
Section · §3.4 · Tables 2–3
Projections
| Approach | a (Nopt ∝ Ca) | b (Dopt ∝ Cb) |
|---|---|---|
| 1. Minimum over training curves | 0.50 | 0.50 |
| 2. IsoFLOP profiles | 0.49 | 0.51 |
| 3. Parametric loss | 0.46 | 0.54 |
| Kaplan et al. (2020) | 0.73 | 0.27 |
| Parameters | FLOPs | Tokens (Approach 1, Table 3) | Tokens / parameter |
|---|---|---|---|
| 400M | 1.92e19 | 8.0B | 20 |
| 1B | 1.21e20 | 20.2B | 20 |
| 10B | 1.23e22 | 205.1B | 21 |
| 67B | 5.76e23 | 1.5T | 22 |
| 175B | 3.85e24 | 3.7T | 21 |
| 280B | 9.90e24 | 5.9T | 21 |
| 520B | 3.43e25 | 11.0T | 21 |
| 1T | 1.27e26 | 21.2T | 21 |
| 10T | 1.30e28 | 216.2T | 22 |
The last column is derived here, not printed in the paper; "20 tokens per parameter" is the community's summary of this table. The paper's own phrasing is that for every doubling of model size the number of training tokens should also be doubled.
Section · §4 · Tables 4–6
Chinchilla: the confirming run
Chinchilla vs Gopher (Table 4): both 80 layers with key/value size 128; dmodel 8,192 vs 16,384; 64 vs 128 heads; max learning rate 1 × 10−4 vs 4 × 10−5; batch 1.5M → 3M tokens vs 3M → 6M. Same MassiveText data with a slightly different subset mix to support 1.4T tokens; AdamW instead of Adam; a SentencePiece tokenizer without NFKC normalisation. Appendix G reports AdamW-trained models ending better than Adam-trained ones; in training loss, AdamW only overtook Adam about 80% of the way through the cosine cycle (footnote 8, Fig. A7).
Results: MMLU 67.6% average 5-shot (Table 6; the abstract rounds to 67.5%) vs 60.0%, better on 51 of 57 tasks; BIG-bench 65.1% vs 54.4%, Gopher better on only 4 of 62 tasks; better on every subset of The Pile, with WikiText-103 perplexity 7.16 vs 7.75 (the authors caution about possible train/test leakage given 4× more data); LAMBADA 77.4% vs 74.5%; RACE-m 86.8% vs 75.1%, RACE-h 82.3% vs 71.6%; Natural Questions 64-shot 35.5% vs 28.2%.
Section · §5 · lineage
Limitations and what came next
- Few large confirmations. Only two runs at the largest scale (Chinchilla and Gopher), no intermediate-scale tests (§5).
- Power-law assumption. The authors observe concavity in log Nopt at high compute (Appendix E), which would make optimal models at very large budgets even smaller than projected.
- Single epoch. All runs see each token at most once; the multi-epoch regime is unexplored. Muennighoff et al. (2023, "Scaling data-constrained language models") later found up to about four epochs of repeated data are nearly as good as fresh data.
- Approach 3 fit. Besiroglu et al. (2024) reconstructed the data from the paper's figures and found the reported Approach 3 estimates fit it poorly, with implausibly narrow confidence intervals; their refit agrees with Approaches 1–2 (about 20 tokens per parameter).
- Training-only objective. Compute-optimal ignores inference. Sardana & Frankle (2023) extend the analysis to total lifetime cost, recommending smaller, longer-trained models when demand is high, which is what LLaMA and successors do.
- Kaplan reconciliation. Pearce & Song (2024) and Porian et al. (2024) attribute Kaplan et al.'s 0.73 to counting non-embedding parameters at small scale, warmup duration, uncounted last-layer compute, and untuned optimiser settings at small scale. Porian et al. found the learning-rate schedule was not the main factor, contrary to Hoffmann et al.'s own hypothesis.
Check yourself
- Under Eq. 4, what determines whether parameters or data should grow faster with compute?
- The ratio of the exponents: a = β/(α + β). If data-limited loss falls faster with D (larger β) than model-limited loss falls with N (α), more of the budget goes to parameters, and vice versa. With α ≈ β, they grow equally.
- Why did Kaplan et al.'s fixed learning-rate schedule bias them towards larger models?
- Smaller models need more tokens for the same compute; reading their losses at intermediate points of a long schedule, before the learning rate had decayed, understated what they would reach with a schedule matched to that length, so data looked less valuable relative to parameters.
Reference
Glossary
- Compute-optimal
- The (N, D) that minimises loss for a given training FLOP budget.
- IsoFLOP profile
- Final loss as a function of model size at fixed training compute.
- E
- Irreducible loss in Eq. 2, the estimated entropy of the text.
- MassiveText
- DeepMind's multi-source English text corpus used for Gopher and Chinchilla.
- MMLU
- Massive Multitask Language Understanding: 57 exam-style subjects.
- Tokens per parameter
- D/N; about 20 at the compute-optimal point by this paper.
- Huber loss
- A robust loss, quadratic for small residuals and linear for large ones.