The paper in four sentences
The authors trained hundreds of transformer language models, from tiny (about a thousand adjustable numbers) to large (1.5 billion), on different amounts of text and with different amounts of computing power, and measured how well each predicts the next word.
The prediction error fell along remarkably straight lines when plotted on logarithmic axes, a pattern called a power law, separately for model size, data size and compute, as long as none of the three was holding the others back.
Details such as the model's exact shape (deep and narrow versus shallow and wide) mattered far less than its total size.
Because bigger models learn more from each word they read, the paper concluded that with a fixed budget you should build a much bigger model and stop training it early, growing model size much faster than data; Chinchilla later showed data should grow about as fast as model size.
Idea · §1
Why you would want to predict a training run
A language model is trained to guess the next word (more precisely, the next token, a word or piece of a word) given the text so far. Its score is the loss: roughly, how surprised it is by the actual next token, averaged over a lot of text. Lower is better.
Training a big model costs a great deal of money and months of computer time. Before you start, you have to decide how big to make it, how much text to feed it and how long to train. Get it wrong and you waste the budget. The authors asked whether the loss can be predicted in advance from three numbers: N, the model's size (its count of adjustable parameters), D, the amount of training text in tokens, and C, the total computation used.
TakeawayThe goal is to predict a model's quality from its size, its data and its compute before paying for the run.
Idea · §1.2 · Fig. 1 · Eqs. 1.1–1.3
Straight lines on log paper
The answer is a power law. Make the model 10 times bigger and the loss is multiplied by a fixed factor, always the same factor, whether you go from 1 million to 10 million parameters or from 100 million to a billion. For model size, that factor is about 0.84: each tenfold increase cuts the loss by about 16%.
Plotted on ordinary axes, a power law is a curve that drops steeply and then flattens, and it is hard to see whether it will keep going. Plotted with both axes on a logarithmic scale (where each step along the axis means "10 times more"), it becomes a straight line, and you can extend it with a ruler. The paper found the same kind of straight line for data and for compute too.
TakeawayLoss falls as a power law in model size, data and compute: a straight line on log–log axes, which is what makes it predictable.
Check yourself
- If the exponent is 0.076, roughly how much bigger must a model be to halve its loss?
- You need 10 to the power (log 2 / 0.076) ≈ 104, so about 9,000 times more parameters, with enough data and compute. Gains are steady but expensive.
Idea · §1.1 · §3
Predicting the big run from small ones
Here is why the straight line matters in practice. Train a handful of small, cheap models, plot them, draw the line, and read off where a model 100 times larger should land. Below, the circles are small "experiments", generated from the paper's law with a little random noise. Choose how many of them you are allowed to use, and see how well the fitted line predicts a model far beyond them.
The paper also found that the details of a model's shape hardly matter: for a fixed number of parameters, changing the ratio of depth to width by a factor of 40 changed the loss only slightly. And a model's loss on other kinds of text (books, Wikipedia) tracked its loss on its training text, just with a constant offset.
TakeawayBecause the trend is a straight line on log axes, a few cheap runs let you forecast an expensive one, and the forecast mostly ignores architectural details.
Idea · §5–6 · Figs. 2–3 · Eq. 1.6
Bigger models learn faster, so stop early
A striking finding is that larger models are more sample-efficient: they reach a given loss after seeing fewer tokens, and in fewer training steps, than small models. A big model gets more out of every sentence it reads.
That changes how to spend a fixed computing budget. Each step of a big model costs more, so you can afford fewer steps. But because big models improve so fast per step, the paper found the best use of a budget is a model much larger than you might expect, trained for a relatively short time and stopped well before it has finished improving. Below, each curve is a model size; the horizontal axis is computing spent. Slide the budget and see which size is lowest at that point.
TakeawayLarge models learn more per token, so for a fixed budget Kaplan et al. recommend a bigger model trained for less time than you would naively choose.
Check yourself
- Why is it not wasteful to stop training a model before it has converged?
- The last stretch of training improves the loss very slowly. The same compute spent on a larger model, even one trained for fewer steps, buys more improvement.
Idea · §6.1 · Fig. 3
The advice that was later revised
Putting it together, the paper recommended that when your compute grows tenfold, most of it should go into model size (about 5.4 times bigger) and relatively little into more data (about 1.9 times more tokens). This guided the giant models of 2020–2021: GPT-3 had 175 billion parameters but was trained on only about 300 billion tokens.
The paper also warned that its laws must break down somewhere: extrapolated far enough, they contradict each other, at around a trillion parameters and loss of about 1.7. The authors flagged their estimate of that point as highly uncertain.
TakeawayThe paper's method (fit power laws on small runs, extrapolate) became standard; its specific split of compute between size and data did not survive.
Abstract, restated
Empirical scaling laws for language-model performance on cross-entropy loss: the loss scales as a power law with model size, dataset size and training compute, with trends spanning more than seven orders of magnitude. Other architectural details such as width or depth have minimal effect within a wide range. Simple equations govern the dependence of overfitting on model and dataset size, and the dependence of training speed on model size. These relationships determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, so optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence. Kaplan, McCandlish, Henighan, Brown, Chess, Child, Gray, Radford, Wu & Amodei (OpenAI), arXiv 2001.08361, 2020.
Section · §2 · Eq. 2.1
Setup and notation
Decoder-only transformers trained on WebText2 (an extended WebText; about 2.29 × 1010 tokens, 6.6 × 108 held out for test), byte-pair encoding with vocabulary 50,257, context 1024. Loss L is the cross-entropy in nats per token, averaged over the context. Training default: Adam for 2.5 × 105 steps with batch 512 sequences × 1024 tokens; models above 1B parameters used Adafactor; 3000-step linear warmup then cosine decay to zero.
N counts non-embedding parameters, N ≈ 2dmodelnlayer(2dattn + dff) = 12nlayerdmodel2 for the standard dattn = dff/4 = dmodel. Excluding embeddings is what makes the trends clean across depths; including them makes small shallow models look anomalous. Compute is estimated as C ≈ 6NBS (forward plus backward, ignoring the context-dependent attention term), quoted in PF-days (1 PF-day = 8.64 × 1019 FLOPs). Model sizes ranged from 768 to 1.5 × 109 non-embedding parameters, datasets from 2.2 × 107 to 2.3 × 1010 tokens.
Section · §1.2 · Eqs. 1.1–1.6
The laws and their constants
Each single-variable law holds when the other two factors are not bottlenecks:
Cmin and Smin are the compute and steps the run would need at the critical batch size (§5.1), which the paper estimates from the gradient noise scale; training at B ≫ Bcrit wastes compute, at B ≪ Bcrit wastes steps. Refitting Eq. 1.5 jointly gives αN = 0.076, αD = 0.103, Nc = 6.4 × 1013, Dc = 1.8 × 1013 (Table 2), and Eq. 1.6 uses αN = 0.077, Nc = 6.5 × 1013 (Table 3). The form of Eq. 1.5 was chosen so that the loss depends on overfitting only through NαN/αD/D (Fig. 9) and so that L → L(N) as D → ∞. Note it has no irreducible-loss term: the laws describe the reducible part above whatever entropy natural text has, a choice later work changed.
Section · §4, §3.2
Overfitting, shape, and transfer
- Overfitting (§4.2). Under Eq. 1.5 the overfitting penalty depends on N and D only through NαN/αD/D ≈ N0.74/D. Keeping it below about 0.02 nats, roughly seed-to-seed variance, requires
- so an 8× larger model needs only about 5× more data to avoid overfitting. By this rule, WebText2's 22 billion tokens would suffice for models of roughly 109 parameters.
- Shape (§3.1, Fig. 5). At fixed N, the loss varies only a few per cent over wide ranges of aspect ratio dmodel/nlayer (a factor of 40 changes it only slightly), feed-forward ratio and attention head dimension.
- Transfer (§3.2.2). Loss on other distributions (Books, Common Crawl, Wikipedia, Internet Books) improves in parallel with WebText2 validation loss, with a roughly constant offset: generalisation depends almost entirely on in-distribution validation loss, not on training duration or proximity to convergence.
- LSTMs (§3.2.1). Transformers beat LSTMs on per-token loss, especially later in the context, while LSTMs plateau after about 100 tokens.
Section · §6 · Eqs. 1.7–1.8, 6.1–6.2 · Fig. 14
Compute-efficient allocation
Choosing N, B and S to minimise L at fixed C, using Eq. 1.6 and B = Bcrit, gives
The predicted αCmin ≈ 0.054 is close to the direct fit 0.050. Empirically (Fig. 14), N(Cmin) ≈ (1.3 × 109) Cmin0.73 and Smin ≈ (5.4 × 103) Cmin0.03 with Cmin in PF-days, and B ∝ C0.24; hence data processed grows as D = BS ∝ C0.27. Serial steps barely increase; almost all extra compute goes into bigger models and bigger batches. Fig. 12 adds that models between 0.6× and 2.2× the optimal size need only about 20% more compute, so the optimum is broad.
The calculator below takes these fits at face value and compares them with the Chinchilla rule of thumb (about 20 tokens per parameter, C ≈ 6ND).
Section · §6.3 · Fig. 15 · lineage
The contradiction, and what replaced the allocation
Breakdown (§6.3). Compute-efficient training uses data D ∝ C0.27, but Eq. 1.2 says loss in the data-limited regime falls only as D−0.095, about C−0.03, slower than L(Cmin) ∝ C−0.050. The two trends must cross, which the authors estimate at C* ~ 104 PF-days, N* ~ 1012 parameters, D* ~ 1012 tokens and L* ~ 1.7 nats/token, with an uncertainty of an order of magnitude either way. They conjecture L* may approximate the entropy of natural language, so the laws must change before then.
Learning-rate schedule (App. D.6). The paper reports that the schedule is mostly irrelevant provided the total summed learning rate is large enough and it decays to near zero, and it used one fixed schedule length for all runs. Hoffmann et al. (2022) argued this mattered, since a model evaluated partway through a cosine schedule tuned for a longer run looks worse than a model whose schedule ends at that point, though Porian et al. (2024) later found the schedule was not the main source of the discrepancy.
Check yourself
- Why does excluding embedding parameters from N matter for the fitted exponent?
- For small models the embedding matrix is a large fraction of all parameters. Counting it or not changes the apparent size of the small models much more than the large ones, which tilts the fitted lines and therefore the inferred optimal allocation.
- Kaplan says an 8× larger model needs about 5× more data to avoid overfitting. Why is that not the same as the compute-optimal data ratio?
- Eq. 4.4 is about when overfitting starts to hurt a converged model. Compute-optimal allocation is about what minimises loss for a fixed number of FLOPs when training stops early, a different question.
Reference
Glossary
- N
- Non-embedding parameter count.
- D
- Dataset size in tokens.
- C, PF-day
- Training compute ≈ 6NBS; 1 PF-day = 1015 FLOP/s × 1 day.
- Bcrit
- Critical batch size, beyond which larger batches give diminishing returns in steps.
- Smin, Cmin
- Steps and compute needed at the critical batch size.
- Power law
- y = ax−k: a straight line of slope −k on log–log axes.
- Nats
- Natural-log units of cross-entropy; 1 nat ≈ 1.44 bits.
- Sample efficiency
- Loss reached per token processed.