Study guide · Machine learning & training · 2020

Scaling Laws for Neural Language Models

Jared Kaplan et al. · arXiv

If you make a language model bigger, give it more text, or train it with more computing power, how much better does it get? This paper measured it across seven orders of magnitude and found startlingly simple answers: the error falls along smooth power laws, so the result of a huge, expensive training run can be predicted from cheap small ones. Its advice on how to split a budget between model size and data was later revised by Chinchilla, but the idea of planning AI training with scaling laws stuck.

The paper in four sentences

The authors trained hundreds of transformer language models, from tiny (about a thousand adjustable numbers) to large (1.5 billion), on different amounts of text and with different amounts of computing power, and measured how well each predicts the next word.

The prediction error fell along remarkably straight lines when plotted on logarithmic axes, a pattern called a power law, separately for model size, data size and compute, as long as none of the three was holding the others back.

Details such as the model's exact shape (deep and narrow versus shallow and wide) mattered far less than its total size.

Because bigger models learn more from each word they read, the paper concluded that with a fixed budget you should build a much bigger model and stop training it early, growing model size much faster than data; Chinchilla later showed data should grow about as fast as model size.

Idea · §1

Why you would want to predict a training run

A language model is trained to guess the next word (more precisely, the next token, a word or piece of a word) given the text so far. Its score is the loss: roughly, how surprised it is by the actual next token, averaged over a lot of text. Lower is better.

Training a big model costs a great deal of money and months of computer time. Before you start, you have to decide how big to make it, how much text to feed it and how long to train. Get it wrong and you waste the budget. The authors asked whether the loss can be predicted in advance from three numbers: N, the model's size (its count of adjustable parameters), D, the amount of training text in tokens, and C, the total computation used.

Everyday pictureEngineers testing ship designs build small models and drag them through a water tank, then use known scaling rules to predict how the full-size ship will behave. This paper found the equivalent rules for language models.

TakeawayThe goal is to predict a model's quality from its size, its data and its compute before paying for the run.

Idea · §1.2 · Fig. 1 · Eqs. 1.1–1.3

Straight lines on log paper

The answer is a power law. Make the model 10 times bigger and the loss is multiplied by a fixed factor, always the same factor, whether you go from 1 million to 10 million parameters or from 100 million to a billion. For model size, that factor is about 0.84: each tenfold increase cuts the loss by about 16%.

Plotted on ordinary axes, a power law is a curve that drops steeply and then flattens, and it is hard to see whether it will keep going. Plotted with both axes on a logarithmic scale (where each step along the axis means "10 times more"), it becomes a straight line, and you can extend it with a ruler. The paper found the same kind of straight line for data and for compute too.

The paper's fitted law for model size, Eq. 1.1: loss = (8.8 × 1013 / N)0.076, in nats per token (a unit of surprise), for models trained to near convergence on plenty of data. The slider sets N as a power of 10 (9.0 means 109 = one billion). The paper measured from about 103 to 1.5 × 109 parameters; beyond that the line is an extrapolation.

TakeawayLoss falls as a power law in model size, data and compute: a straight line on log–log axes, which is what makes it predictable.

Check yourself
If the exponent is 0.076, roughly how much bigger must a model be to halve its loss?
You need 10 to the power (log 2 / 0.076) ≈ 104, so about 9,000 times more parameters, with enough data and compute. Gains are steady but expensive.

Idea · §1.1 · §3

Predicting the big run from small ones

Here is why the straight line matters in practice. Train a handful of small, cheap models, plot them, draw the line, and read off where a model 100 times larger should land. Below, the circles are small "experiments", generated from the paper's law with a little random noise. Choose how many of them you are allowed to use, and see how well the fitted line predicts a model far beyond them.

The paper also found that the details of a model's shape hardly matter: for a fixed number of parameters, changing the ratio of depth to width by a factor of 40 changed the loss only slightly. And a model's loss on other kinds of text (books, Wikipedia) tracked its loss on its training text, just with a constant offset.

Log–log axes. Filled circles: the small models you are allowed to fit; hollow circles: larger runs you have not done yet. Amber line: the straight-line fit, extended to 1011 parameters. Dashed: the "true" law used to generate the data (Eq. 1.1). With no noise any range of small models predicts perfectly. With noise, a small error in the fitted slope is magnified by the long extrapolation, so a wider range of small models, reaching closer to the target, gives a much better forecast. Illustrative: real deviations from a power law are what this cannot show.

TakeawayBecause the trend is a straight line on log axes, a few cheap runs let you forecast an expensive one, and the forecast mostly ignores architectural details.

Idea · §5–6 · Figs. 2–3 · Eq. 1.6

Bigger models learn faster, so stop early

A striking finding is that larger models are more sample-efficient: they reach a given loss after seeing fewer tokens, and in fewer training steps, than small models. A big model gets more out of every sentence it reads.

That changes how to spend a fixed computing budget. Each step of a big model costs more, so you can afford fewer steps. But because big models improve so fast per step, the paper found the best use of a budget is a model much larger than you might expect, trained for a relatively short time and stopped well before it has finished improving. Below, each curve is a model size; the horizontal axis is computing spent. Slide the budget and see which size is lowest at that point.

PF-days

Learning curves from the paper's Eq. 1.6, L(N, S) = (Nc/N)0.077 + (Sc/S)0.76, with the constants of its Table 3, for six model sizes. Compute is 6 × N × batch × steps; for simplicity a fixed batch of 0.5 million tokens is assumed, whereas the paper's S counts steps at the ideal batch size. One PF-day is 8.64 × 1019 operations. The vertical line is your budget; the highlighted model is the best one for that budget. Notice that bigger budgets favour bigger models that have not yet levelled off.

TakeawayLarge models learn more per token, so for a fixed budget Kaplan et al. recommend a bigger model trained for less time than you would naively choose.

Check yourself
Why is it not wasteful to stop training a model before it has converged?
The last stretch of training improves the loss very slowly. The same compute spent on a larger model, even one trained for fewer steps, buys more improvement.

Idea · §6.1 · Fig. 3

The advice that was later revised

Putting it together, the paper recommended that when your compute grows tenfold, most of it should go into model size (about 5.4 times bigger) and relatively little into more data (about 1.9 times more tokens). This guided the giant models of 2020–2021: GPT-3 had 175 billion parameters but was trained on only about 300 billion tokens.

The paper also warned that its laws must break down somewhere: extrapolated far enough, they contradict each other, at around a trillion parameters and loss of about 1.7. The authors flagged their estimate of that point as highly uncertain.

Later refinementHoffmann et al. (2022) redid the analysis with learning-rate schedules matched to each run's length and with larger models, and found data should grow about as fast as model size, roughly 20 tokens per parameter. Their 70-billion-parameter Chinchilla, trained on 1.4 trillion tokens, beat the 280-billion-parameter Gopher on the same compute. Later analyses traced most of the disagreement to Kaplan et al.'s counting of parameters without embeddings, their focus on small models, and details such as warmup length and optimiser tuning; Hoffmann et al.'s own explanation, the fixed learning-rate schedule, turned out to matter less. The power-law idea itself held up and is now used to plan essentially every large model.

TakeawayThe paper's method (fit power laws on small runs, extrapolate) became standard; its specific split of compute between size and data did not survive.

Follow the idea

Led to: Chinchilla (2022)