Study guide · Machine learning & training · 2022

Training Compute-Optimal Large Language Models

Jordan Hoffmann et al. · NeurIPS 2022

Given a fixed amount of computing power, should you build a bigger language model or train a smaller one on more text? The influential Kaplan et al. scaling laws said: mostly bigger. This DeepMind paper trained over 400 models and found that model size and training data should grow in equal proportion, about 20 tokens of text per parameter. It proved the point with Chinchilla, a 70-billion-parameter model that beat the four-times-larger Gopher using the same compute.

The paper in four sentences

The cost of training a language model is roughly proportional to its size times the amount of text it reads, so for a fixed budget, a bigger model must read less text and a smaller one can read more.

The authors trained more than 400 models of different sizes on different amounts of text and, using three independent methods, found the best split: every time you double the model, also double the text.

By that rule, the giant models of the day (GPT-3, Gopher, Megatron-Turing NLG) were far too big for the amount of text they had been trained on.

To test it, they trained Chinchilla with the same compute as their own 280-billion-parameter Gopher but with a quarter of the parameters and four times the data, and it was better on nearly every benchmark while being cheaper to use.

Idea · §1 · Eq. 1

A fixed budget, two ways to spend it

A language model has a number of adjustable parameters (its size, N) and is trained by reading a number of tokens (word pieces) of text (D). Every token read costs work proportional to the model's size, so the total computation is roughly 6 × N × D arithmetic operations. If your budget of computation is fixed, a bigger model means fewer tokens and vice versa.

Both help: bigger models can represent more, and more text teaches them more. The question the paper asks is what split gives the lowest error for a given budget. That question matters enormously, because these training runs cost millions of dollars and are done once.

The curve is the paper's loss formula (Eq. 2), here with the corrected constants of Besiroglu et al. (2024) because the paper's printed ones are inconsistent with its own other results (see the In depth tab). It predicts the loss of a model of size N trained on whatever data the budget allows, D = budget / (6N). The first slider sets the budget as a power of 10 times Gopher's training compute (0.0 = the same budget, 1.0 = ten times more). Drag the second slider to try model sizes. Too small and the model can't use the data; too big and it barely gets to read. At the Gopher budget (0.0), grey dots mark Gopher's and Chinchilla's sizes; both were trained with roughly this much compute. The vertical line is the bottom of the curve. The low point is broad, so anything within a factor of two or so is nearly as good.

TakeawayWith compute fixed, model size and data trade off, and there is a best size in the middle.

Check yourself
If you double the model size with a fixed budget, what happens to the number of tokens it can read?
It halves, because compute is roughly 6 × size × tokens.

Idea · §3 · Figs. 2–4 · Table 2

Three ways to find the sweet spot, one answer

The authors trained over 400 models, from 70 million to over 16 billion parameters, on 5 to 500 billion tokens, and analysed them three different ways:

  • Approach 1: train each model size for several different lengths, then for every amount of compute, note which run reached the lowest loss.
  • Approach 2: fix nine compute budgets, train many model sizes at each, and find the bottom of each U-shaped curve (an "IsoFLOP profile", a curve of equal compute, like the one above).
  • Approach 3: fit a single formula for loss as a function of size and data to all the runs, and solve it for the best split.

All three agreed: as compute grows, the best model size and the best amount of data should grow at about the same rate. Doubling one means doubling the other. In round numbers, the best amount of training text is about 20 tokens per parameter. The earlier Kaplan et al. analysis had said model size should grow almost three times as fast as data (on log scales, 0.73 versus 0.27).

Why the difference? The Chinchilla authors blamed mainly the fact that Kaplan et al. used the same training schedule length for every run (later studies found other details, such as how parameters were counted, mattered more). A learning rate that is still high partway through a long schedule makes a model look worse than it would if its schedule had been set for that shorter run, which made long training on more data look less useful than it is.

Everyday pictureJudging runners by their time at the 5 km mark of a marathon, while they are pacing themselves for 42 km, would make you think short races favour other runners. Chinchilla's authors timed each runner over a race of the length they had trained for.

TakeawayThree methods agree that model size and training tokens should scale equally with compute, roughly 20 tokens per parameter.

Idea · Tables 1, 3

The giants were undertrained

Apply the rule to the biggest models of 2020 and 2021 and they all look the same way: far too many parameters for the text they were trained on. Most were trained on roughly 300 billion tokens, whatever their size. By the paper's estimates, a 175-billion-parameter model should see several trillion tokens. Click a model below to see what the rule says.

Sizes and training tokens from the paper's Table 1. "Tokens per parameter" divides one by the other; the dashed line at 20 is the rule of thumb implied by the paper's Table 3. "Compute-optimal size" keeps each model's compute (6 × size × tokens) fixed and shows the size the 20-to-1 rule would choose instead, with the tokens it would then read. Click a bar for numbers.

TakeawayBy this analysis, every large model of its day could have been several times smaller and trained on several times more text for the same cost.

Idea · §4 · Tables 4–6

Chinchilla versus Gopher

The test: Gopher, DeepMind's own 280-billion-parameter model, trained on 300 billion tokens, against Chinchilla, 70 billion parameters trained on 1.4 trillion tokens of the same kind of text, with roughly the same training compute. Chinchilla also switched to AdamW for training, which the authors found helped.

Chinchilla won almost everywhere. On MMLU, a broad exam-style test covering 57 subjects, it averaged 67.6% against Gopher's 60.0%. It also beat Gopher on 58 of 62 BIG-bench tasks, on reading comprehension, on closed-book question answering, and on language modelling. And because it is four times smaller, it is about four times cheaper to run and to fine-tune afterwards.

BenchmarkGopher 280BChinchilla 70B
MMLU, 5-shot average (Table 6)60.0%67.6%
BIG-bench, average of 62 tasks54.4%65.1%
LAMBADA (zero-shot)74.5%77.4%
RACE-h (reading comprehension)71.6%82.3%
Natural Questions, 64-shot (closed book)28.2%35.5%
Later refinement"Chinchilla-optimal" became a standard term, but practice moved past it. Compute-optimal means cheapest to train to a given quality; a model that will answer billions of queries is cheaper overall if it is smaller and trained much longer. LLaMA (2023) trained a 7-billion-parameter model on a trillion tokens, far beyond 20 per parameter, and later open models went further. Independent re-analysis (Besiroglu et al., 2024) also found errors in the paper's third method, though the corrected version agrees with the 20-tokens-per-parameter conclusion.

TakeawayA model a quarter of Gopher's size, trained on four times the data for the same compute, was better on almost every benchmark and cheaper to run.

Check yourself
Why might a company deliberately train a model that is smaller than Chinchilla-optimal?
Because the model will be run many times after training. A smaller model trained on more tokens can reach the same quality at a higher training cost but a much lower cost per answer.
What does the paper imply is the next bottleneck after compute?
Data. Compute-optimal models need trillions of high-quality tokens; the authors note that scaling datasets is only useful if the data stays high quality.

Follow the idea

Built on: Scaling laws (2020)