Study guide · Machine learning & training · 2015

Distilling the Knowledge in a Neural Network

Geoffrey Hinton, Oriol Vinyals, Jeff Dean · NeurIPS 2014 workshop

Big models, and especially ensembles of many models, are accurate but too slow and expensive to deploy. Can a small model learn to do what the big one does? This paper says yes, if the small model is trained to copy the big model's full set of output probabilities, softened with a "temperature", instead of just the right answers. Those soft probabilities carry the big model's sense of which wrong answers are nearly right, and that turns out to be a great deal of knowledge.

The paper in four sentences

When a trained classifier looks at a picture, it does not just name one class: it gives a probability for every class, and the small probabilities it gives to wrong classes reveal what it has learned about how things resemble one another.

Those small probabilities are normally so close to zero that they are invisible, so the authors divide the model's raw scores by a "temperature" before turning them into probabilities, which softens the distribution and brings the hidden resemblances into view.

A small "student" model trained to match these softened outputs of a big "teacher" model (or an ensemble of them) learns far more per example than one trained on right-or-wrong labels alone, and can even recognise a class it never saw in training.

On handwritten digits and on a commercial speech recogniser, a single distilled model recovered most of the accuracy of a much bigger model or a ten-model ensemble, which is why "knowledge distillation" is now a standard way to make large models cheap to run.

Idea · §1

Training and deployment want different models

The authors start with an analogy from biology: many insects have a larva that is good at eating and an adult that is good at travelling and breeding. Machine learning has the same split. During training, you can afford huge models, or an ensemble (many models whose answers are averaged), because the job is to soak up as much structure from the data as possible. At deployment, the model must answer millions of users quickly and cheaply.

The question is how to move what the big model knows into a small one. A common but misleading view is that a model's "knowledge" is its particular weight values. The paper suggests a better view: the knowledge is the function it has learned, the mapping from inputs to outputs. If the small model reproduces that function, it has the knowledge, whatever its insides look like.

Everyday pictureA master chef cannot hand an apprentice the contents of their brain. But the apprentice can learn a lot by tasting every dish the master makes, including the master's comments on near misses ("this is almost a tarte tatin, nothing like a soup").

TakeawayTreat a model's knowledge as its input-to-output behaviour, and you can transfer it to a different, smaller model by teaching it to copy that behaviour.

Idea · §1–2 · Eq. 1

The knowledge hiding in wrong answers

A classifier ends with a softmax: it takes one raw score per class (called a logit) and turns the scores into probabilities that add up to 1. A well-trained model shown a photo of a BMW puts nearly all the probability on "car". But the tiny remainder is not random. The paper's example: a BMW may have only a small chance of being mistaken for a garbage truck, but that mistake is still many times more likely than mistaking it for a carrot.

That ranking of mistakes is exactly the kind of thing a small model would struggle to work out from labels alone. The trouble is that the numbers are microscopic, say one in a million versus one in a billion, so they barely affect training.

The fix is temperature. Before the softmax, divide every logit by a number T. At T = 1 you get the usual sharp probabilities. Raise T and the distribution flattens, so the small differences between wrong answers become large enough to learn from. Try it.

Bars: probabilities from a softmax at temperature T. The small numbers under each bar are the model's raw logits, which never change. At T = 1 almost everything is on one class; by T = 5 or so the ordering of the wrong classes becomes visible. The logits are made up for illustration, inspired by the paper's BMW and digit examples.

TakeawayDividing the logits by a temperature softens a model's probabilities and exposes its knowledge of how classes resemble each other.

Check yourself
Does raising the temperature change which class the model thinks is most likely?
No. Dividing every logit by the same positive number keeps their order, so the top class stays on top. It only changes how spread out the probabilities are.
What happens as T becomes very large?
All probabilities approach 1/(number of classes): the distribution becomes flat and eventually carries no information, so T must be chosen moderately.

Idea · §2–3

Distillation: copy the soft answers

Here is the recipe. Run the big teacher model at a high temperature on a set of examples (the transfer set) and record its softened probabilities, the soft targets. Train the small student, at the same temperature, to produce the same probabilities. Optionally also train it, at normal temperature, on the true labels; the paper found a small weight on the true labels helped. After training, the student is used at temperature 1 like any other model.

Why does this work so well? A hard label ("this is a 2") only says which class is right. A full soft distribution says how much it looks like a 3, a 7 and every other class, so each example carries much more information, and the student's training signal is much less noisy. Below, a small student learns a three-class problem from very few examples. Compare hard labels with soft targets, then try hiding one class entirely from the transfer set.

Left: the teacher's answer at every point (three classes: blue, amber, green). Right: the student's answer after training on the transfer examples (circles). Both are simple linear classifiers here, so this is a toy, but it shows the paper's two effects: soft targets let a student match its teacher from a handful of examples, and a student that never sees an amber example can still learn amber from the amber probability inside the other examples' soft targets. With hard labels it almost never predicts amber.

TakeawaySoft targets carry far more information per example than hard labels, so a student learns more from less data, including about classes it never sees.

Check yourself
In the paper's digit experiment, the student never saw a single 3 during distillation. How could it still recognise 3s?
Other digits' soft targets contain small probabilities of "3" (some 8s and 5s look a bit like 3s). Matching those taught the student what makes something 3-like. Its main weakness was a bias for class 3 that was too low, which the authors fixed by raising it.

Idea · §3–4 · Table 1

Results: digits and a real speech recogniser

Handwritten digits (MNIST). A large, heavily regularised network made 67 errors on the 10,000 test images. A smaller network trained normally made 146. The same small network trained to match the large one's soft targets at temperature 20 made 74, almost as good as its teacher.

Speech recognition. The authors took the acoustic model of a production speech system (Google's, at the time) and trained ten copies to form an ensemble. The ensemble improved accuracy on individual sound frames from 58.9% to 61.1%. A single model distilled from the ensemble reached 60.8%, keeping more than 80% of the improvement, and matched the ensemble's word error rate of 10.7% (versus 10.9% for the baseline). Same size and speed as the original, most of the ensemble's benefit.

Soft targets as a regulariser. Trained on only 3% of the speech data with hard labels, the model badly overfit (44.5% test accuracy). Trained on the same 3% with soft targets from a model that had seen all the data, it reached 57.0%, close to the 58.9% of a model trained on everything.

Later refinementDistillation became the standard way to make big models cheap. DistilBERT (Sanh et al., 2019) shrank BERT by 40% while keeping 97% of its language-understanding score. Switch Transformer distilled huge sparse models into dense ones. Today's small language models are often trained on a larger model's outputs, as with the distilled versions of DeepSeek-R1. Later studies (Stanton et al., 2021) found students often do not match their teachers as closely as you might hope, even when they generalise well.

TakeawayA distilled model of the same size as the original captured most of a ten-model ensemble's gain, and soft targets let a model trained on 3% of the data nearly match one trained on all of it.

Follow the idea

This is the newest step on its timeline so far.