The paper in four sentences
When a trained classifier looks at a picture, it does not just name one class: it gives a probability for every class, and the small probabilities it gives to wrong classes reveal what it has learned about how things resemble one another.
Those small probabilities are normally so close to zero that they are invisible, so the authors divide the model's raw scores by a "temperature" before turning them into probabilities, which softens the distribution and brings the hidden resemblances into view.
A small "student" model trained to match these softened outputs of a big "teacher" model (or an ensemble of them) learns far more per example than one trained on right-or-wrong labels alone, and can even recognise a class it never saw in training.
On handwritten digits and on a commercial speech recogniser, a single distilled model recovered most of the accuracy of a much bigger model or a ten-model ensemble, which is why "knowledge distillation" is now a standard way to make large models cheap to run.
Idea · §1
Training and deployment want different models
The authors start with an analogy from biology: many insects have a larva that is good at eating and an adult that is good at travelling and breeding. Machine learning has the same split. During training, you can afford huge models, or an ensemble (many models whose answers are averaged), because the job is to soak up as much structure from the data as possible. At deployment, the model must answer millions of users quickly and cheaply.
The question is how to move what the big model knows into a small one. A common but misleading view is that a model's "knowledge" is its particular weight values. The paper suggests a better view: the knowledge is the function it has learned, the mapping from inputs to outputs. If the small model reproduces that function, it has the knowledge, whatever its insides look like.
TakeawayTreat a model's knowledge as its input-to-output behaviour, and you can transfer it to a different, smaller model by teaching it to copy that behaviour.
Idea · §1–2 · Eq. 1
The knowledge hiding in wrong answers
A classifier ends with a softmax: it takes one raw score per class (called a logit) and turns the scores into probabilities that add up to 1. A well-trained model shown a photo of a BMW puts nearly all the probability on "car". But the tiny remainder is not random. The paper's example: a BMW may have only a small chance of being mistaken for a garbage truck, but that mistake is still many times more likely than mistaking it for a carrot.
That ranking of mistakes is exactly the kind of thing a small model would struggle to work out from labels alone. The trouble is that the numbers are microscopic, say one in a million versus one in a billion, so they barely affect training.
The fix is temperature. Before the softmax, divide every logit by a number T. At T = 1 you get the usual sharp probabilities. Raise T and the distribution flattens, so the small differences between wrong answers become large enough to learn from. Try it.
TakeawayDividing the logits by a temperature softens a model's probabilities and exposes its knowledge of how classes resemble each other.
Check yourself
- Does raising the temperature change which class the model thinks is most likely?
- No. Dividing every logit by the same positive number keeps their order, so the top class stays on top. It only changes how spread out the probabilities are.
- What happens as T becomes very large?
- All probabilities approach 1/(number of classes): the distribution becomes flat and eventually carries no information, so T must be chosen moderately.
Idea · §2–3
Distillation: copy the soft answers
Here is the recipe. Run the big teacher model at a high temperature on a set of examples (the transfer set) and record its softened probabilities, the soft targets. Train the small student, at the same temperature, to produce the same probabilities. Optionally also train it, at normal temperature, on the true labels; the paper found a small weight on the true labels helped. After training, the student is used at temperature 1 like any other model.
Why does this work so well? A hard label ("this is a 2") only says which class is right. A full soft distribution says how much it looks like a 3, a 7 and every other class, so each example carries much more information, and the student's training signal is much less noisy. Below, a small student learns a three-class problem from very few examples. Compare hard labels with soft targets, then try hiding one class entirely from the transfer set.
TakeawaySoft targets carry far more information per example than hard labels, so a student learns more from less data, including about classes it never sees.
Check yourself
- In the paper's digit experiment, the student never saw a single 3 during distillation. How could it still recognise 3s?
- Other digits' soft targets contain small probabilities of "3" (some 8s and 5s look a bit like 3s). Matching those taught the student what makes something 3-like. Its main weakness was a bias for class 3 that was too low, which the authors fixed by raising it.
Idea · §3–4 · Table 1
Results: digits and a real speech recogniser
Handwritten digits (MNIST). A large, heavily regularised network made 67 errors on the 10,000 test images. A smaller network trained normally made 146. The same small network trained to match the large one's soft targets at temperature 20 made 74, almost as good as its teacher.
Speech recognition. The authors took the acoustic model of a production speech system (Google's, at the time) and trained ten copies to form an ensemble. The ensemble improved accuracy on individual sound frames from 58.9% to 61.1%. A single model distilled from the ensemble reached 60.8%, keeping more than 80% of the improvement, and matched the ensemble's word error rate of 10.7% (versus 10.9% for the baseline). Same size and speed as the original, most of the ensemble's benefit.
Soft targets as a regulariser. Trained on only 3% of the speech data with hard labels, the model badly overfit (44.5% test accuracy). Trained on the same 3% with soft targets from a model that had seen all the data, it reached 57.0%, close to the 58.9% of a model trained on everything.
TakeawayA distilled model of the same size as the original captured most of a ten-model ensemble's gain, and soft targets let a model trained on 3% of the data nearly match one trained on all of it.
Abstract, restated
Averaging the predictions of many models improves almost any learner, but ensembles are cumbersome to deploy. Building on Caruana and colleagues' model compression, the paper develops a different compression technique, distillation, that transfers an ensemble's or a large model's knowledge into a single small model. It shows surprising results on MNIST and significantly improves the acoustic model of a heavily used commercial speech system by distilling an ensemble into one model. It also introduces ensembles of a generalist plus many specialist models that learn to distinguish fine-grained classes the full model confuses, which can be trained rapidly and in parallel. Hinton, Vinyals & Dean, NIPS 2014 Deep Learning Workshop (arXiv 1503.02531, 2015).
Section · §2 · Eqs. 1–2
Softmax temperature and the distillation objective
The cumbersome model produces soft targets pi at high T. The distilled model is trained at the same T to minimise cross-entropy C = −Σi pi log qi, and used at T = 1 afterwards. When true labels are known, the best results came from a weighted average of two objectives: the soft-target cross-entropy at high T, and the hard-label cross-entropy at T = 1 with a considerably lower weight. The per-logit gradient of the soft term is
with v the teacher's logits. Because the gradient magnitudes of the soft term scale as 1/T2, the authors multiply it by T2, which keeps the relative contribution of hard and soft targets roughly unchanged when T is varied. This T2 factor is a standard detail of distillation implementations and is easy to forget.
Section · §2.1 · Eqs. 3–4
Matching logits is the high-temperature limit
For T large compared with the logits, ex/T ≈ 1 + x/T, so with N classes
So in the high-temperature limit, distillation is equivalent to minimising ½(zi − vi)2, the logit-matching of Caruana et al. (Buciluǎ, Caruana & Niculescu-Mizil, 2006, "Model compression"). At lower temperatures, distillation pays much less attention to matching logits that are far more negative than the average. That is a potential advantage, since such logits are almost unconstrained by the teacher's training signal and may be noisy, and a potential disadvantage, since they might carry useful knowledge. Empirically, intermediate temperatures worked best when the student was too small to capture all the teacher's knowledge.
Section · §3–4, §6 · Tables 1, 5
MNIST and speech experiments
MNIST (§3). Teacher: two hidden layers of 1200 ReLUs, trained on all 60,000 cases with dropout, weight constraints and inputs jittered by up to two pixels: 67 test errors. Student: two hidden layers of 800 ReLUs, no regularisation: 146 errors trained on hard labels; 74 errors when trained with the additional soft-target task at T = 20. Students with 300 or more units per layer did similarly well for all T above 8; with only 30 units per layer, T between 2.5 and 4 worked much better than higher or lower values.
Omitted class. With every 3 removed from the transfer set, the distilled model made 206 errors, 133 of them on the 1010 test 3s. Most errors were caused by the learned bias for class 3 being too low; raising it by 3.5 gave 109 errors, of which 14 were on 3s, i.e. 98.6% of test 3s correct despite never seeing one. With a transfer set containing only 7s and 8s, the error was 47.3%, falling to 13.2% after lowering the biases of 7 and 8 by 7.6.
| Speech system (§4, Table 1) | Test frame accuracy | Word error rate |
|---|---|---|
| Baseline | 58.9% | 10.9% |
| 10× ensemble | 61.1% | 10.7% |
| Distilled single model | 60.8% | 10.7% |
The acoustic model: 8 hidden layers of 2560 ReLUs, softmax over 14,000 labels (HMM states), about 85M parameters, trained on about 2000 hours of English speech (about 700M examples). Ensemble members differed only in random initialisation. For distillation the authors tried T ∈ {1, 2, 5, 10} and used a relative weight of 0.5 on the hard-target cross-entropy. More than 80% of the ensemble's frame-accuracy gain transferred.
Soft targets as regulariser (§6, Table 5). 100% of data, hard targets: 63.4% train / 58.9% test frame accuracy. 3% of data, hard targets: 67.3% / 44.5% (severe overfitting). 3% of data, soft targets: 65.4% / 57.0%.
Section · §5, §7 · Tables 3–4
Specialists on JFT
JFT is an internal Google dataset of 100 million labelled images in 15,000 classes. Training an ensemble of full models was impractical (the baseline took about six months on many cores). Instead the authors train one generalist plus 61 specialists, each covering 300 easily confused classes plus a "dustbin" class for everything else. Specialists are initialised from the generalist, trained on data enriched with their classes, and can be trained in parallel. The confusable subsets are found by clustering the covariance matrix of the generalist's predictions (§5.3). At test time, the specialists relevant to the generalist's top classes are combined with it by finding a full distribution that minimises a sum of KL divergences to the generalist and active specialists (§5.4).
Result (§5.5, Table 3): conditional test accuracy 43.1% → 45.9%, overall top-1 test accuracy 25.0% → 26.1%, a 4.4% relative improvement; gains grew with the number of specialists covering a class (Table 4). The specialists were not distilled back into a single model in this paper; that is described as future work. §6.1 notes that soft targets from the generalist can stop specialists, which see enriched and therefore biased data, from overfitting. §7 contrasts this with mixtures of experts: specialists are assigned by a fixed clustering rather than a learned gating network, which makes them easy to parallelise.
Section · Lineage
Limitations and what came next
- Priority. Model compression by training a small net on an ensemble's outputs is due to Buciluǎ, Caruana & Niculescu-Mizil (2006); Ba & Caruana (2014) matched logits with shallow students. This paper's contributions are the temperature, the unifying view (logit matching as a limit), and the large-scale demonstrations.
- Vocabulary. "Dark knowledge", the name commonly attached to the information in the small probabilities, comes from Hinton's talks rather than the paper itself.
- Fidelity. Stanton et al. (2021, "Does knowledge distillation really work?") found that students often generalise well but agree with their teachers much less than expected, largely because the optimisation is hard, not because the student lacks capacity.
- Why it works remains debated: explanations include richer per-example information, label-smoothing-like regularisation (Müller et al., 2019, showed label smoothing hurts subsequent distillation), and "privileged information".
- Extensions. Intermediate-feature matching (FitNets, 2015), self-distillation (born-again networks, 2018), sequence-level distillation for translation (Kim & Rush, 2016), task-agnostic language-model distillation (DistilBERT, TinyBERT), and distillation of sparse into dense models in Switch Transformers (which kept about 30% of the quality gain). For large language models, training small models on the outputs or full token distributions of a larger one is now common.
Check yourself
- Why multiply the soft-target loss by T2?
- Eq. 4 shows its gradients scale as 1/T2; without compensation, raising T would silently shrink the soft term relative to the hard-label term.
- When might a lower temperature beat logit matching?
- When the student is small: low T lets it ignore the teacher's very negative logits, which are poorly constrained and noisy, and spend capacity on the classes that matter.
Reference
Glossary
- Logit
- Pre-softmax score zi.
- Temperature T
- Divisor of the logits before the softmax; larger means softer.
- Soft targets
- Teacher's probabilities at temperature T.
- Transfer set
- Inputs on which the student is trained to match the teacher (labelled or not).
- Cumbersome model
- The paper's term for the teacher: a large model or an ensemble.
- Specialist
- Model trained on a confusable subset of classes plus a dustbin class.
- Frame accuracy
- Fraction of 10 ms speech frames assigned the correct HMM state.