Study guide · Neural networks · 2014

Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov · JMLR

Big neural networks can memorise their training data, quirks and noise included, and then do badly on anything new. This paper's remedy is almost absurdly simple: during training, switch off a random half of the hidden units for every example, so no unit can rely on particular partners, and at test time use the whole network with its weights scaled down.

The paper in four sentences

Large networks overfit: they learn the training examples so exactly, noise and all, that they generalise poorly to new ones.

Dropout fights this by randomly deleting units (and their connections) during training, a fresh random choice for every example, typically keeping each hidden unit with probability one half.

Each unit therefore has to be useful on its own, alongside whatever random set of other units happens to be present, instead of forming fragile partnerships that only work on the training data.

At test time nothing is dropped; each unit's outgoing weights are scaled by its keep probability, which approximates averaging the predictions of the enormous family of thinned networks seen in training. It improved results across vision, speech, text and biology benchmarks.

Idea · §1

Overfitting: learning the noise

A neural network learns by adjusting its weights (connection strengths) to reduce its error on training examples. What we actually care about is its error on new examples it has never seen. When a network has many weights and few examples, it can drive the training error to nearly zero by fitting every random wobble in the data. That is overfitting: perfect on the homework, poor in the exam.

The usual cures at the time were stopping training early, penalising large weights, or averaging the predictions of many separately trained models. Averaging works very well, but training and running dozens of large networks was far too expensive.

Below, two identical small networks learn the same 16 noisy points. The blue one trains normally. The amber one uses dropout. Watch how the blue curve contorts itself to pass through every point while the amber one stays closer to the true pattern (dashed).

Both networks have two hidden layers of 40 units and the same starting weights, trained with the same optimiser on the same 16 points (true curve: a sine wave; noise added). "New-data error" is the average squared distance from the true curve over the whole range. The keep probability applies to the amber network's hidden units; at 1.00 it is no longer using dropout. A scaled-down illustration of the paper's idea, not an experiment from it.

TakeawayA big network can fit its training data perfectly and still predict badly. Dropout trades a slightly worse fit on training data for a much better fit on new data.

Check yourself
Why is the blue network's training error lower but its new-data error higher?
It has bent itself to pass through the random noise in each training point, and the noise does not repeat in new data.

Idea · §1, §4 · Figs. 1, 3

Switch units off at random

Here is the whole method. Every time a training example goes through the network, flip a biased coin for each unit: keep it with probability p (the paper suggests one half for hidden units, and higher, around 0.8, for input units), otherwise set its output to zero for this example. The dropped unit sends nothing forward and receives no blame on the backward pass. The next example gets a new set of coin flips.

Each pattern of coin flips gives a different thinned network. A network with n units that can be dropped has 2n possible thinned versions, all sharing the same weights. With a few thousand units that number is beyond astronomical, so almost every training step uses a thinned network that has never been used before.

A network with 4 inputs, two hidden layers of 7 units and 2 outputs. Input units are kept with probability 0.8 (as the paper often uses), hidden units with the slider value; output units are never dropped. Greyed units and their connections are switched off for this example only.
Everyday pictureA football team that trains with a random half of its players missing each day. No one can lean on a favourite teammate always being there, so everyone learns to be useful in many combinations, and the full team on match day is more robust.

TakeawayDropout trains a different random thinned network on every example, all sharing one set of weights.

Check yourself
With 10 droppable units, how many thinned networks are there?
210 = 1,024: each unit is either in or out.
Does a dropped unit's weight change on that training step?
Its outgoing weights get no update from that example, because the unit sent nothing forward and so carried none of the blame.

Idea · §2, §7.1–7.2 · Figs. 7–8

Why it helps: breaking up co-adaptations

The authors' explanation is that in an ordinary network, units can develop co-adaptations: one unit learns to fix up the mistakes of another, and the pair only works together, on these exact training examples. Such conspiracies fit the training data but break on new data. If any partner might vanish at any moment, a unit cannot rely on such fixes and must learn something useful by itself.

The paper motivates this with an analogy to sexual reproduction: genes are shuffled into new combinations each generation, so a gene that only works with a particular set of partners is less successful than one that works with many. The evidence is visual. On handwritten digits, features learned without dropout look like noise, while with dropout each unit clearly detects a stroke or edge (Fig. 7). Dropout also made the hidden activity sparser, with fewer units strongly active at once, without anyone asking for that (Fig. 8).

TakeawayRandom absences stop units from forming fragile partnerships, which pushes each unit toward a feature that is useful on its own.

Check yourself
In one sentence, what is a co-adaptation?
A set of units whose weights only make sense together, for example one unit correcting another's errors on the training set, so the arrangement does not generalise.

Idea · §1, §7.5 · Figs. 2, 11

At test time: one network, scaled down

After training we have one set of weights but a gigantic family of thinned networks. The ideal prediction would average them all, which is impossible. The paper's shortcut: use the full network, nothing dropped, but multiply each unit's outgoing weights by its keep probability p. That way, the signal a unit sends at test time matches what it sent on average during training, when it was present only a fraction p of the time.

Is the shortcut good? The authors checked on MNIST by actually sampling many thinned networks and averaging their outputs. After about 50 samples, the sampled average matched the scaled network, and more samples were only slightly better (Fig. 11). So one cheap pass gets most of the benefit of averaging a huge ensemble.

Later refinementModern libraries use inverted dropout: they divide the kept units' outputs by p during training, so the test-time network needs no change at all. The two are mathematically equivalent for the expected signal. A later line of work (Gal & Ghahramani, 2016) keeps dropout switched on at test time and uses the spread of many sampled predictions as a measure of the model's uncertainty.

TakeawayScaling weights by the keep probability turns the whole network into a cheap stand-in for the average of all the thinned networks.

Check yourself
A unit outputs 2.0 and is kept with probability 0.5. What does the next layer receive from it, on average, during training?
1.0 (2.0 half the time, 0 the other half). Scaling the outgoing weight by 0.5 at test time reproduces that average.

Idea · §6, §11 · Tables 2–9

Results, costs and legacy

The paper tested dropout on digit recognition (MNIST), house-number photos (SVHN), small natural images (CIFAR-10 and -100), ImageNet, speech (TIMIT), news articles (Reuters) and a genetics task. It helped on every one. On MNIST, for example, a standard network made 1.60% errors, while dropout networks got to 0.95%, and 0.79% when combined with pretraining. The convolutional network that won the 2012 ImageNet competition, AlexNet, used dropout.

The price: training takes longer, typically two to three times as long, because each update is noisier. And on very large datasets, where overfitting is less of a risk, the gains shrink (the text-classification gain was small for this reason).

Later refinementBatch normalisation (2015) reported that it often made dropout unnecessary in convolutional networks. Dropout remains common in transformers (the original transformer used a rate of 0.1), though very large language models trained on vast data often use little or none.

TakeawayDropout was a cheap, general regulariser that improved almost everything it touched in 2012–2014; it costs extra training time and matters less when data is plentiful.

Follow the idea

This is the newest step on its timeline so far.