The paper in four sentences
Large networks overfit: they learn the training examples so exactly, noise and all, that they generalise poorly to new ones.
Dropout fights this by randomly deleting units (and their connections) during training, a fresh random choice for every example, typically keeping each hidden unit with probability one half.
Each unit therefore has to be useful on its own, alongside whatever random set of other units happens to be present, instead of forming fragile partnerships that only work on the training data.
At test time nothing is dropped; each unit's outgoing weights are scaled by its keep probability, which approximates averaging the predictions of the enormous family of thinned networks seen in training. It improved results across vision, speech, text and biology benchmarks.
Idea · §1
Overfitting: learning the noise
A neural network learns by adjusting its weights (connection strengths) to reduce its error on training examples. What we actually care about is its error on new examples it has never seen. When a network has many weights and few examples, it can drive the training error to nearly zero by fitting every random wobble in the data. That is overfitting: perfect on the homework, poor in the exam.
The usual cures at the time were stopping training early, penalising large weights, or averaging the predictions of many separately trained models. Averaging works very well, but training and running dozens of large networks was far too expensive.
Below, two identical small networks learn the same 16 noisy points. The blue one trains normally. The amber one uses dropout. Watch how the blue curve contorts itself to pass through every point while the amber one stays closer to the true pattern (dashed).
TakeawayA big network can fit its training data perfectly and still predict badly. Dropout trades a slightly worse fit on training data for a much better fit on new data.
Check yourself
- Why is the blue network's training error lower but its new-data error higher?
- It has bent itself to pass through the random noise in each training point, and the noise does not repeat in new data.
Idea · §1, §4 · Figs. 1, 3
Switch units off at random
Here is the whole method. Every time a training example goes through the network, flip a biased coin for each unit: keep it with probability p (the paper suggests one half for hidden units, and higher, around 0.8, for input units), otherwise set its output to zero for this example. The dropped unit sends nothing forward and receives no blame on the backward pass. The next example gets a new set of coin flips.
Each pattern of coin flips gives a different thinned network. A network with n units that can be dropped has 2n possible thinned versions, all sharing the same weights. With a few thousand units that number is beyond astronomical, so almost every training step uses a thinned network that has never been used before.
TakeawayDropout trains a different random thinned network on every example, all sharing one set of weights.
Check yourself
- With 10 droppable units, how many thinned networks are there?
- 210 = 1,024: each unit is either in or out.
- Does a dropped unit's weight change on that training step?
- Its outgoing weights get no update from that example, because the unit sent nothing forward and so carried none of the blame.
Idea · §2, §7.1–7.2 · Figs. 7–8
Why it helps: breaking up co-adaptations
The authors' explanation is that in an ordinary network, units can develop co-adaptations: one unit learns to fix up the mistakes of another, and the pair only works together, on these exact training examples. Such conspiracies fit the training data but break on new data. If any partner might vanish at any moment, a unit cannot rely on such fixes and must learn something useful by itself.
The paper motivates this with an analogy to sexual reproduction: genes are shuffled into new combinations each generation, so a gene that only works with a particular set of partners is less successful than one that works with many. The evidence is visual. On handwritten digits, features learned without dropout look like noise, while with dropout each unit clearly detects a stroke or edge (Fig. 7). Dropout also made the hidden activity sparser, with fewer units strongly active at once, without anyone asking for that (Fig. 8).
TakeawayRandom absences stop units from forming fragile partnerships, which pushes each unit toward a feature that is useful on its own.
Check yourself
- In one sentence, what is a co-adaptation?
- A set of units whose weights only make sense together, for example one unit correcting another's errors on the training set, so the arrangement does not generalise.
Idea · §1, §7.5 · Figs. 2, 11
At test time: one network, scaled down
After training we have one set of weights but a gigantic family of thinned networks. The ideal prediction would average them all, which is impossible. The paper's shortcut: use the full network, nothing dropped, but multiply each unit's outgoing weights by its keep probability p. That way, the signal a unit sends at test time matches what it sent on average during training, when it was present only a fraction p of the time.
Is the shortcut good? The authors checked on MNIST by actually sampling many thinned networks and averaging their outputs. After about 50 samples, the sampled average matched the scaled network, and more samples were only slightly better (Fig. 11). So one cheap pass gets most of the benefit of averaging a huge ensemble.
TakeawayScaling weights by the keep probability turns the whole network into a cheap stand-in for the average of all the thinned networks.
Check yourself
- A unit outputs 2.0 and is kept with probability 0.5. What does the next layer receive from it, on average, during training?
- 1.0 (2.0 half the time, 0 the other half). Scaling the outgoing weight by 0.5 at test time reproduces that average.
Idea · §6, §11 · Tables 2–9
Results, costs and legacy
The paper tested dropout on digit recognition (MNIST), house-number photos (SVHN), small natural images (CIFAR-10 and -100), ImageNet, speech (TIMIT), news articles (Reuters) and a genetics task. It helped on every one. On MNIST, for example, a standard network made 1.60% errors, while dropout networks got to 0.95%, and 0.79% when combined with pretraining. The convolutional network that won the 2012 ImageNet competition, AlexNet, used dropout.
The price: training takes longer, typically two to three times as long, because each update is noisier. And on very large datasets, where overfitting is less of a risk, the gains shrink (the text-classification gain was small for this reason).
TakeawayDropout was a cheap, general regulariser that improved almost everything it touched in 2012–2014; it costs extra training time and matters less when data is plentiful.
Abstract, restated
Deep networks with many parameters are powerful but overfit, and combining the predictions of many large networks at test time is slow. Dropout randomly drops units, with their connections, during training, which prevents units from co-adapting too much. Training samples from an exponential number of thinned networks; at test time a single unthinned network with smaller weights approximates averaging their predictions. Dropout significantly reduces overfitting and gives major improvements over other regularisation methods, achieving state-of-the-art results on many benchmarks in vision, speech, document classification and computational biology. JMLR 15 (2014), 1929–1958; the method first appeared in Hinton et al., arXiv 2012.
Section · §4 · Figs. 2–3
The model
Layer l has outputs y(l). A standard feed-forward step is zi(l+1) = wi(l+1)y(l) + bi(l+1), yi(l+1) = f(zi(l+1)). With dropout:
Here ∗ is the elementwise product and p is the retention probability (note: the paper's p is the probability of keeping a unit, whereas most libraries' "dropout rate" is 1 − p). A network with n droppable units defines 2n thinned networks with shared weights; each is trained very rarely, if ever, and weight sharing is what makes training them all feasible. Test-time scaling makes each unit's expected output under the training distribution equal its actual test output.
§3 frames dropout as a generalisation of denoising auto-encoders (noise on hidden units, not just inputs) and as an efficient form of model averaging; the authors report that retaining hidden units with p = 0.5 is close to optimal for many tasks, while input units should be retained with higher probability (typically 0.8).
Section · §5, Appendix A
Training dropout networks
Training is ordinary minibatch SGD: for each training case in a minibatch, sample a thinned network, forward- and back-propagate through it, and average the gradients over the minibatch (a dropped unit contributes zero gradient for its parameters). Pretrained networks (RBMs, auto-encoders) can be fine-tuned with dropout after scaling the pretrained weights by 1/p.
The paper found one regulariser particularly effective alongside dropout: max-norm, constraining each hidden unit's incoming weight vector to ‖w‖2 ≤ c, projecting back onto the ball whenever an update leaves it. Max-norm makes it safe to use very large learning rates, since weights cannot blow up, and the noise from dropout lets the optimiser explore. Appendix A's practical guide:
| Setting | Recommendation (Appendix A) |
|---|---|
| Network size | If n units are right without dropout, use at least n/p with dropout. |
| Learning rate | 10–100 times the rate optimal for a standard net. |
| Momentum | 0.95–0.99 (versus 0.9 typical without dropout). |
| Max-norm c | Typically 3–4. |
| Retention p | 0.5–0.8 for hidden units; about 0.8 for real-valued inputs. |
On MNIST, a regulariser comparison (Table 9, a 784-1024-1024-2048-10 network) gives test errors of 1.62% (L2), 1.60% (L2 + L1), 1.55% (L2 + KL sparsity), 1.35% (max-norm), 1.25% (dropout + L2) and 1.05% (dropout + max-norm).
Check yourself
- Why recommend n/p units?
- Only about np units are active at a time, so to keep the effective capacity of an n-unit network you need about n/p in total.
Section · §6 · Tables 1–8
Experimental results
| Dataset (table) | Without dropout | With dropout |
|---|---|---|
| MNIST, permutation-invariant (Table 2) | 1.60% | 1.35% (logistic, 3×1024) · 1.25% (ReLU) · 0.95% (ReLU + max-norm, 2×8192) · 0.79% (DBM-pretrained + dropout) |
| SVHN (Table 3) | 3.95% | 3.02% (FC layers only) · 2.55% (all layers) |
| CIFAR-10 (Table 4) | 15.60% | 14.32% (FC) · 12.61% (all layers) |
| CIFAR-100 (Table 4) | 43.48% | 41.26% (FC) · 37.20% (all layers) |
| ImageNet ILSVRC-2010 (Table 5) | sparse coding 47.1% / 28.2% | conv net + dropout 37.5% top-1 / 17.0% top-5 |
| TIMIT phone error (Table 7) | 23.4% (6-layer) · 22.7% (DBN-pretrained, 4-layer) | 21.8% · 19.7% |
| Reuters-RCV1 (§6.3) | 31.05% | 29.62% |
| Alternative splicing, code quality in bits, higher is better (Table 8) | 440 (early stopping) | 567 (Bayesian NN: 623) |
Two exceptions are worth reading closely. On Reuters the gain is small, which the authors attribute to the dataset being large enough that overfitting is mild. On alternative splicing, a small dataset, Bayesian neural networks beat dropout: dropout is an approximation to model averaging, and a proper Bayesian average can do better when it is affordable. In ILSVRC-2012, a dropout convolutional network won the competition (top-5 test error 16.4% for a five-model ensemble, Table 6).
Section · §7 · Figs. 7–11
Salient features: features, sparsity, rate, data size, averaging
- Features (§7.1, Fig. 7). A 256-unit ReLU auto-encoder on MNIST learns noisy, co-adapted filters without dropout and localised stroke detectors with p = 0.5.
- Sparsity (§7.2, Fig. 8). Mean hidden activation falls from roughly 2.0 to roughly 0.7, with far fewer strongly active units, without any sparsity penalty.
- Retention rate (§7.3, Fig. 9). With the architecture fixed, test error is flat for 0.4 ≤ p ≤ 0.8. Holding pn fixed instead (so the expected number of active units is constant) makes small p far less harmful.
- Data size (§7.4, Fig. 10). With 100 or 500 MNIST examples dropout does not help; gains are largest at intermediate sizes and shrink as data grows.
- Monte Carlo versus weight scaling (§7.5, Fig. 11). Averaging k sampled thinned networks matches the weight-scaled network at about k = 50; beyond that MC is only slightly better, within one standard deviation.
The figure below repeats the §7.5 comparison on a random untrained toy network with dropout on its inputs and hidden layer. With a nonlinearity between the dropped layers, weight scaling is an approximation, so the scaled output and the true average over thinned networks differ slightly; the question is how many samples MC needs to beat it.
Section · §9–10 · Table 10
Marginalising dropout, and Gaussian dropout
For linear regression with inputs X ∈ ℝN×D and input dropout mask R with entries ∼ Bernoulli(p), the expected loss has a closed form (§9.1):
Substituting w̃ = pw gives ‖y − Xw̃‖² + ((1 − p)/p)‖Γw̃‖²: ridge regression with a per-feature penalty scaled by that feature's magnitude, and strength (1 − p)/p. For logistic regression and deep nets no closed form exists; approximations (e.g. Wang & Manning, 2013) weaken with depth (§9.2). Verify the identity numerically below.
Gaussian dropout (§10). Multiplying activations by r ∼ 𝒩(1, σ²) with σ² = (1 − p)/p matches the mean and variance of scaled Bernoulli dropout and needs no test-time scaling. Table 10 reports it as at least as good: MNIST 0.95% vs 1.08% (Bernoulli), CIFAR-10 12.5% vs 12.6%.
Section · §11 and later work
Limitations and lineage
- Training time. The conclusion notes a dropout net typically takes 2–3 times longer to train than a standard net of the same architecture, because gradients are noisier.
- Approximate averaging. Weight scaling is exact only for a single linear layer; for deep nets it is a heuristic, empirically good (§7.5). Bayesian averaging beat it on a small dataset (Table 8).
- Interaction with batch statistics. Dropout changes activation variance between training and test, which interacts badly with batch normalisation placed after it; BatchNorm's authors also found dropout could be removed or reduced.
- Theory. The paper's explanation (co-adaptation, sexual reproduction) is intuitive rather than formal. Wager, Wang and Liang (2013) showed dropout acts as an adaptive, data-dependent L2 regulariser for generalised linear models; Gal and Ghahramani (2016) interpreted it as approximate variational inference in a Bayesian network (MC dropout).
Reference
Glossary
- Retention p
- Probability a unit is kept (the paper's convention). Library "dropout rate" = 1 − p.
- Thinned network
- The sub-network of units that survived a given dropout sample.
- Co-adaptation
- Units whose usefulness depends on specific other units being present.
- Weight scaling
- Multiplying outgoing weights by p at test time to approximate the ensemble average.
- Inverted dropout
- Dividing kept activations by p during training instead, so test time is unchanged.
- Max-norm
- Constraint ‖w‖2 ≤ c on each unit's incoming weights, enforced by projection.
- MC averaging
- Averaging predictions of k sampled thinned networks.
- Gaussian dropout
- Multiplicative 𝒩(1, (1 − p)/p) noise instead of Bernoulli masks.