The paper in four sentences
Inside a deep network, each layer's inputs are the previous layer's outputs, and those keep changing as training adjusts the earlier layers. The authors argued this constant shifting, which they named internal covariate shift, slows training down.
Their fix, batch normalisation: for each unit, take the values it produces across the current mini-batch of examples, subtract their average and divide by their spread, so they are centred on zero with spread one; then apply a learned scale and shift.
Networks with this step tolerated far higher learning rates, cared less about initialisation, could use saturating units like sigmoids, and needed less dropout. On ImageNet a batch-normalised network matched the original accuracy in 14 times fewer training steps.
Later work (Santurkar and colleagues, 2018) showed the benefit probably does not come from reducing internal covariate shift at all, but from making the optimisation landscape smoother.
Idea · §1–2
A moving target for every layer
A deep network is a stack of layers, each feeding the next. Training adjusts all of them at once with backpropagation. The paper's starting observation: when you change the weights of layer 3, the numbers flowing into layer 4 change their typical size and centre. Layer 4 was just learning to deal with the old numbers, and now has to adapt to new ones. The deeper the network, the more these shifts compound.
It was already known that networks learn faster when their inputs are standardised: shifted to average zero and scaled to a typical size of one. The authors call the shifting inside the network internal covariate shift ("covariate shift" being the statistician's term for the input distribution changing) and set out to standardise the input of every layer, not just the first.
A practical symptom was the sigmoid, an S-shaped unit that flattens out for large positive or negative inputs. If a unit's input drifts into the flat part, its slope is nearly zero and it stops learning. Keeping inputs centred keeps units in their responsive middle range.
TakeawayThe paper's premise: inputs to inner layers keep shifting during training, and that slows learning, so standardise them.
Check yourself
- Why is it bad for a sigmoid unit's input to drift to large values?
- The sigmoid is flat there, so its slope is nearly zero and almost no learning signal passes through it.
Idea · §3 · Algorithm 1
Standardise over the mini-batch, then rescale
Networks are trained on small groups of examples at a time, called mini-batches. Batch normalisation works one unit at a time. Take the values the unit produced for every example in the mini-batch. Compute their average (the mean) and their typical distance from it (the standard deviation). Subtract the mean and divide by the standard deviation. Now this unit's values have average 0 and spread 1 within the batch.
Forcing every unit to exactly that range could limit what the network can express: perhaps a unit genuinely works better with large values. So each unit also gets two learned numbers, a scale γ and a shift β, applied after standardising. The network can learn any centre and spread it likes, including undoing the normalisation entirely. What changes is that centre and spread are now controlled directly by two numbers rather than emerging from tangled interactions of all the earlier weights.
TakeawayFor each unit: subtract the batch mean, divide by the batch spread, then apply a learned scale and shift.
Check yourself
- A batch has values 2, 4, 6. What are they after standardising (before γ and β)?
- Mean 4, standard deviation √(8/3) ≈ 1.63, so roughly −1.22, 0, 1.22.
- Why add γ and β if they could just undo the normalisation?
- So the network loses no expressive power. The benefit is that centre and spread become two directly learnable numbers per unit instead of side effects of all earlier weights.
Idea · §3.3, §4.1–4.2 · Figs. 1–3
Much faster, much bolder training
The headline result is speed. Because each layer's inputs stay in a sensible range, the network tolerates a much larger learning rate (the size of each training step) without blowing up. On ImageNet, a batch-normalised version of Google's Inception network reached the original's accuracy in 14 times fewer steps, and with a larger learning rate went on to beat it. It even trained well with sigmoid units, where the original network made essentially no progress.
The figure below re-runs that comparison in miniature: two identical deep networks, four hidden layers of 16 sigmoid units each, learn to separate two interleaved spirals. One has batch normalisation before every sigmoid. Try a few learning rates.
TakeawayNormalising inside the network let deep models train with bigger steps, fewer tricks and in far fewer iterations.
Check yourself
- Why does the plain deep sigmoid network struggle?
- Signals and gradients shrink as they pass through many sigmoids, and units drift into flat regions, so the early layers receive almost no useful learning signal. Normalising keeps every layer's inputs in the sigmoid's responsive range.
Idea · §3.1, §3.4 · Algorithm 2
The batch as a side effect: noise in training, averages at test
A quirk follows from using batch statistics: an example's normalised value depends on which other examples happen to be in its batch. The same image gives slightly different internal values each time it is seen. The paper found this jitter acts as a regulariser, a mild form of noise that discourages overfitting, much like dropout, which could often be reduced or removed.
At test time you want a prediction for one image that does not depend on its neighbours. So after training, the network replaces the batch statistics with averages over the whole training set (in practice, running averages kept during training), and each batch-norm step becomes a fixed rescaling.
TakeawayBatch statistics make training slightly noisy, which helps generalisation; at test time fixed dataset averages are used instead, so predictions are deterministic.
Check yourself
- What goes wrong with a batch size of 1?
- The batch standard deviation is zero and the mean equals the value itself, so every value normalises to 0 (or is undefined): the unit carries no information. Batch norm needs reasonably large batches.
Idea · Later work
Right method, disputed reason
In 2018 Shibani Santurkar and colleagues at MIT tested the paper's explanation directly. They deliberately added random shifts and rescalings after each batch-norm layer, recreating the very instability batch norm was supposed to remove, and the networks still trained about as well as with ordinary batch norm. They also found batch norm does not reliably reduce internal covariate shift by their measures. Their alternative account: batch norm makes the error landscape smoother, so gradients are more predictable and larger steps are safe.
So the method works, but probably not for the reason the title gives. Researchers still debate the full story.
TakeawayBatch norm's speed-up is real and widely reproduced; the "internal covariate shift" explanation is not the accepted reason any more.
Abstract, restated
Training deep networks is complicated by the fact that each layer's input distribution changes as the preceding layers' parameters change, which requires lower learning rates and careful initialisation and makes saturating nonlinearities hard to train. The authors call this internal covariate shift and address it by normalising layer inputs, making normalisation part of the architecture and performing it for each training mini-batch. Batch Normalization permits much higher learning rates and less careful initialisation, and in some cases removes the need for Dropout. Applied to a state-of-the-art image classifier, it reaches the same accuracy with 14 times fewer training steps and beats the original by a significant margin; an ensemble reaches 4.9% top-5 validation error (4.8% test) on ImageNet. ICML 2015.
Section · §2
Internal covariate shift, and why normalisation must be differentiated
Internal covariate shift is defined as the change in the distribution of network activations due to the change in network parameters during training. Whitening inputs (zero mean, unit variance, decorrelated) is known to speed convergence (LeCun et al., 1998), so the goal is to whiten every layer's inputs.
The paper's key argument is that normalisation must sit inside the gradient computation. Consider a layer x = u + b normalised as x̂ = x − E[x], where the mean is computed over the training data. If the gradient step ignores the dependence of E[x] on b, then Δb ∝ −∂ℓ/∂x̂, yet u + (b + Δb) − E[u + (b + Δb)] = u + b − E[u + b]: the update changes nothing in the output or the loss, so b grows without bound. The paper reports observing this blow-up empirically. Full whitening inside backprop, however, needs the covariance matrix, its inverse square root and their derivatives: too costly.
Section · §3 · Algorithm 1
The BN transform
Two simplifications: normalise each scalar feature independently (no decorrelation), and estimate the statistics from the mini-batch. For a mini-batch ℬ = {x1…m} of one activation:
Algorithm 1: Batch Normalizing Transform, applied to activation x over a mini-batch μℬ ← (1/m) Σi xi mini-batch mean σ²ℬ ← (1/m) Σi (xi − μℬ)² mini-batch variance x̂i ← (xi − μℬ) / √(σ²ℬ + ε) normalise yi ← γ x̂i + β ≡ BNγ,β(xi) scale and shift
Setting γ = √Var[x] and β = E[x] recovers the original activation, so the transform can represent the identity. Backpropagation goes through μℬ and σ²ℬ, so the gradient for each example depends on the whole batch. The paper gives the chain-rule terms:
The second line is the paper's three chain-rule terms (through x̂, μℬ and σ²ℬ) collected into one expression. It shows that the input gradient is the upstream gradient with its batch mean and its component along x̂ projected out: BN removes gradient directions that would only shift or rescale the batch. The live training on the Plain English tab uses exactly this backward pass.
Check yourself
- Sum ∂ℓ/∂xi over the batch. What do you get, and why?
- Zero. Shifting all inputs by a constant leaves every x̂i unchanged, so the loss has zero derivative in that direction.
Section · §3.1–3.2 · Algorithm 2
Inference and convolutional layers
Inference (Algorithm 2). After training, use population statistics: E[x] ← Eℬ[μℬ] and Var[x] ← (m/(m − 1)) · Eℬ[σ²ℬ], the unbiased variance estimate, averaged over training mini-batches of size m (moving averages can be used instead, to track the model as it trains). BN then becomes a fixed affine map,
which can be folded into the preceding linear layer.
Placement and convolutions (§3.2). BN is applied to x = Wu + b, immediately before the nonlinearity: z = g(BN(Wu)); the bias b is dropped because mean subtraction cancels it and β takes its role. The authors argue Wu is more likely to have a symmetric, non-sparse, "more Gaussian" distribution than the layer input u. For convolutional layers, normalisation must respect the convolutional property, so statistics are shared over all spatial locations of a feature map: the effective mini-batch is m′ = m · p · q for p × q feature maps, with one γ, β pair per feature map.
Section · §3.3–3.4
Why higher learning rates work; regularisation
BN makes a layer invariant to the scale of its weights. For scalar a:
So backpropagation through the layer is unaffected by the scale of its parameters, and larger weights receive smaller gradients, which stabilises parameter growth. The authors further conjecture that BN may push layer Jacobians toward singular values near 1, which would help gradient propagation; they present this as a conjecture under strong assumptions. Check the scale invariance numerically:
Regularisation (§3.4). A training example is seen together with other examples in its mini-batch, so the network no longer produces deterministic values for it. The authors found this beneficial for generalisation and that Dropout can be removed or reduced.
Section · §4 · Figs. 1–4
Experiments
MNIST (§4.1, Fig. 1). Three fully connected hidden layers of 100 sigmoid units, 50,000 steps with 60 examples per mini-batch. The BN network reaches higher test accuracy, and the 15th, 50th and 85th percentiles of a typical sigmoid input stay much more stable over training than without BN.
ImageNet (§4.2). A variant of Inception (5×5 convolutions replaced by two 3×3 layers; 13.6 million parameters). Simply adding BN is "BN-Baseline". The accelerated versions also changed (§4.2.1): increase the learning rate; remove Dropout; reduce L2 weight regularisation by a factor of 5; accelerate learning-rate decay by 6×; remove Local Response Normalisation; shuffle training examples more thoroughly; reduce photometric distortions.
Ensemble (§4.2.3, Fig. 4). Six BN-x30-based networks with some variations (initial weights, dropout of 5% or 10%, per-activation BN in the last hidden layers) reach 4.9% top-5 validation error and 4.82% top-5 test error, ahead of the previous best (He et al., 2015, 4.94%) and of the reported accuracy of human raters. A single BN-Inception with multi-crop evaluation gets 5.82% top-5, versus 6.67% for the GoogLeNet ensemble.
Section · §5 and later work
Limitations and lineage
- The explanation. Santurkar, Tsipras, Ilyas and Madry ("How Does Batch Normalization Help Optimization?", NeurIPS 2018) injected random, time-varying non-zero-mean, non-unit-variance noise after BN layers and found training performance largely unaffected, and found BN does not necessarily reduce internal covariate shift. They argue BN improves the Lipschitzness of the loss and its gradients, smoothing the landscape. Other accounts emphasise the scale invariance of §3.3 (an effective, automatically adapting learning rate).
- Batch dependence. Small or non-i.i.d. batches give noisy or biased statistics, and the train/test discrepancy (batch versus population statistics) hurts. Batch Renormalization (Ioffe, 2017) and Group Normalization (Wu & He, 2018) address this.
- Sequences. The conclusion lists recurrent networks as future work; per-time-step statistics proved awkward, motivating Layer Normalization.
- Placement. The paper normalises before the nonlinearity; many later architectures normalise elsewhere (e.g. pre-activation residual blocks), with mixed evidence on what is best.
Reference
Glossary
- Internal covariate shift
- Change in the distribution of network activations due to parameter changes during training (the paper's definition).
- Whitening
- Transforming inputs to zero mean, unit variance and no correlation.
- μℬ, σ²ℬ
- Mini-batch mean and (biased) variance of one activation.
- γ, β
- Learned per-activation (per-feature-map for convolutions) scale and shift.
- ε
- Small constant added to the variance for numerical stability.
- Population statistics
- Mean and unbiased variance over the training data, used at inference.
- BN-x5, BN-x30
- BN-Inception with 5× and 30× the original initial learning rate.
- Saturation
- Operating in the flat region of a nonlinearity, where gradients vanish.