Study guide · Neural networks · 2015

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Sergey Ioffe, Christian Szegedy · ICML 2015

Deep networks in 2015 trained slowly and needed small learning rates and careful initialisation. Ioffe and Szegedy insert a step that standardises each unit's input using the statistics of the current mini-batch, then lets the network rescale it as it likes. Training became dramatically faster and more forgiving. Their explanation, reducing "internal covariate shift", was later disputed, but the technique became a standard part of deep learning.

The paper in four sentences

Inside a deep network, each layer's inputs are the previous layer's outputs, and those keep changing as training adjusts the earlier layers. The authors argued this constant shifting, which they named internal covariate shift, slows training down.

Their fix, batch normalisation: for each unit, take the values it produces across the current mini-batch of examples, subtract their average and divide by their spread, so they are centred on zero with spread one; then apply a learned scale and shift.

Networks with this step tolerated far higher learning rates, cared less about initialisation, could use saturating units like sigmoids, and needed less dropout. On ImageNet a batch-normalised network matched the original accuracy in 14 times fewer training steps.

Later work (Santurkar and colleagues, 2018) showed the benefit probably does not come from reducing internal covariate shift at all, but from making the optimisation landscape smoother.

Idea · §1–2

A moving target for every layer

A deep network is a stack of layers, each feeding the next. Training adjusts all of them at once with backpropagation. The paper's starting observation: when you change the weights of layer 3, the numbers flowing into layer 4 change their typical size and centre. Layer 4 was just learning to deal with the old numbers, and now has to adapt to new ones. The deeper the network, the more these shifts compound.

It was already known that networks learn faster when their inputs are standardised: shifted to average zero and scaled to a typical size of one. The authors call the shifting inside the network internal covariate shift ("covariate shift" being the statistician's term for the input distribution changing) and set out to standardise the input of every layer, not just the first.

A practical symptom was the sigmoid, an S-shaped unit that flattens out for large positive or negative inputs. If a unit's input drifts into the flat part, its slope is nearly zero and it stops learning. Keeping inputs centred keeps units in their responsive middle range.

Everyday pictureA relay of translators, each translating the previous one's output. If the person upstream keeps changing their vocabulary, everyone downstream has to keep re-learning. Batch norm is like agreeing on a standard dictionary between every pair of neighbours.

TakeawayThe paper's premise: inputs to inner layers keep shifting during training, and that slows learning, so standardise them.

Check yourself
Why is it bad for a sigmoid unit's input to drift to large values?
The sigmoid is flat there, so its slope is nearly zero and almost no learning signal passes through it.

Idea · §3 · Algorithm 1

Standardise over the mini-batch, then rescale

Networks are trained on small groups of examples at a time, called mini-batches. Batch normalisation works one unit at a time. Take the values the unit produced for every example in the mini-batch. Compute their average (the mean) and their typical distance from it (the standard deviation). Subtract the mean and divide by the standard deviation. Now this unit's values have average 0 and spread 1 within the batch.

Forcing every unit to exactly that range could limit what the network can express: perhaps a unit genuinely works better with large values. So each unit also gets two learned numbers, a scale γ and a shift β, applied after standardising. The network can learn any centre and spread it likes, including undoing the normalisation entirely. What changes is that centre and spread are now controlled directly by two numbers rather than emerging from tangled interactions of all the earlier weights.

Top row: the values one unit produced for the 8 examples in a mini-batch. Middle row: after subtracting the batch mean and dividing by the batch standard deviation, then applying γ and β. Bottom: those values fed through a sigmoid (the S-curve), showing how many land on its flat, unresponsive parts. Move the raw centre far from zero: without normalisation the sigmoid saturates, with it the values stay in the useful middle.

TakeawayFor each unit: subtract the batch mean, divide by the batch spread, then apply a learned scale and shift.

Check yourself
A batch has values 2, 4, 6. What are they after standardising (before γ and β)?
Mean 4, standard deviation √(8/3) ≈ 1.63, so roughly −1.22, 0, 1.22.
Why add γ and β if they could just undo the normalisation?
So the network loses no expressive power. The benefit is that centre and spread become two directly learnable numbers per unit instead of side effects of all earlier weights.

Idea · §3.3, §4.1–4.2 · Figs. 1–3

Much faster, much bolder training

The headline result is speed. Because each layer's inputs stay in a sensible range, the network tolerates a much larger learning rate (the size of each training step) without blowing up. On ImageNet, a batch-normalised version of Google's Inception network reached the original's accuracy in 14 times fewer steps, and with a larger learning rate went on to beat it. It even trained well with sigmoid units, where the original network made essentially no progress.

The figure below re-runs that comparison in miniature: two identical deep networks, four hidden layers of 16 sigmoid units each, learn to separate two interleaved spirals. One has batch normalisation before every sigmoid. Try a few learning rates.

Left: what each network currently predicts across the square (amber or blue class) with the training points. Right: training loss (lower is better) over 2,000 steps of mini-batches of 32; blue line plain network, amber line batch-normalised. Same starting weights for both. At every learning rate in the slider's range the plain network stalls for a long time, while the batch-normalised one learns the spirals. Predictions for the batch-normalised network use running averages of the batch statistics, as at test time. Illustrative toy in the spirit of the paper's MNIST experiment (Fig. 1).

TakeawayNormalising inside the network let deep models train with bigger steps, fewer tricks and in far fewer iterations.

Check yourself
Why does the plain deep sigmoid network struggle?
Signals and gradients shrink as they pass through many sigmoids, and units drift into flat regions, so the early layers receive almost no useful learning signal. Normalising keeps every layer's inputs in the sigmoid's responsive range.

Idea · §3.1, §3.4 · Algorithm 2

The batch as a side effect: noise in training, averages at test

A quirk follows from using batch statistics: an example's normalised value depends on which other examples happen to be in its batch. The same image gives slightly different internal values each time it is seen. The paper found this jitter acts as a regulariser, a mild form of noise that discourages overfitting, much like dropout, which could often be reduced or removed.

At test time you want a prediction for one image that does not depend on its neighbours. So after training, the network replaces the batch statistics with averages over the whole training set (in practice, running averages kept during training), and each batch-norm step becomes a fixed rescaling.

One fixed example whose raw value is 1.0, from a unit whose values across the whole dataset have mean 0 and spread 1. We put it into 2,000 random mini-batches and record its normalised value each time. With a large dataset-wide normalisation it would always be exactly 1.0 (dashed). Small batches make it jump around a lot; large batches pin it down.

TakeawayBatch statistics make training slightly noisy, which helps generalisation; at test time fixed dataset averages are used instead, so predictions are deterministic.

Check yourself
What goes wrong with a batch size of 1?
The batch standard deviation is zero and the mean equals the value itself, so every value normalises to 0 (or is undefined): the unit carries no information. Batch norm needs reasonably large batches.

Idea · Later work

Right method, disputed reason

In 2018 Shibani Santurkar and colleagues at MIT tested the paper's explanation directly. They deliberately added random shifts and rescalings after each batch-norm layer, recreating the very instability batch norm was supposed to remove, and the networks still trained about as well as with ordinary batch norm. They also found batch norm does not reliably reduce internal covariate shift by their measures. Their alternative account: batch norm makes the error landscape smoother, so gradients are more predictable and larger steps are safe.

So the method works, but probably not for the reason the title gives. Researchers still debate the full story.

Later refinementBatch norm's dependence on the batch is awkward for small batches and for recurrent networks. Layer normalisation (2016) normalises across a single example's units instead and became the default in transformers; group normalisation (2018) serves vision models with small batches. Batch norm remains common in convolutional networks such as ResNet.

TakeawayBatch norm's speed-up is real and widely reproduced; the "internal covariate shift" explanation is not the accepted reason any more.

Follow the idea

Led to: Layer norm (2016)