The paper in four sentences
Batch normalisation standardises each unit using the average and spread of its values across a batch of examples, which ties every example to its batch-mates and needs separate statistics at test time.
Layer normalisation instead takes a single example, looks at all the units in one layer, and standardises them using their average and spread. Then, as in batch norm, each unit gets a learned scale and shift.
Because nothing depends on other examples, it works with a batch of one, behaves identically in training and testing, and slots naturally into recurrent networks, where it is applied afresh at every time step.
It sped up training of recurrent models on several language and generation tasks, though batch norm stayed better for convolutional image networks. A year later it became the default in transformers; many recent language models use a simpler variant, RMSNorm.
Idea · §1–2
What was awkward about batch norm
Batch normalisation takes one unit, collects its values across all the examples in a mini-batch, and rescales them to average 0 and spread 1. It made deep image networks train far faster. But because it relies on the batch, it has three awkward properties.
- It needs reasonably large batches. With only a few examples, the batch average and spread are noisy estimates; with one example they are meaningless.
- Training and testing differ. At test time there may be just one example, so batch norm switches to stored averages, a second mode of operation.
- Recurrent networks are a poor fit. A recurrent network reads a sequence step by step, reusing the same weights at every step. Batch norm would need separate statistics for each time step, and test sequences may be longer than any seen in training.
TakeawayBatch norm's reliance on other examples in the batch is what makes it hard to use with small batches, at test time, and in recurrent networks.
Idea · §3 · Eq. 3
Turn it sideways: normalise across the layer
Picture the numbers a layer produces for a batch as a table: one row per example, one column per unit. Batch norm standardises each column (one unit, many examples). Layer norm standardises each row (one example, all the units in the layer). Each example's numbers are shifted to average 0 and scaled to spread 1 using only that example's own values, then every unit applies its own learned scale and shift.
Switch between the two below and shrink the batch. Layer norm gives the same answer for an example however many others are in the batch; batch norm's answer changes, and at batch size 1 it collapses to zeros.
TakeawayLayer norm computes the average and spread over one example's units in a layer, so each example is normalised independently of all others.
Check yourself
- A layer's four units output 1, 3, 5, 7 for one example. What does layer norm give (before scale and shift)?
- Mean 4, standard deviation √5 ≈ 2.24, so about −1.34, −0.45, 0.45, 1.34.
- Does layer norm's output for an example depend on the batch size?
- No. Only that example's own values enter the calculation.
Idea · §3.1 · Eq. 4
A good fit for recurrent networks
A recurrent network keeps a hidden state, a list of numbers it updates at every step of a sequence by mixing the previous state with the new input. If the mixing weights are a bit too large, the numbers grow step after step until the units saturate (stuck at their maximum); a bit too small, and the state fades away. Either way, learning suffers.
Layer norm fixes the scale at every single step: whatever the weights do, the summed inputs are re-centred and rescaled before the activation function. And because it uses only the current step's own numbers, it handles sequences of any length with no stored statistics.
TakeawayNormalising every time step keeps a recurrent network's hidden state in a healthy range, regardless of sequence length or weight scale.
Check yourself
- Why does batch norm need per-time-step statistics in an RNN but layer norm does not?
- The distribution of summed inputs changes over time steps, so batch statistics pooled across steps would be wrong. Layer norm computes fresh statistics from the current step's units for each example, so nothing needs storing.
Idea · §5.1, §6 · Table 1
What it ignores, and how it performed
A useful way to understand a normaliser is to ask which changes it cancels out. Layer norm ignores any change that scales all of a layer's incoming weights by the same factor, or that scales one input example up or down: either way, after normalising, the result is the same. Batch norm instead ignores changes applied to the whole dataset. Neither is better everywhere; they protect against different kinds of drift.
In experiments, layer norm made recurrent models train faster, and often end up better, on image-caption retrieval, question answering over news articles, learning sentence representations ("skip-thought" vectors), handwriting generation and digit-image generation. One honest exception: in convolutional image networks it helped over no normalisation, but batch norm was better. The authors suspect that units at the edges of an image see very different statistics from those in the middle, so pooling them together does not suit images.
TakeawayLayer norm helped recurrent and sequence models most, was weaker for convolutional image models, and went on to become the transformer's normaliser.
Abstract, restated
Training state-of-the-art deep networks is computationally expensive; normalising neuron activities reduces training time. Batch normalisation uses the distribution of a neuron's summed input over a mini-batch, which makes its effect depend on mini-batch size, and it is not obvious how to apply it to recurrent networks. This paper transposes batch normalisation into layer normalisation by computing the mean and variance used for normalisation from all summed inputs to the neurons in a layer on a single training case. Each neuron has its own adaptive bias and gain applied after normalisation and before the nonlinearity. Layer normalisation performs exactly the same computation at training and test time, and is straightforward to apply to recurrent networks by computing the statistics separately at each time step. It is very effective at stabilising hidden-state dynamics in recurrent networks and substantially reduces training time compared with previously published techniques. arXiv, July 2016.
Section · §2–3 · Eqs. 1–4
Definitions
For layer l with incoming weights Wl and input hl, summed inputs are al = Wlhl and hl+1 = f(al + bl) (Eq. 1). Batch norm rescales each summed input by statistics over the data distribution, āi = (gi/σi)(ai − μi), with μi, σi estimated per mini-batch (Eq. 2). Layer norm uses statistics over the H hidden units of the layer:
All units in a layer share μ and σ, but each training case has its own. The normalised activation is āi = (gi/σ)(ai − μ), followed by the bias and nonlinearity. For a recurrent network with at = Whhht−1 + Wxhxt:
Gains g and biases b are shared over time steps and initialised to 1 and 0 (§6). In code a small ε is added inside the square root for stability; the paper's equations omit it. The paper's central selling point: since there is no dependence on other cases, the computation is identical in training and inference, needs no running averages, and imposes no constraint on mini-batch size (it works with batch size 1, enabling pure online learning).
The paper explains the RNN motivation in terms of scale: in a standard RNN the average magnitude of summed inputs to the recurrent units tends to grow or shrink at every step, causing exploding or vanishing gradients; layer normalisation makes the network invariant to re-scaling all summed inputs in a layer, which gives much more stable hidden-to-hidden dynamics.
Check yourself
- In a transformer, over which axis is LayerNorm computed for a tensor of shape (batch, sequence, features)?
- Over the feature axis only, separately for every token of every sequence: each token vector is normalised on its own.
Section · §5.1 · Table 1 · Eqs. 5–7
Invariance analysis
Batch norm, weight normalisation (Salimans & Kingma, 2016: āi = (gi/‖wi‖) wi⊤x) and layer norm can all be written as hi = f((gi/σi)(ai − μi) + bi) (Eq. 5) with different choices of μ and σ. Table 1 records which transformations leave the model's output unchanged.
Layer norm is invariant to scaling the whole weight matrix W → δW and to re-centring it, W → W + 1γ⊤, which adds the same γ⊤x to every unit's summed input (Eq. 6). It is invariant to scaling any single input case, x → δx (Eq. 7). It is not invariant to scaling an individual unit's weight vector, since that changes the layer's shared statistics. Batch and weight norm are invariant to individual weight-vector scaling; batch norm is also invariant to re-scaling and re-centring the dataset, while layer norm is not invariant to re-centring it. The figure computes the table rather than quoting it.
Geometry (§5.2). Using the Fisher information as a Riemannian metric (Eqs. 8–9) for normalised generalised linear models, the authors show that the curvature along a weight vector's direction scales inversely with its norm: as weights grow, the effective learning rate along them falls, an implicit early-stopping effect; and learning of the gain parameters depends only on the magnitude of the prediction error, which they argue makes it more robust to input scaling.
Section · §6 · Tables 2–3 · Figs. 1–6
Experiments
| Task (section) | Model | Result reported |
|---|---|---|
| Image–sentence ranking, MS-COCO (§6.1, Table 2, Fig. 1) | Order embeddings with GRU | R@1 caption 46.6 → 48.5, image 37.8 → 38.9; reaches best validation model in 60% of the baseline's time. |
| Question answering, CNN corpus (§6.2, Fig. 2) | Attentive reader (LSTM) | Trains faster and converges to better validation error than both baseline and recurrent batch norm; insensitive to initial gain scale, unlike recurrent BN. |
| Skip-thought vectors, BookCorpus (§6.3, Table 3, Fig. 3) | GRU encoder–decoder | At 1M iterations, improves all seven evaluation numbers over the same model without LN (e.g. MR 77.3 → 79.5, SICK r 0.842 → 0.854); training about a month gives further gains on most of them. |
| DRAW, binarised MNIST (§6.4, Fig. 4) | Recurrent VAE with attention | Converges almost twice as fast; test NLL 82.36 → 82.09 nats after 200 epochs. |
| Handwriting generation, IAM-OnDB (§6.5, Fig. 5) | 3-layer LSTM, mixture density output, batch size 8 | Comparable final log-likelihood, much faster convergence; sequences average about 700 steps. |
| Permutation-invariant MNIST (§6.6, Fig. 6) | 784-1000-1000-10 MLP | Robust to batch size (128 and 4) and faster than batch norm; LN applied to hidden layers only, not the softmax. |
| Convolutional networks (§6.7) | ConvNet | Faster than no normalisation, but batch normalisation outperforms it. |
The paper's explanation for the convolutional result: the assumption that all units in a layer make similar contributions holds well for fully connected layers but poorly for convolutional ones, where units whose receptive fields lie at the image boundary are rarely active and have very different statistics.
Section · §7 and later work
Limitations and lineage
- Pooling over heterogeneous units. All units share μ and σ, so one dominant unit or feature can distort the normalisation of the rest; this is the paper's own diagnosis for ConvNets (§6.7). Group Normalization (Wu & He, 2018) normalises over groups of channels and spatial positions as a middle ground for vision.
- Cost. Two reductions (mean, then variance) per token per layer is non-trivial on accelerators. Zhang and Sennrich's RMSNorm (2019) argued the re-centring is unnecessary and kept only re-scaling by the root mean square, reporting 7%–64% speed-ups at similar quality.
- Placement in transformers. The original transformer applies LN after each residual addition ("post-LN"). GPT-2 moved it to the input of each sub-block, and Xiong et al. (2020) analysed why this "pre-LN" arrangement gives well-behaved gradients at initialisation and can train without learning-rate warm-up.
Check yourself
- Which invariance does layer norm have that RMSNorm gives up?
- Re-centring: adding the same amount to every summed input of an example (e.g. W → W + 1γ⊤). RMSNorm does not subtract the mean, so such a shift changes its output.
Reference
Glossary
- Summed input a
- Pre-activation Wh of a unit, the quantity being normalised.
- H
- Number of hidden units in the layer, the set LN averages over.
- Gain g, bias b
- Per-unit learned scale and shift applied after normalisation.
- Batch norm
- Normalises each unit over the mini-batch (Ioffe & Szegedy, 2015).
- Weight norm
- Divides each unit's weight vector by its L2 norm (Salimans & Kingma, 2016).
- Recurrent BN
- Batch norm in LSTMs with per-time-step statistics (Cooijmans et al., 2016).
- Pre-LN / post-LN
- LayerNorm placed before each transformer sub-block versus after the residual addition.