Study guide · Neural networks · 2016

Layer Normalization

Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey Hinton · arXiv

Batch normalisation made deep networks train much faster, but it depends on a batch of examples, which is awkward for recurrent networks, small batches and one-example-at-a-time use. Ba, Kiros and Hinton turn it sideways: normalise each example on its own, across all the units of a layer. It does the same thing in training and at test time, works with any batch size, and became the standard normaliser in transformers.

The paper in four sentences

Batch normalisation standardises each unit using the average and spread of its values across a batch of examples, which ties every example to its batch-mates and needs separate statistics at test time.

Layer normalisation instead takes a single example, looks at all the units in one layer, and standardises them using their average and spread. Then, as in batch norm, each unit gets a learned scale and shift.

Because nothing depends on other examples, it works with a batch of one, behaves identically in training and testing, and slots naturally into recurrent networks, where it is applied afresh at every time step.

It sped up training of recurrent models on several language and generation tasks, though batch norm stayed better for convolutional image networks. A year later it became the default in transformers; many recent language models use a simpler variant, RMSNorm.

Idea · §1–2

What was awkward about batch norm

Batch normalisation takes one unit, collects its values across all the examples in a mini-batch, and rescales them to average 0 and spread 1. It made deep image networks train far faster. But because it relies on the batch, it has three awkward properties.

  • It needs reasonably large batches. With only a few examples, the batch average and spread are noisy estimates; with one example they are meaningless.
  • Training and testing differ. At test time there may be just one example, so batch norm switches to stored averages, a second mode of operation.
  • Recurrent networks are a poor fit. A recurrent network reads a sequence step by step, reusing the same weights at every step. Batch norm would need separate statistics for each time step, and test sequences may be longer than any seen in training.

TakeawayBatch norm's reliance on other examples in the batch is what makes it hard to use with small batches, at test time, and in recurrent networks.

Idea · §3 · Eq. 3

Turn it sideways: normalise across the layer

Picture the numbers a layer produces for a batch as a table: one row per example, one column per unit. Batch norm standardises each column (one unit, many examples). Layer norm standardises each row (one example, all the units in the layer). Each example's numbers are shifted to average 0 and scaled to spread 1 using only that example's own values, then every unit applies its own learned scale and shift.

Switch between the two below and shrink the batch. Layer norm gives the same answer for an example however many others are in the batch; batch norm's answer changes, and at batch size 1 it collapses to zeros.

Left: raw summed inputs of 6 units (columns) for up to 6 examples (rows); the outlined strip shows one group that is normalised together. Right: values after normalising (before the learned scale and shift). Amber is positive, blue negative; darker is larger. Watch the first row (example 1) on the right as you change the batch size.
Everyday pictureBatch norm grades on a curve across the class: your mark depends on who else sat the exam. Layer norm compares each student's scores across their own subjects: your profile of strengths and weaknesses, regardless of classmates.

TakeawayLayer norm computes the average and spread over one example's units in a layer, so each example is normalised independently of all others.

Check yourself
A layer's four units output 1, 3, 5, 7 for one example. What does layer norm give (before scale and shift)?
Mean 4, standard deviation √5 ≈ 2.24, so about −1.34, −0.45, 0.45, 1.34.
Does layer norm's output for an example depend on the batch size?
No. Only that example's own values enter the calculation.

Idea · §3.1 · Eq. 4

A good fit for recurrent networks

A recurrent network keeps a hidden state, a list of numbers it updates at every step of a sequence by mixing the previous state with the new input. If the mixing weights are a bit too large, the numbers grow step after step until the units saturate (stuck at their maximum); a bit too small, and the state fades away. Either way, learning suffers.

Layer norm fixes the scale at every single step: whatever the weights do, the summed inputs are re-centred and rescaled before the activation function. And because it uses only the current step's own numbers, it handles sequences of any length with no stored statistics.

A random recurrent network with 32 tanh units reads 60 steps of small random input. Top: the typical size (root mean square) of the hidden state at each step; blue without layer norm, amber with. Bottom: the share of units that are saturated (|value| > 0.95, where tanh is flat and passes almost no gradient). Small weights make the plain network's state fade; large ones saturate it. With layer norm both curves barely move. Illustrative; untrained weights.

TakeawayNormalising every time step keeps a recurrent network's hidden state in a healthy range, regardless of sequence length or weight scale.

Check yourself
Why does batch norm need per-time-step statistics in an RNN but layer norm does not?
The distribution of summed inputs changes over time steps, so batch statistics pooled across steps would be wrong. Layer norm computes fresh statistics from the current step's units for each example, so nothing needs storing.

Idea · §5.1, §6 · Table 1

What it ignores, and how it performed

A useful way to understand a normaliser is to ask which changes it cancels out. Layer norm ignores any change that scales all of a layer's incoming weights by the same factor, or that scales one input example up or down: either way, after normalising, the result is the same. Batch norm instead ignores changes applied to the whole dataset. Neither is better everywhere; they protect against different kinds of drift.

In experiments, layer norm made recurrent models train faster, and often end up better, on image-caption retrieval, question answering over news articles, learning sentence representations ("skip-thought" vectors), handwriting generation and digit-image generation. One honest exception: in convolutional image networks it helped over no normalisation, but batch norm was better. The authors suspect that units at the edges of an image see very different statistics from those in the middle, so pooling them together does not suit images.

Later refinementThe transformer (2017) used layer norm after every attention and feed-forward block, and later models moved it to the start of each block ("pre-norm"), which trains more stably. RMSNorm (2019) dropped the mean subtraction to save computation, and many recent language models, such as LLaMA, use it instead.

TakeawayLayer norm helped recurrent and sequence models most, was weaker for convolutional image models, and went on to become the transformer's normaliser.

Follow the idea

Built on: Batch norm (2015)

Led to: RMSNorm (2019)