Study guide · Neural networks · 2019

Root Mean Square Layer Normalization

Biao Zhang, Rico Sennrich · NeurIPS 2019

Layer normalisation does two things to each example's activations: it shifts them to average zero and scales them to a standard size. Zhang and Sennrich ask whether the shifting is needed at all. Their answer: mostly not. Rescaling by the root mean square alone matches layer norm's accuracy on a range of tasks while running noticeably faster, and this stripped-down normaliser is now standard in large language models.

The paper in four sentences

Layer normalisation takes the numbers a layer produces for one example, subtracts their average (re-centring) and divides by their spread (re-scaling). It stabilises training but adds noticeable computing time at every layer.

The authors guess that the re-scaling is what actually helps, and that the re-centring can be dropped.

RMSNorm divides the numbers by their root mean square (square them, average, take the square root) and then applies a learned gain. No average is subtracted.

Across translation, reading comprehension, image-caption matching and image classification, it performed about as well as layer norm and cut running time by between 7% and 64% depending on the model and implementation. It became the normaliser of choice in many large language models, including LLaMA.

Idea · §1, §3 · Fig. 1

Layer norm works, but it costs time

Deep networks train more smoothly when each layer's numbers are kept at a sensible size. Layer normalisation does this for one example at a time: it takes all the values a layer computes for that example, subtracts their average, divides by their spread (the standard deviation), and then applies a learned gain to each unit.

The catch is speed. Computing an average and then a spread, for every example at every layer at every step, adds up, especially in recurrent networks where it happens at every time step. The paper's first figure shows a translation model where layer norm makes each step of training more productive, but the extra time per step eats into the overall gain when you measure progress against the clock.

TakeawayLayer norm helps each training step but makes each step slower; the authors set out to keep the help and lose the cost.

Idea · §4 · Eq. 4

Keep the rescaling, drop the re-centring

Layer norm makes the output immune to two kinds of change in a layer's numbers. If they all get bigger or smaller by the same factor, the output is unchanged (re-scaling invariance). If they all shift up or down by the same amount, the output is also unchanged (re-centring invariance).

The authors' hypothesis is that only the first matters for stable training. So RMSNorm only rescales: it divides every value by the root mean square (RMS), a measure of typical size that does not subtract the average first. Square each value, take the average of the squares, take the square root. Then each unit applies its own learned gain, as before.

Try it below. "Scale all" leaves both outputs exactly the same. "Shift all" changes RMSNorm's output but not layer norm's. When the values already average to zero, the two are identical.

Eight values one layer produced for one example (left), and the same values after layer norm (middle) and RMSNorm (right), with gains of 1. Bar heights are drawn to the same scale within each panel. Shifting all values far upward makes RMSNorm's outputs nearly equal and positive, since the shared offset then dominates the RMS; layer norm removes the offset.
Everyday pictureAdjusting the volume of a recording. Layer norm removes any constant hum and then sets the volume; RMSNorm just sets the volume. If there was little hum to begin with, the result sounds the same and you skipped a step.

TakeawayRMSNorm divides by the root mean square and keeps a learned gain. It keeps layer norm's insensitivity to scale but gives up insensitivity to a shared shift.

Check yourself
What is the RMS of 3, −4, 0, 0?
Squares 9, 16, 0, 0; average 25/4 = 6.25; square root 2.5.
When are layer norm and RMSNorm exactly the same?
When the values already average to zero: then subtracting the mean does nothing and the standard deviation equals the RMS.

Idea · §6 · Tables 2–10

Same accuracy, less time

The authors swapped RMSNorm in for layer norm across very different models: recurrent translation systems built in three software frameworks, a transformer translation model, a reading-comprehension model, an image-caption matching model, and a small image classifier. Quality was close to layer norm's in each case, sometimes a little better, sometimes a little worse. Running time dropped every time, though by very different amounts: the saving depends heavily on how much of the model's time is spent normalising and on how each framework implements it.

The chart shows the paper's own timings.

Real numbers from the paper. Time per 1,000 training steps (translation tasks, seconds), per 100 steps (reading comprehension and caption retrieval) or per epoch (CIFAR-10). "Baseline" has no normalisation. The transformer baseline failed to train at all without normalisation (BLEU 0), so its time is not a useful comparison.

TakeawayRMSNorm matched layer norm's quality in the paper's tests and was faster every time, by 7% to 64% depending on the model and software.

Check yourself
Why does the speed-up vary so much between tasks?
It depends on what fraction of total time normalisation takes (large in small recurrent models, small in a big transformer) and on how efficiently each framework implements the operations.

Idea · Later work

From a speed tweak to the LLM default

RMSNorm's arrival was quiet. It became prominent when large language models adopted it: T5 (2019) used a layer norm with no mean subtraction and no bias, which is essentially RMSNorm, and Meta's LLaMA (2023) explicitly cites this paper and applies RMSNorm to the input of every transformer sub-layer. Many open models since have followed.

At the scale of these models, every per-token operation is repeated trillions of times, so a cheaper normaliser with no loss in quality is an easy choice. Fewer parameters (no bias) and simpler numerics also help.

Later refinementThe paper's "partial" variant, which estimates the RMS from only the first few percent of the values, did not catch on: the authors themselves found it was not consistently faster in practice. Modern hardware also narrows the gap, since fused kernels compute layer norm's mean and variance in one pass; RMSNorm stays popular mainly for its simplicity and equal quality.

TakeawayDropping one step of layer norm cost nothing in quality, and the simpler normaliser became standard in LLMs.

Follow the idea

Built on: Layer norm (2016)

Led to: LLaMA (2023)