The paper in four sentences
Layer normalisation takes the numbers a layer produces for one example, subtracts their average (re-centring) and divides by their spread (re-scaling). It stabilises training but adds noticeable computing time at every layer.
The authors guess that the re-scaling is what actually helps, and that the re-centring can be dropped.
RMSNorm divides the numbers by their root mean square (square them, average, take the square root) and then applies a learned gain. No average is subtracted.
Across translation, reading comprehension, image-caption matching and image classification, it performed about as well as layer norm and cut running time by between 7% and 64% depending on the model and implementation. It became the normaliser of choice in many large language models, including LLaMA.
Idea · §1, §3 · Fig. 1
Layer norm works, but it costs time
Deep networks train more smoothly when each layer's numbers are kept at a sensible size. Layer normalisation does this for one example at a time: it takes all the values a layer computes for that example, subtracts their average, divides by their spread (the standard deviation), and then applies a learned gain to each unit.
The catch is speed. Computing an average and then a spread, for every example at every layer at every step, adds up, especially in recurrent networks where it happens at every time step. The paper's first figure shows a translation model where layer norm makes each step of training more productive, but the extra time per step eats into the overall gain when you measure progress against the clock.
TakeawayLayer norm helps each training step but makes each step slower; the authors set out to keep the help and lose the cost.
Idea · §4 · Eq. 4
Keep the rescaling, drop the re-centring
Layer norm makes the output immune to two kinds of change in a layer's numbers. If they all get bigger or smaller by the same factor, the output is unchanged (re-scaling invariance). If they all shift up or down by the same amount, the output is also unchanged (re-centring invariance).
The authors' hypothesis is that only the first matters for stable training. So RMSNorm only rescales: it divides every value by the root mean square (RMS), a measure of typical size that does not subtract the average first. Square each value, take the average of the squares, take the square root. Then each unit applies its own learned gain, as before.
Try it below. "Scale all" leaves both outputs exactly the same. "Shift all" changes RMSNorm's output but not layer norm's. When the values already average to zero, the two are identical.
TakeawayRMSNorm divides by the root mean square and keeps a learned gain. It keeps layer norm's insensitivity to scale but gives up insensitivity to a shared shift.
Check yourself
- What is the RMS of 3, −4, 0, 0?
- Squares 9, 16, 0, 0; average 25/4 = 6.25; square root 2.5.
- When are layer norm and RMSNorm exactly the same?
- When the values already average to zero: then subtracting the mean does nothing and the standard deviation equals the RMS.
Idea · §6 · Tables 2–10
Same accuracy, less time
The authors swapped RMSNorm in for layer norm across very different models: recurrent translation systems built in three software frameworks, a transformer translation model, a reading-comprehension model, an image-caption matching model, and a small image classifier. Quality was close to layer norm's in each case, sometimes a little better, sometimes a little worse. Running time dropped every time, though by very different amounts: the saving depends heavily on how much of the model's time is spent normalising and on how each framework implements it.
The chart shows the paper's own timings.
TakeawayRMSNorm matched layer norm's quality in the paper's tests and was faster every time, by 7% to 64% depending on the model and software.
Check yourself
- Why does the speed-up vary so much between tasks?
- It depends on what fraction of total time normalisation takes (large in small recurrent models, small in a big transformer) and on how efficiently each framework implements the operations.
Idea · Later work
From a speed tweak to the LLM default
RMSNorm's arrival was quiet. It became prominent when large language models adopted it: T5 (2019) used a layer norm with no mean subtraction and no bias, which is essentially RMSNorm, and Meta's LLaMA (2023) explicitly cites this paper and applies RMSNorm to the input of every transformer sub-layer. Many open models since have followed.
At the scale of these models, every per-token operation is repeated trillions of times, so a cheaper normaliser with no loss in quality is an easy choice. Fewer parameters (no bias) and simpler numerics also help.
TakeawayDropping one step of layer norm cost nothing in quality, and the simpler normaliser became standard in LLMs.
Abstract, restated
Layer normalisation helps deep networks by stabilising training and speeding convergence through its re-centring and re-scaling invariances, but its computational overhead slows networks, notably RNNs. The paper hypothesises that re-centring invariance is dispensable and proposes root mean square layer normalisation (RMSNorm), which regularises the summed inputs to a neuron by their root mean square alone. RMSNorm is invariant to re-scaling and gives the model an implicit learning-rate adaptation, and is computationally simpler than LayerNorm. A partial variant, pRMSNorm, estimates the RMS from p% of the summed inputs. Across several tasks and architectures RMSNorm achieves performance comparable to LayerNorm while reducing running time by 7%–64%. NeurIPS 2019.
Section · §3–4 · Eqs. 1–4
From LayerNorm to RMSNorm
With summed inputs ai = Σj wijxj and yi = f(ai + bi) (Eq. 1), LayerNorm is
with gain g initialised to 1. RMSNorm drops μ:
When μ = 0, RMSNorm equals LayerNorm. Since RMS(a)² = σ² + μ², RMSNorm divides by a larger quantity whenever the mean is non-zero, and the residual mean passes through scaled rather than removed. The normalised vector lies on a sphere of radius √n. The authors note that the plain L2 norm (which differs from RMS by the factor √n) did not work as a layer normaliser in their experiments (it appears as "L2-Norm" in Table 2), and suggest the √n scaling matters for robustness across layer sizes. In practice an ε is added inside the root, and modern implementations usually omit the bias b.
Cost: LayerNorm needs the mean before it can compute the variance (two dependent reductions in a straightforward implementation, plus a subtraction per element); RMSNorm needs one reduction of squares.
Check yourself
- For a with mean μ and standard deviation σ, by what factor does RMSNorm's output differ from LayerNorm's for a mean-zero component?
- LayerNorm divides by σ, RMSNorm by √(σ² + μ²), so mean-zero components are shrunk by σ/√(σ² + μ²) relative to LayerNorm, and the mean itself survives as μ/√(σ² + μ²).
Section · §4.1–4.2 · Table 1 · Eqs. 5–10
Invariance and gradient analysis
RMS is linear in scale, RMS(αx) = αRMS(x) (Eq. 6), so scaling the weight matrix W → δW or the input x → δx leaves the output unchanged (Eq. 7). Unlike LayerNorm, RMSNorm is not invariant to re-centring. Table 1 of the paper extends LayerNorm's invariance table with RMSNorm; the figure recomputes it numerically.
Gradients (§4.2). The gain gradient ∂ℒ/∂g is proportional to the normalised summed inputs, so it is invariant to scaling of x and W (Eq. 8). For the weights, the gradient (Eq. 9) involves a matrix R that scales as 1/δ when x or W is scaled by δ (Eq. 10). So ∂ℒ/∂W is invariant to input scaling and inversely proportional to weight scale: large weights get small gradients, an implicit learning-rate adaptation of the same kind that BatchNorm §3.3 noted. The authors argue this, not re-centring, is the source of normalisation's stabilising effect.
Section · §5
Partial RMSNorm
If the summed inputs are roughly i.i.d., the RMS can be estimated from a subset. pRMSNorm uses the first k = ⌈n · p⌉ elements:
It shares RMSNorm's invariances. The paper uses p = 6.25% in its experiments (12.5% on CIFAR-10) and notes the estimate is biased and can make gradients unstable for small layers. In practice slicing overhead often cancelled the saving.
Section · §6 · Tables 2–10
Experiments
| Task (table) | Quality: LN → RMSNorm | Time: LN → RMSNorm |
|---|---|---|
| WMT14 En–De, RNNSearch, TensorFlow (Table 2); BLEU newstest2014 / 2017 | 22.6 / 23.6 → 22.4 / 23.7 | 665 → 501 s (24.7%) |
| Same, Theano (Table 3) | 22.3 / 23.8 → 22.5 / 23.2 | 988 → 652 s (34.0%) |
| Same, PyTorch (Table 3) | 23.2 / 24.3 → 22.9 / 24.5 | 857 → 763 s (11.0%) |
| Transformer base, TensorFlow (Table 4) | 26.6 / 27.7 → 26.8 / 27.7 | 248 → 231 s (6.9%) |
| CNN reading comprehension (Table 6, Fig. 5) | lower validation error than BatchNorm variants; similar to LN | 392 → 333 s (15.1%) |
| MS-COCO caption retrieval, OE model (Tables 7–8) | R@1 caption 47.9 → 48.7, image 38.4 → 39.0 | 12.02 → 7.12 s (40.8%; pRMSNorm 63.9%) |
| CIFAR-10, ConvPool-CNN-C (Table 10); test error | 10.49% → 8.83% (BatchNorm 8.25%) | 39 → 31 s/epoch (20.5%) |
Times are per 1k steps (translation), per 0.1k steps (reading comprehension and retrieval) or per epoch (CIFAR-10); percentages are speed-ups relative to LayerNorm as the paper reports them. Without normalisation the Transformer failed to train (BLEU 0, Table 4). Table 5 shows both LN and RMSNorm keep the decoder's recurrent activations at a stable scale, and Fig. 4 shows RMSNorm is more robust than LN when weights are initialised with a non-zero mean.
Note the CIFAR-10 row: for convolutional image models, BatchNorm (and WeightNorm, 8.28%) beat both LN-style methods, consistent with LayerNorm's own §6.7.
Section · §7 and later work
Limitations and lineage
- Speed-ups are implementation-dependent. The 7%–64% range spans frameworks, hardware and architectures, timed on single runs; the transformer's gain was the smallest (6.9%). Fused kernels narrow the gap.
- The hypothesis is empirical. The paper shows re-centring is dispensable on its tasks, not why in general. It also gives up an invariance, so a model with large shared offsets behaves differently under RMSNorm.
- pRMSNorm was not consistently faster and can be unstable on small layers; it is rarely used.
- Vision. On CIFAR-10, BatchNorm stayed better (Table 10).
Reference
Glossary
- RMS
- Root mean square: √(mean of squares). Equals the standard deviation when the mean is zero.
- Re-centring invariance
- Output unchanged when the same constant is added to all summed inputs.
- Re-scaling invariance
- Output unchanged when all summed inputs are multiplied by the same positive factor.
- Gain g
- Learned per-unit scale applied after normalisation.
- pRMSNorm
- RMSNorm estimated from the first p% of summed inputs.
- Pre-norm
- Normalising the input of each transformer sub-layer rather than its output.
- RNNSearch
- The attention-based recurrent translation model of Bahdanau et al.