Study guide · Machine learning & training · 2017

Decoupled Weight Decay Regularization

Ilya Loshchilov, Frank Hutter · ICLR 2019

Two common ways of stopping a network's weights from growing too large, "L2 regularisation" and "weight decay", were treated as the same thing. This paper shows they are the same only for plain gradient descent: inside Adam they behave differently, and the usual implementation quietly weakens the regulariser for exactly the weights that most need it. Applying the decay directly to the weights, outside Adam's machinery, fixes it and narrows Adam's gap with SGD on image classification.

The paper in four sentences

To keep networks from overfitting, people shrink every weight slightly towards zero at each step, either by adding a penalty on large weights to the loss ("L2 regularisation") or by directly multiplying the weights by a number just below 1 ("weight decay").

For plain gradient descent these two are exactly the same, which is why libraries implemented weight decay as an L2 penalty, including for Adam.

But Adam rescales every gradient by that weight's typical gradient size, so a penalty that travels through the gradient gets rescaled too: weights with large, busy gradients are barely regularised at all.

Taking the decay out of the gradient and applying it directly to the weights ("decoupled weight decay", giving AdamW) made Adam generalise better, made its two main settings independent of each other, and became the standard way large models are trained.

Idea · §1

Why shrink the weights at all?

A network with lots of adjustable numbers ("weights") can memorise its training examples instead of learning the general pattern. That is called overfitting: perfect on the practice questions, poor on the exam.

A simple, old remedy is to keep weights small. Large weights let a network build sharp, twitchy functions that chase every quirk of the training data; small weights force smoother functions. So at every training step you pull each weight a little towards zero, and let the data pull back only where it really matters.

There are two ways to write that pull down:

  • L2 regularisation: add a penalty to the error, proportional to the sum of the squared weights. The gradient of that penalty is just "weight × a constant", so it shows up as an extra push towards zero inside the gradient.
  • Weight decay: after each step, multiply every weight by a number just below 1, such as 0.9999.

For ordinary gradient descent you can do the algebra and these are identical, once you match the constants. Everyone therefore used the words interchangeably, and software implemented "weight decay" by adding the L2 term to the gradient.

Everyday pictureA garden where every plant is trimmed back a little each week, so only the ones that keep getting strong sunlight grow big. You can either give the gardener a rule ("trim 1% off everything") or bake the trimming into the plant food. With plain plant food the effect is the same.

TakeawayL2 regularisation and weight decay both pull weights towards zero, and for plain gradient descent they are the same thing.

Idea · §2 · Prop. 2

Adam bends the penalty

Adam gives each weight its own step size: it divides that weight's gradient by the typical size of its recent gradients. A weight that always sees big gradients gets its steps shrunk; a weight with tiny gradients gets them enlarged. That is what makes Adam work so well.

Now put the L2 penalty into the gradient. It too gets divided by the typical gradient size. For a weight with large, noisy gradients, the shrink-towards-zero signal is divided by a big number and almost vanishes. For a weight with small gradients it is amplified. The regulariser is no longer the same for every weight; it is weakest exactly on the weights the data is pushing around the most.

Below, two weights start at 1. The data gives them only noise, a gradient that flips between plus and minus every step and averages to zero, so the only steady force is the pull to zero. Weight A gets large noisy gradients, weight B small ones. Compare how fast each shrinks under each method.

Solid lines: decoupled weight decay (subtract λ × weight directly). Dashed: L2 penalty added to the gradient. Amber: weight A (noise level set by the slider). Blue: weight B (noise 0.01). The noise is a gradient of ±noise level, alternating each step. 400 steps, λ = 0.01 per step; Adam uses α = 0.01, and plain SGD uses step 0.01 with the L2 strength set to λ/0.01 so the two methods match. Under SGD the dashed and solid lines coincide exactly. Under Adam the dashed amber line barely moves: a busy weight is hardly regularised. Illustrative toy, same seed every time.

TakeawayInside Adam, an L2 penalty is rescaled per weight, so weights with large gradients are barely regularised. Weight decay applied directly treats every weight the same.

Check yourself
Why does the dashed blue line (weight B, small noise) still shrink reasonably under Adam with L2?
Its gradient noise is small, so the L2 term is a sizeable part of its gradient, and after Adam's division the pull towards zero becomes a step of nearly α every time. It is the noisy weight A whose pull is divided by a big number and drowned out.
What happens to Adam + L2 if you make weight A's noise tiny too?
Then the L2 term dominates both gradients, Adam normalises it to a step of about α each time, and both weights shrink at a steady rate set by α, not by λ. The penalty strength has stopped mattering, which is the other face of the same problem.

Idea · §2 · Algorithm 2

The fix: decay the weights directly

The paper's change is one line of code. Keep Adam exactly as it is for the data's gradient, and afterwards, separately, shrink each weight by a small fixed fraction. The decay no longer passes through Adam's per-weight rescaling, so every weight is pulled towards zero at the same proportional rate. The authors call this decoupled weight decay; applied to Adam it is AdamW, and applied to SGD with momentum it is SGDW.

A second benefit is that the two knobs people tune stop interfering. With L2 inside the gradient, if you change the learning rate you also change how strong the regulariser effectively is, so the best weight-decay setting moves whenever the learning rate moves. Decoupled, the two settings can be tuned one at a time.

A cartoon of the paper's Figs. 1–2, not real data. Each cell is a training run with one learning rate (across) and one weight-decay setting (up); darker means lower test error. In this toy, the error depends on how far the learning rate is from its best value and how far the effective decay is from its best value. With L2 the effective decay is learning rate × setting, so the good region is a diagonal stripe: change one knob and you must re-tune the other. Decoupled, the good region is a compact blob you can find one axis at a time. Click any cell to see why.

TakeawayDecoupled weight decay is one line of code, and it also makes the learning rate and the decay strength independently tunable.

Idea · §4 · Figs. 3–4

What it bought

At the time, a common belief was that Adam trains fast but generalises worse than plain SGD with momentum on image tasks. The authors argued that much of that gap came from the broken regulariser, not from Adam itself. On CIFAR-10 (small colour images) and a downsampled 32×32 ImageNet, using residual networks, AdamW reached clearly lower test error than Adam with L2, and with a learning-rate schedule that periodically restarts (AdamWR) it closed most of the gap to SGD.

The authors also noticed that the best decay strength depends on how long you train and how big the batches are, and suggested a rule to rescale it, so a setting found on a short run carries over to a longer one.

Later refinementAdamW became the standard optimiser for transformers: GPT-style and BERT-style models and Chinchilla all use decoupled weight decay, and Chinchilla's authors report AdamW beat Adam in their language-model runs. The main lasting tweak is how the decay is tied to the learning rate: popular libraries multiply it by the learning rate, which partly re-couples the two knobs the paper set out to separate.

TakeawayFixing the regulariser made Adam competitive on image classification and gave the field the optimiser it still uses for large models.

Check yourself
A colleague says "AdamW is just Adam with weight decay turned on". What is missing?
Where the decay is applied. Adam with a weight-decay option usually adds an L2 term to the gradient, which is then rescaled per weight. AdamW subtracts the decay from the weights directly, outside the rescaling.

Follow the idea

Built on: Adam (2014)