The paper in four sentences
To keep networks from overfitting, people shrink every weight slightly towards zero at each step, either by adding a penalty on large weights to the loss ("L2 regularisation") or by directly multiplying the weights by a number just below 1 ("weight decay").
For plain gradient descent these two are exactly the same, which is why libraries implemented weight decay as an L2 penalty, including for Adam.
But Adam rescales every gradient by that weight's typical gradient size, so a penalty that travels through the gradient gets rescaled too: weights with large, busy gradients are barely regularised at all.
Taking the decay out of the gradient and applying it directly to the weights ("decoupled weight decay", giving AdamW) made Adam generalise better, made its two main settings independent of each other, and became the standard way large models are trained.
Idea · §1
Why shrink the weights at all?
A network with lots of adjustable numbers ("weights") can memorise its training examples instead of learning the general pattern. That is called overfitting: perfect on the practice questions, poor on the exam.
A simple, old remedy is to keep weights small. Large weights let a network build sharp, twitchy functions that chase every quirk of the training data; small weights force smoother functions. So at every training step you pull each weight a little towards zero, and let the data pull back only where it really matters.
There are two ways to write that pull down:
- L2 regularisation: add a penalty to the error, proportional to the sum of the squared weights. The gradient of that penalty is just "weight × a constant", so it shows up as an extra push towards zero inside the gradient.
- Weight decay: after each step, multiply every weight by a number just below 1, such as 0.9999.
For ordinary gradient descent you can do the algebra and these are identical, once you match the constants. Everyone therefore used the words interchangeably, and software implemented "weight decay" by adding the L2 term to the gradient.
TakeawayL2 regularisation and weight decay both pull weights towards zero, and for plain gradient descent they are the same thing.
Idea · §2 · Prop. 2
Adam bends the penalty
Adam gives each weight its own step size: it divides that weight's gradient by the typical size of its recent gradients. A weight that always sees big gradients gets its steps shrunk; a weight with tiny gradients gets them enlarged. That is what makes Adam work so well.
Now put the L2 penalty into the gradient. It too gets divided by the typical gradient size. For a weight with large, noisy gradients, the shrink-towards-zero signal is divided by a big number and almost vanishes. For a weight with small gradients it is amplified. The regulariser is no longer the same for every weight; it is weakest exactly on the weights the data is pushing around the most.
Below, two weights start at 1. The data gives them only noise, a gradient that flips between plus and minus every step and averages to zero, so the only steady force is the pull to zero. Weight A gets large noisy gradients, weight B small ones. Compare how fast each shrinks under each method.
TakeawayInside Adam, an L2 penalty is rescaled per weight, so weights with large gradients are barely regularised. Weight decay applied directly treats every weight the same.
Check yourself
- Why does the dashed blue line (weight B, small noise) still shrink reasonably under Adam with L2?
- Its gradient noise is small, so the L2 term is a sizeable part of its gradient, and after Adam's division the pull towards zero becomes a step of nearly α every time. It is the noisy weight A whose pull is divided by a big number and drowned out.
- What happens to Adam + L2 if you make weight A's noise tiny too?
- Then the L2 term dominates both gradients, Adam normalises it to a step of about α each time, and both weights shrink at a steady rate set by α, not by λ. The penalty strength has stopped mattering, which is the other face of the same problem.
Idea · §2 · Algorithm 2
The fix: decay the weights directly
The paper's change is one line of code. Keep Adam exactly as it is for the data's gradient, and afterwards, separately, shrink each weight by a small fixed fraction. The decay no longer passes through Adam's per-weight rescaling, so every weight is pulled towards zero at the same proportional rate. The authors call this decoupled weight decay; applied to Adam it is AdamW, and applied to SGD with momentum it is SGDW.
A second benefit is that the two knobs people tune stop interfering. With L2 inside the gradient, if you change the learning rate you also change how strong the regulariser effectively is, so the best weight-decay setting moves whenever the learning rate moves. Decoupled, the two settings can be tuned one at a time.
TakeawayDecoupled weight decay is one line of code, and it also makes the learning rate and the decay strength independently tunable.
Idea · §4 · Figs. 3–4
What it bought
At the time, a common belief was that Adam trains fast but generalises worse than plain SGD with momentum on image tasks. The authors argued that much of that gap came from the broken regulariser, not from Adam itself. On CIFAR-10 (small colour images) and a downsampled 32×32 ImageNet, using residual networks, AdamW reached clearly lower test error than Adam with L2, and with a learning-rate schedule that periodically restarts (AdamWR) it closed most of the gap to SGD.
The authors also noticed that the best decay strength depends on how long you train and how big the batches are, and suggested a rule to rescale it, so a setting found on a short run carries over to a longer one.
TakeawayFixing the regulariser made Adam competitive on image classification and gave the field the optimiser it still uses for large models.
Check yourself
- A colleague says "AdamW is just Adam with weight decay turned on". What is missing?
- Where the decay is applied. Adam with a weight-decay option usually adds an L2 term to the gradient, which is then rescaled per weight. AdamW subtracts the decay from the weights directly, outside the rescaling.
Abstract, restated
L2 regularisation and weight decay are equivalent for standard SGD (once the coefficient is rescaled by the learning rate) but not for adaptive gradient methods such as Adam. Common implementations of these methods use L2 and call it weight decay, which may be misleading. A simple modification recovers the original formulation by decoupling weight decay from the optimisation steps taken with respect to the loss. This (i) decouples the optimal weight-decay factor from the learning rate for both SGD and Adam, and (ii) substantially improves Adam's generalisation, letting it compete with SGD with momentum on image classification. Loshchilov & Hutter, ICLR 2019 (arXiv 1711.05101; first circulated in 2017 as "Fixing Weight Decay Regularization in Adam").
Section · §2 · Props. 1–2
When are L2 and weight decay the same?
Weight decay in Hanson & Pratt's (1988) sense is
whereas L2 regularisation minimises ftreg(θ) = ft(θ) + (λ′/2)‖θ‖22, whose gradient adds λ′θ.
Proposition 1. For SGD, the two are identical with λ′ = λ/α: α(∇f + (λ/α)θ) = α∇f + λθ. Note the dependence on α: if L2 is used to implement decay, the "right" λ′ shifts whenever α changes, which is the coupling visible as diagonal valleys in the paper's Fig. 2.
Proposition 2. Consider any optimiser whose step is −αMt∇ft with a preconditioner Mt ≠ kI. Running it on freg gives decay −αMtλ′θ, so no λ′ reproduces uniform decay λθ. For Adam, Mt = diag(1/(√v̂t + ε)) (with momentum on top), so coordinate i's effective decay rate is αλ′/(√v̂t,i + ε): weights with large historical gradient magnitude are regularised less than they would be by weight decay.
Proposition 3 (fixed preconditioner M = diag(s)−1) shows that decoupled decay on such an optimiser equals gradient steps on a scale-adjusted L2 penalty in which each coordinate's penalty is weighted by its preconditioner scale √si, so weights with large typical gradients are regularised relatively more than under plain L2. The authors flag that real adaptive methods have time-varying preconditioners, so this is an intuition rather than an exact statement.
Section · §2 · Algorithms 1–2
SGDW and AdamW
# Algorithm 2 (Adam with L2 vs AdamW). Defaults α=0.001, β1=0.9, β2=0.999, ε=1e-8, λ∈ℝ t ← t + 1 ∇ft ← SelectBatch(θ(t−1)) ; gt ← ∇ft(θ(t−1)) + λθ(t−1) # L2: decay inside the gradient mt ← β1 m(t−1) + (1 − β1) gt vt ← β2 v(t−1) + (1 − β2) gt² m̂t ← mt/(1 − β1^t) ; v̂t ← vt/(1 − β2^t) ηt ← SetScheduleMultiplier(t) θt ← θ(t−1) − ηt ( α m̂t/(√v̂t + ε) + λθ(t−1) ) # AdamW: decay outside it
Exactly one of the two highlighted terms is used. Two details matter. First, in the paper the decay is multiplied by the schedule multiplier ηt but not by α, so changing α leaves the decay strength alone; this is the decoupling of λ from α. Second, ηt scales both terms, so with a cosine schedule the decay anneals together with the step size. Algorithm 1 (SGDW) is analogous: θt ← θt−1 − mt − ηtλθt−1, with the decay kept out of the momentum buffer.
Implementation drift. PyTorch's torch.optim.AdamW applies θ ← θ(1 − lr·λ), where lr already includes α and the schedule. The decay is therefore proportional to the learning rate, so the "decoupled" λ is again tied to α (though still not rescaled per coordinate). Comparing λ values across codebases requires knowing which convention is used.
Section · §3
A Bayesian-filtering justification
§3 summarises Aitchison's (2018) view of adaptive gradient methods as Bayesian filtering: the optimiser tracks a Gaussian belief over the optimal parameters, the preconditioner plays the role of posterior uncertainty, and each gradient is an observation (μpost = μprior + Σpostg). In this view, weight decay belongs in the state transition, not the observation: assume the optimal θ drifts according to N((I − A)θt, Q). Taking A = λI multiplies the mean by (1 − λ) each step, which is decoupled weight decay, applied independently of the per-parameter uncertainty. L2, by contrast, enters as part of the observed gradient and is therefore scaled by Σpost. This is a motivation rather than a derivation of AdamW's exact update.
Section · §4 · App. B · Figs. 1–4
Experiments, schedules and normalised decay
Setup. 26 2×64d ResNet with Shake-Shake regularisation (11.6M parameters) on CIFAR-10, plus 26 2×96d (25.6M) for long runs, and ImageNet32×32 (1.2M images at 32×32); batch size 128; budgets 100 to 1800 epochs.
- Figs. 1–2 (§4.1–4.2). Test-error heatmaps over (learning rate, weight decay) for fixed, step-drop and cosine schedules. With L2, the good settings lie on a diagonal (coupled); with decoupled decay they are more separable. Cosine annealing gave the best results for both optimisers.
- Fig. 3 (§4.3). Over 1800 epochs with the 26 2×96d net, AdamW achieved lower training loss and lower test error than Adam, and better test error at the same training loss, i.e. better generalisation, not just faster optimisation. The paper reports a relative test-error improvement of roughly 15% over Adam with L2 (at Adam's default learning rate of 0.001).
- Fig. 4 (§4.4). Warm restarts (SGDR-style cosine cycles) give AdamWR, improving anytime performance, and reaching performance comparable to SGDWR on ImageNet32×32 while closing most of the gap on CIFAR-10.
Normalised weight decay (App. B.1). The best λ shrinks as the number of weight updates grows, so the authors propose
with batch size b, training-set size B and epochs T (per restart for AdamWR). λnorm is read as the decay to use if training for one pass with one batch. The authors note the square-root form comes from limited experiments. The figure below shows the cosine-with-restarts multiplier ηt (App. B.2) and the λ this rule gives.
Section · §5 and lineage
Limitations and later work
- Scope. All of the authors' own experiments are image classification with one architecture family; they say results must be verified on a wider range of tasks, and conjecture (untested) that similar results hold for AdaGrad and AMSGrad. §4.5 cites third-party results (EEG decoding, and the use of AdamW in transformer pre-training by Radford et al., 2018).
- The scaling rule for λ is heuristic. Later work treats AdamW's decay as setting an averaging timescale of roughly 1/(ηλ) updates and derives how λ should change with model and dataset size.
- Which parameters to decay. Common practice (not from this paper) excludes biases and normalisation gains from decay.
- Interaction with normalisation. For scale-invariant weights (followed by BatchNorm or LayerNorm), decay mainly controls the effective learning rate rather than model complexity (van Laarhoven, 2017; Zhang et al., 2019), a different reason it helps.
Check yourself
- Under the paper's AdamW update, does doubling α change the strength of weight decay?
- No: the decay term is ηtλθ, independent of α. (Under PyTorch's AdamW it would double, because decay is lr·λ.)
- For a coordinate with √v̂ = 100 and α = 10−3, what per-step decay rate does Adam + L2 with λ′ = 10−2 give?
- About αλ′/√v̂ = 10−7, a factor of 100 weaker than for a coordinate with √v̂ = 1.
Reference
Glossary
- L2 regularisation
- Adding (λ′/2)‖θ‖² to the loss; its gradient is λ′θ.
- Weight decay
- Multiplying weights by (1 − λ) each step, independent of the loss gradient.
- Decoupled
- Applied directly to θ, outside the optimiser's preconditioning and momentum.
- Preconditioner
- The matrix (here diagonal) that rescales the gradient; for Adam 1/(√v̂ + ε).
- ηt
- Schedule multiplier (e.g. cosine annealing) applied to both the step and the decay.
- SGDR / AdamWR
- Cosine annealing with warm restarts, applied to SGD(W) or AdamW.
- λnorm
- Normalised weight decay, rescaled by √(b/(BT)).