The paper in four sentences
Training a network means nudging millions of numbers ("weights") downhill on an error landscape, and plain gradient descent uses one step size for all of them, which is a poor fit when some weights get huge, noisy signals and others get tiny, rare ones.
Adam keeps two running averages for every weight: the average recent slope (which way to go) and the average recent squared slope (how big and how noisy the slopes usually are).
Each step moves the weight by the first average divided by the square root of the second, so every weight takes a step of roughly the same, sensible size, no matter how large its raw gradients are.
A small correction fixes the fact that both averages start at zero, and with default settings that rarely need changing it trained logistic regression, multilayer networks and convolutional networks as fast as or faster than the alternatives.
Idea · §1
One step size does not fit all
A neural network learns by gradient descent. For every weight, the gradient says which direction would reduce the error and how steeply. You move each weight a little in that direction, multiplied by a single number called the learning rate, and repeat millions of times.
In practice the gradient is computed on a small random batch of examples rather than the whole dataset, so it is noisy: a wobbly estimate of the true slope. That is why this is called stochastic (random) optimisation, the word in the paper's title.
The trouble is that different weights live in very different worlds. Some get large gradients on every batch. Others, say the weight for a rare word, get a useful signal only once in a while. A learning rate small enough to keep the first group stable leaves the second group crawling. A learning rate big enough for the second makes the first explode.
TakeawayGradient descent with one shared step size struggles when weights see gradients of very different sizes or noise levels.
Idea · §2 · Algorithm 1
Two running averages per weight
Adam stands for adaptive moment estimation. A "moment" is a statistician's word for a kind of average. Adam keeps two for each weight:
- The first moment, m: a running average of recent gradients. Noise in different batches points in different directions and partly cancels, so m tracks the consistent direction. This is the same idea as momentum, a ball that keeps rolling the way it has been going.
- The second moment, v: a running average of recent squared gradients. Squaring removes the sign, so v measures how big the gradients typically are, whether they agree or not.
"Running average" here means an exponential moving average: each step, keep 90% of the old value and mix in 10% of the new gradient (for m), or keep 99.9% and mix in 0.1% of the new squared gradient (for v). Old information fades gradually instead of being forgotten all at once.
The step is then m divided by the square root of v, times a base step size α (0.001 by default). Dividing by √v undoes the weight's typical gradient size, so a weight with huge gradients and a weight with tiny ones both move by roughly α per step.
TakeawayAdam averages the gradient to find a steady direction, and divides by the typical gradient size so every weight gets a step of about the same length.
Check yourself
- A weight's gradients alternate +5, −5, +5, −5. Roughly what are m and v?
- m stays near zero, because the signs cancel; v settles near 25. So the step m/√v is tiny: Adam slows down when the gradient has no consistent direction.
- Why does dividing by √v rather than v give a step of about α?
- v is in squared units. Its square root is in the same units as the gradient, so m/√v is a pure number near ±1 when gradients are consistent.
Idea · §2.1
A step size you can reason about
Because of that division, the size of each step barely depends on how large the gradients are. If you multiplied every gradient by 1,000, both the top (m) and the bottom (√v) would grow 1,000 times and the step would not change at all. The paper points out that this makes α easy to interpret: it is roughly the most a weight will move in one step, a kind of "trust region" you can set in advance from knowing what sensible weight values look like.
The ratio m/√v also behaves like a signal-to-noise ratio. When recent gradients agree, the ratio is near 1 and Adam takes a full step. When they disagree, which tends to happen near the bottom of the valley where the noise dominates, the ratio shrinks and Adam automatically takes smaller steps. The paper calls this a form of automatic annealing (gradually cooling down).
TakeawayAdam's step is roughly α when the gradient is consistent and smaller when it is noisy, and it does not care about the overall scale of the gradients.
Idea · §3 · Fig. 4
Fixing the slow start
Both running averages start at zero. After one step, m is only 10% of the first gradient and v only 0.1% of its square. They are biased towards zero, and badly so for the slow-moving v, which takes thousands of steps to warm up.
That bias does not cancel. On the very first step, the top is 0.1 × the gradient and the bottom is √0.001 ≈ 0.032 × the gradient, so the step comes out about three times too large. The paper's fix, called bias correction, divides each average by exactly how much of it has "filled up" so far. After one step that is 10% for m and 0.1% for v; after many steps the correction fades to nothing.
Try it below. The gradient is a steady 1 with some noise. Watch the uncorrected step overshoot early on.
TakeawayAverages that start at zero are biased; Adam divides them by how full they are, which matters most for the slow second average.
Check yourself
- Why is the first uncorrected step too big rather than too small, when both averages are too small?
- The second average is shrunk far more (to 0.1%) than the first (to 10%), and it sits under a square root in the denominator. 0.1 / √0.001 ≈ 3.16, so the step is about three times too large.
- Does bias correction still matter after 100,000 steps?
- No. The correction factors 1 − βt are essentially 1 by then. It only changes the early steps.
Idea · §4, §6
Fast in practice, flawed in theory
The paper tested Adam on logistic regression for handwritten digits and movie reviews, on multilayer networks with and without dropout, and on convolutional networks for small images. Adam was as fast as or faster than the best alternative in each case, and it did this with the same default settings that rarely needed changing. That robustness is why it spread so widely.
The paper also included a mathematical guarantee that, for a simplified class of problems (convex ones, where the landscape is a single bowl), Adam would get as close to the best answer as the best methods known. In 2018 Reddi, Kale and Kumar showed that this proof contained an error, and built a simple bowl-shaped problem on which Adam goes to the worst possible answer. The cause: Adam's averages can forget a rare but large gradient, so its effective step size can grow again when it should be shrinking.
TakeawayAdam's practical success outlived its proof: it works very well on real networks, but it is not guaranteed to converge even on simple problems.
Check yourself
- In one sentence, why can Adam fail on the Reddi counterexample?
- A large, informative gradient that appears only rarely is quickly forgotten by the running average of squares, so the many small misleading gradients in between get relatively large steps and win.
Abstract, restated
Adam is a first-order method for stochastic objectives based on adaptive estimates of lower-order moments of the gradient. It is simple, computationally efficient, has small memory requirements, is invariant to diagonal rescaling of the gradients, and suits problems that are large in data or parameters, non-stationary, or have very noisy or sparse gradients. Hyperparameters have intuitive interpretations and typically need little tuning. The paper relates Adam to AdaGrad and RMSProp, gives a regret bound in the online convex setting, reports experiments, and introduces AdaMax, an infinity-norm variant. Kingma & Ba, ICLR 2015 (arXiv 1412.6980, final revision v9, 2017).
Section · §2 · Algorithm 1
The algorithm
Given step size α, decay rates β1, β2 ∈ [0, 1), a stochastic objective f(θ) with realisations f1, …, fT, and initial θ0. All operations are element-wise.
# defaults: α = 0.001, β1 = 0.9, β2 = 0.999, ε = 1e-8 m0 ← 0; v0 ← 0; t ← 0 while θ not converged: t ← t + 1 gt ← ∇θ ft(θ(t−1)) mt ← β1·m(t−1) + (1 − β1)·gt # biased first moment vt ← β2·v(t−1) + (1 − β2)·gt² # biased second raw moment m̂t ← mt / (1 − β1^t) # bias-corrected v̂t ← vt / (1 − β2^t) θt ← θ(t−1) − α·m̂t / (√v̂t + ε)
The paper notes the last three lines can be folded into αt = α·√(1 − β2t)/(1 − β1t) and θt ← θt−1 − αt mt/(√vt + ε̂), which is how most libraries implement it (with ε moved slightly). Memory cost is two extra vectors the size of θ.
Step through it below for one scalar parameter. Choose a gradient pattern, then scale every gradient by c to see the invariance of §2.1.
Section · §2.1
Effective step size and the signal-to-noise ratio
With ε = 0, the step is Δt = α·m̂t/√v̂t. The paper gives two upper bounds:
The first case only arises under severe sparsity, when a gradient has been zero for all but the current step. With the defaults, (1 − β1) = 0.1 exceeds √(1 − β2) ≈ 0.032, so the first bound is the operative one: an isolated gradient spike after a long run of zeros can produce a step of up to about 0.1/0.032 ≈ 3.16α once both bias corrections have faded (large t). In the common case |𝔼[g]/√𝔼[g2]| ≤ 1 gives |Δt| ≲ α, so α sets an approximate trust region in parameter space.
The paper calls m̂t/√v̂t the signal-to-noise ratio (SNR). As the optimum is approached, the SNR typically falls and steps shrink: a built-in annealing. Invariance: replacing g by c·g scales m̂ by c and √v̂ by |c|, leaving Δ unchanged. More generally Adam is invariant to any diagonal rescaling of the gradient, since the normalisation is per coordinate.
Section 5 relates the preconditioner 1/√v̂ to a diagonal approximation of the Fisher information, but with the square root it is more conservative than natural-gradient preconditioning (which would use 1/v̂).
Check yourself
- Why is Adam not invariant to a full (non-diagonal) linear change of coordinates, as Newton's method is?
- Its preconditioner is diagonal: it rescales each coordinate independently from that coordinate's own gradient history. A rotation mixes coordinates, changing each coordinate's statistics, and Adam's diagonal estimate cannot undo it.
Section · §3, §5
Bias correction, and Adam's relatives
Unrolling the second-moment recursion with v0 = 0:
where ζ = 0 if the second moment is stationary, and is small otherwise because β2 gives little weight to the distant past. Dividing by (1 − β2t) removes the bias; the same argument gives the correction for m. The practical case is β2 close to 1, required for robust estimates with sparse gradients, which is exactly when the uncorrected bias is largest and lasts longest. §6.4 (Fig. 4) trains a variational autoencoder (one 500-unit softplus hidden layer, 50-dimensional latent) over β1 ∈ [0, 0.9], β2 ∈ {0.99, 0.999, 0.9999} and log10α ∈ [−5, −1]: without correction, β2 near 1 leads to instability, especially early in training, and the best results came from small 1 − β2 with correction.
RMSProp (Tieleman & Hinton, 2012) is Adam without bias correction and with momentum applied to the rescaled gradient rather than the gradient. AdaGrad (Duchi et al., 2011) divides by the root of the sum of all past squared gradients. The paper shows that Adam with β1 = 0, (1 − β2) infinitesimal and αt = α/√t reduces to AdaGrad, and that this correspondence needs bias correction. Adam thus aims to combine AdaGrad's strength on sparse gradients with RMSProp's on non-stationary objectives.
Section · §4 · Reddi et al. (2018)
The regret bound, and why it fails
§4 analyses Adam in Zinkevich's online convex optimisation framework, with regret R(T) = Σt [ft(θt) − ft(θ*)]. Theorem 4.1 assumes bounded gradients (‖∇ft‖2 ≤ G, ‖∇ft‖∞ ≤ G∞), bounded distances between iterates, β12/√β2 < 1, αt = α/√t and a decaying β1,t = β1λt−1, and claims R(T) = O(√T), with a data-dependent bound that can be much better than O(√(dT)) for sparse gradients. Corollary 4.2: R(T)/T = O(1/√T).
Reddi, Kale & Kumar ("On the Convergence of Adam and Beyond", ICLR 2018) identified the error. Define
the change in the inverse effective learning rate. The proof implicitly assumes Γt ⪰ 0 (step sizes never increase). That holds for SGD and AdaGrad but not for exponential moving averages: vt can fall, so the effective step can grow. Their Theorem 1 gives an explicit counterexample on ℱ = [−1, 1]: ft(x) = Cx when t mod 3 = 1 and −x otherwise, with C > 2. The average gradient is (C − 2)/3 > 0, so the optimum is x = −1, yet with β1 = 0 and β2 = 1/(1 + C2) Adam converges to x = +1. Theorem 2 extends this to any β1 < √β2, and Theorem 3 gives a stochastic version. Their fix, AMSGrad, uses v̂t = max(v̂t−1, vt), which forces Γt ⪰ 0.
Section · §6 · Figs. 1–3 · §7
Experiments and extensions
Hyperparameters for every method were chosen from a dense grid and the best result reported.
| Setting | Model | Finding |
|---|---|---|
| §6.1, Fig. 1 left | L2-regularised multiclass logistic regression, MNIST, minibatch 128, αt = α/√t | Adam converges like SGD with Nesterov momentum, both faster than AdaGrad. |
| §6.1, Fig. 1 right | Logistic regression on IMDB bag-of-words (10,000 features, 50% input dropout) | Sparse features: AdaGrad beats SGD+Nesterov by a large margin; Adam matches AdaGrad. |
| §6.2, Fig. 2 | MLP, two hidden layers of 1000 ReLUs, MNIST, minibatch 128 | With dropout, Adam beats AdaGrad, RMSProp, SGD+Nesterov and AdaDelta; SFO fails to converge. Deterministic loss: Adam beats SFO in iterations and wall clock (SFO is 5–10× slower per iteration). |
| §6.3, Fig. 3 | CNN c64-c64-c128-1000 on CIFAR-10, whitened input, dropout | Adam and SGD both much faster than AdaGrad over 45 epochs; Adam marginally better than SGD+momentum. The authors observe v̂ vanishes to ε within a few epochs here, so the first moment does most of the work. |
AdaMax (§7.1, Algorithm 2). Generalise the L2 norm in v to Lp and let p → ∞. The recursion becomes
which needs no bias correction for u and has the simpler bound |Δt| ≤ α; suggested defaults α = 0.002, β1 = 0.9, β2 = 0.999. Temporal averaging (§7.2): keep an exponential moving average of θ with its own bias correction, a Polyak–Ruppert-style trick to reduce noise in the final iterate (now standard as "EMA weights").
Section · Lineage
Limitations and what came next
- Convergence. The §4 proof is wrong (Reddi et al., 2018); see above. Non-convex guarantees, which are what matter for deep nets, were not attempted.
- Generalisation. Wilson et al. (2017, "The marginal value of adaptive gradient methods") found adaptive methods often generalised worse than well-tuned SGD on image classification. Loshchilov & Hutter traced much of this to how L2 regularisation interacts with Adam's preconditioner, giving AdamW, now the default for large models.
- ε and the denominator. ε is not just a numerical guard: in the CIFAR-10 experiment the paper itself notes v̂ falls to the scale of ε, where Adam behaves more like momentum SGD. Later implementations expose ε as a tunable hyperparameter.
- Memory. Two extra state vectors per parameter, a real cost at billions of parameters. Adafactor (Shazeer & Stern, 2018) factorises v to save memory, and was used for the largest models in Kaplan et al.'s scaling-law study. 8-bit optimiser states (Dettmers et al., 2022) shrink it further.
- Large-batch and stability variants. LAMB (You et al., 2019) adds per-layer trust ratios; Lion (Chen et al., 2023) uses only the sign of a momentum; β2 = 0.95 rather than 0.999 became common for large language models for stability.
TakeawayAdam's core (per-coordinate normalisation by an EMA of squared gradients, with bias correction) survives almost untouched; what changed is weight decay, β2 and how the theory is justified.
Check yourself
- AMSGrad is provably convergent on the counterexample. Why didn't it replace Adam?
- On practical deep learning problems it gave little or no benefit, and the counterexample requires a very small β2; with β2 = 0.999 the failure mode rarely bites.
- Which part of Algorithm 1 makes Adam invariant to rescaling the loss by a constant?
- The ratio m̂/√v̂: the numerator scales with the constant and the denominator with its absolute value.
Reference
Glossary
- First moment m
- Exponential moving average of gradients (momentum).
- Second raw moment v
- Exponential moving average of element-wise squared gradients (uncentred variance).
- β1, β2
- Decay rates of the two averages; their effective memory is about 1/(1 − β) steps (10 and 1000 by default).
- Bias correction
- Division by 1 − βt to undo the zero initialisation of the averages.
- SNR
- The paper's name for m̂/√v̂.
- Regret
- Cumulative excess loss of the online iterates over the best fixed parameter in hindsight.
- AdaGrad / RMSProp
- Earlier adaptive methods dividing by the root of the sum / EMA of squared gradients.
- AMSGrad
- Reddi et al.'s variant using the running maximum of v.
- AdaMax
- The paper's infinity-norm variant.