Study guide · Machine learning & training · 2014

Adam: A Method for Stochastic Optimization

Diederik Kingma, Jimmy Ba · ICLR 2015

How big a step should each of a network's millions of weights take when it learns? Adam's answer is to let every weight set its own step size, using a running average of its recent gradients and a running average of their squares. The recipe is a few lines long, needs almost no tuning, and became the default way to train neural networks for a decade, even though its original convergence proof turned out to be wrong.

The paper in four sentences

Training a network means nudging millions of numbers ("weights") downhill on an error landscape, and plain gradient descent uses one step size for all of them, which is a poor fit when some weights get huge, noisy signals and others get tiny, rare ones.

Adam keeps two running averages for every weight: the average recent slope (which way to go) and the average recent squared slope (how big and how noisy the slopes usually are).

Each step moves the weight by the first average divided by the square root of the second, so every weight takes a step of roughly the same, sensible size, no matter how large its raw gradients are.

A small correction fixes the fact that both averages start at zero, and with default settings that rarely need changing it trained logistic regression, multilayer networks and convolutional networks as fast as or faster than the alternatives.

Idea · §1

One step size does not fit all

A neural network learns by gradient descent. For every weight, the gradient says which direction would reduce the error and how steeply. You move each weight a little in that direction, multiplied by a single number called the learning rate, and repeat millions of times.

In practice the gradient is computed on a small random batch of examples rather than the whole dataset, so it is noisy: a wobbly estimate of the true slope. That is why this is called stochastic (random) optimisation, the word in the paper's title.

The trouble is that different weights live in very different worlds. Some get large gradients on every batch. Others, say the weight for a rare word, get a useful signal only once in a while. A learning rate small enough to keep the first group stable leaves the second group crawling. A learning rate big enough for the second makes the first explode.

Everyday pictureImagine one volume knob that controls every instrument in an orchestra at once. The drums are deafening and the flute is inaudible. What you want is a separate knob per instrument that adjusts itself to how loud that instrument has recently been.

TakeawayGradient descent with one shared step size struggles when weights see gradients of very different sizes or noise levels.

Idea · §2 · Algorithm 1

Two running averages per weight

Adam stands for adaptive moment estimation. A "moment" is a statistician's word for a kind of average. Adam keeps two for each weight:

  • The first moment, m: a running average of recent gradients. Noise in different batches points in different directions and partly cancels, so m tracks the consistent direction. This is the same idea as momentum, a ball that keeps rolling the way it has been going.
  • The second moment, v: a running average of recent squared gradients. Squaring removes the sign, so v measures how big the gradients typically are, whether they agree or not.

"Running average" here means an exponential moving average: each step, keep 90% of the old value and mix in 10% of the new gradient (for m), or keep 99.9% and mix in 0.1% of the new squared gradient (for v). Old information fades gradually instead of being forgotten all at once.

The step is then m divided by the square root of v, times a base step size α (0.001 by default). Dividing by √v undoes the weight's typical gradient size, so a weight with huge gradients and a weight with tiny ones both move by roughly α per step.

A toy error landscape: a long valley that is 200 times steeper across (up and down) than along it (left to right). The minimum is the cross. All three start top left. Plain gradient descent (blue) uses the learning rate directly; momentum (dashed) uses it with momentum 0.9; Adam (amber) uses it as α with the default β values. Plain gradient descent creeps along the flat direction, and above a learning rate of 0.1 it bounces out of the valley because the steep direction cannot tolerate the step. Momentum helps a lot until about 0.19, where it too blows up. Adam's per-direction scaling keeps its step near α in both directions, so it never diverges; at large α it just jitters around the minimum. Illustrative; not from the paper.

TakeawayAdam averages the gradient to find a steady direction, and divides by the typical gradient size so every weight gets a step of about the same length.

Check yourself
A weight's gradients alternate +5, −5, +5, −5. Roughly what are m and v?
m stays near zero, because the signs cancel; v settles near 25. So the step m/√v is tiny: Adam slows down when the gradient has no consistent direction.
Why does dividing by √v rather than v give a step of about α?
v is in squared units. Its square root is in the same units as the gradient, so m/√v is a pure number near ±1 when gradients are consistent.

Idea · §2.1

A step size you can reason about

Because of that division, the size of each step barely depends on how large the gradients are. If you multiplied every gradient by 1,000, both the top (m) and the bottom (√v) would grow 1,000 times and the step would not change at all. The paper points out that this makes α easy to interpret: it is roughly the most a weight will move in one step, a kind of "trust region" you can set in advance from knowing what sensible weight values look like.

The ratio m/√v also behaves like a signal-to-noise ratio. When recent gradients agree, the ratio is near 1 and Adam takes a full step. When they disagree, which tends to happen near the bottom of the valley where the noise dominates, the ratio shrinks and Adam automatically takes smaller steps. The paper calls this a form of automatic annealing (gradually cooling down).

Everyday pictureA hiker in fog asks ten passers-by which way the village is. If all ten point the same way, she strides confidently. If they point in all directions, she shuffles. The length of her stride depends on how much the answers agree, not on how loudly people shout.

TakeawayAdam's step is roughly α when the gradient is consistent and smaller when it is noisy, and it does not care about the overall scale of the gradients.

Idea · §3 · Fig. 4

Fixing the slow start

Both running averages start at zero. After one step, m is only 10% of the first gradient and v only 0.1% of its square. They are biased towards zero, and badly so for the slow-moving v, which takes thousands of steps to warm up.

That bias does not cancel. On the very first step, the top is 0.1 × the gradient and the bottom is √0.001 ≈ 0.032 × the gradient, so the step comes out about three times too large. The paper's fix, called bias correction, divides each average by exactly how much of it has "filled up" so far. After one step that is 10% for m and 0.1% for v; after many steps the correction fades to nothing.

Try it below. The gradient is a steady 1 with some noise. Watch the uncorrected step overshoot early on.

Each bar is the size of Adam's step, in units of α, over the first 60 steps for a weight whose gradient is 1 plus noise. The dashed line at 1 is the step you would expect for a consistent gradient. Toggle correction off and the early steps balloon, worst when β₂ is close to 1. The paper's Fig. 4 shows the same effect causing unstable training of a variational autoencoder.

TakeawayAverages that start at zero are biased; Adam divides them by how full they are, which matters most for the slow second average.

Check yourself
Why is the first uncorrected step too big rather than too small, when both averages are too small?
The second average is shrunk far more (to 0.1%) than the first (to 10%), and it sits under a square root in the denominator. 0.1 / √0.001 ≈ 3.16, so the step is about three times too large.
Does bias correction still matter after 100,000 steps?
No. The correction factors 1 − βt are essentially 1 by then. It only changes the early steps.

Idea · §4, §6

Fast in practice, flawed in theory

The paper tested Adam on logistic regression for handwritten digits and movie reviews, on multilayer networks with and without dropout, and on convolutional networks for small images. Adam was as fast as or faster than the best alternative in each case, and it did this with the same default settings that rarely needed changing. That robustness is why it spread so widely.

The paper also included a mathematical guarantee that, for a simplified class of problems (convex ones, where the landscape is a single bowl), Adam would get as close to the best answer as the best methods known. In 2018 Reddi, Kale and Kumar showed that this proof contained an error, and built a simple bowl-shaped problem on which Adam goes to the worst possible answer. The cause: Adam's averages can forget a rare but large gradient, so its effective step size can grow again when it should be shrinking.

Later refinementReddi et al.'s fix, AMSGrad, keeps the largest second average seen so far, so step sizes never grow. In practice, though, the bigger change came from AdamW (Loshchilov and Hutter), which fixed how Adam interacts with weight decay. AdamW is what most large language models, including Chinchilla, are trained with.

TakeawayAdam's practical success outlived its proof: it works very well on real networks, but it is not guaranteed to converge even on simple problems.

Check yourself
In one sentence, why can Adam fail on the Reddi counterexample?
A large, informative gradient that appears only rarely is quickly forgotten by the running average of squares, so the many small misleading gradients in between get relatively large steps and win.

Follow the idea

Led to: AdamW (2017)