Study guide · Neural networks · 2013

Auto-Encoding Variational Bayes

Diederik Kingma, Max Welling · ICLR 2014

Can a neural network learn to describe each example with a few hidden "causes" and also generate brand-new examples from them, when the maths of working backwards from an example to its causes is impossible to do exactly? Kingma and Welling show that it can: pair a decoder with an encoder that guesses the causes, and rewrite the random guessing step so that ordinary backpropagation can train both together.

The paper in four sentences

Imagine every picture of a face is produced from a short list of hidden settings (pose, smile, lighting). A decoder network turns settings into a picture; we would like to learn it, and also be able to make new faces.

To learn, we need to know which settings probably produced each real picture. That is exactly what we cannot calculate, so a second network, the encoder, learns to guess a fuzzy range of settings for each picture.

The guess involves random sampling, and you cannot normally pass learning signals back through a dice roll. The reparameterisation trick moves the dice roll to the side, so the sampling becomes "fixed noise, shifted and stretched", which backpropagation handles easily.

Training balances two pulls: make the decoded picture match the original, and keep the guessed settings close to a simple standard bell curve, so that random settings later decode into sensible new pictures.

Idea · §2.1 · Fig. 1

Hidden causes behind the data

A generative model is a recipe for producing data. This paper uses a two-step recipe. First pick a small set of hidden numbers, called the latent variables or the code and written z, at random from a simple distribution. Then feed them through a network that turns them into a data point x, such as an image.

If the recipe is good, it can do three useful things: produce new, realistic examples; describe any real example compactly by its code; and say how plausible a given example is.

The hard part is learning the recipe from data. For each real image we would like to know which codes could have produced it (statisticians call this the posterior). When the decoder is a neural network, there is no formula for that. Earlier methods either needed simplifying assumptions or ran a slow sampling procedure separately for every single data point, which is hopeless for large datasets.

Everyday pictureA sound engineer hears a finished song and wants to know the mixing-desk settings that produced it. Turning the knobs forward to make a song is easy. Working backwards from the song to the knobs is the hard direction, and that is the direction learning needs.

TakeawayA latent-variable model makes data from a few hidden numbers. Going forward is easy; inferring the hidden numbers from data is the hard step that blocked training.

Check yourself
Which direction is hard: from code to image, or from image to code?
From image to code. The decoder computes image from code directly, but no formula inverts a neural network into a probability distribution over codes.

Idea · §2.1–2.2, §3

An encoder that guesses, a decoder that rebuilds

The paper's answer is to train a second network to do the hard direction approximately. This encoder looks at an example and outputs a guess about its code. The guess is not a single point but a fuzzy region: a centre (the mean, μ) and a spread (the standard deviation, σ) for each latent number. The paper calls this a recognition model, and notes that when it is a neural network, the whole system is a variational auto-encoder, or VAE.

Training then looks like an auto-encoder: an example goes in, the encoder proposes a fuzzy code, we pick one code from that fuzzy region, the decoder rebuilds the example from it, and we score how well it matches. Because the encoder is shared across all examples, guessing a new example's code costs one quick pass instead of a slow per-example search.

The score being maximised has a precise meaning: it is a guaranteed lower bound on how probable the model finds the real data (more on that in the In depth tab). Pushing the bound up improves both the decoder and the encoder's guesses.

image x encoder μ (centre)σ (spread) z pick one code decoder rebuiltimage training: rebuild the image well, and keep (μ, σ) near a bell curve generating: skip the encoder, draw z from the bell curve, decode
The VAE as an auto-encoder with a fuzzy middle. Only the decoder is needed to make new examples.

TakeawayThe encoder learns to do the hard direction approximately, giving a centre and spread for each example's code, and the decoder learns to rebuild the example from a code picked there.

Check yourself
Why output a spread, not just a single best code?
The true answer is a range of plausible codes, not one. Representing the range is what makes the training score a bound on the data's probability, and it forces nearby codes to decode to similar examples.
What does sharing one encoder across all examples buy?
Speed: a new example's code is guessed in a single forward pass rather than by a separate optimisation or sampling run for that example.

Idea · §2.3–2.4 · Eq. 4

The reparameterisation trick

Here is the snag. In the middle of the pipeline we pick a random code from the encoder's fuzzy region. Backpropagation works by asking "if I nudged this number a little, how would the final score change?" But a random draw has no such answer: nudge μ and you just get a different, unrelated random number.

The trick is to draw the randomness first, from a fixed bell curve that has nothing to learn, and then compute the code from it: code = μ + σ × noise. The randomness is the same; what changes is that the code is now an ordinary calculation from μ and σ. Nudge μ and every code slides by exactly that amount. Stretch σ and they spread out smoothly. Backpropagation can follow that.

Try it below. In "fresh draws" mode, moving the sliders re-rolls the dice and the dots jump about at random. In "reparameterised" mode the noise is drawn once and kept, so the dots glide. Then press Learn to let gradient descent move μ and σ so the dots land in the target zone.

Top row: 24 noise values drawn from a standard bell curve. Bottom row: the codes μ + σ × noise. Lines show which noise value made which code. The amber band is a target region; the score is the average squared distance of the codes from its centre (2.0). Learn takes gradient steps on μ and σ through the formula, which is only possible in reparameterised mode.
Everyday pictureA dart player's throws scatter randomly. You cannot coach "the randomness", but you can coach the aim (where the centre of the scatter is) and the steadiness (how wide it is). Writing each throw as "aim plus steadiness times a random wobble" separates the coachable parts from the luck.

TakeawayWrite the random code as "centre plus spread times fixed noise". The randomness stays, but the code becomes a smooth function of what the network outputs, so backpropagation can train through it.

Check yourself
After the trick, which quantity is random and which are learned?
The noise is random and has no learnable parts; μ and σ are produced by the encoder and are learned.
If the score improves when codes move right, what does backprop tell the encoder?
Increase μ: each code moves one-for-one with μ, so the derivative of the score with respect to μ is just the average derivative with respect to the codes.

Idea · Eq. 3, Eq. 10

A tug of war: rebuild well, but keep codes tidy

The training score has two parts. The reconstruction part rewards rebuilding each example accurately from a code picked in its fuzzy region. The regulariser part (a "KL divergence", a measure of how different two distributions are) penalises fuzzy regions that stray from a standard bell curve centred on zero with spread one.

Each pull alone goes wrong. With only reconstruction, the encoder shrinks every spread to almost nothing and pushes codes far apart: a plain auto-encoder, whose space has gaps that decode to nonsense. With too much regulariser, every example gets the same bell-curve code and the decoder learns to output the average example regardless. The balance gives codes that are distinct enough to rebuild from and packed together enough that a random draw from the bell curve lands somewhere meaningful.

The toy below trains a real (tiny) VAE in your browser on eight one-dimensional data points, using the reparameterisation trick. The slider changes the weight on the regulariser. The paper uses weight 1; turning it up or down is a later variation.

Top: the eight data points (circles), their reconstructions (ticks), and any generated points (triangles). Bottom: each point's fuzzy code (bell shapes, matching colours) against the standard bell curve (dashed). Try weight 0 (spreads shrink to slivers and nothing ties the codes to the bell curve), 1, and 10 (every code collapses onto the bell curve and every reconstruction becomes the average). Press Train after each change. Illustrative toy with a straight-line decoder and fixed decoder noise; the collapse threshold depends on that noise level.

TakeawayReconstruction keeps codes informative; the regulariser keeps them packed near a standard bell curve so that random codes decode to sensible new examples.

Check yourself
Why does generation fail if the regulariser weight is zero?
Codes can sit anywhere with tiny spreads, so most of the standard bell curve covers regions no real example uses, and those codes decode to things unlike the data.
What does the model do when the regulariser dominates?
It ignores the input: every code distribution equals the bell curve, and the decoder outputs the average example. This is later called posterior collapse.

Idea · §5 · Figs. 2–5

What it showed, and where it led

The experiments are modest by today's standards: handwritten digits (MNIST) and a set of face images (Frey Face). The new method reached a better score faster than the main older alternative, the wake-sleep algorithm, in every setting tested, and it worked on the full MNIST training set where a sampling-based method was too slow. With only two latent numbers, the learned code space could be laid out as a map, and walking across it morphed one digit or facial expression smoothly into another.

The bigger legacy is the trick itself. "Sample noise, then transform it differentiably" became a standard tool across machine learning, and the encoder-decoder-with-fuzzy-codes design became a basic building block.

Later refinementVAE images tend to look blurry next to those from GANs or diffusion models. VAEs survived as components: latent diffusion (the Stable Diffusion family) first compresses images with an auto-encoder that is lightly regularised in this KL style, then runs diffusion in that smaller code space. The same reparameterisation idea was published independently by Rezende, Mohamed and Wierstra in 2014.

TakeawayThe VAE made deep latent-variable models trainable at scale with plain backpropagation; its sampling trick outlived its image quality.

Follow the idea

Led to: Latent diffusion (2021)