The paper in four sentences
Imagine every picture of a face is produced from a short list of hidden settings (pose, smile, lighting). A decoder network turns settings into a picture; we would like to learn it, and also be able to make new faces.
To learn, we need to know which settings probably produced each real picture. That is exactly what we cannot calculate, so a second network, the encoder, learns to guess a fuzzy range of settings for each picture.
The guess involves random sampling, and you cannot normally pass learning signals back through a dice roll. The reparameterisation trick moves the dice roll to the side, so the sampling becomes "fixed noise, shifted and stretched", which backpropagation handles easily.
Training balances two pulls: make the decoded picture match the original, and keep the guessed settings close to a simple standard bell curve, so that random settings later decode into sensible new pictures.
Idea · §2.1 · Fig. 1
Hidden causes behind the data
A generative model is a recipe for producing data. This paper uses a two-step recipe. First pick a small set of hidden numbers, called the latent variables or the code and written z, at random from a simple distribution. Then feed them through a network that turns them into a data point x, such as an image.
If the recipe is good, it can do three useful things: produce new, realistic examples; describe any real example compactly by its code; and say how plausible a given example is.
The hard part is learning the recipe from data. For each real image we would like to know which codes could have produced it (statisticians call this the posterior). When the decoder is a neural network, there is no formula for that. Earlier methods either needed simplifying assumptions or ran a slow sampling procedure separately for every single data point, which is hopeless for large datasets.
TakeawayA latent-variable model makes data from a few hidden numbers. Going forward is easy; inferring the hidden numbers from data is the hard step that blocked training.
Check yourself
- Which direction is hard: from code to image, or from image to code?
- From image to code. The decoder computes image from code directly, but no formula inverts a neural network into a probability distribution over codes.
Idea · §2.1–2.2, §3
An encoder that guesses, a decoder that rebuilds
The paper's answer is to train a second network to do the hard direction approximately. This encoder looks at an example and outputs a guess about its code. The guess is not a single point but a fuzzy region: a centre (the mean, μ) and a spread (the standard deviation, σ) for each latent number. The paper calls this a recognition model, and notes that when it is a neural network, the whole system is a variational auto-encoder, or VAE.
Training then looks like an auto-encoder: an example goes in, the encoder proposes a fuzzy code, we pick one code from that fuzzy region, the decoder rebuilds the example from it, and we score how well it matches. Because the encoder is shared across all examples, guessing a new example's code costs one quick pass instead of a slow per-example search.
The score being maximised has a precise meaning: it is a guaranteed lower bound on how probable the model finds the real data (more on that in the In depth tab). Pushing the bound up improves both the decoder and the encoder's guesses.
TakeawayThe encoder learns to do the hard direction approximately, giving a centre and spread for each example's code, and the decoder learns to rebuild the example from a code picked there.
Check yourself
- Why output a spread, not just a single best code?
- The true answer is a range of plausible codes, not one. Representing the range is what makes the training score a bound on the data's probability, and it forces nearby codes to decode to similar examples.
- What does sharing one encoder across all examples buy?
- Speed: a new example's code is guessed in a single forward pass rather than by a separate optimisation or sampling run for that example.
Idea · §2.3–2.4 · Eq. 4
The reparameterisation trick
Here is the snag. In the middle of the pipeline we pick a random code from the encoder's fuzzy region. Backpropagation works by asking "if I nudged this number a little, how would the final score change?" But a random draw has no such answer: nudge μ and you just get a different, unrelated random number.
The trick is to draw the randomness first, from a fixed bell curve that has nothing to learn, and then compute the code from it: code = μ + σ × noise. The randomness is the same; what changes is that the code is now an ordinary calculation from μ and σ. Nudge μ and every code slides by exactly that amount. Stretch σ and they spread out smoothly. Backpropagation can follow that.
Try it below. In "fresh draws" mode, moving the sliders re-rolls the dice and the dots jump about at random. In "reparameterised" mode the noise is drawn once and kept, so the dots glide. Then press Learn to let gradient descent move μ and σ so the dots land in the target zone.
TakeawayWrite the random code as "centre plus spread times fixed noise". The randomness stays, but the code becomes a smooth function of what the network outputs, so backpropagation can train through it.
Check yourself
- After the trick, which quantity is random and which are learned?
- The noise is random and has no learnable parts; μ and σ are produced by the encoder and are learned.
- If the score improves when codes move right, what does backprop tell the encoder?
- Increase μ: each code moves one-for-one with μ, so the derivative of the score with respect to μ is just the average derivative with respect to the codes.
Idea · Eq. 3, Eq. 10
A tug of war: rebuild well, but keep codes tidy
The training score has two parts. The reconstruction part rewards rebuilding each example accurately from a code picked in its fuzzy region. The regulariser part (a "KL divergence", a measure of how different two distributions are) penalises fuzzy regions that stray from a standard bell curve centred on zero with spread one.
Each pull alone goes wrong. With only reconstruction, the encoder shrinks every spread to almost nothing and pushes codes far apart: a plain auto-encoder, whose space has gaps that decode to nonsense. With too much regulariser, every example gets the same bell-curve code and the decoder learns to output the average example regardless. The balance gives codes that are distinct enough to rebuild from and packed together enough that a random draw from the bell curve lands somewhere meaningful.
The toy below trains a real (tiny) VAE in your browser on eight one-dimensional data points, using the reparameterisation trick. The slider changes the weight on the regulariser. The paper uses weight 1; turning it up or down is a later variation.
TakeawayReconstruction keeps codes informative; the regulariser keeps them packed near a standard bell curve so that random codes decode to sensible new examples.
Check yourself
- Why does generation fail if the regulariser weight is zero?
- Codes can sit anywhere with tiny spreads, so most of the standard bell curve covers regions no real example uses, and those codes decode to things unlike the data.
- What does the model do when the regulariser dominates?
- It ignores the input: every code distribution equals the bell curve, and the decoder outputs the average example. This is later called posterior collapse.
Idea · §5 · Figs. 2–5
What it showed, and where it led
The experiments are modest by today's standards: handwritten digits (MNIST) and a set of face images (Frey Face). The new method reached a better score faster than the main older alternative, the wake-sleep algorithm, in every setting tested, and it worked on the full MNIST training set where a sampling-based method was too slow. With only two latent numbers, the learned code space could be laid out as a map, and walking across it morphed one digit or facial expression smoothly into another.
The bigger legacy is the trick itself. "Sample noise, then transform it differentiably" became a standard tool across machine learning, and the encoder-decoder-with-fuzzy-codes design became a basic building block.
TakeawayThe VAE made deep latent-variable models trainable at scale with plain backpropagation; its sampling trick outlived its image quality.
Abstract, restated
How can one perform efficient approximate inference and learning in directed probabilistic models whose continuous latent variables have intractable posteriors, on large datasets? The paper introduces a stochastic variational inference and learning algorithm that scales to large data and works under mild differentiability conditions. First, a reparameterisation of the variational lower bound yields an estimator that can be optimised with standard stochastic gradient methods (SGVB). Second, for i.i.d. data with per-datapoint latents, posterior inference is made efficient by fitting an approximate inference (recognition) model to the intractable posterior with that estimator (AEVB). arXiv December 2013; ICLR 2014.
Section · §2.1–2.2 · Eqs. 1–3
Setting and the variational bound
Data X = {x(i)}i=1Nz(i) ∼ pθ*(z) and then x(i) ∼ pθ*(x|z). The paper explicitly allows the case where the marginal pθ(x) = ∫ pθ(z)pθ(x|z) dz and the posterior pθ(z|x) are intractable (so EM and closed-form mean-field VB do not apply) and the dataset is too large for per-datapoint sampling loops. It targets three problems: ML or MAP estimation of θ, approximate posterior inference of z, and approximate marginal inference of x.
A recognition model qφ(z|x), the probabilistic encoder, approximates the posterior; pθ(x|z) is the probabilistic decoder. Unlike mean-field, q need not factorise and its parameters are not computed from a closed-form expectation; φ is learned jointly with θ. The marginal log-likelihood decomposes per datapoint as
Since the first KL in (1) is non-negative, ℒ is a lower bound (the ELBO), tight when q equals the true posterior. Maximising ℒ over θ and φ jointly both raises the likelihood and pulls q toward the posterior. The gradient in θ is easy to estimate by sampling; the gradient in φ is the problem, because φ parameterises the distribution the expectation is taken under. The naive Monte Carlo (score-function) estimator, Eq[f(z) ∇φ log qφ(z)], is unbiased but, the paper notes, has very high variance and is impractical for this purpose.
Check yourself
- Why does maximising ℒ over φ make q a better posterior approximation?
- log pθ(x) does not depend on φ, so by Eq. 1 raising ℒ in φ must lower KL(q ‖ p(z|x)).
- In Eq. 3, which term acts like an auto-encoder's reconstruction error?
- The expected log-likelihood Eq[log pθ(x|z)]; the KL to the prior acts as a regulariser.
Section · §2.3 · Eqs. 4–8 · Algorithm 1
SGVB estimators and the AEVB algorithm
Assume z̃ ∼ qφ(z|x) can be written as a differentiable transform of auxiliary noise:
Applying (5) to (2) gives the generic estimator; applying it only to the reconstruction term of (3), when the KL is analytic, gives a second estimator that typically has lower variance; and a minibatch of M points from N gives the dataset-level estimator:
Algorithm 1 (AEVB, minibatch version) θ, φ ← initialise repeat XM ← random minibatch of M datapoints ε ← L samples per datapoint from p(ε) g ← ∇θ,φ ℒ̃M(θ, φ; XM, ε) gradient of Eq. 8, by backprop θ, φ ← update with g SGD or Adagrad until convergence The paper uses M = 100 and L = 1.
The point of the reparameterisation is that ∇φ now passes through f ∘ gφ by the chain rule, using the derivative of f, instead of weighting f by the score ∇ log q. The figure compares the two unbiased estimators of the same gradient, d/dμ Ez∼𝒩(μ, σ²)[(z − 2)²] = 2(μ − 2).
Check yourself
- Why does the paper get away with L = 1?
- The reparameterised estimator has low variance, and with a minibatch of M = 100 the gradient already averages over 100 independent samples.
- What does ℒ̃B require that ℒ̃A does not?
- A closed-form KL between qφ(z|x) and the prior, which holds for Gaussian q and Gaussian prior.
Section · §2.4
When can you reparameterise?
For the univariate Gaussian z ∼ 𝒩(μ, σ²), z = μ + σε with ε ∼ 𝒩(0, 1), so E𝒩(z; μ, σ²)[f(z)] ≃ (1/L) Σl f(μ + σε(l)). The paper lists three general routes to a suitable gφ and p(ε):
| Route | Construction | Examples given in §2.4 |
|---|---|---|
| Tractable inverse CDF | ε ∼ 𝒰(0, 1), g = inverse CDF of q | Exponential, Cauchy, Logistic, Rayleigh, Pareto, Weibull, Reciprocal, Gompertz, Gumbel, Erlang |
| Location–scale family | standard member as ε, g = location + scale · ε | Laplace, Elliptical, Student's t, Logistic, Uniform, Triangular, Gaussian |
| Composition | express the variable as a transform of auxiliary variables | Log-Normal (exponentiated normal), Gamma, Dirichlet, Beta, Chi-squared, F |
If none apply, good approximations to the inverse CDF exist with cost comparable to the density. The conspicuous gap is discrete latents: there is no differentiable g for a categorical variable, which is why wake-sleep (which handles discrete latents) remained relevant and why continuous relaxations were introduced later.
Section · §3 · Eqs. 9–10 · App. B–C
The variational auto-encoder
The worked example: prior pθ(z) = 𝒩(z; 0, I) with no parameters; decoder pθ(x|z) a multivariate Gaussian (real-valued data) or Bernoulli (binary data) whose parameters are output by an MLP with one hidden layer (Appendix C); encoder a diagonal Gaussian whose mean and log-variance are output by an MLP:
The first sum is −KL(q ‖ p) in closed form (derived in Appendix B), so this is estimator ℒ̃B. A footnote stresses that the diagonal covariance is a simplifying choice, not a limitation of the method. Check the closed form against Monte Carlo below.
Check yourself
- What KL does the encoder pay for μ = 0, σ = 1, and what does that mean?
- Zero. The code carries no information about x; if every dimension sits there, the model has collapsed to ignoring its input.
Section · §4–5 · Figs. 2–5
Related work and experiments
Related work (§4). The wake-sleep algorithm (Hinton et al., 1995) is described as the only other online method in the literature applicable to the same general class of continuous latent-variable models; it also uses a recognition model but optimises two objectives that together do not correspond to a bound on the marginal likelihood. The authors relate the KL term to regularised auto-encoders (denoising, contractive, sparse) and note a linear-Gaussian special case connects to PCA. An independent paper (Rezende, Mohamed and Wierstra) connected auto-encoders, directed models and the same reparameterisation.
Setup (§5). Encoder and decoder each have one hidden layer: 500 hidden units on MNIST, 200 on Frey Face (smaller, to limit overfitting); Frey Face decoder means are passed through a sigmoid into (0, 1). Parameters are initialised from 𝒩(0, 0.01); optimisation is minibatch SGVB with Adagrad, step size chosen from {0.01, 0.02, 0.1}; M = 100, L = 1. A weight-decay term corresponding to a prior p(θ) = 𝒩(0, I) makes the parameter estimate approximately MAP.
| Experiment | What was compared | Reported outcome |
|---|---|---|
| Lower bound vs training time (Fig. 2) | AEVB vs wake-sleep; latent sizes Nz = 3, 5, 10, 20, 200 (MNIST) and 2, 5, 10, 20 (Frey Face) | AEVB converged considerably faster and reached a better bound in all experiments; extra latent dimensions did not cause overfitting, attributed to the regularising KL. |
| Estimated marginal likelihood (Fig. 3) | AEVB, wake-sleep, Monte Carlo EM with HMC; 3 latents, 100 hidden units; Ntrain = 1,000 and 50,000 | AEVB compared favourably with both; MCEM is not an online algorithm and could not be run efficiently on the full MNIST set. |
| Visualisation (Figs. 4–5, App. A) | 2-D latent manifolds; random samples for several latent sizes | Smooth manifolds of digits and facial expressions. |
Section · §6–7 and later work
Limitations and lineage
- Continuous latents only. The trick needs a differentiable gφ. Discrete latents later got continuous relaxations (Gumbel-softmax / Concrete, Jang et al. and Maddison et al., 2017) or a straight-through quantiser (VQ-VAE, van den Oord et al., 2017).
- The bound's slack. A diagonal Gaussian q leaves a gap in Eq. 1. Importance-weighted bounds (IWAE, Burda et al.) tighten it; normalising flows (Rezende & Mohamed, 2015) and hierarchical latents (e.g. NVAE, 2020) make q and the prior richer.
- Posterior collapse. With a powerful decoder (e.g. an autoregressive text model), the optimum can set q = prior and ignore z. Bowman et al. (2016) observed this for sentences and used KL annealing.
- Sample quality. Gaussian-likelihood VAEs produce blurry samples relative to GANs and later diffusion models. β-VAE (Higgins et al., 2017) deliberately up-weights the KL to encourage disentangled codes, trading reconstruction.
The authors' future-work list (§7) included hierarchical models with convolutional encoders and decoders, time-series models, SGVB for global parameters, and supervised latent-variable models; all were pursued.
Check yourself
- Why can't the Gaussian reparameterisation be used directly for a categorical latent?
- A categorical sample is a piecewise-constant function of the parameters, so its derivative is zero almost everywhere; there is no useful differentiable path from φ to the sample.
Reference
Glossary
- Latent z
- Unobserved variable (the code) from which data are generated.
- Posterior
- pθ(z|x): distribution of codes given an observation; intractable here.
- Recognition model
- qφ(z|x), the encoder; an amortised approximation to the posterior.
- ELBO / ℒ
- Evidence lower bound, Eqs. 2–3; lower bound on log pθ(x).
- KL divergence
- Eq[log q − log p] ≥ 0; zero iff the distributions are equal.
- SGVB
- Stochastic Gradient Variational Bayes: the reparameterised Monte Carlo estimators of ℒ (Eqs. 6–8).
- AEVB
- Auto-Encoding Variational Bayes: Algorithm 1, joint SGD on θ and φ with a recognition model.
- Reparameterisation
- Writing z = gφ(ε, x) with parameter-free noise ε, so gradients pass through the sample.
- Score-function estimator
- E[f ∇ log q]; unbiased, high variance (also called REINFORCE).
- Posterior collapse
- q(z|x) ≈ p(z) for all x: the decoder ignores the code.