The paper in four sentences
Models that generate data usually have to choose between being flexible (able to capture complicated data) and being tractable (easy to train, sample from and evaluate). This paper tries to get both.
Start with real data and blur it with a tiny bit of random noise, over and over, until nothing but noise is left. That direction is easy and needs no learning.
The trick is that each single step is so small that undoing it is also simple: a bell-shaped guess. A neural network learns those small undo steps.
Chain the learned undo steps together, starting from pure noise, and structure reappears: new samples of the data. The idea lay mostly dormant until 2020, when DDPM made it produce excellent images.
Idea · §1
The generative modelling dilemma
A generative model is a program that has learned what a kind of data looks like, so it can make up new examples (a new face, a new handwritten digit) and say how likely a given example is. Formally it learns a probability distribution: a rule that assigns every possible image a plausibility score.
The paper opens with a tension. Tractable models, such as a single bell curve, are easy to fit and easy to evaluate, but far too simple for real images. Flexible models can describe almost anything, but computing their scores usually involves a sum over astronomically many possibilities, so training and sampling become slow or approximate.
The authors' goal is a model that is flexible and still tractable: you can train it, draw samples from it, and compute (or at least tightly bound) how likely the data is under it.
TakeawayThe paper wants a model of data that is rich enough for real images but still easy to train, sample and score.
Idea · §2.1 · Fig. 1
Destroy the data slowly with noise
Picture a drop of ink in a glass of water. Over time the molecules jiggle randomly and the drop spreads out until the water is evenly tinted. Physicists call this diffusion. Every trace of the drop's original shape is gone.
The paper does the same to data. Take a data point (say, a dot on a spiral) and at every step shrink it very slightly towards the centre and add a pinch of random noise. After enough steps, every dot ends up in a featureless round cloud, the familiar bell-shaped Gaussian distribution. This is the forward process. It is fixed in advance and involves no learning at all.
Drag the slider to watch a spiral (the paper's first toy example is a "Swiss roll" like this) dissolve over 40 steps, the number the paper uses for this toy.
TakeawayAdding a little noise many times turns any data into plain noise. This direction is easy and needs nothing learned.
Check yourself
- Why does the forward process need no training?
- It is a fixed recipe (shrink slightly, add a known amount of noise). Nothing about it depends on a model; only the reverse direction is learned.
- Where do all the points end up after many steps, whatever data you start with?
- In the same simple, round Gaussian cloud. That is what makes it a good starting point for generating.
Idea · §2.2
Small steps are easy to undo
Running the movie backwards, from noise to data, sounds hopeless: you can't un-stir ink. But the paper relies on an old mathematical fact (it cites Feller, 1949): if each forward step is tiny, then the reverse of that step has the same simple form. Given a slightly noisy point, the place it probably came from a moment ago is described by a single bell curve, nudged in some direction.
That turns an impossible problem into many easy ones. A neural network only needs to learn, for each step and each location, where the centre of that bell curve is and how wide it is.
Try it below. The data here is one-dimensional: values cluster around −1 and +1. We take a noisy value and ask "where was it a few steps earlier?". The answer is computed exactly. With a small jump it is one tidy bump (the dashed bell curve fits it). With a big jump it splits into two bumps, which a single bell curve cannot describe.
TakeawayEach tiny noise step can be reversed with a simple bell-curve guess, so the hard reverse journey becomes many easy, learnable steps.
Check yourself
- Why not reverse the whole process in one go?
- Over a large jump, the set of places a noisy point could have come from is complicated (here, two separate clusters). Only for small jumps is it a simple bell curve that a network can easily output.
- What does the network actually have to output at each step?
- The centre and the width of the bell curve for the previous step, given the current noisy point and the step number.
Idea · §2.2–2.4 · Figs. 1–2
Generate by walking back from noise
To make a new sample: draw a point from the plain noise cloud, then apply the learned undo step, then again, and again, all the way back to step 0. Each undo step removes a little noise and adds a little structure. After the last step, the point looks like data.
Training checks how well the chain of undo steps retraces the forward process. Because every piece is a simple bell curve, the paper can compute a guaranteed lower bound on how likely the data is under the model, and improve the network by pushing that bound up.
Below, the "network" is replaced by the best possible bell-curve step, computed exactly, so you can see the effect of the number of steps on its own. With few, big steps the samples come out in the wrong places. With many small steps they land on the two clusters.
TakeawayNew data comes from running the learned small undo steps from pure noise. More, smaller steps make each one simpler and the result better.
Check yourself
- With two steps, why do many samples land between the clusters?
- Each step can only move a sample by a bell-curve-shaped amount. With so few steps, the last step has to cover a large jump whose true answer has two bumps, and a single bell curve averages them.
- What does the model start from when it generates?
- Pure random noise from the simple Gaussian cloud that the forward process ends in.
Idea · §2.5, §3 · Figs. 3–5
Repairing images, and what happened next
Because the model is built from simple steps, it can be combined with other information. The paper uses this to clean up noisy images (CIFAR-10 photos with heavy added noise, Fig. 3) and to fill in a missing patch of a bark texture (Fig. 5), by steering each reverse step towards what is already known.
The authors tested the approach on toy data, handwritten digits (MNIST), small photos (CIFAR-10), and natural textures. The samples were recognisable but blurry by later standards, and the method attracted little attention for five years.
TakeawayThis paper set out the full diffusion recipe; five years of engineering later, the same recipe powers modern image generators.
Abstract, restated
The goal is a family of probabilistic models that is both highly flexible and tractable to learn, sample and evaluate. Inspired by non-equilibrium statistical physics, the method slowly destroys structure in a data distribution through an iterative forward diffusion process, then learns a reverse diffusion process that restores it. This yields deep generative models with thousands of layers or time steps whose likelihood can be bounded, and which allow conditional and posterior probabilities to be computed cheaply. A reference implementation was released. ICML 2015 (PMLR 37); arXiv:1503.03585.
Section · §2.1 · Eqs. 1–3 · Table App.1
Forward trajectory
Data distribution q(x(0)) is converted into an analytically tractable distribution π(y) by repeatedly applying a Markov diffusion kernel Tπ for which π is stationary:
Two kernels are used (Table App.1). For continuous data, a Gaussian kernel N(x(t); x(t−1)√(1 − βt), Iβt), whose stationary distribution is the standard normal. For binary data, a binomial kernel B(x(t); x(t−1)(1 − βt) + 0.5βt), whose stationary distribution is independent fair coins. βt is the diffusion rate of step t.
Composing Gaussian steps gives x(t) = √ᾱt x(0) + √(1 − ᾱt) ε with ᾱt = ∏s≤t(1 − βs). This closed form is not emphasised in the 2015 paper; it became central in DDPM, which writes it as its Eq. 4.
Section · §2.2 · Eqs. 4–5
Reverse trajectory and the small-step argument
The generative model is the same chain run backwards from the stationary distribution:
For continuous diffusion (the limit of small β), the reversal of the diffusion process has "the identical functional form as the forward process" (§2.2, citing Feller 1949). So if βt is small, q(x(t−1) | x(t)) is itself close to Gaussian (or binomial), and the model uses p(x(t−1) | x(t)) = N(x(t−1); fμ(x(t), t), fΣ(x(t), t)) for Gaussian diffusion, or B(x(t−1); fb(x(t), t)) for binomial. The functions f are neural networks (multi-layer perceptrons in §2.2; the image models use a multi-scale convolutional architecture, Appendix D). The cost of learning is then "the cost of these functions, times the number of time-steps".
The figure measures how wrong the Gaussian assumption is. For a 1-D two-cluster data distribution, it integrates the exact reverse kernel on a grid and computes its KL divergence from the moment-matched Gaussian (the best Gaussian in that KL direction), summed over all T steps and averaged over x(t). By the chain rule for KL this sum upper-bounds the gap between the bound K of §2.4 and the true log-likelihood for an ideal Gaussian-kernel model.
TakeawayThe whole method rests on one approximation, Gaussian reverse kernels, and that approximation becomes exact as the number of steps grows.
Section · §2.3–2.4 · Eqs. 9–14
Model probability and the training bound
The model probability p(x(0)) = ∫ dx(1…T) p(x(0…T)) is intractable directly, but can be written as an average over forward trajectories of the ratio p/q (§2.3). The paper notes this is the same move as annealed importance sampling and the Jarzynski equality (§1.2: "the Jarzynski equality … known in machine learning as Annealed Importance Sampling"): if forward and reverse trajectories coincide (a quasi-static process), a single sample from q evaluates p(x(0)) exactly.
Training maximises the log-likelihood L = ∫ q(x(0)) log p(x(0)). Jensen's inequality (Eq. 12) gives L ≥ K (Eq. 13), and with the derivation of Appendix B:
Every term is analytic for Gaussian kernels: the entropies are fixed by the schedule, and each KL is between two Gaussians, because conditioning the forward process on x(0) makes the posterior q(x(t−1) | x(t), x(0)) Gaussian. The bound is tight when the reverse kernels equal the true reverse of the forward process. Training is regression: find the fμ, fΣ maximising K (Eq. 15).
Schedule (§2.4.1). For Gaussian diffusion, β1 is fixed to a small constant (to prevent overfitting) and β2…T are learned by gradient ascent on K, holding the sampled noise fixed so the gradient is low-variance. For binomial diffusion the schedule is fixed: βt = (T − t + 1)−1, which removes a 1/T share of the original signal per step.
Entropy bounds (§2.6, App. A). The entropy of each reverse step is bounded above and below by quantities computable from the forward process, which constrains fΣ.
# Training, as in §2.4 (Gaussian case) repeat: x0 ~ data; sample a forward trajectory x1..xT with Eq. 2 for each t: compute the analytic Gaussian KL between q(x_{t-1} | x_t, x0) and N(f_mu(x_t,t), f_Sigma(x_t,t)) K = -(sum of KLs) + schedule-dependent entropy terms # Eq. 14 gradient ascent on K w.r.t. network weights (and beta_2..T)
Check yourself
- Why condition the posterior on x(0) inside K?
- The marginal reverse q(x(t−1) | x(t)) depends on the unknown data distribution, but conditioned on x(0) it is a Gaussian with closed-form mean and variance, so every KL term is analytic and low-variance.
- What links the method to annealed importance sampling?
- The likelihood is an importance-weighted average over forward trajectories of p/q; this is AIS (equivalently the Jarzynski equality), and it is exact with one sample when forward and reverse trajectories agree.
Section · §2.5 · Eqs. 16–23 · Figs. 3, 5
Multiplying distributions: posteriors, denoising, inpainting
Many tasks need the product of the model with another function, p̃(x(0)) ∝ p(x(0)) r(x(0)): a posterior given a noisy observation, or a constraint that some pixels are known. For most models this is hard. Here, the paper multiplies every intermediate distribution by a corresponding r(x(t)) (Eq. 16) and obtains a modified reverse kernel (Eq. 22). When r is smooth relative to the kernel, it acts as a small perturbation of each Gaussian step, computable in closed form (§2.5.3, App. C). The simplest choice is a constant r(x(t)) = r(x(0)) (Eq. 23).
Applications: denoising CIFAR-10 images corrupted with Gaussian noise of variance 1 (Fig. 3), and inpainting a 100 × 100 region of a bark texture replaced by noise (Fig. 5). This "steer each reverse step with extra information" idea is the ancestor of classifier guidance in Diffusion Models Beat GANs and of later inpainting and editing methods.
Section · §3 · Tables 1–2 · App. D
Experiments
Toy problems (§3.1). A 2-D Swiss roll with T = 40 and a radial-basis-function network with 16 hidden units (Fig. 1), and binary "heartbeat" sequences (a periodic pulse) with binomial diffusion, T = 2000 and an MLP with three hidden layers of 50 units (Fig. 2). On the heartbeat, the bound K = −2.414 bits per sequence against a true log-likelihood of −2.322.
Images (§3.2). MNIST, CIFAR-10, dead-leaf images and bark textures, using T = 1000 (500 for bark). Image networks are multi-scale convolutional (mean-pooling to several scales, convolution, upsampling and summing), with a dense pathway for MNIST and CIFAR-10; the mean is parameterised as a perturbation of the forward kernel and the covariance is diagonal (App. D.2.1). Training used SFO, except RMSprop for CIFAR-10.
| Dataset (Table 1) | Bound K | K − Lnull |
|---|---|---|
| Swiss roll | 2.35 bits | 6.45 bits |
| Binary heartbeat | −2.414 bits/seq | 12.024 bits/seq |
| Bark | −0.55 bits/pixel | 1.5 bits/pixel |
| Dead leaves | 1.489 bits/pixel | 3.536 bits/pixel |
| CIFAR-10 | 5.4 ± 0.2 bits/pixel | 11.5 ± 0.2 bits/pixel |
All datasets except the heartbeat were scaled to unit variance, so these are not the bits-per-dimension numbers later papers report for 8-bit images. A footnote records that an earlier version reported higher CIFAR-10 bounds because of pixel quantisation, fixed by adding uniform noise. Table 2 compares with prior work: on dead leaves, the bound of 1.489 bits/pixel beats the MCGSM's 1.244; on MNIST, Parzen-window log-likelihood estimates are 317 ± 2.7 bits for diffusion against 309 ± 1.6 for a deep GSN and 325 ± 2.9 for an adversarial net. Parzen-window estimates were soon shown to be unreliable (Theis, van den Oord and Bethge, 2016), so that comparison carries little weight.
Section · §4 and lineage
Limitations and lineage
- Sample quality. Image samples (CIFAR-10, MNIST) were recognisable but well behind contemporary GANs; the paper's strength was tractable likelihood bounds and posterior computation, not visual quality.
- Many steps. Accuracy of the Gaussian reverse kernel needs small β, hence hundreds to thousands of network evaluations per sample. Reducing this became a research field of its own (DDIM, distillation, consistency models, flow matching).
- What the network predicts. Predicting the reverse mean directly, plus learning a covariance and the schedule, made optimisation harder than it needed to be.
Check yourself
- Which ingredient of the 2015 model did DDPM change most consequentially?
- The parameterisation and loss: predicting the added noise with an unweighted squared error, rather than predicting the reverse mean and covariance under the full bound with a learned schedule.
Reference
Glossary
- Forward trajectory
- The fixed Markov chain q that turns data into noise (Eq. 3).
- Reverse trajectory
- The learned Markov chain p from noise to data (Eq. 5).
- βt
- Diffusion rate (noise variance) of step t.
- Stationary distribution π
- The distribution the kernel leaves unchanged; the end point of the forward process and starting point of generation.
- K
- The variational lower bound on log-likelihood (Eq. 14), maximised during training.
- AIS
- Annealed importance sampling: estimating a normaliser by averaging importance weights along a chain of intermediate distributions; the Jarzynski equality in physics.
- Moment matching
- Fitting a Gaussian with the same mean and variance as a target; it minimises KL(target ‖ Gaussian).
- Inpainting
- Filling in a missing region of an image consistently with the rest.