The paper in four sentences
Take a picture and add random noise to it in 1,000 small steps until it is pure static. That part is fixed and needs no learning.
Train a neural network on one simple question: "here is a noisy picture and how noisy it is; what noise was added?"
To make a new picture, start from pure static and use the network 1,000 times, each time removing a little of the noise it thinks is there.
This is the 2015 diffusion model with better choices, and it was the first time diffusion images rivalled the best GANs (on CIFAR-10 it edged past them).
Idea · §1 · Fig. 2
Generating by removing noise
Before this paper, the best image generators were GANs: two networks in a contest, one forging images and one spotting fakes. They made sharp pictures but were notoriously fiddly to train. A 2015 paper had proposed a calmer alternative, the diffusion model, but its pictures were blurry.
The diffusion idea runs a film in two directions. Forward: blur a real picture with a little random noise, many times, until nothing is left but static. Backward: learn to undo one small step of that blurring. Chain the backward steps together from fresh static, and a new picture emerges.
Ho, Jain and Abbeel kept that idea and changed the details: what the network predicts, how the noise is scheduled, and how training is scored. Those choices turned out to matter enormously.
TakeawayDDPM generates images by starting from static and repeatedly removing a little predicted noise.
Idea · §2 · Eq. 4
Adding noise, and a shortcut
The forward process has 1,000 steps. At each one, the picture is faded very slightly and a pinch of random noise is sprinkled on. The amount of noise per step grows slowly from tiny (0.0001) to small (0.02), a "linear schedule".
A handy fact: you don't need to run the steps one by one. Because adding random noise to random noise just gives more random noise, the picture at any step t is simply "a faded copy of the original plus one dose of noise", with the exact fade and dose known in advance. That makes training fast: pick any step, jump straight there.
TakeawayThe noisy picture at any step is a known mix of the original and fresh noise, so training can jump to any step instantly.
Check yourself
- Why is the shortcut useful for training?
- Each training example can be made at a random step in one calculation, instead of simulating hundreds of noise steps first.
- The noise multiplier overtakes the picture multiplier at about step 260. Why is the ring still recognisable well after that?
- The picture's pattern is spread consistently over many pixels while the noise is independent in each pixel, so your eye averages the noise away. Recognition fails only when the picture multiplier gets much smaller (in the figure, somewhere around steps 600 to 800).
Idea · §3.2, §3.4 · Algorithm 1
Training: guess the noise
The key choice in this paper is what the network should predict. Instead of asking "what did the slightly cleaner picture look like?", it asks "which noise was added?". Knowing the noise is equivalent to knowing the clean picture, since one is the noisy picture minus the other, but the noise version trains much better.
Training is then almost embarrassingly simple: take a real picture, pick a random step, add the right amount of random noise, show the network the noisy picture and the step number, and penalise the squared difference between its guess and the real noise. Repeat millions of times.
The figure shows what the best possible guess looks like when the "training set" is just four tiny pictures. At low noise the guess points to exactly the right picture. At high noise, the best guess is a blend of all four: the network honestly can't tell yet.
TakeawayThe network learns one skill: look at a noisy picture and say what noise is in it. Training is just a squared-error penalty on that guess.
Check yourself
- If the network knows the noise exactly, what else does it know?
- The clean picture: subtract the (scaled) noise from the noisy picture and undo the fading.
- Why is a blended guess correct at high noise, rather than a mistake?
- With lots of noise, several originals are plausible. Squared-error training rewards the average of the plausible answers, so blending is the best possible guess.
Idea · §3.2 · Algorithm 2
Sampling: 1,000 small clean-ups
To make a new picture, start from pure random static. Ask the network what noise is there, remove a small part of it, then (except at the very last step) sprinkle a little fresh noise back in. Do this 1,000 times, counting down from step 1,000 to step 1.
Sprinkling noise back in seems odd, but it keeps the process honest: each intermediate picture should look like a genuine noisy picture at that step, which is what the network was trained on. Watch below: the rough layout is decided early, and the fine detail last.
TakeawayGeneration is the training skill used in a loop: guess the noise, remove some of it, repeat from static to picture.
Check yourself
- Which features of the picture get decided first?
- The coarse layout, at high noise levels. Fine details are settled near the end, when little noise is left.
- What is the main practical cost of this method?
- Speed: a picture needs 1,000 passes through the network, against one for a GAN.
Idea · §4 · Tables 1, 3
The results, and what came next
Image quality is often scored with FID: lower means the generated pictures are statistically closer to real ones. On CIFAR-10 (tiny 32 × 32 photos), DDPM scored 3.17, a new best for models not told the image class. On 256 × 256 pictures of bedrooms and churches, its quality was comparable to ProgressiveGAN, one of the strongest GANs of the time.
The paper also found something surprising: these models are not great at the "how likely is this image?" score that the 2015 paper emphasised. Most of that score is spent on details too faint for people to see.
TakeawayDDPM made diffusion work in practice, and nearly every modern image generator descends from its recipe.
Abstract, restated
High-quality image synthesis with diffusion probabilistic models, latent-variable models inspired by non-equilibrium thermodynamics. The best results come from training on a weighted variational bound designed from a novel connection between diffusion models and denoising score matching with Langevin dynamics. The models admit a progressive lossy decompression scheme that can be read as a generalisation of autoregressive decoding. Unconditional CIFAR-10: Inception score 9.46, FID 3.17 (state of the art). 256 × 256 LSUN: quality similar to ProgressiveGAN. NeurIPS 2020; arXiv:2006.11239.
Section · §2 · Eqs. 1–7
Forward process, closed forms and posterior
The forward process is a fixed Markov chain with variance schedule β1, …, βT:
With αt = 1 − βt and ᾱt = ∏s≤t αs, any step can be sampled directly from the data:
and the forward posterior is Gaussian with closed-form parameters:
The variational bound (Eq. 5) decomposes into LT (prior matching; constant here because β is fixed), Lt−1 = DKL(q(xt−1 | xt, x0) ‖ pθ(xt−1 | xt)) for 1 < t ≤ T, and a decoder term L0. Unlike Sohl-Dickstein et al., the schedule is not learned: T = 1000 and βt rises linearly from 10−4 to 0.02 (§4), chosen so that ᾱT is close to 0 while each step stays small.
Section · §3.2, §3.4 · Eqs. 11–14 · Algorithm 1
ε-prediction and the simplified objective
The reverse process is pθ(xt−1 | xt) = N(μθ(xt, t), σt2I) with untrained, time-dependent variances. Since xt = √ᾱtx0 + √(1 − ᾱt)ε, substituting x0 into Eq. 7 shows the target mean is a function of xt and ε. The paper therefore parameterises
which turns each Lt−1 into a weighted noise-regression loss (Eq. 12) with weight βt2/(2σt2αt(1 − ᾱt)). Dropping that weight gives the objective used for the best samples:
Relative to the true bound, Lsimple down-weights the small-t terms (easy, low-noise denoising) so the network spends capacity on harder, noisier steps (§3.4). It is a better sample-quality objective and a worse likelihood objective.
Algorithm 1 Training
repeat
x0 ~ q(x0)
t ~ Uniform({1, ..., T})
eps ~ N(0, I)
take gradient step on || eps - eps_theta( sqrt(abar_t) x0 + sqrt(1 - abar_t) eps , t ) ||^2
until converged
Check yourself
- Why does Eq. 12 look like denoising score matching?
- For Gaussian corruption, the score of the noised distribution is ∇ log q(xt) = −E[ε | xt]/√(1 − ᾱt), so regressing ε is score matching at noise scale √(1 − ᾱt), summed over many scales.
- What does Lsimple sacrifice?
- Log-likelihood. Table 1 shows the true-bound model (3.70 bits/dim) beats Lsimple (3.75) on NLL while being far worse on FID (13.51 vs 3.17).
Section · §3.2–3.3 · Algorithm 2
Sampling as Langevin-like dynamics
Algorithm 2 Sampling
xT ~ N(0, I)
for t = T, ..., 1:
z ~ N(0, I) if t > 1, else z = 0
x_{t-1} = (1/sqrt(alpha_t)) ( x_t - ((1 - alpha_t)/sqrt(1 - abar_t)) eps_theta(x_t, t) ) + sigma_t z
return x0
With εθ as a scaled score estimate, each step is a gradient step on log density plus injected noise, resembling annealed Langevin dynamics (Song and Ermon, 2019). For σt2, βt and β̃t gave similar results; they are the optimal choices for x0 ~ N(0, I) and for deterministic x0 respectively, the two extremes of reverse-process entropy (§3.2). Data are scaled to [−1, 1] and the last step uses a discretised Gaussian decoder (Eq. 13), with the mean displayed noiselessly (§3.3).
Below, Algorithm 2 runs exactly as written on 1-D data (two clusters at −1 and +1), with the exact optimal εθ(xt, t) = E[ε | xt] substituted for the network.
Section · §4.1–4.2 · Tables 1–3 · App. B
Experiments
Architecture. A U-Net backbone similar to an unmasked PixelCNN++, built from Wide-ResNet blocks, with group normalisation, self-attention at the 16 × 16 resolution, and the step t given by a Transformer sinusoidal embedding added in each residual block. 35.7M parameters for CIFAR-10, 114M for LSUN and CelebA-HQ (about 256M for a larger LSUN Bedroom model). Training on a TPU v3-8: CIFAR-10 for 800k steps at batch 128 (about 10.6 hours); sampling 256 CIFAR images took about 17 s. Adam, learning rate 2 × 10−4 (2 × 10−5 at 256 × 256), EMA 0.9999, dropout 0.1 on CIFAR-10.
| CIFAR-10, Table 1 | IS | FID | NLL test (train), bits/dim |
|---|---|---|---|
| Ours, L (true bound, fixed isotropic Σ) | 7.67 ± 0.13 | 13.51 | ≤ 3.70 (3.69) |
| Ours, Lsimple | 9.46 ± 0.11 | 3.17 | ≤ 3.75 (3.72) |
| StyleGAN2 + ADA (v1), unconditional | 9.74 ± 0.05 | 3.26 | – |
| StyleGAN2 + ADA (v1), class-conditional | 10.06 | 2.67 | – |
| Diffusion (original, 2015) | – | – | ≤ 5.40 |
| NCSN (Song & Ermon) | 8.87 ± 0.12 | 25.32 | – |
| Ablation, Table 2 (CIFAR-10) | IS | FID |
|---|---|---|
| predict μ̃, true bound, learned diagonal Σ | 7.28 | 23.69 |
| predict μ̃, true bound, fixed isotropic Σ | 8.06 | 13.22 |
| predict μ̃, unweighted MSE | unstable | |
| predict ε, true bound, learned diagonal Σ | unstable | |
| predict ε, true bound, fixed isotropic Σ | 7.67 | 13.51 |
| predict ε, Lsimple | 9.46 | 3.17 |
Table 3 (LSUN 256 × 256 FID): Church 7.89 (ProgressiveGAN 6.42), Bedroom 6.36 and 4.90 for the large model (ProgressiveGAN 8.34), Cat 19.75 (ProgressiveGAN 37.52). CIFAR-10 FID against the test set is 5.24; the train/test NLL gap of at most 0.03 bits/dim indicates no overfitting.
TakeawayThe ablation is the paper in miniature: ε-prediction with the unweighted loss is what turns diffusion from FID ~13 into FID ~3.
Check yourself
- Which StyleGAN2 number is lower than DDPM's FID, and why is it not a like-for-like comparison?
- StyleGAN2 + ADA's 2.67 is class-conditional; DDPM's 3.17 is unconditional. The like-for-like unconditional StyleGAN2 + ADA result is 3.26, slightly worse than DDPM.
Section · §4.3–4.4 · Table 4 · Figs. 6–9
Progressive coding and interpolation
Lossless codelengths. 3.75 bits/dim is better than energy-based models and score matching but worse than other likelihood models (the paper cites 2.80 for the Sparse Transformer). Treating L1 + … + LT as rate and L0 as distortion, the CIFAR-10 model has rate 1.78 bits/dim and distortion 1.97 bits/dim, an RMSE of 0.95 on a 0–255 scale. So more than half the lossless codelength describes imperceptible distortions: diffusion models have an inductive bias towards perceptually good lossy compression.
Progressive generation. Predicting x̂0 from xt during sampling shows large-scale features appearing first and details last (Fig. 6). Sampling from the same xt shares high-level attributes when t is small and only coarse layout when t is large (Fig. 7). The authors read Algorithm 2 as a generalisation of autoregressive decoding along a bit ordering.
Interpolation (Figs. 8–9). Encode two images to xt with the forward process (t = 500), interpolate linearly, and decode with the reverse process; on CelebA-HQ this smoothly varies pose, skin tone, hairstyle, expression and background, "but not eyewear".
Section · §5–6 and lineage
Limitations and lineage
- Sampling cost. 1,000 sequential network evaluations; sampling 128 images at 256 × 256 took about 300 s on the TPU v3-8.
- Likelihood. NLL is uncompetitive with autoregressive models, and the best-sample objective is not the best-likelihood objective.
- Schedule and variances by hand. Fixed linear β and fixed σt leave quality on the table, especially with fewer steps.
Reference
Glossary
- βt, αt, ᾱt
- Per-step noise variance, 1 − βt, and the cumulative product of α.
- ε-prediction
- Network outputs the noise that was added, rather than the mean or the clean image.
- Lsimple
- Unweighted mean-squared error on ε with uniformly sampled t (Eq. 14).
- FID
- Fréchet Inception Distance: distance between Gaussian fits to Inception-network features of real and generated images. Lower is better.
- IS
- Inception Score: rewards confident and diverse class predictions on samples. Higher is better.
- bits/dim
- Negative log-likelihood per pixel channel, in bits.
- Langevin dynamics
- Sampling by repeatedly following the score (gradient of log density) plus Gaussian noise.
- EMA
- Exponential moving average of weights, used at sampling time.