The paper in four sentences
Diffusion models already covered the variety of real data better than GANs, but their individual pictures were not as convincing on big, varied datasets like ImageNet.
A systematic search over network designs (more attention, better ways to feed in the noise level and class) closed much of the gap on its own.
The bigger step was classifier guidance: a separate classifier, trained to recognise classes in noisy images, nudges every denoising step towards "more like a corgi", with a dial for how hard to push.
Turning the dial up gives more convincing but less varied images, and the best setting beats BigGAN-deep. A year later, classifier-free guidance got the same effect without a separate classifier.
Idea · §1–2
The scoreboard: quality and variety
Image generators are judged on two things that pull against each other. Fidelity: does each picture look real? Diversity: do the pictures, taken together, cover everything the real data contains, or does the model play it safe and repeat a few favourites?
GANs were very good at fidelity but known to drop parts of the data ("mode collapse"). Diffusion models covered the data well but, on hard datasets, each picture looked less convincing. The common score, FID, mixes both: it compares the statistics of thousands of generated images with real ones, and lower is better. The paper also reports precision (roughly, the share of generated images that look like they belong to the real data) and recall (roughly, the share of the real data the generator can reach).
The authors suspected two reasons diffusion lagged: its networks had been tuned less intensively than GANs', and GANs had a built-in way to trade diversity for fidelity (the "truncation trick") that diffusion lacked.
TakeawayGenerators are judged on how real each image looks and how much of the real variety they cover. Diffusion led on variety; GANs on looks.
Idea · §3 · Tables 1–3
A better-tuned network
The denoising network in diffusion models is a U-Net: it shrinks the image step by step to capture the big picture, then expands it back, with shortcuts that carry fine detail across. The authors tried many changes one at a time and kept the ones that helped:
- Attention at more scales. Attention lets every part of the image consult every other part. Using it at three image sizes instead of one helped.
- More attention heads. Splitting attention into more, smaller "heads" that look for different things helped.
- Better resizing blocks, borrowed from BigGAN.
- Adaptive normalisation. The network is told the noise level (and, when there is one, the class) by letting them rescale and shift its internal signals, rather than just adding them in.
They call the result ADM ("ablated diffusion model"). Its improvements alone brought diffusion close to GANs on ImageNet.
TakeawayCareful engineering of the denoising network was worth a lot: no new theory, just many small, tested improvements.
Check yourself
- What does "adaptive" normalisation adapt to?
- The step (noise level) and class: they are turned into a scale and a shift applied to each layer's normalised signals.
Idea · §4.1 · Algorithm 1
Steering with a classifier
Suppose you want a picture of a corgi. Train a separate classifier, a network that looks at an image and says how likely each class is. Crucially, train it on noisy images, so it can make sensible guesses at every stage of the denoising.
At each denoising step, ask the classifier: "which small change to this noisy picture would make it look more like a corgi?". That direction is called the classifier's gradient. Nudge the step a little in that direction. Repeat for every step. A guidance scale multiplies the nudge: 0 means no steering, larger means pushing harder.
Below is a one-dimensional version. Data come from three overlapping classes. The diffusion model knows nothing about classes; a perfect noisy-image classifier steers it towards the class you pick.
TakeawayClassifier guidance adds, at every step, a nudge in the direction a noisy-image classifier says makes the picture more like the wanted class.
Check yourself
- Why must the classifier be trained on noisy images?
- It is consulted at every step, including early ones where the picture is mostly noise. An ordinary classifier has never seen such inputs and its gradients would be meaningless there.
- In the figure, what happens to samples near the border with a neighbouring class as the scale grows?
- They are pushed away from the border, towards the region where the classifier is most confident, so the samples get less varied.
Idea · §4.3 · Fig. 3 · Table 4
The fidelity–diversity dial
The guidance scale turned out to be a powerful dial. On an unconditional ImageNet model (one never told the class), a scale of 1 gave pictures the classifier rated about 50% likely to be the right class, yet they often didn't look like it. Scaling up to 10 made them convincingly on-class (the paper's corgi example, Fig. 3).
The table below holds the paper's real numbers at 256 × 256. Guidance raises precision (pictures look more real) and lowers recall (less variety). FID, which balances both, is best somewhere in between: for the class-conditional model, a scale of 1.0.
TakeawayTurning up guidance buys realism with variety. The best overall score comes from a moderate setting.
Idea · §5, §7
Results, costs, and what replaced the classifier
With both changes, the guided model (ADM-G) beat BigGAN-deep's FID on ImageNet at 128 × 128 (2.97 vs 6.02), 256 × 256 (4.59 vs 6.95) and 512 × 512 (7.72 vs 8.43), while keeping better recall. With only 25 denoising steps it roughly matched BigGAN-deep. Unguided ADM also beat the best GANs on LSUN bedrooms, horses and cats.
The costs: sampling still takes many network passes, training uses more compute than a GAN, and guidance needs labelled data plus a second, noise-aware classifier.
TakeawayThis paper ended GANs' lead in image quality; its guidance idea survived in classifier-free form.
Abstract, restated
Diffusion models can achieve image sample quality superior to the state-of-the-art generative models. Unconditional synthesis improves through a series of architecture ablations. For conditional synthesis, classifier guidance, a simple, compute-efficient method that uses gradients from a classifier, trades off diversity for fidelity. FID 2.97 on ImageNet 128 × 128, 4.59 on 256 × 256 and 7.72 on 512 × 512; BigGAN-deep is matched with as few as 25 forward passes per sample while keeping better distribution coverage. Combined with upsampling diffusion models, guidance improves FID to 3.94 at 256 × 256 and 3.85 at 512 × 512. NeurIPS 2021; arXiv:2105.05233.
Section · §2 · Eq. 1
Background: learned variances, DDIM, metrics
The paper builds on DDPM with two upgrades from Improved DDPM (Nichol and Dhariwal, 2021). First, learned reverse variances, interpolating in log space between the two DDPM extremes:
trained with the hybrid objective Lsimple + λLvlb; this allows sampling with fewer steps without much loss of quality. Second, DDIM (Song et al., 2020), a deterministic sampler that shares the training objective and is used for few-step sampling (25 steps below).
Metrics (§2.2). FID (Inception-V3 pool features), sFID (spatial features, more sensitive to layout), Inception Score, and improved precision/recall (Kynkäänniemi et al., 2019), which separate fidelity (precision) from coverage (recall). FID is the primary metric.
Section · §3 · Tables 1–3
Architecture ablations
Starting from the DDPM U-Net on ImageNet 128 × 128, the authors ablate: depth vs width (deeper helps but trains slower in wall-clock time, so not adopted), number of attention heads / channels per head, attention at 32, 16 and 8 rather than only 16, BigGAN residual blocks for up- and downsampling, and rescaling residual connections by 1/√2. All except the residual rescaling improved FID. The final model uses 64 channels per head, attention at 32, 16 and 8, BigGAN up/downsampling, and adaptive group normalisation:
| Ablation (ImageNet 128, FID) | Result |
|---|---|
| Table 1 baseline (1 head, attention at 16 only), 700K / 1200K iters | 15.33 / 13.21 |
| Table 1 best combination (4 heads, attn 32/16/8, BigGAN up/down, no rescale) | −3.14 / −3.00 |
| Table 2: 1 head (baseline), and the change with 32 channels per head | 14.08; −1.36 |
| Table 3: Addition + GroupNorm (DDPM style) | 15.08 |
| Table 3: AdaGN | 13.06 |
Check yourself
- Which tested change did not help, and was dropped?
- Rescaling residual connections by 1/√2.
Section · §4.1–4.2 · Eqs. 2–14 · Algorithm 1
Classifier guidance
Train a classifier pφ(y | xt, t) on noised images (the downsampling half of the U-Net with an attention pool at 8 × 8, trained on the same noising distribution as the diffusion model). A conditional reverse step can be written
With pθ(xt | xt+1) = N(μ, Σ) and log pφ having low curvature relative to Σ−1 (true in the limit of infinitely many steps, where ‖Σ‖ tends to 0), a first-order Taylor expansion around xt = μ gives log pφ(y | xt) ≈ (xt − μ)g + C with g = ∇xt log pφ(y | xt)|xt=μ. Completing the square, the product is approximately N(μ + Σg, Σ) (Eqs. 3–10): guidance is a mean shift by the covariance-scaled classifier gradient.
Algorithm 1 Classifier-guided diffusion sampling
input: class label y, gradient scale s
x_T ~ N(0, I)
for t = T, ..., 1:
mu, Sigma = mu_theta(x_t), Sigma_theta(x_t)
x_{t-1} ~ N( mu + s * Sigma * grad_{x_t} log p_phi(y | x_t), Sigma )
return x_0
DDIM version (§4.2). Deterministic samplers have no Σ to scale, so use the score identity ∇ log pθ(xt) = −εθ(xt)/√(1 − ᾱt) and add the classifier's score to the joint:
then run ordinary DDIM with ε̂ (Algorithm 2 in the paper). Below, the same 1-D toy as the Plain English tab is run for a sweep of scales s with exact scores, to show the trade-off curve that Fig. 4 shows for ImageNet.
Section · §4.3 · Figs. 3–5 · Table 4
Why scale the gradient?
Multiplying the gradient by s is guidance towards a sharpened classifier: s·∇ log p(y | x) = ∇ log [p(y | x)s/Z]. For s > 1 the distribution p(y | x)s concentrates on inputs the classifier is most sure about, so samples become higher-fidelity and less diverse. With an unconditional model, s = 1 gave samples rated about 50% on-class by the classifier but which often did not look on-class; s = 10 gave nearly 100% and visually convincing samples (Fig. 3, Pembroke Welsh corgi; FID 33.0 vs 12.0). Guidance can also be applied to a conditional model, and combining both gives the best results (Table 4).
| Table 4 (ImageNet 256) | Scale | FID | Prec | Rec |
|---|---|---|---|---|
| Unconditional | – | 26.21 | 0.61 | 0.63 |
| Unconditional | 1.0 | 33.03 | 0.56 | 0.65 |
| Unconditional | 10.0 | 12.00 | 0.76 | 0.44 |
| Conditional | – | 10.94 | 0.69 | 0.63 |
| Conditional | 1.0 | 4.59 | 0.82 | 0.52 |
| Conditional | 10.0 | 9.11 | 0.88 | 0.32 |
Fig. 5 compares guidance with BigGAN-deep's truncation trick: guidance is strictly better on the FID–IS trade-off, but not on the precision–recall trade-off beyond a certain precision.
Check yourself
- Why does the unconditional model at s = 1 have a worse FID than at no guidance?
- At s = 1 the samples are pulled off the model's distribution towards classifier-pleasing regions without becoming convincingly on-class, so they lose diversity and realism; only a stronger push makes them clearly on-class and improves FID.
Section · §5 · Tables 5–6
Results
| Table 5 | Model | FID | Prec | Rec |
|---|---|---|---|---|
| ImageNet 64 | BigGAN-deep (trained by the authors) | 4.06 | 0.79 | 0.48 |
| ADM (dropout) | 2.07 | 0.74 | 0.63 | |
| ImageNet 128 | BigGAN-deep | 6.02 | 0.86 | 0.35 |
| ADM-G (25 steps) | 5.98 | 0.78 | 0.51 | |
| ADM-G | 2.97 | 0.78 | 0.59 | |
| ImageNet 256 | BigGAN-deep | 6.95 | 0.87 | 0.28 |
| ADM-G | 4.59 | 0.82 | 0.52 | |
| ImageNet 512 | BigGAN-deep | 8.43 | 0.88 | 0.29 |
| ADM-G | 7.72 | 0.87 | 0.42 | |
| LSUN Bedroom | StyleGAN / ADM (dropout) | 2.35 / 1.90 | ||
| LSUN Horse | StyleGAN2 / ADM (dropout) | 3.84 / 2.57 | ||
| LSUN Cat | StyleGAN2 / ADM (dropout) | 7.25 / 5.57 |
Upsampling (§5.2, Table 6). A two-stage pipeline (a base model at 64 or 128, then an upsampling diffusion model) with guidance on the base model reaches FID 3.94 at 256 × 256 and 3.85 at 512 × 512 (250 + 250 steps). Guidance and upsampling are complementary. At 256 × 256, upsampling alone improves FID from 10.94 to 7.49 while keeping recall at 0.63, whereas guidance alone raises precision from 0.69 to 0.82 at the cost of recall (0.52). Note how much of the gain at 512 comes from guidance alone (from 23.24 for ADM to 7.72 for ADM-G).
Section · §7 and lineage
Limitations and lineage
- Sampling speed. Still slower than GANs at sampling time because of multiple denoising steps; 25-step DDIM costs some FID.
- Training compute. More than the corresponding GANs, though upsampling stacks narrow the gap.
- Labels and a second network. Guidance needs labelled data and a classifier trained on noisy inputs; the authors suggest clustering for synthetic labels on unlabelled data.
- Adversarial gradients. Classifier gradients can push towards features that fool the classifier rather than improve images; this was one motivation stated by later work for removing the classifier.
Reference
Glossary
- ADM / ADM-G / ADM-U
- Ablated diffusion model; with classifier guidance; upsampling variant.
- AdaGN
- Adaptive group normalisation: GroupNorm followed by a scale and shift predicted from the timestep and class embedding.
- Guidance scale s
- Multiplier on the classifier gradient; equivalent to guiding with p(y | x)s.
- Precision / recall
- Fraction of samples within the real-data manifold; fraction of real data within the sample manifold (Kynkäänniemi et al., 2019).
- sFID
- FID on spatial (intermediate) Inception features, more sensitive to spatial structure.
- Truncation trick
- Sampling a GAN's latent from a truncated normal to trade diversity for fidelity.
- DDIM
- Denoising diffusion implicit model: deterministic sampler allowing far fewer steps.