Study guide · Computer vision · 2021

Diffusion Models Beat GANs on Image Synthesis

Prafulla Dhariwal, Alex Nichol · NeurIPS 2021

In 2021 GANs still made the best-looking images, and diffusion models trailed on hard datasets like ImageNet. Dhariwal and Nichol close the gap with two moves: a carefully tuned network architecture, and classifier guidance, which uses an image classifier's gradient to steer each denoising step towards the requested class. The result beats the best GAN (BigGAN-deep) on ImageNet at every resolution they test, with a dial that trades variety for fidelity.

The paper in four sentences

Diffusion models already covered the variety of real data better than GANs, but their individual pictures were not as convincing on big, varied datasets like ImageNet.

A systematic search over network designs (more attention, better ways to feed in the noise level and class) closed much of the gap on its own.

The bigger step was classifier guidance: a separate classifier, trained to recognise classes in noisy images, nudges every denoising step towards "more like a corgi", with a dial for how hard to push.

Turning the dial up gives more convincing but less varied images, and the best setting beats BigGAN-deep. A year later, classifier-free guidance got the same effect without a separate classifier.

Idea · §1–2

The scoreboard: quality and variety

Image generators are judged on two things that pull against each other. Fidelity: does each picture look real? Diversity: do the pictures, taken together, cover everything the real data contains, or does the model play it safe and repeat a few favourites?

GANs were very good at fidelity but known to drop parts of the data ("mode collapse"). Diffusion models covered the data well but, on hard datasets, each picture looked less convincing. The common score, FID, mixes both: it compares the statistics of thousands of generated images with real ones, and lower is better. The paper also reports precision (roughly, the share of generated images that look like they belong to the real data) and recall (roughly, the share of the real data the generator can reach).

The authors suspected two reasons diffusion lagged: its networks had been tuned less intensively than GANs', and GANs had a built-in way to trade diversity for fidelity (the "truncation trick") that diffusion lacked.

Everyday pictureA street artist who only ever paints the Eiffel Tower paints it beautifully (high precision) but cannot paint Paris (low recall). One who tries everything covers the city but each picture is rougher. FID rewards being good at both.

TakeawayGenerators are judged on how real each image looks and how much of the real variety they cover. Diffusion led on variety; GANs on looks.

Idea · §3 · Tables 1–3

A better-tuned network

The denoising network in diffusion models is a U-Net: it shrinks the image step by step to capture the big picture, then expands it back, with shortcuts that carry fine detail across. The authors tried many changes one at a time and kept the ones that helped:

  • Attention at more scales. Attention lets every part of the image consult every other part. Using it at three image sizes instead of one helped.
  • More attention heads. Splitting attention into more, smaller "heads" that look for different things helped.
  • Better resizing blocks, borrowed from BigGAN.
  • Adaptive normalisation. The network is told the noise level (and, when there is one, the class) by letting them rescale and shift its internal signals, rather than just adding them in.

They call the result ADM ("ablated diffusion model"). Its improvements alone brought diffusion close to GANs on ImageNet.

TakeawayCareful engineering of the denoising network was worth a lot: no new theory, just many small, tested improvements.

Check yourself
What does "adaptive" normalisation adapt to?
The step (noise level) and class: they are turned into a scale and a shift applied to each layer's normalised signals.

Idea · §4.1 · Algorithm 1

Steering with a classifier

Suppose you want a picture of a corgi. Train a separate classifier, a network that looks at an image and says how likely each class is. Crucially, train it on noisy images, so it can make sensible guesses at every stage of the denoising.

At each denoising step, ask the classifier: "which small change to this noisy picture would make it look more like a corgi?". That direction is called the classifier's gradient. Nudge the step a little in that direction. Repeat for every step. A guidance scale multiplies the nudge: 0 means no steering, larger means pushing harder.

Below is a one-dimensional version. Data come from three overlapping classes. The diffusion model knows nothing about classes; a perfect noisy-image classifier steers it towards the class you pick.

Class:

Grey: the whole data distribution, made of three classes (thin outlines). Amber bars: 1,500 samples from a 100-step diffusion sampler with the chosen class and guidance scale. Scale 0 gives all three classes. Scale 1 roughly recovers the chosen class. Larger scales pile samples into the part of the class that is least like its neighbours. Toy illustration with exact denoiser and exact classifier.
Everyday pictureA sculptor (the diffusion model) chips away at a block. A critic standing beside them (the classifier) says after every cut "a bit more corgi-like that way". With a gentle critic you get a varied range of corgis; with a pushy one you get the most stereotypical corgi possible, every time.

TakeawayClassifier guidance adds, at every step, a nudge in the direction a noisy-image classifier says makes the picture more like the wanted class.

Check yourself
Why must the classifier be trained on noisy images?
It is consulted at every step, including early ones where the picture is mostly noise. An ordinary classifier has never seen such inputs and its gradients would be meaningless there.
In the figure, what happens to samples near the border with a neighbouring class as the scale grows?
They are pushed away from the border, towards the region where the classifier is most confident, so the samples get less varied.

Idea · §4.3 · Fig. 3 · Table 4

The fidelity–diversity dial

The guidance scale turned out to be a powerful dial. On an unconditional ImageNet model (one never told the class), a scale of 1 gave pictures the classifier rated about 50% likely to be the right class, yet they often didn't look like it. Scaling up to 10 made them convincingly on-class (the paper's corgi example, Fig. 3).

The table below holds the paper's real numbers at 256 × 256. Guidance raises precision (pictures look more real) and lowers recall (less variety). FID, which balances both, is best somewhere in between: for the class-conditional model, a scale of 1.0.

Data: Table 4 of the paper (ImageNet 256 × 256, models trained for 2M iterations, 250 sampling steps). FID: lower is better. Precision and recall: higher is better.

TakeawayTurning up guidance buys realism with variety. The best overall score comes from a moderate setting.

Idea · §5, §7

Results, costs, and what replaced the classifier

With both changes, the guided model (ADM-G) beat BigGAN-deep's FID on ImageNet at 128 × 128 (2.97 vs 6.02), 256 × 256 (4.59 vs 6.95) and 512 × 512 (7.72 vs 8.43), while keeping better recall. With only 25 denoising steps it roughly matched BigGAN-deep. Unguided ADM also beat the best GANs on LSUN bedrooms, horses and cats.

The costs: sampling still takes many network passes, training uses more compute than a GAN, and guidance needs labelled data plus a second, noise-aware classifier.

Later refinementClassifier-free guidance (Ho and Salimans, 2021) removed the classifier: train one diffusion model both with and without the label, and at each step push away from the unlabelled guess towards the labelled one. It works with text prompts, where a noisy-image classifier is impractical, and became the standard in systems like Stable Diffusion.

TakeawayThis paper ended GANs' lead in image quality; its guidance idea survived in classifier-free form.

Follow the idea

Built on: GAN (2014), DDPM (2020)

Led to: Classifier-free guidance (2021)