Study guide · Computer vision · 2021

Classifier-Free Diffusion Guidance

Jonathan Ho, Tim Salimans · NeurIPS 2021 workshop

Classifier guidance made diffusion images sharper, but it needed a second network: a classifier trained on noisy images. Ho and Salimans show you can get the same effect from the diffusion model alone. Train it to denoise both with and without the label, then at every sampling step push the prediction away from the unlabelled guess and towards the labelled one. This one-line trick is how nearly every text-to-image model is steered today.

The paper in four sentences

Classifier guidance sharpened diffusion images by nudging every step with a separate classifier, which had to be specially trained on noisy images.

Instead, train a single diffusion model that sometimes gets the label (such as "dog") and sometimes gets a blank, so it learns both "a dog picture" and "any picture".

When generating, compute both guesses and exaggerate the difference between them: move further in the direction that the label adds.

A guidance weight sets how far to exaggerate: more gives more typical, convincing images and less variety, exactly the trade-off classifier guidance gave, with no classifier at all.

Idea · §1, §3.1

The trouble with an extra classifier

A diffusion model makes a picture by starting from random static and removing a little noise at a time. To ask for a particular kind of picture, you can give the model a label (a class such as "malamute", or later a text prompt). On its own, though, a labelled model often produces pictures that only vaguely match.

Dhariwal and Nichol fixed this with classifier guidance: a separate image classifier nudges each step towards "more malamute". It worked, but it had costs. The classifier had to be trained on noisy images at every noise level, so off-the-shelf classifiers couldn't be used. And some people suspected the trick might work partly by "fooling" the classifier, a bit like the adversarial examples that trick image recognisers, rather than by genuinely improving the images.

Everyday pictureA painter (the diffusion model) works with a critic standing beside them (the classifier). The question this paper asks: could the painter learn to be their own critic?

TakeawayGuidance improves conditional diffusion, but the classifier it needed was awkward to build and raised doubts about what it was really doing.

Idea · §3.2 · Algorithm 1 · Eq. 6

One model, two guesses

Training changes by one line. Normally every training picture comes with its label. Here, a random 10 to 20% of the time the label is replaced by a "blank" token. So the same network learns two jobs: denoise given the label, and denoise without it.

At each sampling step, ask the network twice: once with the label, once with the blank. The difference between the two answers is "what the label adds". Classifier-free guidance takes the labelled answer and adds an extra helping of that difference, scaled by a guidance weight w. At w = 0 you get the plain labelled model; larger w exaggerates.

The figure shows these directions for three classes of 2-D points. Each arrow shows which way a noisy point is pushed. The blue arrows use the label alone; the amber arrows add guidance. Turn up w and the amber arrows turn away from the other classes.

Label:

Circles: where each class's noisy points lie at this noise level (one standard deviation; they shrink towards the centre and spread out as noise grows). Each dot is a noisy point; the lines leaving it show where it is pushed. Blue: the direction the labelled model moves a noisy point. Amber: the guided direction, (1 + w) × labelled − w × unlabelled. Noise level runs from 5 (little noise) to 95 (mostly noise) on a 100-step scale. Toy illustration with exact denoisers.

TakeawayOne network learns both "with label" and "without label". Guidance moves each step further along the difference between them.

Check yourself
What does the network see during training when the label is dropped?
A special "null" label. It then has to denoise without knowing the class, which teaches it the unconditional model.
Why does subtracting the unlabelled guess help?
The unlabelled guess points towards "any plausible picture". Moving away from it emphasises what is specific to the requested label, so samples become more distinctively that class.

Idea · §3 · Fig. 2 · §4.1

The guidance dial

Now run the whole sampling process with those guided arrows. The paper's Figure 2 does this for three Gaussian blobs; the figure below does the same live. With w = 0 the samples cover the requested class, including the parts that overlap its neighbours. As w grows, they crowd into the region that is most clearly that class and least like the others.

On real images the effect is the same: more typical, more recognisable pictures, at the cost of variety. In the paper's ImageNet tests, a small amount of guidance gave the best overall quality score, while larger amounts gave the most convincing individual images.

Label:

600 samples from a 100-step sampler using classifier-free guidance with the exact labelled and unlabelled denoisers. Dashed circles: two standard deviations of each class. Try label A, then raise w from 0 to 4: the cloud shrinks and moves away from B and C. Toy illustration modelled on the paper's Fig. 2.

TakeawayTurning up the guidance weight trades variety for typicality, the same dial classifier guidance offered.

Check yourself
At high w, why do samples move away from the other classes rather than just shrinking to the class centre?
Guidance rewards points that are much more likely under the label than overall. Points near other classes are fairly likely overall, so they are penalised, and the cloud drifts to the side facing away from its neighbours.

Idea · §4 · Tables 1–2

Does it work on real images?

The authors trained class-conditional diffusion models on ImageNet at 64 × 64 and 128 × 128 pixels. A small guidance weight (w = 0.1 at 64 × 64, 0.3 at 128 × 128) gave the best FID, the usual realism-plus-variety score. At 128 × 128 it scored 2.43, better than the classifier-guided model from Dhariwal and Nichol (2.97). Larger weights steadily raised the Inception Score (how confidently a classifier recognises each image) while FID got worse, which is the variety cost showing up.

Dropping the label 10% or 20% of the time worked about equally well; 50% gave a worse balance of quality and variety. So only a small share of training has to go to the unlabelled job.

TakeawayWithout any classifier, guidance matched and beat classifier guidance on ImageNet.

Idea · §5 and legacy

The price, and why it took over

The cost is speed: every sampling step now runs the network twice, once with and once without the label. The authors also note that less diverse outputs might under-represent rare kinds of data.

The benefits won out. The trick needs no extra model, works with any kind of conditioning (including text prompts, where a noisy-image classifier would be impractical), and is a couple of lines of code. The "guidance scale" slider in Stable Diffusion and similar tools is this paper's w, shifted by one: their default of 7.5 corresponds to w = 6.5 here.

Later refinementGLIDE, DALL·E 2, Imagen and Stable Diffusion all generate with classifier-free guidance, usually at much larger weights than the paper's best-FID settings, because people prefer prompt-faithful images over maximal variety. Later work tames side effects such as over-saturation at high weights, for example by applying guidance only in part of the noise range, or by guiding with a weaker version of the model itself ("autoguidance").

TakeawayClassifier-free guidance is simple, general and cheap enough that it became the standard way to steer diffusion models.

Follow the idea

Built on: Diffusion beats GANs (2021)

Led to: Latent diffusion (2021)