The paper in four sentences
Classifier guidance sharpened diffusion images by nudging every step with a separate classifier, which had to be specially trained on noisy images.
Instead, train a single diffusion model that sometimes gets the label (such as "dog") and sometimes gets a blank, so it learns both "a dog picture" and "any picture".
When generating, compute both guesses and exaggerate the difference between them: move further in the direction that the label adds.
A guidance weight sets how far to exaggerate: more gives more typical, convincing images and less variety, exactly the trade-off classifier guidance gave, with no classifier at all.
Idea · §1, §3.1
The trouble with an extra classifier
A diffusion model makes a picture by starting from random static and removing a little noise at a time. To ask for a particular kind of picture, you can give the model a label (a class such as "malamute", or later a text prompt). On its own, though, a labelled model often produces pictures that only vaguely match.
Dhariwal and Nichol fixed this with classifier guidance: a separate image classifier nudges each step towards "more malamute". It worked, but it had costs. The classifier had to be trained on noisy images at every noise level, so off-the-shelf classifiers couldn't be used. And some people suspected the trick might work partly by "fooling" the classifier, a bit like the adversarial examples that trick image recognisers, rather than by genuinely improving the images.
TakeawayGuidance improves conditional diffusion, but the classifier it needed was awkward to build and raised doubts about what it was really doing.
Idea · §3.2 · Algorithm 1 · Eq. 6
One model, two guesses
Training changes by one line. Normally every training picture comes with its label. Here, a random 10 to 20% of the time the label is replaced by a "blank" token. So the same network learns two jobs: denoise given the label, and denoise without it.
At each sampling step, ask the network twice: once with the label, once with the blank. The difference between the two answers is "what the label adds". Classifier-free guidance takes the labelled answer and adds an extra helping of that difference, scaled by a guidance weight w. At w = 0 you get the plain labelled model; larger w exaggerates.
The figure shows these directions for three classes of 2-D points. Each arrow shows which way a noisy point is pushed. The blue arrows use the label alone; the amber arrows add guidance. Turn up w and the amber arrows turn away from the other classes.
TakeawayOne network learns both "with label" and "without label". Guidance moves each step further along the difference between them.
Check yourself
- What does the network see during training when the label is dropped?
- A special "null" label. It then has to denoise without knowing the class, which teaches it the unconditional model.
- Why does subtracting the unlabelled guess help?
- The unlabelled guess points towards "any plausible picture". Moving away from it emphasises what is specific to the requested label, so samples become more distinctively that class.
Idea · §3 · Fig. 2 · §4.1
The guidance dial
Now run the whole sampling process with those guided arrows. The paper's Figure 2 does this for three Gaussian blobs; the figure below does the same live. With w = 0 the samples cover the requested class, including the parts that overlap its neighbours. As w grows, they crowd into the region that is most clearly that class and least like the others.
On real images the effect is the same: more typical, more recognisable pictures, at the cost of variety. In the paper's ImageNet tests, a small amount of guidance gave the best overall quality score, while larger amounts gave the most convincing individual images.
TakeawayTurning up the guidance weight trades variety for typicality, the same dial classifier guidance offered.
Check yourself
- At high w, why do samples move away from the other classes rather than just shrinking to the class centre?
- Guidance rewards points that are much more likely under the label than overall. Points near other classes are fairly likely overall, so they are penalised, and the cloud drifts to the side facing away from its neighbours.
Idea · §4 · Tables 1–2
Does it work on real images?
The authors trained class-conditional diffusion models on ImageNet at 64 × 64 and 128 × 128 pixels. A small guidance weight (w = 0.1 at 64 × 64, 0.3 at 128 × 128) gave the best FID, the usual realism-plus-variety score. At 128 × 128 it scored 2.43, better than the classifier-guided model from Dhariwal and Nichol (2.97). Larger weights steadily raised the Inception Score (how confidently a classifier recognises each image) while FID got worse, which is the variety cost showing up.
Dropping the label 10% or 20% of the time worked about equally well; 50% gave a worse balance of quality and variety. So only a small share of training has to go to the unlabelled job.
TakeawayWithout any classifier, guidance matched and beat classifier guidance on ImageNet.
Idea · §5 and legacy
The price, and why it took over
The cost is speed: every sampling step now runs the network twice, once with and once without the label. The authors also note that less diverse outputs might under-represent rare kinds of data.
The benefits won out. The trick needs no extra model, works with any kind of conditioning (including text prompts, where a noisy-image classifier would be impractical), and is a couple of lines of code. The "guidance scale" slider in Stable Diffusion and similar tools is this paper's w, shifted by one: their default of 7.5 corresponds to w = 6.5 here.
TakeawayClassifier-free guidance is simple, general and cheap enough that it became the standard way to steer diffusion models.
Abstract, restated
Classifier guidance trades mode coverage for sample fidelity in conditional diffusion models, but requires training an image classifier separately from the diffusion model. This paper shows guidance can be performed by a pure generative model without such a classifier: jointly train a conditional and an unconditional diffusion model, and combine their score estimates to obtain a trade-off between sample quality and diversity similar to that of classifier guidance. A short version appeared at the NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications; arXiv:2207.12598 (2022).
Section · §2 · Eqs. 1–5
Continuous-time setup
The paper uses a variance-preserving continuous-time formulation indexed by log signal-to-noise ratio λ = log(αλ2/σλ2):
Training samples λ = −2 log tan(au + b) with u ~ U[0, 1], b = arctan(e−λmax/2), a = arctan(e−λmin/2) − b: a hyperbolic secant distribution truncated to [λmin, λmax] = [−20, 20], inspired by the cosine schedule of Nichol and Dhariwal. The model predicts ε, with xθ(zλ) = (zλ − σλεθ(zλ))/αλ, trained on E‖εθ(zλ) − ε‖2 (Eq. 5), so εθ(zλ) ≈ −σλ∇zλ log p(zλ). Sampling uses ancestral steps with a variance interpolation exponent v (Eq. 4): v = 0.3 at 64 × 64, 0.2 at 128 × 128.
Section · §3.1
Classifier guidance in this notation
so sampling targets p̃θ(zλ | c) ∝ pθ(zλ | c) pθ(c | zλ)w, up-weighting points where the classifier is confident. The paper notes that guiding an unconditional model with weight w + 1 is in principle the same as guiding a conditional model with weight w, since p(z | c)p(c | z)w ∝ p(z)p(c | z)w+1, but that the conditional form gave the best results in practice (as in Dhariwal and Nichol). Fig. 2 illustrates the effect on a mixture of three Gaussians.
Section · §3.2 · Eq. 6 · Algorithms 1–2
Classifier-free guidance
A single network parameterises both models: εθ(zλ) = εθ(zλ, c = ∅). Sampling uses the linear combination
Why this is guidance. Define the implicit classifier pi(c | zλ) ∝ p(zλ | c)/p(zλ). With exact scores ε*, ∇ log pi(c | zλ) = −(1/σλ)[ε*(zλ, c) − ε*(zλ)]. Plugging this into classifier guidance gives ε̃*(zλ, c) = (1 + w)ε*(zλ, c) − wε*(zλ), which is Eq. 6. Equivalently, the target is p(z | c)1+w/p(z)w.
The paper stresses a subtlety: Eq. 6 uses unconstrained networks, so ε̃θ need not be the score of any distribution (the vector field need not be conservative), and the guided sampler does not exactly sample pi-tilted distributions. Even with exact scores, tilting each noise level separately is not the same as tilting the clean data distribution and then diffusing it.
Algorithm 1 Joint training input: p_uncond repeat (x, c) ~ p(x, c) c <- null with probability p_uncond lambda ~ p(lambda); eps ~ N(0, I) z = alpha_lambda x + sigma_lambda eps gradient step on || eps_theta(z, c) - eps ||^2 until converged Algorithm 2 Conditional sampling input: guidance w, condition c, increasing log-SNR sequence lambda_1..lambda_T z_1 ~ N(0, I) for t = 1..T: eps~ = (1 + w) eps_theta(z_t, c) - w eps_theta(z_t) # Eq. 6 x~ = (z_t - sigma_t eps~) / alpha_t z_{t+1} ~ N( mu~(z_t, x~), sigma~^2 ) if t < T else z_{t+1} = x~ return z_{T+1}
The figure plots the ideal clean-data target of Eq. 6 in one dimension, p(x | c)1+w p(x)−w normalised, for three overlapping classes.
Check yourself
- Write classifier-free guidance in the "scale s" form used by most libraries.
- εθ(z) + s[εθ(z, c) − εθ(z)] with s = 1 + w. So s = 1 (w = 0) is the plain conditional model.
- Why can't the guided sampler be read as an adversarial attack on a classifier?
- There is no classifier: the update is a difference of two generative-model outputs. The paper argues this shows guidance gains are obtainable by a purely generative model.
Section · §4 · Tables 1–2 · Figs. 3–5
Experiments
Class-conditional ImageNet at 64 × 64 and 128 × 128, using the architecture and hyperparameters of Dhariwal and Nichol's guided models, with the class embedding able to take the null token. FID and IS on 50k samples. §4.1 varies w; §4.2 varies puncond ∈ {0.1, 0.2, 0.5}; §4.3 varies the number of sampling steps T ∈ {128, 256, 1024}.
Best FIDs: 1.55 at 64 × 64 (w = 0.1, puncond = 0.1), against ADM's 2.07, though slightly above the cascaded diffusion model's 1.48; 2.43 at 128 × 128 (w = 0.3, T = 256 or 1024), against ADM-G's 2.97, LOGAN's 3.36 and BigGAN-deep's 5.7. (Table 2 labels the third step count 1024; the Fig. 5 legend says 512.) puncond = 0.1 and 0.2 perform about equally, and the paper finds 0.5 gives a worse trade-off: at a given FID it reaches a lower IS (its raw FID at moderate w is lower simply because its IS, and hence its guidance effect, is weaker). More sampling steps help: at w = 0.3, T = 128 gives 3.04 and T = 256 gives 2.43.
Section · §5–6 and lineage
Limitations and lineage
- Two forward passes per step. Classifier guidance can be cheaper when the classifier is smaller than the denoiser. The authors suggest injecting conditioning late in the network to share computation, leaving it to future work. In practice the two passes are batched together.
- Diversity. The authors flag that reduced diversity could be harmful where some data are underrepresented.
- Not an exact sampler. Eq. 6 does not sample any well-defined tilted distribution exactly, and high weights push intermediate states off the data manifold (over-saturation, simplified images).
Reference
Glossary
- λ
- Log signal-to-noise ratio of zλ, log(αλ2/σλ2).
- w
- Guidance strength; w = 0 is the unguided conditional model.
- puncond
- Probability of replacing the condition with the null token during training.
- ∅ (null token)
- A learned "no condition" input that turns the conditional network into an unconditional one.
- Implicit classifier
- p(c | z) ∝ p(z | c)/p(z), defined by the two generative models rather than trained separately.
- Conservative field
- A vector field that is the gradient of some scalar function; Eq. 6's field need not be.
- IS / FID
- Inception Score (higher = more confident, varied classes) and Fréchet Inception Distance (lower = closer to real statistics).