Study guide · Computer vision · 2020

Denoising Diffusion Probabilistic Models

Jonathan Ho, Ajay Jain, Pieter Abbeel · NeurIPS 2020

Can a model that only ever learns to remove a little noise generate high-quality images from scratch? Ho, Jain and Abbeel show that it can: with a fixed noise schedule and a network trained simply to guess the noise that was added, the 2015 diffusion idea produces CIFAR-10 images with a record FID of 3.17 and 256 × 256 LSUN images of similar quality to ProgressiveGAN.

The paper in four sentences

Take a picture and add random noise to it in 1,000 small steps until it is pure static. That part is fixed and needs no learning.

Train a neural network on one simple question: "here is a noisy picture and how noisy it is; what noise was added?"

To make a new picture, start from pure static and use the network 1,000 times, each time removing a little of the noise it thinks is there.

This is the 2015 diffusion model with better choices, and it was the first time diffusion images rivalled the best GANs (on CIFAR-10 it edged past them).

Idea · §1 · Fig. 2

Generating by removing noise

Before this paper, the best image generators were GANs: two networks in a contest, one forging images and one spotting fakes. They made sharp pictures but were notoriously fiddly to train. A 2015 paper had proposed a calmer alternative, the diffusion model, but its pictures were blurry.

The diffusion idea runs a film in two directions. Forward: blur a real picture with a little random noise, many times, until nothing is left but static. Backward: learn to undo one small step of that blurring. Chain the backward steps together from fresh static, and a new picture emerges.

Ho, Jain and Abbeel kept that idea and changed the details: what the network predicts, how the noise is scheduled, and how training is scored. Those choices turned out to matter enormously.

Everyday pictureA restorer cleaning a grimy painting doesn't repaint it in one stroke. They remove a thin layer of dirt, look again, remove another. DDPM generates images the same way, except it starts from a canvas that is all grime.

TakeawayDDPM generates images by starting from static and repeatedly removing a little predicted noise.

Idea · §2 · Eq. 4

Adding noise, and a shortcut

The forward process has 1,000 steps. At each one, the picture is faded very slightly and a pinch of random noise is sprinkled on. The amount of noise per step grows slowly from tiny (0.0001) to small (0.02), a "linear schedule".

A handy fact: you don't need to run the steps one by one. Because adding random noise to random noise just gives more random noise, the picture at any step t is simply "a faded copy of the original plus one dose of noise", with the exact fade and dose known in advance. That makes training fast: pick any step, jump straight there.

A 16 × 16 picture (pixel values from −1 to 1, as in the paper) jumped straight to step t using the paper's schedule. The bars show the multipliers on the original picture and on the noise. By step 1,000 almost nothing of the original is left.

TakeawayThe noisy picture at any step is a known mix of the original and fresh noise, so training can jump to any step instantly.

Check yourself
Why is the shortcut useful for training?
Each training example can be made at a random step in one calculation, instead of simulating hundreds of noise steps first.
The noise multiplier overtakes the picture multiplier at about step 260. Why is the ring still recognisable well after that?
The picture's pattern is spread consistently over many pixels while the noise is independent in each pixel, so your eye averages the noise away. Recognition fails only when the picture multiplier gets much smaller (in the figure, somewhere around steps 600 to 800).

Idea · §3.2, §3.4 · Algorithm 1

Training: guess the noise

The key choice in this paper is what the network should predict. Instead of asking "what did the slightly cleaner picture look like?", it asks "which noise was added?". Knowing the noise is equivalent to knowing the clean picture, since one is the noisy picture minus the other, but the noise version trains much better.

Training is then almost embarrassingly simple: take a real picture, pick a random step, add the right amount of random noise, show the network the noisy picture and the step number, and penalise the squared difference between its guess and the real noise. Repeat millions of times.

The figure shows what the best possible guess looks like when the "training set" is just four tiny pictures. At low noise the guess points to exactly the right picture. At high noise, the best guess is a blend of all four: the network honestly can't tell yet.

Left: the true picture. Middle: the noisy version at step t. Right: the clean picture implied by the best possible noise guess. Bottom: how much weight that guess gives each of the four training pictures. Slide from 1,000 down to see the blend resolve into one picture. Illustrative toy: a real network sees millions of images and generalises rather than choosing among four.

TakeawayThe network learns one skill: look at a noisy picture and say what noise is in it. Training is just a squared-error penalty on that guess.

Check yourself
If the network knows the noise exactly, what else does it know?
The clean picture: subtract the (scaled) noise from the noisy picture and undo the fading.
Why is a blended guess correct at high noise, rather than a mistake?
With lots of noise, several originals are plausible. Squared-error training rewards the average of the plausible answers, so blending is the best possible guess.

Idea · §3.2 · Algorithm 2

Sampling: 1,000 small clean-ups

To make a new picture, start from pure random static. Ask the network what noise is there, remove a small part of it, then (except at the very last step) sprinkle a little fresh noise back in. Do this 1,000 times, counting down from step 1,000 to step 1.

Sprinkling noise back in seems odd, but it keeps the process honest: each intermediate picture should look like a genuine noisy picture at that step, which is what the network was trained on. Watch below: the rough layout is decided early, and the fine detail last.

The real sampling algorithm (Algorithm 2, 1,000 steps, the paper's schedule), driven by the perfect noise-guesser for the four tiny training pictures. Left: the current noisy picture. Right: the clean picture it currently points to. Toy illustration: with only four pictures to learn from, the ideal model just reproduces one of them, whereas a trained network produces new ones.

TakeawayGeneration is the training skill used in a loop: guess the noise, remove some of it, repeat from static to picture.

Check yourself
Which features of the picture get decided first?
The coarse layout, at high noise levels. Fine details are settled near the end, when little noise is left.
What is the main practical cost of this method?
Speed: a picture needs 1,000 passes through the network, against one for a GAN.

Idea · §4 · Tables 1, 3

The results, and what came next

Image quality is often scored with FID: lower means the generated pictures are statistically closer to real ones. On CIFAR-10 (tiny 32 × 32 photos), DDPM scored 3.17, a new best for models not told the image class. On 256 × 256 pictures of bedrooms and churches, its quality was comparable to ProgressiveGAN, one of the strongest GANs of the time.

The paper also found something surprising: these models are not great at the "how likely is this image?" score that the 2015 paper emphasised. Most of that score is spent on details too faint for people to see.

Later refinementThe 1,000-step sampling was the obvious weakness. DDIM (Song, Meng and Ermon, 2020) reused DDPM networks with 50 steps or fewer. Dhariwal and Nichol (2021) improved the network and added guidance, beating GANs outright. Latent diffusion ran the process on compressed images, which became Stable Diffusion, and flow matching recast it as learning a velocity along straight paths.

TakeawayDDPM made diffusion work in practice, and nearly every modern image generator descends from its recipe.

Follow the idea

Built on: U-Net (2015), Diffusion models (2015)

Led to: Diffusion beats GANs (2021), Latent diffusion (2021), Diffusion Policy (2023), Flow matching (2022)