Study guide · Computer vision · 2015

Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, Surya Ganguli · ICML 2015

How do you build a model that can invent new, realistic data but is still simple enough to train and to score? Sohl-Dickstein and colleagues borrow an idea from physics: destroy the structure in data by adding a little noise many times over, then train a neural network to undo each small step. Run the learned undo steps starting from pure noise and you get new samples. This is the original diffusion model.

The paper in four sentences

Models that generate data usually have to choose between being flexible (able to capture complicated data) and being tractable (easy to train, sample from and evaluate). This paper tries to get both.

Start with real data and blur it with a tiny bit of random noise, over and over, until nothing but noise is left. That direction is easy and needs no learning.

The trick is that each single step is so small that undoing it is also simple: a bell-shaped guess. A neural network learns those small undo steps.

Chain the learned undo steps together, starting from pure noise, and structure reappears: new samples of the data. The idea lay mostly dormant until 2020, when DDPM made it produce excellent images.

Idea · §1

The generative modelling dilemma

A generative model is a program that has learned what a kind of data looks like, so it can make up new examples (a new face, a new handwritten digit) and say how likely a given example is. Formally it learns a probability distribution: a rule that assigns every possible image a plausibility score.

The paper opens with a tension. Tractable models, such as a single bell curve, are easy to fit and easy to evaluate, but far too simple for real images. Flexible models can describe almost anything, but computing their scores usually involves a sum over astronomically many possibilities, so training and sampling become slow or approximate.

The authors' goal is a model that is flexible and still tractable: you can train it, draw samples from it, and compute (or at least tightly bound) how likely the data is under it.

Everyday pictureDescribing a city with one sentence ("it's round-ish") is easy but useless. Describing every brick is complete but impossible to work with. The paper looks for a description built out of many tiny, simple pieces that add up to something detailed.

TakeawayThe paper wants a model of data that is rich enough for real images but still easy to train, sample and score.

Idea · §2.1 · Fig. 1

Destroy the data slowly with noise

Picture a drop of ink in a glass of water. Over time the molecules jiggle randomly and the drop spreads out until the water is evenly tinted. Physicists call this diffusion. Every trace of the drop's original shape is gone.

The paper does the same to data. Take a data point (say, a dot on a spiral) and at every step shrink it very slightly towards the centre and add a pinch of random noise. After enough steps, every dot ends up in a featureless round cloud, the familiar bell-shaped Gaussian distribution. This is the forward process. It is fixed in advance and involves no learning at all.

Drag the slider to watch a spiral (the paper's first toy example is a "Swiss roll" like this) dissolve over 40 steps, the number the paper uses for this toy.

Left: 300 points after the chosen number of steps. Right: how much of each point's original position survives (amber) versus how much is accumulated noise (blue). Each step uses the fixed schedule the paper gives for binomial diffusion, βt = 1/(40 − t + 1), applied here to Gaussian noise (footnote 2 reports this works too), so the signal is exactly gone at step 40. The paper's own Swiss-roll model learns its schedule instead.

TakeawayAdding a little noise many times turns any data into plain noise. This direction is easy and needs nothing learned.

Check yourself
Why does the forward process need no training?
It is a fixed recipe (shrink slightly, add a known amount of noise). Nothing about it depends on a model; only the reverse direction is learned.
Where do all the points end up after many steps, whatever data you start with?
In the same simple, round Gaussian cloud. That is what makes it a good starting point for generating.

Idea · §2.2

Small steps are easy to undo

Running the movie backwards, from noise to data, sounds hopeless: you can't un-stir ink. But the paper relies on an old mathematical fact (it cites Feller, 1949): if each forward step is tiny, then the reverse of that step has the same simple form. Given a slightly noisy point, the place it probably came from a moment ago is described by a single bell curve, nudged in some direction.

That turns an impossible problem into many easy ones. A neural network only needs to learn, for each step and each location, where the centre of that bell curve is and how wide it is.

Try it below. The data here is one-dimensional: values cluster around −1 and +1. We take a noisy value and ask "where was it a few steps earlier?". The answer is computed exactly. With a small jump it is one tidy bump (the dashed bell curve fits it). With a big jump it splits into two bumps, which a single bell curve cannot describe.

Data: two narrow clusters at −1 and +1. We start at step 20 of 40 (half the signal gone). The solid amber curve is the exact probability of each earlier value; the dashed blue curve is the best single bell curve. Try "Steps undone at once" = 20 with the noisy value near 0: the true answer is "near −1 or near +1", and a single bell curve misplaces it in the middle.
Everyday pictureIf you see a footprint in the sand and are asked where the walker was one second earlier, "about one stride back" is a good answer. Asked where they were an hour earlier, the honest answer is a complicated map of possibilities. Small steps keep the answer simple.

TakeawayEach tiny noise step can be reversed with a simple bell-curve guess, so the hard reverse journey becomes many easy, learnable steps.

Check yourself
Why not reverse the whole process in one go?
Over a large jump, the set of places a noisy point could have come from is complicated (here, two separate clusters). Only for small jumps is it a simple bell curve that a network can easily output.
What does the network actually have to output at each step?
The centre and the width of the bell curve for the previous step, given the current noisy point and the step number.

Idea · §2.2–2.4 · Figs. 1–2

Generate by walking back from noise

To make a new sample: draw a point from the plain noise cloud, then apply the learned undo step, then again, and again, all the way back to step 0. Each undo step removes a little noise and adds a little structure. After the last step, the point looks like data.

Training checks how well the chain of undo steps retraces the forward process. Because every piece is a simple bell curve, the paper can compute a guaranteed lower bound on how likely the data is under the model, and improve the network by pushing that bound up.

Below, the "network" is replaced by the best possible bell-curve step, computed exactly, so you can see the effect of the number of steps on its own. With few, big steps the samples come out in the wrong places. With many small steps they land on the two clusters.

Each sample starts as pure noise and takes the chosen number of reverse steps, each the best single bell curve for that step. Grey outline: the real data distribution. Amber bars: the generated samples. "Matches the data" is the overlap between the two shapes. Toy illustration; the real paper learns the steps with a neural network.

TakeawayNew data comes from running the learned small undo steps from pure noise. More, smaller steps make each one simpler and the result better.

Check yourself
With two steps, why do many samples land between the clusters?
Each step can only move a sample by a bell-curve-shaped amount. With so few steps, the last step has to cover a large jump whose true answer has two bumps, and a single bell curve averages them.
What does the model start from when it generates?
Pure random noise from the simple Gaussian cloud that the forward process ends in.

Idea · §2.5, §3 · Figs. 3–5

Repairing images, and what happened next

Because the model is built from simple steps, it can be combined with other information. The paper uses this to clean up noisy images (CIFAR-10 photos with heavy added noise, Fig. 3) and to fill in a missing patch of a bark texture (Fig. 5), by steering each reverse step towards what is already known.

The authors tested the approach on toy data, handwritten digits (MNIST), small photos (CIFAR-10), and natural textures. The samples were recognisable but blurry by later standards, and the method attracted little attention for five years.

Later refinementIn 2020, DDPM (Ho, Jain and Abbeel) kept the same forward-and-reverse structure but fixed the noise schedule, had the network predict the added noise instead of the bell curve's centre, and simplified the training loss. The results rivalled the best image generators of the day, and diffusion became the dominant way to generate images.

TakeawayThis paper set out the full diffusion recipe; five years of engineering later, the same recipe powers modern image generators.

Follow the idea

Led to: DDPM (2020)