Study guide · Computer vision · 2021

High-Resolution Image Synthesis with Latent Diffusion Models

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer · CVPR 2022

Pixel-space diffusion models made excellent images but cost hundreds of GPU-days to train and minutes to sample. Rombach and colleagues split the job in two: an autoencoder first compresses images into a much smaller "latent" grid that keeps what the eye cares about, and the diffusion model then works only in that small space. Add cross-attention so the model can read text, and you have the design that became Stable Diffusion.

The paper in four sentences

Diffusion models that work directly on pixels spend most of their effort on fine details the eye barely notices, which makes them very expensive.

So first train a separate compressor (an autoencoder) that shrinks an image to a small grid, say 8 times smaller in each direction, and can expand it back with the details intact.

Run the diffusion model on the small grid instead of the pixels; at the end, the decoder turns its output into a full-size image.

Feed in conditions such as text through "cross-attention", and a single design handles text-to-image, inpainting, super-resolution and more, at a fraction of the cost. Its public release as Stable Diffusion, with CLIP's text encoder, put it on millions of computers.

Idea · §1 · Fig. 2

Pixels are an expensive place to work

A diffusion model makes an image by starting from random static and removing noise in many small steps, running a large neural network at every step. In the earlier designs, that network worked on every single pixel. A 512 × 512 colour image has 786,432 numbers, and the network must process all of them, hundreds of times per image.

The authors observe that most of those numbers encode fine texture: the exact grain of a carpet, the precise edge of a leaf. Those details matter for realism but carry little of the image's meaning. They describe learning as having two parts: perceptual compression (dropping detail the eye won't miss) and semantic compression (learning what the image is actually of). Pixel-space diffusion wastes enormous effort on the first.

The paper quotes the cost: the most powerful pixel diffusion models of the time needed 150 to 1,000 GPU-days to train, and generating 50,000 samples took about 5 days on a single high-end GPU.

Everyday pictureAn architect sketches a building's layout on a small plan, not by placing every brick. The bricks are filled in later by a builder who knows how bricks go. Latent diffusion makes the diffusion model the architect and the decoder the builder.

TakeawayMost of an image's numbers are fine detail. Diffusing over all of them is costly and mostly unnecessary.

Idea · §3.1, §4.1 · Figs. 6–7

Compress first, then diffuse

Step one is an autoencoder: an encoder network shrinks the image to a small grid of numbers (the latent), and a decoder rebuilds the image from it. It is trained so that the rebuilt image looks right to people, using a perceptual loss and a GAN-style critic that punishes blurry, fake-looking patches. It is trained once and then frozen.

Step two is ordinary diffusion, but on the latent grid. The model learns to generate latents; the decoder turns a generated latent into a picture.

How much to shrink? The paper tries downsampling factors f from 1 (no compression) to 32. Too little compression and diffusion stays slow to train; too much and the autoencoder loses detail, capping quality. Factors of 4 to 16 struck a good balance, with 4 and 8 best overall.

Downsampling factor f:

Left: a 512 × 512 image; each square is one latent position (shown up to a 64 × 64 grid; finer grids are drawn shaded). The autoencoder stores a few numbers per position and its decoder recreates the detail inside each square. Right: grid positions the diffusion model must process, on a log scale, and the paper's verdict for that factor (§4.1, Figs. 6–7).

TakeawayA learned compressor shrinks images by a factor of 4 to 16 per side; diffusion then works on that small grid, and a decoder restores the detail.

Check yourself
Why is the autoencoder trained with a GAN-style critic rather than just "match the pixels"?
Matching pixels on average produces blurry reconstructions. The critic and perceptual loss push the decoder to produce sharp, realistic texture, which is what makes strong compression acceptable.
What goes wrong with f = 32?
The compression is so strong that the autoencoder can no longer reconstruct the image faithfully, so even a perfect diffusion model would give limited quality.

Idea · §3.3 · Fig. 3

Reading the prompt: cross-attention

To control what gets generated, the model needs to read a condition: a text prompt, a class label, a sketch of where objects go. The paper turns the condition into a list of vectors (for text, one per word piece) using a separate encoder, then lets the denoising network look at them through cross-attention.

Cross-attention works like this: every position in the latent grid asks a question ("what should I be?"), every word offers an answer tag, and each position pays most attention to the words whose tags match its question. It then mixes in information from those words. Different parts of the image end up listening to different words.

Click a cell in the toy scene below to see which words it attends to.

Prompt:

Each latent cell has a hand-made "query" describing what kind of region it is (sky-like, ground-like, object-like); each word has a "key" of the same kind. Bars: softmax(query · key × sharpness) for the selected cell. Toy illustration of the mechanism; in the real model queries and keys are learned vectors of hundreds of dimensions, with many heads and layers.

TakeawayCross-attention lets each part of the image pull information from the words most relevant to it, which is how text steers the picture.

Check yourself
In cross-attention, which side asks and which side answers?
The image (latent positions) asks, through queries; the condition (words) answers, through keys and values.

Idea · §4 · Tables 1–7

One design, many jobs, much cheaper

The same recipe, with different conditions, was tested on many tasks:

  • Unconditional faces (CelebA-HQ): a record quality score at the time (FID 5.11).
  • ImageNet classes: better than the strongest pixel model from Dhariwal and Nichol (FID 3.60 vs 4.59) with about two-thirds of its parameters and, by the authors' account, much less training compute.
  • Text-to-image: a 1.45-billion-parameter model trained on LAION-400M, close to the much larger GLIDE on COCO captions when using classifier-free guidance.
  • Super-resolution, inpainting (filling holes, a new best score on the Places dataset) and layout-to-image.

Because the denoiser is convolutional, it can also be run on larger latent grids than it was trained on, producing images around a megapixel in size.

TakeawayDiffusing in a compressed space matched or beat pixel models across tasks at a fraction of the cost.

Idea · §5 and legacy

Limits, and Stable Diffusion

The authors list two limits: sampling is still step-by-step and so slower than a GAN, and the autoencoder caps how precise the output can be, which matters for tasks needing exact pixels. They also discuss misuse risks such as easier creation of fake images and the reproduction of biases in training data.

Later refinementIn August 2022 the authors' group, with Stability AI and Runway, released Stable Diffusion: a latent diffusion model with f = 8 and a 4-channel latent, trained on a subset of LAION-5B, that swapped the paper's own text transformer for OpenAI's frozen CLIP text encoder. Released with its weights, it could run on a consumer graphics card, and it became the base of a huge ecosystem of fine-tuned models and tools. Later versions kept the latent idea but changed the denoiser (larger U-Nets in SDXL, then transformers trained with flow matching in Stable Diffusion 3).

Takeaway"Compress, then diffuse" made high-quality text-to-image generation cheap enough to give away, and it remains the standard design.

Follow the idea

Built on: VAE (2013), DDPM (2020), Classifier-free guidance (2021)