The paper in four sentences
Diffusion models that work directly on pixels spend most of their effort on fine details the eye barely notices, which makes them very expensive.
So first train a separate compressor (an autoencoder) that shrinks an image to a small grid, say 8 times smaller in each direction, and can expand it back with the details intact.
Run the diffusion model on the small grid instead of the pixels; at the end, the decoder turns its output into a full-size image.
Feed in conditions such as text through "cross-attention", and a single design handles text-to-image, inpainting, super-resolution and more, at a fraction of the cost. Its public release as Stable Diffusion, with CLIP's text encoder, put it on millions of computers.
Idea · §1 · Fig. 2
Pixels are an expensive place to work
A diffusion model makes an image by starting from random static and removing noise in many small steps, running a large neural network at every step. In the earlier designs, that network worked on every single pixel. A 512 × 512 colour image has 786,432 numbers, and the network must process all of them, hundreds of times per image.
The authors observe that most of those numbers encode fine texture: the exact grain of a carpet, the precise edge of a leaf. Those details matter for realism but carry little of the image's meaning. They describe learning as having two parts: perceptual compression (dropping detail the eye won't miss) and semantic compression (learning what the image is actually of). Pixel-space diffusion wastes enormous effort on the first.
The paper quotes the cost: the most powerful pixel diffusion models of the time needed 150 to 1,000 GPU-days to train, and generating 50,000 samples took about 5 days on a single high-end GPU.
TakeawayMost of an image's numbers are fine detail. Diffusing over all of them is costly and mostly unnecessary.
Idea · §3.1, §4.1 · Figs. 6–7
Compress first, then diffuse
Step one is an autoencoder: an encoder network shrinks the image to a small grid of numbers (the latent), and a decoder rebuilds the image from it. It is trained so that the rebuilt image looks right to people, using a perceptual loss and a GAN-style critic that punishes blurry, fake-looking patches. It is trained once and then frozen.
Step two is ordinary diffusion, but on the latent grid. The model learns to generate latents; the decoder turns a generated latent into a picture.
How much to shrink? The paper tries downsampling factors f from 1 (no compression) to 32. Too little compression and diffusion stays slow to train; too much and the autoencoder loses detail, capping quality. Factors of 4 to 16 struck a good balance, with 4 and 8 best overall.
TakeawayA learned compressor shrinks images by a factor of 4 to 16 per side; diffusion then works on that small grid, and a decoder restores the detail.
Check yourself
- Why is the autoencoder trained with a GAN-style critic rather than just "match the pixels"?
- Matching pixels on average produces blurry reconstructions. The critic and perceptual loss push the decoder to produce sharp, realistic texture, which is what makes strong compression acceptable.
- What goes wrong with f = 32?
- The compression is so strong that the autoencoder can no longer reconstruct the image faithfully, so even a perfect diffusion model would give limited quality.
Idea · §3.3 · Fig. 3
Reading the prompt: cross-attention
To control what gets generated, the model needs to read a condition: a text prompt, a class label, a sketch of where objects go. The paper turns the condition into a list of vectors (for text, one per word piece) using a separate encoder, then lets the denoising network look at them through cross-attention.
Cross-attention works like this: every position in the latent grid asks a question ("what should I be?"), every word offers an answer tag, and each position pays most attention to the words whose tags match its question. It then mixes in information from those words. Different parts of the image end up listening to different words.
Click a cell in the toy scene below to see which words it attends to.
TakeawayCross-attention lets each part of the image pull information from the words most relevant to it, which is how text steers the picture.
Check yourself
- In cross-attention, which side asks and which side answers?
- The image (latent positions) asks, through queries; the condition (words) answers, through keys and values.
Idea · §4 · Tables 1–7
One design, many jobs, much cheaper
The same recipe, with different conditions, was tested on many tasks:
- Unconditional faces (CelebA-HQ): a record quality score at the time (FID 5.11).
- ImageNet classes: better than the strongest pixel model from Dhariwal and Nichol (FID 3.60 vs 4.59) with about two-thirds of its parameters and, by the authors' account, much less training compute.
- Text-to-image: a 1.45-billion-parameter model trained on LAION-400M, close to the much larger GLIDE on COCO captions when using classifier-free guidance.
- Super-resolution, inpainting (filling holes, a new best score on the Places dataset) and layout-to-image.
Because the denoiser is convolutional, it can also be run on larger latent grids than it was trained on, producing images around a megapixel in size.
TakeawayDiffusing in a compressed space matched or beat pixel models across tasks at a fraction of the cost.
Idea · §5 and legacy
Limits, and Stable Diffusion
The authors list two limits: sampling is still step-by-step and so slower than a GAN, and the autoencoder caps how precise the output can be, which matters for tasks needing exact pixels. They also discuss misuse risks such as easier creation of fake images and the reproduction of biases in training data.
Takeaway"Compress, then diffuse" made high-quality text-to-image generation cheap enough to give away, and it remains the standard design.
Abstract, restated
Diffusion models achieve state-of-the-art synthesis by decomposing generation into sequential denoising, but operating in pixel space makes training cost hundreds of GPU-days and inference expensive. Training them in the latent space of powerful pretrained autoencoders reaches a near-optimal point between complexity reduction and detail preservation, greatly boosting visual fidelity per unit of compute. Cross-attention layers turn diffusion models into flexible generators for general conditioning inputs such as text or bounding boxes, and allow convolutional high-resolution synthesis. Latent diffusion models (LDMs) set new states of the art for image inpainting and class-conditional synthesis and are highly competitive on unconditional generation, text-to-image, and super-resolution, while needing far less compute than pixel-based diffusion models. CVPR 2022; arXiv:2112.10752.
Section · §3.1 · Figs. 1–2
Perceptual compression stage
An encoder ℰ maps x ∈ ℝH×W×3 to z = ℰ(x) ∈ ℝh×w×c, and a decoder 𝒟 reconstructs x̃ = 𝒟(ℰ(x)). The downsampling factor is f = H/h = W/w = 2m. Training combines a perceptual loss with a patch-based adversarial objective, keeping reconstructions on the image manifold and avoiding the blur of pure L1/L2 losses. To avoid high-variance latents, two regularisers are compared:
- KL-reg. A slight KL penalty towards a standard normal on the latent, as in a VAE but with a very small weight.
- VQ-reg. A vector-quantisation layer absorbed into the decoder, a VQGAN-like model.
Unlike earlier two-stage models (VQ-VAE-2, VQGAN + transformer), which flatten the latent into a 1-D sequence and therefore need aggressive compression for a tractable autoregressive prior, LDMs keep a 2-D latent and use a convolutional U-Net prior that exploits its spatial structure. So mild compression suffices, and reconstructions stay good. The autoencoder is trained once and reused across diffusion models and tasks.
Section · §3.2–3.3 · Eqs. 1–3 · Fig. 3
Latent diffusion and cross-attention conditioning
Eq. 2 is the DDPM Lsimple applied to latents; εθ is a time-conditional U-Net; samples are decoded with a single pass of 𝒟. For conditioning, a domain-specific encoder τθ maps y to τθ(y) ∈ ℝM×dτ, injected into intermediate U-Net layers by cross-attention:
φi(zt) is the flattened U-Net activation at layer i. τθ and εθ are optimised jointly through Eq. 3. For text, τθ is a transformer over BERT-tokenised input (§4.3.1). Spatially aligned conditions (semantic maps, low-resolution images, masked images) are instead concatenated to zt at the input. The attention mechanism is the transformer's, with queries from the image and keys/values from the condition.
The calculator shows what moving to latent space buys at the U-Net's input, for an image of a given size.
Section · §4.1–4.2 · Figs. 6–7 · Table 1
Choosing f; unconditional results
§4.1 trains class-conditional ImageNet LDMs with f ∈ {1, 2, 4, 8, 16, 32} (LDM-1 is pixel space), all on a single A100 with the same number of steps and parameters. Small f (LDM-1, 2) trains slowly; after 2M steps there is an FID gap of 38 between LDM-1 and LDM-8. Large f (LDM-32) stagnates at low fidelity after few steps, because the first stage compresses too much. "LDM-{4-16} strike a good balance"; for sample quality against sampling steps (Fig. 7), LDM-4 and LDM-8 offer the best conditions.
| Unconditional, 256² (Table 1) | Model | FID | Note |
|---|---|---|---|
| CelebA-HQ | LDM-4 (500 DDIM steps) | 5.11 | new state of the art |
| FFHQ | LDM-4 (200 steps) | 4.98 | |
| LSUN-Churches | LDM-8 | 4.02 | |
| LSUN-Bedrooms | LDM-4 | 2.95 | ADM is better (1.90) |
Section · §4.3–4.5 · Tables 2–7, 9
Conditional results
| Task | Model | Result | Comparison |
|---|---|---|---|
| Text-to-image, MS-COCO 256² (Table 2) | LDM-KL-8-G, 1.45B params, LAION-400M, 250 DDIM steps, guidance s = 1.5 | FID 12.63 | unguided 23.31; GLIDE (6B) 12.24 |
| Class-conditional ImageNet 256² (Table 3) | LDM-4-G, 400M params | FID 3.60, IS 247.67 | ADM-G (608M) 4.59 |
| 4× super-resolution, ImageNet (Table 5) | LDM-4, 169M, 100 steps | FID 2.8 | SR3 (625M) 5.2; SR3 has higher IS |
| Inpainting, Places 512² (Table 7) | LDM-4 big, fine-tuned | FID 1.50 | LaMa 2.21 |
| Layout-to-image, COCO (Table 9) | LDM-4 fine-tuned from OpenImages | FID 40.91 |
Against ADM on ImageNet, Against ADM-G on ImageNet, LDM-4-G uses about two-thirds of the parameters (400M vs 608M) and, per the authors, significantly less training compute (App. E.3.5). The paper's "half the parameters, 4× less compute" figure refers to LSUN-Bedrooms. For inpainting, latent models train and sample at least 2.7× faster than pixel models while improving FID by at least 1.6× (§4.5, Table 6). Convolutional sampling (§4.3.2). Models conditioned on spatially aligned inputs and trained at 256² generalise to larger latent grids, e.g. 512 × 1024 landscapes from semantic maps (Fig. 9), and the paper claims consistent images around 1024² px; the scale of the latent (KL-reg needs rescaling) affects how well this works.
Check yourself
- Why does LDM not beat ADM on LSUN-Bedrooms, but beat it on class-conditional ImageNet?
- The paper does not give a single cause. One reading: unconditional bedrooms reward exact fine texture, which the autoencoder's reconstruction ceiling limits, while class-conditional ImageNet rewards semantic composition and benefits from guidance, where the latent model's compute savings let it train a strong prior.
Section · §5 and lineage
Limitations and lineage
- Sequential sampling. Still slower than GANs.
- Reconstruction ceiling. When fine-grained pixel accuracy is needed, the first stage can be the bottleneck; the authors suspect their super-resolution models are somewhat limited by this.
- Societal impact. The paper discusses easier creation of manipulated images and disinformation, reproduction of data biases, and the possibility of revealing training data.
Reference
Glossary
- ℰ, 𝒟
- Encoder and decoder of the first-stage autoencoder.
- f
- Spatial downsampling factor H/h; LDM-f names the model.
- KL-reg / VQ-reg
- Latent regularisation by a small KL penalty, or by vector quantisation.
- τθ
- Condition encoder producing the keys and values for cross-attention.
- Cross-attention
- Attention whose queries come from one sequence (image latents) and keys/values from another (condition tokens).
- Perceptual loss
- Distance between deep-network features of two images, closer to human judgements than pixel distance.
- -G suffix
- Sampled with classifier-free guidance.