Study guide · Computer vision · 2022

Flow Matching for Generative Modeling

Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, Matt Le · ICLR 2023

Diffusion models turn noise into data by learning to undo a noising process. Lipman and colleagues ask a more direct question: why not learn the velocity that carries each bit of noise to data, and choose the simplest possible paths for it to follow? Their flow matching objective trains such a velocity field with a plain regression loss, and with straight-line paths it trains faster and samples in fewer steps than diffusion.

The paper in four sentences

Picture generation as moving a cloud of random points (noise) until it takes the shape of the data, like wind blowing sand into the shape of dunes.

A neural network learns the "wind": at every place and moment, which direction and how fast a point should move. Generating means releasing a noise point and letting it drift.

The trick that makes training easy: pair a random noise point with a random data point, draw a path between them, and teach the network the speed along that path. Averaged over many pairs, it learns the right wind for the whole cloud.

Choosing straight-line paths, rather than the curved paths diffusion implies, gives a simpler wind that needs fewer steps to follow. Recent image models and the robot model π0 use this.

Idea · §1–2

Generating as moving a cloud of points

Every generative model has to turn something simple and random into something that looks like real data. Diffusion models do it by learning to remove noise step by step. There is another way to see the same job: as transport. Start with a cloud of random points, then move every point smoothly over a fixed stretch of time until the cloud has the shape of the data.

A rule that says, for every location and every moment, which way and how fast to move is called a velocity field (or vector field). If a neural network can represent that field, generating a sample is just following it: start at a random point and take small steps in the direction the network says. Models built this way are called continuous normalising flows. They were known before this paper, but they were slow and awkward to train, because training required simulating the whole journey.

Everyday pictureThink of a weather map of wind arrows. Drop a feather anywhere and the arrows tell you where it drifts. Flow matching learns a wind map that blows a pile of randomly scattered feathers into the shape of, say, a face.

TakeawayGeneration can be seen as moving noise into data along a learned velocity field; the challenge is training that field efficiently.

Idea · §4.1 · Figs. 2–4

Straight paths beat curved ones

Many different velocity fields can carry the noise cloud to the data. The paper's flexible framework lets you choose the paths. One choice reproduces the curved paths that diffusion models follow. Another, which the paper calls the optimal transport path, moves each point in a straight line at constant speed.

Straight paths matter because generating means following the field in a limited number of steps. On a curved path, each straight step overshoots the bend; on a straight path, even one big step lands close to where it should. Below, both kinds of flow carry the same noise to the same three data clusters, using the exact velocity field for each. Lower the number of steps and see which holds up.

500 noise points follow the exact velocity field with the chosen number of simple straight-line steps (Euler steps). Faint lines: the routes of 50 of them. Dashed circles: the three data clusters. With 4 steps the straight-path flow lands roughly 90% of points on clusters, the diffusion-path flow fewer than half. With a single step both fail: from pure noise, one step can only aim at the average of the data. Toy illustration in two dimensions, with exact rather than learned velocities.

TakeawayStraight-line paths make the velocity field simple, so it can be followed accurately in few steps.

Check yourself
Why does a curved route need more steps to follow?
Each step moves in a straight line in the current direction. On a curve the direction keeps changing, so big steps cut corners and go off course; on a straight route the direction doesn't change.

Idea · §3 · Theorems 1–2

The training trick: learn from single pairs

Here is the problem. To train the network you would want to show it the correct overall velocity at each place and time, but nobody knows that field; it is exactly what we're trying to find.

The paper's insight: pick one noise point and one data point, join them with a simple path, and use the velocity along that path as the training target. For a straight path the target is simply "data point minus noise point". Each individual target is "wrong", since a point in the middle of the cloud could be heading to many places. But the network, trained with a squared-error loss over millions of random pairs, learns the average velocity at each place and time. The paper proves that this average is exactly the field that moves the whole cloud correctly.

In the figure, time runs left to right. Grey lines are individual training pairs: they cross each other freely. The amber curves are what the learned, averaged field actually does: they never cross, and each noise point is routed smoothly to one of the two data clusters.

One-dimensional data at −1 and +1 (right edge), noise from a bell curve (left edge). Grey: 40 random noise-to-data pairs joined by straight lines (training targets). Amber: 25 paths of the exact averaged velocity field, started from evenly spread noise values. Toy illustration.

TakeawayTrain on simple single-pair targets; the squared-error loss averages them into the correct velocity field for the whole distribution.

Check yourself
Why can the learned paths not cross, even though the training pairs do?
The learned field gives one velocity per place and time. Two paths meeting at the same place and time would have to continue identically, so they cannot cross.
What is the training target for a straight path from noise point A to data point B?
The constant velocity B − A (with a tiny correction in the paper for a minimum noise level).

Idea · §6 · Tables 1–2

How well it worked

The authors trained one U-Net architecture with different objectives on CIFAR-10 and downsampled ImageNet (32, 64 and 128 pixels). Flow matching with straight paths gave the best likelihood and image-quality scores among their trained baselines, and generated images with fewer network evaluations: on ImageNet 64 × 64, an FID of 14.45 using about 138 evaluations, against 17.36 and about 264 for the DDPM baseline. It also trained faster and handled 4× super-resolution well.

Even training the network with diffusion-shaped paths, but via flow matching, worked better than the usual score-matching loss, showing the training method itself is robust.

TakeawayFlow matching with straight paths matched or beat diffusion training in quality while sampling with fewer steps.

Idea · §5, §7 and legacy

A simpler alternative to diffusion

Flow matching does not throw diffusion away; it contains it. Diffusion paths are one choice inside the framework, and the training loss is a cousin of diffusion's noise-prediction loss. What it adds is freedom to choose the path, and a clean, simulation-free way to train any of them.

The paper was one of several in 2022 that reached closely related ideas at the same time, notably "rectified flow" (Liu and colleagues) and "stochastic interpolants" (Albergo and Vanden-Eijnden).

Later refinementStraight-path flow matching (often called rectified flow) became a standard way to train large generators: Stable Diffusion 3 and Flux for images, Meta's Movie Gen for video, and robot policies such as π0, which generates action sequences with a flow-matching "action expert", much as Diffusion Policy did with diffusion. Note that only the individual training paths are straight; the learned overall paths are generally curved, which later work such as rectification and distillation addresses to reach one or two steps.

TakeawayFlow matching reframed diffusion as learning velocities along chosen paths, and straight paths became the default for many state-of-the-art generators.

Follow the idea

Built on: DDPM (2020)

Led to: π0 (2024)