Study guide · Robotics & reinforcement learning · 2023

Diffusion Policy: Visuomotor Policy Learning via Action Diffusion

Cheng Chi et al. · RSS 2023

When people demonstrate a task to a robot, they don't always do it the same way: sometimes they go left around an obstacle, sometimes right. A robot that learns by averaging the demonstrations goes straight into the obstacle. Chi and colleagues borrow the machinery of image diffusion models to generate a whole short sequence of robot actions from noise, which captures all the ways of doing the task and commits to one of them. Across 12 tasks from four benchmarks (15 in the 2024 extended version), it beat the previous best methods by 46.9% on average.

The paper in four sentences

A common way to teach robots is by demonstration: a person controls the robot a few hundred times and the robot learns to copy what it sees ("behaviour cloning").

Demonstrations often solve a task in more than one way, and standard learners blur those ways together, producing actions that match none of them.

Diffusion Policy treats the next stretch of robot motion like an image to be generated: start from random noise and refine it step by step, guided by what the cameras see. Like an image generator, it can produce any of the valid options, cleanly.

It plans about a second of motion at a time, carries out part of it, then plans again. This made it markedly more reliable in simulation and on real robots, and the same idea now sits inside large robot models such as π0.

Idea · §1, §4.1 · Fig. 3

The averaging trap

In behaviour cloning, a robot is shown many examples of "what the cameras saw" paired with "what the person made the robot do", and a neural network learns to predict the action from the observation. The simplest version predicts a single best action, trained to be as close as possible to the demonstrated ones.

That works when there is one right answer. But suppose half the demonstrations go left around a cup and half go right. The prediction that is "closest on average" to both is straight down the middle, into the cup. In the jargon, the action distribution is multimodal (it has several separate peaks), and a single averaged prediction lands between them.

Below, the dashed lines are demonstrations of getting from the bottom to the top around an obstacle. Compare the averaged prediction with samples from a diffusion model trained on the same demonstrations.

Dashed: 24 demonstrated paths of 16 waypoints each, half passing left of the obstacle, half right. Blue: the average path, what a squared-error regression learns. Amber: paths generated by a diffusion process (50 denoising steps) using the exact best denoiser for these demos. Each click adds a new sample. Toy illustration, not the paper's task.
Everyday pictureAsk two friends for directions round a lake: one says "go left", the other "go right". Averaging their advice tells you to swim.

TakeawayWhen demonstrations solve a task in several ways, predicting the average action can be worse than any of them. The learner must be able to choose one way.

Check yourself
Why does a squared-error learner end up in the middle?
For a given observation, the single prediction that minimises the average squared distance to all demonstrated actions is their mean, which here lies between the two routes.
What does "multimodal" mean here?
For the same situation, there are several distinct good actions (modes), such as going left or going right.

Idea · §2 · Fig. 1

Robot motion, generated like an image

Image diffusion models generate a picture by starting from random static and removing noise step by step. Diffusion Policy applies the same procedure to robot actions. The thing being generated is not a picture but a short action sequence: the next 16 target positions for the robot's hand, say. The noise-removing network is also given what the robot currently sees, so the actions it generates fit the situation.

Because a diffusion model learns the whole spread of possible outputs, not just their average, each run lands on one of the demonstrated ways. Which one depends on the random starting noise. And because the whole sequence is generated together, it is a coherent plan rather than a jumble of steps from different strategies.

Drag the slider to watch one action sequence emerge from noise.

Step 50 is pure noise; step 0 is the finished plan. Amber dots: the 16 waypoints at the chosen step. Faint line: the clean plan the denoiser currently expects. The same demonstrations and exact denoiser as above. Toy illustration.

TakeawayDiffusion Policy generates a whole sequence of future actions by denoising, conditioned on what the robot sees, so it can represent several valid strategies and commit to one.

Check yourself
In Diffusion Policy, what plays the role of the "image"?
A short sequence of future robot actions (for example, target positions for the next 16 time steps).
What decides whether a given run goes left or right?
The random noise it starts from (and the randomness added during denoising). Different noise leads to different valid plans.

Idea · §2.3, §4.3 · Figs. 2, 5

Plan a stretch, do part of it, plan again

Generating takes time, many network passes per plan, so the robot can't afford to plan from scratch at every tiny time step. And committing to a very long plan would make it slow to react if something changes. The paper's compromise, borrowed from control engineering, is receding-horizon control: look at the last couple of camera frames, plan the next 16 steps, carry out the first 8, then look again and make a new plan.

Carrying out several steps of one plan has a second benefit: the robot sticks with one strategy instead of flip-flopping between "left" and "right" at every moment. The paper found 8 executed steps to be the best setting for most of its tasks; fewer gave jittery motion, more made the robot slow to react.

Each row is one planning cycle over a 48-step episode. Blue: the 2 most recent observations it looks at. Grey: the 16 steps it predicts. Amber: the steps it actually carries out before re-planning. Fewer executed steps means more re-planning (more reactive, more computation); more means longer commitment.

TakeawayDiffusion Policy plans 16 steps, executes about 8, and re-plans: a balance between consistency and reacting to change.

Idea · §5–7 · Tables 1–6

Results in simulation and on real robots

The authors compared Diffusion Policy with three strong earlier methods: a recurrent network that outputs a mixture of bell curves (LSTM-GMM), an "implicit" energy-based policy (IBC), and a transformer that clusters actions (BET). Across 12 simulated tasks (15 in v5) from four benchmarks, from lifting objects to multi-stage kitchen chores, it improved on the best of them by 46.9% on average.

On real robots: pushing a T-shaped block into place (95% success, the baselines 0 to 20%), flipping a mug (90% vs 0% for LSTM-GMM), and pouring and spreading sauce on pizza dough. An extended version adds two-armed tasks such as unrolling a mat and folding a shirt (75% success on each).

Two practical findings stood out. Commanding target positions worked better than commanding velocities, the opposite of what earlier methods preferred. And training was stable: unlike the energy-based method, its performance did not swing wildly between training checkpoints.

TakeawayDiffusion Policy outperformed the best previous behaviour-cloning methods by a wide margin, in simulation and on real hardware.

Idea · §9 and legacy

Limits, and where the idea went

Diffusion Policy still only imitates: it needs good demonstrations and can't improve beyond them on its own. And generating each plan takes many network passes, which makes it slower and more computationally demanding than simpler policies, an issue for very fast control loops.

Later refinement"Generate a chunk of actions with a generative model" became a standard design for robot learning. Large vision-language-action models such as π0 (2024) generate action chunks with flow matching, a close cousin of diffusion that needs fewer steps; others speed up diffusion policies with consistency-model distillation. Action chunking was also central to ACT (Zhao et al., 2023), developed in parallel for low-cost bimanual arms.

TakeawayTreating robot actions as something to generate, not just predict, handled multiple strategies cleanly and shaped how today's robot foundation models produce actions.

Follow the idea

Built on: DDPM (2020)

Led to: π0 (2024)