Study guide · Robotics & reinforcement learning · 2024

π0: A Vision-Language-Action Flow Model for General Robot Control

Kevin Black et al. (Physical Intelligence) · arXiv

How do you get one model to fold laundry, bus tables and build boxes on several different robots? π0 starts from a vision-language model that already understands images and words, bolts on a small "action expert" that turns noise into smooth arm movements by flow matching, and trains the whole thing on over 10,000 hours of robot data from seven robot types. The result handles long, dexterous tasks that earlier robot models could not.

The paper in four sentences

Earlier "robot brains" built from language models wrote movements as a handful of numbered tokens, a few times per second: good for pick-and-place, too coarse and slow for folding a shirt.

π0 keeps a pretrained vision-language model to understand the camera images and the instruction, but adds a separate, smaller network that produces a whole second's worth of smooth movement (50 small steps) at once.

That network is trained with flow matching: it learns to turn random noise, step by step, into a realistic movement, which lets it represent "there are two good ways to do this" without averaging them into a bad one.

Pretrained on a large mixture of data from seven robot configurations, then fine-tuned on high-quality demonstrations, it folds laundry from a basket, buses tables and assembles boxes, outperforming earlier generalist robot models.

Idea · §I, §IV · Fig. 3

A language-model brain with a motor cortex attached

π0 has two parts that share one network. The first is PaliGemma, a 3-billion-parameter vision-language model (VLM) from Google that was trained on web images and text. It reads the robot's camera images (up to three cameras) and the instruction, such as "fold the shirt".

The second part is an action expert of about 300 million parameters, trained from scratch. It reads the robot's current joint positions and everything the VLM has worked out, and outputs the next 50 movement commands. This bundle of upcoming commands is called an action chunk. At 50 commands per second, one chunk covers a second of motion.

The two parts are separate sets of weights, but they talk through attention inside one transformer, so the action expert can "look at" what the VLM noticed in the image.

camera images "fold the shirt" joint angles noisy actions VLM (PaliGemma)≈3B parameters, pretrained Action expert≈300M parameters, new attention 50-step chunk≈1 s of motion repeat 10×
Simplified from the paper's Fig. 3. The action expert is run ten times per chunk, each time refining the noisy actions a little (next idea).

Takeawayπ0 = a pretrained vision-language model for understanding plus a small new network that writes a second of smooth movement at a time.

Idea · §IV · Eq. 1–2

From noise to movement: flow matching

Suppose a cup is in the way and the arm can pass it on the left or on the right. Both are fine. A model trained to predict "the best guess" for each moment learns the average of the demonstrations, which is straight through the cup.

Flow matching avoids this. During training, the model is shown a real movement blended with random noise and learns which direction points from the noise towards the real movement. To act, it starts from pure noise and takes a few small steps in the direction it has learned, ten in π0's case. Different starting noise ends up at different good answers: some runs go left, some go right, and almost none go through the middle.

A one-number action (sideways position of the hand). Demonstrations go left (top band) or right (bottom band) of an obstacle (shaded). Each line is one sample: it starts as random noise on the left (time 0) and follows the flow to an action on the right (time 1). The flow used here is the exact ideal one for this two-band data, computed in closed form, standing in for what π0's network learns. With one step every sample lands on the average, in the obstacle; with ten it splits cleanly.
Everyday pictureA sculptor starts with a rough block and removes a little at a time. Flow matching starts with static on a TV screen and nudges it, a few times, into a picture. Which picture you get depends on the static you started with.

TakeawayFlow matching turns noise into an action in a few small steps, so it can produce any of several good movements instead of their bad average.

Check yourself
Why does a single step send every sample to the middle?
At time 0 the input is pure noise and carries no information about which side to choose, so the best single prediction of the destination is the average of all demonstrations. Later steps can see which way the sample has started to lean.
Why did RT-2-style models with token outputs not hit this averaging problem as badly?
Tokens give a probability over 256 bins per dimension, which can also have two peaks. Their weakness is coarseness and speed, not averaging. Flow matching gives continuous values and a whole chunk at once.

Idea · §I, §IV, App. D

A second of motion at a time

Laundry folding needs fast, fine motion. π0 controls robots at up to 50 commands per second. Running a 3-billion-parameter model fifty times a second is not practical, so instead π0 predicts a chunk of 50 future commands in one go (about 73 milliseconds on a single gaming graphics card) and the robot plays out part of the chunk before asking for the next.

The trade-off is responsiveness. While the robot is playing out a chunk, it is not looking. Asking more often reacts faster but costs more computing.

Illustrative. Two seconds at 50 commands per second. The target (dashed) jumps at 0.9 s, say because someone nudged the shirt. Each new chunk (marked by a tick) aims at wherever the target is when it is computed, and the hand (amber) follows the chunk. The paper does not tie π0 to one re-planning interval; this shows the general trade-off.

TakeawayPredicting a chunk of 50 commands makes 50-per-second control affordable, at the price of reacting only when a new chunk is computed.

Idea · §V · Fig. 4

Learn broadly, then polish

The training recipe copies how language models are made. Pre-training uses a huge, varied mixture: over 10,000 hours of the company's own robot data across 68 tasks and seven robot configurations (single arms, pairs of arms, arms on mobile bases), plus about 9% public data including Open X-Embodiment. This teaches broad abilities and how to recover from mistakes, because messy data contains many.

Post-training then fine-tunes on a smaller set of careful, high-quality demonstrations of one task, from a few hours to over 100 hours, to make the robot fluent and efficient at it. The authors found both stages matter: the pretrained-only model is robust but clumsy; a model trained only on the polished data is fluent but brittle.

Everyday pictureA chef's training: years of cooking everything (broad, messy experience), then weeks perfecting one restaurant's menu. Either alone gives a worse cook.

TakeawayDiverse, imperfect data teaches robustness; a little perfect data teaches fluency. π0 uses both, in that order.

Idea · §VI–VII

What it can do, and what is still open

Straight after pre-training, π0 beat earlier open generalist robot models (OpenVLA and Octo) on every test task, including shirt folding and clearing tables. After post-training it did long, multi-stage jobs: taking laundry out of a dryer, folding it, bussing a table, assembling a cardboard box, and packing eggs into a carton. On every one of these the full model scored above half of the maximum possible points, which earlier methods could not approach.

For long jobs it can be paired with a higher-level language model that breaks a request into steps ("pick up the napkin", "put the cup in the bin"), an echo of SayCan.

The authors are frank about the limits: nobody yet knows how to choose and weight the pre-training data, not every task works reliably, and it has not been tried on very different domains such as driving or walking robots.

Later refinementThe same group released π0-FAST (2025), which goes back to tokens but compresses each action chunk with a frequency transform so tokens stay efficient, and π0.5 (2025), which trains on more varied data so that the robot can clean kitchens and bedrooms in homes it has never seen. The weights of π0 were released openly in early 2025.

Takeawayπ0 showed general robot models can do long, dexterous household tasks; how to build their training data remains open.

Follow the idea

Built on: Flow matching (2022), Diffusion Policy (2023), RT-2 (2023), Open X-Embodiment (2023)