The paper in four sentences
Earlier "robot brains" built from language models wrote movements as a handful of numbered tokens, a few times per second: good for pick-and-place, too coarse and slow for folding a shirt.
π0 keeps a pretrained vision-language model to understand the camera images and the instruction, but adds a separate, smaller network that produces a whole second's worth of smooth movement (50 small steps) at once.
That network is trained with flow matching: it learns to turn random noise, step by step, into a realistic movement, which lets it represent "there are two good ways to do this" without averaging them into a bad one.
Pretrained on a large mixture of data from seven robot configurations, then fine-tuned on high-quality demonstrations, it folds laundry from a basket, buses tables and assembles boxes, outperforming earlier generalist robot models.
Idea · §I, §IV · Fig. 3
A language-model brain with a motor cortex attached
π0 has two parts that share one network. The first is PaliGemma, a 3-billion-parameter vision-language model (VLM) from Google that was trained on web images and text. It reads the robot's camera images (up to three cameras) and the instruction, such as "fold the shirt".
The second part is an action expert of about 300 million parameters, trained from scratch. It reads the robot's current joint positions and everything the VLM has worked out, and outputs the next 50 movement commands. This bundle of upcoming commands is called an action chunk. At 50 commands per second, one chunk covers a second of motion.
The two parts are separate sets of weights, but they talk through attention inside one transformer, so the action expert can "look at" what the VLM noticed in the image.
Takeawayπ0 = a pretrained vision-language model for understanding plus a small new network that writes a second of smooth movement at a time.
Idea · §IV · Eq. 1–2
From noise to movement: flow matching
Suppose a cup is in the way and the arm can pass it on the left or on the right. Both are fine. A model trained to predict "the best guess" for each moment learns the average of the demonstrations, which is straight through the cup.
Flow matching avoids this. During training, the model is shown a real movement blended with random noise and learns which direction points from the noise towards the real movement. To act, it starts from pure noise and takes a few small steps in the direction it has learned, ten in π0's case. Different starting noise ends up at different good answers: some runs go left, some go right, and almost none go through the middle.
TakeawayFlow matching turns noise into an action in a few small steps, so it can produce any of several good movements instead of their bad average.
Check yourself
- Why does a single step send every sample to the middle?
- At time 0 the input is pure noise and carries no information about which side to choose, so the best single prediction of the destination is the average of all demonstrations. Later steps can see which way the sample has started to lean.
- Why did RT-2-style models with token outputs not hit this averaging problem as badly?
- Tokens give a probability over 256 bins per dimension, which can also have two peaks. Their weakness is coarseness and speed, not averaging. Flow matching gives continuous values and a whole chunk at once.
Idea · §I, §IV, App. D
A second of motion at a time
Laundry folding needs fast, fine motion. π0 controls robots at up to 50 commands per second. Running a 3-billion-parameter model fifty times a second is not practical, so instead π0 predicts a chunk of 50 future commands in one go (about 73 milliseconds on a single gaming graphics card) and the robot plays out part of the chunk before asking for the next.
The trade-off is responsiveness. While the robot is playing out a chunk, it is not looking. Asking more often reacts faster but costs more computing.
TakeawayPredicting a chunk of 50 commands makes 50-per-second control affordable, at the price of reacting only when a new chunk is computed.
Idea · §V · Fig. 4
Learn broadly, then polish
The training recipe copies how language models are made. Pre-training uses a huge, varied mixture: over 10,000 hours of the company's own robot data across 68 tasks and seven robot configurations (single arms, pairs of arms, arms on mobile bases), plus about 9% public data including Open X-Embodiment. This teaches broad abilities and how to recover from mistakes, because messy data contains many.
Post-training then fine-tunes on a smaller set of careful, high-quality demonstrations of one task, from a few hours to over 100 hours, to make the robot fluent and efficient at it. The authors found both stages matter: the pretrained-only model is robust but clumsy; a model trained only on the polished data is fluent but brittle.
TakeawayDiverse, imperfect data teaches robustness; a little perfect data teaches fluency. π0 uses both, in that order.
Idea · §VI–VII
What it can do, and what is still open
Straight after pre-training, π0 beat earlier open generalist robot models (OpenVLA and Octo) on every test task, including shirt folding and clearing tables. After post-training it did long, multi-stage jobs: taking laundry out of a dryer, folding it, bussing a table, assembling a cardboard box, and packing eggs into a carton. On every one of these the full model scored above half of the maximum possible points, which earlier methods could not approach.
For long jobs it can be paired with a higher-level language model that breaks a request into steps ("pick up the napkin", "put the cup in the bin"), an echo of SayCan.
The authors are frank about the limits: nobody yet knows how to choose and weight the pre-training data, not every task works reliably, and it has not been tried on very different domains such as driving or walking robots.
Takeawayπ0 showed general robot models can do long, dexterous household tasks; how to build their training data remains open.
Abstract, restated
Robot learning promises flexible, general, dexterous systems but faces data, generalisation and robustness obstacles. π0 is a flow-matching architecture built on a pretrained VLM to inherit Internet-scale semantic knowledge, trained on a large, diverse dataset from multiple dexterous platforms (single-arm, dual-arm and mobile manipulators). It is evaluated on zero-shot performance after pre-training, following language instructions from people and from a high-level VLM policy, and acquiring new skills by fine-tuning, across tasks such as laundry folding, table cleaning and box assembly. Black et al. (Physical Intelligence), arXiv 2410.24164, October 2024; later published at RSS 2025.
Section · §IV · App. B · Fig. 3
Architecture: VLM backbone, action expert, blockwise causal mask
The policy models p(At | ot) where At = [at, …, at+H−1] is an action chunk with H = 50 and ot = [I1t, …, Int, ℓt, qt] holds 2 or 3 RGB images, a language command and the proprioceptive joint-angle vector qt. Images pass through PaliGemma's vision encoder into the language token space; qt and the noisy actions are linearly projected in. Action tokens also receive a sinusoidal embedding of the flow time τ, fused via an MLP.
Backbone: PaliGemma (≈3B, Gemma-2B language model). The action expert is a second set of transformer weights (width 1024, MLP dimension 4096, ≈300M parameters) that processes the state and action tokens; the two weight sets interact only through self-attention, a mixture-of-experts-style design in the manner of Transfusion. Total ≈3.3B parameters. Attention follows a blockwise causal mask with three blocks: [images, language], [qt], [noisy actions]. Click a row below to see what it may attend to.
Check yourself
- Why is the prefix (images + language) forbidden from attending to the state and actions?
- So the pretrained VLM sees inputs like those it was trained on, minimising distribution shift, and so the prefix's keys and values do not depend on the noisy actions and can be computed once and cached across all integration steps.
Section · §IV · App. B, D
Conditional flow matching for action chunks
With a linear-Gaussian (optimal-transport) probability path, noisy actions and the training loss are
The conditional target is the time derivative of the path, u(Aτ | A) = A − ε. (The first arXiv version writes ε − A, which is inconsistent with integrating from noise at τ = 0 to data at τ = 1; the later revision gives A − ε.) The equation numbers above are this guide's; the paper's HTML version leaves its display equations unnumbered. Inference uses forward Euler with 10 steps, caching the prefix's attention keys and values so only the action tokens are recomputed per step.
Timestep distribution. Rather than uniform τ or the logit-normal of image flow models, π0 samples τ from a shifted beta distribution emphasising low τ (noisier inputs): p(τ) = Beta((s − τ)/s; 1.5, 1) with cutoff s = 0.999. The rationale given is that, unlike image generation, predicting the mean action at high noise is hard but at low noise the observation strongly constrains the action, so training effort is shifted towards the harder, noisier regime; τ above s is never sampled because steps that small are not needed with δ = 0.1.
Check yourself
- Verify that A − ε is the conditional velocity of the path.
- d/dτ [τA + (1 − τ)ε] = A − ε, independent of τ. The marginal field that vθ learns is 𝔼[A − ε | Aτ, o], which is what the Plain English toy computes in closed form.
- What fraction of training draws have τ < 0.5 under π0's distribution?
- With u = (s − τ)/s ∼ Beta(1.5, 1), the CDF is u1.5. τ < 0.5 means u > (0.999 − 0.5)/0.999 ≈ 0.4995, with probability 1 − 0.49951.5 ≈ 0.65.
Section · §V · Fig. 4 · Table I
Data, recipe and robot system
- Own data: 903M timesteps (106M single-arm, 797M dual-arm), 68 tasks defined broadly (for example "bussing" covers many objects and placements), more than 10,000 hours overall.
- Open data: Open X-Embodiment, Bridge v2 and DROID, 9.1% of the mixture.
- Weighting: each task–robot combination with n samples is weighted by n0.43, down-weighting over-represented combinations.
- Heterogeneous robots: 7 configurations (single-arm, bimanual and mobile platforms) with configuration and action spaces padded with zeros to the largest, 18 dimensions; robots with fewer cameras have missing image slots masked.
- Post-training: 5 to 100+ hours of curated data per downstream task.
- Inference (Table I, RTX 4090, three cameras): 73 ms on-board in total, of which the 10 flow steps take 27 ms; 86 ms off-board including 13 ms network latency.
Section · §VI · Figs. 7, 9, 11, 13
Experiments
Results are reported as normalised task-progress scores in figures, so the main text gives few exact numbers; the qualitative findings are clear.
- Out of the box (§VI-A, Fig. 7): shirt folding, bussing easy and hard, grocery bagging, toast out of toaster. π0 is best on all, near-perfect on shirt folding and easy bussing. Even a "parity" π0 trained for only 160k steps beats all baselines. π0-small (no VLM initialisation) beats OpenVLA and Octo but trails π0. OpenVLA suffers from lacking action chunking and its discrete tokenisation at these control rates.
- Language following (§VI-B, Fig. 9): on bussing, table setting and grocery bagging, π0 follows commands markedly better than π0-small, showing the value of VLM initialisation; intermediate commands from a human expert or from a high-level VLM policy improve performance.
- New dexterous tasks (§VI-C, Fig. 11): stacking bowls, towel folding, Tupperware in microwave, paper-towel replacement, items into a Franka drawer, versus OpenVLA, Octo, ACT and Diffusion Policy. π0 generally wins; pre-training helps most on tasks close to the pre-training data, with improvements up to about 2× over training from scratch.
- Complex multi-stage tasks (§VI-D, Fig. 13): laundry folding (static and mobile), dryer unloading, table bussing, box building, to-go box and packing eggs. The full pre-train + post-train model scores above 50% of the maximum on every task; both scratch-trained and pre-trained-only models do markedly worse.
Section · §VII · Lineage
Limitations and lineage
- No principled method for composing or weighting the pre-training mixture.
- Not all tasks work reliably; the amount of post-training data needed for near-perfect performance is unpredictable.
- When and why positive transfer arises across very diverse data is not understood.
- Untested beyond manipulation (driving, navigation, legged locomotion).
- Zero-shot pre-trained behaviour lacks the fluent strategies seen after post-training.
Reference
Glossary
- VLA
- Vision-language-action model.
- Action chunk
- A block of H future actions predicted jointly (H = 50).
- Action expert
- π0's separate transformer weights for state and action tokens.
- Flow matching
- Training a velocity field that transports noise to data along prescribed paths.
- Forward Euler
- Integrating dx/dτ = v with fixed steps x ← x + δv.
- Blockwise causal mask
- Attention mask where tokens attend within their block and to earlier blocks only.
- Proprioception
- The robot's sensing of its own joint positions.
- Post-training
- Fine-tuning a pretrained model on curated task-specific data.