The paper in four sentences
Vision-language models, trained on huge collections of web images and text, can describe pictures and answer questions about them, but they have never controlled a body.
RT-2 turns every robot movement into eight whole numbers between 0 and 255, writes them out like a sentence, and teaches the vision-language model to output that "sentence" when shown the robot's camera image and an instruction.
It trains on robot demonstrations and the original web data together, so the model keeps what it knew about the world while learning to move.
The result matches an earlier robot model on familiar tasks and roughly doubles its success on unfamiliar ones, and it can follow instructions that need web knowledge, like "move the banana to the sum of two plus one".
Idea · §1–2
Robots have little data; the web has a lot
A robot learns a skill like "pick up the can" mostly by imitation: people steer the robot by remote control, and the recordings become training examples. That is slow and expensive. The dataset behind RT-2's predecessor, RT-1, took 13 robots about 17 months to collect, and it still covers only several hundred kinds of task in a few office kitchens.
Meanwhile a vision-language model (VLM), a model that takes an image and some text and writes text back, has seen billions of pictures with captions. It knows what a Coke can, a dinosaur toy or the Google logo looks like. The question RT-2 asks is whether that knowledge can be put to work on the motor side, not just for describing scenes.
TakeawayRobot data is scarce; web data is plentiful. RT-2 tries to transfer what is learned from the web into robot control.
Idea · §3.2 · Fig. 1
Writing movements as words
A VLM only ever outputs text, one small piece (a token) at a time. So RT-2 makes actions look like text. Each movement command for the arm has eight parts: whether to stop the episode, how far to move the hand in three directions, how much to rotate it about three axes, and how far to open the gripper.
Each continuous amount is chopped into 256 equal steps, called bins, numbered 0 to 255. "Move a little left" becomes, say, 112. The whole command becomes a string such as 1 128 124 136 121 158 111 255 (an action string quoted in the paper's chain-of-thought example, §4.4). The model is then trained just as it was trained on captions: predict the next token.
TakeawayChop each movement into 256 levels and write the levels as numbers. Now "what should the robot do?" is just another question the model answers in text.
Check yourself
- Why does RT-2 need actions to be discrete numbers rather than any decimal value?
- A language model chooses from a fixed vocabulary of tokens. Discretising each action dimension into 256 bins gives it 256 choices per dimension, which can be mapped onto 256 tokens.
- What is lost by using 256 bins?
- Precision: any value inside a bin is rounded to that bin, so the finest movement step is one 256th of the range.
Idea · §3.2, §4.3
Keep the web data in the mix
If you fine-tune a big model only on robot data, it can forget what it learned before. RT-2 instead uses co-fine-tuning: each training batch mixes robot examples with the model's original web tasks, such as captioning and answering questions about images. Robot examples are given extra weight so they make up about half of what the model sees.
When asked for a robot action, the model is only allowed to output valid action tokens. When asked an ordinary question, it can answer in normal words. So one model is both a chatbot about pictures and a robot controller.
TakeawayTrain on robot data and web data together, so the model learns to act without forgetting what it knows.
Idea · §4.1–4.2 · Fig. 4, Table 4
It generalises to things it never practised
The team ran about 6,000 trials on real robots. On tasks seen in robot training, RT-2 and RT-1 are about equally good (above 90%). The difference shows on unseen conditions: new objects, new table cloths and backgrounds, and whole new rooms. There RT-2 succeeds roughly twice as often as RT-1.
It also does things that need knowledge only the web could have supplied: "pick up the extinct animal" (the toy dinosaur), "move the Coke can to Taylor Swift" (a photo), "move the banana to the sum of two plus one" (a card with a 3). None of these concepts appear in the robot data.
TakeawaySame skill on familiar tasks, about twice as good on unfamiliar ones, plus new abilities borrowed from web knowledge.
Check yourself
- Why would web pretraining help with an unseen table cloth but not with an unseen motion such as wiping?
- Web data teaches what things look like and what words mean, so new appearances are familiar. It contains no robot motor commands, so motions the robot never demonstrated remain out of reach.
Idea · §5
Limits, and what came next
RT-2 does not learn new physical skills from the web. It can apply its known motions (pick, place, move near) to new things and new instructions, but it cannot invent wiping or tool use. The biggest model also has 55 billion parameters and runs on cloud computers at only 1 to 3 commands per second, too slow for fast, delicate work. And few such large models were openly available to build on.
TakeawayRT-2 transfers knowledge about the world, not new motor skills, and it is slow; later work fixed speed and smoothness while keeping the core idea.
Abstract, restated
The paper studies how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control to boost generalisation and enable emergent semantic reasoning. Actions are expressed as text tokens and the VLM is co-fine-tuned on robotic trajectory data and Internet-scale vision-language tasks, yielding a vision-language-action (VLA) model, RT-2. Over roughly 6,000 evaluation trials, RT-2 generalises to novel objects, interprets commands not present in robot data, and performs rudimentary reasoning, and chain-of-thought variants can do multi-stage semantic reasoning. Brohan et al. (Google DeepMind), CoRL 2023; arXiv 2307.15818.
Section · §3.2
Actions as tokens
The action space is RT-1's: 6-DoF end-effector displacement (Δposx,y,z, Δrotx,y,z), gripper extension, and a discrete episode-termination flag. Each continuous dimension is uniformly discretised into 256 bins, so an action is 8 integers, serialised as
Mapping to vocabulary differs by backbone. PaLI-X tokenises integers up to 1000 as unique tokens, so bin k maps to the token for k. PaLM-E has no such representation, so the 256 least frequently used tokens are overwritten as action tokens (a form of symbol tuning). The input is formatted as VQA: an image plus "Q: what action should the robot take to [task instruction]? A:". Training uses the ordinary next-token cross-entropy, which on robot data is exactly a behavioural-cloning loss over discretised actions.
Output constraint. When prompted for an action, decoding is restricted to valid action tokens, guaranteeing an executable command; on vision-language prompts the full vocabulary is allowed.
The cost of tokenisation is quantisation. The figure below discretises a smooth trajectory with a chosen number of bins per dimension.
Check yourself
- Why does the PaLM-E variant overwrite rare tokens instead of adding new ones?
- Reusing existing vocabulary entries keeps the embedding table and output softmax unchanged, so the pretrained model can be fine-tuned without architectural surgery; choosing the least-used tokens minimises interference with language ability.
- Each dimension is predicted as a separate token. What dependency does this introduce?
- Autoregressive decoding factorises the joint action distribution over dimensions in a fixed order, so each dimension is conditioned on the previous ones. This lets the model represent correlated, even multimodal, joint actions, at the cost of 8 sequential decoding steps per action.
Section · §3.1, §3.3 · App. B, E
Backbones, co-fine-tuning and real-time inference
Two VLM families: PaLI-X (ViT encoder plus UL2 encoder-decoder; 5B and 55B) and PaLM-E (12B, a decoder-only LM with image embeddings injected as tokens). Robot data is the RT-1 dataset (13 robots, 17 months, office-kitchen tasks). Co-fine-tuning mixes it with each VLM's original web data, upweighting robot samples to about 50% of the mixture for PaLI-X and about 66% for PaLM-E.
| Model | Learning rate | Batch | Gradient steps |
|---|---|---|---|
| RT-2-PaLI-X-55B | 1e-3 | 2048 | 80K |
| RT-2-PaLI-X-5B | 1e-3 | 2048 | 270K |
| RT-2-PaLM-E-12B | 4e-4 | 512 | 1M |
| RT-2-PaLI-3B (Language Table) | 1e-3 | 128 | 300K |
Inference (§3.3). The 55B model is, per the paper, over an order of magnitude larger than any previous model used for direct closed-loop robot control. It runs in a multi-TPU cloud service queried over the network: about 1–3 Hz for 55B and about 5 Hz for 5B.
Section · §4.1–4.2 · Tables 1, 4, 5
Results: generalisation and emergent skills
Seen tasks: over 200; unseen evaluation: over 280 tasks across object, background and environment shifts, each split into easy and hard. Success rates (%) from Table 4:
| Model | Seen | Obj E/H | BG E/H | Env E/H | Unseen avg |
|---|---|---|---|---|---|
| R3M | 45 | 32 / 14 | 13 / 9 | 0 / 2 | 12 |
| VC-1 | 63 | 34 / 10 | 13 / 3 | 0 / 0 | 10 |
| RT-1 | 92 | 31 / 43 | 71 / 9 | 26 / 14 | 32 |
| MOO | 75 | 58 / 48 | 38 / 41 | 19 / 3 | 35 |
| RT-2-PaLI-X-55B | 91 | 70 / 62 | 96 / 48 | 63 / 35 | 62 |
| RT-2-PaLM-E-12B | 93 | 84 / 76 | 75 / 71 | 36 / 33 | 62 |
The paper summarises this as roughly 2× over RT-1 and MOO and roughly 6× over R3M and VC-1 on unseen conditions. In simulation, on Language Table (Table 1), RT-2-PaLI-3B reaches 90 ± 10% versus BC-Zero 72 ± 3, RT-1 74 ± 13 and LAVA 77 ± 4.
Emergent capabilities (§4.2, Table 5) were tested with A/B instructions requiring symbol understanding, reasoning (math, logos, nutrition, multilingual commands) and person recognition. Averages: VC-1 11%, RT-1 17%, RT-2-PaLM-E-12B 40%, RT-2-PaLI-X-55B 60% (with symbol understanding 82%). PaLM-E was relatively stronger on math, which the authors attribute to its pretraining mixture.
Section · §4.3–4.4 · Fig. 6b, Table 6, Fig. 7
Ablations: size, pretraining, co-training; chain of thought
Three conclusions follow. Pretrained weights are essential (9% from scratch versus 42% fine-tuned at 5B). Co-fine-tuning helps, more so at 55B (from 52 to 63) than at 5B (from 42 to 44). And scale helps generalisation.
Chain of thought (§4.4). The PaLM-E variant was fine-tuned for a few hundred steps on data augmented with a natural-language "Plan" before the action, for example "Instruction: I'm hungry. Plan: pick rxbar chocolate. Action: 1 128 124 136 121 158 111 255." This let it handle multi-step semantic requests such as choosing a rock as an improvised hammer. The evidence is qualitative (Fig. 7, App. I).
Check yourself
- Why might co-fine-tuning matter more for the 55B model than for 5B?
- One plausible reading is that the larger model has more web knowledge to lose by catastrophic forgetting when fine-tuned only on narrow robot data, so retaining web data preserves more. The paper reports the numbers but does not test this explanation.
Section · §5 · App. G · Lineage
Limitations and lineage
- No new motor primitives. Web pretraining transfers semantics and visual concepts, not skills; physical skills remain bounded by the robot data distribution. Failure cases (App. G) include grasping by specific parts such as handles, novel motions such as wiping, dexterous tasks such as towel folding, and extended multi-step reasoning.
- Compute. 1–3 Hz cloud inference for 55B limits high-frequency control; the authors suggest quantisation and distillation.
- Availability. Few VLMs were open or fine-tunable, constraining replication.
- Discretised, single-step actions. 256 uniform bins per dimension, predicted one timestep at a time.
Reference
Glossary
- VLM
- Vision-language model: image + text in, text out.
- VLA
- Vision-language-action model: a VLM whose outputs include robot actions.
- Token
- One unit of a model's vocabulary; RT-2 maps action bins to tokens.
- Bin
- One of 256 equal intervals into which each action dimension is discretised.
- Co-fine-tuning
- Fine-tuning on a mixture of the new task data and the original pretraining tasks.
- Behavioural cloning
- Supervised learning of actions from demonstrations.
- RT-1
- Robotics Transformer 1 (2022), a 35M-parameter robot-only policy and its 13-robot dataset.
- Emergent capability
- Here: a behaviour requiring concepts absent from robot data, inherited from web pretraining.