Study guide · Robotics & reinforcement learning · 2023

RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

Anthony Brohan et al. · CoRL 2023

Can the knowledge a model picks up from billions of web images and captions help a robot arm move? RT-2 says yes, with a simple trick: write each robot movement as a short string of numbers, treat those numbers as words, and fine-tune a large vision-language model to "answer" a camera image with the next movement. The resulting robot handles objects, backgrounds and instructions it never saw in robot training far better than earlier robot-only models.

The paper in four sentences

Vision-language models, trained on huge collections of web images and text, can describe pictures and answer questions about them, but they have never controlled a body.

RT-2 turns every robot movement into eight whole numbers between 0 and 255, writes them out like a sentence, and teaches the vision-language model to output that "sentence" when shown the robot's camera image and an instruction.

It trains on robot demonstrations and the original web data together, so the model keeps what it knew about the world while learning to move.

The result matches an earlier robot model on familiar tasks and roughly doubles its success on unfamiliar ones, and it can follow instructions that need web knowledge, like "move the banana to the sum of two plus one".

Idea · §1–2

Robots have little data; the web has a lot

A robot learns a skill like "pick up the can" mostly by imitation: people steer the robot by remote control, and the recordings become training examples. That is slow and expensive. The dataset behind RT-2's predecessor, RT-1, took 13 robots about 17 months to collect, and it still covers only several hundred kinds of task in a few office kitchens.

Meanwhile a vision-language model (VLM), a model that takes an image and some text and writes text back, has seen billions of pictures with captions. It knows what a Coke can, a dinosaur toy or the Google logo looks like. The question RT-2 asks is whether that knowledge can be put to work on the motor side, not just for describing scenes.

Everyday pictureA new employee who has read a lot but never worked in a warehouse. Training them only on the warehouse floor would take years. Better to keep everything they already know and add a few weeks of hands-on practice.

TakeawayRobot data is scarce; web data is plentiful. RT-2 tries to transfer what is learned from the web into robot control.

Idea · §3.2 · Fig. 1

Writing movements as words

A VLM only ever outputs text, one small piece (a token) at a time. So RT-2 makes actions look like text. Each movement command for the arm has eight parts: whether to stop the episode, how far to move the hand in three directions, how much to rotate it about three axes, and how far to open the gripper.

Each continuous amount is chopped into 256 equal steps, called bins, numbered 0 to 255. "Move a little left" becomes, say, 112. The whole command becomes a string such as 1 128 124 136 121 158 111 255 (an action string quoted in the paper's chain-of-thought example, §4.4). The model is then trained just as it was trained on captions: predict the next token.

Each slider is one part of the movement, scaled to a range of −1 to 1 (the gripper 0 to 1). The ruler shows where it lands among the 256 bins, and the bottom row is the string the model would have to write. Rotations are held at zero here to keep the picture small. The ranges are normalised for illustration; the real robot's ranges are set per dimension.

TakeawayChop each movement into 256 levels and write the levels as numbers. Now "what should the robot do?" is just another question the model answers in text.

Check yourself
Why does RT-2 need actions to be discrete numbers rather than any decimal value?
A language model chooses from a fixed vocabulary of tokens. Discretising each action dimension into 256 bins gives it 256 choices per dimension, which can be mapped onto 256 tokens.
What is lost by using 256 bins?
Precision: any value inside a bin is rounded to that bin, so the finest movement step is one 256th of the range.

Idea · §3.2, §4.3

Keep the web data in the mix

If you fine-tune a big model only on robot data, it can forget what it learned before. RT-2 instead uses co-fine-tuning: each training batch mixes robot examples with the model's original web tasks, such as captioning and answering questions about images. Robot examples are given extra weight so they make up about half of what the model sees.

When asked for a robot action, the model is only allowed to output valid action tokens. When asked an ordinary question, it can answer in normal words. So one model is both a chatbot about pictures and a robot controller.

TakeawayTrain on robot data and web data together, so the model learns to act without forgetting what it knows.

Idea · §4.1–4.2 · Fig. 4, Table 4

It generalises to things it never practised

The team ran about 6,000 trials on real robots. On tasks seen in robot training, RT-2 and RT-1 are about equally good (above 90%). The difference shows on unseen conditions: new objects, new table cloths and backgrounds, and whole new rooms. There RT-2 succeeds roughly twice as often as RT-1.

It also does things that need knowledge only the web could have supplied: "pick up the extinct animal" (the toy dinosaur), "move the Coke can to Taylor Swift" (a photo), "move the banana to the sum of two plus one" (a card with a 3). None of these concepts appear in the robot data.

Real success rates from the paper's Table 4 (appendix H.1). RT-1 is the earlier robot-only model; R3M and VC-1 use visual features pretrained on other data; MOO uses a pretrained detector to find objects. Flip from "Seen" to any unseen condition and watch the robot-only models fall away.

TakeawaySame skill on familiar tasks, about twice as good on unfamiliar ones, plus new abilities borrowed from web knowledge.

Check yourself
Why would web pretraining help with an unseen table cloth but not with an unseen motion such as wiping?
Web data teaches what things look like and what words mean, so new appearances are familiar. It contains no robot motor commands, so motions the robot never demonstrated remain out of reach.

Idea · §5

Limits, and what came next

RT-2 does not learn new physical skills from the web. It can apply its known motions (pick, place, move near) to new things and new instructions, but it cannot invent wiping or tool use. The biggest model also has 55 billion parameters and runs on cloud computers at only 1 to 3 commands per second, too slow for fast, delicate work. And few such large models were openly available to build on.

Later refinementOpen X-Embodiment trained RT-2 on data from many different robots (RT-2-X) and found skills transferring between them. OpenVLA (2024) released an open 7-billion-parameter model in the same style. π0 kept the "vision-language model as robot brain" idea but replaced number tokens with a separate action module that outputs smooth, continuous movements at up to 50 per second, better suited to dexterous tasks like folding laundry.

TakeawayRT-2 transfers knowledge about the world, not new motor skills, and it is slow; later work fixed speed and smoothness while keeping the core idea.

Follow the idea

Built on: CLIP (2021), SayCan (2022)

Led to: Open X-Embodiment (2023), π0 (2024)