Study guide · Robotics & reinforcement learning · 2023

Open X-Embodiment: Robotic Learning Datasets and RT-X Models

Open X-Embodiment Collaboration · ICRA 2024

Every robotics lab collects data on its own robot and trains its own model, so each dataset is small. This paper, written by a collaboration of dozens of labs, asks what happens if you pour all of it into one pot. With more than a million demonstrations from 22 kinds of robot, a single model trained on the mixture outperformed the specialised models the individual labs had built for their own robots.

The paper in four sentences

Robot learning is held back by small datasets, and every lab's data comes from a different robot, so the data has rarely been combined.

The Open X-Embodiment collaboration converted 60 existing datasets from 22 robot types (over a million recorded episodes) into one shared format and released them.

They trained two existing model designs, RT-1 and RT-2, on a mixture of these datasets, calling the results RT-1-X and RT-2-X, with only a rough alignment of what each robot's "move" commands mean.

On five labs' own small-data tasks, RT-1-X beat the labs' own methods by about 50% on average, and RT-2-X roughly tripled the success of RT-2 on skills that only existed in another robot's data.

Idea · §I

Many small piles of data, each on a different robot

In language and vision, progress came from training one big model on a huge, varied collection of data, then adapting it. Robotics has not had that collection. A robot dataset is gathered by people steering a particular robot through tasks, which is slow, and every lab uses different arms, grippers, cameras and rooms.

The obvious worry is that data from one robot is useless, or even harmful, for another. A small tabletop arm and a large mobile robot don't move the same way. The paper calls the alternative positive transfer: training on other robots' data makes a model better on your robot. The question is whether it happens.

An illustrative toy, not the paper's model. The dashed curve is a skill all robots share, but each robot sees it shifted by its own quirk (for example a different arm length). Our robot (amber dots) only ever practised on the left part of the range. Grey dots are other robots' data, each covering a different region. Amber line: a model trained on our data alone. Green line: a model trained on everyone's data. Switch off "adjust for quirks" and most of the benefit disappears, because the other robots' shifted data drags the model away from ours.
Everyday pictureLearning to drive by watching only your own street is limited. Watching hundreds of drivers in different cars and cities teaches you far more, as long as you remember that a lorry turns wider than your hatchback.

TakeawayOther robots' data can fill gaps in your own, if the differences between robots are accounted for. The paper tests whether that works at scale.

Check yourself
In the toy, why does the pooled model help most in the right-hand region?
Our robot never saw that region, so a model trained on our data alone has to guess. Other robots did see it, and the shared shape of the skill carries over once their quirks are subtracted.

Idea · §III · Fig. 2

The dataset: one format for 22 robots

The collaboration gathered 60 datasets from 21 institutions, covering 22 robot embodiments (types of robot body): single arms, two-armed robots and even four-legged ones. Together they hold over a million real-robot episodes and 527 skills, though most of the data is variations of picking and placing. The Franka arm appears in the most datasets.

All of it was converted to one file format so that a single program can load any of it. That sounds like plumbing, and it is, but it is the part that lets everyone else reuse the data.

TakeawayThe main contribution is the pooled, standardised, public dataset; the models show what it can do.

Idea · §IV-A

Making different robots speak roughly the same language

Each model sees a camera picture and a text instruction and outputs a seven-part movement for the robot's hand: three for position, three for rotation, one for the gripper. Different robots have different ranges of movement, so each dataset's movements are rescaled to a common range before being chopped into 256 levels, the trick inherited from RT-2.

Crucially, the alignment is only rough. One robot's "x" might point forward and another's sideways; one robot might be told where to go, another how fast to move. The authors deliberately didn't fix this, and the model had to work out from the camera image which robot it was driving.

Illustrative ranges. A small arm whose step commands span ±2 cm and a large robot whose span ±8 cm. Each dataset is scaled by its own range, so the same token means "this fraction of my usual movement", not "this many centimetres". Moves beyond a robot's range are clipped.

TakeawayEach robot's actions are squeezed into the same 256 levels using its own scale; the model learns the rest from what it sees.

Idea · §V · Fig. 4, Tables I–II

What pooling bought

Small datasets. Five labs tested on their own robots and tasks (such as kitchen manipulation, cable routing and door opening). The small RT-1-X model, trained on the mixture, beat each lab's own method on four of the five, and was about 50% better on average.

Large datasets. For robots that already had lots of data, the small RT-1-X did worse than models trained on that robot alone: it was too small to absorb everything. The much larger RT-2-X (55 billion parameters) did as well or better.

Skills borrowed from other robots. The Google robot was asked to do things that appeared only in a different robot's data, such as "move the apple between the can and the orange". RT-2 trained on Google robot data managed 27% of these; RT-2-X trained on the mixture managed 76%. Remove the WidowX arm's data from the mix and it fell to 43%, so the skills really were coming from that other robot.

TakeawaySmall-data robots gain a lot from pooling; big-data robots gain only if the model is big enough; and skills really do cross between robot bodies.

Check yourself
Why did RT-1-X underperform on the large datasets while RT-2-X did not?
RT-1 has about 35 million parameters, too few to fit a very diverse mixture without sacrificing performance on any single robot. RT-2-X has 55 billion and enough capacity for both.
What experiment shows the emergent skills came from another robot's data?
Removing the Bridge (WidowX) datasets from RT-2-X's training mix dropped emergent-skill success from 75.8% to 42.8%.

Idea · §VI

Open questions, and where it led

The authors list what they did not show: whether the model works on a brand-new robot it has no data for, whether robots with very different senses or controls (touch sensors, legged motion) benefit, and when transfer will help versus hurt. They also used only a subset of the robots for training, because the dataset kept growing during the project.

Later refinementThe dataset became a standard starting point. Octo (2024) and OpenVLA (2024) are open generalist policies trained on Open X-Embodiment data. π0 mixes it (with Bridge and DROID) into a much larger private dataset, where it makes up about 9% of the training mixture, and cross-embodiment training is now the default for robot foundation models.

TakeawayThe paper made "train one model on many robots" the default, but left open how to choose and weight the data.

Follow the idea

Built on: RT-2 (2023)

Led to: π0 (2024)