The paper in four sentences
Robot learning is held back by small datasets, and every lab's data comes from a different robot, so the data has rarely been combined.
The Open X-Embodiment collaboration converted 60 existing datasets from 22 robot types (over a million recorded episodes) into one shared format and released them.
They trained two existing model designs, RT-1 and RT-2, on a mixture of these datasets, calling the results RT-1-X and RT-2-X, with only a rough alignment of what each robot's "move" commands mean.
On five labs' own small-data tasks, RT-1-X beat the labs' own methods by about 50% on average, and RT-2-X roughly tripled the success of RT-2 on skills that only existed in another robot's data.
Idea · §I
Many small piles of data, each on a different robot
In language and vision, progress came from training one big model on a huge, varied collection of data, then adapting it. Robotics has not had that collection. A robot dataset is gathered by people steering a particular robot through tasks, which is slow, and every lab uses different arms, grippers, cameras and rooms.
The obvious worry is that data from one robot is useless, or even harmful, for another. A small tabletop arm and a large mobile robot don't move the same way. The paper calls the alternative positive transfer: training on other robots' data makes a model better on your robot. The question is whether it happens.
TakeawayOther robots' data can fill gaps in your own, if the differences between robots are accounted for. The paper tests whether that works at scale.
Check yourself
- In the toy, why does the pooled model help most in the right-hand region?
- Our robot never saw that region, so a model trained on our data alone has to guess. Other robots did see it, and the shared shape of the skill carries over once their quirks are subtracted.
Idea · §III · Fig. 2
The dataset: one format for 22 robots
The collaboration gathered 60 datasets from 21 institutions, covering 22 robot embodiments (types of robot body): single arms, two-armed robots and even four-legged ones. Together they hold over a million real-robot episodes and 527 skills, though most of the data is variations of picking and placing. The Franka arm appears in the most datasets.
All of it was converted to one file format so that a single program can load any of it. That sounds like plumbing, and it is, but it is the part that lets everyone else reuse the data.
TakeawayThe main contribution is the pooled, standardised, public dataset; the models show what it can do.
Idea · §IV-A
Making different robots speak roughly the same language
Each model sees a camera picture and a text instruction and outputs a seven-part movement for the robot's hand: three for position, three for rotation, one for the gripper. Different robots have different ranges of movement, so each dataset's movements are rescaled to a common range before being chopped into 256 levels, the trick inherited from RT-2.
Crucially, the alignment is only rough. One robot's "x" might point forward and another's sideways; one robot might be told where to go, another how fast to move. The authors deliberately didn't fix this, and the model had to work out from the camera image which robot it was driving.
TakeawayEach robot's actions are squeezed into the same 256 levels using its own scale; the model learns the rest from what it sees.
Idea · §V · Fig. 4, Tables I–II
What pooling bought
Small datasets. Five labs tested on their own robots and tasks (such as kitchen manipulation, cable routing and door opening). The small RT-1-X model, trained on the mixture, beat each lab's own method on four of the five, and was about 50% better on average.
Large datasets. For robots that already had lots of data, the small RT-1-X did worse than models trained on that robot alone: it was too small to absorb everything. The much larger RT-2-X (55 billion parameters) did as well or better.
Skills borrowed from other robots. The Google robot was asked to do things that appeared only in a different robot's data, such as "move the apple between the can and the orange". RT-2 trained on Google robot data managed 27% of these; RT-2-X trained on the mixture managed 76%. Remove the WidowX arm's data from the mix and it fell to 43%, so the skills really were coming from that other robot.
TakeawaySmall-data robots gain a lot from pooling; big-data robots gain only if the model is big enough; and skills really do cross between robot bodies.
Check yourself
- Why did RT-1-X underperform on the large datasets while RT-2-X did not?
- RT-1 has about 35 million parameters, too few to fit a very diverse mixture without sacrificing performance on any single robot. RT-2-X has 55 billion and enough capacity for both.
- What experiment shows the emergent skills came from another robot's data?
- Removing the Bridge (WidowX) datasets from RT-2-X's training mix dropped emergent-skill success from 75.8% to 42.8%.
Idea · §VI
Open questions, and where it led
The authors list what they did not show: whether the model works on a brand-new robot it has no data for, whether robots with very different senses or controls (touch sensors, legged motion) benefit, and when transfer will help versus hurt. They also used only a subset of the robots for training, because the dataset kept growing during the project.
TakeawayThe paper made "train one model on many robots" the default, but left open how to choose and weight the data.
Abstract, restated
Large models pretrained on diverse data have been adapted to many downstream tasks in NLP and vision; this paper asks whether the same is possible for robot manipulation, with an "X-robot" policy that adapts to new robots, tasks and environments. It releases standardised data from 22 robots collected through a collaboration of 21 institutions, demonstrating 527 skills (160,266 tasks), and shows that a high-capacity model trained on it (RT-X) exhibits positive transfer, improving several robots' capabilities by leveraging other platforms' experience. Open X-Embodiment Collaboration, ICRA 2024; arXiv 2310.08864.
Section · §III · Fig. 2
The Open X-Embodiment repository
60 existing datasets, 22 embodiments, 1M+ real trajectories. Fig. 2 breaks the data down: Franka is the most common robot by number of datasets and visually distinct scenes, xArm and the Google Robot contribute the most trajectories, and skills are dominated by pick-and-place with a long tail (wiping, assembling, routing cables). Objects are household items: appliances, food, utensils.
Format. Data is stored in RLDS (Reinforcement Learning Datasets), serialised as TFRecord, which supports differing action spaces and observation modalities (varying numbers of RGB cameras, depth, point clouds) and efficient parallel loading in TensorFlow, PyTorch and JAX pipelines.
The paper's text gives both 21 institutions and 34 research labs as the contributing groups; these presumably count different units (institutions versus individual labs).
Section · §IV · Fig. 3
RT-X: data consolidation and architectures
Consolidation (§IV-A). Inputs: a history of images and a language instruction. One canonical camera view per dataset, resized to a shared resolution. Output: a 7-DoF end-effector action (x, y, z, roll, pitch, yaw, gripper opening, or their rates) plus termination. Each dataset's actions are normalised before discretisation into 256 bins; at inference the output is de-normalised per embodiment. Action spaces are coarsely aligned only: coordinate frames are not aligned across robots, and actions may be absolute or delta depending on the robot's controller. The same action vector can therefore mean very different motions on different robots.
RT-1-X. RT-1 (~35M parameters): EfficientNet image encoder with FiLM conditioning on a Universal Sentence Encoder embedding of the instruction, a 15-image history compressed to 81 vision-language tokens, and a decoder-only Transformer outputting discretised actions. Trained on the robotics mixture only.
RT-2-X. RT-2 on a PaLI-X backbone (ViT plus UL2, pretrained mainly on WebLI), 55B or 5B, actions as text tokens (eight dimensions, 256 bins). Co-fine-tuned with roughly 1:1 web VLM data to robot data.
Training mixture (§IV-C). 9 manipulators from 12 datasets: RT-1, QT-Opt, Bridge, Task Agnostic Robot Play, Jaco Play, Cable Routing, RoboTurk, NYU VINN, Austin VIOLA, Berkeley UR5, TOTO and Language Table. Loss: categorical cross-entropy over action tokens. Inference at 3–10 Hz (RT-1 on-robot, RT-2 via cloud).
Check yourself
- If frames are not aligned, how can the model output the right action for a given robot?
- It must infer the embodiment from the observation (the camera view shows the robot and scene) and condition its action distribution on that, much as a language model infers the language from context. Per-dataset normalisation keeps the token ranges comparable but does not supply the frame.
Section · §V-A · Fig. 4, Table I
In-distribution performance
3,600 evaluation trials on 6 robots. Small-data domains (Fig. 4): Kitchen Manipulation, Cable Routing, NYU Door Opening, AUTOLab UR5, Robot Play. Each "Original Method" is the model the dataset's creators published. RT-1-X outperforms it on 4 of 5, with mean success about 50% higher than both the Original Method and RT-1 trained on that dataset alone. Large-data domains (Table I) follow.
The pattern is a capacity effect: the 35M-parameter RT-1-X underfits when the mixture adds data that the target domain does not need, while the 55B RT-2-X absorbs the mixture and improves or matches.
Section · §V-B, §V-C · Table II, Fig. 5
Out-of-distribution transfer and design ablations
Evaluated on the Google Robot. Emergent skills: tasks whose concepts appear only in Bridge (WidowX) data, such as spatial relations and object placements not in the RT-1 dataset (Fig. 5). Generalisation: the unseen objects/backgrounds/environments suite from RT-2.
- Transfer is real: row 2 vs 1, 75.8% vs 27.3% emergent (~2.8×, "about 3×" in the text) with generalisation unchanged (61% vs 62%), which the authors attribute to the VLM backbone already supplying generalisation.
- Its source is identifiable: removing Bridge (row 3) drops emergent skills to 42.8%.
- History matters: 2-step image history (row 4) beats none (row 5), 44.4% vs 14.5%.
- Web pretraining is essential: from scratch (row 6) scores 0% / 1%.
- Scale: 55B (row 2) vs 5B (row 4), 75.8% vs 44.4%.
- Co-fine-tuning vs fine-tuning (row 7 vs 4) is roughly a wash here, which the authors attribute to the diversity of the robot mixture itself.
Section · §VI · Lineage
Limitations and what followed
- No evaluation on robots with very different sensing or actuation (tactile, legged locomotion).
- No study of generalisation to an entirely new, unseen robot.
- No criterion for when positive transfer will occur; data selection and weighting are ad hoc.
- Training used 9 of the 22 embodiments; the architectures were reused, not designed for cross-embodiment.
- Action alignment is coarse; mixing absolute and delta controllers is left to the model.
Reference
Glossary
- Embodiment
- A robot's physical body: kinematics, gripper, sensors, controller.
- Cross-embodiment
- Training or deploying one policy across different robot bodies.
- Positive transfer
- Training on other domains' data improves performance on the target domain.
- RLDS
- Reinforcement Learning Datasets: an episode/step data format on TFRecord.
- RT-1-X / RT-2-X
- RT-1 or RT-2 architectures trained on the X-embodiment mixture.
- Bridge
- A large WidowX tabletop dataset from UC Berkeley, the main source of emergent skills here.
- Emergent skill
- Here: a task performable on robot A only because it was demonstrated on robot B.