The paper in four sentences
Simulators can generate unlimited labelled training images for robots, but networks trained on them usually fail on real camera images because the simulated ones look different: the "reality gap".
Instead of making the simulator more realistic, the authors made it wildly unrealistic in random ways: random colours and patterns on every surface, random lights, a jittered camera, random clutter and random noise.
A network trained only on these images has to learn what stays the same across all the variation (the shape and position of the object), and so it works on the real world without ever having seen a real training image.
The resulting detector found eight kinds of geometric objects on a real table with an average error of about 1.5 cm, and a robot used it to pick objects out of clutter in 38 of 40 attempts.
Idea · §I
The reality gap
Training a robot in the real world is slow, expensive and occasionally destructive. In a simulator you can run thousands of experiments an hour, and every image comes with perfect labels for free: you know exactly where every object is, because you put it there.
The trouble is that a network trained on simulated images learns the simulator's quirks. Real cameras have odd colours, glare, shadows, blur and messy backgrounds that the simulator never showed. A network that relied on, say, "the table is always this exact shade of grey" falls apart on a real table. Closing this reality gap usually meant painstaking photo-realistic rendering, or collecting at least some real images to adapt with.
TakeawaySimulated images are cheap and perfectly labelled, but a network trained on them learns the simulator's look along with the task.
Idea · §III.A · Fig. 1
Randomise everything that should not matter
The paper's idea, domain randomisation, is to make every visual detail that is irrelevant to the task change at random in every training image. The authors used a basic, deliberately non-photorealistic renderer and randomised:
- the number and shapes of distractor objects on the table (0 to 10);
- the position and texture (colour pattern) of every object, plus the textures of the table, floor, background and robot;
- the camera's position, angle and field of view, within small ranges around the real camera's placement;
- the number of lights and their positions, directions and shininess;
- the type and amount of random noise added to the image.
Textures were simple: a single random colour, a gradient between two random colours, or a checkerboard of two random colours. The result looks nothing like reality, which is the point. Generate a few scenes below.
TakeawayIf colours, lighting, camera and clutter change in every training image, the only thing the network can rely on is what actually matters: the object's shape and position.
Check yourself
- Why randomise the camera position slightly, rather than fixing it exactly where the real camera is?
- The real camera's position is never measured perfectly. Training across a small range makes the network robust to that error (and the paper's ablation found this mattered).
- Why add distractor objects?
- So the network learns to find the target shape rather than "the only object on the table". Without distractors in training, its error with real distractors rose several-fold.
Idea · §IV.C · Fig. 5
How much variety is enough?
A network trained on a few textures can still do well on the simulator, because it learns to "subtract" those particular backgrounds. On a new background it has never seen, that trick fails. Only when it has seen a huge variety does it learn to ignore backgrounds in general. The paper found that real-world accuracy degraded significantly with fewer than about 1,000 distinct textures in training.
The toy below shows the same effect in miniature. Each "image" is a row of 32 pixels: a bright blob (the object) on a background pattern. A simple learner estimates the blob's position. It trains on 600 images using a limited set of background patterns, then is tested on images with brand-new patterns, playing the part of the real world.
TakeawayDoing well on simulated images is easy; doing well on unseen ones needs a very wide variety of randomised appearances.
Idea · §IV · Tables I–II
Results, and what mattered
The network looks at a single image from an ordinary webcam and outputs the position of one particular object. On real images, its average error was about 1.5 cm across eight shapes (cone, cube, cylinder, prisms, pyramid, tetrahedron), and it coped with distractors and partial occlusion. On simulated images it was much more accurate (0.3 to 0.5 cm), so it still overfits to the simulator somewhat.
The authors switched ingredients off one at a time. Removing distractors from training was the most damaging: with real distractors on the table, the average error jumped from 1.8 cm to 7.2 cm. Removing camera randomisation made every condition worse by half a centimetre or more. Removing noise barely changed accuracy, though it helped training converge. Starting from a network pre-trained on ordinary photos (ImageNet) helped most when training data were scarce; with tens of thousands of simulated images, starting from scratch did almost as well.
Finally, a Fetch robot used the detector to grasp target objects from increasingly cluttered tables, succeeding in 38 of 40 attempts.
TakeawayTrained only on cartoonish random images, a detector reached about 1.5 cm real-world accuracy. Variety in clutter and camera mattered most.
Abstract, restated
Bridging the reality gap between simulation and physical robots is a major obstacle to using simulated data. Domain randomisation trains models on simulated images with randomised rendering so that, with enough variability, the real world appears to the model as just another variation. Applied to object localisation, it yields the first successful transfer, to the authors' knowledge, of a deep network trained only on simulated RGB images (without pre-training on real images) to the real world for robotic control. The detector is accurate to 1.5 cm and robust to distractors and partial occlusions, and is used to grasp in clutter. IROS 2017, arXiv:1703.06907.
Section · §III.A
Problem and randomisation
Task. Learn a detector fθ mapping one monocular RGB frame I to the Cartesian coordinates (x, y, z) of each object of interest, in the presence of distractors to be ignored. The table height is fixed, so the task is effectively 2-D pose estimation on the table plane.
Randomised at training time (each sample is drawn independently):
- number and shape of distractors on the table (0–10);
- position and texture of all objects; textures of table, floor, skybox and robot;
- camera position within a 10 × 5 × 10 cm box, orientation offset up to 0.1 rad, field of view scaled by up to 5%;
- number, position, orientation and specular characteristics of lights;
- type and amount of random noise added to images.
Textures are drawn from: (a) a random RGB value, (b) a gradient between two random RGB values, (c) a checker pattern between two random RGB values. Rendering uses MuJoCo's built-in renderer, which is not intended to be photorealistic; the authors stress that physically plausible appearance is unnecessary.
Section · §III.B · Fig. 2
Model and training
A modified VGG-16: the convolutional stack of VGG-16 (convolutions, ReLU, max-pooling), input a 224 × 224 downsized webcam frame, followed by fully connected layers of 256 and 64 units without dropout, and a linear output of (x, y, z). Loss: L2 between predicted and true position; optimiser Adam with learning rate around 10−4 (rather than the standard 10−3). Convolutional layers were usually initialised from ImageNet weights, though random initialisation "works as well in most cases".
Note the nuance in the "trained only on simulated images" claim: the reported detectors typically started from ImageNet-pretrained convolutional weights, but the authors show (Fig. 4) that with enough simulated data, random initialisation reaches similar accuracy, so real images are not required. With pretraining, as few as 5,000 simulated samples gave reasonable real-world accuracy, improving up to around 50,000.
Section · §IV.A–C · Tables I–II
Localisation accuracy and ablations
Real-world evaluation: objects on a table imaged by the webcam, under three conditions: object alone, with distractors, and partially occluded. Average errors are about 1.5 cm overall; on simulated validation images the error was 0.3–0.5 cm, indicating residual overfitting to simulation.
Readings. Distractors in training are critical for robustness to real distractors (1.8 → 7.2 cm without them). Camera randomisation improves accuracy in all three conditions. Noise has a negligible effect on accuracy but helps convergence. Fig. 4: pretraining matters more with less data. Fig. 5 (10,000 examples): accuracy degrades significantly with fewer than 1,000 unique textures. The worst Table I entries (tetrahedron and cylinder under occlusion) have large standard deviations, consistent with occasional gross failures rather than uniform imprecision.
Robot experiments (§IV.D). On a Fetch robot, the detector drove a grasping pipeline that succeeded in 38 of 40 trials across 20 increasingly cluttered scenes. A detector for a YCB Spam can, with unseen food items as distractors, gave 9 of 10 successful trials.
Check yourself
- Why is a low simulated error combined with a higher real error evidence of overfitting to simulation, not of a bad model?
- The model can fit the simulator's distribution well; the gap shows that some sim-specific cues still help it, and the real images fall partly outside the randomised distribution.
- Which ablation shows the largest effect, and why does it make sense?
- Removing distractors: a network that never saw other objects can rely on "the only object present", a shortcut that fails as soon as real clutter appears.
Section · §V and lineage
Limitations and lineage
- Precision. About 1.5 cm suffices for grasping simple shapes but not for contact-rich or high-precision manipulation, which the authors name as future work.
- Narrow task. Simple geometric objects on one table, fixed table height, from a camera near a known position.
- Hand-chosen ranges. What to randomise and by how much is designed by hand; too little leaves a gap, too much makes the task needlessly hard.
- Visual only. Only appearance is randomised; the detector supplies positions, and dynamics are not part of what is learned.
Reference
Glossary
- Reality gap
- Mismatch between simulated and real observations or dynamics that breaks transfer.
- Domain randomisation
- Training across randomised simulator parameters so that the real world falls within the training distribution.
- Domain adaptation
- Using (usually some) real data to align a model trained on one domain with another.
- Distractor
- A non-target object placed in the scene to prevent shortcut solutions.
- Skybox
- The background surrounding a rendered scene.
- MuJoCo
- A physics simulator with a simple built-in renderer.
- VGG-16
- A 16-layer convolutional network for image classification, often used as a pretrained feature extractor.
- Fetch
- A mobile manipulator robot with a single arm and parallel gripper.