Study guide · Robotics & reinforcement learning · 2017

Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World

Josh Tobin et al. · IROS 2017

Robots could learn cheaply in simulation if only the simulated camera images looked like the real world, and they never quite do. This paper turns the problem around: render the simulated scenes with absurd random textures, lighting, camera angles and clutter, so that a network trained on them treats the real world as just one more variation. A detector trained only on such images located objects on a real table to within about 1.5 cm, well enough to grasp them in clutter.

The paper in four sentences

Simulators can generate unlimited labelled training images for robots, but networks trained on them usually fail on real camera images because the simulated ones look different: the "reality gap".

Instead of making the simulator more realistic, the authors made it wildly unrealistic in random ways: random colours and patterns on every surface, random lights, a jittered camera, random clutter and random noise.

A network trained only on these images has to learn what stays the same across all the variation (the shape and position of the object), and so it works on the real world without ever having seen a real training image.

The resulting detector found eight kinds of geometric objects on a real table with an average error of about 1.5 cm, and a robot used it to pick objects out of clutter in 38 of 40 attempts.

Idea · §I

The reality gap

Training a robot in the real world is slow, expensive and occasionally destructive. In a simulator you can run thousands of experiments an hour, and every image comes with perfect labels for free: you know exactly where every object is, because you put it there.

The trouble is that a network trained on simulated images learns the simulator's quirks. Real cameras have odd colours, glare, shadows, blur and messy backgrounds that the simulator never showed. A network that relied on, say, "the table is always this exact shade of grey" falls apart on a real table. Closing this reality gap usually meant painstaking photo-realistic rendering, or collecting at least some real images to adapt with.

Everyday pictureA student who only ever practised with one textbook's diagrams can be thrown by a slightly different drawing in the exam. A student who practised with diagrams from twenty different books, in different styles, focuses on what the diagrams have in common.

TakeawaySimulated images are cheap and perfectly labelled, but a network trained on them learns the simulator's look along with the task.

Idea · §III.A · Fig. 1

Randomise everything that should not matter

The paper's idea, domain randomisation, is to make every visual detail that is irrelevant to the task change at random in every training image. The authors used a basic, deliberately non-photorealistic renderer and randomised:

  • the number and shapes of distractor objects on the table (0 to 10);
  • the position and texture (colour pattern) of every object, plus the textures of the table, floor, background and robot;
  • the camera's position, angle and field of view, within small ranges around the real camera's placement;
  • the number of lights and their positions, directions and shininess;
  • the type and amount of random noise added to the image.

Textures were simple: a single random colour, a gradient between two random colours, or a checkerboard of two random colours. The result looks nothing like reality, which is the point. Generate a few scenes below.

A cartoon of the paper's scene generator (its renderer was MuJoCo's built-in one, and its real images came from a webcam). The target is the object in the dashed box; its true position is the label the network learns to output. Switch factors off to see what the network would otherwise be able to rely on. "Plain stand-in" draws one sober, consistent scene to play the role of the real world: from the network's point of view it is just one more variation.

TakeawayIf colours, lighting, camera and clutter change in every training image, the only thing the network can rely on is what actually matters: the object's shape and position.

Check yourself
Why randomise the camera position slightly, rather than fixing it exactly where the real camera is?
The real camera's position is never measured perfectly. Training across a small range makes the network robust to that error (and the paper's ablation found this mattered).
Why add distractor objects?
So the network learns to find the target shape rather than "the only object on the table". Without distractors in training, its error with real distractors rose several-fold.

Idea · §IV.C · Fig. 5

How much variety is enough?

A network trained on a few textures can still do well on the simulator, because it learns to "subtract" those particular backgrounds. On a new background it has never seen, that trick fails. Only when it has seen a huge variety does it learn to ignore backgrounds in general. The paper found that real-world accuracy degraded significantly with fewer than about 1,000 distinct textures in training.

The toy below shows the same effect in miniature. Each "image" is a row of 32 pixels: a bright blob (the object) on a background pattern. A simple learner estimates the blob's position. It trains on 600 images using a limited set of background patterns, then is tested on images with brand-new patterns, playing the part of the real world.

An illustrative toy, not the paper's model: a linear (ridge) regression from 32 pixel values to the blob position, trained on 600 images. Backgrounds are random flat colours, gradients or checkers, like the paper's texture types. Each strip is one image; the ring marks the true position, the amber tick the prediction. Left: images using training patterns. Right: images with patterns never seen in training. With 1 or 2 patterns the gap between left and right is large; by 100 or so it nearly closes.

TakeawayDoing well on simulated images is easy; doing well on unseen ones needs a very wide variety of randomised appearances.

Idea · §IV · Tables I–II

Results, and what mattered

The network looks at a single image from an ordinary webcam and outputs the position of one particular object. On real images, its average error was about 1.5 cm across eight shapes (cone, cube, cylinder, prisms, pyramid, tetrahedron), and it coped with distractors and partial occlusion. On simulated images it was much more accurate (0.3 to 0.5 cm), so it still overfits to the simulator somewhat.

The authors switched ingredients off one at a time. Removing distractors from training was the most damaging: with real distractors on the table, the average error jumped from 1.8 cm to 7.2 cm. Removing camera randomisation made every condition worse by half a centimetre or more. Removing noise barely changed accuracy, though it helped training converge. Starting from a network pre-trained on ordinary photos (ImageNet) helped most when training data were scarce; with tens of thousands of simulated images, starting from scratch did almost as well.

Finally, a Fetch robot used the detector to grasp target objects from increasingly cluttered tables, succeeding in 38 of 40 attempts.

Later refinementDomain randomisation became a standard tool. Peng et al. (2018) randomised physics (masses, friction, delays) as well as appearance. OpenAI's robot hand (2018) learned to rotate a block, and later (2019) to solve a Rubik's Cube, trained entirely in randomised simulation; the second used "automatic domain randomisation", which widens the random ranges as the robot improves. Other work learned to translate randomised images into a canonical look (RCAN, 2019) or adapted with a little real data.

TakeawayTrained only on cartoonish random images, a detector reached about 1.5 cm real-world accuracy. Variety in clutter and camera mattered most.

Follow the idea

This is the newest step on its timeline so far.