Study guide · Robotics & reinforcement learning · 2022

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances

Michael Ahn et al. · CoRL 2022

A large language model knows a lot about how tasks are done, but it has never seen the room a robot is standing in. SayCan asks how to make its advice usable, and answers by multiplying two scores for every possible next step: how useful the language model thinks the step is, and how likely the robot's own learned skill thinks it is to succeed from here.

The paper in four sentences

Language models can break a request like "I spilled my drink, can you help?" into sensible steps, but they don't know what the robot can see, reach or hold, so their plans are often impossible.

The robot already has a few hundred trained skills ("pick up the sponge", "go to the table"), and each skill comes with a learned estimate of how likely it is to succeed in the current situation.

SayCan scores every skill twice, once by the language model ("is this a useful next step?") and once by the skill's own estimate ("can I actually do this now?"), multiplies the two, does the winner, and repeats.

On 101 real kitchen instructions, a mobile robot planned correctly 84% of the time and carried the plan out 74% of the time, and the grounding step was what made the difference.

Idea · §1

A clever adviser who has never seen the room

A large language model (LLM) is a program trained on huge amounts of text to predict what words come next. Because so much human writing describes how things are done, an LLM "knows" that cleaning a spill involves a sponge, and that a sponge has to be fetched before it can be used.

But the model has no body and no eyes. Ask it to help a robot and it may say "use the vacuum cleaner" when there isn't one, or "wipe the table" when the robot can't wipe anything. The paper calls this the problem of grounding: connecting the words to what is actually possible in this place, for this robot, right now.

The title is a twist on "do as I say, not as I do". Here the robot is saying to the language model: do as I can, not as you would say.

Everyday pictureYou phone a friend who is a brilliant cook for help making dinner. They suggest a recipe that needs an oven, but you're in a hotel room with a kettle. Their knowledge is real; it just needs to be filtered through what your room allows.

TakeawayLanguage models know how tasks are usually done, but not what is possible here and now. That missing link is called grounding.

Idea · §3 · Fig. 1, Fig. 3

Say times Can

The robot comes with a fixed menu of skills, each with a short text label such as "find a sponge" or "pick up the apple". SayCan never lets the language model write free text. Instead, for each label on the menu, it asks the model how likely it would be to say exactly that as the next step. That is the Say score: is this step useful for the request?

Each skill also has its own learned predictor that looks at the camera image and estimates how likely the skill is to succeed from where the robot is now. That is the Can score: is this step possible?

SayCan multiplies the two and does the skill with the highest product. Then it adds that step to the plan so far, looks at the world again, and scores the whole menu once more. It stops when the best choice is "done".

Request: "I spilled my coke, can you bring me something to clean it up?" Blue bars are the Say score, green the Can score, amber their product. The numbers are made up for illustration, but the logic is the paper's. Turn the Can score off and the robot follows the language model alone: it tries to pick up a sponge it is not standing next to, which fails.

TakeawayFor every skill on the menu, multiply "would the language model suggest this?" by "can the robot do this from here?", do the best one, and repeat.

Check yourself
Why does SayCan score a fixed menu of skills instead of letting the language model write its own next step?
Free text might name an action the robot has no skill for, or phrase it in a way no skill matches. Scoring only the menu guarantees every chosen step is something the robot knows how to try.
If a step is very useful but the robot is very unlikely to manage it right now, what happens to its combined score?
It becomes small, because a large number times a number near zero is near zero. A less ideal but feasible step (such as "find a sponge" first) wins instead.

Idea · §2, §4 · Fig. 2

Where the Can score comes from

The Can score is not written by a person. Each skill was trained with reinforcement learning, where the robot tries things and gets a reward of 1 for success and 0 for failure. A by-product of that training is a value function: a learned guess, from the current camera image, of how much reward the skill will collect if you run it now. With a reward of exactly 1 for success, that guess is simply the chance of success.

So "pick up the apple" has a high value when an apple is in front of the robot, and a low one when the robot is staring at an empty corridor. The authors call these affordances, a psychology word for "what the surroundings let you do". The robot has 551 skills in total, built from seven families of actions (such as picking, placing and moving to a location) applied to different objects and places.

Each bar is a skill's learned estimate of its own chance of success in the chosen situation (illustrative values in the spirit of the paper's Fig. 2). Notice that the language request plays no part here: the Can score depends only on what the robot sees.
Everyday pictureYour hands "know" before you try whether a jar lid will open: a dry, loose lid feels easy; a greasy, stuck one doesn't. The value function is the robot's version of that gut feeling, one for each skill.

TakeawayThe Can score is a learned chance of success, read from the camera image. It is what ties the language model's ideas to the real scene.

Check yourself
Why does a 1-for-success, 0-otherwise reward make the value function a probability?
The value is the expected reward. If reward is 1 on success and 0 on failure, the expected reward equals the probability of success.
Does the Can score know what the person asked for?
No. It only depends on the skill and the current observation. Usefulness for the request is the language model's job.

Idea · §5 · Tables 2–3

What happened in a real kitchen

The team tested a mobile robot with one arm on 101 instructions, from simple ("pick up the apple") to long and vague ("I just worked out, can you bring me a drink and a snack to recover?"). Each run was judged by human raters on two things: was the plan right, and did the robot actually execute it?

Using Google's 540-billion-parameter PaLM language model, SayCan planned correctly 84% of the time in a mock kitchen and executed 74% successfully. In a real office kitchen it dropped only a little, to 81% and 60%. Removing the Can score, so that the language model plans blind, cut planning success to 67%.

Real numbers from §5 and Tables 2–3 of the paper, all over the same 101 instructions. "No Can score" plans with the language model alone; "Generative" lets the model write a step freely and then snaps it to the nearest skill on the menu. The ablations were scored for planning only.

TakeawayWith grounding, the robot got most real requests right; without it, planning success fell by about a fifth.

Idea · §5.1, §8

What still goes wrong, and what came next

The authors traced about two-thirds of failures to the language model (for example misunderstanding "a snack that isn't an apple", or stopping a long task too early) and about a third to the skills' value functions misjudging the scene. They also say plainly that SayCan can only ever be as good as its skill menu: if the robot has no "open the fridge" skill, no amount of clever planning will open the fridge.

Another limit is that the planner only looks at the robot's situation through the Can scores. If a skill reports high value but then fails, the language model is not told what went wrong.

Later refinementInner Monologue (Huang et al., 2022) fed success detectors and scene descriptions back into the language model as text, so it could react to failures. RT-2 went further and removed the split between "planner" and "skills": one vision-language model reads the camera and writes the low-level motor commands itself. π0 keeps a SayCan-like split for long tasks, with a high-level language policy choosing short subtasks for a low-level model.

TakeawaySayCan showed that a language model can plan for a robot if every suggestion is checked against what the robot can do; its ceiling is the size of its skill menu.

Check yourself
Why can't a better language model alone fix SayCan's biggest bottleneck?
Because the robot can only execute skills on its fixed menu, and their reliability limits execution. A smarter planner still has to pick from the same skills.

Follow the idea

Led to: RT-2 (2023)