The paper in four sentences
Language models can break a request like "I spilled my drink, can you help?" into sensible steps, but they don't know what the robot can see, reach or hold, so their plans are often impossible.
The robot already has a few hundred trained skills ("pick up the sponge", "go to the table"), and each skill comes with a learned estimate of how likely it is to succeed in the current situation.
SayCan scores every skill twice, once by the language model ("is this a useful next step?") and once by the skill's own estimate ("can I actually do this now?"), multiplies the two, does the winner, and repeats.
On 101 real kitchen instructions, a mobile robot planned correctly 84% of the time and carried the plan out 74% of the time, and the grounding step was what made the difference.
Idea · §1
A clever adviser who has never seen the room
A large language model (LLM) is a program trained on huge amounts of text to predict what words come next. Because so much human writing describes how things are done, an LLM "knows" that cleaning a spill involves a sponge, and that a sponge has to be fetched before it can be used.
But the model has no body and no eyes. Ask it to help a robot and it may say "use the vacuum cleaner" when there isn't one, or "wipe the table" when the robot can't wipe anything. The paper calls this the problem of grounding: connecting the words to what is actually possible in this place, for this robot, right now.
The title is a twist on "do as I say, not as I do". Here the robot is saying to the language model: do as I can, not as you would say.
TakeawayLanguage models know how tasks are usually done, but not what is possible here and now. That missing link is called grounding.
Idea · §3 · Fig. 1, Fig. 3
Say times Can
The robot comes with a fixed menu of skills, each with a short text label such as "find a sponge" or "pick up the apple". SayCan never lets the language model write free text. Instead, for each label on the menu, it asks the model how likely it would be to say exactly that as the next step. That is the Say score: is this step useful for the request?
Each skill also has its own learned predictor that looks at the camera image and estimates how likely the skill is to succeed from where the robot is now. That is the Can score: is this step possible?
SayCan multiplies the two and does the skill with the highest product. Then it adds that step to the plan so far, looks at the world again, and scores the whole menu once more. It stops when the best choice is "done".
TakeawayFor every skill on the menu, multiply "would the language model suggest this?" by "can the robot do this from here?", do the best one, and repeat.
Check yourself
- Why does SayCan score a fixed menu of skills instead of letting the language model write its own next step?
- Free text might name an action the robot has no skill for, or phrase it in a way no skill matches. Scoring only the menu guarantees every chosen step is something the robot knows how to try.
- If a step is very useful but the robot is very unlikely to manage it right now, what happens to its combined score?
- It becomes small, because a large number times a number near zero is near zero. A less ideal but feasible step (such as "find a sponge" first) wins instead.
Idea · §2, §4 · Fig. 2
Where the Can score comes from
The Can score is not written by a person. Each skill was trained with reinforcement learning, where the robot tries things and gets a reward of 1 for success and 0 for failure. A by-product of that training is a value function: a learned guess, from the current camera image, of how much reward the skill will collect if you run it now. With a reward of exactly 1 for success, that guess is simply the chance of success.
So "pick up the apple" has a high value when an apple is in front of the robot, and a low one when the robot is staring at an empty corridor. The authors call these affordances, a psychology word for "what the surroundings let you do". The robot has 551 skills in total, built from seven families of actions (such as picking, placing and moving to a location) applied to different objects and places.
TakeawayThe Can score is a learned chance of success, read from the camera image. It is what ties the language model's ideas to the real scene.
Check yourself
- Why does a 1-for-success, 0-otherwise reward make the value function a probability?
- The value is the expected reward. If reward is 1 on success and 0 on failure, the expected reward equals the probability of success.
- Does the Can score know what the person asked for?
- No. It only depends on the skill and the current observation. Usefulness for the request is the language model's job.
Idea · §5 · Tables 2–3
What happened in a real kitchen
The team tested a mobile robot with one arm on 101 instructions, from simple ("pick up the apple") to long and vague ("I just worked out, can you bring me a drink and a snack to recover?"). Each run was judged by human raters on two things: was the plan right, and did the robot actually execute it?
Using Google's 540-billion-parameter PaLM language model, SayCan planned correctly 84% of the time in a mock kitchen and executed 74% successfully. In a real office kitchen it dropped only a little, to 81% and 60%. Removing the Can score, so that the language model plans blind, cut planning success to 67%.
TakeawayWith grounding, the robot got most real requests right; without it, planning success fell by about a fifth.
Idea · §5.1, §8
What still goes wrong, and what came next
The authors traced about two-thirds of failures to the language model (for example misunderstanding "a snack that isn't an apple", or stopping a long task too early) and about a third to the skills' value functions misjudging the scene. They also say plainly that SayCan can only ever be as good as its skill menu: if the robot has no "open the fridge" skill, no amount of clever planning will open the fridge.
Another limit is that the planner only looks at the robot's situation through the Can scores. If a skill reports high value but then fails, the language model is not told what went wrong.
TakeawaySayCan showed that a language model can plan for a robot if every suggestion is checked against what the robot can do; its ceiling is the size of its skill menu.
Check yourself
- Why can't a better language model alone fix SayCan's biggest bottleneck?
- Because the robot can only execute skills on its fixed menu, and their reliability limits execution. A smarter planner still has to pick from the same skills.
Abstract, restated
Large language models encode rich semantic knowledge useful for following high-level, temporally extended instructions, but they lack real-world experience, so their suggestions may be infeasible for a given embodiment and scene. SayCan grounds the LLM with pretrained skills: the LLM provides task-grounding (which skill is useful for the instruction) and value functions of the skills provide world-grounding (which skill is feasible in the current state). Evaluated on a real mobile manipulator over 101 long-horizon, abstract instructions in two kitchens. Ahn et al. (Robotics at Google and Everyday Robots), CoRL 2022; arXiv 2204.01691 (v2 added PaLM results, drawer skills, chain-of-thought, multilingual queries and an open-source tabletop version).
Section · §2–3
Problem statement and factorisation
The system receives a natural-language instruction i and has a set of skills Π. Each skill π ∈ Π is a policy with a short language description ℓπ (for example "find a sponge") and an affordance function p(cπ | s, ℓπ), the probability that executing π from state s completes ℓπ (cπ is the Bernoulli completion variable). The LLM supplies p(ℓπ | i), the probability that ℓπ is a valid next step for i. SayCan assumes the probability that skill π makes progress on i factorises as
The paper calls the LLM term task-grounding and the affordance term world-grounding. The LLM is used in scoring mode: rather than sampling text, SayCan computes the likelihood of each candidate skill description given the prompt, instruction and steps so far. This sidesteps two failure modes of generative use, namely outputs that map to no available skill and outputs that are phrased so that they cannot be parsed.
Affordances come from reinforcement learning (§2, "Value functions and RL"). With a sparse reward of 1 on success and 0 otherwise and no discount, the value function Vπ(s) of a language-conditioned policy equals the probability of success, so it is directly an affordance function. The paper's Fig. 2 visualises pick-skill values: high for objects in view, uniformly low when the robot faces empty space.
Check yourself
- What independence assumption hides in the product?
- That usefulness for the instruction (judged from language and plan history only) and feasibility (judged from the state and the skill only) contribute separately. The LLM never sees the image, and the value function never sees the instruction.
- Why does undiscounted sparse reward matter for the interpretation?
- With discounting, values would also encode how long success takes; with shaped rewards they would not be probabilities. With reward in {0, 1} at termination and γ = 1, V(s) = Pr(success | s).
Section · §3 · Algorithm 1 · App. D
The planning loop, prompt and termination
Algorithm 1 SayCan given instruction i, initial state s₀, skills Π with descriptions ℓ_Π n ← 0, π ← ∅ while ℓ_{π_{n−1}} ≠ "done": C ← ∅ for π ∈ Π, ℓ_π ∈ ℓ_Π: p_LLM ← p(ℓ_π | i, ℓ_{π_{n−1}}, …, ℓ_{π_0}) # task-grounding p_aff ← p(c_π | s_n, ℓ_π) # world-grounding C ← C ∪ { p_aff · p_LLM } π_n ← argmax_{π ∈ Π} C execute π_n(s_n), observe s_{n+1}; n ← n + 1
The step chosen at each iteration is appended to the LLM context, so the next query scores candidates conditioned on the plan so far. The planner is greedy: there is no search over full plans and no backtracking.
Prompt. A dialogue-style few-shot prompt (App. D, Listing 1) opens "Robot: Hi there, I'm a robot operating in an office kitchen" and then gives worked examples in the form "Human: … Robot: I would: 1. find a sponge, 2. pick up the sponge, 3. bring it to you, 4. done." The full prompt has 17 examples.
Termination. "done" is scored as a skill like any other, with a fixed affordance so it remains available. Getting the LLM to terminate turned out to depend heavily on prompting: the appendix reports planning success of only 10% on termination-requiring tasks with no examples, rising to 88% with the full 17-example prompt. Early termination was the dominant failure on long-horizon instructions.
Step through the loop below with your own numbers. Each candidate's Say score is fixed for the step; drag the Can sliders to see how scene feasibility re-ranks them.
Section · §4 · App. C
Skills, policies and value functions
Robot. Everyday Robots mobile manipulator with a 7-DoF arm and a two-finger gripper, RGB camera observations. Action space: 6-DoF end-effector pose, gripper open/close, base x–y and yaw deltas, and a terminate action.
Skills. 551 skills spanning 7 skill families and 17 objects (families such as picking, placing, rearranging and navigating to locations). The evaluation scene uses 15 objects and 5 semantic locations (two counters, a table, a trash can and the user).
Policies. Two routes were used. Image-based behavioural cloning following BC-Z, trained on teleoperated demonstrations, and multi-task RL following MT-Opt, bootstrapped from simulation (with RetinaGAN sim-to-real image transfer) and then improved with real-robot data. Policies and value functions are conditioned on the skill description through a frozen Universal Sentence Encoder (USE) embedding. BC policies had higher success at the time, so they were used for execution where available, while the RL-trained Q-functions supplied the affordances for every skill.
Rewards are sparse and come from human raters watching episode videos, with a positive label requiring agreement from two of three raters. Value functions are trained with temporal-difference backups on these labels. For some skills the paper calibrates raw values into probabilities with fixed ranges (App. D gives pick skills vmax = 0.5, vmin = 0.2).
The equation is the generic TD objective of §2 written in squared form; MT-Opt's specific target network and cross-entropy variant are details of that cited work.
Section · §5 · Tables 1–3
Evaluation and results
Benchmark (Table 1). 101 instructions in seven families: NL Single Primitive (15), NL Nouns (15), NL Verbs (15), Structured Language (15), Embodiment (11, where the right plan depends on the robot's current state), Crowd-Sourced (15) and Long-Horizon (15). Two environments: the mock kitchen where skills were trained and a real office kitchen. Plan success and execution success are both judged by two of three human raters.
| Variant (101 instructions; mock kitchen unless stated) | Plan | Execute |
|---|---|---|
| PaLM-SayCan (540B) | 84% | 74% |
| PaLM-SayCan, real kitchen | 81% | 60% |
| FLAN-SayCan (137B) | 70% | 61% |
| No value function (LLM scoring only) | 67% | – |
| Generative (sample, then project to nearest skill with USE) | 74% | – |
| BC NL (full instruction straight to the policy) | – | 0% |
| BC USE (instruction projected to nearest skill) | – | 60% on NL Single only, 0% elsewhere |
Error analysis (§5.1). 65% of failures were attributed to the LLM and 35% to affordances. LLM failures included negation ("a snack that isn't an apple"), ambiguous references, and early termination. Model size (App. E). PaLM 8B, 62B and 540B and FLAN 137B were compared; larger is generally better, with a small gap between 62B and 540B. The planning performance tracks LLM quality, which the authors highlight as a way robotics benefits directly from language-model progress.
New capabilities (§5.2). Adding drawer skills: 21 queries, 100% plan success, 33% execution, with failures in the chained manipulation policy. Chain-of-thought: the prompt gains an "Explanation" line generated before scoring, which helped with negation and multi-step reasoning (qualitative examples in Table 4). Multilingual queries in Chinese, French and Spanish showed almost no drop in planning success.
Check yourself
- Why does BC NL score 0% while SayCan scores 74%, given that both use the same low-level policies?
- The policies were trained on short skill descriptions. A full abstract instruction like "bring me something to clean a spill" is out of distribution for them; SayCan decomposes it into in-distribution skill commands.
- Which ablation isolates the value of world-grounding?
- "No VF": the same LLM scoring with affordances removed drops planning from 84% to 67%.
Section · §6–8 · Lineage
Limitations and what replaced it
- Skill bottleneck. §8 states that the primary bottleneck is the range and capability of the underlying skills. The planner cannot exceed the skill library.
- Open loop at the plan level. The LLM sees only the history of chosen skill names, not whether they succeeded. A skill with high value that fails is not easily recovered from.
- Greedy, factorised decisions. No lookahead; the LLM and affordance terms never see each other's inputs.
- Inherited LLM biases and the cost of querying a 540B model for every skill at every step (the paper scores all candidates).
- An open-source tabletop version (§6) replaces the real robot with a simulated pick-and-place environment using CLIPort-style policies and ViLD-detected objects for affordances.
Reference
Glossary
- Grounding
- Connecting language to what is perceivable and achievable in the physical world.
- Affordance
- What an environment permits an agent to do; here p(cπ | s, ℓπ), a skill's success probability from state s.
- Task-grounding
- The LLM term p(ℓπ | i): usefulness of a skill for the instruction.
- World-grounding
- The affordance term: feasibility in the current state.
- Scoring mode
- Using an LLM to compute likelihoods of given strings rather than sampling new text.
- Value function
- Expected future reward from a state; with 0/1 terminal reward, a success probability.
- BC-Z / MT-Opt
- Prior Google systems for language-conditioned behavioural cloning and multi-task RL, used to train the skills.
- USE
- Universal Sentence Encoder, a frozen text embedding used to condition policies on skill descriptions.