Study guide · Language models & AI · 2022

ReAct: Synergizing Reasoning and Acting in Language Models

Shunyu Yao et al. · ICLR 2023

Language models could either "think out loud" (reason step by step) or take actions such as searching the web, but not both at once. ReAct prompts a model to alternate: write a thought, take an action, read the result, think again. Grounding the reasoning in what it actually finds cut down made-up facts, and the same loop let it play text games and shop online. The pattern became the basic template for LLM agents.

The paper in four sentences

Chain-of-thought prompting lets a language model reason step by step, but only from memory, so it confidently reasons from facts it has made up.

Other work let models act (search, click, move in a game) but without explicit reasoning, so they lose track of what they are doing.

ReAct shows the model a few examples in which "Thought", "Action" and "Observation" lines alternate, so it learns to plan in words, act, read the result, and update its plan.

On fact-checking and multi-step questions this made the model more trustworthy and, combined with plain reasoning, more accurate; on a household text game and an online shopping site it beat methods trained on thousands of examples, using just one or two examples in the prompt.

Idea · §1 · Fig. 1

Thinking without looking, looking without thinking

Two lines of work were developing separately in 2022. Chain-of-thought prompting showed that if you give a large language model examples where the reasoning is written out, it reasons step by step and solves harder problems. But it reasons only from what it remembers. If its memory is wrong, every later step builds on a mistake. This is called hallucination: fluent, confident, made-up facts.

Meanwhile, other work used language models to choose actions in an environment, such as which web page to open or which button to press. These models act but do not explain to themselves why, so on long tasks they repeat themselves or wander off.

ReAct ("reason + act") puts both in one stream of text. Step through the three styles on the same question below.

An illustrative trajectory written in the style of the paper's Fig. 1, on a small made-up search tool (real facts: Hemingway wrote The Old Man and the Sea, was born in Oak Park, Illinois, and later lived in Key West and Cuba). Amber lines are the model's thoughts, blue its actions, grey what the tool returns. Reason only answers from memory and picks a place associated with Hemingway but not his birthplace. Act only finds the right page but, with no thoughts to track the goal, grabs the wrong fact.

TakeawayReasoning alone can drift into made-up facts; acting alone loses track of the goal. Interleaving them lets each fix the other's weakness.

Check yourself
What does "hallucination" mean for a language model?
Producing a confident, fluent statement that is false, typically a made-up fact, because it is generated from patterns in memory rather than checked against a source.

Idea · §2 · §3.1

A thought is an action that changes nothing outside

The paper's formal idea is simple. An agent normally chooses from a fixed set of actions, like "search" or "go to the kitchen". ReAct adds one more kind of action: writing a thought. A thought does not change the world and gets no reply from it. It only adds text to the model's own notes, which then shape what it does next.

Thoughts do different jobs: breaking the goal into steps ("I need to find who wrote it, then where they were born"), pulling out the key fact from a long result, noticing a mistake ("this page is about the film, not the book"), or deciding when to stop.

For the question-answering tasks, the model got a deliberately simple Wikipedia tool with three actions: search for a page (returns its first few sentences), lookup a word in the current page (like Ctrl+F), and finish with an answer. No training was involved: the model, Google's 540-billion-parameter PaLM, just saw six worked examples in its prompt.

Everyday pictureDoing a research task with a notepad beside the keyboard. You jot "first find the author", search, jot "born in Oak Park; that's in Illinois", then write the answer. The notepad doesn't change the internet, but it keeps your searches on track.

TakeawayReAct treats "think" as one more action the model can take, so planning and tool use live in a single text stream.

Idea · §3.3 · Table 2

Fewer invented facts, different mistakes

The authors read through 200 of the model's attempts at multi-step questions (HotpotQA) and sorted what went right and wrong. Choose a style below.

Real numbers from Table 2 (PaLM-540B on HotpotQA). Top: among correct answers, the share with a genuinely sound path versus a lucky or hallucinated one. Bottom: among wrong answers, the share caused by each kind of mistake. "Label ambiguity" means the answer was arguably right but did not match the expected wording.

The pattern is clear. Hallucination caused 56% of chain-of-thought's failures and none of ReAct's. But ReAct has its own weaknesses: it more often gets its reasoning tangled (sometimes looping on the same search), and an unhelpful search result can derail it completely.

TakeawayLooking things up nearly eliminates invented facts, but makes the model dependent on good search results and good bookkeeping.

Check yourself
Why is ReAct's answer more trustworthy even when the accuracy is similar?
Its reasoning is tied to observations you can read, so you can check where each fact came from; chain-of-thought's correct answers more often rest on hallucinated steps (14% vs 6% of correct answers).

Idea · §3.2–3.3 · Table 1

Best of both: fall back when unsure

On raw accuracy, ReAct alone did not beat chain-of-thought on the question benchmark (27.4% against 29.4% exact match), though it did better on fact-checking. The two make different mistakes, so the authors combined them: if ReAct fails to finish within a few steps, fall back to chain-of-thought; or if chain-of-thought's answers disagree with each other a lot, switch to ReAct. Combinations scored best on both benchmarks among the prompting methods, though still well below systems trained specifically for these tasks.

TakeawayInternal knowledge and external lookup are complementary; knowing when to trust each gave the best results.

Idea · §4 · Tables 3–4

Acting in games and shops

ReAct was also tested on two decision-making environments. ALFWorld is a text adventure set in a home ("go to countertop 1", "take mug 2", "heat mug with microwave"). WebShop is a simulated shopping site where the model must find a product matching a request such as a particular size and price.

Here thoughts are used sparingly, just when needed: to set subgoals ("first I need to find a mug; it is probably in a cabinet") and to check progress. With only one or two examples in the prompt, ReAct succeeded on 71% of ALFWorld tasks (best of six prompts) against 37% for a system trained by imitating about 100,000 expert demonstrations per task type. On WebShop it raised the success rate by 10 percentage points over methods trained by imitation and reinforcement learning (40.0% against 28.7–29.1%), though still well short of human experts (59.6%).

Later refinementThe Thought / Action / Observation loop became the default design for LLM "agents" and was built into popular frameworks. Later work let models write their own feedback and retry (Reflexion), learn when to call tools during training (Toolformer), and explore several reasoning branches (Tree of Thoughts, by the same first author). Chat models now have structured "tool calling" built in, which is ReAct's idea moved from prompt format into model training. Giving models tools also opened new risks: text returned by a tool can try to take over the agent (indirect prompt injection).

TakeawayThe same think-act-observe loop worked for question answering, household tasks and web shopping, from a handful of examples: the template for today's agents.

Follow the idea

Built on: RAG (2020), Chain of thought (2022)