The paper in four sentences
Chain-of-thought prompting lets a language model reason step by step, but only from memory, so it confidently reasons from facts it has made up.
Other work let models act (search, click, move in a game) but without explicit reasoning, so they lose track of what they are doing.
ReAct shows the model a few examples in which "Thought", "Action" and "Observation" lines alternate, so it learns to plan in words, act, read the result, and update its plan.
On fact-checking and multi-step questions this made the model more trustworthy and, combined with plain reasoning, more accurate; on a household text game and an online shopping site it beat methods trained on thousands of examples, using just one or two examples in the prompt.
Idea · §1 · Fig. 1
Thinking without looking, looking without thinking
Two lines of work were developing separately in 2022. Chain-of-thought prompting showed that if you give a large language model examples where the reasoning is written out, it reasons step by step and solves harder problems. But it reasons only from what it remembers. If its memory is wrong, every later step builds on a mistake. This is called hallucination: fluent, confident, made-up facts.
Meanwhile, other work used language models to choose actions in an environment, such as which web page to open or which button to press. These models act but do not explain to themselves why, so on long tasks they repeat themselves or wander off.
ReAct ("reason + act") puts both in one stream of text. Step through the three styles on the same question below.
TakeawayReasoning alone can drift into made-up facts; acting alone loses track of the goal. Interleaving them lets each fix the other's weakness.
Check yourself
- What does "hallucination" mean for a language model?
- Producing a confident, fluent statement that is false, typically a made-up fact, because it is generated from patterns in memory rather than checked against a source.
Idea · §2 · §3.1
A thought is an action that changes nothing outside
The paper's formal idea is simple. An agent normally chooses from a fixed set of actions, like "search" or "go to the kitchen". ReAct adds one more kind of action: writing a thought. A thought does not change the world and gets no reply from it. It only adds text to the model's own notes, which then shape what it does next.
Thoughts do different jobs: breaking the goal into steps ("I need to find who wrote it, then where they were born"), pulling out the key fact from a long result, noticing a mistake ("this page is about the film, not the book"), or deciding when to stop.
For the question-answering tasks, the model got a deliberately simple Wikipedia tool with three actions: search for a page (returns its first few sentences), lookup a word in the current page (like Ctrl+F), and finish with an answer. No training was involved: the model, Google's 540-billion-parameter PaLM, just saw six worked examples in its prompt.
TakeawayReAct treats "think" as one more action the model can take, so planning and tool use live in a single text stream.
Idea · §3.3 · Table 2
Fewer invented facts, different mistakes
The authors read through 200 of the model's attempts at multi-step questions (HotpotQA) and sorted what went right and wrong. Choose a style below.
The pattern is clear. Hallucination caused 56% of chain-of-thought's failures and none of ReAct's. But ReAct has its own weaknesses: it more often gets its reasoning tangled (sometimes looping on the same search), and an unhelpful search result can derail it completely.
TakeawayLooking things up nearly eliminates invented facts, but makes the model dependent on good search results and good bookkeeping.
Check yourself
- Why is ReAct's answer more trustworthy even when the accuracy is similar?
- Its reasoning is tied to observations you can read, so you can check where each fact came from; chain-of-thought's correct answers more often rest on hallucinated steps (14% vs 6% of correct answers).
Idea · §3.2–3.3 · Table 1
Best of both: fall back when unsure
On raw accuracy, ReAct alone did not beat chain-of-thought on the question benchmark (27.4% against 29.4% exact match), though it did better on fact-checking. The two make different mistakes, so the authors combined them: if ReAct fails to finish within a few steps, fall back to chain-of-thought; or if chain-of-thought's answers disagree with each other a lot, switch to ReAct. Combinations scored best on both benchmarks among the prompting methods, though still well below systems trained specifically for these tasks.
TakeawayInternal knowledge and external lookup are complementary; knowing when to trust each gave the best results.
Idea · §4 · Tables 3–4
Acting in games and shops
ReAct was also tested on two decision-making environments. ALFWorld is a text adventure set in a home ("go to countertop 1", "take mug 2", "heat mug with microwave"). WebShop is a simulated shopping site where the model must find a product matching a request such as a particular size and price.
Here thoughts are used sparingly, just when needed: to set subgoals ("first I need to find a mug; it is probably in a cabinet") and to check progress. With only one or two examples in the prompt, ReAct succeeded on 71% of ALFWorld tasks (best of six prompts) against 37% for a system trained by imitating about 100,000 expert demonstrations per task type. On WebShop it raised the success rate by 10 percentage points over methods trained by imitation and reinforcement learning (40.0% against 28.7–29.1%), though still well short of human experts (59.6%).
TakeawayThe same think-act-observe loop worked for question answering, household tasks and web shopping, from a handful of examples: the template for today's agents.
Abstract, restated
LLMs' abilities for reasoning (e.g. chain-of-thought prompting) and acting (e.g. action-plan generation) had been studied separately. ReAct uses LLMs to generate reasoning traces and task-specific actions in an interleaved manner: traces help induce, track and update action plans and handle exceptions, while actions let the model interface with external sources such as knowledge bases or environments. On HotpotQA and FEVER, interacting with a simple Wikipedia API overcomes hallucination and error propagation in chain-of-thought and produces more interpretable, human-like trajectories. On ALFWorld and WebShop, ReAct outperforms imitation and reinforcement learning methods by an absolute success rate of 34% and 10% respectively, prompted with only one or two in-context examples. Yao et al., ICLR 2023.
Section · §2
Formulation: augmenting the action space with language
An agent at time t receives observation ot ∈ 𝒪 and takes action at ∈ 𝒜 following policy π(at | ct), where the context is ct = (o1, a1, …, ot−1, at−1, ot). Learning a policy is hard when the mapping ct ↦ at is highly implicit and needs extensive computation.
ReAct augments the action space to 𝒜̂ = 𝒜 ∪ ℒ, with ℒ the space of language. A thought ât ∈ ℒ does not affect the environment and yields no observation; it only updates the context:
Because ℒ is unbounded, learning in this space is hard without a strong prior; the paper uses a frozen LLM, PaLM-540B, prompted with few-shot human-written trajectories, to generate both domain actions and free-form thoughts. For knowledge-intensive reasoning, thoughts and actions alternate densely (thought–action–observation steps); for decision making with many actions, thoughts appear sparsely and the model itself decides when to think.
Action space (§3.1). search[entity] returns the first 5 sentences of the entity's Wikipedia page if it exists, else the top-5 similar entities; lookup[string] returns the next sentence containing the string in the current page; finish[answer] ends the episode. The API is deliberately weaker than state-of-the-art retrievers, forcing the model to retrieve via explicit reasoning.
Check yourself
- Formally, why can't a thought be "wrong" with respect to the environment?
- It has no effect on the environment state and produces no observation; it only changes the context the policy conditions on. Its value is entirely in how it changes later environment actions.
Section · §3.2–3.3 · Table 1
Knowledge-intensive reasoning: HotpotQA and FEVER
HotpotQA (multi-hop QA, exact match) and FEVER (claim verification: SUPPORTS / REFUTES / NOT ENOUGH INFO, accuracy), question-only setting, with 6 (HotpotQA) and 3 (FEVER) hand-written trajectories in the prompt. Baselines: Standard (no thoughts or actions), CoT, CoT-SC (self-consistency: 21 sampled CoT trajectories at temperature 0.7, majority vote) and Act (actions without thoughts).
Combining internal and external knowledge. Two heuristics: ReAct → CoT-SC backs off to self-consistency when ReAct does not return an answer within 7 (HotpotQA) or 5 (FEVER) steps; CoT-SC → ReAct switches to ReAct when the majority answer among n CoT-SC samples occurs fewer than n/2 times, signalling low internal confidence. Try the decision rule:
Error analysis (Table 2). Of 200 sampled trajectories, among successes ReAct had 94% grounded true positives vs 86% for CoT (6% vs 14% with hallucinated reasoning or facts); among failures, hallucination accounts for 56% of CoT's and 0% of ReAct's, while ReAct shows more reasoning errors (47% vs 16%, including repetitive loops) and search-result errors (23%). The authors attribute ReAct's lower HotpotQA EM to reduced flexibility: the structural constraint of grounding in observations makes reasoning harder when retrieval is uninformative.
Fine-tuning (Fig. 3). Bootstrapping 3,000 correct ReAct trajectories and fine-tuning PaLM-8B and PaLM-62B reverses the picture: prompted ReAct is the weakest of the four methods at small scale, but fine-tuned ReAct is the strongest, with fine-tuned PaLM-8B ReAct beating all PaLM-62B prompting methods and fine-tuned 62B beating all 540B prompting methods.
Section · §4 · Tables 3–4
Decision making: ALFWorld and WebShop
ALFWorld (text-based household tasks aligned with the embodied ALFRED benchmark; six task types such as pick-and-place, clean, heat, cool, examine-in-light, pick two). Prompts contain sparse thoughts for goal decomposition, subgoal tracking, commonsense about where objects are, and progress checks; 3 annotated trajectories per task type, combined into 6 prompts per type. WebShop: 1.18M real products and 12k human instructions; the agent searches, clicks and chooses options, and is scored by attribute match (score) and full success rate (SR). One-shot prompts.
| ALFWorld (Table 3) | Overall success |
|---|---|
| Act (best of 6 prompts) | 45% |
| ReAct (average / best of 6) | 57% / 71% |
| ReAct-IM (average / best of 6) | 48% / 53% |
| BUTLER (imitation learning, best of 8) | 37% |
| WebShop (Table 4) | Score | Success rate |
|---|---|---|
| IL (imitation learning) | 59.9 | 29.1 |
| IL + RL | 62.4 | 28.7 |
| Act | 62.3 | 30.1 |
| ReAct | 66.6 | 40.0 |
| Human expert | 82.1 | 59.6 |
ReAct vs Inner Monologue. ReAct-IM restricts thoughts to the style of Inner Monologue (Huang et al., 2022): dense descriptions of the environment and of what remains to be done. It drops to 53% (best of 6) because it lacks high-level goal decomposition and commonsense about object locations, and often fails to recognise when a subgoal is complete. The point is that the content of thoughts matters, not just their presence. Compare SayCan, where an LLM proposes steps and learned value functions score feasibility, without free-form reasoning.
Section · §3.3 · §6 · Lineage
Limitations and lineage
- Prompted accuracy. Alone, ReAct is slightly worse than CoT on HotpotQA EM; most of the gain needs the CoT-SC combinations or fine-tuning.
- Brittleness to tools. Uninformative search results derail reasoning (23% of failures), and the model can loop, repeating thoughts and actions.
- Context length. Complex tasks with large action spaces need more demonstrations, which can exceed the in-context input limit; the authors point to fine-tuning and multi-task training as remedies.
- Gap to humans. WebShop success remains far below human experts.
- Evaluation. Results use a single large proprietary model (PaLM-540B), with GPT-3 replications in the appendix.
Check yourself
- Why does fine-tuning help ReAct more than CoT at small scale?
- Prompting a small model with a few trajectories is not enough to teach the joint skill of reasoning and acting; fine-tuning on 3,000 trajectories teaches the generalisable skill of retrieving and reasoning from observations, whereas fine-tuned CoT mainly memorises (possibly hallucinated) facts.
Reference
Glossary
- Trajectory
- The full sequence of thoughts, actions and observations for one task.
- CoT
- Chain-of-thought prompting: demonstrations include written-out reasoning.
- CoT-SC
- Self-consistency: sample many CoT answers and take the majority.
- Act
- The ablation with actions and observations but no thoughts.
- EM
- Exact match with the reference answer.
- ALFWorld
- Text-game household environment aligned with ALFRED.
- WebShop
- Simulated e-commerce site with real product data and human instructions.
- Hallucination
- A generated statement not supported by fact or source.