Study guide · Language models & AI · 2022

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Jason Wei et al. · NeurIPS 2022

Large language models were surprisingly poor at multi-step word problems, even when shown examples. Wei and colleagues find a remarkably simple fix: in the examples you give the model, write out the intermediate reasoning, not just the answer. Large models then produce their own step-by-step reasoning, and their accuracy on maths, commonsense and symbolic puzzles jumps, sometimes past fine-tuned systems.

The paper in four sentences

You can teach a large language model a task just by putting a few solved examples in front of the question ("few-shot prompting"), but for problems that need several steps, showing only the final answers works poorly.

If the examples instead include the reasoning in plain words ("Roger started with 5 balls. 2 cans of 3 tennis balls each is 6…"), the model writes similar reasoning before its own answer, a "chain of thought".

This helped only very large models (around 100 billion parameters and up); smaller models wrote fluent but wrong reasoning and sometimes got worse.

With eight such examples, the 540-billion-parameter PaLM model solved 56.9% of grade-school maths problems (up from 17.9%), beating the previous best, a fine-tuned GPT-3 with a separate answer checker. A few months later, others showed that simply adding "Let's think step by step" also works.

Idea · §1

Teaching by example, inside the prompt

A language model continues whatever text it is given. GPT-3 showed you can exploit this to "teach" a task without any retraining: write a few example questions with their answers, then the new question, and the model continues with an answer in the same style. This is few-shot prompting; the examples are called exemplars.

It works well for simple tasks but badly for problems needing several steps of reasoning, like maths word problems. And making models bigger helped less on these than on other tasks; the curve was nearly flat. Older approaches trained models specifically to write out solutions, which needs large datasets of worked solutions for every task.

Everyday pictureShowing a student an answer key ("Q1: 11, Q2: 9") teaches them nothing about how to solve Q3. Showing them the worked solutions does.

TakeawayFew-shot prompting teaches a task through examples in the prompt, but answer-only examples don't teach multi-step reasoning.

Idea · §2 · Fig. 1

Show the working

The paper's idea is simply to put the reasoning into the exemplars. Each example now goes question, then a short chain of reasoning in plain language, then the answer. Faced with a new question, the model imitates the pattern: it writes its own chain of reasoning, then an answer.

Toggle between the two prompts below. These are the paper's own example and the model's actual outputs from its Figure 1.

Text from the paper's Fig. 1. Top box: the prompt, one exemplar plus the new question (the real experiments used eight exemplars). Bottom box: what the model wrote. The only difference between the prompts is the highlighted reasoning in the exemplar's answer.

TakeawayWrite the reasoning into the examples and a large model will write reasoning into its answers.

Check yourself
Was the model retrained or changed in any way?
No. Only the prompt changed. Chain-of-thought prompting works with an off-the-shelf model, which is one of its main attractions.
Why might writing intermediate steps help a model that produces one word at a time?
Each step it writes becomes part of the input for the next words, so it can use earlier results (like "23 − 20 = 3") instead of having to jump to the final number in one go. It also spends more computation on harder problems, since longer problems get longer chains.

Idea · §3.2 · Fig. 4, Table 2

Only big models can do it

The authors tried three families of models at many sizes. For small models, chain-of-thought prompting did nothing or made things worse: they produced reasoning that read smoothly but didn't make sense. Only at around 100 billion parameters did the benefit appear, and then it was large.

The authors describe this as an emergent ability of scale: something that is absent in small models and appears in large ones, rather than improving gradually. Pick a family below to see the paper's numbers on GSM8K, a set of grade-school maths word problems.

Real numbers from the paper's Table 2 (GSM8K solve rate, %). Model size on a log scale. Grey: standard prompting. Amber: chain-of-thought prompting. Dashed line: the previous best (55%, a GPT-3 fine-tuned on worked solutions, plus a trained verifier, Cobbe et al. 2021).

TakeawayChain of thought is useless or harmful for small models and dramatically helpful for very large ones.

Check yourself
What happens to the smallest GPT-3 model with chain-of-thought prompting?
It gets worse: 2.2% with standard prompting, 0.5% with chain of thought. Small models imitate the form of reasoning without being able to do it.

Idea · §4–5 · Tables 4–5

Not just arithmetic

The same trick helped on commonsense questions. For example, on deciding whether a sports sentence is plausible ("Joao Moutinho caught the screen pass in the NFC championship"; he is a footballer, so no), PaLM 540B went from 80.5% to 95.4%, above the 84% of an unaided sports enthusiast. On StrategyQA, questions like "Would a pear sink in water?" that need several implicit steps, it went from 68.6% to 77.8%.

It also helped on symbolic puzzles such as "take the last letter of each word and join them". Most strikingly, after seeing examples with only two words, models using chain of thought could handle three or four words, which standard prompting failed at almost completely. Writing the steps lets the model extend a procedure to longer inputs.

TakeawayChain of thought helps on any task a person would solve by reasoning in words, and helps models generalise to longer problems than in the examples.

Idea · §6 and later

Is it really reasoning? And what came next

The authors are careful: the chains look like human reasoning, but whether the model is truly "reasoning" is left open. Chains can also be wrong while the final answer is right (especially on multiple-choice questions), and the method only helps models that are expensive to run.

Later refinementLater the same year, Kojima et al. (2022) showed that you don't even need worked examples: adding "Let's think step by step" to the question raised InstructGPT's (text-davinci-002) accuracy on MultiArith from 17.7% to 78.7% and on GSM8K from 10.4% to 40.7%. "Self-consistency" (Wang et al., 2022) sampled many chains and took a majority vote over their answers. ReAct interleaved reasoning with actions such as searches. Most recently, reasoning models like DeepSeek-R1 are trained with reinforcement learning to produce long chains of thought by themselves, no prompt needed.

TakeawayPrompting for step-by-step reasoning started a line of work that led to models trained to think before they answer.

Follow the idea

Built on: GPT-3 (2020)

Led to: ReAct (2022), DeepSeek-R1 (2025)