Study guide · Language models & AI · 2025

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

DeepSeek-AI · Nature 2025

Can a language model learn to reason, rather than just imitate worked examples, if you only tell it whether its final answer was right? DeepSeek-R1 says yes: pure reinforcement learning on a strong base model, rewarded for correct answers and a simple format, produced long, self-checking chains of thought on its own (R1-Zero), and a short supervised warm-up plus further training stages made that reasoning readable (R1).

The paper in four sentences

The team took a large pretrained model and trained it with reinforcement learning on maths, coding and logic problems, rewarding it only when its final answer was correct and its reasoning was placed inside <think> tags.

Without ever being shown how to reason, this model, R1-Zero, learned to think for longer and longer, to re-check its work and to try other approaches, and its score on the AIME 2024 maths competition rose from 15.6% to 71.0% (77.9% in the revised version of the paper).

Its reasoning was hard to read and switched between languages, so the full DeepSeek-R1 adds a small set of readable example solutions first, then more rounds of reinforcement learning and fine-tuning, reaching performance comparable to OpenAI's o1 on reasoning tests.

Finally, R1's reasoning traces were used to train much smaller models, which beat the same small models trained with reinforcement learning directly.

Idea · §1

Can reasoning be rewarded into existence?

Large language models solve harder problems when they write out their reasoning step by step, an idea popularised by chain-of-thought prompting. In 2024 OpenAI's o1 showed that training a model to produce long reasoning before answering helps even more, but did not say how.

The usual way to teach a model to reason is to show it many human-written worked solutions. That is expensive, and it limits the model to reasoning the way people wrote it down. DeepSeek asked whether a model could discover good reasoning by itself if it were only rewarded for getting the answer right.

Everyday pictureTeaching chess by only saying "you won" or "you lost" at the end of each game, never explaining moves. It sounds hopeless for a beginner, but a player who already knows a lot can work out, over thousands of games, which habits lead to wins.

TakeawayThe paper tests whether reasoning can emerge from outcome rewards alone, without examples of how to reason.

Idea · §2.1–2.2 · Table 1

A reward a computer can check

Reinforcement learning needs a score for each attempt. Many systems use a second neural network, a "reward model", to score answers, but a model being trained hard will find tricks that fool it (called reward hacking). DeepSeek avoided this for reasoning tasks by using simple rules a computer can check exactly:

  • Accuracy reward: is the final answer correct? For maths the answer must be in a set format so it can be compared; for code, the program is run against test cases.
  • Format reward: is the reasoning placed between <think> and </think> tags?

The prompt template only says to think first and then answer; it says nothing about how to think. Training uses GRPO: for each problem, sample a group of answers (16 in the revised paper) and reward each one by how it compares with the group's average. Click the answers below.

Problem: "What is 7 × 8 + 4?" (correct answer 60). Click a response to inspect it.

A made-up group of six responses scored with the paper's two rules (accuracy and format, weighted equally, as in Eq. 4 of the revised paper). The right column is each response's GRPO advantage: its reward minus the group average, divided by the group's standard deviation. Correct, well-formatted answers get the biggest push. The rule never looks at the reasoning itself.

TakeawayR1-Zero's only teachers are two checkable rules: right final answer, reasoning inside the tags.

Check yourself
Why not use a neural network to judge whether the reasoning is good?
Because during long, intense training the model tends to find outputs that fool a learned judge without actually being better. A rule that checks the final answer is much harder to game.

Idea · §2.3 · Fig. 1(b), Table 2 (revised) / Fig. 3, Table 3 (v1)

Thinking longer, on its own

As training went on, R1-Zero's responses grew steadily longer, from hundreds to thousands of reasoning tokens. Nobody asked for length. Longer reasoning simply led to more correct answers, so it was rewarded. Along with length came new behaviours: checking intermediate results, noticing mistakes, and trying a different approach.

The authors highlight an "aha moment" in which an intermediate checkpoint, partway through a solution, writes "Wait, wait. Wait. That's an aha moment I can flag here." and goes back to re-evaluate its steps. They report a sharp rise in the use of the word "wait" during training.

The toy below shows why length can emerge. A "model" chooses how many checking steps to take before answering; each extra check gives another chance to catch an error. It is rewarded only for correct answers.

Illustrative toy, not the paper's experiment. The policy picks a number of checking steps from 0 to 12; an answer with k checks is right with probability 1 − (1 − q)k+1. Training uses GRPO with groups of 16 and a correct/incorrect reward only. Left: how likely each length is now. Right: average length (amber) and accuracy (green) over training. Harder problems (small q) drive the model to think longer.

TakeawayLong, self-correcting reasoning was not taught; it was the behaviour that earned reward, so it grew.

Check yourself
In the toy, why does nothing stop the length from growing?
There is no penalty for length, only a reward for correctness, and more checks never hurt. The real paper notes the same side effect: R1 can "overthink" easy questions.

Idea · §2.3–3 (revised) / §2.3 (v1)

From R1-Zero to R1: making it readable

R1-Zero reasoned well but wrote badly: its chains of thought were hard to follow and mixed English and Chinese in the same answer. It was also trained only on checkable problems, so it was not a good general assistant.

DeepSeek-R1 adds stages around the reinforcement learning:

  1. Cold start: fine-tune on thousands of long, readable example solutions, so reasoning starts out in a human-friendly style.
  2. Reasoning RL: the same rule-based reinforcement learning, plus a reward for keeping the reasoning in one language.
  3. Rejection sampling and fine-tuning: generate many answers, keep only good ones (about 600,000 reasoning examples), add about 200,000 examples of ordinary tasks like writing and factual questions, and fine-tune on them.
  4. Final RL: train on all kinds of prompts, using rule-based rewards for reasoning and learned reward models for helpfulness and harmlessness.

The result performed about on par with OpenAI's o1 (December 2024 version) on maths and coding benchmarks: for example 79.8% on AIME 2024 against o1's 79.2%, and 97.3% on the MATH-500 test against 96.4%.

Everyday pictureA self-taught prodigy who solves problems brilliantly but whose notes are illegible. You first show them a few tidy worked examples, let them keep practising, collect their best solutions, and finally polish their manners for general conversation.

TakeawayR1 keeps R1-Zero's reinforcement learning at its core, and adds a little supervised data before and after to make it readable and generally useful.

Idea · §2.4, §4.1 (v1) · Tables 5–6

Teaching small models by example

The team took about 800,000 of R1's reasoning and general examples and used them to fine-tune smaller open models from the Qwen and Llama families, with no reinforcement learning at all. This is called distillation: a big model's outputs become a small model's textbook.

The distilled 7-billion-parameter model scored 55.5% on AIME 2024. And in a direct comparison on a 32B model, distillation beat running large-scale reinforcement learning on the small model itself (72.6% vs 47.0% on AIME). The authors' conclusion: reasoning patterns found by a large model transfer well, and discovering them from scratch needs a strong base model.

Later refinementR1 was released openly in January 2025, with its weights, and sparked many replications of "R1-Zero-style" training. The paper was later published in Nature (2025) in revised form, which this guide cites as "revised". Follow-up studies questioned how much RL adds beyond what the base model can already do when sampled many times, refined GRPO's details (Dr. GRPO, DAPO), and worked on reducing "overthinking" on easy questions.

TakeawayA big model's discovered reasoning can be cheaply handed to small models, more effectively than letting the small models discover it themselves.

Follow the idea

Built on: GRPO (2024), Chain of thought (2022)