The paper in four sentences
The team took a large pretrained model and trained it with reinforcement learning on maths, coding and logic problems, rewarding it only when its final answer was correct and its reasoning was placed inside <think> tags.
Without ever being shown how to reason, this model, R1-Zero, learned to think for longer and longer, to re-check its work and to try other approaches, and its score on the AIME 2024 maths competition rose from 15.6% to 71.0% (77.9% in the revised version of the paper).
Its reasoning was hard to read and switched between languages, so the full DeepSeek-R1 adds a small set of readable example solutions first, then more rounds of reinforcement learning and fine-tuning, reaching performance comparable to OpenAI's o1 on reasoning tests.
Finally, R1's reasoning traces were used to train much smaller models, which beat the same small models trained with reinforcement learning directly.
Idea · §1
Can reasoning be rewarded into existence?
Large language models solve harder problems when they write out their reasoning step by step, an idea popularised by chain-of-thought prompting. In 2024 OpenAI's o1 showed that training a model to produce long reasoning before answering helps even more, but did not say how.
The usual way to teach a model to reason is to show it many human-written worked solutions. That is expensive, and it limits the model to reasoning the way people wrote it down. DeepSeek asked whether a model could discover good reasoning by itself if it were only rewarded for getting the answer right.
TakeawayThe paper tests whether reasoning can emerge from outcome rewards alone, without examples of how to reason.
Idea · §2.1–2.2 · Table 1
A reward a computer can check
Reinforcement learning needs a score for each attempt. Many systems use a second neural network, a "reward model", to score answers, but a model being trained hard will find tricks that fool it (called reward hacking). DeepSeek avoided this for reasoning tasks by using simple rules a computer can check exactly:
- Accuracy reward: is the final answer correct? For maths the answer must be in a set format so it can be compared; for code, the program is run against test cases.
- Format reward: is the reasoning placed between <think> and </think> tags?
The prompt template only says to think first and then answer; it says nothing about how to think. Training uses GRPO: for each problem, sample a group of answers (16 in the revised paper) and reward each one by how it compares with the group's average. Click the answers below.
TakeawayR1-Zero's only teachers are two checkable rules: right final answer, reasoning inside the tags.
Check yourself
- Why not use a neural network to judge whether the reasoning is good?
- Because during long, intense training the model tends to find outputs that fool a learned judge without actually being better. A rule that checks the final answer is much harder to game.
Idea · §2.3 · Fig. 1(b), Table 2 (revised) / Fig. 3, Table 3 (v1)
Thinking longer, on its own
As training went on, R1-Zero's responses grew steadily longer, from hundreds to thousands of reasoning tokens. Nobody asked for length. Longer reasoning simply led to more correct answers, so it was rewarded. Along with length came new behaviours: checking intermediate results, noticing mistakes, and trying a different approach.
The authors highlight an "aha moment" in which an intermediate checkpoint, partway through a solution, writes "Wait, wait. Wait. That's an aha moment I can flag here." and goes back to re-evaluate its steps. They report a sharp rise in the use of the word "wait" during training.
The toy below shows why length can emerge. A "model" chooses how many checking steps to take before answering; each extra check gives another chance to catch an error. It is rewarded only for correct answers.
TakeawayLong, self-correcting reasoning was not taught; it was the behaviour that earned reward, so it grew.
Check yourself
- In the toy, why does nothing stop the length from growing?
- There is no penalty for length, only a reward for correctness, and more checks never hurt. The real paper notes the same side effect: R1 can "overthink" easy questions.
Idea · §2.3–3 (revised) / §2.3 (v1)
From R1-Zero to R1: making it readable
R1-Zero reasoned well but wrote badly: its chains of thought were hard to follow and mixed English and Chinese in the same answer. It was also trained only on checkable problems, so it was not a good general assistant.
DeepSeek-R1 adds stages around the reinforcement learning:
- Cold start: fine-tune on thousands of long, readable example solutions, so reasoning starts out in a human-friendly style.
- Reasoning RL: the same rule-based reinforcement learning, plus a reward for keeping the reasoning in one language.
- Rejection sampling and fine-tuning: generate many answers, keep only good ones (about 600,000 reasoning examples), add about 200,000 examples of ordinary tasks like writing and factual questions, and fine-tune on them.
- Final RL: train on all kinds of prompts, using rule-based rewards for reasoning and learned reward models for helpfulness and harmlessness.
The result performed about on par with OpenAI's o1 (December 2024 version) on maths and coding benchmarks: for example 79.8% on AIME 2024 against o1's 79.2%, and 97.3% on the MATH-500 test against 96.4%.
TakeawayR1 keeps R1-Zero's reinforcement learning at its core, and adds a little supervised data before and after to make it readable and generally useful.
Idea · §2.4, §4.1 (v1) · Tables 5–6
Teaching small models by example
The team took about 800,000 of R1's reasoning and general examples and used them to fine-tune smaller open models from the Qwen and Llama families, with no reinforcement learning at all. This is called distillation: a big model's outputs become a small model's textbook.
The distilled 7-billion-parameter model scored 55.5% on AIME 2024. And in a direct comparison on a 32B model, distillation beat running large-scale reinforcement learning on the small model itself (72.6% vs 47.0% on AIME). The authors' conclusion: reasoning patterns found by a large model transfer well, and discovering them from scratch needs a strong base model.
TakeawayA big model's discovered reasoning can be cheaply handed to small models, more effectively than letting the small models discover it themselves.
Abstract, restated
General reasoning in LLMs has relied on human-annotated demonstrations. This work shows reasoning abilities can be incentivised through pure reinforcement learning, without human-labelled reasoning trajectories. The RL framework facilitates emergent reasoning patterns such as self-reflection, verification and dynamic strategy adaptation; the trained model achieves superior performance on verifiable tasks such as mathematics, coding competitions and STEM fields, surpassing counterparts trained by supervised learning on human demonstrations; and the emergent patterns can be used to guide and enhance smaller models. DeepSeek-AI, arXiv 2501.12948 (v1 January 2025; revised version corresponding to the Nature 2025 publication). Where the versions differ, both are given.
Section · §2.1–2.2 (revised) · Eqs. 1–4 · Table 1
R1-Zero: GRPO with rule-based rewards
Base model: DeepSeek-V3-Base (a 671B-parameter mixture-of-experts, 37B active). Optimiser: GRPO. For question q, sample {o1, …, oG} from πθold and maximise
Rewards. Accuracy: deterministic answer checking (boxed final answer for maths, compiler and test cases for code). Format: reasoning enclosed in <think>…</think>. No neural outcome or process reward model is used for reasoning, because "neural reward models are susceptible to reward hacking during large-scale reinforcement learning", and retraining them adds cost and complexity.
Template (Table 1). The system prompt describes a conversation in which the assistant first thinks about the reasoning process and then provides the answer, with the two enclosed in <think> and <answer> tags. The constraint is deliberately limited to structure, to avoid content-specific biases such as mandating reflection or particular strategies, so that the natural progression under RL can be observed.
# R1-Zero settings (revised paper, §2.1)
learning rate 3e-6, KL coefficient β = 0.001, rollout temperature 1
G = 16 samples per question, 32 questions per step (batch 512)
max length 32,768 tokens, raised to 65,536 after step 8.2k
10,400 steps (~1.6 epochs); π_ref reset to current policy every 400 steps
each rollout: 8,192 outputs split into 16 mini-batches, one inner epoch
Section · §2.3 · Fig. 1, Table 2 (revised); §2.2, Table 2, Fig. 2–3 (v1)
R1-Zero results and emergent behaviour
AIME 2024. Pass@1 rises from 15.6% at the start of RL to 71.0% at the end in v1, and to 77.9% in the revised paper's longer run; with majority voting it reaches 86.7% (cons@64 in v1; the revised paper labels it cons@16). In v1's Table 2, R1-Zero is compared with OpenAI's o1-0912: AIME pass@1 71.0 vs 74.4 (cons@64 86.7 vs 83.3), MATH-500 95.9 vs 94.8, GPQA Diamond 73.3 vs 77.3, LiveCodeBench 50.0 vs 63.4, Codeforces rating 1444 vs 1843.
Self-evolution. Average response length grows steadily through training (revised Fig. 1b, v1 Fig. 3), from hundreds to thousands of reasoning tokens, i.e. the model learns to allocate more test-time compute. Reflection and exploration of alternatives appear without being specified. The revised paper links the "aha moment" (Table 2) to a sudden increase in the use of "wait" during reflections (Supplementary Fig. 9b).
Drawbacks. Poor readability and language mixing (mainly English and Chinese in one chain of thought), and a narrow focus on verifiable tasks.
Check yourself
- Why does the format reward exist if the accuracy reward is what matters?
- It forces a separation between reasoning and answer, which makes answers extractable for checking and makes the emergent reasoning observable, without constraining its content.
Section · §3 (revised) · §2.3 (v1)
The DeepSeek-R1 pipeline
The language consistency reward is the proportion of target-language words in the chain of thought, added to the accuracy reward. The v1 ablation found it slightly degrades benchmark performance, a cost accepted for readability. In the revised paper, the final RL stage runs 1,700 steps, with general instruction and preference-based rewards used only in the last 400 steps, a guard against reward hacking of learned reward models.
Section · §3 Tables 4–6 (v1); Table 3 (revised)
Results and distillation
R1 is weaker than o1-1217 on GPQA Diamond (71.5 vs 75.7) and slightly on MMLU (90.8 vs 91.8) and Codeforces percentile (96.3 vs 96.6). Distillation from a stronger model beats RL on the small model by a wide margin (AIME 72.6 vs 47.0 at 32B), and the authors note that pushing beyond this may still require more powerful base models and larger-scale RL.
Unsuccessful attempts (v1 §4.2). Process reward models: hard to define fine-grained steps in general reasoning, hard to judge intermediate correctness, model-based annotation invites reward hacking, and retraining adds cost; useful for reranking but not worth it for large-scale RL here. Monte Carlo Tree Search: the token-level search space is vastly larger than in games, per-node extension limits lead to local optima, and training a fine-grained value model to guide search is hard, so iterative self-improvement via search did not work out.
Section · §6 (revised), §5 (v1) and later work
Limitations and lineage
- Structured output and tool use lag behind existing models.
- Token efficiency. Overthinking on simple questions.
- Language mixing for languages other than Chinese and English.
- Prompt sensitivity. Few-shot prompting degrades performance; zero-shot with a clear format is recommended.
- Software engineering. Limited gains over DeepSeek-V3, because long evaluation times make large-scale RL on such tasks expensive.
- Reward hacking remains the obstacle for tasks without reliable verifiers, such as writing.
Check yourself
- What does the distillation result imply about where R1's gains come from?
- That the reasoning patterns themselves, once discovered by a large model, are transferable data; small models struggle to discover them by RL, so base-model capacity matters for discovery, not for use.
Reference
Glossary
- R1-Zero
- DeepSeek-V3-Base trained with RL only, no SFT.
- Cold start
- Small SFT set of long, readable CoT examples before RL.
- Rule-based reward
- Programmatic correctness and format checks.
- Pass@1
- Accuracy of a single sample (estimated by averaging several samples).
- cons@64
- Accuracy of the majority answer over 64 samples.
- Rejection sampling
- Generating many outputs and keeping only those that pass a check.
- Distillation
- Training a smaller model on a larger model's outputs.
- PRM
- Process reward model, scoring intermediate steps.
- MCTS
- Monte Carlo Tree Search over partial solutions.