The paper in four sentences
The team filtered 120 billion tokens of mathematical text out of a huge web crawl and used it to keep training a 7-billion-parameter model; after fine-tuning and reinforcement learning it reached 51.7% on a hard competition-maths test without any calculator or code tools.
To improve it further with reinforcement learning, they replaced the usual PPO method's second "critic" network, which tries to predict how good each answer will be, with something much simpler.
GRPO asks the model the same question many times (64 in the paper), scores every answer, and rewards each one by how much better or worse it did than the group's average: grading on a curve.
This saves the memory and effort of a whole extra model, lifted the scores further, and became the training method behind DeepSeek-R1 a year later.
Idea · §2
First, feed it a lot of maths
Most of the paper is about data. The authors started from OpenWebMath, a public collection of mathematical web pages, and trained a simple text classifier to recognise "pages that look like maths". They ran it over Common Crawl, a giant public snapshot of the web, kept the best-scoring pages, had people help label which website sections were mathematical, retrained the classifier and repeated. After four rounds they had 35.5 million web pages totalling 120 billion tokens (word pieces).
They also removed anything that matched the test questions, so the model could not simply memorise answers. A code-trained 7B model, further trained on this corpus plus some code and ordinary text, became DeepSeekMath-Base. After supervised fine-tuning on worked solutions it became DeepSeekMath-Instruct, and after reinforcement learning, DeepSeekMath-RL.
TakeawayCareful data selection, not a bigger model, did most of the work; reinforcement learning was the finishing step.
Idea · §4.1.1
Learning from right and wrong answers, and the expensive critic
In reinforcement learning (RL) the model tries something, gets a score (a reward), and is nudged to do more of what scored well. For maths, the model writes a solution and a reward model judges it.
A raw score alone is a poor teacher. If almost every answer scores well, "you scored 0.8" tells you little; what matters is whether this answer was better than expected. That difference is called the advantage. The popular method PPO, used in InstructGPT, computes it with a second neural network, the critic or value model, trained alongside the main one to predict the expected score at every point in the answer.
The critic is usually as big as the model being trained. That means another few billion numbers in GPU memory, another network to train, and a tricky job: predicting the final score from a half-written solution when the reward only arrives at the very end.
TakeawayReinforcement learning needs a baseline, "what score did we expect?", and PPO pays for it with a whole second model.
Idea · §4.1.2 · Eq. 3
Grade on a curve instead
GRPO's idea is to get the baseline from the model's own attempts. For each question, sample a group of answers. Score them all. The expected score is just the group's average; each answer's advantage is how far above or below the average it landed, divided by how spread out the scores were (the standard deviation), so that every question's lesson has a similar size.
Answers above average get reinforced, those below get discouraged. No critic is needed. Try it below with a right-or-wrong (1 or 0) score.
TakeawayGRPO replaces the critic with the group: the average score of several attempts at the same question is the baseline.
Check yourself
- A question is so easy that all 64 answers are correct. What does GRPO learn from it?
- Nothing: every advantage is zero. Questions that are always solved or never solved give no signal, which is why the mix of difficulty in the training set matters.
- Why does a rare correct answer get a large positive advantage?
- Because the average is low and the spread small, so being right is far above expectation. The method pushes hard on surprising successes.
Idea · §4.1 · Algorithm 1
Watch GRPO train
Here is a tiny stand-in for a language model. For one type of question it can choose between four solution strategies, each with a different (hidden) chance of reaching the right answer. Every step, it samples a group of attempts, scores each 1 or 0, computes the group-relative advantages, and nudges its preferences. Watch it discover the best strategy without ever being told which one that is.
TakeawayComparing attempts with each other is enough signal to move a model towards strategies that reach correct answers.
Idea · §4.2, §5.2 · Table 5, Fig. 7
What reinforcement learning actually changed
GRPO, trained only on questions related to two maths test sets (GSM8K and MATH), raised the Instruct model from 82.9% to 88.2% on GSM8K and from 46.8% to 51.7% on MATH. It also improved on tests it was not trained for, such as a Chinese maths benchmark.
The authors then asked a sharp question. If you let the model try K times, does RL raise the chance that at least one try is right (called Pass@K)? Mostly not. What it raised was the chance that the most common answer is right (Maj@K), and the chance a single try is right. Their reading: RL made the model more reliable at producing answers it could already sometimes produce, rather than teaching fundamentally new abilities.
TakeawayGRPO improved accuracy cheaply, but mainly by making the model's existing good answers more likely, a finding later work keeps revisiting.
Check yourself
- If RL raises Maj@K but not Pass@K, what has changed in the model?
- The correct answer was already somewhere among its possible outputs; RL made it more probable, so it wins the vote more often, without adding new correct answers it could never find before.
Abstract, restated
DeepSeekMath 7B continues pre-training DeepSeek-Coder-Base-v1.5 7B on 120B math-related tokens from Common Crawl, together with natural language and code. It scores 51.7% on competition-level MATH without external toolkits or voting, approaching Gemini-Ultra and GPT-4, and 60.9% with self-consistency over 64 samples. Two factors are credited: a carefully engineered data-selection pipeline over public web data, and Group Relative Policy Optimization (GRPO), a PPO variant that improves mathematical reasoning while reducing PPO's memory usage. Shao, Wang, Zhu, Xu, Song, Bi, Zhang, Zhang, Li, Wu, Guo, arXiv 2402.03300 (2024).
Section · §2–3
Corpus, base and instruct models
- Data collection (§2.1). A fastText classifier is trained on 500,000 OpenWebMath positives and 500,000 Common Crawl negatives and applied to a deduplicated Common Crawl of 40B HTML pages. Domains with over 10% of pages recalled are marked math-related; annotators then label math URL paths, uncollected pages from those paths are added as positives, and the classifier is retrained. After four iterations (by iteration 4, about 98% of the data had already been collected in iteration 3) the DeepSeekMath Corpus holds 35.5M pages, 120B tokens.
- Decontamination. Text segments with exact 10-gram matches to benchmarks (GSM8K, MATH, CMATH, AGIEval) are removed; shorter benchmark texts use exact matching down to 3-grams.
- Base model (§2.3). Initialised from DeepSeek-Coder-Base-v1.5 7B; the authors find (§5.1) that code pre-training first helps mathematical reasoning, while arXiv papers, perhaps surprisingly, gave no clear improvement on the benchmarks they used.
- SFT (§3). Chain-of-thought, program-of-thought and tool-integrated solutions in English and Chinese, giving DeepSeekMath-Instruct 7B.
Section · §4.1.1 · Eqs. 1–4 · Fig. 4
From PPO to GRPO
PPO maximises the clipped surrogate
with At from Generalised Advantage Estimation (GAE) over rewards and a learned value function Vψ, and a per-token KL penalty folded into the reward:
The value model is typically as large as the policy, and in LLM RL only the last token receives a reward, which makes a per-token value function hard to train. GRPO drops it. For each question q, sample G outputs {o1, …, oG} from πθold and maximise
Two differences besides the advantage: the KL term is added directly to the loss rather than to the reward, so it does not complicate Â; and it uses the unbiased, always non-negative estimator
The clipped surrogate is what keeps each update "proximal". For one token, the objective as a function of the ratio ρ:
Check yourself
- Why is Eq. 4 always non-negative?
- For x = πref/πθ > 0, x − log x − 1 ≥ 0 with equality only at x = 1 (since log x ≤ x − 1). Its expectation under πθ equals the true KL, so it is unbiased.
Section · §4.1.2–4.1.4 · Algorithm 1
Outcome and process supervision, iterative RL
Outcome supervision. A reward model scores each complete output, r = {r1, …, rG}. Every token of output i gets the same normalised advantage:
Process supervision. A process reward model scores the end of each reasoning step. All step rewards across the group are normalised by their mean and standard deviation, and a token's advantage is the sum of normalised rewards of the steps ending at or after it: Âi,t = Σindex(j) ≥ t r̃iindex(j).
Iterative RL. As the policy improves, the old reward model may no longer supervise it well. Algorithm 1 therefore alternates: periodically generate new training data for the reward model from the current policy's samples, continue training the reward model with a replay mechanism that keeps 10% of historical data, and reset the reference model to the current policy.
Algorithm 1 (paraphrased)
for iteration = 1..I:
π_ref ← π_θ
for step = 1..M:
sample batch D_b of questions; π_θold ← π_θ
for each q in D_b: sample G outputs {o_i} ~ π_θold(·|q)
score each o_i with reward model r_φ; compute Â_{i,t}
for GRPO iteration = 1..μ: update π_θ by maximising Eq. 3
update r_φ by continued training on new samples (replay 10%)
Settings (§4.2). Training questions: about 144K chain-of-thought questions related to GSM8K and MATH from the SFT data (deliberately narrow, to test generalisation). Policy learning rate 1e-6, KL coefficient β = 0.04, G = 64 outputs per question, max length 1,024, batch size 1,024, and the policy is updated only once after each exploration stage. (A single update means each batch is used on-policy, so in this setting the ratio starts at 1 and the clip plays little role; this is our reading, not a statement in the paper.) The reward model was initialised from DeepSeekMath-Base 7B with learning rate 2e-5.
Section · §4.2 · Table 5
Results
| Benchmark | Type | Instruct 7B | RL 7B |
|---|---|---|---|
| GSM8K (CoT) | in-domain | 82.9 | 88.2 |
| MATH (CoT) | in-domain | 46.8 | 51.7 |
| MGSM-zh (CoT) | out-of-domain | 73.2 | 79.6 |
| CMATH (CoT) | out-of-domain | 84.6 | 88.8 |
| GSM8K (tool-integrated) | in-domain | 83.7 | 86.7 |
| MATH (tool-integrated) | in-domain | 57.4 | 58.8 |
RL used only chain-of-thought-format GSM8K/MATH questions, yet improved tool-integrated and Chinese benchmarks too. With self-consistency over 64 samples, the RL model reaches 60.9% on MATH (abstract).
Section · §5.2 · Eq. 5 · Table 10 · Fig. 7
A unified view, and why RL works
§5.2.1 writes SFT, rejection-sampling fine-tuning (RFT), DPO, online RFT, PPO and GRPO as the same gradient form with different data sources, reward functions and per-token gradient coefficients:
| Method | Data source | Reward |
|---|---|---|
| SFT | human-curated (q, o) | none (coefficient 1) |
| RFT | SFT model samples, offline | rule |
| DPO | SFT model pairs, offline | rule |
| Online RFT | current policy samples | rule |
| PPO | current policy samples | model |
| GRPO | groups from current policy | model |
Experiments along these axes find that online data beats offline, and that GRPO's reward-scaled coefficients (which can penalise wrong answers by different amounts) beat online RFT's binary keep/discard.
Why RL works (§5.2.2, Fig. 7). RL improves Maj@K but not Pass@K. The authors conclude that RL "enhances the model's overall performance by rendering the output distribution more robust": it boosts correct responses already in the model's top-K rather than enhancing fundamental capabilities. The toy below reproduces the effect with exactly that mechanism.
Section · §5.2.3, §6 and later work
Limitations and lineage
- Scope. A 7B model on maths; geometry and theorem proving are weaker than closed models, and the authors attribute some limits to model scale and data.
- Reward model. Results depend on a learned reward model whose reliability and generalisation to harder questions §5.2.3 names as a key direction, along with better exploration (e.g. tree search) and data sources.
- No headline memory number. The memory and compute savings of dropping the critic are argued qualitatively; no measured figure is given.
- Narrow gains. Fig. 7 suggests RL mainly sharpens rather than extends capability.
Check yourself
- Why does per-output length normalisation bias GRPO?
- Each output's token losses are averaged over its own length, so a long wrong answer's per-token penalty is diluted while a short correct answer's per-token reward is concentrated. Over training this can favour longer incorrect outputs.
Reference
Glossary
- Advantage
- How much better an action or output did than a baseline expectation.
- Value model / critic
- Network predicting expected future reward; PPO's baseline.
- GAE
- Generalised Advantage Estimation: exponentially weighted TD errors from a value function.
- Group G
- Number of outputs sampled per question (64 in the paper).
- πref
- Reference policy for the KL penalty (initially the SFT model).
- ε
- Clip range of the probability ratio.
- Pass@K
- Probability that at least one of K samples is correct.
- Maj@K
- Accuracy of the majority-vote answer over K samples (self-consistency).
- ORM / PRM
- Outcome / process reward model: scores whole answers or individual steps.