Study guide · Machine learning & training · 2024

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Zhihong Shao et al. · arXiv

DeepSeekMath asks how far an open 7-billion-parameter model can get at competition mathematics. The answer combines a 120-billion-token maths corpus mined from the web with a new, leaner reinforcement-learning method, Group Relative Policy Optimization (GRPO), which scores each answer against the other answers to the same question instead of training a separate network to predict how good an answer should be.

The paper in four sentences

The team filtered 120 billion tokens of mathematical text out of a huge web crawl and used it to keep training a 7-billion-parameter model; after fine-tuning and reinforcement learning it reached 51.7% on a hard competition-maths test without any calculator or code tools.

To improve it further with reinforcement learning, they replaced the usual PPO method's second "critic" network, which tries to predict how good each answer will be, with something much simpler.

GRPO asks the model the same question many times (64 in the paper), scores every answer, and rewards each one by how much better or worse it did than the group's average: grading on a curve.

This saves the memory and effort of a whole extra model, lifted the scores further, and became the training method behind DeepSeek-R1 a year later.

Idea · §2

First, feed it a lot of maths

Most of the paper is about data. The authors started from OpenWebMath, a public collection of mathematical web pages, and trained a simple text classifier to recognise "pages that look like maths". They ran it over Common Crawl, a giant public snapshot of the web, kept the best-scoring pages, had people help label which website sections were mathematical, retrained the classifier and repeated. After four rounds they had 35.5 million web pages totalling 120 billion tokens (word pieces).

They also removed anything that matched the test questions, so the model could not simply memorise answers. A code-trained 7B model, further trained on this corpus plus some code and ordinary text, became DeepSeekMath-Base. After supervised fine-tuning on worked solutions it became DeepSeekMath-Instruct, and after reinforcement learning, DeepSeekMath-RL.

Everyday pictureBefore coaching a student for a maths olympiad, you give them a library. The library here is the web's maths pages, found by a sniffer dog that gets better at finding them each round.

TakeawayCareful data selection, not a bigger model, did most of the work; reinforcement learning was the finishing step.

Idea · §4.1.1

Learning from right and wrong answers, and the expensive critic

In reinforcement learning (RL) the model tries something, gets a score (a reward), and is nudged to do more of what scored well. For maths, the model writes a solution and a reward model judges it.

A raw score alone is a poor teacher. If almost every answer scores well, "you scored 0.8" tells you little; what matters is whether this answer was better than expected. That difference is called the advantage. The popular method PPO, used in InstructGPT, computes it with a second neural network, the critic or value model, trained alongside the main one to predict the expected score at every point in the answer.

The critic is usually as big as the model being trained. That means another few billion numbers in GPU memory, another network to train, and a tricky job: predicting the final score from a half-written solution when the reward only arrives at the very end.

TakeawayReinforcement learning needs a baseline, "what score did we expect?", and PPO pays for it with a whole second model.

Idea · §4.1.2 · Eq. 3

Grade on a curve instead

GRPO's idea is to get the baseline from the model's own attempts. For each question, sample a group of answers. Score them all. The expected score is just the group's average; each answer's advantage is how far above or below the average it landed, divided by how spread out the scores were (the standard deviation), so that every question's lesson has a similar size.

Answers above average get reinforced, those below get discouraged. No critic is needed. Try it below with a right-or-wrong (1 or 0) score.

Each column is one sampled answer: tick = correct (reward 1), cross = wrong (reward 0). Bars show its advantage, (reward − group average) ÷ standard deviation: up (amber) reinforces, down (blue) discourages. Set the chance to 0 or 1, or use a small group on an easy question: when every answer gets the same score, all advantages are zero and the group teaches nothing.
Everyday pictureA teacher who grades on a curve doesn't need to know in advance how hard the test was. The class's own average tells her: whoever beat it did well.

TakeawayGRPO replaces the critic with the group: the average score of several attempts at the same question is the baseline.

Check yourself
A question is so easy that all 64 answers are correct. What does GRPO learn from it?
Nothing: every advantage is zero. Questions that are always solved or never solved give no signal, which is why the mix of difficulty in the training set matters.
Why does a rare correct answer get a large positive advantage?
Because the average is low and the spread small, so being right is far above expectation. The method pushes hard on surprising successes.

Idea · §4.1 · Algorithm 1

Watch GRPO train

Here is a tiny stand-in for a language model. For one type of question it can choose between four solution strategies, each with a different (hidden) chance of reaching the right answer. Every step, it samples a group of attempts, scores each 1 or 0, computes the group-relative advantages, and nudges its preferences. Watch it discover the best strategy without ever being told which one that is.

Left: how often the model picks each strategy (hidden success rates underneath). Right: expected accuracy over training. "Wasted groups" counts groups where all answers scored the same, so all advantages were zero. With group size 2, many groups are wasted and learning is slower and noisier. An illustration of the update rule, not the paper's experiment.

TakeawayComparing attempts with each other is enough signal to move a model towards strategies that reach correct answers.

Idea · §4.2, §5.2 · Table 5, Fig. 7

What reinforcement learning actually changed

GRPO, trained only on questions related to two maths test sets (GSM8K and MATH), raised the Instruct model from 82.9% to 88.2% on GSM8K and from 46.8% to 51.7% on MATH. It also improved on tests it was not trained for, such as a Chinese maths benchmark.

The authors then asked a sharp question. If you let the model try K times, does RL raise the chance that at least one try is right (called Pass@K)? Mostly not. What it raised was the chance that the most common answer is right (Maj@K), and the chance a single try is right. Their reading: RL made the model more reliable at producing answers it could already sometimes produce, rather than teaching fundamentally new abilities.

Later refinementA year later DeepSeek-R1 used GRPO with simple rule-based rewards (is the final answer right?) at much larger scale, and long, self-checking reasoning emerged. Others then refined GRPO itself: "Dr. GRPO" (2025) argued that dividing by the group's spread and by answer length introduces biases and removed them; DAPO (2025) changed clipping and sampling to keep training stable. The debate over whether RL creates new abilities or sharpens existing ones, raised here, continues.

TakeawayGRPO improved accuracy cheaply, but mainly by making the model's existing good answers more likely, a finding later work keeps revisiting.

Check yourself
If RL raises Maj@K but not Pass@K, what has changed in the model?
The correct answer was already somewhere among its possible outputs; RL made it more probable, so it wins the vote more often, without adding new correct answers it could never find before.

Follow the idea

Built on: InstructGPT (RLHF) (2022), PPO (2017)

Led to: DeepSeek-R1 (2025)