Study guide · Machine learning & training · 2023

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafael Rafailov et al. · NeurIPS 2023

The standard way to make a language model follow human preferences (RLHF) trains a separate "reward model" and then runs a finicky reinforcement-learning loop against it. Rafailov and colleagues show that the reward model can be written in terms of the language model itself, so the whole pipeline collapses into one simple classification-style loss on pairs of preferred and rejected answers.

The paper in four sentences

To teach a chatbot what people like, labs collect pairs of answers where a person said "this one is better", train a reward model to score answers, then use reinforcement learning to push the chatbot towards high scores without drifting too far from where it started.

The authors notice that this goal has an exact mathematical solution: the best chatbot is the starting chatbot with each answer's probability boosted according to its reward.

Turning that solution around, any chatbot implicitly defines a reward: how much more likely it makes an answer than the starting model did. So you can train the chatbot directly on the preference pairs, with no separate reward model and no reinforcement-learning loop.

This "Direct Preference Optimization" is a single, stable loss, and it matched or beat the reinforcement-learning method (PPO) on sentiment control, summarisation and dialogue in models up to 6 billion parameters.

Idea · §1, §3

The old recipe: three stages and a referee

A freshly trained language model is good at continuing text but not necessarily at being helpful. The recipe made famous by InstructGPT, called RLHF (reinforcement learning from human feedback), fixes this in three stages:

  1. Supervised fine-tuning: show the model good example answers.
  2. Reward model: show people two answers to the same question, ask which is better, and train a second network to predict those judgements as a score.
  3. Reinforcement learning: let the model write answers, have the reward model score them, and nudge the model towards higher scores, with a penalty for drifting too far from the stage-1 model.

Stage 3 is the troublesome one. It uses an algorithm called PPO, needs several copies of large networks in memory at once, requires generating text during training, and is notoriously sensitive to its settings.

Everyday pictureA cook learning to please diners. RLHF hires a food critic who has studied the diners' comment cards, then has the cook make dish after dish for the critic to score. DPO skips the critic and has the cook study the comment cards directly.

TakeawayRLHF works, but its reinforcement-learning stage is complex, expensive and unstable. DPO's goal is to get the same result without it.

Idea · §3 · Eq. 1

From "which is better?" to a score

People don't give answers a score out of ten; they pick the better of two. To connect choices to scores, the paper uses a classic model from 1952 called Bradley–Terry, also used to rank chess players: each answer has a hidden score, and the bigger the gap between two scores, the more reliably people pick the higher one. Equal scores mean a coin flip.

A reward model is trained to find scores that make the observed choices as likely as possible. DPO keeps exactly this assumption; it just finds the scores in a different place.

The S-shaped curve is the Bradley–Terry rule: chance that A is preferred = 1 / (1 + e−(score A − score B)). Each "Ask 20 people" draws 20 random judgements with that chance. Note that only the difference in scores matters: adding the same amount to both changes nothing.

TakeawayPreferences are modelled as noisy comparisons of hidden scores; only score differences matter.

Check yourself
A has score 2 and B has score 2. How often is A preferred?
Half the time. Only the difference counts, and it is zero.
Why does "only the difference matters" turn out to be useful later?
Because it means a reward can be shifted by any amount that depends only on the question without changing any predictions. DPO uses this freedom to cancel an otherwise impossible-to-compute term.

Idea · §4 · Eqs. 4–5

Your language model is secretly a reward model

RLHF's stage 3 asks for a model that earns high reward but stays close to the starting model (called the reference model). The authors point out that this goal has a known exact answer: take the reference model's probability for each possible answer and multiply it by a boost that grows with the answer's reward. A setting called β (beta) controls how strong the boost is: small β means "chase reward hard", large β means "stay close to the reference".

That exact answer can't be used directly, because it would require summing over every possible piece of text. But it can be turned around. If the best model is "reference times boost", then the boost, and hence the reward, can be read off any model by comparing it with the reference: an answer's reward is β times how much more likely the model makes it than the reference did (on a log scale).

So instead of training a reward model and then a chatbot, you write the Bradley–Terry training rule in terms of this implicit reward and train the chatbot directly. The impossible sum appears equally for both answers in a pair, and because only differences matter, it cancels.

Everyday pictureYou can't see a student's private marking scheme, but you can infer it from how their answers changed since the start of term: whatever they now say much more often than before must be what they think scores well.

TakeawayEvery language model defines a reward, "how much more likely than the reference", so fitting the model to preferences directly is the same as fitting a reward model and then optimising it.

Check yourself
What happens to the implicit reward if the model is identical to the reference?
It is zero for every answer: the model hasn't boosted anything.

Idea · §4 · Eq. 7

Training on pairs, live

The DPO rule for one pair is simple. Make the preferred answer more likely and the rejected answer less likely, relative to the reference. Push hardest on pairs the model currently gets wrong (where it rates the rejected answer higher), and barely at all on pairs it already gets right.

Below, a tiny "model" chooses between five possible replies to one question. Hidden human scores generate the preferences (through the Bradley–Terry rule). Train it with DPO and watch two things: the model's probabilities shift, and its implicit rewards line up with the hidden scores it never saw directly.

Question: "My code throws an error, can you help?" Grey bars: the reference model's probability for each reply. Amber bars: the model being trained. Right column: the implicit reward β·log(model ÷ reference), shifted so the top reply matches, next to the hidden score. Training uses the exact expected DPO gradient over all pairs. Try β = 0.1 (the model piles almost everything onto the best reply) and β = 2 (it stays near the reference). Either way the implicit rewards converge to the hidden scores. A toy, but the maths is the paper's.

TakeawayDPO is ordinary supervised training on preference pairs; β sets how far the model may move from the reference to satisfy them.

Check yourself
Why does a smaller β let the model move further from the reference?
The implicit reward is β times the log-ratio. To express the same reward differences with a smaller β, the log-ratios, and so the changes in probability, must be larger.

Idea · §6–7 · Fig. 2–3, Table 2

Did it work, and what came next

The authors tested three tasks: making movie-review continuations positive, summarising Reddit posts, and answering single-turn chat questions. On summaries, judged by GPT-4 against human-written reference summaries, DPO won about 61% of the time against PPO's best of 57%, and it was much less sensitive to the sampling temperature. In a check with human raters, DPO's summaries were preferred over PPO's 58% of the time.

All experiments used models of at most 6 billion parameters. Even so, DPO spread quickly because it is so easy to run: it needs no sampling during training and no reward model.

Later refinementDPO became a standard step in open models (for example Zephyr and Llama 3 used it alongside supervised fine-tuning). Follow-ups adjusted the loss: IPO (2023) guards against overfitting when preferences are near-deterministic, KTO (2024) learns from single thumbs-up or thumbs-down labels instead of pairs, and SimPO (2024) drops the reference model. For reasoning tasks with checkable answers, labs returned to online reinforcement learning, as in GRPO and DeepSeek-R1.

TakeawayDPO matched or beat PPO-based RLHF on the paper's tasks with a far simpler method, and became a default tool for preference tuning.

Follow the idea

Built on: InstructGPT (RLHF) (2022)