The paper in four sentences
To teach a chatbot what people like, labs collect pairs of answers where a person said "this one is better", train a reward model to score answers, then use reinforcement learning to push the chatbot towards high scores without drifting too far from where it started.
The authors notice that this goal has an exact mathematical solution: the best chatbot is the starting chatbot with each answer's probability boosted according to its reward.
Turning that solution around, any chatbot implicitly defines a reward: how much more likely it makes an answer than the starting model did. So you can train the chatbot directly on the preference pairs, with no separate reward model and no reinforcement-learning loop.
This "Direct Preference Optimization" is a single, stable loss, and it matched or beat the reinforcement-learning method (PPO) on sentiment control, summarisation and dialogue in models up to 6 billion parameters.
Idea · §1, §3
The old recipe: three stages and a referee
A freshly trained language model is good at continuing text but not necessarily at being helpful. The recipe made famous by InstructGPT, called RLHF (reinforcement learning from human feedback), fixes this in three stages:
- Supervised fine-tuning: show the model good example answers.
- Reward model: show people two answers to the same question, ask which is better, and train a second network to predict those judgements as a score.
- Reinforcement learning: let the model write answers, have the reward model score them, and nudge the model towards higher scores, with a penalty for drifting too far from the stage-1 model.
Stage 3 is the troublesome one. It uses an algorithm called PPO, needs several copies of large networks in memory at once, requires generating text during training, and is notoriously sensitive to its settings.
TakeawayRLHF works, but its reinforcement-learning stage is complex, expensive and unstable. DPO's goal is to get the same result without it.
Idea · §3 · Eq. 1
From "which is better?" to a score
People don't give answers a score out of ten; they pick the better of two. To connect choices to scores, the paper uses a classic model from 1952 called Bradley–Terry, also used to rank chess players: each answer has a hidden score, and the bigger the gap between two scores, the more reliably people pick the higher one. Equal scores mean a coin flip.
A reward model is trained to find scores that make the observed choices as likely as possible. DPO keeps exactly this assumption; it just finds the scores in a different place.
TakeawayPreferences are modelled as noisy comparisons of hidden scores; only score differences matter.
Check yourself
- A has score 2 and B has score 2. How often is A preferred?
- Half the time. Only the difference counts, and it is zero.
- Why does "only the difference matters" turn out to be useful later?
- Because it means a reward can be shifted by any amount that depends only on the question without changing any predictions. DPO uses this freedom to cancel an otherwise impossible-to-compute term.
Idea · §4 · Eqs. 4–5
Your language model is secretly a reward model
RLHF's stage 3 asks for a model that earns high reward but stays close to the starting model (called the reference model). The authors point out that this goal has a known exact answer: take the reference model's probability for each possible answer and multiply it by a boost that grows with the answer's reward. A setting called β (beta) controls how strong the boost is: small β means "chase reward hard", large β means "stay close to the reference".
That exact answer can't be used directly, because it would require summing over every possible piece of text. But it can be turned around. If the best model is "reference times boost", then the boost, and hence the reward, can be read off any model by comparing it with the reference: an answer's reward is β times how much more likely the model makes it than the reference did (on a log scale).
So instead of training a reward model and then a chatbot, you write the Bradley–Terry training rule in terms of this implicit reward and train the chatbot directly. The impossible sum appears equally for both answers in a pair, and because only differences matter, it cancels.
TakeawayEvery language model defines a reward, "how much more likely than the reference", so fitting the model to preferences directly is the same as fitting a reward model and then optimising it.
Check yourself
- What happens to the implicit reward if the model is identical to the reference?
- It is zero for every answer: the model hasn't boosted anything.
Idea · §4 · Eq. 7
Training on pairs, live
The DPO rule for one pair is simple. Make the preferred answer more likely and the rejected answer less likely, relative to the reference. Push hardest on pairs the model currently gets wrong (where it rates the rejected answer higher), and barely at all on pairs it already gets right.
Below, a tiny "model" chooses between five possible replies to one question. Hidden human scores generate the preferences (through the Bradley–Terry rule). Train it with DPO and watch two things: the model's probabilities shift, and its implicit rewards line up with the hidden scores it never saw directly.
TakeawayDPO is ordinary supervised training on preference pairs; β sets how far the model may move from the reference to satisfy them.
Check yourself
- Why does a smaller β let the model move further from the reference?
- The implicit reward is β times the log-ratio. To express the same reward differences with a smaller β, the log-ratios, and so the changes in probability, must be larger.
Idea · §6–7 · Fig. 2–3, Table 2
Did it work, and what came next
The authors tested three tasks: making movie-review continuations positive, summarising Reddit posts, and answering single-turn chat questions. On summaries, judged by GPT-4 against human-written reference summaries, DPO won about 61% of the time against PPO's best of 57%, and it was much less sensitive to the sampling temperature. In a check with human raters, DPO's summaries were preferred over PPO's 58% of the time.
All experiments used models of at most 6 billion parameters. Even so, DPO spread quickly because it is so easy to run: it needs no sampling during training and no reward model.
TakeawayDPO matched or beat PPO-based RLHF on the paper's tasks with a far simpler method, and became a default tool for preference tuning.
Abstract, restated
Large unsupervised language models are hard to steer precisely. RLHF fits a reward model to human preference labels and then fine-tunes the model with reinforcement learning to maximise that reward under a KL constraint to the original model, a complex and often unstable procedure. DPO introduces a reparameterisation of the reward model that allows the corresponding optimal policy to be extracted in closed form, so the standard RLHF problem can be solved with only a simple classification loss. DPO is stable, performant and computationally lightweight, needing no sampling from the policy during fine-tuning and little hyperparameter tuning. It controls sentiment better than PPO-based RLHF and matches or improves response quality in summarisation and single-turn dialogue. Rafailov, Sharma, Mitchell, Ermon, Manning, Finn, NeurIPS 2023 (arXiv 2305.18290).
Section · §3 · Eqs. 1–3
The RLHF problem DPO solves
Starting from an SFT model πSFT, prompts x yield answer pairs (y1, y2) labelled yw ≻ yl. Preferences are assumed generated by a latent reward r* through the Bradley–Terry model:
A reward model rφ is fit by maximum likelihood, a binary classification:
and the policy is then optimised under a KL constraint to a reference πref (normally πSFT):
Because generation is discrete, Eq. 3 is not differentiable through sampling; it is usually optimised with PPO on the shaped reward rφ − β(log πθ − log πref).
Section · §4 · Eqs. 4–7 · Appendix A.1
The derivation
For any reward r, the KL-constrained objective (3) has the closed-form optimum (derived in Appendix A.1 following prior work on KL-regularised RL):
Z(x) sums over all sequences and is intractable. Taking logs and rearranging expresses the reward through its optimal policy:
Substitute (5) into Bradley–Terry (1). Only reward differences between two completions of the same x appear, so β log Z(x) cancels, and the preference probability depends only on the policy and the reference. Maximum likelihood over a parametrised πθ gives the DPO objective:
The figure applies Eq. 4 to a six-answer toy problem. The reward-versus-KL trade-off that PPO has to search for by sampling is, here, a closed-form curve traced by β.
Check yourself
- Why does DPO need pairs rather than single labelled answers?
- The β log Z(x) term only cancels in a difference of rewards for the same prompt. A single answer's reward would still contain the intractable partition function.
Section · §4 "What does the DPO update do?"
The gradient and its weighting
With the implicit reward r̂θ(x, y) = β log (πθ(y|x) / πref(y|x)):
The update raises the log-likelihood of yw and lowers that of yl, each example weighted by how wrongly the implicit reward model currently orders the pair, scaled by β. The paper stresses that this weighting matters: a naive version that just raises yw and lowers yl with constant weight (unlikelihood training) can cause the language model to degenerate (Appendix Table 3).
Check yourself
- A pair already has margin +5. How much does it contribute to the gradient?
- Its weight is σ(−5) ≈ 0.0067, so almost nothing; the model concentrates on misordered pairs.
Section · §5 · Theorem 1, Lemmas 1–2
Theory: no loss of generality, and why PPO is unstable
Equivalence classes (Definition 1). Rewards r and r′ are equivalent iff r(x, y) − r′(x, y) = f(x). Lemma 1: equivalent rewards induce the same preference distribution under Plackett–Luce (and Bradley–Terry). Lemma 2: they induce the same optimal policy under (3).
Theorem 1. Under mild assumptions (πref(y|x) > 0, β > 0), every reward class consistent with Plackett–Luce or Bradley–Terry can be represented as r(x, y) = β log (π(y|x) / πref(y|x)) for some model π. So restricting the reward model to this form loses nothing: the language model is a reward model whose own partition function is 1.
Instability of actor-critic (§5.2). Viewed through the same lens, PPO-style RLHF optimises rφ(x, y) − β log Σy πref(y|x) exp(rφ(x, y)/β) − β log(πθ/πref). The normalising term (a soft value function of the reference) does not change the optimum, but without it the policy-gradient estimate has high variance; it must be approximated by a learned value function or a single-sample human baseline, both sources of instability. DPO's reparameterisation yields a reward that needs no baseline.
Section · §6 · Figs. 2–3, Tables 1–2, Appendix B
Experiments
Setup. Default β = 0.1 (β = 0.5 for TL;DR), batch size 64, RMSprop, learning rate 1e-6 with 150-step linear warm-up. Evaluations with GPT-4 (gpt-4-0314) as judge unless stated.
- Controlled sentiment (IMDb, GPT-2-large). Prefixes of movie reviews; a sentiment classifier provides ground-truth reward, so the true reward-KL frontier can be measured. Over 22 runs sweeping PPO target KL ∈ {3, 6, 9, 12}, DPO β ∈ {0.05, 0.1, 1, 5} and baselines, DPO gives the best frontier, dominating PPO and even PPO with access to the ground-truth reward (PPO-GT) (Fig. 2 left).
- Summarisation (Reddit TL;DR, GPT-J-based SFT). Win rate versus reference summaries: DPO about 61% at temperature 0.0, PPO's best 57% at 0.0; DPO is far more robust across temperatures (Fig. 2 right). Best-of-N is a strong but expensive baseline.
- Single-turn dialogue (Anthropic HH, Pythia-2.8B). Reference model trained on chosen completions. DPO is the only computationally efficient method that improves over the dataset's chosen completions, and matches or exceeds Best-of-128 (Fig. 3).
- Distribution shift (Table 1). TL;DR policies applied to CNN/DailyMail: GPT-4 win rate vs ground truth 0.36 (DPO) vs 0.26 (PPO) at temperature 0, 0.31 vs 0.23 at 0.25.
| Table 2 (TL;DR, vs PPO at temp 0) | DPO | SFT | PPO-1 |
|---|---|---|---|
| Respondents | 272 | 122 | 199 |
| GPT-4 (C) win % | 54 | 32 | 12 |
| Human win % | 58 | 43 | 17 |
| GPT-4 (C)–human agreement % | 67 | 79 | 85 |
| Human–human agreement % | 65 | – | 87 |
GPT-4 agrees with humans about as often as humans agree with each other, which the authors use to justify GPT-4 as a proxy judge (§6.4). Raters were 25 volunteers, mostly Stanford STEM students.
Section · §7 and later work
Limitations and lineage
- Scale. Experiments stopped at 6B parameters; scaling to frontier models was left open.
- Generalisation and over-optimisation. How DPO policies generalise out of distribution versus an explicit reward model, and how reward over-optimisation shows up in DPO (perhaps the slight late decline in Fig. 3 right), are named as open questions.
- Offline data. DPO learns from a fixed dataset of pairs, typically generated by another model; the paper suggests self-labelling with the DPO policy as future work.
- Judge sensitivity. GPT-4 win rates depend on the prompt used.
Check yourself
- Why can the chosen response's likelihood decrease under DPO even as the loss falls?
- The loss depends only on the difference of log-ratios. Lowering both log-probabilities, with the rejected one falling faster, increases the margin and reduces the loss.
Reference
Glossary
- πref
- Reference policy, usually the SFT model; the anchor of the KL constraint.
- β
- KL strength; in DPO, the scale of the implicit reward.
- Implicit reward
- r̂(x, y) = β log(πθ/πref).
- Bradley–Terry
- Pairwise preference model with P(a ≻ b) = σ(ra − rb).
- Plackett–Luce
- Its generalisation to rankings of more than two items.
- Partition function Z(x)
- Normaliser of the optimal policy; intractable, cancels in DPO.
- PPO
- Proximal Policy Optimization, the actor-critic RL algorithm used in RLHF.
- Win rate
- Fraction of comparisons a judge prefers one system's output.