The paper in four sentences
A large language model trained only to predict the next word often ignores instructions, makes things up, or says harmful things, because that is what "continue this text" rewards.
The authors hired about 40 people to write good answers to real user requests, and to rank several model answers from best to worst.
They fine-tuned GPT-3 on the written answers, trained a separate "reward model" to predict the rankings, and then used reinforcement learning to make GPT-3 produce answers the reward model scores highly.
Labellers preferred the resulting InstructGPT's answers to GPT-3's most of the time, even when InstructGPT was over 100 times smaller; it was also more truthful and somewhat less toxic, though no less biased.
Idea · §1
Predicting text is not following instructions
GPT-3 learned from a vast amount of internet text, one task only: guess the next word. In OpenAI's own illustration, GPT-3 given "Explain the moon landing to a 6 year old" may continue with more questions in the same style, because on the internet a list of questions is often followed by more questions.
The authors call this a problem of alignment: the model's training goal (predict text) differs from what users want (be helpful, honest and harmless). Bigger models are better at predicting text, but that does not by itself make them better at doing what they are asked.
TakeawayA model's training goal shapes its behaviour. "Continue internet text" is not the same goal as "help the user", so more scale alone doesn't fix it.
Idea · §3.1 · Fig. 2
Three steps: show, rank, practise
The method has three steps, each using a different kind of human input. Step through them below.
TakeawayDemonstrations teach the format, rankings teach a score, and reinforcement learning optimises against that score.
Check yourself
- Why not just collect more demonstrations instead of rankings?
- Judging which of several answers is better is much quicker and easier for a person than writing an excellent answer from scratch, and comparisons can be collected on the model's own outputs, telling it about its specific mistakes.
Idea · §3.5 "Reward modeling"
Turning a ranking into many comparisons
For each prompt, labellers saw between 4 and 9 model answers and put them in order. A ranking of K answers contains a comparison for every pair: "A beat B", "A beat C", and so on. That is K×(K−1)/2 comparisons from one ranking, so a single ranking of 9 answers yields 36 lessons for the reward model.
The reward model is a neural network that reads a prompt and an answer and outputs a single number. It is trained so that, for every pair, the preferred answer gets the higher number. Click the answers to change the ranking.
TakeawayOne ranking of K answers gives K(K−1)/2 pairwise lessons; the reward model learns a score that respects all of them.
Check yourself
- How many comparisons come from ranking 6 answers?
- 6 × 5 / 2 = 15.
Idea · §4 · Fig. 1
Smaller but preferred
The headline result: labellers preferred answers from the 1.3-billion-parameter InstructGPT to those of the 175-billion-parameter GPT-3, a model more than 100 times larger. At the same 175B size, InstructGPT's answers were preferred 85 ± 3% of the time over GPT-3's, and 71 ± 4% of the time even when GPT-3 was given a carefully written few-shot prompt.
Other effects:
- Truthfulness: on the TruthfulQA test, InstructGPT gave truthful and informative answers about twice as often as GPT-3.
- Making things up: on tasks like summarising a given text, where the answer should come only from the input, it invented information 21% of the time against GPT-3's 41%.
- Toxicity: about 25% fewer toxic outputs when asked to be respectful.
- Bias: no significant improvement on two standard bias tests.
There was a cost, which the authors call the alignment tax: RLHF made the model slightly worse on some standard academic tests. Mixing in some of the original next-word training during the RL step (a variant called PPO-ptx) mostly fixed this.
TakeawayFine-tuning on human preferences made a model much more useful to people than a 100× increase in size did.
Idea · §5.2–5.3
Aligned to whom?
The authors are explicit that "aligned" here means aligned to a particular group: the roughly 40 labellers (mostly English-speaking), the researchers who wrote their instructions, and the API customers whose requests were the prompts. They do not claim this is the right standard for everyone.
The model still made up facts, still followed harmful instructions if asked, and could be overly cautious with simple questions. Most comparisons were labelled by only one person.
TakeawayRLHF aligns a model to whoever supplies the feedback, so who those people are, and what they were told, matters.
Abstract, restated
Making language models bigger does not inherently make them better at following a user's intent; large models can be untruthful, toxic or unhelpful. Starting from labeller-written prompts and prompts submitted through the OpenAI API, the authors collect labeller demonstrations of desired behaviour and fine-tune GPT-3 with supervised learning, then collect rankings of model outputs and further fine-tune with reinforcement learning from human feedback. In human evaluations on the prompt distribution, outputs from the 1.3B InstructGPT are preferred to those of 175B GPT-3, despite 100× fewer parameters. InstructGPT improves truthfulness and reduces toxic generation with minimal performance regressions on public NLP datasets, though it still makes simple mistakes. Ouyang et al., NeurIPS 2022 (arXiv 2203.02155).
Section · §3.2–3.4 · Table 1
Data and labellers
Prompts come mainly from the OpenAI API (InstructGPT beta playground), deduplicated by common prefix and limited to 200 per user ID, with train/validation/test split by user; plus prompts written by labellers to bootstrap. About 40 contractors were hired via Upwork and ScaleAI after a screening test of sensitivity to diverse demographic preferences and ability to identify harmful outputs. Labellers prioritised helpfulness during training data collection but truthfulness and harmlessness in final evaluations.
| Dataset (§3.2) | Prompts | Supervision |
|---|---|---|
| SFT | ~13k | labeller demonstrations |
| RM | ~33k | labeller rankings of K = 4–9 outputs |
| PPO | ~31k | none (API prompts only) |
| Use case (Table 1) | % | Use case | % |
|---|---|---|---|
| Generation | 45.6 | Summarization | 4.2 |
| Open QA | 12.4 | Classification | 3.5 |
| Brainstorming | 11.2 | Other | 3.5 |
| Chat | 8.4 | Closed QA | 2.6 |
| Rewrite | 6.6 | Extract | 1.9 |
Note the mismatch with academic benchmarks: classification and QA, which dominate public NLP datasets, are a small share of what API users actually asked for.
Section · §3.5 · Eq. 1
SFT and the reward model
SFT. GPT-3 fine-tuned on demonstrations for 16 epochs with cosine learning-rate decay; the SFT models overfit on validation loss after one epoch, but more epochs helped both RM score and human preference.
Reward model. Start from the SFT model with the unembedding layer replaced by a scalar head. Only 6B RMs are used: they save compute, and 175B RM training was found to be unstable and thus less suitable as the value function in RL. For a prompt x with K ranked completions:
All C(K, 2) comparisons from one prompt form a single batch element: one forward pass per completion rather than per comparison, and no overfitting from seeing correlated pairs in separate updates (shuffled pairs overfit after one epoch). The RM is normalised with a bias so that demonstrations score 0 on average before RL. Try the loss for one prompt with K = 4 and the true ranking A ≻ B ≻ C ≻ D:
Section · §3.5 "Reinforcement learning" · Eq. 2
PPO and PPO-ptx
The policy is initialised from SFT and trained with PPO in a bandit setting: a random API prompt, a sampled response, a reward from the RM, episode over. A per-token KL penalty against the SFT model mitigates over-optimisation of the RM, and the value function is initialised from the RM. PPO-ptx additionally mixes in pretraining gradients:
β sets the KL strength; γ = 0 gives plain "PPO" models, γ > 0 gives "PPO-ptx" (the models called InstructGPT unless stated otherwise). Why the KL term matters: the RM is a learned proxy trained on a finite set of comparisons, so a policy pushed far enough will find outputs the RM overrates. The toy below illustrates the trade-off.
Section · §4 · Figs. 1, 3–7
Results
- API prompt distribution (§4.1). Labeller preference ordering is PPO-ptx ≈ PPO > SFT > GPT-3 prompted > GPT-3 at every size (1.3B, 6B, 175B). 175B InstructGPT beats 175B GPT-3 85 ± 3% of the time and few-shot GPT-3 71 ± 4%. The 1.3B PPO-ptx model is preferred to 175B GPT-3. Results hold for held-out labellers who produced no training data, and on prompts submitted to GPT-3 (not InstructGPT) models.
- FLAN and T0 (§4.1). GPT-3 fine-tuned on these public instruction datasets does slightly worse than SFT on API prompts; public NLP tasks do not reflect real usage.
- Truthfulness. On TruthfulQA, about twice as often truthful and informative as GPT-3. On closed-domain API tasks (summarisation, closed QA), hallucination 21% vs 41%.
- Toxicity (RealToxicityPrompts). About 25% fewer toxic outputs than GPT-3 when instructed to be respectful; no advantage without that instruction, and when explicitly instructed to be toxic, its outputs are much more toxic than GPT-3's.
- Bias. No significant improvement on Winogender and CrowS-Pairs.
- Alignment tax. PPO regresses on SQuAD, DROP, HellaSwag and WMT 2015 Fr→En; PPO-ptx greatly reduces the regressions without hurting labeller preference, though some gaps remain (DROP, SQuADv2, translation). Raising the KL coefficient alone is less effective: it costs reward and never fully recovers DROP and SQuAD.
- Generalisation (§4.3). Qualitatively follows instructions in non-English languages and on code, which were rare in fine-tuning data, suggesting the behaviour generalises beyond the supervised distribution.
Check yourself
- Why mix pretraining gradients into PPO rather than increasing β?
- β constrains the policy towards SFT outputs on the RL prompt distribution, which does not protect capabilities exercised only on other inputs. Pretraining gradients directly maintain likelihood on broad text, which the paper finds reduces regressions more effectively.
Section · §5.2–5.4 and later work
Limitations and lineage
- Whose preferences (§5.2). Labellers, the researchers' instructions, and API customers (from a waitlist seeded by OpenAI employees). Not representative of all users or of people affected by the model.
- Labelling (§5.3). Mostly one labeller per comparison; labellers mostly English-speaking; inter-labeller agreement was imperfect (training labellers agreed with each other 72.6 ± 1.5% of the time, held-out labellers 77.3 ± 1.3%).
- Behaviour. Still fabricates, still follows harmful instructions when asked, sometimes over-hedges, and can be misled by false premises.
- Open questions (§5.4). Adversarial data collection, filtering pretraining data, steerability, alternative algorithms to PPO, and richer feedback such as edits or critiques.
Reference
Glossary
- SFT
- Supervised fine-tuning on human demonstrations.
- RM
- Reward model: scalar score for (prompt, completion), trained on comparisons.
- PPO
- Proximal Policy Optimization, clipped policy-gradient RL with a value function.
- PPO-ptx
- PPO with pretraining log-likelihood gradients mixed in (γ > 0).
- KL penalty β
- Per-token penalty on log(πRL/πSFT).
- Alignment tax
- Performance lost on other tasks as a result of alignment training.
- Hallucination
- Content not supported by the input in closed-domain tasks.
- C(K, 2)
- Number of pairs among K items, K(K−1)/2.