Study guide · Language models & AI · 2022

Training language models to follow instructions with human feedback

Long Ouyang et al. · NeurIPS 2022

GPT-3 was trained to continue internet text, which is not the same as doing what a user asks. Ouyang and colleagues at OpenAI fine-tune it with human demonstrations and human rankings of its answers (reinforcement learning from human feedback, RLHF), and people end up preferring the answers of a 1.3-billion-parameter InstructGPT over those of the 175-billion-parameter GPT-3.

The paper in four sentences

A large language model trained only to predict the next word often ignores instructions, makes things up, or says harmful things, because that is what "continue this text" rewards.

The authors hired about 40 people to write good answers to real user requests, and to rank several model answers from best to worst.

They fine-tuned GPT-3 on the written answers, trained a separate "reward model" to predict the rankings, and then used reinforcement learning to make GPT-3 produce answers the reward model scores highly.

Labellers preferred the resulting InstructGPT's answers to GPT-3's most of the time, even when InstructGPT was over 100 times smaller; it was also more truthful and somewhat less toxic, though no less biased.

Idea · §1

Predicting text is not following instructions

GPT-3 learned from a vast amount of internet text, one task only: guess the next word. In OpenAI's own illustration, GPT-3 given "Explain the moon landing to a 6 year old" may continue with more questions in the same style, because on the internet a list of questions is often followed by more questions.

The authors call this a problem of alignment: the model's training goal (predict text) differs from what users want (be helpful, honest and harmless). Bigger models are better at predicting text, but that does not by itself make them better at doing what they are asked.

Everyday pictureSomeone who has read every book in the library but has never had a job. They know an enormous amount, but if you hand them a request they might respond with a quotation, a parody, or a related question, rather than doing the task.

TakeawayA model's training goal shapes its behaviour. "Continue internet text" is not the same goal as "help the user", so more scale alone doesn't fix it.

Idea · §3.1 · Fig. 2

Three steps: show, rank, practise

The method has three steps, each using a different kind of human input. Step through them below.

The pipeline of the paper's Fig. 2. Prompt counts are the approximate dataset sizes from §3.2. Prompts came mainly from real requests submitted to OpenAI's API (with user consent), plus some written by the labellers.
Everyday pictureTraining a new waiter. First, they shadow experienced staff (demonstrations). Then a manager watches them serve several tables and ranks the visits from best to worst, and from many such rankings the waiter forms a sense of "what the manager likes" (the reward model). Finally they serve many more tables, adjusting to that sense, while not straying too far from how they were taught.

TakeawayDemonstrations teach the format, rankings teach a score, and reinforcement learning optimises against that score.

Check yourself
Why not just collect more demonstrations instead of rankings?
Judging which of several answers is better is much quicker and easier for a person than writing an excellent answer from scratch, and comparisons can be collected on the model's own outputs, telling it about its specific mistakes.

Idea · §3.5 "Reward modeling"

Turning a ranking into many comparisons

For each prompt, labellers saw between 4 and 9 model answers and put them in order. A ranking of K answers contains a comparison for every pair: "A beat B", "A beat C", and so on. That is K×(K−1)/2 comparisons from one ranking, so a single ranking of 9 answers yields 36 lessons for the reward model.

The reward model is a neural network that reads a prompt and an answer and outputs a single number. It is trained so that, for every pair, the preferred answer gets the higher number. Click the answers to change the ranking.

Left: the labeller's ranking, best at the top. Click an answer to move it up one place. Right: every pairwise comparison it implies, as a grid (amber = row answer beats column answer). The paper trains on all of a prompt's comparisons together as one batch element, because comparisons from the same ranking are strongly correlated and treating them as separate examples caused overfitting.

TakeawayOne ranking of K answers gives K(K−1)/2 pairwise lessons; the reward model learns a score that respects all of them.

Check yourself
How many comparisons come from ranking 6 answers?
6 × 5 / 2 = 15.

Idea · §4 · Fig. 1

Smaller but preferred

The headline result: labellers preferred answers from the 1.3-billion-parameter InstructGPT to those of the 175-billion-parameter GPT-3, a model more than 100 times larger. At the same 175B size, InstructGPT's answers were preferred 85 ± 3% of the time over GPT-3's, and 71 ± 4% of the time even when GPT-3 was given a carefully written few-shot prompt.

Other effects:

  • Truthfulness: on the TruthfulQA test, InstructGPT gave truthful and informative answers about twice as often as GPT-3.
  • Making things up: on tasks like summarising a given text, where the answer should come only from the input, it invented information 21% of the time against GPT-3's 41%.
  • Toxicity: about 25% fewer toxic outputs when asked to be respectful.
  • Bias: no significant improvement on two standard bias tests.

There was a cost, which the authors call the alignment tax: RLHF made the model slightly worse on some standard academic tests. Mixing in some of the original next-word training during the RL step (a variant called PPO-ptx) mostly fixed this.

TakeawayFine-tuning on human preferences made a model much more useful to people than a 100× increase in size did.

Idea · §5.2–5.3

Aligned to whom?

The authors are explicit that "aligned" here means aligned to a particular group: the roughly 40 labellers (mostly English-speaking), the researchers who wrote their instructions, and the API customers whose requests were the prompts. They do not claim this is the right standard for everyone.

The model still made up facts, still followed harmful instructions if asked, and could be overly cautious with simple questions. Most comparisons were labelled by only one person.

Later refinementThis recipe became the basis of ChatGPT later in 2022. Later work tried to reduce its cost and complexity: Constitutional AI replaced many human harm labels with AI feedback guided by written principles, DPO removed the separate reward model and the reinforcement-learning loop, and GRPO removed PPO's value network.

TakeawayRLHF aligns a model to whoever supplies the feedback, so who those people are, and what they were told, matters.

Follow the idea

Built on: GPT-3 (2020), PPO (2017)

Led to: Constitutional AI (2022), DPO (2023), GRPO (2024)