Study guide · Language models & AI · 2019

Language Models are Unsupervised Multitask Learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever · OpenAI

Can a language model learn to do tasks such as translating, summarising or answering questions without ever being trained on them? OpenAI trained a 1.5-billion-parameter transformer to do one thing, predict the next word of 40 GB of web pages, and found it could attempt all of these when the task was simply written into the prompt. It set new records on 7 of 8 language-modelling benchmarks without seeing their training data.

The paper in four sentences

Until 2019, a model for each language task (translation, question answering, summarising) was trained on a hand-labelled dataset made for that task.

The authors argue that ordinary web text already contains countless examples of these tasks, such as a sentence followed by its French translation, so a model that predicts web text well must partly learn to do them.

They collected 8 million web pages that Reddit users had linked to, called WebText, and trained a transformer of up to 1.5 billion parameters, GPT-2, simply to predict the next piece of text.

With no task-specific training at all ("zero-shot"), GPT-2 beat the best previous results on 7 of 8 language-modelling benchmarks and made a real start on reading comprehension, summarising and translation, getting steadily better as the model grew.

Idea · §1–2

The task is written in the text

A language model is a program that reads some text and guesses what comes next, one small piece at a time. Ask it to continue "The capital of France is" and a good one puts most of its bets on "Paris".

The usual way to build, say, a translator was different: collect a big table of sentence pairs and train a model only on that. The model is a specialist, and it does nothing else.

The paper's key observation is that you can describe a task in words, as part of the input. A translation request is just text like "translate to French: the cat", and a question is just text ending in "A:". If the model has read enough of the web, it has seen many passages shaped like that, and the most likely continuation is the answer. The authors even found sentences in their data such as an English quote followed by "in French, that is …" (they list examples in Table 1).

Try it on a tiny pretend model below. It has only "read" a dozen sentences, and it predicts the next word by looking for the longest matching phrase it has seen before. Notice how the same final words, "dog is", lead to different guesses depending on what comes before them.

An illustrative toy, not GPT-2: a word-counting model "trained" on 14 short sentences (French word lessons, quiz questions, everyday sentences). It looks back up to three words and backs off to shorter matches when it has no exact match. Bars show its probability for each possible next word. GPT-2 does something far richer with a neural network, but the lesson carries over: the words before the gap tell the model which "task" it is in.
Everyday pictureA child who has read thousands of recipe books, quiz books and phrase books has never had a lesson in translating, but if you write "Hello in French is …" on the board they will probably say "bonjour". They learned the task as a side effect of reading.

TakeawayIf a task can be written as text, predicting text well means learning a little of that task, for free.

Check yourself
In this framing, how do you tell the model which task to do?
By writing the task into the input, for example with a format like "question … A:" or a sentence pair pattern, so the most likely continuation is the answer.
Why did the toy give different answers after "dog is"?
Because it uses the longer context: after "in french , dog is" the only continuation it had seen was "chien", while "my dog is" led to "happy". The earlier words selected the task.

Idea · §2.1

WebText: let Reddit pick the pages

To learn many tasks from reading, the model needs a lot of varied, decent-quality text. The whole web is huge but full of junk, and cleaning it by hand is impossible.

The authors used a shortcut. They took every outbound link posted on the discussion site Reddit that had received at least 3 "karma" (net upvotes), on the theory that a human had found the page interesting, funny or useful. After removing duplicates and cleaning, this gave a little over 8 million documents, 40 GB of text, which they called WebText.

They removed all Wikipedia pages, because many of the tests they planned use Wikipedia and they did not want the model to have simply read the answers.

Everyday pictureInstead of reading every book in the world, you read only the ones a friend has recommended. You miss some gems, but you skip most of the rubbish.

TakeawayHuman upvotes served as a cheap quality filter, giving a large and varied training set without hand-labelling.

Idea · §2.3 · §3.1 · Table 3

Bigger keeps getting better

The model itself is a transformer, the same design as GPT-1 with small changes. The authors trained four sizes, from 117 million to 1.5 billion parameters (the adjustable numbers inside the network that training tunes).

They then tested each size, with no extra training, on standard language benchmarks. Some measure perplexity: roughly, how many words the model is "torn between" at each step, so lower is better. Others measure accuracy on fill-in-the-blank tests. Pick a benchmark below. The dashed line is the best published result before GPT-2, achieved by models trained on that benchmark's own data.

Real numbers from Table 3 of the paper. The four dots are the four model sizes; the dashed line is the previous state of the art. On every benchmark the largest model is better than the smallest. One Billion Word is the exception that stays far behind: its sentences are shuffled, which destroys the long-range structure GPT-2 relies on.

Two details stand out. First, the gains were biggest on small benchmarks and on ones that need long-range context, such as LAMBADA, where you must read a whole paragraph to guess its last word. Second, even the largest model was still improving on its own training data as it grew (Fig. 4): it had not run out of things to learn.

TakeawayWithout seeing any benchmark's training data, the largest GPT-2 set a new record on 7 of 8 language-modelling tests, and every step up in size helped.

Check yourself
Is lower or higher perplexity better, and why?
Lower. It measures how uncertain the model is about the next word; a perplexity of 10 is like choosing among about ten equally likely words each step.
Why did GPT-2 do poorly on One Billion Word?
That benchmark's sentences are shuffled, so there is no long-range context to use; the paper notes this "destructive pre-processing".

Idea · §3.5–3.8

Tasks it was never taught: promising, not yet good

The more surprising tests asked GPT-2 to do things that are not language modelling at all, using only a cleverly shaped prompt:

  • Reading comprehension. Give it a story, a conversation about it, and end with "A:". It scored 55 on the CoQA test's F1 measure, matching or beating 3 of 4 earlier systems that had been trained on more than 127,000 labelled examples.
  • Summarising. Paste a news article and add "TL;DR:" (internet slang for "too long, didn't read: here's the short version"). The summaries were weak by the standards of trained systems, but removing the "TL;DR:" hint made them clearly worse, showing the hint really does call up the task.
  • Translation. Show a few "English sentence = French sentence" pairs, then an English sentence and "=". French to English reached 11.5 BLEU (a translation-quality score), far below the best systems but remarkable given that only about 10 MB of the training text was French.
  • Trivia questions. It answered 4.1% of Natural Questions exactly right; on the 1% of questions it was most confident about, it was right 63.1% of the time.

The authors are frank that most of these results are far from useful. The point is the trend: performance on every one rose with model size, which suggested that much bigger models might cross into usefulness.

Later refinementThat bet was tested a year later by GPT-3: over 100 times larger, and given a few worked examples in the prompt ("few-shot"), it became competitive on many of these tasks. Later still, instruction tuning with human feedback made models follow plain requests instead of needing prompt tricks like "TL;DR:".

TakeawayZero-shot task performance was weak but real, and it climbed steadily with size: the first strong hint that scale alone unlocks new abilities.

Check yourself
What does the "TL;DR:" experiment show?
That the prompt wording selects the task: with the hint the summaries score higher than without it, so the model has learned what usually follows "TL;DR:" on the web.

Idea · §4 · Discussion

Did it just memorise? And should it be released?

A model that has read 8 million web pages might have read the test questions too. The authors checked by chopping texts into runs of 8 words and seeing how many runs from each test set also appear in WebText. Overlap was small, between 1% and 6% for most test sets (3.2% on average), and removing it changed scores only slightly. Several test sets overlapped more with their own training sets than with WebText.

The paper itself is a research report, but GPT-2 is also remembered for how it was released. Worried that fluent fake text could be misused, OpenAI at first published only the smallest model and released the full 1.5-billion-parameter version in stages over 2019. Whatever one thinks of that choice, it started a lasting debate about releasing powerful models.

TakeawayThe results were mostly genuine generalisation, not memorised test answers; and GPT-2's staged release made "should we publish this model?" a real question.

Follow the idea

Built on: GPT-1 (2018)

Led to: GPT-3 (2020)