Study guide · Language models & AI · 2018

Improving Language Understanding by Generative Pre-Training

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever · OpenAI

Labelled examples for language tasks are scarce and expensive, but plain text is everywhere. Radford and colleagues at OpenAI first train a Transformer simply to predict the next word across 7,000 unpublished books, then lightly adapt that same network to each task. With almost no task-specific design, it set new records on 9 of the 12 benchmarks they tried: the first "GPT".

The paper in four sentences

Teaching a computer a language task, like judging whether one sentence follows from another, normally needs many thousands of examples labelled by people, which few tasks have.

This paper first trains a large network on a task that needs no labels at all: read books and guess each next word. To get good at that, the network has to pick up grammar, facts and some reasoning.

Then the same network is trained a little more on each labelled task, with the task's inputs simply written out as one stretch of text with marker symbols in between.

This "pre-train, then fine-tune" recipe beat specially designed systems on 9 of 12 benchmarks and became the blueprint for GPT-2, GPT-3 and today's chatbots.

Idea · §1–2

Labels are scarce, text is plentiful

Most machine-learning systems of 2018 learned a task from labelled examples: thousands of sentence pairs marked "follows", "contradicts" or "unrelated", say. Every new task meant a new labelled dataset and often a new model design. Meanwhile, the world has vast amounts of ordinary text with no labels at all.

People had already found a way to use unlabelled text: learn word vectors from it, then feed those into a task-specific model. But that only transfers knowledge about single words. The question here is whether a whole network, one that understands words in context, can be learned from raw text and then reused for many tasks with minimal changes.

Everyday pictureRather than training a new employee from scratch for every job, hire someone who has read widely for years, then give them a short induction for each role.

TakeawayThe aim is to learn general language ability from unlabelled text, so each task needs only a little labelled data.

Idea · §3.1 · Eq. 1

Step one: guess the next word, millions of times

The pre-training task is simple to state: given the words so far, predict the next one. The text itself supplies the right answer, so no human labelling is needed. A model that does this well must learn a great deal: which words fit grammatically, what tends to follow what, who did what in the story so far.

The more context the model can use, the better its guesses. The toy below uses the crudest possible predictor, counting which word followed which in a short passage, to show how extra context sharpens guesses. GPT does the same job with a 12-layer Transformer that looks back over up to 512 tokens (words and word pieces), and it generalises instead of just counting.

A word-counting model "trained" on a 150-word story written for this guide. With 1 word of context the guesses are vague; with 3 they become confident (here, too confident: the model mostly recites the story, because the text is tiny). A neural language model learns from billions of words and can guess sensibly about word sequences it has never seen. Illustrative only.

TakeawayPredicting the next word is a free, endless source of training signal, and doing it well forces a model to learn a lot about language.

Check yourself
Why is next-word prediction called "unsupervised" or "self-supervised"?
Nobody labels anything: the correct answer for each position is just the next word in the text.

Idea · §4.1

Why books, and why a Transformer

The authors pre-trained on BooksCorpus, over 7,000 unpublished books in genres such as adventure, fantasy and romance. They chose books on purpose: they contain long stretches of continuous text, so the model can learn to use context from far back. A similar-sized collection of shuffled single sentences would not teach that.

The network is a stack of 12 Transformer layers of the "decoder" kind: each word may only look at words before it, which is what next-word prediction requires. Compared with the recurrent networks used in similar earlier work, the Transformer's attention gives it direct access to distant words, and the paper finds it transfers markedly better.

TakeawayLong continuous text plus a Transformer that can attend far back let the model learn long-range structure.

Idea · §3.2–3.3 · Fig. 1

Step two: a short fine-tune, with tasks written as text

After pre-training, the network is trained for a few more passes on each labelled task. The only new part is one small final layer that turns the network's last output into an answer (for example "follows" / "contradicts" / "neutral").

The neat trick is how inputs are presented. The network only knows how to read a single stretch of text. So a pair of sentences is just written one after the other with a special separator symbol between them, framed by special start and end symbols. A multiple-choice question becomes several stretches of text, one per answer, each scored separately. No new machinery per task.

During fine-tuning the model also keeps practising next-word prediction on the task's text as a side exercise, which the authors found helped, especially on larger datasets.

After the paper's Fig. 1 (right). Each row is one pass through the same pre-trained Transformer. The output at the final "end" token feeds a small new linear layer. For similarity, where order shouldn't matter, both orderings are run and their outputs added. For multiple choice, each answer gets its own pass and a softmax picks among them. Example sentences are made up.

TakeawayEvery task is turned into text the pre-trained model can already read, so almost nothing new has to be learned from scratch.

Check yourself
Why run both orderings for similarity tasks?
The model reads left to right, so "A then B" and "B then A" produce different outputs. Similarity is symmetric, so combining both removes the arbitrary order.

Idea · §4.2 · §5 · What came next

Results, and the seed of GPT-2 and GPT-3

One model, lightly adapted, beat the best published systems on 9 of 12 datasets, many of which had been designed by hand for their task. The biggest jumps were on a common-sense story-ending test (8.9 points better) and on school reading-comprehension exams (5.7 points better). It fell short on the other three (RTE, SST-2 and MRPC), two of which are among the smallest datasets tested.

Two analyses pointed to the future. Removing pre-training dropped average performance by about 15 points, so the pre-training was doing most of the work. And even before any fine-tuning, the pre-trained model could do some tasks a little, using simple hand-written rules on its predictions, and this "zero-shot" ability grew steadily as pre-training went on (Fig. 2, right).

Later refinementA few months later BERT showed that letting the model look at words on both sides (filling in blanks rather than predicting the next word) did even better on these understanding tasks. But GPT-2 (2019) and GPT-3 (2020) followed the zero-shot thread instead: make the same next-word model far larger, train it on far more text, and skip fine-tuning altogether by describing the task in the prompt. That path led to today's chatbots.

TakeawayGenerative pre-training plus light fine-tuning beat task-specific designs, and the model's emerging zero-shot skills hinted at the scaled-up GPTs to come.

Follow the idea

Built on: Transformer (2017)

Led to: BERT (2018), GPT-2 (2019)