The paper in four sentences
Teaching a computer a language task, like judging whether one sentence follows from another, normally needs many thousands of examples labelled by people, which few tasks have.
This paper first trains a large network on a task that needs no labels at all: read books and guess each next word. To get good at that, the network has to pick up grammar, facts and some reasoning.
Then the same network is trained a little more on each labelled task, with the task's inputs simply written out as one stretch of text with marker symbols in between.
This "pre-train, then fine-tune" recipe beat specially designed systems on 9 of 12 benchmarks and became the blueprint for GPT-2, GPT-3 and today's chatbots.
Idea · §1–2
Labels are scarce, text is plentiful
Most machine-learning systems of 2018 learned a task from labelled examples: thousands of sentence pairs marked "follows", "contradicts" or "unrelated", say. Every new task meant a new labelled dataset and often a new model design. Meanwhile, the world has vast amounts of ordinary text with no labels at all.
People had already found a way to use unlabelled text: learn word vectors from it, then feed those into a task-specific model. But that only transfers knowledge about single words. The question here is whether a whole network, one that understands words in context, can be learned from raw text and then reused for many tasks with minimal changes.
TakeawayThe aim is to learn general language ability from unlabelled text, so each task needs only a little labelled data.
Idea · §3.1 · Eq. 1
Step one: guess the next word, millions of times
The pre-training task is simple to state: given the words so far, predict the next one. The text itself supplies the right answer, so no human labelling is needed. A model that does this well must learn a great deal: which words fit grammatically, what tends to follow what, who did what in the story so far.
The more context the model can use, the better its guesses. The toy below uses the crudest possible predictor, counting which word followed which in a short passage, to show how extra context sharpens guesses. GPT does the same job with a 12-layer Transformer that looks back over up to 512 tokens (words and word pieces), and it generalises instead of just counting.
TakeawayPredicting the next word is a free, endless source of training signal, and doing it well forces a model to learn a lot about language.
Check yourself
- Why is next-word prediction called "unsupervised" or "self-supervised"?
- Nobody labels anything: the correct answer for each position is just the next word in the text.
Idea · §4.1
Why books, and why a Transformer
The authors pre-trained on BooksCorpus, over 7,000 unpublished books in genres such as adventure, fantasy and romance. They chose books on purpose: they contain long stretches of continuous text, so the model can learn to use context from far back. A similar-sized collection of shuffled single sentences would not teach that.
The network is a stack of 12 Transformer layers of the "decoder" kind: each word may only look at words before it, which is what next-word prediction requires. Compared with the recurrent networks used in similar earlier work, the Transformer's attention gives it direct access to distant words, and the paper finds it transfers markedly better.
TakeawayLong continuous text plus a Transformer that can attend far back let the model learn long-range structure.
Idea · §3.2–3.3 · Fig. 1
Step two: a short fine-tune, with tasks written as text
After pre-training, the network is trained for a few more passes on each labelled task. The only new part is one small final layer that turns the network's last output into an answer (for example "follows" / "contradicts" / "neutral").
The neat trick is how inputs are presented. The network only knows how to read a single stretch of text. So a pair of sentences is just written one after the other with a special separator symbol between them, framed by special start and end symbols. A multiple-choice question becomes several stretches of text, one per answer, each scored separately. No new machinery per task.
During fine-tuning the model also keeps practising next-word prediction on the task's text as a side exercise, which the authors found helped, especially on larger datasets.
TakeawayEvery task is turned into text the pre-trained model can already read, so almost nothing new has to be learned from scratch.
Check yourself
- Why run both orderings for similarity tasks?
- The model reads left to right, so "A then B" and "B then A" produce different outputs. Similarity is symmetric, so combining both removes the arbitrary order.
Idea · §4.2 · §5 · What came next
Results, and the seed of GPT-2 and GPT-3
One model, lightly adapted, beat the best published systems on 9 of 12 datasets, many of which had been designed by hand for their task. The biggest jumps were on a common-sense story-ending test (8.9 points better) and on school reading-comprehension exams (5.7 points better). It fell short on the other three (RTE, SST-2 and MRPC), two of which are among the smallest datasets tested.
Two analyses pointed to the future. Removing pre-training dropped average performance by about 15 points, so the pre-training was doing most of the work. And even before any fine-tuning, the pre-trained model could do some tasks a little, using simple hand-written rules on its predictions, and this "zero-shot" ability grew steadily as pre-training went on (Fig. 2, right).
TakeawayGenerative pre-training plus light fine-tuning beat task-specific designs, and the model's emerging zero-shot skills hinted at the scaled-up GPTs to come.
Abstract, restated
Natural language understanding spans tasks such as textual entailment, question answering, semantic similarity assessment and document classification. Unlabelled text is abundant but labelled data for each task is scarce, making discriminatively trained models hard to make perform well. Large gains come from generative pre-training of a language model on a diverse unlabelled corpus followed by discriminative fine-tuning on each task, using task-aware input transformations so the architecture changes minimally. The task-agnostic model beats discriminatively trained models with task-specific architectures, improving the state of the art on 9 of the 12 tasks studied: absolute gains of 8.9% on commonsense reasoning (Story Cloze), 5.7% on question answering (RACE) and 1.5% on textual entailment (MultiNLI). OpenAI technical report, June 2018.
Section · §3.1 · Eqs. 1–2
Unsupervised pre-training
Given an unlabelled token corpus 𝒰 = {u1, …, un}, maximise the standard language-modelling likelihood with context window k:
using a multi-layer Transformer decoder (the decoder-only variant of Liu et al., 2018): masked multi-head self-attention followed by position-wise feed-forward layers.
U is the one-hot context matrix, We the token embedding (tied with the output projection), Wp learned position embeddings. Because of the causal mask, one forward pass yields a prediction at every position, so a 512-token sequence provides 511 training targets in parallel.
Section · §3.2–3.3 · Eqs. 3–5 · Fig. 1
Supervised fine-tuning and input transformations
For a labelled dataset 𝒞 of token sequences x1, …, xm with label y, the final block's activation at the last token, hlm, feeds one added linear layer Wy:
The auxiliary LM term on the task data improves generalisation and speeds convergence. The only new parameters are Wy and embeddings for the delimiter tokens. Input transformations (§3.3): all add randomly initialised start and end tokens ⟨s⟩, ⟨e⟩. Entailment: premise $ hypothesis. Similarity: both orderings processed independently, final representations hlm added element-wise before Wy. Question answering and commonsense reasoning: for context z, question q and answers {ak}, each [z; q; $; ak] is processed independently and normalised with a softmax over answers. This "traversal-style" approach (after Rocktäschel et al., 2015) avoids task-specific architectures on top of the transferred network, unlike ELMo-style feature extraction.
Section · §4.1
Setup
Data. BooksCorpus (over 7,000 unique unpublished books). The authors contrast it with the 1B Word Benchmark, similar in size but shuffled at sentence level, which destroys long-range structure. The model reaches a very low token-level perplexity of 18.4 on BooksCorpus.
Model. 12-layer decoder-only Transformer, dmodel = 768, 12 heads, feed-forward inner size 3072; learned position embeddings; GELU activations; BPE vocabulary with 40,000 merges; weights initialised N(0, 0.02); text cleaned with ftfy and pre-tokenised with spaCy. (Later reported as about 117M parameters; the paper does not state the count.)
Pre-training. Adam, max learning rate 2.5·10−4, linear warm-up over the first 2,000 updates then cosine annealing to 0; 100 epochs on minibatches of 64 random contiguous 512-token sequences; residual, embedding and attention dropout 0.1; a modified L2 regularisation (w = 0.01) on non-bias/gain weights.
Fine-tuning. Reuse the pre-training hyperparameters except: classifier dropout 0.1, learning rate 6.25·10−5, batch size 32, 3 epochs for most tasks, linear decay with warm-up over 0.2% of training, λ = 0.5.
Section · §4.2 · Tables 2–4
Results on 12 datasets
Notable: Story Cloze 86.5 vs 77.6; RACE 59.0 vs 53.3 (a 9-model ensemble); SciTail 88.3 vs 83.3; QNLI 88.1 vs 82.3; MNLI 82.1/81.4 vs 80.6/80.1 (a 3-model ensemble); CoLA 45.4 vs 35.0, which the authors attribute to linguistic knowledge learned in pre-training; GLUE overall 72.8 vs 68.9. Misses: RTE (2,490 examples) 56.0 vs 61.7 for a multi-task BiLSTM; SST-2 91.3 vs 93.2; MRPC 82.3 vs 86.0 F1.
Section · §5 · Table 5 · Fig. 2 · Lineage
Analysis, limitations and lineage
Layers transferred (Fig. 2 left). Transferring embeddings helps, and each additional Transformer layer adds more, up to about 9% on MultiNLI for full transfer: every pre-trained layer contains useful functionality.
Zero-shot behaviours (Fig. 2 right). Heuristic zero-shot solutions using the generative model alone (e.g. for SST-2, append "very" and compare the LM's probability of "positive" vs "negative"; for linguistic acceptability, threshold average token log-probability) improve steadily over pre-training, more stably than for an LSTM, suggesting the LM objective itself teaches task-relevant abilities.
| Ablation (Table 5) | Avg. score |
|---|---|
| Transformer with auxiliary LM (full) | 74.7 |
| Transformer without pre-training | 59.9 |
| Transformer without auxiliary LM | 75.0 |
| LSTM with auxiliary LM | 69.1 |
The auxiliary objective helps on the larger datasets (NLI, QQP) but not on smaller ones, so its net average effect is slightly negative. The single-layer 2048-unit LSTM loses 5.6 points on average versus the Transformer, outperforming it only on MRPC.
- Contemporaries. ELMo (Peters et al., 2018) transferred bidirectional LSTM features into task-specific models; ULMFiT (Howard & Ruder, 2018) fine-tuned an LSTM LM for classification; Dai & Le (2015) pre-trained sequence autoencoders and LMs for fine-tuning. GPT's distinctive choices were the Transformer and a single architecture for all task types.
- Unidirectionality. BERT (Oct. 2018) argued the left-to-right constraint is a key limitation for understanding tasks and replaced it with masked-LM pre-training, beating GPT across GLUE with a same-size model (BERT-base).
- Scaling and prompting. GPT-2 scaled the same decoder to 1.5B parameters on WebText, moved LayerNorm to the start of each block, and emphasised zero-shot task transfer; GPT-3 (175B) introduced in-context few-shot learning; InstructGPT added human-feedback fine-tuning.
- Data concerns. BooksCorpus was later criticised for copyright and duplication issues (e.g. Bandy & Vincent, 2021).
Reference
Glossary
- Language model
- A model of the probability of the next token given the previous ones.
- Pre-training
- Training on a generic, usually self-supervised objective before task training.
- Fine-tuning
- Continuing training of all weights on a labelled task.
- Causal mask
- Attention mask allowing each position to attend only to earlier positions.
- BPE
- Byte-pair encoding subword vocabulary.
- GELU
- Gaussian error linear unit, x·Φ(x).
- Auxiliary objective
- The LM loss L1 added during fine-tuning with weight λ.
- Zero-shot
- Performing a task with no task-specific training.
- GLUE
- General Language Understanding Evaluation benchmark of nine tasks.
- RACE / Story Cloze
- Exam reading comprehension; choosing the right ending to a four-sentence story.