The paper in four sentences
Until 2019, a model for each language task (translation, question answering, summarising) was trained on a hand-labelled dataset made for that task.
The authors argue that ordinary web text already contains countless examples of these tasks, such as a sentence followed by its French translation, so a model that predicts web text well must partly learn to do them.
They collected 8 million web pages that Reddit users had linked to, called WebText, and trained a transformer of up to 1.5 billion parameters, GPT-2, simply to predict the next piece of text.
With no task-specific training at all ("zero-shot"), GPT-2 beat the best previous results on 7 of 8 language-modelling benchmarks and made a real start on reading comprehension, summarising and translation, getting steadily better as the model grew.
Idea · §1–2
The task is written in the text
A language model is a program that reads some text and guesses what comes next, one small piece at a time. Ask it to continue "The capital of France is" and a good one puts most of its bets on "Paris".
The usual way to build, say, a translator was different: collect a big table of sentence pairs and train a model only on that. The model is a specialist, and it does nothing else.
The paper's key observation is that you can describe a task in words, as part of the input. A translation request is just text like "translate to French: the cat", and a question is just text ending in "A:". If the model has read enough of the web, it has seen many passages shaped like that, and the most likely continuation is the answer. The authors even found sentences in their data such as an English quote followed by "in French, that is …" (they list examples in Table 1).
Try it on a tiny pretend model below. It has only "read" a dozen sentences, and it predicts the next word by looking for the longest matching phrase it has seen before. Notice how the same final words, "dog is", lead to different guesses depending on what comes before them.
TakeawayIf a task can be written as text, predicting text well means learning a little of that task, for free.
Check yourself
- In this framing, how do you tell the model which task to do?
- By writing the task into the input, for example with a format like "question … A:" or a sentence pair pattern, so the most likely continuation is the answer.
- Why did the toy give different answers after "dog is"?
- Because it uses the longer context: after "in french , dog is" the only continuation it had seen was "chien", while "my dog is" led to "happy". The earlier words selected the task.
Idea · §2.1
WebText: let Reddit pick the pages
To learn many tasks from reading, the model needs a lot of varied, decent-quality text. The whole web is huge but full of junk, and cleaning it by hand is impossible.
The authors used a shortcut. They took every outbound link posted on the discussion site Reddit that had received at least 3 "karma" (net upvotes), on the theory that a human had found the page interesting, funny or useful. After removing duplicates and cleaning, this gave a little over 8 million documents, 40 GB of text, which they called WebText.
They removed all Wikipedia pages, because many of the tests they planned use Wikipedia and they did not want the model to have simply read the answers.
TakeawayHuman upvotes served as a cheap quality filter, giving a large and varied training set without hand-labelling.
Idea · §2.3 · §3.1 · Table 3
Bigger keeps getting better
The model itself is a transformer, the same design as GPT-1 with small changes. The authors trained four sizes, from 117 million to 1.5 billion parameters (the adjustable numbers inside the network that training tunes).
They then tested each size, with no extra training, on standard language benchmarks. Some measure perplexity: roughly, how many words the model is "torn between" at each step, so lower is better. Others measure accuracy on fill-in-the-blank tests. Pick a benchmark below. The dashed line is the best published result before GPT-2, achieved by models trained on that benchmark's own data.
Two details stand out. First, the gains were biggest on small benchmarks and on ones that need long-range context, such as LAMBADA, where you must read a whole paragraph to guess its last word. Second, even the largest model was still improving on its own training data as it grew (Fig. 4): it had not run out of things to learn.
TakeawayWithout seeing any benchmark's training data, the largest GPT-2 set a new record on 7 of 8 language-modelling tests, and every step up in size helped.
Check yourself
- Is lower or higher perplexity better, and why?
- Lower. It measures how uncertain the model is about the next word; a perplexity of 10 is like choosing among about ten equally likely words each step.
- Why did GPT-2 do poorly on One Billion Word?
- That benchmark's sentences are shuffled, so there is no long-range context to use; the paper notes this "destructive pre-processing".
Idea · §3.5–3.8
Tasks it was never taught: promising, not yet good
The more surprising tests asked GPT-2 to do things that are not language modelling at all, using only a cleverly shaped prompt:
- Reading comprehension. Give it a story, a conversation about it, and end with "A:". It scored 55 on the CoQA test's F1 measure, matching or beating 3 of 4 earlier systems that had been trained on more than 127,000 labelled examples.
- Summarising. Paste a news article and add "TL;DR:" (internet slang for "too long, didn't read: here's the short version"). The summaries were weak by the standards of trained systems, but removing the "TL;DR:" hint made them clearly worse, showing the hint really does call up the task.
- Translation. Show a few "English sentence = French sentence" pairs, then an English sentence and "=". French to English reached 11.5 BLEU (a translation-quality score), far below the best systems but remarkable given that only about 10 MB of the training text was French.
- Trivia questions. It answered 4.1% of Natural Questions exactly right; on the 1% of questions it was most confident about, it was right 63.1% of the time.
The authors are frank that most of these results are far from useful. The point is the trend: performance on every one rose with model size, which suggested that much bigger models might cross into usefulness.
TakeawayZero-shot task performance was weak but real, and it climbed steadily with size: the first strong hint that scale alone unlocks new abilities.
Check yourself
- What does the "TL;DR:" experiment show?
- That the prompt wording selects the task: with the hint the summaries score higher than without it, so the model has learned what usually follows "TL;DR:" on the web.
Idea · §4 · Discussion
Did it just memorise? And should it be released?
A model that has read 8 million web pages might have read the test questions too. The authors checked by chopping texts into runs of 8 words and seeing how many runs from each test set also appear in WebText. Overlap was small, between 1% and 6% for most test sets (3.2% on average), and removing it changed scores only slightly. Several test sets overlapped more with their own training sets than with WebText.
The paper itself is a research report, but GPT-2 is also remembered for how it was released. Worried that fluent fake text could be misused, OpenAI at first published only the smallest model and released the full 1.5-billion-parameter version in stages over 2019. Whatever one thinks of that choice, it started a lasting debate about releasing powerful models.
TakeawayThe results were mostly genuine generalisation, not memorised test answers; and GPT-2's staged release made "should we publish this model?" a real question.
Abstract, restated
Language tasks such as question answering, machine translation, reading comprehension and summarisation are usually tackled with supervised learning on task-specific datasets. The authors show that language models begin to learn these tasks without explicit supervision when trained on WebText, a new dataset of millions of web pages. Conditioned on a document and questions, the model reaches 55 F1 on CoQA, matching or exceeding 3 of 4 baselines without their 127k+ training examples. Capacity is essential: performance improves log-linearly across tasks with model size. The largest model, GPT-2 (1.5B parameters), achieves state-of-the-art results on 7 of 8 language-modelling datasets zero-shot while still underfitting WebText, and its samples contain coherent paragraphs. OpenAI technical report, February 2019.
Section · §2
Multitask learning as conditional language modelling
A language model factorises the probability of a symbol sequence autoregressively:
A single-task system estimates p(output | input); a general system should estimate p(output | input, task). Earlier work conditioned on the task architecturally (task-specific encoders/decoders) or algorithmically (meta-learning). §2 observes that language offers a flexible way to specify the task, input and output all as one symbol sequence, citing McCann et al.'s decaNLP: a translation example becomes (translate to french, english text, french text) and reading comprehension becomes (answer the question, document, question, answer).
The central conjecture: since the supervised objective is the unsupervised objective evaluated on a subset of the sequence, the global minimum of the unsupervised objective is also a minimum of the supervised one. A sufficiently capable LM trained on sufficiently diverse text will therefore learn to infer and perform tasks demonstrated in natural language, in order to predict them better; the authors call this unsupervised multitask learning. Table 1 lists naturally occurring English–French translation pairs found in WebText to make the point concrete.
Check yourself
- Why should minimising the unsupervised LM loss also minimise a supervised loss?
- The supervised targets are just some of the tokens of the sequence, so the supervised loss is the LM loss restricted to those positions; a model optimal on all positions is optimal on that subset too (the paper's argument; in practice optimisation is the bottleneck, which is why scale matters).
Section · §2.1–2.2
WebText and byte-level BPE
Data. Outbound links from Reddit with at least 3 karma, about 45 million links, text extracted with the Dragnet and Newspaper extractors. The version used excludes links created after December 2017 and, after de-duplication and heuristic cleaning, has slightly over 8 million documents (40 GB). All Wikipedia documents were removed to reduce test-set overlap.
Input representation. The model should be able to assign probability to any string, so it works over bytes rather than pre-processed words. Plain byte-level LMs were not competitive with word-level ones at the time, so the authors use byte pair encoding (BPE) on UTF-8 bytes. BPE starts from a base vocabulary of 256 byte symbols and repeatedly merges the most frequent adjacent pair. Doing BPE over Unicode code points instead would need a base vocabulary of more than 130,000 symbols, against the 32,000 to 64,000 tokens typical of BPE vocabularies.
A naive byte-level BPE wastes vocabulary on variants such as dog. dog! dog?, because the greedy frequency heuristic happily merges across punctuation. GPT-2 blocks merges across character categories, with an exception for spaces, which keeps word pieces clean while preserving full coverage. The final vocabulary has 50,257 tokens. Run both variants below.
Check yourself
- Why not simply use a byte-level model with a 256-symbol vocabulary?
- It can represent anything, but at the time byte-level LMs underperformed word-level ones on large datasets, and sequences become several times longer. BPE keeps full coverage while producing word-like units.
Section · §2.3 · Table 2
Model changes from GPT-1
The architecture largely follows GPT-1 (a decoder-only transformer) with these modifications:
- Layer normalisation moved to the input of each sub-block (pre-activation, as in pre-activation residual networks), plus an extra layer norm after the final self-attention block.
- Residual-layer weights scaled at initialisation by 1/√N, with N the number of residual layers, to account for accumulation along the residual path.
- Vocabulary expanded to 50,257; context size increased from 512 to 1,024 tokens; batch size 512.
| Parameters | Layers | dmodel |
|---|---|---|
| 117M | 12 | 768 |
| 345M | 24 | 1024 |
| 762M | 36 | 1280 |
| 1542M | 48 | 1600 |
Table 2 of the paper. The smallest is equivalent to GPT-1, the second to the largest BERT. Learning rates were tuned per size for the best perplexity on a 5% held-out split of WebText; all sizes still underfit WebText, with held-out perplexity improving with size (Fig. 4). (OpenAI's code release later relabelled the two smaller models 124M and 355M after a recount of their parameters.)
Pre-norm and scaled residual initialisation became standard for deep transformers; later models such as LLaMA keep pre-norm but swap LayerNorm for RMSNorm.
Section · §3 · Tables 3–4
Zero-shot results
Language modelling (§3.1, Table 3). Because byte-level BPE needs no lossy pre-processing, GPT-2 can be evaluated on any benchmark; its log-probabilities are rescaled to the benchmark's own tokenisation, and de-tokenisers invert benchmark-specific artefacts (which alone gives 2.5 to 5 perplexity points). The 1542M model sets the state of the art on 7 of 8 datasets:
| Dataset (metric) | Prev. SOTA | 117M | 345M | 762M | 1542M |
|---|---|---|---|---|---|
| LAMBADA (PPL) | 99.8 | 35.13 | 15.60 | 10.87 | 8.63 |
| LAMBADA (ACC) | 59.23 | 45.99 | 55.48 | 60.12 | 63.24 |
| CBT-CN (ACC) | 85.7 | 87.65 | 92.35 | 93.45 | 93.30 |
| CBT-NE (ACC) | 82.3 | 83.4 | 87.1 | 88.0 | 89.05 |
| WikiText-2 (PPL) | 39.14 | 29.41 | 22.76 | 19.93 | 18.34 |
| PTB (PPL) | 46.54 | 65.85 | 47.33 | 40.31 | 35.76 |
| enwik8 (BPB) | 0.99 | 1.16 | 1.01 | 0.97 | 0.93 |
| text8 (BPC) | 1.08 | 1.17 | 1.06 | 1.02 | 0.98 |
| WikiText-103 (PPL) | 18.3 | 37.50 | 26.37 | 22.05 | 17.48 |
| 1BW (PPL) | 21.8 | 75.20 | 55.72 | 44.575 | 42.16 |
Raw LAMBADA accuracy rises from the previous LM result of 19% to 52.66%; the 63.24% in the table adds a stop-word filter that approximates the constraint that the answer must be the final word of a sentence. Winograd Schema (§3.4): 70.70%, a 7-point improvement over the previous best, on a test set of only 273 examples.
Task transfer.
- CoQA (§3.5): 55 F1 on the development set with greedy decoding, conditioned on document, dialogue history and a final "A:". The paper notes that many answers seem to rely on simple retrieval heuristics such as answering "who" questions with names in the document.
- CNN/Daily Mail summarisation (§3.6, Table 4): append "TL;DR:", sample 100 tokens with top-k random sampling (k = 2), keep the first 3 generated sentences. ROUGE-1/2/L = 29.34 / 8.27 / 26.58 (average 21.40); with no hint, average 15.03, a drop of 6.4 points. That only just beats picking 3 random sentences from the article (21.40 vs 20.98 average).
- WMT-14 translation (§3.7): conditioned on example pairs in the format "english sentence = french sentence". En→Fr 5 BLEU; Fr→En 11.5 BLEU, above several unsupervised MT baselines but well below the best unsupervised system at the time (33.5 BLEU). A byte-level language detector found only about 10 MB of French in WebText.
- Natural Questions (§3.8): 4.1% exact match, 5.3 times a trivial most-common-answer baseline (the smallest model does not beat that baseline's 1.0%), and 63.1% accuracy on its most confident 1% of questions, against 30 to 50% for open-domain QA systems with retrieval.
Section · §4 · Tables 5–6
Generalisation versus memorisation
To estimate train–test contamination, the authors built Bloom filters over the 8-grams of WebText training text (normalised to lower-case alphanumeric words separated by single spaces), with the false-positive rate bounded above by 1/108. Test sets of common LM benchmarks have 1 to 6% of their 8-grams in WebText train (average 3.2%), while those benchmarks' own training splits overlap their test sets more (average 5.9%); 1BW's test set overlaps its own training set by nearly 13.2%. For LAMBADA the average overlap is 1.2%, and removing examples with any overlap moves accuracy and perplexity only slightly. WebText's own held-out set also tracks its training set closely as models grow (Fig. 4), so the models underfit rather than memorise. The figure below applies the same 8-gram test to text you supply.
Check yourself
- Why is "test overlaps its own training split by 5.9%" an important comparison?
- It shows some overlap is normal even for curated splits, so WebText's 3.2% is not unusually contaminating; models trained on the benchmark's own train set already enjoy more leakage than GPT-2.
Section · §5–7 and lineage
Limitations and what came next
- Usefulness. The authors state that zero-shot performance on summarisation, translation and QA is still far from usable; the contribution is a baseline and a trend, with performance rising roughly log-linearly in capacity.
- Unclear ceiling. They note it is unknown whether fine-tuning on top would remove the remaining gap, and planned to fine-tune on decaNLP and GLUE.
- Prompt sensitivity. Task specification relies on hand-chosen formats ("TL;DR:", "A:", "english = french"), which later work showed can swing results substantially.
- Data. Reddit-karma filtering inherits Reddit's demographics and tastes; WebText was not released, prompting open replications such as OpenWebText.
Check yourself
- What distinguishes GPT-2's evaluation from GPT-3's "few-shot" one?
- GPT-2 mostly uses zero-shot task cues (a format or hint), with only translation given example pairs; GPT-3 systematically places K demonstrations in the context and compares zero-, one- and few-shot settings.
Reference
Glossary
- Zero-shot
- Evaluated on a task without any training examples from that task, here without any parameter updates either.
- Perplexity (PPL)
- exp of the average negative log-likelihood per token; lower is better.
- BPB / BPC
- Bits per byte / per character, the compression-style metric for enwik8 and text8.
- BPE
- Byte pair encoding: a vocabulary built by repeatedly merging the most frequent adjacent symbol pair.
- WebText
- 8M documents (40 GB) from Reddit outbound links with ≥ 3 karma, Wikipedia removed.
- Pre-norm
- Applying layer normalisation at the input of each residual sub-block rather than after the residual sum.
- Bloom filter
- A compact probabilistic set that can report false positives but never false negatives.
- Top-k sampling
- Sample the next token only from the k most likely tokens.