The paper in four sentences
The standard way to get a language model to do a task was to fine-tune it: train it further on thousands of labelled examples of that task.
GPT-3 is a 175-billion-parameter language model trained only to predict the next word, on about 300 billion tokens (word pieces) of text from the web, books and Wikipedia.
Instead of fine-tuning, the authors simply wrote a task description and a few solved examples into the prompt, then asked for one more; the model's weights never change.
This "few-shot" prompting worked surprisingly well, sometimes matching fine-tuned systems, and it got much better with model size, though GPT-3 still failed on some kinds of reasoning and comparison tasks.
Idea · §1 · Fig. 2.1
Zero, one or a few examples in the prompt
A language model continues text: it reads a prompt and predicts what comes next, one small piece (a token, roughly a word fragment) at a time.
The paper tests three ways of asking it to do a task, with no further training in any of them:
- Zero-shot: just a description, such as "Translate English to French: cheese =>".
- One-shot: the description plus one solved example, such as "sea otter => loutre de mer", before the question.
- Few-shot: the description plus several solved examples, usually 10 to 100, as many as fit in the model's 2,048-token window.
The model has to work out from the examples what you want, and then do it. The authors call this in-context learning: "learning" that happens inside a single prompt and is forgotten the moment the prompt ends.
The toy below acts out what this asks of the model. It knows eight possible "rules" for turning a word into something else, and keeps a score of how believable each rule is given the examples so far. Add examples and watch it narrow down.
TakeawayFew-shot prompting replaces "retrain the model on the task" with "show it a few examples of the task", and the model's weights never change.
Check yourself
- What is the difference between few-shot prompting and fine-tuning?
- Fine-tuning changes the model's weights by training on many labelled examples. Few-shot prompting puts a few examples in the input text; the weights are untouched and nothing is remembered after the prompt.
- Why are there at most about 10 to 100 examples in the few-shot setting?
- They have to fit, along with the question, inside the model's context window of 2,048 tokens.
Idea · §2.1–2.3 · Tables 2.1–2.2
Same recipe, much bigger
GPT-3 is not a new design. It is the GPT-2 transformer with one change to how attention is computed, made much larger. The authors trained eight sizes, from 125 million to 175 billion parameters (the adjustable numbers that training tunes). The largest is more than 100 times GPT-2.
All eight read the same 300 billion tokens. The text came mostly from a filtered crawl of the web (Common Crawl), shrunk from 45 TB of compressed text to 570 GB by keeping pages that resembled known good sources and removing near-duplicates. Smaller, higher-quality sources such as Wikipedia and books were shown to the model more often than their size alone would suggest, so it read Wikipedia more than three times over but most of the web crawl less than once.
Training the largest model took roughly 3 × 1023 arithmetic operations, on a cluster of graphics processors provided by Microsoft. That cost is why the authors could not simply retrain when they later found a mistake in their data cleaning (Idea 5).
TakeawayGPT-3's novelty is scale and the way it is used, not a new architecture.
Idea · §3 · Fig. 1.2 · Tables 3.2–3.10
What it could and could not do
The authors tested GPT-3 on more than two dozen benchmarks, plus some new tasks they invented to make sure it could not have seen the answers. Three patterns stand out. Examples help, more examples usually help more, and the benefit of examples grows with model size: big models are much better at learning from the prompt (Fig. 1.2 shows this on a word-unscrambling task).
Choose a view below. All numbers are from the paper, for the 175-billion-parameter model.
The arithmetic results were striking in 2020: 100% on two-digit addition, from examples alone. But accuracy falls sharply with more digits. On word puzzles it can undo some scrambling but almost never reverse the letters of a word (0.44%), probably because it sees text as chunks rather than individual letters. In-context learning is real, but it inherits the model's blind spots.
Weak spots the authors report include tasks that compare two sentences (such as deciding whether one follows from the other, where GPT-3 was barely above chance on the hardest test, ANLI) and some reading-comprehension benchmarks.
TakeawayFew-shot GPT-3 was competitive on some benchmarks and poor on others; the consistent pattern was that larger models learn more from examples in the prompt.
Check yourself
- On LAMBADA, one-shot was worse than zero-shot (72.5% vs 76.2%), yet few-shot was best (86.4%). What might explain this?
- The paper suggests the models need several examples to recognise the unusual fill-in-the-blank format; with only one, the format seems to confuse more than help.
- Why might reversing a word's letters be so hard for GPT-3?
- Its input is split into multi-letter tokens, so it does not directly see individual letters.
Idea · §3.9.4 · §6
Fake news that people can't spot
The authors asked GPT-3 to write short news articles from a title and subtitle, then asked people to judge whether each article was written by a human or a machine. For the largest model, people were right only about 52% of the time, barely better than tossing a coin. For a deliberately weak control model they were right 86% of the time.
The paper spends a whole section on what this could mean: misinformation, spam, fraudulent academic essays. It also measured biases the model absorbed from the web, for example in how it associates genders, races and religions with particular words, and the energy cost of training.
TakeawayGPT-3's text was fluent enough that people could not reliably tell it from human writing, which the authors flagged as a risk alongside learned social biases.
Idea · §4–5
Caveats: leaked tests and an unclear kind of learning
Leaked test questions. With so much web text, some benchmark questions were bound to be in the training data. The authors tried to remove them, but a bug meant some overlaps were missed, and the model was too expensive to retrain. They checked each benchmark afterwards on "clean" questions only; for most, scores barely changed, but they flagged a few results (such as PIQA and the Winograd test) as possibly inflated.
Learning, or recognising? The authors admit they cannot tell whether few-shot prompting teaches GPT-3 a genuinely new task, or just helps it recognise a task it already met during training. That question shaped years of later research.
TakeawayGPT-3 proved prompting works at scale, but left open how much it truly learns from the prompt, and some of its scores carry contamination caveats.
Abstract, restated
Pre-training then fine-tuning achieves strong NLP results, but still needs task-specific datasets of thousands of examples; humans can generally do a new language task from a few examples or instructions. The authors show that scaling up language models greatly improves task-agnostic few-shot performance, sometimes reaching competitiveness with prior fine-tuned state of the art. GPT-3, an autoregressive LM with 175B parameters (10× any previous non-sparse LM), is applied without gradient updates or fine-tuning, with tasks and demonstrations specified purely as text. It does well on translation, QA and cloze tasks and on tasks needing on-the-fly reasoning (unscrambling words, novel words in a sentence, 3-digit arithmetic), but struggles on some datasets and faces methodological issues from training on large web corpora. Humans have difficulty distinguishing its news articles from human-written ones. Brown et al., NeurIPS 2020.
Section · §2 · Fig. 2.1
Evaluation settings and scoring
Four points on a spectrum of task-specific data: fine-tuning (not used for GPT-3 in this paper), few-shot (K demonstrations of context and completion plus one final context, K typically 10 to 100, bounded by nctx = 2048), one-shot (K = 1) and zero-shot (a natural-language instruction only). No gradient updates happen in any of the three prompting settings. Demonstrations are drawn at random from the task's training set (from the development set for LAMBADA and StoryCloze, which have no training set), separated by one or two newlines depending on the task.
Scoring (§2.4): for multiple choice, compare the LM likelihood of each completion, normalised per token; for ARC, OpenBookQA and RACE, normalising by the unconditional probability of each completion, P(completion | context) / P(completion | answer context), helped further. Free-form answers use beam search with width 4 and length penalty α = 0.6. SuperGLUE used 32 random training examples as the few-shot context.
The paper frames in-context learning as the inner loop of meta-learning: the outer loop is pre-training by SGD over many documents, some of which happen to contain repeated sub-tasks, and the inner loop is the forward pass over a prompt.
Check yourself
- Why normalise multiple-choice scores per token?
- Raw sequence log-likelihood penalises longer answers simply for having more tokens; per-token normalisation removes most of that length bias.
Section · §2.1–2.3 · Table 2.1
Models, data and compute
Architecture as GPT-2 (pre-norm, modified initialisation, reversible tokenisation), except for alternating dense and locally banded sparse attention patterns in the layers, similar to the Sparse Transformer. Feed-forward width is 4dmodel. Eight models were trained, all on 300B tokens:
Data (§2.2, Table 2.2). Common Crawl filtered by similarity to high-quality reference corpora (45 TB compressed before filtering, 570 GB after), fuzzy-deduplicated at document level within and across datasets, and mixed with curated corpora. Sampling is by quality, not size:
| Dataset | Tokens | Weight in mix | Epochs in 300B |
|---|---|---|---|
| Common Crawl (filtered) | 410B | 60% | 0.44 |
| WebText2 | 19B | 22% | 2.9 |
| Books1 | 12B | 8% | 1.9 |
| Books2 | 55B | 8% | 0.43 |
| Wikipedia | 3B | 3% | 3.4 |
Training (§2.3). Larger models use larger batches (0.5M tokens for 125M up to 3.2M for 175B) and smaller learning rates (6.0 × 10−4 down to 0.6 × 10−4); model parallelism within matrix multiplies and across layers; V100 GPUs on a Microsoft cluster. The 6ND estimate gives about 3.1 × 1023 FLOPs for the 175B run.
Section · §3 · Tables 3.2–3.11
Results in brief
| Task | Zero | One | Few | Reference |
|---|---|---|---|---|
| LAMBADA acc. (Table 3.2) | 76.2 | 72.5 | 86.4 | prior SOTA 68.0 |
| TriviaQA acc., closed book (Table 3.3) | 64.3 | 68.0 | 71.2 | no retrieval |
| SuperGLUE avg. (Table 3.8) | – | – | 71.8 | fine-tuned SOTA 89.0 |
| WMT Fr→En / De→En BLEU (Table 3.4) | – | – | 39.2 / 40.6 | En→Fr 32.6, En→De 29.7 |
| SAT analogies (§3.9.3) | – | – | 65.2 | college applicants 57 |
LAMBADA's few-shot gain comes from casting it as fill-in-the-blank, which tells the model that exactly one word is wanted; one-shot is always worse than zero-shot here, which the authors attribute to models needing several examples to recognise the pattern. Translation is markedly asymmetric: translating into English (the training data is about 93% English by word count) is much stronger than out of it. The synthetic tasks are designed to probe on-the-fly adaptation:
Spot-checking memorisation of arithmetic. The authors searched the training data for the 3-digit problems in the form "<num1> + <num2> =" and "<num1> plus <num2>", and found only 17 matches (0.8%) among 2,000 addition problems and 2 (0.1%) among 2,000 subtraction problems, suggesting the model computes rather than looks up; errors such as failing to carry a 1 also point that way.
Human detection of generated news (§3.9.4, Table 3.11). Mean human accuracy at identifying ~200-word articles from the 175B model was 52% (95% CI 49 to 54%), versus 86% for a deliberately poor control model, falling smoothly with model size.
Section · §4–6
Contamination, limitations and broader impacts
Contamination (§4). A filtering bug left some benchmark overlaps in the training data and retraining was too costly. For each benchmark the authors built a "clean" subset by removing any example with a 13-gram overlap with training data (shorter for very short examples) and compared scores. The method is deliberately conservative, so it over-flags; most benchmarks showed negligible change. PIQA and Winograd showed signs of contamination and are marked with asterisks; for reading-comprehension sets over 90% of examples were flagged, but inspection found the overlap was mostly source passages rather than questions and answers.
Limitations (§5). Text-generation weaknesses (repetition, loss of coherence over long passages, self-contradiction); trouble with "common sense physics"; architectural limits from being unidirectional, which may explain weak results on tasks comparing two sentences (WiC, ANLI) and some reading comprehension; a pre-training objective that weights every token equally and has no grounding in other modalities; poor sample efficiency during pre-training compared with humans; uncertainty whether few-shot learning learns tasks "from scratch" or recognises ones seen in training; expensive inference (distillation is suggested); and the usual lack of interpretability and calibration.
Broader impacts (§6). Misuse (misinformation, spam, phishing, fraudulent essays), bias probes on gender, race and religion (occupation–gender associations, sentiment and co-occurrence of descriptive words), and energy use, with the note that a trained model is cheap to use relative to training.
Section · Lineage
How the ideas were refined
- Compute-optimal scaling. GPT-3 followed Kaplan et al. (2020), who advised growing parameters faster than data. Hoffmann et al. (2022) revised this to roughly equal scaling of both, about 20 tokens per parameter; at 1.7 tokens per parameter GPT-3 175B was heavily undertrained. LLaMA went further, training small models far past that point for cheaper inference.
- Following instructions. InstructGPT fine-tuned GPT-3 with human feedback; people preferred a 1.3B InstructGPT's outputs to 175B GPT-3's. The weak zero-shot behaviour in this paper is largely a property of the raw pre-trained model.
- Reasoning. Chain-of-thought demonstrations (Wei et al., 2022) greatly improved the multi-step arithmetic and reasoning tasks on which few-shot GPT-3 was weak.
- What in-context learning is. Xie et al. (2021) modelled it as implicit Bayesian inference over a latent task; Min et al. (2022) found that randomising the labels in demonstrations often hurts surprisingly little, implying demonstrations mainly convey format, label space and input distribution, consistent with the "recognising" side of the paper's open question. Later analyses (for example induction heads, Olsson et al. 2022) looked for the mechanism inside transformers.
- Emergence. Abrupt jumps with scale, visible here in arithmetic, were catalogued as "emergent abilities" (Wei et al., 2022); Schaeffer et al. (2023) argued many such jumps reflect discontinuous metrics like exact match.
Check yourself
- How many tokens per parameter did GPT-3 175B see, and why does it matter?
- 300B / 175B ≈ 1.7. Chinchilla's analysis puts the compute-optimal ratio near 20, so the same compute would have been better spent on a smaller model trained on more data.
- What does Min et al.'s random-label result suggest about few-shot prompting?
- That much of the benefit comes from showing the format and distribution of inputs and outputs rather than teaching the input–label mapping itself.
Reference
Glossary
- In-context learning
- Adapting to a task from demonstrations or instructions in the prompt, without weight updates.
- Zero/one/few-shot
- K = 0, 1 or typically 10 to 100 demonstrations in the context.
- nctx
- Context window, 2,048 tokens for all GPT-3 models.
- Fine-tuning
- Further gradient training on a task's labelled data.
- Contamination
- Benchmark test items present in the pre-training data.
- Sparse attention
- Attention restricted to a pattern (here locally banded) to cut cost; GPT-3 alternates it with dense layers.
- 6ND
- Standard estimate of transformer training FLOPs: about 6 operations per parameter per token.