The paper in four sentences
Big language models remember a lot of facts, but only implicitly, in their billions of internal numbers: you cannot easily see where an answer came from, correct it, or add new knowledge without retraining.
RAG (retrieval-augmented generation) pairs a text generator with a retriever that searches 21 million short Wikipedia passages and hands the best few to the generator.
The two parts are trained together, end to end, using only question–answer pairs: nobody tells the retriever which passage is "right"; it learns which passages help the generator answer.
RAG set new records on three open-domain question-answering benchmarks, wrote more factual and specific text than a generator alone, and changed its answers when given a newer copy of Wikipedia.
Idea · §1
Two kinds of memory
A language model such as GPT-2 learns facts during training and keeps them in its parameters, the millions or billions of numbers inside the network. The paper calls this parametric memory. It is powerful but opaque: when the model says something false (a "hallucination"), there is no source to check, and teaching it a new fact means more training.
The alternative is non-parametric memory: an external collection of documents the system can look things up in, like a library. Facts there are readable, citable and replaceable.
RAG combines the two. A retriever finds passages related to the input. A generator, a pre-trained model that writes text (here BART), then produces the answer while looking at both the question and the retrieved passages.
TakeawayRAG keeps the language skill in the model's weights but moves much of the factual knowledge into a searchable document collection.
Idea · §2.2 · Fig. 1
How the retriever finds passages
The retriever turns every passage into a list of numbers, a vector, that captures roughly what it is about. It does the same for the question. Passages whose vectors point in a similar direction to the question's get a high score, and the top few are passed on. Because all passage vectors are computed once in advance, searching millions of passages takes a fraction of a second.
The real system learns its vectors with neural networks. The toy below uses a much cruder version (it simply counts shared meaningful words), but the pipeline is the same: score every passage, convert scores into probabilities, keep the top k, and let the generator read them.
Check yourself
- Why can RAG search 21 million passages quickly?
- The passage vectors are computed once and stored in an index; at question time only the question needs encoding, and a fast nearest-neighbour search finds the highest-scoring passages.
Idea · §2.1 · §2.4 · Fig. 2
Learning what to retrieve, without being told
Training data is only question–answer pairs, with no note of which passage contains the answer. So how does the retriever improve?
RAG treats the retrieved passage as an unknown it averages over. For each candidate passage, the generator says how likely the correct answer is given that passage. Passages that make the correct answer more likely get their retrieval score pushed up; unhelpful ones get pushed down. Over many examples the retriever learns which kinds of passages actually help.
The paper offers two ways to do the averaging. RAG-Sequence picks one passage and writes the whole answer from it, then averages over passages. RAG-Token may lean on a different passage for each word. That matters when an answer combines facts from several places: the paper shows the generator drawing on one passage for "The Sun Also Rises" and another for "A Farewell to Arms" when writing about Hemingway (Fig. 2).
TakeawayThe retriever learns from the generator's success, not from labelled "correct passages", which makes RAG easy to train on any input–output task.
Check yourself
- What is the practical difference between RAG-Sequence and RAG-Token?
- RAG-Sequence commits to one passage per whole answer; RAG-Token can switch passages word by word, so it can combine facts from different documents in one answer.
Idea · §4.1–4.4 · Tables 1–4
More accurate, more specific, more factual
On four open-domain question benchmarks (questions with no document provided), RAG beat both a 11-billion-parameter model answering from memory (T5) and earlier "retrieve, then extract a span" systems on three of them, while having far fewer trainable parameters than T5-11B. Because it generates rather than copies, it could even answer correctly when no retrieved passage contained the exact answer: 11.8% of such Natural Questions cases, where a copying system would score zero.
The authors also had it write quiz-show (Jeopardy-style) clues. Human judges compared RAG's clues with those of the same generator without retrieval (BART): RAG was judged more factual in 42.7% of comparisons and BART in only 7.1%; for specificity it was 37.4% against 16.8%.
TakeawayAdding retrieval made a mid-sized generator more accurate than a far larger memory-only model, and made its writing measurably more factual.
Idea · §4.5
Update the knowledge by swapping the library
Because facts live in the index, you can change what RAG knows without retraining it. The authors built a test of 82 world leaders ("Who is the President of Peru?" and so on) whose holders changed between 2016 and 2018, and swapped a 2016 Wikipedia index for a 2018 one.
TakeawaySwap the documents and the answers change: knowledge in an index can be updated, inspected and cited in a way knowledge in weights cannot.
Check yourself
- What does the 70% versus 4% result show?
- That RAG's answers really come from the index, not from memorised facts: with the old index it gives the old leader, and switching to a newer index updates its answers without any training.
Abstract, restated
Large pre-trained LMs store factual knowledge in their parameters, but their ability to access and precisely manipulate it is limited, so they lag task-specific architectures on knowledge-intensive tasks; provenance and knowledge updates are open problems. The authors propose a general fine-tuning recipe for retrieval-augmented generation (RAG), combining a pre-trained seq2seq model as parametric memory with a dense vector index of Wikipedia, accessed by a pre-trained neural retriever, as non-parametric memory. Two formulations condition on the same retrieved passages for a whole sequence or per token. Fine-tuned and evaluated on a wide range of knowledge-intensive tasks, RAG sets the state of the art on three open-domain QA tasks, beating parametric seq2seq models and task-specific retrieve-and-extract architectures, and generates more specific, diverse and factual language than a parametric-only baseline. Lewis et al., NeurIPS 2020.
Section · §2.1
Two marginalisations
Input x, retrieved text document z, target y. A retriever pη(z | x) returns a top-k truncated distribution over passages; a generator pθ(yi | x, z, y1:i−1) produces tokens. The latent document is marginalised in one of two ways:
RAG-Sequence is a mixture of k complete sequence models; RAG-Token is a single autoregressive model whose next-token distribution is a mixture, so the effective document can change at each step. For classification (e.g. FEVER), the target is a single token and the two coincide. The calculator below evaluates both formulas on a two-document, two-slot example.
Check yourself
- In the calculator, why does RAG-Sequence give "S, F" a low probability even when both passages are retrieved with equal weight?
- Under RAG-Sequence each whole answer is generated from one passage; neither passage alone strongly supports both titles, so every mixture component assigns the combination a low probability. RAG-Token re-mixes the passages for slot 2.
Section · §2.2–2.5
Retriever, generator, training, decoding
Retriever (DPR). A bi-encoder, pη(z|x) ∝ exp(d(z)⊤q(x)), with d(z) = BERTd(z) and q(x) = BERTq(x), both BERT-base. Top-k retrieval is maximum inner product search, approximated in sub-linear time with FAISS and an HNSW index. The retriever is initialised from DPR trained on TriviaQA and Natural Questions. The index (the "non-parametric memory") is the December 2018 Wikipedia dump split into disjoint 100-word chunks, 21M documents.
Generator. BART-large (400M parameters); input x and passage z are simply concatenated.
Training (§2.4). Minimise Σj −log p(yj|xj) with Adam, with no supervision on which document to retrieve. Updating the document encoder would require periodically re-encoding and re-indexing 21M passages (as REALM does); the authors found this unnecessary, so BERTd and the index stay fixed and only BERTq and BART are fine-tuned. k ∈ {5, 10} in training; test-time k is chosen on dev data (e.g. 15 for RAG-Token QA, 50 for RAG-Sequence QA).
Decoding (§2.5). RAG-Token is a standard autoregressive model with transition p′θ(yi|x, y1:i−1) = Σz pη(z|x) pθ(yi|x, z, y1:i−1), so ordinary beam search applies. RAG-Sequence's likelihood does not factor per token, so beam search runs per document and hypotheses are scored by summing over documents. "Thorough Decoding" runs extra forward passes for hypotheses missing from some document's beam; "Fast Decoding" approximates their probability as zero.
Section · §3–4 · Tables 1–2, 6
Results and ablations
- Open-domain QA (Table 1). New state of the art on NQ, WQ and CT, and on the T5-comparable TriviaQA split. RAG-Sequence: NQ 44.5, TQA 56.8 / 68.0, WQ 45.2, CT 52.2.
- Generation (Table 2). MS-MARCO: RAG-Sequence beats BART by 2.6 BLEU-1 and 2.6 ROUGE-L (44.2 / 40.8 vs 41.6 / 38.2). Jeopardy question generation: RAG-Token Q-BLEU-1 22.2 vs BART 19.7.
- FEVER (Table 2). RAG-Token 72.5 (3-way) and 89.5 (2-way) label accuracy, within 4.3% of pipeline systems that use retrieval supervision.
- Human evaluation (Table 4). Over 452 pairs of Jeopardy generations: factuality, RAG better 42.7% vs BART better 7.1%; specificity, 37.4% vs 16.8%.
- Diversity (Table 5). Distinct-trigram ratio on Jeopardy: BART 32.4%, RAG-Token 46.8%, RAG-Sequence 53.8% (gold 90.0%).
- Ablations (Table 6). Learned retrieval matters: freezing the retriever drops RAG-Token's NQ from 43.5 to 37.8; BM25 gives 29.7. BM25 does best on FEVER, where claims are entity-centric and word overlap is a strong signal.
- More documents (Fig. 3). RAG-Sequence's NQ score improves monotonically with retrieved documents at test time; RAG-Token peaks at 10.
Section · §4.5 · Broader impact · Lineage
Hot-swapping, limitations, and what came next
Index hot-swapping. With a December 2016 DrQA Wikipedia index and the 2018 index, on 82 world leaders queried as "Who is {position}?": 70% with the 2016 index for 2016 leaders, 68% with the 2018 index for 2018 leaders, and 12% and 4% when mismatched (2018 index on 2016 leaders, 2016 index on 2018 leaders). Knowledge can be updated without retraining.
Parametric and non-parametric together. In the Hemingway example the retrieved document's posterior is high when a title begins, then falls once the first tokens are produced, because BART can complete the title from its own parameters: the index "cues" knowledge the generator already holds.
Limitations. Wikipedia is neither complete nor free of bias; retrieval is limited to 100-word passages and a fixed number k; the document encoder is never updated; on some tasks (e.g. story generation in Appendix H) retrieval "collapsed", returning the same documents for every input, after which RAG behaves like BART. The broader-impact section notes misuse risks similar to GPT-2's.
Check yourself
- Why not fine-tune the document encoder too?
- Every update would invalidate the stored passage vectors, requiring re-encoding 21M passages and rebuilding the index; the authors found fine-tuning only the query encoder sufficient.
- Why might BM25 beat dense retrieval on FEVER?
- FEVER claims name specific entities; exact word overlap with an entity's article is a strong retrieval signal there.
Reference
Glossary
- Parametric memory
- Knowledge stored in model weights (here BART).
- Non-parametric memory
- An external, swappable store (here 21M Wikipedia passages).
- DPR
- Dense Passage Retriever: two BERT encoders scoring passages by inner product.
- MIPS
- Maximum inner product search; approximate versions (FAISS, HNSW) run in sub-linear time.
- Marginalise
- Sum a probability over the possible values of a hidden variable (the retrieved document).
- Exact match (EM)
- Fraction of answers that equal a reference answer after normalisation.
- Open-domain QA
- Question answering with no document given; the system must find or remember the knowledge.
- Hot-swapping
- Replacing the document index at test time without retraining.