The paper in four sentences
Neural translation systems read the source sentence into one summary, a fixed-length list of numbers, and then wrote the translation from that summary alone.
A summary of fixed size has limited room, so translation quality dropped sharply as sentences got longer.
This paper keeps a separate description of every source word and, for each word it writes, lets the translator compute a set of weights saying which source words to focus on, then blends their descriptions accordingly.
The weights are learned with no one telling the model which words correspond, yet they line up with sensible word alignments, and long sentences stop being a problem.
Idea · §1–2
Translating through a keyhole
In 2014, a new style of translation system had just appeared: the encoder–decoder. One neural network, the encoder, reads the English sentence word by word and ends with a single summary: a list of, say, 1,000 numbers. A second network, the decoder, reads that summary and writes the French sentence word by word.
Everything the decoder knows about the source has to pass through that one summary, and it is the same size whether the sentence has 5 words or 50. The authors suspected this was the weak point. In their words, the network must "compress all the necessary information of a source sentence into a fixed-length vector", and they conjectured that this fixed-length vector is a bottleneck.
The toy below shows why. Each "sentence" is a list of random words. Version one squashes the whole list into a single fixed-size summary and then tries to read back each word. Version two keeps one note per word and simply looks up the right note.
TakeawayForcing a whole sentence through one fixed-size summary throws information away, and the loss grows with sentence length.
Check yourself
- Why would a bigger summary help but not solve the problem?
- It gives more room, so the breakdown happens at longer sentences, but any fixed size is eventually too small for long enough input.
Idea · §3.1
Look back while you write
The fix: don't throw away the per-word information. The encoder produces a description for every source word (the paper calls these annotations). Then, each time the decoder is about to write a word, it does three things:
- It scores every source word: how relevant is this word to what I am about to write, given what I have written so far?
- It turns the scores into weights that are positive and add up to 1 (a "softmax", which also exaggerates the differences so the best few dominate).
- It forms a context: a blend of the source-word descriptions, mixed according to the weights. Then it writes the next word using that context.
The scoring rule is itself a small neural network, trained along with everything else. Nobody tells it which English word matches which French word; it works that out because doing so helps it translate. The paper calls this learning to align. Everyone else soon called it attention.
TakeawayFor every word it writes, the decoder learns where to look in the source and blends what it finds there. That is attention.
Check yourself
- Why do the weights need to add up to 1?
- So that the context is an average of source-word descriptions: a blend in which focusing more on one word means focusing less on others.
- Who decides which source words match which output words?
- No one. The scoring network is trained only to make good translations; sensible alignments emerge because they help.
Idea · §3.2
Describing each word with both sides of the sentence
For the look-back to work, each source word's description should say something about the word in its setting, not just the word alone. "Bank" means different things in "river bank" and "bank loan".
So the encoder reads the sentence twice: once left to right and once right to left (a bidirectional recurrent network). Each word's description is the two readings stuck together: one summarising everything up to and including that word, the other everything from that word to the end. The description is mostly about the word itself and its near neighbours, because recurrent networks remember recent words best.
TakeawayEach source word gets its own description that reflects the words on both sides of it, ready for the decoder to consult.
Idea · §5 · Table 1 · Figs. 2–3
Long sentences stop being a problem
The authors trained English-to-French systems on a large collection of parallel text from the WMT 2014 shared task. Scores are given in BLEU, a standard measure of how many word sequences a machine translation shares with human reference translations (higher is better).
The old encoder–decoder, trained on sentences up to 50 words, scored 17.8 BLEU. The attention model with the same training data scored 26.8, and a version trained longer scored 28.5. More tellingly, the old system's quality fell steeply as sentences got longer, while the attention model's barely changed, even beyond 50 words (Fig. 2).
The best attention model still trailed Moses, a mature phrase-based translation system built over many years, on all sentences (28.5 vs 33.3 BLEU), but matched and slightly beat it when sentences with rare words were left out, even though Moses had also used an extra 418 million words of French text.
The learned weights, drawn as grids (Fig. 3), mostly run along the diagonal, as you'd expect for English and French, but bend where word order differs, such as adjectives that come after the noun in French.
TakeawayWith attention, translation quality no longer collapsed on long sentences, and the weights turned out to be readable word alignments.
Check yourself
- Why is the attention grid mostly diagonal for English to French?
- The two languages mostly keep the same word order, so the n-th French word usually corresponds to a word near the n-th English word.
Idea · §6–7 · What came next
From a helper to the whole machine
Here attention is an add-on to recurrent networks, which still read the sentence one word at a time. Within three years, the Transformer asked: what if attention is all you need? It dropped the recurrent parts entirely and let every word attend to every other word, in parallel. Today's large language models are built from that.
The paper's own limits were practical: the vocabulary was capped at 30,000 words, so rarer words became an "unknown" token, which the authors name as the main challenge left. And computing a weight for every source word at every output step costs time proportional to the product of the two lengths, which they judged fine for sentences but possibly limiting elsewhere.
TakeawayThis paper introduced learned soft attention as a fix for translation; it went on to become the core operation of modern language models.
Abstract, restated
Neural machine translation builds a single network, jointly tuned to maximise translation performance; existing models are encoder–decoders that encode a source sentence into a fixed-length vector. The authors conjecture that the fixed-length vector is a bottleneck and propose letting the model automatically (soft-)search for parts of the source sentence relevant to predicting each target word, without forming hard segments explicitly. The resulting system matches the English–French performance of the existing phrase-based state of the art, and qualitative analysis shows the soft alignments agree with intuition. ICLR 2015 (arXiv 1409.0473, Sept 2014).
Section · §2.1 · Eqs. 1–3
The RNN encoder–decoder baseline
Given source x = (x1, …, xTx), an encoder RNN computes hidden states and a summary
Typically q just takes the last state, c = hTx (Sutskever et al. 2014 used LSTMs; Cho et al. 2014 the gated unit later called the GRU). The single c is shared by every decoding step.
Section · §3.1 · Eqs. 4–6 · App. A
The decoder with an alignment model
The proposed model conditions each target word on a distinct context ci:
The alignment model a is a one-hidden-layer MLP (the "additive" or "concat" score), trained jointly by backpropagation; because α is a deterministic soft function, no latent alignment variable or sampling is needed. The paper interprets ci as an expected annotation, the expectation of hj under the alignment distribution αi·. Since a must be evaluated Tx × Ty times per sentence pair, Uahj does not depend on i and can be precomputed (App. A.1.2). Note the score uses the previous decoder state si−1: the model decides where to look before updating its state.
Check yourself
- Why must ci lie in the convex hull of the annotations?
- The αij are non-negative and sum to 1, so ci is a convex combination of the hj.
- What is gained by computing eij from si−1 instead of si?
- si itself depends on ci (Eq. 4), so scoring with it would be circular; using si−1 keeps the computation a single forward pass. Luong et al. (2015) later scored with the current state by reordering the computation.
Section · §3.2 · §4 · App. A–B
Encoder, data and training
Encoder. A bidirectional RNN; the annotation for word j is the concatenation hj = [h⃗j⊤; h⃖j⊤]⊤. Both directions and the decoder use gated hidden units (the GRU of Cho et al., 2014) with n = 1,000 units each; n′ = 1,000 for the alignment MLP. The decoder's initial state is s0 = tanh(Ws h⃖1). The output g uses a single maxout hidden layer followed by softmax over the target vocabulary.
Data (§4.1). WMT'14 English–French parallel corpora (Europarl, news commentary, UN, two crawled corpora), 850M words, reduced to 348M by the data-selection method of Axelrod et al. (2011). 30,000-word shortlists per language; everything else maps to [UNK]. Development: news-test-2012 + 2013. Test: news-test-2014, 3,003 sentences.
Training (App. B). Minibatch SGD with Adadelta (ε = 10−6, ρ = 0.95), minibatches of 80 sentences, gradient norm clipped to 1; about 5 days per model. Decoding with beam search. Two length regimes: models trained on sentences up to 30 words (RNNencdec-30, RNNsearch-30) and up to 50 words (-50).
Section · §5 · Table 1 · Figs. 2–3
Results
Length robustness (Fig. 2). BLEU against sentence length: both RNNencdec models degrade sharply as length grows, RNNsearch-30 degrades more gently, and RNNsearch-50 shows no deterioration even for sentences of 50 words or more. Appendix C shows long-sentence translations where RNNencdec drifts from the source meaning after about 30 words.
Alignments (§5.2.1, Fig. 3). Visualised α matrices are largely monotonic with non-trivial reorderings. In "European Economic Area" → "zone économique européenne", the model aligns "zone" to "Area", then steps back through the two adjectives. For "the man" → "l'homme", a hard alignment of "the" to "l'" is insufficient, since the article form depends on the following noun; soft alignment lets the model attend to both "the" and "man".
Check yourself
- RNNsearch-30 beats RNNencdec-50 on all sentences (21.5 vs 17.8). Why is that notable?
- The attention model trained only on sentences of up to 30 words outperforms a fixed-vector model trained on longer ones, consistent with the bottleneck being the fixed-length vector, not the amount of training data.
Section · §6–7 · Lineage
Limitations and what replaced what
- Rare words. The 30k shortlist and [UNK] handling is the main weakness the conclusion names. It was later addressed by subword segmentation (byte-pair encoding, Sennrich et al., 2016) and word pieces.
- Cost. Attention needs Tx·Ty score evaluations; the authors (§6.1) consider this acceptable for 15–40-word sentences but a possible limitation for other tasks. This quadratic cost, now n2 for self-attention, is what FlashAttention engineers around.
- Precursors. Graves (2013) used a learned soft window over the input for handwriting synthesis; the paper discusses it in §6.1 and notes its alignment is constrained to move monotonically, whereas here any source position can be chosen at each step.
- Score functions. Luong, Pham & Manning (2015) compared dot, general (s⊤W h) and concat (this paper's additive) scores, and global versus local attention. Google's GNMT (Wu et al., 2016) scaled attention-based RNN translation to production.
- Self-attention. Vaswani et al. (2017) removed recurrence, applied attention within a sequence as well as between encoder and decoder, used scaled dot-product scores and multiple heads. The additive score survives mostly in textbooks.
Reference
Glossary
- NMT
- Neural machine translation: one network trained end to end to map source to target sentences.
- Encoder–decoder
- One RNN reads the source, another generates the target conditioned on it.
- Annotation hj
- The bidirectional encoder state for source word j.
- Alignment model a
- MLP scoring how well source position j matches target position i.
- Soft alignment
- A probability distribution over source positions, rather than one chosen position.
- Context ci
- Weighted average of annotations used to emit target word i.
- BLEU
- Modified n-gram precision against reference translations with a brevity penalty.
- Maxout
- A unit outputting the maximum of several linear functions.
- Moses
- Open-source phrase-based statistical MT system, the conventional baseline.