The paper in four sentences
Recurrent networks read a sentence one word after another, so they are slow to train and struggle to connect words that are far apart.
Attention, a way for each word to pull in information from whichever other words are relevant, had been used as an add-on to them.
The Transformer is built from attention alone: in every layer, every word looks at every other word directly and in parallel, with some extra information added to say where each word sits in the sentence.
It set new records on English–German and English–French translation at a fraction of the training cost, and its design (with a few later tweaks) is what GPT, BERT and today's chatbots are built on.
Idea · §1–2
The problem with reading one word at a time
Before this paper, the standard way to process a sentence with a neural network was a recurrent network (such as an LSTM). It reads word 1, updates a running memory, reads word 2, updates the memory again, and so on. Word 50 cannot be processed until words 1 to 49 are done.
That causes two problems. First, speed: modern chips (GPUs) are fast because they do thousands of calculations at the same time, and a strict one-after-another chain wastes that. Second, distance: information about word 1 has to survive 49 memory updates to influence word 50, and it tends to fade on the way.
A fix already existed in part. Bahdanau and colleagues had let a translator "attend" back to any source word directly. But the recurrent chain was still there underneath. The Transformer's authors asked whether attention could do the whole job.
TakeawayRecurrent networks are slow because each step waits for the last, and they lose track of far-apart words. Attention offered a direct path between any two words.
Idea · §3.2.1
Attention as a soft search
Here is how one word gathers information from the others. Each word produces three lists of numbers:
- a query: what am I looking for?
- a key: what do I contain, as a label others can match against?
- a value: what information do I hand over if someone picks me?
To update a word, compare its query with every word's key. The comparison is a dot product: multiply matching entries and add them up, which gives a big number when the two lists point the same way. Turn those scores into weights that are positive and add up to 1 (a "softmax"), and take the weighted blend of the values. All of these lists are produced by the network itself, and it learns what to put in them during training.
Below, the word "it" is deciding what it refers to. Change the last word of the sentence and its query changes.
TakeawayEach word asks a question (query), every word offers a label (key), and the answer is a blend of the words' contents (values) weighted by how well labels match the question.
Check yourself
- Why use a weighted blend instead of picking the single best match?
- A blend can combine information from several words, and, unlike a hard pick, it changes smoothly when the numbers change, which is what training by gradient descent needs.
- What would happen if every key were identical?
- All scores would be equal, so every word gets the same weight and the result is a plain average of all values.
Idea · §4 · Table 1
Everything at once
In a Transformer layer, every word does this search at the same time. Nothing waits for anything else, so a GPU can compute the whole layer in one go. And any word can reach any other in a single step, however far apart they are.
The price is that the number of comparisons grows with the square of the sentence length: 10 words means 100 query–key comparisons, 1,000 words means a million. For the sentence lengths in translation that was cheap; for long documents it became the main cost, which later work like FlashAttention attacks.
TakeawaySelf-attention swaps a long chain of steps for one big parallel step, at the cost of comparing every pair of words.
Check yourself
- A sentence doubles in length. How do the sequential steps change for an RNN and for self-attention? And the comparisons?
- RNN steps double; self-attention stays at one step per layer. But the self-attention comparisons quadruple.
Idea · §3.2.2 · §3.5
Many heads, and a sense of order
Many heads. One search can only look for one kind of thing at a time. So each layer runs several searches side by side (8 in the base model, 16 in the big one), called heads, each with its own queries, keys and values. One head might track which noun a pronoun refers to, another the verb that goes with a subject. Their results are stuck together and mixed.
Order. Attention on its own ignores word order: "dog bites man" and "man bites dog" would look the same, because the search compares contents, not positions. So before the first layer, each word gets a positional encoding added to it: a pattern of wave values (sines and cosines of different wavelengths) unique to its position, like a barcode. The network learns to read these barcodes.
Each layer also has a small ordinary network applied to every word separately (the "feed-forward" part), plus shortcuts that add a layer's input back to its output and a normalisation step that keeps numbers in a sensible range. Six such layers make the encoder, which reads the source sentence; six more make the decoder, which writes the translation one word at a time and is blocked from peeking at words it hasn't written yet.
TakeawaySeveral attention heads look for different relationships in parallel, and added position signals give the model the word order that attention alone ignores.
Check yourself
- Why does the decoder need to be blocked from looking ahead?
- At use time it generates words one by one, so future words don't exist yet. During training it must predict each word without seeing it, or it would just copy the answer.
Idea · §6 · Tables 2–3 · What came next
Better, cheaper, and everywhere
On the standard English-to-German translation test (WMT 2014) the big Transformer scored 28.4 BLEU, more than 2 points above the best previous results, including ensembles of several models. On English-to-French it scored 41.8, a new single-model record. The small version trained in 12 hours on one machine with 8 GPUs; the big one in 3.5 days, a small fraction of the computing cost of the previous best systems.
The paper also showed the design wasn't specific to translation: with little tuning, it did well at parsing English sentences into grammatical trees.
Within two years, GPT took just the decoder half and BERT took just the encoder half, and pre-trained them on huge amounts of text. Later it spread to images (ViT), audio, proteins and robotics.
TakeawayAttention-only models were more accurate and much cheaper to train, and with a handful of later tweaks they became the standard architecture of modern AI.
Abstract, restated
Dominant sequence transduction models are complex recurrent or convolutional encoder–decoders, the best connected through attention. The Transformer is based solely on attention, dispensing with recurrence and convolutions. On WMT 2014 English–German it reaches 28.4 BLEU, over 2 BLEU above existing best results including ensembles; on WMT 2014 English–French it sets a single-model state of the art of 41.8 BLEU after 3.5 days on eight GPUs, a small fraction of the training cost of the best models in the literature. It also generalises to English constituency parsing. NeurIPS 2017 (arXiv 1706.03762).
Section · §3.1 · Fig. 1
Architecture
Encoder: N = 6 identical layers, each with a multi-head self-attention sub-layer and a position-wise feed-forward sub-layer. Decoder: 6 layers, each with masked self-attention, encoder–decoder attention (queries from the decoder, keys and values from the encoder output) and a feed-forward sub-layer. Every sub-layer is wrapped as
(post-norm), with all sub-layers and embeddings of width dmodel = 512 so the residuals type-check. Decoder self-attention is masked by setting scores of illegal (future) connections to −∞ before the softmax; together with offsetting the output embeddings by one position, this makes prediction at position i depend only on outputs before i.
Embeddings (§3.4). Input embedding, output embedding and pre-softmax projection share one weight matrix; embedding weights are multiplied by √dmodel.
Section · §3.2 · Eq. 1 · footnote 4
Scaled dot-product and multi-head attention
Dot-product attention is chosen over additive attention (Bahdanau et al.) because it is a pair of matrix multiplications and therefore much faster and more memory-efficient in practice. The scale matters: if the components of q and k are independent with mean 0 and variance 1, then q·k = Σi=1dk qiki has mean 0 and variance dk. Large logits push the softmax into saturated regions where its gradient is tiny; dividing by √dk restores unit variance. The authors note additive attention outperforms unscaled dot-product attention for larger dk, which motivates the scaling.
With WiQ, WiK ∈ ℝdmodel×dk, WiV ∈ ℝdmodel×dv, WO ∈ ℝhdv×dmodel, h = 8 and dk = dv = dmodel/h = 64, the total cost matches single-head attention at full width. The paper's rationale: a single head averages, which inhibits attending to information from different representation subspaces at different positions. Ablation (Table 3, row A, newstest2013 En–De dev BLEU): 1 head 24.9, 4 heads 25.5, 8 heads 25.8, 16 heads 25.8, 32 heads 25.4. Reducing dk alone (row B) to 16 or 32 hurts (25.1, 25.4).
Check yourself
- Why doesn't additive attention need the 1/√dk factor in the same way?
- Its score passes through a tanh and a learned vector va, so its magnitude is bounded by ‖va‖ rather than growing with dimension like a raw sum of dk products.
- Given the mean-0, variance-1 assumption, what is the standard deviation of an unscaled logit at dk = 64?
- √64 = 8, so differences of tens between logits are common and exp makes one weight dominate.
Section · §3.3 · §3.5 · Eq. 2
Feed-forward layers and sinusoidal positions
Applied identically at every position (equivalently two kernel-size-1 convolutions), with inner width dff = 2048. Positions are injected by adding to the embeddings
Wavelengths form a geometric progression from 2π to 10000·2π. The authors chose sinusoids because for any fixed offset k, PEpos+k is a linear function of PEpos (each sin/cos pair is rotated by an angle proportional to k), hypothesising this would make relative positions easy to learn, and because it might extrapolate to longer sequences. Learned position embeddings gave nearly identical results (Table 3, row E: 25.7 vs 25.8).
Section · §4–6 · Tables 1–4 · Eq. 3
Why self-attention, training, and results
| Layer type (Table 1) | Complexity / layer | Sequential ops | Max path length |
|---|---|---|---|
| Self-attention | O(n²·d) | O(1) | O(1) |
| Recurrent | O(n·d²) | O(n) | O(n) |
| Convolutional | O(k·n·d²) | O(1) | O(logk n) |
| Self-attention (restricted, r) | O(r·n·d) | O(1) | O(n/r) |
Self-attention is cheaper than recurrence per layer when n < d, true for sentence-level translation with word-piece vocabularies, and its O(1) path length makes long-range dependencies easier to learn.
Training (§5). WMT 2014 En–De: about 4.5M sentence pairs, byte-pair encoding with a shared source–target vocabulary of about 37,000 tokens. En–Fr: 36M sentences, 32,000 word-piece vocabulary. Batches of about 25,000 source and 25,000 target tokens. One machine with 8 NVIDIA P100 GPUs: base model 100,000 steps at about 0.4 s/step (12 hours); big model 300,000 steps at 1.0 s/step (3.5 days). Adam with β1 = 0.9, β2 = 0.98, ε = 10−9, and
Regularisation: residual dropout Pdrop = 0.1 (base) on sub-layer outputs and on embedding + PE sums; label smoothing εls = 0.1, which hurts perplexity but improves BLEU.
| Model (Table 2) | EN–DE BLEU | EN–FR BLEU | Train FLOPs (EN–DE) |
|---|---|---|---|
| ByteNet | 23.75 | – | – |
| GNMT + RL | 24.6 | 39.92 | 2.3·10¹⁹ |
| ConvS2S | 25.16 | 40.46 | 9.6·10¹⁸ |
| MoE | 26.03 | 40.56 | 2.0·10¹⁹ |
| ConvS2S ensemble | 26.36 | 41.29 | 7.7·10¹⁹ |
| Transformer (base, ~65M params) | 27.3 | 38.1 | 3.3·10¹⁸ |
| Transformer (big, ~213M params) | 28.4 | 41.8 | 2.3·10¹⁹ |
The arXiv text of §6.1 gives 41.0 for big En–Fr while Table 2 and the abstract give 41.8; the paper is internally inconsistent here. Ablations (Table 3) also show bigger models help (dmodel 1024: 26.0; dff 4096: 26.2) and dropout matters (Pdrop = 0: 24.6).
Constituency parsing (§6.3, Table 4). A 4-layer Transformer with dmodel = 1024 on the WSJ portion of the Penn Treebank (about 40K sentences) reaches 91.3 F1 trained on WSJ only and 92.7 semi-supervised, better than all previously reported models except the Recurrent Neural Network Grammar.
Section · §7 · Limitations and lineage
What modern models changed
- Pre-norm. Post-norm (LayerNorm after the residual add) makes deep stacks hard to train without warm-up. Moving the norm inside the residual branch, x + Sublayer(LN(x)), gives better-behaved gradients (Xiong et al., 2020) and is used by GPT-2 onwards.
- RMSNorm. Zhang & Sennrich (2019) drop LayerNorm's mean-centring and bias; standard in LLaMA-style models.
- Positions. Additive sinusoids were replaced first by learned absolute embeddings (GPT, BERT), then by relative schemes; RoPE (2021) rotates queries and keys so the score depends on the offset. The hope that sinusoids would extrapolate to longer sequences was not borne out in practice (Press et al., 2021).
- Feed-forward. Gated variants such as SwiGLU (Shazeer, 2020) replace ReLU FFNs, usually with a reduced inner width to keep parameters constant. Sparse mixture-of-experts layers (Switch) scale the FFN further.
- Attention cost and memory. The O(n2) cost limited context length; FlashAttention makes exact attention IO-efficient, and multi-query / grouped-query attention shrink the key–value cache at inference.
- Decoder-only. GPT showed the decoder stack alone, trained as a language model, transfers broadly; BERT did the same with the encoder. Scaling laws (Kaplan et al., Chinchilla) were measured on this architecture.
Check yourself
- When is a self-attention layer cheaper than a recurrent one, according to Table 1?
- When sequence length n is smaller than representation width d, since the costs are O(n²d) versus O(nd²).
Reference
Glossary
- Self-attention
- Attention where queries, keys and values all come from the same sequence.
- Q, K, V
- Queries, keys, values: linear projections of the token representations.
- Head
- One attention computation with its own projections; h of them run in parallel.
- dmodel
- Width of the residual stream (512 base, 1024 big).
- Masking
- Setting disallowed scores to −∞ so future tokens get zero weight.
- Positional encoding
- A position-dependent vector added to token embeddings.
- Post-norm / pre-norm
- LayerNorm after the residual addition (this paper) or before the sub-layer (modern).
- Label smoothing
- Training against a target distribution with ε of the mass spread over other tokens.
- BPE
- Byte-pair encoding: a subword vocabulary built by merging frequent symbol pairs.
- BLEU
- N-gram overlap metric for translation quality.