Study guide · Language models & AI · 2017

Attention Is All You Need

Ashish Vaswani et al. · NeurIPS 2017

In 2017 the best translation systems still read sentences one word at a time with recurrent networks, using attention as a helper. This paper throws out the recurrence and builds the whole model from attention, so every word can look at every other word at once. It translated better, trained in a fraction of the time, and became the blueprint for nearly every large language model since.

The paper in four sentences

Recurrent networks read a sentence one word after another, so they are slow to train and struggle to connect words that are far apart.

Attention, a way for each word to pull in information from whichever other words are relevant, had been used as an add-on to them.

The Transformer is built from attention alone: in every layer, every word looks at every other word directly and in parallel, with some extra information added to say where each word sits in the sentence.

It set new records on English–German and English–French translation at a fraction of the training cost, and its design (with a few later tweaks) is what GPT, BERT and today's chatbots are built on.

Idea · §1–2

The problem with reading one word at a time

Before this paper, the standard way to process a sentence with a neural network was a recurrent network (such as an LSTM). It reads word 1, updates a running memory, reads word 2, updates the memory again, and so on. Word 50 cannot be processed until words 1 to 49 are done.

That causes two problems. First, speed: modern chips (GPUs) are fast because they do thousands of calculations at the same time, and a strict one-after-another chain wastes that. Second, distance: information about word 1 has to survive 49 memory updates to influence word 50, and it tends to fade on the way.

A fix already existed in part. Bahdanau and colleagues had let a translator "attend" back to any source word directly. But the recurrent chain was still there underneath. The Transformer's authors asked whether attention could do the whole job.

Everyday picturePassing a message down a line of fifty people by whispering, versus putting everyone in one room where anyone can ask anyone directly. The room is faster and the message doesn't get garbled.

TakeawayRecurrent networks are slow because each step waits for the last, and they lose track of far-apart words. Attention offered a direct path between any two words.

Idea · §3.2.1

Attention as a soft search

Here is how one word gathers information from the others. Each word produces three lists of numbers:

  • a query: what am I looking for?
  • a key: what do I contain, as a label others can match against?
  • a value: what information do I hand over if someone picks me?

To update a word, compare its query with every word's key. The comparison is a dot product: multiply matching entries and add them up, which gives a big number when the two lists point the same way. Turn those scores into weights that are positive and add up to 1 (a "softmax"), and take the weighted blend of the values. All of these lists are produced by the network itself, and it learns what to put in them during training.

Below, the word "it" is deciding what it refers to. Change the last word of the sentence and its query changes.

Left: keys of the other words (blue) and the query of "it" (amber) as arrows in a two-dimensional space. Right: dot-product score and resulting weight for each word. The vectors are hand-made to illustrate the mechanism; in a real model they have 64 or more numbers and are learned. The paper's own attention visualisations (Figs. 3–5) show heads resolving references like this.
Everyday pictureSearching a library. Your query is what you type, the keys are the catalogue entries, the values are the books. Instead of taking the one best match, you take a bit of every book, more from the better matches.

TakeawayEach word asks a question (query), every word offers a label (key), and the answer is a blend of the words' contents (values) weighted by how well labels match the question.

Check yourself
Why use a weighted blend instead of picking the single best match?
A blend can combine information from several words, and, unlike a hard pick, it changes smoothly when the numbers change, which is what training by gradient descent needs.
What would happen if every key were identical?
All scores would be equal, so every word gets the same weight and the result is a plain average of all values.

Idea · §4 · Table 1

Everything at once

In a Transformer layer, every word does this search at the same time. Nothing waits for anything else, so a GPU can compute the whole layer in one go. And any word can reach any other in a single step, however far apart they are.

The price is that the number of comparisons grows with the square of the sentence length: 10 words means 100 query–key comparisons, 1,000 words means a million. For the sentence lengths in translation that was cheap; for long documents it became the main cost, which later work like FlashAttention attacks.

Top: a recurrent network must take one step per word, and word 1 reaches word n only after n − 1 hops. Bottom: one self-attention layer connects every pair of words directly in a single parallel step (every line is one query–key comparison). The counts follow Table 1 of the paper: O(n) sequential steps versus O(1), and n² comparisons per attention layer.

TakeawaySelf-attention swaps a long chain of steps for one big parallel step, at the cost of comparing every pair of words.

Check yourself
A sentence doubles in length. How do the sequential steps change for an RNN and for self-attention? And the comparisons?
RNN steps double; self-attention stays at one step per layer. But the self-attention comparisons quadruple.

Idea · §3.2.2 · §3.5

Many heads, and a sense of order

Many heads. One search can only look for one kind of thing at a time. So each layer runs several searches side by side (8 in the base model, 16 in the big one), called heads, each with its own queries, keys and values. One head might track which noun a pronoun refers to, another the verb that goes with a subject. Their results are stuck together and mixed.

Order. Attention on its own ignores word order: "dog bites man" and "man bites dog" would look the same, because the search compares contents, not positions. So before the first layer, each word gets a positional encoding added to it: a pattern of wave values (sines and cosines of different wavelengths) unique to its position, like a barcode. The network learns to read these barcodes.

Each layer also has a small ordinary network applied to every word separately (the "feed-forward" part), plus shortcuts that add a layer's input back to its output and a normalisation step that keeps numbers in a sensible range. Six such layers make the encoder, which reads the source sentence; six more make the decoder, which writes the translation one word at a time and is blocked from peeking at words it hasn't written yet.

Everyday pictureA committee reading a contract. One member checks names, another dates, another money. Each reads the whole document at once and they pool notes. The page numbers (positions) are printed on every page so nobody loses the order.

TakeawaySeveral attention heads look for different relationships in parallel, and added position signals give the model the word order that attention alone ignores.

Check yourself
Why does the decoder need to be blocked from looking ahead?
At use time it generates words one by one, so future words don't exist yet. During training it must predict each word without seeing it, or it would just copy the answer.

Idea · §6 · Tables 2–3 · What came next

Better, cheaper, and everywhere

On the standard English-to-German translation test (WMT 2014) the big Transformer scored 28.4 BLEU, more than 2 points above the best previous results, including ensembles of several models. On English-to-French it scored 41.8, a new single-model record. The small version trained in 12 hours on one machine with 8 GPUs; the big one in 3.5 days, a small fraction of the computing cost of the previous best systems.

The paper also showed the design wasn't specific to translation: with little tuning, it did well at parsing English sentences into grammatical trees.

Within two years, GPT took just the decoder half and BERT took just the encoder half, and pre-trained them on huge amounts of text. Later it spread to images (ViT), audio, proteins and robotics.

Later refinementToday's language models keep the core of attention plus feed-forward blocks with shortcuts, but usually change four details: the normalisation is moved to the start of each block ("pre-norm") and simplified (RMSNorm); the wave-pattern positions are replaced by rotating queries and keys (RoPE); and the feed-forward part uses a "gated" variant (SwiGLU). Most also drop the encoder and use the decoder alone.

TakeawayAttention-only models were more accurate and much cheaper to train, and with a handful of later tweaks they became the standard architecture of modern AI.

Follow the idea

Built on: LSTM (1997), Attention (2014)

Led to: GPT-1 (2018), BERT (2018), GPT-3 (2020), ViT (2020), FlashAttention (2022), RoPE (2021)