Study guide · Language models & AI · 2014

Neural Machine Translation by Jointly Learning to Align and Translate

Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio · ICLR 2015

Neural translators of 2014 squeezed a whole source sentence into one fixed-size list of numbers before writing the translation, and they fell apart on long sentences. Bahdanau, Cho and Bengio let the translator look back at every source word while it writes, choosing where to look for each output word. That learned "where to look" is what we now call attention.

The paper in four sentences

Neural translation systems read the source sentence into one summary, a fixed-length list of numbers, and then wrote the translation from that summary alone.

A summary of fixed size has limited room, so translation quality dropped sharply as sentences got longer.

This paper keeps a separate description of every source word and, for each word it writes, lets the translator compute a set of weights saying which source words to focus on, then blends their descriptions accordingly.

The weights are learned with no one telling the model which words correspond, yet they line up with sensible word alignments, and long sentences stop being a problem.

Idea · §1–2

Translating through a keyhole

In 2014, a new style of translation system had just appeared: the encoder–decoder. One neural network, the encoder, reads the English sentence word by word and ends with a single summary: a list of, say, 1,000 numbers. A second network, the decoder, reads that summary and writes the French sentence word by word.

Everything the decoder knows about the source has to pass through that one summary, and it is the same size whether the sentence has 5 words or 50. The authors suspected this was the weak point. In their words, the network must "compress all the necessary information of a source sentence into a fixed-length vector", and they conjectured that this fixed-length vector is a bottleneck.

The toy below shows why. Each "sentence" is a list of random words. Version one squashes the whole list into a single fixed-size summary and then tries to read back each word. Version two keeps one note per word and simply looks up the right note.

Each word (from a 50-word vocabulary) is a random vector. The fixed summary adds up all the words after scrambling each one by a different random "position" shuffle (a permutation with sign flips); reading word t back means unscrambling with position t's transformation and picking the closest vocabulary word, while the other words add noise. The per-word notes keep every word separately, so look-up is exact. A bigger summary helps, but every fixed size eventually runs out of room. Illustrative, not a real encoder.
Everyday pictureAn interpreter who must listen to a whole speech, close their notebook, and then deliver the translation from memory. Fine for one sentence; hopeless for a paragraph. Let them keep the notebook open and glance back, and length stops mattering.

TakeawayForcing a whole sentence through one fixed-size summary throws information away, and the loss grows with sentence length.

Check yourself
Why would a bigger summary help but not solve the problem?
It gives more room, so the breakdown happens at longer sentences, but any fixed size is eventually too small for long enough input.

Idea · §3.1

Look back while you write

The fix: don't throw away the per-word information. The encoder produces a description for every source word (the paper calls these annotations). Then, each time the decoder is about to write a word, it does three things:

  1. It scores every source word: how relevant is this word to what I am about to write, given what I have written so far?
  2. It turns the scores into weights that are positive and add up to 1 (a "softmax", which also exaggerates the differences so the best few dominate).
  3. It forms a context: a blend of the source-word descriptions, mixed according to the weights. Then it writes the next word using that context.

The scoring rule is itself a small neural network, trained along with everything else. Nobody tells it which English word matches which French word; it works that out because doing so helps it translate. The paper calls this learning to align. Everyone else soon called it attention.

The sentence from the paper's Fig. 3: "The agreement on the European Economic Area was signed in August 1992." Click a French word (left) to see how much weight it puts on each English word (darker = more). Weights here are hand-set to match the pattern the paper describes; the real figure comes from the trained model. Note the reversed block for "zone économique européenne", and "l'" looking at both "the" and the following noun.

TakeawayFor every word it writes, the decoder learns where to look in the source and blends what it finds there. That is attention.

Check yourself
Why do the weights need to add up to 1?
So that the context is an average of source-word descriptions: a blend in which focusing more on one word means focusing less on others.
Who decides which source words match which output words?
No one. The scoring network is trained only to make good translations; sensible alignments emerge because they help.

Idea · §3.2

Describing each word with both sides of the sentence

For the look-back to work, each source word's description should say something about the word in its setting, not just the word alone. "Bank" means different things in "river bank" and "bank loan".

So the encoder reads the sentence twice: once left to right and once right to left (a bidirectional recurrent network). Each word's description is the two readings stuck together: one summarising everything up to and including that word, the other everything from that word to the end. The description is mostly about the word itself and its near neighbours, because recurrent networks remember recent words best.

Everyday pictureHighlighting a word in a paragraph and writing a margin note that says what it means here, having read the text both forwards and backwards.

TakeawayEach source word gets its own description that reflects the words on both sides of it, ready for the decoder to consult.

Idea · §5 · Table 1 · Figs. 2–3

Long sentences stop being a problem

The authors trained English-to-French systems on a large collection of parallel text from the WMT 2014 shared task. Scores are given in BLEU, a standard measure of how many word sequences a machine translation shares with human reference translations (higher is better).

The old encoder–decoder, trained on sentences up to 50 words, scored 17.8 BLEU. The attention model with the same training data scored 26.8, and a version trained longer scored 28.5. More tellingly, the old system's quality fell steeply as sentences got longer, while the attention model's barely changed, even beyond 50 words (Fig. 2).

The best attention model still trailed Moses, a mature phrase-based translation system built over many years, on all sentences (28.5 vs 33.3 BLEU), but matched and slightly beat it when sentences with rare words were left out, even though Moses had also used an extra 418 million words of French text.

The learned weights, drawn as grids (Fig. 3), mostly run along the diagonal, as you'd expect for English and French, but bend where word order differs, such as adjectives that come after the noun in French.

TakeawayWith attention, translation quality no longer collapsed on long sentences, and the weights turned out to be readable word alignments.

Check yourself
Why is the attention grid mostly diagonal for English to French?
The two languages mostly keep the same word order, so the n-th French word usually corresponds to a word near the n-th English word.

Idea · §6–7 · What came next

From a helper to the whole machine

Here attention is an add-on to recurrent networks, which still read the sentence one word at a time. Within three years, the Transformer asked: what if attention is all you need? It dropped the recurrent parts entirely and let every word attend to every other word, in parallel. Today's large language models are built from that.

The paper's own limits were practical: the vocabulary was capped at 30,000 words, so rarer words became an "unknown" token, which the authors name as the main challenge left. And computing a weight for every source word at every output step costs time proportional to the product of the two lengths, which they judged fine for sentences but possibly limiting elsewhere.

Later refinementA year later, Luong, Pham and Manning (2015) showed a simpler scoring rule, a plain dot product between the decoder state and each source description, worked as well or better. The Transformer uses a scaled version of that dot product rather than this paper's small scoring network. Rare words were tackled by splitting words into smaller pieces (subword units), now standard.

TakeawayThis paper introduced learned soft attention as a fix for translation; it went on to become the core operation of modern language models.

Follow the idea

Built on: LSTM (1997)

Led to: Transformer (2017)