Study guide · Language models & AI · 2018

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova · NAACL 2019

GPT had shown that a Transformer pre-trained on plain text could be fine-tuned for many tasks, but it read only left to right. Devlin and colleagues at Google ask what happens if the model may use the words on both sides of every word. To make that possible they train it to fill in deliberately hidden words, and the resulting model, BERT, set new records on eleven language tasks.

The paper in four sentences

To understand a word you often need the words after it as well as before it, but a model trained to predict the next word can only look backwards.

BERT is instead trained on a fill-in-the-blanks game: hide about 15% of the words in a passage and make the model guess them using everything around them, on both sides.

It also practises telling whether a second sentence really followed the first, so it learns about relationships between sentences.

Fine-tuned with one small extra layer per task, the same pre-trained model beat the previous best on eleven benchmarks, often by a wide margin, and became the standard starting point for language understanding systems.

Idea · §1 · Fig. 3

Reading in both directions

Take the sentence "I went to the bank to deposit my pay." When you reach "bank", the words that tell you it's a money bank, not a river bank, come after it. A model that reads strictly left to right, like GPT, has to form its picture of "bank" before seeing them.

Earlier work (ELMo) had tried to fix this by training one model left to right and another right to left, then gluing their outputs together. The BERT authors call that "shallow": the two directions never get to influence each other while the model is thinking. They wanted a single deep model in which every word, at every layer, can draw on both sides.

The toy below makes the case with simple word counts. Click a word to hide it, and compare guessing it from the word before only, versus from the words on both sides.

A counting model built from a made-up text of about 120 words (short sentences about animals). "Left only" looks up which words followed the previous word; "both sides" looks up which words appeared between the same previous and next words. The hidden occurrence itself is left out of the counts, so it can't cheat. The readout scores both methods on every word of the text. BERT learns this with a Transformer rather than counts, so it can use many words of context on each side. Illustrative only.
Everyday pictureA crossword clue is much easier when you already have letters from the crossing words on both sides, not just the first few letters.

TakeawayWords are often best understood from what comes after them as well as before, and BERT is designed so every word can use both.

Check yourself
Why is ELMo's approach called "shallow" bidirectionality?
Its left-to-right and right-to-left models are trained separately and only combined at the end, so neither direction's reasoning can use the other's within the network.

Idea · §3.1 · Task 1 · App. A.1

The fill-in-the-blanks game

Why not just train a model that reads both ways to predict the next word? Because if it can see both ways, it can simply look at the next word. Some of the information would leak, and the task becomes trivial.

BERT's answer is the masked language model. In each training passage, pick 15% of the word pieces at random. Replace most of them with a special [MASK] symbol and ask the model to guess the originals, using everything else in the passage. Since the hidden words are actually missing, there is nothing to copy.

A detail: [MASK] never appears in real tasks later, so to avoid the model relying on it, the chosen words aren't always masked. Of the chosen 15%, 80% become [MASK], 10% become a random word, and 10% are left as they are. The model never knows which words are genuine, so it has to keep a good understanding of every word.

Top: the original. Middle: what the model is shown (amber = [MASK], red = swapped for a random word, green outline = chosen but left unchanged). The model must predict the original at every outlined position, and only there. Bottom: running percentages of all tokens. Expect about 15% chosen: 12% masked, 1.5% random, 1.5% unchanged.

TakeawayHiding words and asking the model to restore them lets it use context on both sides without cheating.

Check yourself
What fraction of all tokens are replaced with a random word?
10% of the chosen 15%, which is 1.5% of all tokens.
What is a cost of only predicting 15% of the words?
The model gets fewer training signals per passage than a left-to-right model, which predicts every word, so it may need more training. The paper finds it is slightly slower to converge but quickly does better.

Idea · §3.1 · Task 2

Does sentence B follow sentence A?

Many tasks are about pairs of sentences: does the answer fit the question, does one statement follow from another. Guessing hidden words doesn't directly teach that. So BERT plays a second game during pre-training: it's shown two passages and asked whether the second really came next in the original document (half the time it did) or was pulled from somewhere random (the other half). This is next sentence prediction.

The input always starts with a special [CLS] symbol whose final output is used for yes/no style decisions, and the two passages are separated by [SEP] and marked as "segment A" and "segment B".

Later refinementThis second game turned out to matter less than the paper suggested. RoBERTa (2019) found that dropping it, and instead training longer on more data with full-length passages, worked as well or better. Most later models of this kind dropped it.

TakeawayA second pre-training game taught BERT about relationships between sentences, though later work found it largely unnecessary.

Idea · §3.2 · §4 · Tables 1–4

One model, eleven records

BERT came in two sizes: BASE, deliberately the same size as GPT so the two could be compared fairly, and LARGE, about three times bigger. Both were pre-trained on English Wikipedia and a large collection of books, about 3.3 billion words, for about four days on Google's TPU chips.

For each task, one small output layer is added and the whole model is trained a little more, usually for under an hour on a single TPU. On the GLUE collection of nine understanding tasks, BERT-LARGE scored 80.5 on the official leaderboard against GPT's 72.8. On a question-answering test (SQuAD), where the answer is a span of the given paragraph, its best entry beat the best previous system and also the human F1 score listed in the paper (93.2 against 91.2). Even BERT-BASE, the same size as GPT, beat GPT on every GLUE task.

Bigger was consistently better, even on tasks with only a few thousand labelled examples, which surprised people at the time.

TakeawayDeep two-way pre-training plus brief fine-tuning beat everything before it, including GPT at the same size.

Check yourself
Why did the authors make BERT-BASE the same size as GPT?
So any difference in results could be attributed to the training approach (bidirectional masked LM and NSP) rather than to model size.

Idea · What came next

BERT's family, and why chatbots went the other way

BERT became the default tool for understanding text: search engines, classifiers and information-extraction systems were rebuilt on it within a year. A large family followed: RoBERTa trained it better, ALBERT made it smaller, ELECTRA replaced fill-in-the-blanks with a more efficient "spot the replaced word" game, and DeBERTa improved how position is handled.

But BERT can't easily write text: it fills in blanks, it doesn't continue a passage. The models behind today's chatbots descend from the other branch, GPT-2 and GPT-3, which kept next-word prediction and grew enormous. BERT-style encoders remain widely used for search, classification and embeddings, where reading both ways matters and generation does not.

Later refinementThe word2vec idea of one fixed vector per word was fully superseded here: BERT gives each word a different representation in every sentence, computed from the whole context.

TakeawayBERT defined the encoder branch of pre-trained Transformers; the generative chatbots came from the decoder branch.

Follow the idea

Built on: word2vec (2013), Transformer (2017), GPT-1 (2018)