The paper in four sentences
To understand a word you often need the words after it as well as before it, but a model trained to predict the next word can only look backwards.
BERT is instead trained on a fill-in-the-blanks game: hide about 15% of the words in a passage and make the model guess them using everything around them, on both sides.
It also practises telling whether a second sentence really followed the first, so it learns about relationships between sentences.
Fine-tuned with one small extra layer per task, the same pre-trained model beat the previous best on eleven benchmarks, often by a wide margin, and became the standard starting point for language understanding systems.
Idea · §1 · Fig. 3
Reading in both directions
Take the sentence "I went to the bank to deposit my pay." When you reach "bank", the words that tell you it's a money bank, not a river bank, come after it. A model that reads strictly left to right, like GPT, has to form its picture of "bank" before seeing them.
Earlier work (ELMo) had tried to fix this by training one model left to right and another right to left, then gluing their outputs together. The BERT authors call that "shallow": the two directions never get to influence each other while the model is thinking. They wanted a single deep model in which every word, at every layer, can draw on both sides.
The toy below makes the case with simple word counts. Click a word to hide it, and compare guessing it from the word before only, versus from the words on both sides.
TakeawayWords are often best understood from what comes after them as well as before, and BERT is designed so every word can use both.
Check yourself
- Why is ELMo's approach called "shallow" bidirectionality?
- Its left-to-right and right-to-left models are trained separately and only combined at the end, so neither direction's reasoning can use the other's within the network.
Idea · §3.1 · Task 1 · App. A.1
The fill-in-the-blanks game
Why not just train a model that reads both ways to predict the next word? Because if it can see both ways, it can simply look at the next word. Some of the information would leak, and the task becomes trivial.
BERT's answer is the masked language model. In each training passage, pick 15% of the word pieces at random. Replace most of them with a special [MASK] symbol and ask the model to guess the originals, using everything else in the passage. Since the hidden words are actually missing, there is nothing to copy.
A detail: [MASK] never appears in real tasks later, so to avoid the model relying on it, the chosen words aren't always masked. Of the chosen 15%, 80% become [MASK], 10% become a random word, and 10% are left as they are. The model never knows which words are genuine, so it has to keep a good understanding of every word.
TakeawayHiding words and asking the model to restore them lets it use context on both sides without cheating.
Check yourself
- What fraction of all tokens are replaced with a random word?
- 10% of the chosen 15%, which is 1.5% of all tokens.
- What is a cost of only predicting 15% of the words?
- The model gets fewer training signals per passage than a left-to-right model, which predicts every word, so it may need more training. The paper finds it is slightly slower to converge but quickly does better.
Idea · §3.1 · Task 2
Does sentence B follow sentence A?
Many tasks are about pairs of sentences: does the answer fit the question, does one statement follow from another. Guessing hidden words doesn't directly teach that. So BERT plays a second game during pre-training: it's shown two passages and asked whether the second really came next in the original document (half the time it did) or was pulled from somewhere random (the other half). This is next sentence prediction.
The input always starts with a special [CLS] symbol whose final output is used for yes/no style decisions, and the two passages are separated by [SEP] and marked as "segment A" and "segment B".
TakeawayA second pre-training game taught BERT about relationships between sentences, though later work found it largely unnecessary.
Idea · §3.2 · §4 · Tables 1–4
One model, eleven records
BERT came in two sizes: BASE, deliberately the same size as GPT so the two could be compared fairly, and LARGE, about three times bigger. Both were pre-trained on English Wikipedia and a large collection of books, about 3.3 billion words, for about four days on Google's TPU chips.
For each task, one small output layer is added and the whole model is trained a little more, usually for under an hour on a single TPU. On the GLUE collection of nine understanding tasks, BERT-LARGE scored 80.5 on the official leaderboard against GPT's 72.8. On a question-answering test (SQuAD), where the answer is a span of the given paragraph, its best entry beat the best previous system and also the human F1 score listed in the paper (93.2 against 91.2). Even BERT-BASE, the same size as GPT, beat GPT on every GLUE task.
Bigger was consistently better, even on tasks with only a few thousand labelled examples, which surprised people at the time.
TakeawayDeep two-way pre-training plus brief fine-tuning beat everything before it, including GPT at the same size.
Check yourself
- Why did the authors make BERT-BASE the same size as GPT?
- So any difference in results could be attributed to the training approach (bidirectional masked LM and NSP) rather than to model size.
Idea · What came next
BERT's family, and why chatbots went the other way
BERT became the default tool for understanding text: search engines, classifiers and information-extraction systems were rebuilt on it within a year. A large family followed: RoBERTa trained it better, ALBERT made it smaller, ELECTRA replaced fill-in-the-blanks with a more efficient "spot the replaced word" game, and DeBERTa improved how position is handled.
But BERT can't easily write text: it fills in blanks, it doesn't continue a passage. The models behind today's chatbots descend from the other branch, GPT-2 and GPT-3, which kept next-word prediction and grew enormous. BERT-style encoders remain widely used for search, classification and embeddings, where reading both ways matters and generation does not.
TakeawayBERT defined the encoder branch of pre-trained Transformers; the generative chatbots came from the decoder branch.
Abstract, restated
BERT (Bidirectional Encoder Representations from Transformers) pre-trains deep bidirectional representations from unlabelled text by jointly conditioning on left and right context in all layers. The pre-trained model can be fine-tuned with just one additional output layer for a wide range of tasks without substantial task-specific architecture changes. It obtains new state-of-the-art results on eleven NLP tasks, including GLUE 80.5 (7.7 points absolute improvement), MultiNLI accuracy 86.7% (4.6), SQuAD v1.1 test F1 93.2 (1.5) and SQuAD v2.0 test F1 83.1 (5.1). NAACL 2019 (arXiv 1810.04805, Oct. 2018).
Section · §3 · Fig. 2
Architecture and input representation
A multi-layer bidirectional Transformer encoder, essentially the original implementation of Vaswani et al., with L layers, hidden size H, A heads and feed-forward size 4H:
- BERTBASE: L = 12, H = 768, A = 12, 110M parameters (chosen to match OpenAI GPT's size).
- BERTLARGE: L = 24, H = 1024, A = 16, 340M parameters.
Input: WordPiece tokens (30,000 vocabulary). Every sequence starts with [CLS]; sentence pairs are packed into one sequence separated by [SEP]. Each token's input embedding is the sum of a token embedding, a learned segment embedding (A or B) and a learned position embedding (Fig. 2), with at most 512 positions. The final hidden vector of [CLS] is C ∈ ℝH and of token i is Ti ∈ ℝH.
Section · §3.1 · App. A.1–A.2
Pre-training: MLM and NSP
Why not a bidirectional LM. Standard conditional language models can only be trained left-to-right or right-to-left, since bidirectional conditioning would allow each word to indirectly "see itself" in a multi-layer network, making prediction trivial.
Task 1: masked LM. Choose 15% of WordPiece positions uniformly at random. For a chosen token: 80% of the time replace it with [MASK], 10% with a random token, 10% leave it unchanged. Predict the original token from Ti with a softmax over the vocabulary (cross-entropy only on chosen positions). The mixing mitigates the pre-train/fine-tune mismatch, as [MASK] never appears at fine-tuning time, and since the encoder doesn't know which tokens are corrupted it must keep a distributional contextual representation of every token. Appendix A.1 illustrates with "my dog is hairy" (chosen token "hairy").
Task 2: next sentence prediction. For segments A, B: 50% IsNext (B actually follows A), 50% NotNext (random B from the corpus); predicted from C. The final model reaches 97–98% NSP accuracy. The paper reports NSP is very beneficial to QA and NLI.
Data and schedule. BooksCorpus (800M words) + English Wikipedia (2,500M words, text passages only); a document-level corpus is critical for long contiguous sequences. Batch 256 sequences × 512 tokens (128,000 tokens) for 1,000,000 steps, about 40 epochs over 3.3B words. Adam, learning rate 10−4, β1 = 0.9, β2 = 0.999, L2 weight decay 0.01, 10,000 warm-up steps then linear decay; dropout 0.1; GELU. 90% of steps at sequence length 128, the last 10% at 512 to learn the position embeddings. BASE: 4 Cloud TPUs (16 chips); LARGE: 16 Cloud TPUs (64 chips); 4 days each.
Check yourself
- Why keep 10% of chosen tokens unchanged rather than always masking?
- To bias the representation towards the actual observed word and reduce the pre-train/fine-tune mismatch; Table 8 shows always masking hurts the feature-based NER setting most (94.0 vs 94.9 dev F1).
- Why does random replacement at 1.5% of tokens not damage language understanding?
- It is rare relative to the model's capacity, and it forces the model to check whether each visible token is plausible in context rather than trusting it blindly.
Section · §3.2 · §4 · Tables 1–4
Fine-tuning and results
All parameters are fine-tuned end to end. Classification: a layer W ∈ ℝK×H on C, loss log softmax(CW⊤). Typical ranges: batch 16 or 32, learning rate 5·10−5, 3·10−5 or 2·10−5, 2–4 epochs. Every reported result takes at most about an hour on one Cloud TPU from the same pre-trained model.
Span extraction (§4.2). Learn start and end vectors S, E ∈ ℝH. Pi = eS·Ti / Σj eS·Tj (likewise for E); span (i, j) scores S·Ti + E·Tj with j ≥ i. SQuAD v2.0 (§4.3) treats "no answer" as the span at [CLS]: snull = S·C + E·C, and a non-null span is predicted when its best score exceeds snull + τ, with τ tuned on dev.
| GLUE test (Table 1) | MNLI m/mm | QQP | QNLI | SST-2 | CoLA | STS-B | MRPC | RTE | Avg |
|---|---|---|---|---|---|---|---|---|---|
| Pre-OpenAI SOTA | 80.6/80.1 | 66.1 | 82.3 | 93.2 | 35.0 | 81.0 | 86.0 | 61.7 | 74.0 |
| BiLSTM+ELMo+Attn | 76.4/76.1 | 64.8 | 79.8 | 90.4 | 36.0 | 73.3 | 84.9 | 56.8 | 71.0 |
| OpenAI GPT | 82.1/81.4 | 70.3 | 87.4 | 91.3 | 45.4 | 80.0 | 82.3 | 56.0 | 75.1 |
| BERTBASE | 84.6/83.4 | 71.2 | 90.5 | 93.5 | 52.1 | 85.8 | 88.9 | 66.4 | 79.6 |
| BERTLARGE | 86.7/85.9 | 72.1 | 92.7 | 94.9 | 60.5 | 86.5 | 89.3 | 70.1 | 82.1 |
The table's average excludes WNLI; the official GLUE leaderboard score (which includes it) is 80.5 for BERTLARGE versus 72.8 for GPT. BERTLARGE fine-tuning was sometimes unstable on small datasets, so several random restarts were run and the best on dev was chosen. SQuAD v1.1 (Table 2): single BERTLARGE dev F1 90.9; with TriviaQA pre-fine-tuning, an ensemble reaches test EM/F1 87.4/93.2. SQuAD v2.0 (Table 3): single BERTLARGE test EM/F1 80.0/83.1, +5.1 F1 over the previous best. SWAG (Table 4): BERTLARGE test 86.3 vs GPT 78.0 and ESIM+ELMo 59.2.
Section · §5 · Tables 5–7 · Fig. 5
Ablations
Pre-training tasks (Table 5). Removing NSP hurts QNLI, MNLI and SQuAD v1.1. Switching to left-to-right is much worse still, especially on MRPC and SQuAD; for SQuAD the token-level predictions have no right context. A BiLSTM on top recovers some SQuAD performance but hurts GLUE. The authors also note that training separate LTR and RTL models and concatenating them (as ELMo does) would cost twice as much, be non-intuitive for QA, and be strictly less powerful than a deep bidirectional model.
| #L | #H | #A | LM ppl | MNLI-m | MRPC | SST-2 |
|---|---|---|---|---|---|---|
| 3 | 768 | 12 | 5.84 | 77.9 | 79.8 | 88.4 |
| 6 | 768 | 3 | 5.24 | 80.6 | 82.2 | 90.7 |
| 6 | 768 | 12 | 4.68 | 81.9 | 84.8 | 91.3 |
| 12 | 768 | 12 | 3.99 | 84.4 | 86.7 | 92.9 |
| 12 | 1024 | 16 | 3.54 | 85.7 | 86.9 | 93.3 |
| 24 | 1024 | 16 | 3.23 | 86.6 | 87.8 | 93.7 |
Model size (Table 6, dev, mean of 5 restarts). Every task improves with size, including MRPC with only 3,600 training examples: the first convincing demonstration that very large scale helps small tasks too, given sufficient pre-training. Feature-based use (Table 7, CoNLL-2003 NER). Without fine-tuning, concatenating the top four hidden layers of BERTBASE as features for a BiLSTM gives 96.1 dev F1, only 0.3 behind fine-tuning BERTBASE (96.4). Training steps (Fig. 5). MNLI accuracy gains almost 1.0% going from 500k to 1M steps; MLM converges slightly more slowly than LTR but outperforms it almost immediately.
Section · Limitations and lineage
What was revised
- NSP and training budget. RoBERTa (Liu et al., 2019) removed NSP, used dynamic masking, much more data, larger batches and longer training, and matched or exceeded later models: BERT was significantly undertrained. ALBERT (2019) replaced NSP with sentence-order prediction and shared parameters across layers.
- Sample efficiency of MLM. Only 15% of tokens give a learning signal. ELECTRA (Clark et al., 2020) trains a discriminator to detect replaced tokens at every position, matching RoBERTa with a fraction of the compute. SpanBERT masks contiguous spans; whole-word masking became a released BERT variant.
- Mismatch and independence. MLM predicts masked tokens independently given the rest, and [MASK] never appears downstream; XLNet (Yang et al., 2019) addressed both with permutation language modelling.
- Positions and length. Learned absolute positions cap input at 512 tokens; DeBERTa (2020) used disentangled relative position attention; ModernBERT (2024) adopted RoPE, longer context and FlashAttention.
- Unified text-to-text and generation. T5 (Raffel et al., 2020) cast all tasks as text-to-text with an encoder–decoder and span corruption. For open-ended generation and in-context learning, decoder-only models (GPT-2, GPT-3) won out.
Reference
Glossary
- Encoder
- A Transformer stack with unmasked (bidirectional) self-attention.
- MLM
- Masked language model: predict randomly selected, corrupted tokens from full context.
- NSP
- Next sentence prediction: binary classification of whether segment B follows A.
- [CLS], [SEP]
- Special tokens: sequence-level classification slot; segment separator.
- Segment embedding
- Learned vector marking whether a token belongs to sentence A or B.
- WordPiece
- Subword tokenisation with a fixed vocabulary (30k here).
- Feature-based
- Using frozen pre-trained representations as input features (ELMo style).
- Fine-tuning
- Updating all pre-trained weights on the downstream task (GPT, BERT style).
- GLUE, SQuAD, SWAG
- Benchmarks for sentence understanding, extractive QA and commonsense continuation.