The paper in four sentences
Cut an image into a grid of small squares ("patches"), turn each into a list of numbers, and give the sequence to a standard Transformer, the same design used for text.
Unlike a convolutional network, the Transformer is not told that nearby pixels belong together; every patch can look at every other patch from the very first layer, and it has to learn what matters.
That lack of built-in assumptions hurts when training data is modest: on ImageNet's 1.3 million images alone, it trails similar-sized convolutional networks.
Pre-trained on 14 to 300 million images, it matches or beats the best convolutional networks on many benchmarks, with substantially less training compute.
Idea · §1 · §3.1 · Fig. 1
An image as a sentence of patches
The Transformer took over language processing around 2018. It reads a sentence as a sequence of tokens (roughly, words), and every token can directly consult every other token through a mechanism called attention. Vision, meanwhile, was still dominated by convolutional networks such as ResNet.
Applying a Transformer to raw pixels is impractical: a 224 by 224 image has over 50,000 pixels, and attention compares every token with every other token, so the work grows with the square of the count. The paper's simple trick is to use small squares of pixels as the tokens instead. With 16 by 16 squares, a 224 by 224 image becomes a 14 by 14 grid: just 196 tokens, a sentence of moderate length.
Each patch is flattened into a list of numbers and multiplied by a learned table to make its token. A label saying where the patch came from (its position) is added on, because otherwise the Transformer would see the patches as an unordered bag. One extra learned token, the class token, is put at the front; at the end, whatever it holds is used to name the image.
TakeawayTreat patches as words and a standard language-model Transformer can classify images, with almost no vision-specific design.
Check yourself
- Why not use each pixel as a token?
- There would be tens of thousands of tokens, and attention's cost grows with the square of that number, which is far too expensive.
- Why add a position label to each patch?
- Attention by itself ignores order. Without positions, a shuffled image would look identical to the original.
Idea · §3.1 · §4.5 · Fig. 7
Every patch can look at every other
In a convolutional layer, each position only sees a small neighbourhood, typically 3 by 3. Information from far-apart parts of an image only meets after many layers. In a Transformer layer, every patch computes how relevant every other patch is to it and gathers information from them, weighted by that relevance. Two patches at opposite corners can exchange information in the first layer.
Click any patch below to see where it "looks" in a toy attention computation. Patches showing similar content attract each other strongly, wherever they are. Switch to the convolutional view to compare.
Real ViTs show both behaviours. The paper measures how far each attention head looks on average (Fig. 7, right): even in the lowest layers some heads attend across most of the image, while others stay local, rather like a convolution. Deeper in the network, almost all heads look far.
TakeawayAttention lets the model combine information from anywhere in the image at once, and it learns for itself when to look near and when to look far.
Check yourself
- Click a leaf patch of the left tree. Why does attention light up the right tree too?
- The two trees' leaf patches have similar content, and attention scores depend on content similarity, not on distance (unless a head learns a distance preference).
Idea · §3.1 · §4.2–4.3 · Figs. 3–4
Fewer assumptions, so more data needed
A convolutional network comes with useful assumptions baked in: nearby pixels are related, and a pattern means the same thing wherever it appears. These are called inductive biases. They are like a head start: the network doesn't need to learn them from data. ViT has far fewer. Apart from cutting the image into patches, it must discover from examples that neighbouring patches tend to belong together.
So the amount of training data decides the winner. Trained on ImageNet's 1.3 million images alone, the large ViTs did worse than comparable ResNets. Pre-trained on 14 million images (ImageNet-21k) they caught up, and on 303 million images (Google's internal JFT-300M collection) they pulled ahead. The best model, ViT-H/14, then reached 88.55% accuracy on ImageNet, against 87.54% for the best convolutional network compared (Big Transfer, BiT-L), while needing about a quarter of BiT-L's training compute (Table 2).
TakeawayBuilt-in assumptions help with little data; with enough data, learning from scratch wins.
Check yourself
- Why is "transformers beat CNNs" not quite the right summary of the paper?
- The result depends on data scale. On ImageNet alone, ResNets of similar size did better; ViT wins only after pre-training on much larger datasets.
Idea · §4.5 · Fig. 7
It rediscovers the picture's geometry
ViT's position labels start out random; nothing tells it that patch 15 sits directly below patch 1. Yet after training, the paper finds that the position labels of nearby patches have become similar, and that patches in the same row or column share a pattern (Fig. 7, centre). The model has worked out the two-dimensional layout of the image by itself. Hand-designed 2D position labels gave no meaningful improvement over these learned ones.
Similarly, the learned first-layer filters, which turn each patch into numbers, resemble the edge-and-texture detectors that convolutional networks learn (Fig. 7, left).
TakeawayGiven enough data, a general-purpose model learned the very structure that vision networks used to have built in.
Abstract, restated
While the Transformer has become the de facto standard for NLP, in vision attention was mostly used together with convolutional networks or to replace parts of them while keeping their overall structure. The paper shows this reliance on CNNs is not necessary: a pure Transformer applied directly to sequences of image patches performs very well on image classification. Pre-trained on large data and transferred to mid-sized or small benchmarks (ImageNet, CIFAR-100, VTAB and others), the Vision Transformer attains excellent results compared with state-of-the-art convolutional networks while requiring substantially fewer computational resources to train. ICLR 2021 (arXiv October 2020).
Section · §3.1 · Eqs. 1–4 · Table 1
The model
An image x ∈ ℝH×W×C is reshaped into N = HW/P2 flattened patches xp ∈ ℝN×(P2·C), linearly projected to the model width D, prefixed with a learnable class embedding, and given learnable 1D position embeddings. The encoder alternates multi-head self-attention (MSA) and MLP blocks (two layers, GELU), with LayerNorm before each block and residual connections after:
A classification head is attached to zL0: an MLP with one hidden layer during pre-training, a single linear layer during fine-tuning. Naming: ViT-L/16 is the Large variant with 16 × 16 patches. Compute the parameter count and sequence length for any configuration:
Check yourself
- Where are most of ViT-B's parameters?
- In the 12 encoder layers: each has about 7.1 M (2.4 M attention, 4.7 M MLP). The patch projection and position embeddings together are under 1 M.
Section · §3.1–3.2
Inductive bias, hybrids and resolution
Inductive bias. In CNNs, locality, 2D neighbourhood structure and translation equivariance are baked into every layer. In ViT "only MLP layers are local and translationally equivariant, while the self-attention layers are global." The 2D structure is used sparingly: in cutting the image into patches, and when adjusting position embeddings for a different resolution at fine-tuning time. Otherwise the position embeddings carry no 2D information at initialisation, and all spatial relations must be learned.
Hybrid. Instead of raw patches, the input sequence can be formed from a CNN feature map (patch size 1 × 1, i.e. each spatial position of a ResNet stage). Hybrids slightly outperform ViT at small compute budgets, but the difference vanishes for larger models (§4.4).
Fine-tuning (§3.2). The pre-training head is removed and a zero-initialised D × K layer attached. It is often beneficial to fine-tune at higher resolution than pre-training; the patch size is kept, so the sequence is longer, and the pre-trained position embeddings are 2D-interpolated according to their location in the original image.
Check yourself
- Fine-tuning ViT-B/16 at 384 × 384 instead of 224, what happens to the sequence length?
- It grows from 14 × 14 = 196 to 24 × 24 = 576 patches (plus the class token), so the 196 learned position embeddings must be interpolated onto a 24 × 24 grid.
Section · §4.1–4.2 · Table 2
Setup and headline results
Data. Pre-training on ImageNet (1.3 M images, 1 k classes), ImageNet-21k (14 M images, 21 k classes) or JFT (303 M high-resolution images, 18 k classes), de-duplicated against the downstream test sets. Transfer to ImageNet (original and cleaned-up ReaL labels), CIFAR-10/100, Oxford-IIIT Pets, Oxford Flowers-102, and the 19-task VTAB suite (natural, specialised, structured groups). Training. Adam with β1 = 0.9, β2 = 0.999, batch 4,096, weight decay 0.1 (found useful for transfer of all models, ResNets included), linear learning-rate warm-up and decay; fine-tuning with SGD with momentum. Baselines are Big Transfer (BiT) ResNets and Noisy Student (EfficientNet-L2).
ViT-L/16 pre-trained on JFT matches or beats BiT-L (pre-trained on the same dataset) on every column of Table 2 (VTAB is a tie, 76.28 vs 76.29) while taking 0.68 k TPUv3-core-days against 9.9 k. ViT-L/16 pre-trained on public ImageNet-21k also does well (85.30% ImageNet) and trains in about 30 days on a standard cloud TPUv3 with 8 cores.
Section · §4.3–4.6 · Figs. 3–7
Data requirements, scaling and inspection
Dataset size (Fig. 3). Pre-trained on ImageNet, ViT-Large underperforms ViT-Base despite moderate regularisation (weight decay, dropout, label smoothing), and both trail BiT ResNets; with ImageNet-21k they are similar; only with JFT-300M do the larger ViTs show their full benefit. Subsets (Fig. 4). With linear few-shot evaluation on random JFT subsets of 9 M, 30 M, 90 M and the full 300 M, without extra regularisation, "Vision Transformers overfit more than ResNets with comparable computational cost on smaller datasets," while ResNets plateau sooner as data grows.
Scaling study (Fig. 5). Across ResNets, ViTs and hybrids, ViT dominates on the performance/compute trade-off, using roughly 2–4× less compute for the same average transfer performance over five datasets, and shows no sign of saturating within the range tried.
Inspection (Fig. 7). The top principal components of the learned patch embedding resemble plausible basis functions for a low-dimensional representation of each patch's fine structure. Position embeddings learn 2D topology (closer patches have more similar embeddings; row-column structure appears; sinusoidal structure for larger grids). Mean attention distance, "analogous to receptive field size in CNNs", is already large for some heads in the lowest layers and grows with depth; Appendix figures show attention focusing on semantically relevant regions.
Self-supervision (§4.6). A preliminary masked patch prediction objective, mimicking BERT's masked language modelling, gives ViT-B/16 79.9% on ImageNet: about 2% better than training from scratch, but 4% behind supervised pre-training.
Section · §5 · lineage
Limitations and lineage
- Data hunger. The strongest results depend on JFT-300M, a private dataset; on ImageNet alone ViT is weaker than ResNets of similar size.
- Quadratic cost. Global attention over (N+1) tokens is O(N2), which limits small patches and high resolutions, and hence dense tasks such as detection and segmentation (left as future work in §5).
- Self-supervision unresolved. Masked patch prediction lags supervised pre-training by about 4%.
Reference
Glossary
- Patch embedding
- Linear projection E of each flattened P × P × C patch to width D.
- Class token
- Learnable token prepended to the sequence; its final state represents the image.
- Position embedding
- Learnable vector added per token position (1D in ViT).
- MSA
- Multi-head self-attention.
- Inductive bias
- Assumptions built into an architecture, such as locality and translation equivariance.
- JFT-300M
- Google's internal dataset of about 303 M images with 18 k classes.
- VTAB
- Visual Task Adaptation Benchmark: 19 tasks with 1,000 training examples each.
- TPUv3-core-day
- One TPUv3 core used for one day; the paper's compute unit.
- Hybrid
- ViT applied to CNN feature maps instead of raw patches.