Study guide · Computer vision · 2020

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Alexey Dosovitskiy et al. · ICLR 2021

Do computers need the special machinery of convolutional networks to see, or would the plain Transformer that powers language models work on images too? This paper cuts each image into 16 by 16 pixel squares, treats them like the words of a sentence, and feeds them to an almost unchanged Transformer. Trained on enough images, it matches or beats the best convolutional networks while using less compute to train.

The paper in four sentences

Cut an image into a grid of small squares ("patches"), turn each into a list of numbers, and give the sequence to a standard Transformer, the same design used for text.

Unlike a convolutional network, the Transformer is not told that nearby pixels belong together; every patch can look at every other patch from the very first layer, and it has to learn what matters.

That lack of built-in assumptions hurts when training data is modest: on ImageNet's 1.3 million images alone, it trails similar-sized convolutional networks.

Pre-trained on 14 to 300 million images, it matches or beats the best convolutional networks on many benchmarks, with substantially less training compute.

Idea · §1 · §3.1 · Fig. 1

An image as a sentence of patches

The Transformer took over language processing around 2018. It reads a sentence as a sequence of tokens (roughly, words), and every token can directly consult every other token through a mechanism called attention. Vision, meanwhile, was still dominated by convolutional networks such as ResNet.

Applying a Transformer to raw pixels is impractical: a 224 by 224 image has over 50,000 pixels, and attention compares every token with every other token, so the work grows with the square of the count. The paper's simple trick is to use small squares of pixels as the tokens instead. With 16 by 16 squares, a 224 by 224 image becomes a 14 by 14 grid: just 196 tokens, a sentence of moderate length.

Each patch is flattened into a list of numbers and multiplied by a learned table to make its token. A label saying where the patch came from (its position) is added on, because otherwise the Transformer would see the patches as an unordered bag. One extra learned token, the class token, is put at the front; at the end, whatever it holds is used to name the image.

Left: a 224 × 224 image (a drawing standing in for a photo) with the patch grid. Right: the first patches of the sequence, read row by row, after the class token. Smaller patches mean more tokens, finer detail and much more attention work; the paper's models use 14, 16 or 32 pixel patches.
Everyday pictureImagine cutting a photo into a jigsaw of equal squares, numbering each piece by where it came from, and handing the pile to someone who has only ever read books. With enough practice, they learn to recognise the picture from the pieces and their numbers.

TakeawayTreat patches as words and a standard language-model Transformer can classify images, with almost no vision-specific design.

Check yourself
Why not use each pixel as a token?
There would be tens of thousands of tokens, and attention's cost grows with the square of that number, which is far too expensive.
Why add a position label to each patch?
Attention by itself ignores order. Without positions, a shuffled image would look identical to the original.

Idea · §3.1 · §4.5 · Fig. 7

Every patch can look at every other

In a convolutional layer, each position only sees a small neighbourhood, typically 3 by 3. Information from far-apart parts of an image only meets after many layers. In a Transformer layer, every patch computes how relevant every other patch is to it and gathers information from them, weighted by that relevance. Two patches at opposite corners can exchange information in the first layer.

Click any patch below to see where it "looks" in a toy attention computation. Patches showing similar content attract each other strongly, wherever they are. Switch to the convolutional view to compare.

An 8 × 8 grid of patches. Each patch is described by a few numbers for its content (sky, sun, leaves, trunk, grass) plus its position; attention weight = softmax of the similarity between the selected patch and every patch. "Position preference" adds a bonus for nearby patches, a little like what some attention heads learn. Hand-made features for illustration; a real ViT learns them.

Real ViTs show both behaviours. The paper measures how far each attention head looks on average (Fig. 7, right): even in the lowest layers some heads attend across most of the image, while others stay local, rather like a convolution. Deeper in the network, almost all heads look far.

TakeawayAttention lets the model combine information from anywhere in the image at once, and it learns for itself when to look near and when to look far.

Check yourself
Click a leaf patch of the left tree. Why does attention light up the right tree too?
The two trees' leaf patches have similar content, and attention scores depend on content similarity, not on distance (unless a head learns a distance preference).

Idea · §3.1 · §4.2–4.3 · Figs. 3–4

Fewer assumptions, so more data needed

A convolutional network comes with useful assumptions baked in: nearby pixels are related, and a pattern means the same thing wherever it appears. These are called inductive biases. They are like a head start: the network doesn't need to learn them from data. ViT has far fewer. Apart from cutting the image into patches, it must discover from examples that neighbouring patches tend to belong together.

So the amount of training data decides the winner. Trained on ImageNet's 1.3 million images alone, the large ViTs did worse than comparable ResNets. Pre-trained on 14 million images (ImageNet-21k) they caught up, and on 303 million images (Google's internal JFT-300M collection) they pulled ahead. The best model, ViT-H/14, then reached 88.55% accuracy on ImageNet, against 87.54% for the best convolutional network compared (Big Transfer, BiT-L), while needing about a quarter of BiT-L's training compute (Table 2).

Everyday pictureA student given a rulebook learns faster at first. A student who works things out from scratch is slower, but with enough examples may end up understanding more, because the rulebook was never quite right.

TakeawayBuilt-in assumptions help with little data; with enough data, learning from scratch wins.

Check yourself
Why is "transformers beat CNNs" not quite the right summary of the paper?
The result depends on data scale. On ImageNet alone, ResNets of similar size did better; ViT wins only after pre-training on much larger datasets.

Idea · §4.5 · Fig. 7

It rediscovers the picture's geometry

ViT's position labels start out random; nothing tells it that patch 15 sits directly below patch 1. Yet after training, the paper finds that the position labels of nearby patches have become similar, and that patches in the same row or column share a pattern (Fig. 7, centre). The model has worked out the two-dimensional layout of the image by itself. Hand-designed 2D position labels gave no meaningful improvement over these learned ones.

Similarly, the learned first-layer filters, which turn each patch into numbers, resemble the edge-and-texture detectors that convolutional networks learn (Fig. 7, left).

Later refinementThe need for huge private datasets did not last. DeiT (2021) trained ViTs competitively on ImageNet alone, using strong augmentation and distillation from a CNN teacher. Self-supervised methods such as DINO and masked autoencoders (MAE, 2021) learned excellent ViTs without labels. Hybrids such as the Swin Transformer reintroduced local windows for efficiency. ViT became the standard image encoder for later systems in this archive, including CLIP and Segment Anything.

TakeawayGiven enough data, a general-purpose model learned the very structure that vision networks used to have built in.

Follow the idea

Built on: Transformer (2017), ResNet (2015)

Led to: CLIP (2021), Segment Anything (2023)