The paper in four sentences
Vision models were usually trained to pick one of a fixed set of labels, like ImageNet's 1,000, which is expensive to annotate and limits what they can recognise.
CLIP instead trains two networks, one for images and one for text, so that an image and its caption end up as nearby points, and mismatched pairs far apart; the 400 million training pairs were collected from the internet.
To classify a new image, you write a short description for each possible category ("a photo of a dog", "a photo of a cat") and pick the description that lands closest to the image: no extra training needed. This is called zero-shot classification.
Without seeing a single ImageNet training example, the best CLIP model matched the accuracy of a ResNet-50 trained on all 1.28 million of them, and held up much better on unusual images such as sketches.
Idea · §1 · §2.1–2.2
Captions are free labels
Image recognisers like AlexNet and ResNet learn from datasets where people have labelled every image with one category from a fixed list. That has two costs. Labelling millions of images is slow and expensive. And the model only ever knows those categories: to recognise something new, you need new labels and new training.
Meanwhile, the internet is full of images that already come with text: captions, alt text, titles. That text is noisy, but it is free, plentiful, and describes far more than 1,000 categories. Language models such as GPT-2 had shown what learning from raw web text could do. CLIP asks whether vision can learn the same way.
The authors built a new dataset of 400 million (image, text) pairs from the internet. To cover a wide range of concepts, they searched for pairs whose text contained one of 500,000 query words and phrases, keeping up to 20,000 pairs per query.
TakeawayNatural-language captions give an almost unlimited, open-ended source of supervision for vision.
Idea · §2.3 · Figs. 2–3
Learning by matching, not by describing
The authors first tried teaching a model to write each image's caption word for word. It learned slowly, because there are countless correct ways to caption a photo and the model is penalised for not guessing the exact one. Simply predicting which words appear, ignoring order, was 3 times faster to train. Then they switched to an even easier task: given a batch of images and a batch of captions, just work out which caption goes with which image. That was another 4 times more efficient.
This matching approach is called contrastive learning. An image network turns each image into a point (a list of numbers called an embedding), and a text network does the same for each caption. Training pulls each image towards its own caption and pushes it away from all the other captions in the batch. Press Train to watch it happen in a small, two-dimensional version.
TakeawayMatching images to captions among many distractors teaches a model what images contain, far more efficiently than generating the captions.
Check yourself
- In a batch of 32,768 pairs, how many wrong pairings does each image get pushed away from?
- 32,767: every other caption in the batch. Large batches give many "negatives" per step, which is part of why CLIP used such a large batch.
- Why was predicting the exact caption a slow way to learn?
- Many different captions fit the same image, so the model spends its effort on wording rather than on recognising what is in the picture.
Idea · §3.1 · Fig. 1
Classify by writing the category names
After training, CLIP can classify images into categories it never saw as labels. Write a sentence for each category, such as "a photo of a dog." and "a photo of a cat.", turn each sentence into a point with the text network, turn the image into a point with the image network, and pick the closest sentence. Because nothing is trained for the new task, this is called zero-shot classification.
Wording matters. Using "A photo of a {label}." instead of the bare label raised ImageNet accuracy by 1.3 points, partly because single words are ambiguous ("crane" the bird or the machine?) and web captions rarely consist of a single word. Choose an image and the candidate categories below.
TakeawayA text description becomes a classifier: change the list of descriptions and you have a new classifier, with no retraining.
Check yourself
- What happens if the correct category is not in your list?
- The model still picks one, the closest description it has. Zero-shot classification can only choose among the options you write.
Idea · §3.1 · §3.3 · Table 1 · Fig. 13
How well it works, and why it is robust
The best CLIP model classified ImageNet zero-shot with 76.2% accuracy. That equals the original ResNet-50, which was trained on ImageNet's 1.28 million labelled images; CLIP used none of them. An earlier attempt at the same zero-shot idea (Visual N-Grams, 2017) had managed 11.5%.
The bigger surprise was robustness. Models trained on ImageNet often fail on images that look different from ImageNet photos: sketches, cartoons, unusual angles. Zero-shot CLIP and a ResNet-101 have the same ImageNet accuracy, but on sketches CLIP scores 60.2% against 25.2%, and on artistic renditions 88.9% against 37.7% (Fig. 13). Because CLIP never trained on ImageNet specifically, it hasn't latched onto the quirks of that one dataset.
| Dataset (Fig. 13) | ResNet-101 | Zero-shot CLIP |
|---|---|---|
| ImageNet | 76.2 | 76.2 |
| ImageNet Sketch | 25.2 | 60.2 |
| ImageNet-R (renditions) | 37.7 | 88.9 |
| ImageNet-A (hard natural images) | 2.7 | 77.1 |
TakeawayLearning from broad web data rather than one curated dataset gave accuracy that transfers to new kinds of images.
Idea · §6 · §7
Limits, and a model everyone built on
CLIP is not good at everything. It got only 88% on handwritten digits (MNIST), which simple models handle almost perfectly, because few web images look like those digits. It struggles to count objects and with very fine distinctions such as car models or aircraft types. The authors estimate it would need around 1,000 times more compute for zero-shot CLIP to reach the best task-specific results overall. And because it learns from unfiltered internet data, it picks up the internet's social biases; the paper's own bias probes found that, depending on the category names offered, it could give offensive labels to images of people.
TakeawayCLIP turned language into a universal interface for vision, at the price of the web's blind spots and biases.
Abstract, restated
State-of-the-art vision systems are trained to predict a fixed set of predetermined categories, which limits their generality, since more labelled data is needed for any new concept. Learning directly from raw text about images uses a much broader source of supervision. The simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn state-of-the-art image representations from scratch, on 400 million (image, text) pairs from the internet. Natural language then references learned visual concepts (or describes new ones), enabling zero-shot transfer. Benchmarked on over 30 datasets spanning OCR, action recognition in videos, geo-localisation and fine-grained classification, the model transfers non-trivially to most tasks and is often competitive with fully supervised baselines; it matches the original ResNet-50 on ImageNet zero-shot without using any of its 1.28 million training examples. ICML 2021.
Section · §2.2–2.3 · Figs. 2–3
Data and the contrastive objective
WIT. 400 M (image, text) pairs collected by searching for text containing one of 500,000 queries (the base query list is all words occurring at least 100 times in English Wikipedia, augmented with high-frequency bigrams, the names of Wikipedia articles above a certain search volume, and WordNet synsets), class-balanced to at most 20,000 pairs per query. The total word count is similar to the WebText dataset used for GPT-2.
Choosing the objective (Fig. 2). A transformer language model predicting captions learned ImageNet zero-shot accuracy 3× more slowly than a bag-of-words prediction baseline; replacing prediction with a contrastive objective gave a further 4× efficiency gain. Given a batch of N pairs, CLIP jointly trains an image and text encoder to maximise the cosine similarity of the N real pairs and minimise it for the N2 − N incorrect ones, with a symmetric cross-entropy over the similarity scores (the multi-class N-pair / InfoNCE loss):
# Fig. 3, numpy-like pseudocode (abridged) I_f = image_encoder(I) # [n, d_i] T_f = text_encoder(T) # [n, d_t] I_e = l2_normalize(np.dot(I_f, W_i), axis=1) T_e = l2_normalize(np.dot(T_f, W_t), axis=1) logits = np.dot(I_e, T_e.T) * np.exp(t) # [n, n] labels = np.arange(n) loss_i = cross_entropy_loss(logits, labels, axis=0) loss_t = cross_entropy_loss(logits, labels, axis=1) loss = (loss_i + loss_t)/2
Simplifications relative to prior work (ConVIRT, which CLIP describes itself as a simplified version of): trained from scratch with no ImageNet or text pre-training; linear rather than non-linear projections into the joint space; random square crops of resized images as the only augmentation; the temperature τ is a learned log-parameterised scalar, initialised to the equivalent of 0.07 and clipped so the logit scale et never exceeds 100. Explore how the logit scale and batch size shape the loss:
Check yourself
- Why does a higher logit scale (lower temperature) sharpen the softmax, and why cap it?
- Cosine similarities lie in [−1, 1], so without scaling the softmax over thousands of candidates is nearly flat. Multiplying by a large scale lets small similarity differences dominate. The cap at 100 prevents training instability from ever-sharper logits.
Section · §2.4–2.5
Encoders and training
Image encoders. Five ResNets (ResNet-50, ResNet-101, and EfficientNet-style scalings RN50x4, RN50x16, RN50x64 using about 4×, 16× and 64× the compute of ResNet-50), modified with ResNet-D improvements, antialiased blur pooling, and global average pooling replaced by an attention pooling layer (a single layer of "transformer-style" multi-head QKV attention whose query is conditioned on the global average-pooled representation). Three Vision Transformers (ViT-B/32, ViT-B/16, ViT-L/14), with an extra layer norm added to the combined patch and position embeddings before the transformer.
Text encoder. A 63 M-parameter, 12-layer, 512-wide Transformer with 8 attention heads, operating on a lower-cased byte-pair-encoded text with a 49,152-token vocabulary, maximum sequence length 76, bracketed with [SOS] and [EOS]; the top layer's activation at [EOS] is layer-normalised and linearly projected into the joint space. Masked self-attention is used to keep the option of initialising from a pre-trained language model or adding language modelling as an auxiliary objective.
Training. All models train for 32 epochs with Adam with decoupled weight decay and a cosine schedule, minibatch 32,768, mixed precision. The largest ResNet, RN50x64, took 18 days on 592 V100 GPUs; the largest ViT took 12 days on 256 V100s. ViT-L/14 was also trained for one extra epoch at 336-pixel resolution (ViT-L/14@336px); results reported simply as "CLIP" use this model.
Section · §3.1 · Table 1 · Figs. 4–6
Zero-shot transfer
For a dataset with K classes, embed each class name (inside a prompt) with the text encoder; the normalised text embeddings form the weights of a linear classifier, scaled by the temperature, applied to the normalised image embedding. The text encoder is thus a "hypernetwork" generating classifier weights; the class embeddings are computed once and cached.
Prompts (§3.1.4). The default template "A photo of a {label}." adds 1.3% on ImageNet over bare class names; task-specific context helps ("A photo of a {label}, a type of pet." for Oxford Pets); ensembling over 80 different prompts, averaged in embedding space, adds another 3.5%. Together prompt engineering and ensembling improve ImageNet accuracy by almost 5%, and average accuracy over 36 datasets by almost 5 points (Fig. 4).
| Zero-shot top-1 (Table 1) | aYahoo | ImageNet | SUN |
|---|---|---|---|
| Visual N-Grams (2017) | 72.4 | 11.5 | 23.0 |
| CLIP | 98.4 | 76.2 | 58.5 |
Against supervised baselines. Zero-shot CLIP beats a fully supervised logistic-regression classifier on ResNet-50 features on 16 of 27 datasets, including ImageNet (Fig. 5); it is weakest on specialised, complex or abstract tasks such as satellite imagery, lymph-node tumour detection, counting objects in synthetic scenes, and traffic-sign recognition. Zero-shot CLIP roughly matches a 4-shot linear probe on its own features (Fig. 6); few-shot probes only overtake it with more examples per class.
Section · §3.3 · Figs. 13–14
Robustness to distribution shift
Following Taori et al. (2020), "effective robustness" is accuracy under distribution shift beyond what in-distribution (ImageNet) accuracy predicts. ImageNet-trained models sit on a common trend line; zero-shot CLIP models lie well above it, reducing the gap between ImageNet accuracy and accuracy under shift by up to 75%.
Adapting to ImageNet (Fig. 14). Fitting a logistic-regression classifier on CLIP features with ImageNet labels raises ImageNet accuracy by 9.2 points but slightly lowers average accuracy across the shifted datasets: much of the gain reflects ImageNet-specific patterns that do not transfer. This is evidence that the robustness comes from avoiding training on the target distribution, not only from the representation.
Check yourself
- Why is comparing to a ResNet-101 with the same ImageNet accuracy the right control?
- Accuracy under shift correlates strongly with in-distribution accuracy; matching ImageNet accuracy isolates robustness beyond what better ImageNet performance alone would give.
Section · §6–7 · lineage
Limitations and lineage
- Compute to close the gap. Zero-shot CLIP is on average competitive with a linear classifier on ResNet-50 features, well below the state of the art; the authors estimate about a 1,000× increase in compute would be needed for zero-shot CLIP to reach overall state-of-the-art performance.
- Task blind spots. Weak on fine-grained classification (car models, flower species, aircraft variants), abstract and systematic tasks such as counting, and truly out-of-distribution data: 88% on MNIST, below logistic regression on raw pixels.
- Only chooses. CLIP selects among given concepts rather than generating outputs such as captions.
- Data efficiency. It does not fix deep learning's poor data efficiency; it compensates with scale (32 epochs over 400 M images is about 12.8 billion images seen).
- Bias and surveillance (§7). Class design strongly affects outcomes, including denigration harms; the authors probe gender and race classification and surveillance-style uses, and caution against deployment without task-specific testing.
Reference
Glossary
- Contrastive loss
- Classifies the correct pair among a batch of candidates (InfoNCE / N-pair).
- Embedding
- A vector representing an image or text in the shared space.
- Logit scale
- et = 1/τ, multiplying cosine similarities before softmax.
- Zero-shot
- Applying the model to a new task with no task-specific training examples.
- Prompt ensembling
- Averaging text embeddings of several templates per class.
- Linear probe
- Logistic regression trained on frozen features.
- Effective robustness
- Accuracy under distribution shift beyond the trend predicted by in-distribution accuracy.
- WIT
- WebImageText, CLIP's 400 M-pair training set.