Study guide · Computer vision · 2021

Learning Transferable Visual Models From Natural Language Supervision

Alec Radford et al. · ICML 2021

Instead of training a vision model on a fixed list of hand-labelled categories, what if it learned from the captions that already come with images on the internet? CLIP learns to match 400 million images with their own captions, and the result can sort images into categories it was never trained on, just from a written description of each category.

The paper in four sentences

Vision models were usually trained to pick one of a fixed set of labels, like ImageNet's 1,000, which is expensive to annotate and limits what they can recognise.

CLIP instead trains two networks, one for images and one for text, so that an image and its caption end up as nearby points, and mismatched pairs far apart; the 400 million training pairs were collected from the internet.

To classify a new image, you write a short description for each possible category ("a photo of a dog", "a photo of a cat") and pick the description that lands closest to the image: no extra training needed. This is called zero-shot classification.

Without seeing a single ImageNet training example, the best CLIP model matched the accuracy of a ResNet-50 trained on all 1.28 million of them, and held up much better on unusual images such as sketches.

Idea · §1 · §2.1–2.2

Captions are free labels

Image recognisers like AlexNet and ResNet learn from datasets where people have labelled every image with one category from a fixed list. That has two costs. Labelling millions of images is slow and expensive. And the model only ever knows those categories: to recognise something new, you need new labels and new training.

Meanwhile, the internet is full of images that already come with text: captions, alt text, titles. That text is noisy, but it is free, plentiful, and describes far more than 1,000 categories. Language models such as GPT-2 had shown what learning from raw web text could do. CLIP asks whether vision can learn the same way.

The authors built a new dataset of 400 million (image, text) pairs from the internet. To cover a wide range of concepts, they searched for pairs whose text contained one of 500,000 query words and phrases, keeping up to 20,000 pairs per query.

Everyday pictureA child doesn't learn what a giraffe is from a labelled flashcard deck alone. They hear "look at the giraffe!" while looking at one, and connect the word with the sight. CLIP learns from millions of such loose pairings.

TakeawayNatural-language captions give an almost unlimited, open-ended source of supervision for vision.

Idea · §2.3 · Figs. 2–3

Learning by matching, not by describing

The authors first tried teaching a model to write each image's caption word for word. It learned slowly, because there are countless correct ways to caption a photo and the model is penalised for not guessing the exact one. Simply predicting which words appear, ignoring order, was 3 times faster to train. Then they switched to an even easier task: given a batch of images and a batch of captions, just work out which caption goes with which image. That was another 4 times more efficient.

This matching approach is called contrastive learning. An image network turns each image into a point (a list of numbers called an embedding), and a text network does the same for each caption. Training pulls each image towards its own caption and pushes it away from all the other captions in the batch. Press Train to watch it happen in a small, two-dimensional version.

Left: embeddings on a circle (CLIP uses 512 or more dimensions; we use 2). Squares are images, circles are captions; matching pairs share a number (captions: 1 "a dog on grass", 2 "waves at sunset", 3 "red sports car", 4 "birthday cake", 5 "cat asleep"). Right: the batch's match table, where row = image, column = caption, and darker means a stronger match; training aims to make the diagonal dark. The update is the real CLIP loss and its gradient, run on this toy.
Everyday pictureA pairs game: a pile of photos and a pile of captions, all shuffled. You don't need to write a perfect caption, only to recognise which one belongs to which photo. That is much easier to learn, yet still forces you to understand what is in each picture.

TakeawayMatching images to captions among many distractors teaches a model what images contain, far more efficiently than generating the captions.

Check yourself
In a batch of 32,768 pairs, how many wrong pairings does each image get pushed away from?
32,767: every other caption in the batch. Large batches give many "negatives" per step, which is part of why CLIP used such a large batch.
Why was predicting the exact caption a slow way to learn?
Many different captions fit the same image, so the model spends its effort on wording rather than on recognising what is in the picture.

Idea · §3.1 · Fig. 1

Classify by writing the category names

After training, CLIP can classify images into categories it never saw as labels. Write a sentence for each category, such as "a photo of a dog." and "a photo of a cat.", turn each sentence into a point with the text network, turn the image into a point with the image network, and pick the closest sentence. Because nothing is trained for the new task, this is called zero-shot classification.

Wording matters. Using "A photo of a {label}." instead of the bare label raised ImageNet accuracy by 1.3 points, partly because single words are ambiguous ("crane" the bird or the machine?) and web captions rarely consist of a single word. Choose an image and the candidate categories below.

Toggle candidate categories on and off. Each becomes "a photo of a {label}." The bars show the probability CLIP-style scoring gives each, computed as a softmax of similarities multiplied by 100 (CLIP's trained scale). Embeddings here are hand-made from a handful of concepts, for illustration only. Note how removing the right answer makes the model confidently pick the nearest wrong one.

TakeawayA text description becomes a classifier: change the list of descriptions and you have a new classifier, with no retraining.

Check yourself
What happens if the correct category is not in your list?
The model still picks one, the closest description it has. Zero-shot classification can only choose among the options you write.

Idea · §3.1 · §3.3 · Table 1 · Fig. 13

How well it works, and why it is robust

The best CLIP model classified ImageNet zero-shot with 76.2% accuracy. That equals the original ResNet-50, which was trained on ImageNet's 1.28 million labelled images; CLIP used none of them. An earlier attempt at the same zero-shot idea (Visual N-Grams, 2017) had managed 11.5%.

The bigger surprise was robustness. Models trained on ImageNet often fail on images that look different from ImageNet photos: sketches, cartoons, unusual angles. Zero-shot CLIP and a ResNet-101 have the same ImageNet accuracy, but on sketches CLIP scores 60.2% against 25.2%, and on artistic renditions 88.9% against 37.7% (Fig. 13). Because CLIP never trained on ImageNet specifically, it hasn't latched onto the quirks of that one dataset.

Dataset (Fig. 13)ResNet-101Zero-shot CLIP
ImageNet76.276.2
ImageNet Sketch25.260.2
ImageNet-R (renditions)37.788.9
ImageNet-A (hard natural images)2.777.1

TakeawayLearning from broad web data rather than one curated dataset gave accuracy that transfers to new kinds of images.

Idea · §6 · §7

Limits, and a model everyone built on

CLIP is not good at everything. It got only 88% on handwritten digits (MNIST), which simple models handle almost perfectly, because few web images look like those digits. It struggles to count objects and with very fine distinctions such as car models or aircraft types. The authors estimate it would need around 1,000 times more compute for zero-shot CLIP to reach the best task-specific results overall. And because it learns from unfiltered internet data, it picks up the internet's social biases; the paper's own bias probes found that, depending on the category names offered, it could give offensive labels to images of people.

Later refinementCLIP became a building block. Its text encoder steers image generators such as DALL·E 2 and Stable Diffusion; its image encoder became the "eyes" of many vision-language chatbots and robot models, in the line that led to RT-2. Open reproductions (OpenCLIP, trained on the public LAION datasets) and variants with simpler losses (SigLIP, 2023) followed. The core recipe, match images and text contrastively at scale, has not been replaced.

TakeawayCLIP turned language into a universal interface for vision, at the price of the web's blind spots and biases.

Follow the idea

Built on: ViT (2020)

Led to: RT-2 (2023)