Study guide · Computer vision · 2023

Segment Anything

Alexander Kirillov et al. · ICCV 2023

Can one model outline any object in any image, the way large language models answer almost any question, without being retrained for each new kind of picture? Segment Anything builds such a model: you click a point or draw a box and it returns the object's outline. It works on images it has never seen because it was trained on a new dataset of 1.1 billion outlines, which the model itself helped to create.

The paper in four sentences

The authors define a simple, general task: given an image and a hint (a click, a box, a rough sketch of a region), produce a sensible outline, or "mask", of the object meant.

The model, SAM, studies each image once with a large network, then answers each new hint in about a twentieth of a second, and when a hint is ambiguous it offers three possible answers (part, larger part, whole object).

To get enough training data, they ran a "data engine": the model helped people draw outlines, was retrained on them, helped more, and finally drew 1.1 billion outlines on 11 million photos by itself.

Tested on 23 segmentation datasets it was never trained on, SAM's outlines from a single click were often as good as or better than specialised tools', though it is not the best at every task.

Idea · §1 · §2 · Fig. 1

A "foundation model" for outlines

Segmentation means finding exactly which pixels belong to an object: not just "there is a cat here" but the cat's precise outline. Earlier tools, like the U-Net, were trained for one job on one kind of image: cells under a microscope, or cars on a road. A new job meant new labelled data and new training.

Language models had shown another way. A model trained on enough text can do many tasks it was never specifically trained for, if you phrase the request as a prompt. CLIP did something similar for recognising images. The paper asks: can we build the same kind of general-purpose, promptable model for segmentation?

It identifies three ingredients, and the paper is organised around them: a task general enough to cover many uses (outline whatever the prompt indicates), a model that supports it and responds fast enough to use interactively, and the data to train it, which did not exist yet.

Everyday pictureThink of a skilled assistant with scissors. You don't explain what a "cup" or a "leaf" is; you just point, and they cut out the thing you pointed at. If your point is ambiguous, they hold up a few options.

TakeawayMake segmentation promptable, like a language model, so one model can serve many tasks without retraining.

Idea · §2 · §3 · Fig. 3

Click, and get an outline (or three)

A prompt can be a click on the object (a positive point), a click on something to leave out (a negative point), a box, or a rough mask. The trouble is that a single click is often ambiguous. Click on the logo printed on a mug: did you mean the logo, the mug's body, or the whole mug including its handle?

SAM's answer is to return three masks for a single click, roughly a small part, a larger part and the whole object, each with its own confidence score. Adding more points removes the ambiguity, so it then returns just one. Try it on this toy scene.

Click the scene. With one point you get three nested candidates (click a thumbnail to show it); with more points, one mask that contains every "include" point and no "exclude" point. A hand-built stand-in: the scene's parts and their nesting are defined by hand, whereas SAM predicts masks from pixels. Try the logo, then add a point on the handle.

TakeawayWhen a prompt could mean several things, giving a few valid answers beats averaging them into one wrong one.

Check yourself
You click once on a plant's leaves. What three masks might SAM return?
Something like one leaf or the leaf cluster, all the foliage, and the whole plant including its pot.
Why does SAM return only one mask when given several points?
Several points usually pin down the intended object, so the three candidates would mostly agree.

Idea · §3 · Fig. 4

One long look, many quick answers

SAM has three parts. A large image encoder (a Vision Transformer) studies the whole image once and produces a compact summary of it. A small prompt encoder turns your clicks or box into numbers. A light mask decoder combines the two and draws the masks.

The split is the clever part. The heavy encoder runs once per image. After that, each new click needs only the prompt encoder and the decoder, which together take about 50 milliseconds, even inside a web browser on an ordinary processor. That is fast enough to feel instant while someone clicks around an image.

image image encoderViT-H, 636 M paramsrun once per image imageembedding prompt enc. clicks, box, mask maskdecoder mask 1mask 2mask 3 about 50 ms per prompt
Redrawn after Fig. 4. Everything inside the dashed area reruns for each prompt; the encoder's work is reused. Each mask also comes with a predicted quality score.

TakeawayDo the expensive understanding once per image; make each question about it cheap.

Idea · §4 · §5 · Figs. 5–6

A data engine: the model helps build its own dataset

No dataset of outlines was anywhere near big enough. Drawing an accurate outline by hand takes a person tens of seconds, and there are millions of objects to outline. So the team built the data in three stages, with the model improving at each one.

Cumulative masks collected (log scale: each grid line is ten times the one before). Numbers from §4 and §5. Stage 3 produced 1.1 billion masks, 99.1% of all masks in the final dataset, SA-1B.

The result, SA-1B, has 11 million licensed, privacy-protected photos (faces and licence plates blurred) and 1.1 billion masks, about 100 per image and roughly 400 times more masks than the largest earlier segmentation dataset. Were the automatic masks any good? When professional annotators corrected a sample, 94% of automatic masks overlapped their corrected versions by more than 90%.

Everyday pictureAn apprentice starts by watching the master, then does the easy parts while the master fixes mistakes and handles the hard parts, and finally works alone. Each round, the apprentice learns from the work just done.

TakeawayThe model, the annotators and the dataset improved together until the model could label data at a scale no team of people could.

Check yourself
Why did stage 2 ask people to focus on objects the model had missed?
The model already found the obvious objects confidently. Human effort was most valuable on the less prominent objects, which made the masks per image more diverse.

Idea · §7 · §8

How well it generalises, and where it falls short

The team tested SAM on 23 existing segmentation datasets covering very different images (underwater scenes, X-rays, microscope images, paintings, driving scenes, and more), none of them seen in training. From a single click, SAM beat a strong specialised interactive tool (RITM) on 16 of the 23, and human raters judged SAM's masks to be of higher quality. Without any retraining, simply by choosing what prompts to give, it could also find edges, propose objects, and outline objects found by a separate detector, though specialised systems trained for those tasks remained better.

The authors list the weaknesses: SAM can miss fine structures, sometimes adds small stray blobs, produces less crisp boundaries than heavier "zoom-in" methods, and is not fully real-time once you count the big image encoder. Text prompts ("a wheel") worked only in an early, exploratory form.

Later refinementSAM 2 (2024) extended promptable segmentation to video, with a memory that tracks an object through frames. Lighter versions (such as MobileSAM and EfficientSAM) replaced the huge image encoder to run on phones. Systems like Grounded-SAM combined it with a text-based object detector to get the "outline every wheel" ability that text prompting in the original only hinted at.

TakeawayA single promptable model trained on enormous data generalised well across many kinds of images, though dedicated systems still win on their own specialities.

Follow the idea

Built on: U-Net (2015), ViT (2020)