The paper in four sentences
The authors define a simple, general task: given an image and a hint (a click, a box, a rough sketch of a region), produce a sensible outline, or "mask", of the object meant.
The model, SAM, studies each image once with a large network, then answers each new hint in about a twentieth of a second, and when a hint is ambiguous it offers three possible answers (part, larger part, whole object).
To get enough training data, they ran a "data engine": the model helped people draw outlines, was retrained on them, helped more, and finally drew 1.1 billion outlines on 11 million photos by itself.
Tested on 23 segmentation datasets it was never trained on, SAM's outlines from a single click were often as good as or better than specialised tools', though it is not the best at every task.
Idea · §1 · §2 · Fig. 1
A "foundation model" for outlines
Segmentation means finding exactly which pixels belong to an object: not just "there is a cat here" but the cat's precise outline. Earlier tools, like the U-Net, were trained for one job on one kind of image: cells under a microscope, or cars on a road. A new job meant new labelled data and new training.
Language models had shown another way. A model trained on enough text can do many tasks it was never specifically trained for, if you phrase the request as a prompt. CLIP did something similar for recognising images. The paper asks: can we build the same kind of general-purpose, promptable model for segmentation?
It identifies three ingredients, and the paper is organised around them: a task general enough to cover many uses (outline whatever the prompt indicates), a model that supports it and responds fast enough to use interactively, and the data to train it, which did not exist yet.
TakeawayMake segmentation promptable, like a language model, so one model can serve many tasks without retraining.
Idea · §2 · §3 · Fig. 3
Click, and get an outline (or three)
A prompt can be a click on the object (a positive point), a click on something to leave out (a negative point), a box, or a rough mask. The trouble is that a single click is often ambiguous. Click on the logo printed on a mug: did you mean the logo, the mug's body, or the whole mug including its handle?
SAM's answer is to return three masks for a single click, roughly a small part, a larger part and the whole object, each with its own confidence score. Adding more points removes the ambiguity, so it then returns just one. Try it on this toy scene.
TakeawayWhen a prompt could mean several things, giving a few valid answers beats averaging them into one wrong one.
Check yourself
- You click once on a plant's leaves. What three masks might SAM return?
- Something like one leaf or the leaf cluster, all the foliage, and the whole plant including its pot.
- Why does SAM return only one mask when given several points?
- Several points usually pin down the intended object, so the three candidates would mostly agree.
Idea · §3 · Fig. 4
One long look, many quick answers
SAM has three parts. A large image encoder (a Vision Transformer) studies the whole image once and produces a compact summary of it. A small prompt encoder turns your clicks or box into numbers. A light mask decoder combines the two and draws the masks.
The split is the clever part. The heavy encoder runs once per image. After that, each new click needs only the prompt encoder and the decoder, which together take about 50 milliseconds, even inside a web browser on an ordinary processor. That is fast enough to feel instant while someone clicks around an image.
TakeawayDo the expensive understanding once per image; make each question about it cheap.
Idea · §4 · §5 · Figs. 5–6
A data engine: the model helps build its own dataset
No dataset of outlines was anywhere near big enough. Drawing an accurate outline by hand takes a person tens of seconds, and there are millions of objects to outline. So the team built the data in three stages, with the model improving at each one.
The result, SA-1B, has 11 million licensed, privacy-protected photos (faces and licence plates blurred) and 1.1 billion masks, about 100 per image and roughly 400 times more masks than the largest earlier segmentation dataset. Were the automatic masks any good? When professional annotators corrected a sample, 94% of automatic masks overlapped their corrected versions by more than 90%.
TakeawayThe model, the annotators and the dataset improved together until the model could label data at a scale no team of people could.
Check yourself
- Why did stage 2 ask people to focus on objects the model had missed?
- The model already found the obvious objects confidently. Human effort was most valuable on the less prominent objects, which made the masks per image more diverse.
Idea · §7 · §8
How well it generalises, and where it falls short
The team tested SAM on 23 existing segmentation datasets covering very different images (underwater scenes, X-rays, microscope images, paintings, driving scenes, and more), none of them seen in training. From a single click, SAM beat a strong specialised interactive tool (RITM) on 16 of the 23, and human raters judged SAM's masks to be of higher quality. Without any retraining, simply by choosing what prompts to give, it could also find edges, propose objects, and outline objects found by a separate detector, though specialised systems trained for those tasks remained better.
The authors list the weaknesses: SAM can miss fine structures, sometimes adds small stray blobs, produces less crisp boundaries than heavier "zoom-in" methods, and is not fully real-time once you count the big image encoder. Text prompts ("a wheel") worked only in an early, exploratory form.
TakeawayA single promptable model trained on enormous data generalised well across many kinds of images, though dedicated systems still win on their own specialities.
Abstract, restated
The Segment Anything (SA) project introduces a new task, model and dataset for image segmentation. Using the model in a data-collection loop, the authors built the largest segmentation dataset to date: over 1 billion masks on 11 million licensed and privacy-respecting images. The model is designed and trained to be promptable, so it can transfer zero-shot to new image distributions and tasks. Evaluated on numerous tasks, its zero-shot performance is impressive, often competitive with or superior to prior fully supervised results. SAM and SA-1B are released to foster research into foundation models for computer vision. ICCV 2023 (arXiv April 2023).
Section · §2
The promptable segmentation task
A prompt is any information indicating what to segment: foreground/background points, a rough box or mask, free-form text. The task is to return a valid mask for any prompt: even when a prompt is ambiguous and could refer to several objects (the paper's example: a point on a shirt could mean the shirt or the person wearing it), "the output should be a reasonable mask for at least one of those objects." This mirrors a language model producing a coherent response to an ambiguous prompt.
Pre-training. For each training mask a sequence of prompts (points, boxes, masks) is simulated and the predicted masks are compared with the ground truth, adapting interactive segmentation; unlike interactive segmentation, the goal is a valid mask for any prompt, even ambiguous ones, which drives the architecture and loss choices. Zero-shot transfer. Downstream tasks are solved by prompt engineering: e.g. a detector's boxes as prompts give instance segmentation.
Section · §3 · Fig. 4 · App. A
Model
Image encoder. An MAE pre-trained ViT-H/16 adapted to high resolution with 14 × 14 windowed attention and four equally spaced global attention blocks. Input 1024 × 1024 (rescaled, longer side, padded); output a 64 × 64 embedding (16× downscale), reduced to 256 channels. ViT-H has 636 M parameters; smaller ViT-B and ViT-L encoders are also reported.
Prompt encoder. Points and boxes: positional encodings summed with learned embeddings per prompt type (foreground point, background point, box corner). Text: CLIP's text encoder. Dense mask prompts: downscaled by convolutions and added element-wise to the image embedding.
Mask decoder. A modified Transformer decoder (two layers) operating on prompt tokens plus learned output tokens: self-attention on tokens, cross-attention token→image and image→token, and an MLP. The image embedding is then upsampled 4× with two transposed convolutions; each output token passes through an MLP to give a dynamic linear classifier, whose dot product with the upsampled embedding gives the mask probability at every location.
Ambiguity. The decoder predicts three masks at once (the authors found three enough for nesting such as whole, part and subpart). During training only the mask with the minimum loss is back-propagated, and a small head predicts each mask's IoU with the target, used to rank masks. With more than one prompt only a single mask is predicted, from a fourth output token, since ambiguity is much rarer and the three outputs usually agree.
Loss and training. Focal loss and dice loss in a 20 : 1 ratio for masks, mean squared error for the IoU head. Interactive simulation: 11 rounds per mask (an initial point or box, then points sampled from error regions of the previous prediction, with the previous mask logits fed back as a prompt, plus two rounds with no new points). Training on SA-1B: 90k iterations (about 2 epochs) across 256 GPUs.
Check yourself
- Why backpropagate only the minimum-loss mask rather than all three?
- Each output is then free to specialise on one valid interpretation; penalising all three against a single target would push them to collapse onto it, or to blur together across examples where the target differs.
- What is the IoU head for, given the masks are already predicted?
- It ranks the three candidates when a single answer is needed (e.g. in the automatic stage of the data engine) and filters unconfident masks.
Section · §4–5 · Figs. 5–7
Data engine and SA-1B
Assisted-manual. Professional annotators click foreground/background points in a browser tool powered by SAM, refining with pixel-precise brush and eraser, labelling "stuff" and "things" freely and moving on once a mask took over 30 seconds. SAM started from public segmentation datasets and was retrained 6 times on new data; time per mask fell from 34 s to 14 s and masks per image rose from 20 to 44: 4.3 M masks from 120 k images. Semi-automatic. Confident masks were pre-filled from a box detector trained on stage-1 masks (with a generic "object" class); annotators labelled the remaining objects. Masks per image rose from 44 to 72: 5.9 M more masks from 180 k images (10.2 M total), with 5 more retrainings. Fully automatic. The ambiguity-aware model was prompted with a 32 × 32 grid of points; masks were kept if the predicted IoU was high and the mask was stable (thresholding the probability map at 0.5 − δ and 0.5 + δ gives similar masks); NMS removed duplicates, and overlapping zoomed-in crops helped small masks. Applied to all 11 M images: 1.1 B masks.
SA-1B (§5). 11 M images averaging 3300 × 4950 pixels, released downsampled to a shortest side of 1500 pixels; about 1.1 B masks (1,129 M), 99.1% generated fully automatically. Quality check: on 500 images (about 50 k masks) professionally corrected, 94% of pairs have IoU above 90% and 97% above 75%. Compared with existing datasets, SA-1B has about 400× more masks than Open Images, more small and medium masks, and mask centres spread more evenly over the image (Figs. 5–7).
Section · §7 · Tables 3–5 · Figs. 9–12
Zero-shot transfer experiments
| Task | Setup | SAM | Reference |
|---|---|---|---|
| Single-point mask (Fig. 9) | 23 datasets, mIoU | better than RITM on 16 of 23 (by up to ~47 IoU); higher human ratings | RITM (interactive) |
| Edge detection (Table 3) | BSDS500, 16 × 16 grid of points, Sobel on mask probabilities | ODS 0.768, OIS 0.786, AP 0.794, R50 0.928 | trained edge detectors are better on ODS; SAM never trained for edges |
| Object proposals (Table 4) | LVIS v1, AR@1000 | 59.3 | ViTDet-H 63.0 |
| Instance segmentation (Table 5) | ViTDet boxes as prompts, mask AP | COCO 46.5, LVIS 44.7 | ViTDet-H 51.0, 46.6 |
| Text-to-mask (Fig. 12) | CLIP image embeddings at training, text embeddings at test | qualitative only | "exploratory and not entirely robust" |
In the single-point setting, SAM is also evaluated with an "oracle" that picks the most relevant of its three masks; with the oracle it beats RITM on all 23 datasets. On instance segmentation, although SAM's mask AP is lower, human raters scored its masks higher than ViTDet's on LVIS: the authors attribute part of the AP gap to SAM not learning the specific annotation biases (such as coarse COCO masks) that a model trained on that dataset absorbs. Human ratings for SAM's single-point masks fall between 7 and 9 on a 1–10 scale.
Check yourself
- Why might lower AP not mean worse masks here?
- AP measures agreement with the benchmark's own annotations, including their systematic quirks; a model trained on that dataset learns the quirks, while a zero-shot model may draw more accurate boundaries that score lower.
Section · §6 · §8 · lineage
Responsible AI, limitations and lineage
Responsible AI (§6). SA-1B has a higher share of images from Europe, Asia and Oceania and middle-income countries than earlier datasets; like all of them, it under-represents Africa and low-income countries, though every region still has tens of millions of masks. On perceived gender presentation, single-point mIoU for people is similar (54.4 feminine vs 55.7 masculine) with overlapping confidence intervals; for perceived skin tone differences are not significant; an appendix finds a gap for segmenting clothing (76.3 feminine vs 81.0 masculine at one point), which the authors flag.
Limitations (§8). Misses fine structures, hallucinates small disconnected components, and its boundaries are less crisp than computationally heavier methods that "zoom in"; dedicated interactive methods can beat it when many points are given; the decoder is fast but the full pipeline is not real-time with ViT-H; text-to-mask is exploratory; and SAM targets general segmentation, so domain-specific tools can outperform it in their domains. It is unclear how to design simple prompts for some tasks, such as semantic or panoptic segmentation.
Reference
Glossary
- Mask
- A binary per-pixel map marking one object or region.
- Prompt
- Points, box, mask or text indicating what to segment.
- Valid mask
- A reasonable mask for at least one object an ambiguous prompt could refer to.
- Output token
- Learned decoder query whose final state defines one mask (and its IoU score).
- IoU
- Intersection over union of two masks.
- Focal / dice loss
- Pixel losses that emphasise hard pixels and region overlap respectively.
- NMS
- Non-maximum suppression: removing near-duplicate predictions.
- Data engine
- Model-in-the-loop annotation, retraining the model as data accumulates.
- SA-1B
- The released dataset: 11 M images, 1.1 B masks.