Learn AI Create the Future — with Pranay
Main site

Segmentation in Depth

Segment Anything

SAM cuts out whatever you point at, without ever being told what it is, and returns several answers because one click is genuinely ambiguous.

Read these first

On this page 9
  1. The short answer
  2. The analogy
  3. Why it exists
  4. How it works
  5. The three answers
  6. What it does not do
  7. Where you have already seen this
  8. Remember this
  9. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

Point at something in a photo, and Segment Anything cuts it out — without knowing what it is called.

The analogy

Stand at a street food stall and point at a dish. The vendor serves it without you naming it.

Now point at a button on your friend's shirt and ask someone to cut around "that". A reasonable person hesitates.

Do you mean the button? The shirt? Your friend? All three are correct readings of the same finger. A helpful person offers all three and lets you choose.

That hesitation is not a flaw in the question. It is a real property of pointing, and the model was built to handle it rather than hide it.

Why it exists

Every model before it had a fixed list. Trained on cars and people, it found cars and people. Show it a circuit board or a cell under a microscope and it found nothing.

Retraining meant collecting thousands of hand-outlined examples, which for most teams is the end of the idea.

Segment Anything moved the question. Not "what are the classes?" but "what did you point at?". Pointing works on any object, including ones nobody has named.

The model was trained on around eleven million photos and over a billion outlines. That is why it handles objects nobody has ever named.

How it works

   photo ---> [ heavy reading step ]  ---> a summary of the photo
                (slow, runs ONCE)              |
                                               |  kept in memory
   your click -> [ tiny cut-out step ] <-------+
                       |
                       +--> three candidate outlines
                       +--> a confidence number for each

The split in that diagram is the practical heart of it. Reading the photo is the expensive part, and it happens once.

After that, every click is answered by a small, fast step. That is why the demos feel instant. The photo was read before you started clicking.

Think of a friend who reads a document once, then answers twenty questions without opening it again.

The three answers

The model returns three outlines for every click, plus its own opinion of how good each one is.

This is the button problem from the analogy, built into the design. One answer would have to average the button, the shirt and the person. An averaged outline is useful to nobody.

What it does not do

It has no idea what anything is called. Point at a dog and you get a very good dog-shaped outline labelled nothing at all.

That is not a bug. Naming was left to other models on purpose. That keeps this one useful for microscope slides and satellite images.

The newest version does accept short phrases like "yellow school bus" and finds every one in the picture. That capability came later, and it is a genuine change in what the family can do.

Where you have already seen this

  • "Select subject" and one-tap cut-out tools in photo editors.
  • Labelling tools where you click an object and the outline appears, instead of tracing it by hand.
  • Video editors that follow a selected object across frames.

The labelling use is the quiet giant. Outlining an object by hand takes a minute; clicking takes a second. Whole datasets are now built that way.

Remember this

  • You point, it outlines. No class list anywhere.
  • Reading the photo is slow and happens once; each click after that is fast.
  • It returns three answers because one click really is ambiguous.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install torch

The experiment below runs on CPU in about forty seconds and downloads nothing. It reproduces the single design decision that makes SAM usable: multiple mask outputs trained by a minimum-over-outputs rule.

Why one output head cannot work

A click on the badge is a valid prompt for the badge, the shirt or the person. The training data contains all three answers, and the model is never told which one the user meant.

ambiguity.py
import torch, torch.nn as nn, torch.nn.functional as F

S = 24
def sample(gen):
    """Nested objects: a badge, on a shirt, on a person. The click always lands on the badge."""
    r = torch.randint(2, 6, (1,), generator=gen).item()
    cy = torch.randint(r + 5, S - r - 5, (1,), generator=gen).item()
    cx = torch.randint(r + 5, S - r - 5, (1,), generator=gen).item()
    def box(h):
        m = torch.zeros(S, S); m[cy - h:cy + h + 1, cx - h:cx + h + 1] = 1.0; return m
    masks = torch.stack([box(r), box(r + 2), box(r + 4)])       # badge, shirt, person
    img = 0.3 * masks[2] + 0.3 * masks[1] + 0.4 * masks[0]
    prompt = torch.zeros(S, S); prompt[cy, cx] = 1.0            # the click, as a second channel
    return torch.stack([img, prompt]), masks

def batch(n, gen):
    xs, ms = zip(*[sample(gen) for _ in range(n)])
    return torch.stack(xs), torch.stack(ms)

class Decoder(nn.Module):
    """n_out mask predictions plus a confidence score for each, like SAM's mask decoder."""
    def __init__(self, n_out):
        super().__init__()
        self.trunk = nn.Sequential(nn.Conv2d(2, 16, 3, padding=1), nn.ReLU(),
                                   nn.Conv2d(16, 16, 3, padding=1), nn.ReLU(),
                                   nn.Conv2d(16, 16, 3, padding=1), nn.ReLU())
        self.mask_head = nn.Conv2d(16, n_out, 1)
        self.iou_head = nn.Linear(16, n_out)
    def forward(self, x):
        h = self.trunk(x)
        return self.mask_head(h), self.iou_head(h.mean((-2, -1)))

def iou(logit, gt):
    p = (logit > 0).float()
    return (p * gt).sum((-2, -1)) / ((p + gt) > 0).float().sum((-2, -1)).clamp(min=1)

def train(n_out, steps=1500, seed=0):
    torch.manual_seed(seed)
    net = Decoder(n_out); opt = torch.optim.Adam(net.parameters(), lr=3e-3)
    gen = torch.Generator().manual_seed(seed + 100)
    for _ in range(steps):
        x, masks = batch(16, gen)
        pick = torch.randint(0, 3, (x.shape[0],), generator=gen)   # what the user meant; the model is not told
        gt = masks[torch.arange(x.shape[0]), pick]
        out, conf = net(x)
        per_head = torch.stack([F.binary_cross_entropy_with_logits(out[:, k], gt, reduction="none").mean((-2, -1))
                                for k in range(n_out)])            # (n_out, batch)
        winner = per_head.argmin(0)
        rows = torch.arange(x.shape[0])
        mask_loss = per_head[winner, rows].mean()                  # only the closest head is corrected
        with torch.no_grad():
            achieved = iou(out[rows, winner], gt)
        conf_loss = F.mse_loss(conf[rows, winner], achieved)       # the head grades its own answer
        opt.zero_grad(); (mask_loss + conf_loss).backward(); opt.step()
    return net

gen = torch.Generator().manual_seed(999)
xv, mv = batch(200, gen)
for n_out in (1, 3):
    net = train(n_out)
    with torch.no_grad():
        out, conf = net(xv)
    print(f"\n{n_out} mask head(s), mean IoU over 200 clicks:")
    print(f"{'':10s}{'badge':>9}{'shirt':>9}{'person':>9}{'said IoU':>11}")
    for k in range(n_out):
        print(f"head {k}    " + "".join(f"{iou(out[:, k], mv[:, j]).mean():>9.2f}" for j in range(3))
              + f"{conf[:, k].mean():>11.2f}")
    best = torch.stack([torch.stack([iou(out[:, k], mv[:, j]) for k in range(n_out)]).max(0).values
                        for j in range(3)]).mean(1)
    print("if the user picks the head they wanted:", "  ".join(f"{v:.2f}" for v in best))
Output

1 mask head(s), mean IoU over 200 clicks:
              badge    shirt   person   said IoU
head 0         0.43     1.00     0.55       0.63
if the user picks the head they wanted: 0.43  1.00  0.55

3 mask head(s), mean IoU over 200 clicks:
              badge    shirt   person   said IoU
head 0         0.24     0.55     1.00       1.02
head 1         1.00     0.43     0.24       1.03
head 2         0.43     1.00     0.55       0.97
if the user picks the head they wanted: 1.00  1.00  1.00

Seeded, on PyTorch 2.5.1 running on CPU. Small variations across versions are expected; the pattern is robust.

Reading that result

The single head picked a favourite and lived with it. It settled on the middle answer, scoring 1.00 on the shirt and 0.43 and 0.55 on the other two. Nothing in the loss let it do better, because the same input arrives with three different correct answers on different steps.

The three heads specialised without being told to. Head 1 took the badge, head 2 took the shirt, head 0 took the person, each at 1.00. No part of the code assigns a head to a scale. The minimum-over-heads rule did it: whichever head happened to fit best got the gradient, and that reinforced its habit.

This is winner-take-all training, and it can collapse. If one head starts ahead on every example, it keeps winning and the others never learn. Diversity in the training data is what breaks the tie here, since object sizes vary from sample to sample. Real SAM training has that diversity in enormous quantity.

The confidence head is not decoration. The single-head model reports about 0.63, honestly signalling that it is often wrong. The three-head model reports near 1.0 on each. Ranking three candidate masks by predicted quality is how an application picks a default without asking the user.

The real models

sam_real.py
import requests, torch
from PIL import Image
from transformers import SamModel, SamProcessor

model = SamModel.from_pretrained("facebook/sam-vit-base", device_map="auto")
processor = SamProcessor.from_pretrained("facebook/sam-vit-base")

img_url = "https://huggingface.co/ybelkada/segment-anything/resolve/main/assets/car.png"
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert("RGB")
input_points = [[[450, 600]]]                       # one click, in pixel coordinates

inputs = processor(raw_image, input_points=input_points, return_tensors="pt").to(model.device)
with torch.no_grad():
    outputs = model(**inputs)

masks = processor.image_processor.post_process_masks(
    outputs.pred_masks.cpu(), inputs["original_sizes"].cpu(), inputs["reshaped_input_sizes"].cpu()
)
scores = outputs.iou_scores                          # the model grading its own three masks

No output block, deliberately. That snippet downloads weights and an image over the network, and the mask contents depend on both. Inventing numbers here would teach you to expect something specific that may not appear.

What is stable: pred_masks carries three masks per prompt by default, and iou_scores carries the matching three confidences. Written against transformers 5.6.2.

Download sizes, checked on the Hub: facebook/sam-vit-base is 375 MB, facebook/sam-vit-huge is 2.56 GB, facebook/sam2.1-hiera-tiny is 156 MB, facebook/sam3 is 3.44 GB. On CPU the base model takes seconds per image for the encoding step. Anything interactive wants a GPU for that step, or a cached embedding.

Text prompting arrived with SAM 3:

sam3_text.py
import requests, torch
from PIL import Image
from transformers import Sam3Model, Sam3Processor

model = Sam3Model.from_pretrained("facebook/sam3", device_map="auto")
processor = Sam3Processor.from_pretrained("facebook/sam3")

image = Image.open(requests.get("http://images.cocodataset.org/val2017/000000077595.jpg",
                                stream=True).raw).convert("RGB")
inputs = processor(images=image, text="ear", return_tensors="pt").to(model.device)
with torch.no_grad():
    outputs = model(**inputs)

results = processor.post_process_instance_segmentation(
    outputs, threshold=0.5, mask_threshold=0.5,
    target_sizes=inputs.get("original_sizes").tolist()
)[0]
print(f"found {len(results['masks'])} objects")

Same reasoning: a real download, so no fabricated output. The shape of the result is masks, boxes and scores, one row per matching instance.

Common mistakes

Recomputing the image embedding for every click. That throws away the entire design. Encode once, then reuse the embedding across prompts. In the Hugging Face API, call model.get_image_embeddings(pixel_values) once and pass image_embeddings= on subsequent calls.

Ignoring the three masks and taking index 0. Index 0 is not "the best". Rank by iou_scores, or expose all three.

Expecting labels. SAM 1 and SAM 2 return geometry, not names. Pair them with a classifier or a detector if you need a label, or use SAM 3's text prompting.

Feeding a box prompt in the wrong coordinate frame. Prompts are in original image pixels, and the processor handles the rescaling. Pre-scaling them yourself gives silently wrong masks.

Treating "everything mode" as free. Automatic mask generation samples a grid of clicks, typically 32 by 32, and runs the decoder for each. It is a thousand decoder passes, not one.

Try it yourself

In the toy script, change n_out to 2 and re-run. Two heads must cover three answers, and you can see which pair they choose and which answer becomes the compromise. Then set pick to always be 1 and watch the three heads become interchangeable, since the ambiguity is gone.

What to learn next

Researcher — Mathematics and papers.

The task

Kirillov et al. (2023) define promptable segmentation. Given a prompt, return a valid mask. A prompt may be points, a box, a coarse mask, or free text. Valid means one reasonable object matching the prompt, even when the prompt is ambiguous.

That definition is doing real work. It replaces "the correct mask" with "a correct mask", which is what makes ambiguity a first-class citizen rather than label noise.

Architecture:

  • Image encoder. MAE-pretrained ViT-H/16, roughly 632M parameters, run once per image, producing a $64 \times 64 \times 256$ embedding for a $1024 \times 1024$ input.
  • Prompt encoder. Positional encodings summed with learned type embeddings for points and boxes; a small convolutional path for mask prompts; a text encoder for free text.
  • Mask decoder. Two modified transformer decoder blocks with bidirectional attention between prompt tokens and image embedding, plus a dynamic mask head. Roughly 4M parameters.

The paper reports the prompt encoder and mask decoder running in about 50 ms in a web browser on CPU, given a precomputed image embedding. That number is the product specification the architecture was designed to hit.

Ambiguity, and the loss that handles it

The decoder emits three masks with predicted IoU scores. During training, the loss is backpropagated only through the mask with the lowest loss against the sampled ground truth. The IoU head is trained by mean squared error against the achieved IoU of that winning mask.

This is the minimum-over-outputs objective reproduced in the developer section. Two properties are worth stating precisely:

  • It is a hard assignment, so gradients are non-smooth and the specialisation that emerges is not unique across seeds.
  • It admits mode collapse when one output dominates early. Diversity in the prompt distribution is the practical safeguard, which is one reason SA-1B's scale matters beyond raw quantity.

The data engine

SA-1B was built in three stages. Assisted-manual, where annotators corrected model output. Semi-automatic, where the model proposed confident masks and annotators added what was missing. Fully automatic, where a $32 \times 32$ point grid ran over each image with quality filtering and de-duplication. The final release is over 1.1 billion masks on 11 million images, and 99.1 percent of masks came from the automatic stage.

The reusable lesson is not the scale. It is that a promptable model is a data engine for its own training data. A prompt is far cheaper to produce than a mask.

SAM 2 and SAM 3

SAM 2 (Ravi et al., 2024, arXiv 2408.00714) generalises to video by adding a streaming memory. A Hiera hierarchical encoder replaces the plain ViT, and a memory encoder writes per-frame representations to a memory bank. Memory attention then lets the current frame cross-attend to past frames and object pointers. It introduces the SA-V dataset, reported as 35.5M masks across 50.9K videos, and reports being about 6 times faster than SAM on images.

SAM 3 (arXiv 2511.16719, submitted November 2025) introduces promptable concept segmentation. Given a short noun phrase, an image exemplar, or both, it returns masks and identities for every matching instance. The architecture pairs an image-level detector with a memory-based video tracker sharing one backbone, and decouples recognition from localisation using a presence head. The reported data engine produced 4M unique concept labels, including hard negatives. The paper reports roughly doubling accuracy over existing systems on its new SA-Co benchmark, for both images and video.

SAM 3.1 (Meta, 27 March 2026) adds object multiplexing, tracking up to 16 objects in a single forward pass. Throughput rises from 16 to 32 frames per second on one H100.

Note the shift across three releases. SAM 1 deliberately excluded semantics; SAM 3 puts a limited, open-vocabulary form of semantics back in. The separation was a useful simplification, not a permanent principle.

Documented limitations

  • Thin structures and fine boundaries. SAM's masks are competitive on IoU and weaker on boundary metrics. HQ-SAM (Ke et al., 2023) addresses this with a learned high-quality output token and a small amount of extra training data.
  • Domain shift in medical imaging. Zero-shot SAM underperforms specialised models on many medical modalities. MedSAM (Ma et al., 2024, Nature Communications) fine-tunes on a large multi-modality medical corpus and is the standard reference point for that gap.
  • Cost. The ViT-H encoder is heavy. MobileSAM, EfficientViT-SAM and FastSAM trade accuracy for one to two orders of magnitude less encoder compute. SAM 2's Hiera-tiny is the mainline answer, at 156 MB.
  • No calibration guarantee. The predicted IoU score is a regression target, not a probability, and it is only as reliable as the training prompt distribution.

Where it fits in practice

The highest-value use today is not inference at all. It is annotation: bootstrapping instance masks from boxes or points, then training a small task-specific model on the result. See weakly supervised segmentation for how that pipeline is evaluated honestly, and why a student model trained on imperfect pseudo-masks can beat its teacher.

Papers

What to learn next