SAM 3: The Eyes for AI — Nikhila & Pengchuan (Meta Superintelligence), ft. Joseph Nelson (Roboflow)

Summary

• SAM 3 turns segmentation into a natural-language interface for detecting, masking, and tracking concepts across images and video. Atomic prompts such as “yellow school bus” find matching instances, while clicks or visual exemplars repair misses; video adds new-object detection alongside persistent tracking. Crucially, SAM 3, SAM 3D Objects, and SAM 3D Body are three separate models—not one system climbing into 3D.
• The strongest advantage claim concerns the data engine behind SAM 3, not merely the released weights. The new SA-Co benchmark expands evaluation from LVIS’s roughly 1.2K concepts to more than 200,000, while automated proposal and verification cut annotation from over two minutes to about 25 seconds per data point. Nikhila Ravi’s framing: competitive advantage increasingly sits in “the data engine to generate that data.”
• Recognition and localization were deliberately separated, including explicit training on concepts absent from an image. More than 70% of the training-dataset annotations are negative phrases describing concepts absent from an image, and a presence token first decides whether a concept exists before localization begins. Ravi’s practical observation was that “a single negative example goes a long way,” with three to five negatives often updating predictions, subject to viewpoint and domain caveats.
• The deployment evidence suggests SAM is already becoming computer-vision infrastructure. Roboflow reports 106 million SAM-powered Smart Polygon annotations—an estimated 100–130 years of saved labeling time—and 8 million SAM 3 inferences during its first five days. Ravi’s preferred standard is correspondingly practical: “The best eval is if it works in the real world.”
• SAM 3’s strategic role may be as both a specialist tool for multimodal models and the training signal that makes visual grounding native. The agent experiments show SAM 3 staying close to the full agent on short atomic prompts while language-model reasoning creates a large advantage on complex requests; the models can also correct one another’s errors. Pengchuan Zhang’s metaphor was simple: “Now we have a very good brain… and we have a very good eye.”
• Video remains the largest technical gap and therefore the clearest future-work surface. Tracking compute scales with detected objects, exhaustive video annotation remains expensive, and whole-masklet scoring improves accuracy only by sacrificing streaming latency. The roadmap discussed smaller edge-oriented models, more efficient video inference, end-to-end video training, and AI video annotators.
• The unresolved commercial problem is specifying intent, not recognizing pixels. A confidence slider cannot know whether a reflected car should count, and even “hand” can mean palm-only or palm-plus-arm depending on the customer’s ontology. Ravi’s formulation captures the last-mile opportunity for tooling, fine-tuning, and feedback loops: systems need to identify “human intention, not necessarily human knowledge.”

Deep dive

1. SAM 3 makes concepts the interface to visual understanding

• Ravi began by correcting the launch’s most common misconception: SAM 3 handles image-and-video understanding, while SAM 3D Objects and SAM 3D Body are “two completely separate models.” This release comprised three models, not a single SAM system extended into another dimension.

• SAM 3 detects, segments, and tracks from short concept prompts. “Watering can” returns its mask; “players in white” finds matching players, tracks them, and detects new instances entering later frames—an expansion from SAM 1 and SAM 2, where users generally clicked each target instance.

• Prompts remain editable rather than final. When “flower” misses an instance, a positive box can become a visual exemplar of the intended concept; the same masks can then drive trails, cloning, spotlights, background treatments, labeling, or downstream analysis.

• The host cited roughly 30 milliseconds for 100 detected objects on one H200. He also read the paper as showing approximately 10 objects on two H200s, 28 on four, and 64 on eight; Pengchuan did not confirm those figures but explained that the public video demo already uses parallel multi-GPU grounding.

2. Atomic concepts required a new benchmark, not another version bump

• Text prompting existed as a SAM 1 proof of concept, Ravi said, but became “the most highly requested feature.” For SAM 3, the team chose to master atomic visual phrases such as “yellow school bus” or “purple umbrella” rather than make unrestricted natural language the core task.

• Existing evaluation was too narrow for that ambition: LVIS contained about 1.2K unique concepts, while the new Segment Anything with Concepts, or SA-Co, benchmark contains more than 200,000. The premise is that natural language has a much larger vocabulary than a thousand concepts.

• Zhang hopes SA-Co will guide progress beyond the current model: stronger systems may arrive quickly, but a benchmark with measured human performance can guide progress toward—and eventually beyond—human-level concept segmentation and video grounding.

3. Real-world usage is becoming the decisive evaluation layer

• Ravi’s test was blunt: “The best eval is if it works in the real world.” Meta does not usually see every downstream deployment, making Roboflow’s production layer valuable both as distribution and as feedback on where an ostensibly general model actually works.

• Roboflow reports 106 million Smart Polygon examples powered by SAM 1, 2, or 3. Nelson estimated that automation saved roughly 100–130 human-years of annotation, depending on how the time is calculated, while SAM 3 alone handled about 8 million inferences during its first five days.

• The strongest specimen came from scientific imaging: the host saw researchers at the CZI Imaging Institute fine-tune SAM to separate structures inside what otherwise looked like “gray mush.” Nelson added examples involving neutrophil analysis, aerial imagery, underwater plastic removal, underwater species monitoring, manufacturing, and logistics.

• Nelson also sees slightly more than two research papers per day citing work from the Roboflow community. Because an individual publication can represent 6, 12, or 24 months of effort, his argument is that annotation acceleration compounds into faster scientific output—not merely cheaper polygons.

4. Negative examples teach SAM 3 when not to see something

• Ravi hoped that fine-tuning with just 10 data points could unlock more than the team can do themselves. Her favorite case is Waymo vehicles: generic “vehicle” works zero-shot, but “Waymo” may not; a 10-second San Francisco clip can provide enough examples for SAM 3 to begin specializing the distinction.

• There was no claimed universal positive-to-negative ratio. Ravi’s practical finding was that three to five negative examples alongside positive examples can substantially update the model’s priors, while viewpoint shifts, unseen subtypes, overhead imagery, and other distribution changes still require testing against the customer’s actual data.

• At pretraining scale, negatives dominate: Zhang said more than 70% of the annotations in the table he identified as Table 24 are phrases absent from their paired images. That volume trains the model “to not detect stuff that is not in the image,” rather than treating every phrase as present.

• The architectural mechanism is a presence token that separates recognition from localization: first determine whether the requested concept exists globally, then locate it. This simplifies the detector’s job and makes the presence decision explicit.

5. Separate detection and tracking resolve an identity conflict

• Ravi called SAM 3 “an entirely new approach,” not a version bump. One architecture now covers interactive segmentation, text prompting, open-vocabulary detection, and tracking—tasks previously served by separate specialist models.

• The detector must be identity-agnostic: every dog should match the shared concept “dog.” The tracker needs the opposite representation, preserving each dog as a distinct object over time; attempts at deeper unification exposed this task conflict, so the final design decouples the two.

• A shared Perception Encoder is a text- and image-aligned visual backbone. The diagram also combines a DETR-based detector, SAM 2 tracking components, Llama, and the data engine. In video, compute scales with the number of detected objects—not merely the number of requested classes—because every instance carries its own track.

6. Multimodal agents pair SAM 3’s eyes with language-model reasoning

• Atomic prompts are intentionally bounded, but users ask relational questions: identify the larger character, explain the feature distinguishing male and female in a picture, or ground a description requiring advanced language understanding and reasoning. The SAM 3 agent therefore lets a multimodal language model reason while calling SAM 3 for precise visual grounding.

• Zhang emphasized synergy rather than a clean brain-eye split: “SAM 3 is not perfect,” and the language model can recognize and correct some grounding errors. Conversely, SAM 3 supplies spatial evidence that a language model’s learned knowledge and reasoning do not reliably recover on their own.

• In an ablation Zhang recalled as roughly 30 on the validation set, SAM 3 alone—without the VLM—performed at that level. SAM 3 alone remained close to the full agent on short, training-like phrases, but the gap widened sharply on complex prompts; the host read Gemini 2.5 as the strongest result in the displayed agent table.

• Nelson’s live gauge comparison used SAM 3, the Gemini 3 Pro API, and Florence 2. SAM 3 was fast and segmented indicator lights, numbers, and a needle; Gemini produced boxes but missed numbers and added erroneous regions, while Florence 2 grouped much of the gauge together. Zhang was surprised because OCR-heavy images were intentionally not prioritized in data collection.

7. Exhaustivity became an automation problem

• The image pipeline begins with sourced images and captions, which are parsed to obtain noun phrases. SAM 3 proposes masks; one stage verifies each mask’s quality, another checks whether the accepted masks exhaust every instance, and humans draw only what remains missing.

• Model-generated proposals reduced annotation from more than two minutes per data point to about 45 seconds. Fine-tuned AI verification then removed the need for humans in those two verification tasks, bringing the process to roughly 25 seconds while retaining manual correction for omitted objects.

• The key breakthrough, Zhang said, was that both mask-quality and exhaustivity judgments could be automated. A fine-tuned Llama 3.2 verifier reached what he described as “superhuman performance” on those two verification tasks, and the resulting metadata also identifies which data points were difficult for the model.

• A fully autonomous engine remains a “dream,” not a completed claim: “There’s no free lunch,” and humans must still inject information on failures. Beyond human performance, Zhang expects vision to enter an RLHF-like regime because imitation from human-drawn answers is bounded by annotator quality, whereas judging “which one is better” is easier than constructing the answer from scratch.

8. Video automation still trails the image pipeline

• SAM 1’s SA-1B dataset could be annotated fully automatically, but Ravi said the team never achieved the equivalent for SAM 2 video or SAM 3 video. Video masks are sufficiently time-intensive that even collecting enough examples to train a capable verifier remains difficult.

• Zhang argued that the missing foundation—strong video multimodal models—only became practically usable later in the project period. Better video-language verifiers, more efficient annotation strategies, and SAM-side perception improvements are all required before image-style bootstrapping transfers cleanly.

• Whole-masklet detection scoring smooths information within a temporal window. Waiting until a partly visible person enters fully lets the model revise an uncertain early judgment, much as a human would; however, gathering evidence across the trajectory sacrifices streaming latency. Future work therefore includes end-to-end video training, smaller edge models, AI video annotators, and more efficient inference.

9. Simple grounding should become native to the AI “brain”

• Zhang’s diagnostic is a six-finger hand: frontier models may answer five from prior knowledge, while SAM 3 can explicitly locate and count six. Such grounding could repair errors that look like reasoning failures but originate in weak perception.

• His preferred future is hybrid but weighted toward native integration. Counting fewer than roughly 20 objects should be a fast, “System 1” capability; counting thousands in a crowded image may appropriately invoke a specialist model, drawing aids, or a longer-running tool workflow.

• Robotics makes that integration consequential: a laboratory robot deciding whether liquid in a test tube has reached the correct level needs perception and reasoning together. The host also raised the field’s competing bet on explicit world models and asked when it might cross over with SAM-style perception; that question remained open.

10. Human intent, not model confidence, defines the last mile

• The host challenged Roboflow’s vehicle demo: a confidence slider cannot express whether reflections in a building should count as cars. Sometimes the reflection is precisely the desired object; usually it is noise, and the initial prompt “vehicle” does not reveal which interpretation the user intends.

• Zhang described an iterative model/interface workflow in which the concept is specified more clearly over multiple passes. Nelson suggested that a model such as Gemini 3 could judge whether reflections are present, lower the threshold, and repeat the process automatically.

• Ravi then argued that a human in the loop remains important because users may not anticipate such failure cases; identifying “human intention—not necessarily human knowledge” is important for last-mile use.

• Even basic ontologies vary. Zhang has seen “hand” mean only the palm in one dataset and include part of the arm in another; both definitions are defensible, so poor benchmark transfer may reflect a user-specific concept rather than deficient eyesight. Few-shot fine-tuning and contextual tooling remain necessary even with an “omniscient” base model.

• Nelson positioned SAM 3 across the full lifecycle: text-prompted autolabeling, domain fine-tuning, and served or eventual edge inference. Ravi added that SAM 3 itself incorporated community datasets, benchmarks, and SAM 2 inference optimizations. Pengchuan’s stated priorities were smaller and more efficient models, better video performance, end-to-end video training, and AI video annotators; Nelson also discussed likely user requests around documents and OCR, spatial relations, action recognition, robotics, and vision-language-action tasks.