The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals

Summary

• OpenAI says SWE-bench Verified is no longer a reliable frontier coding signal because the roughly 500-task benchmark is “effectively saturated” and “highly contaminated.” The host notes that most frontier models are now in the 80-something range; at that level, 0.1-point gains can be meaningless, while contamination and narrow tests can make scores reflect repository familiarity or specific implementation choices rather than broad coding improvement. The field should move toward harder evaluations such as SWE-bench Pro.
• The benchmark’s decline does not erase the extraordinary investment that originally made it useful. OpenAI hired almost 100 software engineers, used three independent reviews, and curated roughly 500 real-world GitHub tasks from the academic SWE-bench set. But a later deep dive into problems o3 could not solve reliably found issues in over half the examined cases, especially tests demanding unspecified names, design choices, or extra features.
• Contamination is a major validity problem, not merely statistical noise. GPT-5.2 sometimes reasoned that a later repository version contained an argument omitted from the prompt, then considered adding what the hidden test expected; OpenAI also found it solving 31 problems in a set judged very hard to solve without contamination knowledge. A separate auditor found evidence across OpenAI models, Claude Opus 4.5, and Gemini Flash, including ground-truth regurgitation and, in some cases, task IDs.
• SWE-bench Pro offers more headroom because its tasks are “bigger and harder,” more diverse, and currently show much less contamination evidence. Roughly 90% of Verified tasks were estimated to take an expert under one hour, while Pro includes one-to-four-hour and four-plus-hour work across more repositories, languages, and problem types. Its contamination evidence was limited to possible light familiarity with one or two repositories.
• The next evaluation frontier is not another patch leaderboard but long-horizon engineering judgment. Mia Glaese and Olivia Watkins discussed tasks lasting hours or days, and tasks demanding work that top engineers might take weeks or months, plus open-ended design decisions, maintainability, performance work, and end-to-end product creation. The hard question is whether an open-source maintainer would have merged the result—something binary unit tests cannot answer.
• Evaluation infrastructure is moving toward expert labor, richer rubrics, and real-world usage metrics. GDPval’s use of professionals across roughly 15–16 white-collar occupations offers one template, while dollars, time, story points, and long-horizon measures are proxies for underlying task complexity. The consequential evidence will increasingly be actual augmentation, job replacement, speed gains, and research automation—not saturated benchmark deltas.

Deep dive

1. SWE-bench Verified was expensive to make—and genuinely useful

• Olivia Watkins’ core call: SWE-bench Verified was one of the field’s “north star coding benchmarks,” but progress has stalled because it is now saturated and contaminated. It no longer measures improvements in coding performance well enough to justify continued reliance.

• The original academic SWE-bench gave an agent a real GitHub repository and issue, then graded its patch through tests. OpenAI discovered that many failures came from defective task setups rather than models simply being incapable, prompting a major cleanup.

• Mia Glaese emphasized the scale: almost 100 real-world software engineers reviewed tasks in full repository context, with three experts independently reviewing each one before OpenAI selected roughly 500. On whether triple review was excessive: “We had to do it.”

2. Narrow tests and repository-history contamination undermine the benchmark

• A deeper dive examined problems o3 could not solve reliably. Reviewers found “one problem or the other” in over half the investigated problems.

• The common failure was an overly narrow test demanding an unspecified implementation detail—such as a particular argument or function name. Another reasonable name could produce a sound solution yet fail; some tests also required features never mentioned in the issue.

• The deeper implication, Mia argued, is asymmetric: passing generally indicates strong work, but failing does not establish bad engineering. The tests accept only a narrow subset of the “whole space of viable and good solutions.”

• Mia’s sharpest contamination example came from GPT-5.2, which reasoned that a later repository version used a particular argument and considered adding it despite no prompt instruction. OpenAI also found GPT-5.2 solving 31 problems in a set judged “very hard to solve without contamination.”

3. SWE-bench Pro provides more headroom, but no benchmark lasts forever

• Mia defended the benchmark’s history: when benchmarks measure important capabilities and models solve perhaps 20% or less, they give the field something meaningful to improve. At very high performance, however, additional 0.1% improvements can become meaningless, and agents may be measured by their ability to “correctly guess how to name a specific function.”

• SWE-bench Pro is harder and more diverse. About 90% of Verified tasks were estimated to take an expert less than an hour; Pro adds one-to-four-hour and four-plus-hour tasks spanning more repositories, languages, and qualitatively different work.

• A contamination-auditor agent probed models using task descriptions, patches, and IDs. It found ground-truth solutions and, in some cases, task IDs around Verified, but only “very light evidence” that some models might know one or two Pro repositories. Olivia’s caveat: Pro will eventually cease being the right benchmark too.

4. Useful coding evals must measure judgment, not just passing tests

• The next target is work beyond short GitHub issues: tasks lasting hours or days, open-ended performance improvements, design choices, code cleanliness, and maintainability. Mia’s framing was practical: “Does it solve the problem the way that my team likes to solve problems?”

• GDPval suggests a human-intensive route. Professionals across roughly 15–16 white-collar occupations helped create tasks, gold solutions, and rubrics requiring domain knowledge; applied to coding, that could assess whether maintainers would merge a PR. The tradeoff is speed: automated tests remain cheap, repeatable, and easy to compare industry-wide.

5. Capability tracking is shifting toward complexity and real-world impact

• Public evaluation of AI-research coding remains difficult because many state-of-the-art codebases are proprietary. Olivia still wants realistic public measures of research workflows, while acknowledging that evaluations built from private systems might not be released.

• Dollars, human time, story points, and long-horizon measures are different projections of an underlying variable: task complexity. The point is to quantify how much complexity—and therefore time, value, or autonomy—an agent can handle.

• OpenAI’s Preparedness Framework is a public framework for tracking dual-use frontier risks. It currently covers biological risk, cybersecurity, and research automation/model autonomy; coding is one important component of the last category. Olivia’s request to the field was concrete: tasks taking “top-notch engineers months or teams weeks,” validated rubrics, end-to-end product benchmarks, and metrics showing actual augmentation, replacement, and speedup in the real world.