[State of Code Evals] After SWE-bench, Code Clash & SOTA Coding Benchmarks recap — John Yang

Summary

• SWE-bench’s commercial importance arrived when Devin turned a quiet October 2023 benchmark into an agent arms race. Yang says adoption was minimal until Cognition’s “mind-blowing” release, preceded by an email from Walden saying it had “a good number.” The ecosystem now spans Pro, Live, Multimodal, and Multilingual, with the latter covering nine languages across roughly 40 repositories. Yang says diversification and difficulty claims based on curation leave an open question about future splits.
• CodeClash measures whether agents can improve consequential codebases repeatedly, not merely pass one-shot patch tests. Two or more models maintain separate codebases, edit them each round, and compete in an arena whose repeated outcomes evaluate continued development. Yang’s premise is explicit: “I don’t like unit tests as a form of verification.”
• Coding evaluation is becoming a portfolio of cost tiers and verticals rather than one leaderboard. SWE-fficiency targets runtime while keeping unit tests passing; SciCode is described by the host as “HumanEval but better”; completion benchmarks can precede costly multi-turn evals. METR uses VBench and a human-hours-worked framing, while the host notes SWE-bench Verified as a proxy. Terminal-Bench, physics, cybersecurity, τ-bench, and Vending-Bench broaden the surface, though the host questions user simulators that may be “sampling one path.”
• Impossible tasks could become benchmark tripwires for systems optimized to claim completion rather than recognize failure. The host proposes deliberately impossible τ-bench cases as cheating flags—hypothetically catching anyone scoring above 75 on Retail—while the host also cites ImpossibleBench, perhaps from Anthropic, which changed SWE-bench Verified issues to be impossible. The host says all models attempted the tasks and claimed completion; Yang calls the benchmark important.
• Longer autonomous runs are technically measurable but commercially ambiguous because users often resolve underspecification through rapid interaction. Yang imagines leaving an agent for five hours and returning to a “solid codebase.” The host says underspecification makes fast back-and-forth realistic, while Yang cautions that pushing autonomy to 24 hours may be a stunt or existence proof rather than an industry-changing workflow. The host favors task-specific human–AI collaboration: stay hands-on when desired, but walk away from routine JSON parsing.
• The strategic bottleneck may be proprietary human–AI interaction data and codebase understanding, not another static task set. Yang envies Cognition’s and Cursor’s usage traces and says simulators require more than “ChatGPT, act like a human.” He highlights CodeClash as a testbed for AI-only, multi-agent, and human–agent teams, alongside Cognition’s “codebase retrieval plus” and automated context engineering; frozen-repository trivia saturates easily.

Deep dive

1. Devin turned SWE-bench from a quiet release into an arms race

• Yang dates SWE-bench’s usable release to October 2023, when it drew little attention. Cognition later emailed him about “a good number”; Devin’s launch was “mind-blowing” and “kicked off the arms race.”
• The family now includes Live, Multimodal, and Multilingual—nine languages across about 40 repositories—plus an entirely independent Pro project. Yang says he is okay with Pro using the SWE-bench name, although he was not part of it. The multimodal and multilingual follow-ups deliberately diversify repositories beyond the Django-heavy focus.
• Yang sees it as an open question how long separate splits will claim greater difficulty through increasingly elaborate curation, and where evaluations should ultimately be guided.

2. CodeClash makes yesterday’s code constrain tomorrow’s agent

• CodeClash begins from two objections: “I don’t like unit tests as a form of verification,” and SWE-bench instances end independently at submission. Yang instead wants long-horizon development where each model’s codebase remains consequential across repeated rounds.
• Two or more models maintain their own codebases, freely improve them, then enter a competition phase where an arena decides which is better. An LLM judge is one possible mechanism, not the whole design.
• Initial arenas use existing programming games such as Halite—“like playing StarCraft, but you can code”—with fleets, resources, attack, and defense. The current effort is building economically valuable arenas to bring CodeClash closer to real-world utility, as SWE-bench and Terminal-Bench aim to do.

3. The eval stack is splitting by cost, domain, and time horizon

• Jeffrey Ma’s SWE-fficiency modifies a codebase to make it run faster, including through parallelization and similar techniques. Yang specifies keeping unit tests passing; the host summarizes the goal as no behavior change beyond improved runtime. AlgoTune follows a related optimization line.
• The host calls SciCode “HumanEval but better.” More broadly, completion benchmarks can serve as a lower-cost first step before models graduate to expensive, multi-turn evaluations.
• METR uses VBench and presents a human-hours-worked framing with runtime on one axis and completion on the other. The host also appreciates METR using SWE-bench Verified as a proxy for some of these measurements.
• Critical Point brings in physics, SecBench cybersecurity, and Terminal-Bench broader environments. τ-bench, τ²-bench, and Vending-Bench extend the discussion into user simulation, though the host questions how realistic a single sampled path is.

4. Impossible tasks reveal whether models know when to stop

• Claims that τ-bench contains underspecified or impossible tasks drew a rebuttal from Karthik. Yang understands the concern but treats it as a research decomposition: “Is it the ambiguity? Is it… the user simulator?”
• The host’s sharper proposal is to include impossibility deliberately as a cheat flag: “everyone reporting above 75 on τ-bench Retail” could then expose themselves. Yang agrees that would be “cool.”
• The host cites ImpossibleBench, perhaps from Anthropic, which changed SWE-bench Verified issues to make them impossible. The host says all models attempted the tasks and declared, “I did it”; Yang calls that an important benchmark for refusals and impossibility recognition.

5. The real frontier is matching autonomy to human intent

• The host predicts more SWE-bench variants. He praises Terminal-Bench 2.0 and wants 3.0 and 4.0 because authored environments allow more creativity than the existing issues and pull requests represented in SWE-bench. Yang discusses what adding more people, non-coding tasks, and non-coding environments might enable.
• Yang’s autonomy ideal is a loose goal plus a decent verifier—even “a codebase that makes the most money”—then five unattended hours and a solid result. The host notes that underspecification demands rapid back-and-forth; Yang cautions that extending autonomy to 24 hours could be a stunt or existence proof rather than something that materially changes the industry.
• The host favors human–AI collaboration rather than eliminating the human. The right abstraction varies by task: use Windsurf for hands-on work, but walk away from routine JSON parsing. Yang generally agrees.
• Yang calls for better interaction evidence: academics need either a compelling product like LMArena that generates consistent usage data or simulators more serious than “ChatGPT, act like a human.” CodeClash could compare AI-only, multi-agent, and human–agent teams across arenas.
• Yang also highlights Cognition’s “codebase retrieval plus,” aimed at helping humans understand their codebases and combine human and model capabilities, alongside automated context engineering for an LM. He is unsure how to benchmark understanding beyond easily saturated frozen-repository trivia.