Summary

ARC-AGI-2 resets the benchmark around tasks that humans can solve but frontier AI systems still largely cannot. Roughly 400 people calibrated the dataset, and every task was solved by at least two humans within two attempts, while pretrained models score effectively 0% and reasoning systems remain in single digits. Mike Knoop rejects the charge that ARC is moving the target: “The goalpost is AGI.”
The early scorecard exposes a vast gap between test-time adaptation and human-level fluid intelligence. GPT-4.5 and similar base models score around 0%; the winning 2024 Kaggle approach gets roughly 3%, an ensemble reaches 4%, and o3 is extrapolated near 4% on low compute, versus average human performance around 60%. François Chollet guesses an o3-style high-compute run might reach 15–20%, but only by spending perhaps $10,000 per task.
o3 is presented as an important discontinuity, not evidence that AGI has arrived. On ARC-AGI-1, its efficient configuration scored about 75%, while a roughly 200-times-more-compute setting reached about 85%; Chollet calls it “a kind of proto-AGI” because it appears to have nonzero fluid intelligence. The two decisive shortcomings are that it remains far below human performance on ARC-AGI-2 and can consume thousands of dollars of compute per task.
Efficiency is the benchmark’s central economic and technical claim, not an optional constraint. Unlimited brute-force program search could theoretically solve ARC tasks, but Chollet argues that intelligence means finding the right program “in very few hops” with little compute, as humans do in two to four minutes. His blunt formulation: “Efficiency is the problem statement; it’s not capability.”
The episode’s key architectural wager is that o3 performs genuine test-time search over natural-language programs. Knoop and Chollet infer that systems such as o1 Pro and o3 can sample, select, and recompose chains of thought, unlike R1 or ordinary autoregressive models; Chollet therefore calls o3 “qualitatively different.” Both hedge that the internals are unconfirmed, but argue that ARC performance, latency, and cost are consistent with active search.
ARC-AGI-2 was engineered to remove the shortcuts that weakened ARC-AGI-1. Roughly half of ARC-AGI-1’s private set could be solved by basic brute-force program search, whereas such methods should reach only 1–2% on version 2. The new tasks preserve the grid format but require deeper chains of interacting rules—such as one operation writing information that another must subsequently read.
ARC Prize 2025 is betting that AGI progress remains idea-constrained enough for small teams to matter. The competition runs through the end of 2025, uses Kaggle plus semi-private and private evaluations, and keeps the unclaimed grand prize conditional on an open-source, highly efficient solution. The foundation’s thesis is that “a lot of sharing” and “a lot of diversity of approach” offer a better innovation environment than a closed, secretive, monocultural one.

Deep dive

1. ARC-AGI-2 measures the human capability gap, not benchmark prestige

Knoop’s response to the moving-goalposts challenge is categorical: “The goalpost is AGI.” ARC Prize exists to drive the gap between humans and computers to zero; while straightforward human tasks remain impossible for machines, the benchmark still has meaningful work to measure.
Most frontier benchmarks pursue “PhD-plus” capabilities that ordinary humans cannot exercise without extensive education. ARC instead asks what simple human abilities AI still lacks, because Knoop sees those residual gaps—not already-superhuman specialist performance—as the more revealing test of general intelligence.
The practical ambition is scientific innovation. A system governed by pretraining and memorization can reflect humanity’s accumulated experience, but Knoop argues it cannot reliably produce the new knowledge and technology needed to “compress science timelines.” Closing the easy-for-humans, hard-for-AI gap is meant to target that missing adaptability.
His product-building lesson from Zapier reinforces the need for benchmarks: classic software tested with five users may behave identically at millions-user scale, while AI must be deployed broadly before its failures become legible. A benchmark supplies the stable measurement that ordinary interaction cannot.

2. Compositional tasks shut down ARC-AGI-1’s brute-force shortcuts

Knoop describes ARC-AGI-1 as having “pretty significant flaws.” After the 2020 Kaggle competition, analysis showed that roughly half its private dataset had been solved at least once by basic brute-force program-search methods, meaning those tasks offered little signal about intelligence.
ARC-AGI-2 retains input-output grids of at most 30 by 30 and the same core-knowledge premise, but removes redundant, single-rule tasks. Brute-force approaches should now score no more than roughly 1–2%, while pretrained language models also remain effectively at zero.
The simplest illustration is flipping objects: alone, flipping can be pretrained or hard-coded. Adding falling introduces composition, but a DSL containing gravity and flipping may still enumerate the answer; the stronger tasks chain interacting rules deeply enough that exhaustive search becomes prohibitively expensive.
The host’s proposed recursive ARC—nesting several puzzles—would make problems harder, but Chollet says simple concatenation is not deep composition because the subtasks remain independently solvable. Worse, it works “only really once”: after developers discover the decomposition, “it’s game over.”

3. Human calibration turns “easy for people” into measured evidence

To replace ARC-AGI-1’s anecdotes, the team recruited roughly 400 people through a San Diego testing center, ranging from Uber drivers and single mothers to UCSD students. Every ARC-AGI-2 task was solved by at least two humans within two attempts, matching the rules imposed on AI systems.
The host pushes back that some puzzles demanded five or six minutes of deep thought and asks whether ARC is “scraping the barrel.” Knoop calls difficulty relative: these tasks were solved in roughly five minutes on a $5-per-task solve-rate budget, yet current AI often cannot solve them at all—or requires thousands of dollars per task.
The resulting bandwidth is wide: pretrained systems such as GPT-4.5 score about 0%, the winning 2024 Kaggle approach about 3%, an ensemble of top entries 4%, and low-compute o3 an extrapolated 4%. Average human performance should be closer to 60%, leaving approximately 95% of the dataset available to measure improvement.

4. o3 establishes fluid intelligence without reaching human intelligence

Knoop recalls expecting a quiet December after publishing the 2024 technical report, only for OpenAI to request urgent o3 testing that same day. The resulting two-week sprint changed his view: “Innovation is a step function,” making smooth timeline forecasts unreliable.
On ARC-AGI-1, o3 scored about 75% within the public leaderboard’s efficiency budget; a configuration using perhaps 200 times more compute reached about 85%. Chollet later references the high-compute system as scoring roughly 88%, preserving the broader conclusion that scale bought a large but extremely expensive gain.
Knoop’s strongest claim is that this constitutes “an existence proof that computers are able to do something that they’ve never been able to do before in the history of humanity”: adapting to novelty, albeit narrowly. He still sees “a long way to go” before AGI and urges serious study rather than treating o3 as more pretraining-scale continuity.
Chollet calls o3 perhaps the first model showing fluid intelligence and “a kind of proto-AGI,” but with two major flaws: efficiency and level. Human-level intelligence would imply something like 60% on ARC-AGI-2; the current estimated o3 score is only about 4–5%.

5. The o3 dispute is about test-time search, not training-set contamination

Critics objected that OpenAI fine-tuned on ARC’s training set, but Knoop calls this “a very poor critique.” ARC explicitly provides training examples to teach the domain, then tests abstraction on a dissimilar private set—his analogy is teaching grade-school mathematics before examining the system on calculus.
The host presses harder, asking whether o3 may have trained on API data and how a vanilla release would compare with OpenAI’s adapted version. Knoop’s answer remains appropriately narrow: OpenAI said it trained only on the provided training set, “and I believe that’s what they did”; ARC Prize will test the released model.
Direct output prediction does not, in Chollet’s account, eliminate structured reasoning. The system may look like end-to-end transformation from outside, but internally it appears to synthesize a chain of thought—a task-specific “recombination artifact”—and then execute that natural-language program to produce the output.
Knoop speculates that o1 Pro and o3 can multisample and recompose chains of thought, creating structures absent from pretraining; he stresses that this is “informed speculation,” not confirmed architecture. His preferred unit of analysis is therefore an AI system combining a deep-learning model with a synthesis engine, not a single model.
Chollet also contrasts the regimes empirically: roughly 50,000-times scaling from GPT-2 to GPT-4.5 moved ARC-AGI-1 from 0% to only about 10%, while ARC-AGI-2 remains effectively at 0% without test-time adaptation.

6. Intelligence is both a threshold and an efficiency spectrum

When the host suggests that half a billion or a billion dollars could saturate ARC-AGI-2, Chollet concedes that unlimited computation could enumerate programs in a Turing-complete DSL until finding the shortest consistent solution. But that brute-force capability already existed conceptually in 2020 and is not what ARC means by intelligence.
Humans solve a task in two, three, or four minutes with “almost zero” energy, while high-compute o3 can use more than $3,000 of compute. Intelligence therefore concerns how efficiently a system acquires and deploys capabilities: “finding that program in very few hops,” not merely proving that some budget can eventually find it.
Fluid intelligence is binary at the boundary between static memorized skills and any nonzero adaptation, then continuous in degree. Once recombination exists, the questions become how deeply knowledge can be recombined, how large a program can be written on the fly, and how quickly and efficiently it can be synthesized.
Chollet says he could not test o3 as extensively on ARC-AGI-2, but reports version-1 failure modes that illuminate the challenge: performance declines exponentially as objects and interacting rules multiply; nonverbal patterns are harder because the model must “talk about” its solution; distant information is harder to combine than nearby information; and applying one rule, then reading its output with another, remains out of reach.

7. ARC Prize is building an open runway toward benchmarks beyond ARC-AGI-2

The foundation emerged after public demand repeatedly exceeded Knoop’s expectations—thousands called for o1 preview testing, then followed the contest results and o3 evaluation. Its intended role is to remain a “North Star for AGI,” continually publishing durable tests of whatever straightforward human capabilities machines still lack.
ARC Prize 2025 keeps the previous contest structure: a Kaggle leaderboard, semi-private testing during the competition, and a private final evaluation through the end of 2025. The grand prize remains unclaimed, and winning it requires an open-source solution with enough efficiency to run under Kaggle constraints.
Chollet expects ARC-AGI-2 to last perhaps one or two years, not ARC-AGI-1’s five. He would “basically rule out” an efficient human-level solution by the end of 2025, says end-2026 is possible, and describes ARC-AGI-3 as challenging systems that do not yet exist; beyond recombination, intelligence must eventually include efficient information gathering, action, goal-setting, and risk-aware exploration.

Gili Raanan - Cybersecurity Investment Playbook - [Invest Like the Best, EP416]Plural Investing's Chris Waller on the entertainment and hospitality turnaround at Seaport $SEG