Summary

ARC Prize 2024 showed that better ideas can neutralize a 1,000-fold compute advantage. Kaggle submissions ran for under 12 hours on one P100—roughly $10—while frontier-model entries could spend $10,000, yet both reached about 55%; even per task, the frontier track used roughly 200 times more compute. Chollet’s investable framing is blunt: “Compute is really a multiplier for ideas,” so raw compute is not a substitute for better methods.
The two successful paradigms were deep learning-guided program synthesis and test-time training, but they adapt in fundamentally different ways. Static LLM transduction stays below roughly 10% on ARC; fine-tuning on each task’s demonstrations lifts it into the 50%-55% range and “probably even 60% soon.” Chollet accepts that as legitimate generalization, yet prefers systems that construct verifiable programs through search: “I don’t think it’s a good idea to try to use gradient descent as a replacement for a programming process.”
ARC 1 is nearing benchmark saturation without anything in the field being close to meaningful AGI. The best 2020 submission scored 20%, but an ensemble reached 49%; in 2024, one submission reached about 55% and the full ensemble 81%, close to the 85% target, while humans would score roughly 97%-99%. That gap between benchmark conquest and intelligence is why Chollet calls ARC “a compass towards AGI,” not a test for it, and plans ARC 2 with more diverse, less redundant tasks and stronger leakage controls.
The competition supports a hybrid architecture because perception and algorithms fail on opposite task classes. Neural transduction handles fuzzy, structurally continuous recognition; symbolic induction handles discrete procedures and can be executed against demonstrations. Chollet’s preferred stack therefore starts with verifiable induction, falls back to transduction, and shares representations across both: “There are problems where vector spaces are the right data structure and problems where symbolic discrete programs are the right data structure.”
Chollet interprets o1—explicitly as speculation—as test-time search over chains of thought, making it a genuine break from static inference. The system appears to explore branches, backtrack and edit until it constructs a near-optimal natural-language program; token volume and latency make a single forward pass “certainly not plausible” in his view. Accuracy should improve logarithmically with search compute, which makes compute-normalized benchmark reporting essential.
His theory of intelligence centers on adapting to novelty, then converting repeated solutions into reusable abstractions. Human System 2 reasoning is, in this account, iterated fuzzy intuition constrained into a symbolic artifact, while consciousness supplies the self-consistency check that prevents the process from becoming “hallucinating” or “dreaming.” This also explains why today’s agents fail when they chain unreliable guesses, yet improve when human-designed workflows provide decomposition and “strong guardrails.”
After leaving Google, Chollet is building a research company around deep learning-guided program synthesis and ultimately human-level AGI. The proposed product vision is programming through natural language, drawings and input-output examples, with the system actively asking users to resolve ambiguity; reusable abstractions would then circulate through a new lifelong, distributed learning architecture. He remains clear that frontier models, “not even o1,” are not there yet—but the end state would let non-programmers offload software complexity rather than merely autocomplete code.

Deep dive

1. ARC Prize helped move the field beyond pre-training-only AGI

Chollet’s central reflection on 2024 is a narrative reversal: the earlier prescription was “100× larger models, 100× more data,” with AGI expected to emerge from scale. The newer realization is that something akin to System 2 reasoning must be explicitly added through test-time search, program synthesis or related mechanisms.
ARC Prize succeeded because it met latent demand from researchers who already suspected “plain LLMs weren’t gonna get to AGI.” Its popularity made that intuition concrete, but Chollet rejects treating the score as a binary AGI indicator: the benchmark is a research tool designed to direct attention toward strong generalization.
That stance is not a conversion to pure symbolism. Chollet has worked with deep learning since roughly 2013, evangelized it through Keras from 2014 and has advocated deep learning-guided program synthesis since 2017: “I’ve always been a deep learning guy.” His proposal augments neural pattern recognition with discrete reasoning rather than replacing it.

2. A $10 submission matched $10,000 of frontier-model compute

The private Kaggle track required self-contained notebooks because its tasks could not be sent to third-party APIs. Each submission received less than 12 hours on a VM with one P100 GPU—about $10 of compute—while preserving the fully private test set.
The public frontier-model track used 400 public-evaluation tasks plus 100 semi-private tasks sent to providers including OpenAI and Anthropic. Entries could consume $10,000 in API credits: 1,000 times the total budget and, because more tasks were evaluated, roughly 200 times more compute per task.
The result surprised Chollet: both tracks scored around 55%. Systems spending $10 delivered the same headline accuracy as systems spending $10,000, demonstrating that “compute is really a multiplier for ideas” and that vastly higher spending cannot rescue a comparatively inefficient method.
Unlimited compute could still brute-force ARC if the search language were suitable; Chollet estimates it might take hundreds of millions of dollars of compute to crack the entire dataset. That would be theoretically valid but “an extremely inefficient and extremely stupid idea,” making score without compute disclosure increasingly meaningless.

3. Test-time training is legitimate, but program search is more structured

Two approach families worked. Deep learning-guided program synthesis used LLMs to generate and iteratively debug code, with DSL-based synthesis remaining underexplored; test-time training instead fine-tuned a pre-trained ARC model on each new task’s demonstration pairs before directly predicting its output grid.
Without adaptation, static LLM transduction remains below roughly 10% because it can only fetch and reapply pre-trained patterns. Per-task fine-tuning recombines latent knowledge into a task-specific model, pushing accuracy into the 50%-55% range and, Chollet believes, “probably even 60% soon.”
Scarfe’s pushback was that task-specific augmentation and fine-tuning might violate developer-aware generalization by relying on human specialization. Chollet disagreed: the demonstrations provide the supervision autonomously, while human awareness of ARC’s format is no more disqualifying than the format-specific machinery inside program induction.
His preference remains discrete adaptation. Test-time training makes gradient descent perform the recombination and stores the task solution in modified weights; program search stores it in an explicit program. Human ARC reasoning seems closer to function composition than gradient descent, making o1-like search more attractive to him.
Chollet’s caveat on deploying test-time training at frontier scale is not primarily infrastructure: serving could be re-engineered, but the method needs clear input-target pairs, which ARC has and most real-world problems do not.

4. Architecture tweaks merely relocate the intelligence problem

Scarfe suggested future transformers might overcome failures on copying, counting and out-of-distribution algorithmic inputs, making transduction viable. Chollet agreed architecture changes can embed strong structural priors and allow gradient descent to discover solutions that generalize much farther.
The catch is that a human engineer must first understand the causal structure and encode an architecture “isomorphic” to it. Fully autonomous novelty adaptation would therefore require a machine that identifies the structure and generates the architecture itself—a problem Chollet considers at least as hard as general program search.
Some solutions were roughly 10 percentage points lower on the semi-private set than on the public evaluation. Program-synthesis solutions showed no comparable decline and scored essentially the same across both sets.
Chollet was nevertheless intrigued by Symbolica’s proposed route: use a symbolic process to turn a task definition into a specialized deep learning architecture, then train it. He called the approach “very, very original” and knew of no one else pursuing the same construction.

5. ARC 1 can be cracked long before intelligence is solved

In Kaggle’s first ARC competition in 2020, the winning Icecuber entry scored only 20% using basic brute-force program enumeration. Combining all submissions, however, yielded 49%—evidence that nearly half the private set was accessible to brute-force enumerators four years before 2024’s advances.
In 2024, the best single submission reached about 55%, still well below the 85% bar; Chollet estimates a human seeing the private set would score around 97%, 98% or 99%. Yet an ensemble of every 2024 submission reached 81%, despite none of those systems—or their ensemble—being meaningfully close to AGI.
The conclusion is uncomfortable but useful: scale can saturate a benchmark by combining narrow coverage. ARC 1 lacked enough task diversity, contained some redundancy and was not consistently challenging enough, so benchmark success increasingly measures coverage and brute-force reach rather than the generalization capability it was meant to isolate.

6. ARC 2 is designed to resist overfitting and leaderboard leakage

ARC 2 was not improvised after the 2024 results. Chollet publicly announced it in early 2022, after the 2020 competition exposed ARC 1’s weaknesses, and began crowdsourcing tasks with Lab42 in Davos before expanding and filtering the collection.
Human testers have attempted the tasks, producing difficulty data from solve rates and numbers of attempts. The goal is three difficulty-calibrated sets—public, semi-private and fully private—so a non-overfit public score should predict performance on the other two.
Reusing one private set across competitions let submitters repeatedly observe scores, creating gradual information leakage and, with enough submissions, the possibility of reconstructing test content. ARC 2 will return semi-private scores during competition and run the fully private evaluation only at the end, when producing the final leaderboard.
That design also enables apples-to-apples comparisons between constrained Kaggle systems and frontier models such as o1 Pro on identical tasks, even when their compute budgets differ. Chollet expected release “early next year,” after final human testing.

7. Perception and algorithms demand different representations

One of 2024’s most revealing findings was that program induction and neural transduction solved “substantially different sets of tasks.” Transduction excelled on perceptual pattern recognition; explicit programs excelled where the transformation was discrete, algorithmic and easy to state procedurally.
Chollet’s letter-A analogy carries the distinction: a visually trained system recognizes varied handwritten As immediately, but writing a symbolic algorithm to recognize them is tremendously difficult. Scarfe’s thought experiment conceded that a Python program equivalent to an MNIST network exists; its simplest representation would probably “look a lot like what the current net is doing.”
Induction should therefore come first when it fits, because a candidate program can be executed against every demonstration and formally checked. Transduction is an unverifiable guess; majority voting assumes wrong guesses scatter across answers while correct guesses converge, an assumption that raises confidence but never supplies proof.

8. Multiple views and latent programs offer practical hybrid routes

Omni-ARC used one model for program writing, program interpretation, transduction, input generation and other ARC-related tasks. Looking at a problem from several angles makes the network more likely to recover its “true shape” while transferring information among complementary objectives.
This multitask design also regularizes neural representations. A network trained through one modality can latch onto noisy correlations; forcing the same representation to support several views lets evidence from one task counter noise learned from another.
Chollet highlighted Clément Bonnet and his colleague’s work as one of his favorite competition papers. Their VAE learns a latent space of programs, then performs gradient descent through that space at test time—a form of “deliberation in latent space” distinct from both discrete search and weight fine-tuning.
Scarfe raised whether gradient search required a smooth, relatively homogeneous or convex landscape. Chollet said gradient descent needs a relatively smooth surface and that the VAE’s structured latent space is key. His proposed extension is to decode promising points into symbolic programs, run them against demonstrations and conduct local discrete search—turning latent guesses into verifiable artifacts.

9. Programs should be searched as graphs, not emitted as token strings

If Chollet spent a year on ARC, he would pursue deep learning-guided program synthesis—but not today’s default of token-by-token code generation. “Programs are not sequences of tokens”; they are graphs of operators, and synthesis should be a tree search in which neural models guide branching and targeted graph edits.
Humans do not mentally enumerate many ARC programs. They first describe grids through objects, properties, relationships and especially causal structure; that task model constrains the candidate space so sharply that only a handful of programs need consideration. Sufficiently strong modeling might “almost entirely remove the need for search.”
Scarfe compared this evolution to commercial LLM software moving from embedded chatbots toward models that orchestrate low-level tools. Chollet agreed: graph search lets the model modify a program locally and deliberately, while regenerating a token sequence makes the same targeted intervention much harder.
Kevin Ellis’s remixing route—starting from 100 handcrafted solutions, generating instances and using retrieval to combine concepts—addresses sparse ARC data rather than search structure. Chollet expects dense synthetic sampling to help LLMs, but warned that it leads to severe overfitting, matching the observed semi-private drop.

10. Reasoning is novelty adaptation disciplined by consciousness

Chollet separates replay from adaptation. Applying a memorized multiplication algorithm is one kind of reasoning and suits LLMs; confronting novelty requires recombining existing building blocks into a new model on the fly. The useful question is therefore not “can they reason?” but “can they adapt to novelty?”
Abstraction follows repeated novelty: when the same recombination recurs across successful solutions, the system can refactor it into a reusable building block. The next related problem then takes fewer steps and less energy. A fixed DSL is consequently limiting; the system must continually enrich its own language, whether atop a DSL or Python.
Chollet does not think reasoning has an intrinsic score. One must define a goal and observe behavior; given a goal, the best model is probably the simplest one that retains the causal factors needed to achieve it. Generalization power—the amount of novelty a system can adapt to—is one possible operational measure.
At the cognitive level, Chollet thinks fuzzy pattern recognition is fundamental. System 2 emerges when intuition is applied iteratively inside a structured artifact: a chess player guesses which branches matter, unfolds only selected moves and focuses on selected board regions, combining continuous judgment with discrete simulation.
Unconstrained iteration drifts into hallucination or dreaming—“repeatedly intuiting what comes next” without regard for the past. His theory is that consciousness supplies the self-consistency guardrail, bringing past and anticipated future into a present “nexus point”; explicit step-by-step reasoning therefore requires awareness.
He also argues that human efficiency is not merely a matter of cheap operations: neurons are more energy-hungry than transistors, yet humans can solve ARC using only a few calories because they use comparatively few neural operations. He expects AGI when AI reaches comparable energy efficiency.

11. o1 points toward search, while the new company targets the full stack

Chollet stressed that he had no insider information about o1. His best hypothesis is test-time tree search over possible chains of thought, including branch evaluation, backtracking and editing, until the model produces a plausibly near-optimal natural-language program. He called the resulting novelty adaptation “a genuine breakthrough” beyond classical deep learning.
Scarfe raised the competing theory that process supervision occurs during training but inference is one forward pass. Chollet called that “certainly not plausible”: tokens, latency and compute clearly indicate search. Training may resemble AlphaZero-style learning to reproduce strong chains, while inference still searches chain-of-thought space; accuracy then improves logarithmically with compute.
Current agents remain constrained by compounded error: an LLM is a useful “guessing machine,” but chaining many fallible guesses rapidly reduces reliability. Scarfe’s AI Scientist example showed decomposition producing much better work than one-shot paper generation; Chollet agreed because human programmers supplied the orchestration and “strong guardrails” that convert iterated System 1 into System 2.
Having left Google a few weeks earlier, Chollet was forming a company and research lab with a friend, hiring a team and making deep learning-guided program synthesis his full-time work. He wants solutions and benchmarks to co-evolve, aims explicitly at human-level AGI, and will not enter the ARC Prize he runs.
His product vision is programming from input-output pairs, natural language, drawings or visual data-flow graphs. The AI would surface ambiguous cases, ask the user which output is correct and collaboratively refine the program; frontier systems, “not even o1,” do not yet support software complexity at a human engineer’s level.
Beyond a single assistant, he anticipates “a completely new type of architecture”: many instances solving problems for different users, detecting commonalities and circulating them as reusable building blocks through lifelong distributed learning. Chollet declined to identify the exact implementation—“I can’t really tell you exactly what we’re building”—but framed that abstraction loop as the engine of increasing capability.

Erik Bernhardsson on Creating Tools That Make AI Feel Effortless AI AMA – Part 2: AI Utopia, Consciousness, and the Future of Work