Summary

Berman’s roughly 30% result on the public ARC-AGI-2 leaderboard came from evolving English instructions instead of executable Python. Every v2 task can be described in 10 plain-English bullets and most in five, while equivalent Python becomes long and brittle. His core bet: “English…is a much more expressive program.”
Natural-language expressivity shifts the bottleneck from generating solutions to verifying them. Berman’s checker directly converts an instruction into a grid and makes errors “all the time”; GPT-5 mini/nano proved inadequate, and “the checker was stronger than the actual instruction creator.” That architecture reached roughly 30% at about $30 per ARC-AGI-2 task.
Model routing and inference efficiency remain economically material because reasoning capability is unusually “spiky.” Grok 4 beat the GPT models Berman tested on grid reasoning, while Sonnet 3.5 remained preferable for code and Opus 4.1 might lead his older Python system. With today’s models, he would not expect anyone to exceed 40%, though he believes his pipeline could become roughly twice as efficient.
The larger opportunity is composable continual learning, not merely larger frozen models. Fine-tuning changes weights that already encode correct answers, so even more examples of a solved problem can cause drift; Berman expects freezing experts or layers to become the next S-curve after reinforcement learning. The target is simple to state but unsolved: add a dataset, retain everything previously learned, and become excellent at the new skill.
Berman and the host agree on the destination but dispute whether neural networks can reach it without a symbolic controller. Berman sees sufficiently large networks plus program-writing tools as capable of human-equivalent computation; the host argues stochastic gradient descent exploits shortcuts and may never discover the rare algorithm that makes finite brains behave like Turing machines. When pressed on that trainability objection, Berman said: “I need to think about that a bit more.”
Berman’s central post-training thesis is that reasoning must replace a memorized knowledge web with a causal, deductive tree. Pre-training connects facts without guaranteeing the right causal structure, while reinforcement learning with verifiable rewards can preserve circuits that yield coherent deductions. His contrarian formulation is that pre-training might impede reasoning: post-training is “turning your consultant into Feynman.”
The next frontier is an “invention circuit” trained to rediscover genuinely withheld knowledge. Berman proposes removing special relativity and its descendants from pre-training, then building an environment where a model must deduce them; the host questions whether creativity is instead domain-specific search requiring a controller. They converge on a useful taxonomy: reasoning builds the tree, understanding possesses it, and intelligence measures how efficiently it can be acquired.

Deep dive

1. ARC’s reasoning gap pulled Berman out of operating

Berman had worked in research for only eight months after four and a half years as CTO of a Y Combinator company. Reading Jeff Hawkins’s A Thousand Brains while learning about language models made “something just click”; believing AGI could be his lifetime’s most important invention, he stepped down despite the company continuing to do well.
His first independent research project attacked ARC after contacting François Chollet and Mike. A top public ARC v1 score led to program-synthesis work with them, but growing confidence in language modeling plus reinforcement learning ultimately took him to Reflection AI, where he now works on reasoning and post-training for frontier foundation models.
ARC presents several input-output grids sharing one transformation rule, then asks the system to extrapolate that rule to a test grid. Berman called it “kind of like an IQ test for machines”: average humans scored around 75% on ARC v1 while GPT-4 and Sonnet 3.5 were around 5% at the time.
The host highlighted a line from Berman’s first paper: “A parrot that lives in a courthouse will regurgitate more correct statements than a parrot that lives in a madhouse.”

2. Python evolution repaired near misses that sampling could not

Inspired by Ryan Greenblatt, Berman initially generated many Python programs representing candidate transformations. Python supplied two decisive properties: deterministic execution and cheap verification against every training example, making a passing program relatively trustworthy on the unseen grid.
Pure breadth failed surprisingly badly. Even after asking Sonnet 3.5 for 1,000 programs, easy tasks often remained wrong by a few cells, despite seeming close enough to the model’s distribution that a valid solution should have appeared.
Berman instead retained the best candidates and supplied granular feedback: the original program, the incorrect cells, and a request to improve it. Repeating this revision as many as 10 times turned near misses into solutions and eventually revealed the broader principle that “evolving solutions is a powerful technique generally.”
The search has a Goldilocks trade-off. Greater breadth reaches unusual hypotheses that local revision cannot discover; greater depth composes and repairs promising ones. ARC-AGI-2 is compositional, with many rules to traverse, which changed the breadth/depth trade-off, and Berman’s ARC v1 system transferred poorly.

3. RL-trained models moved much of the search loop inside

For ARC-AGI-2, Berman found breadth more important than expected because thinking models already perform deep revision inside their thinking blocks. His outer evolutionary loop still helps, but the model now supplies much of the depth that previously had to be engineered around Sonnet 3.5.
Berman admitted that o1, released two weeks after his first post, changed his view enough to make parts of that post embarrassing. Before reinforcement learning taught models to explore, external revision largely simulated thinking through “stochastic guessing”; RL-trained models arrive with “in-built revision loops” and need no prompt to think step by step.
The host challenged the leap from domain-specific competence in math or code to general thinking. Berman’s hedge was temporal: models “are as bad as they’re ever going to be,” and only a subset of post-training compute currently targets general reasoning, so today’s specialization is not proof of a permanent architectural ceiling.

4. English expanded the hypothesis space but broke deterministic verification

Python became the wrong representational language for ARC-AGI-2 because many simple visual rules require “lines and lines and lines” of brittle code. By contrast, Berman said every task fits within 10 plain-English bullets and most within five: “Everything is quite simple. It’s not very hard.”
That representation also mirrors human problem-solving. People inspect a grid, propose a short verbal rule, and check its clauses; even expert programmers would perform worse if forced to express every hypothesis as Python. Natural language lets the model’s learned inductive bias “express itself fully.”
The cost is that English cannot be executed against a grid. Berman’s checker receives a natural-language algorithm and directly outputs the predicted grid, but it fails “all the time.” Attempts to economize with GPT-5 mini or nano went badly; unexpectedly, checker quality mattered more than the instruction generator’s strength.
Berman nevertheless sees the v2 architecture as representationally general in a way his Python system was not. He claimed no ARC-AGI-2 task lies inherently beyond its program space and predicted that, once Grok 6 or GPT-7 supplies stronger reasoning, “you can use my v2 solution and it will win—it will beat ARC.”

5. Performance depends on model spikes, entropy, and inference budget

Berman selected Grok 4 after direct tests against GPT models, not merely from its base leaderboard position. His explanation is distributional specialization: neural networks remain “very spiky” in narrow domains, and Grok 4 may have encountered enough related shapes, grids, or reasoning patterns to acquire an unusual advantage.
That advantage does not automatically transfer to code. Berman still prefers Sonnet 3.5 to Grok for Python generation and suspects Opus 4.1 might lead his ARC v1 system, though he has not run the expensive experiment. “It actually very much matters which model you use.”
Natural language’s higher entropy was deliberate, not collateral damage. With the best system still around 30%, Berman repeatedly pushed the model toward “more entropy, more entropy,” keeping prompts broad even when task-specific constraints could improve a few examples.
The v2 run cost roughly $30 per task versus about $8 on v1; o3’s ARC v1 result was discussed at roughly $200 per task, though the speakers did not establish exactly what search or fine-tuning OpenAI used. With current models, Berman expects optimization to yield perhaps a few accuracy points—not more than 40%—but potentially a twofold efficiency gain.

6. Natural language beat explicit transfer, but reusable traces remain available

Eric Pang’s slightly lower but more efficient system used library learning and transfer. Berman had also built a version that saved training traces and retrieved them during test time; he believes it would improve both efficiency and accuracy because “there is a lot of transfer.”
He omitted that machinery to preserve an elegant, dependency-free result after the simple pipeline scored so highly, joking that he might restore it if someone overtakes him. His stronger claim is that natural language occupies “a much more efficient area to play in,” so he believes he could match Pang’s budget and still score higher.
The host’s reservation was that the current ARC systems remain ephemeral search procedures: each new task largely starts from scratch rather than acquiring and transferring knowledge in Chollet’s sense. Berman agreed that test-time fine-tuning would be the fundamental route from searching to adapting.
On o3, the speakers did not establish exactly what OpenAI did: Berman said they had never figured it out, Mike Knoop thought sampling was probable, and the host said sampling was certain. Berman said the ARC training data was part of the pretraining corpus rather than a fine-tune, which he considered “fair game”; Mike said iterative refinement with verification would have improved the result further.

7. The neural-symbolic argument turns on trainability, not representation alone

Berman’s high-level argument begins with biology: because human brains achieve general reasoning through neural computation, artificial neural networks should not face a physical prohibition. He regards networks as potentially a superset of symbolic systems and generally assumes a language model can access a Python terminal.
The host drew a harder boundary. A finite human brain can expand effective memory through notebooks or whiteboards and therefore run a Turing-complete algorithm, while a fixed neural network is not itself Turing-complete; attaching Python creates a more powerful hybrid but does not show that stochastic gradient descent can learn the controller that searches programs effectively.
Citing Kenneth Stanley’s contrast between SGD and neuroevolution, the host argued that gradient descent tends toward shortcuts while evolutionary search can discover factored, grounded representations that “carve the world up by the joints.” Only a tiny fraction of finite-state algorithms can control an expandable Turing machine, making existence in weight space different from trainability.
When the host asked why a much larger network could not be trained to find the same controller, Berman did not resolve the trainability objection: “I need to think about that a bit more.”

8. Catastrophic forgetting is the missing architecture for adaptive models

The host contrasted today’s training-time RL with an agentic system that acts, receives environmental information, and updates its weights during use. Frozen systems can approximate adaptivity through search, as in AlphaZero or MuZero, but genuinely personalized weight updates would constitute “an entirely different kind of intelligence.”
Berman disputed compute as lasting blocker. He speculated that, with NVIDIA putting $100 billion into OpenAI, Sam Altman’s plan might be to produce a gigawatt of compute a week; he also pointed to a possible local setup running OSS 12B on one GPU and relatively cheap fine-tuning as reasons to consider individualized adaptation plausible within 10 years.
The harder problem is that updating weights can push out previously correct answers. Berman contrasted this with symbolic systems, where a verified result can be stashed in a stable library.
Their shared architectural direction was composability: freeze correct experts or layers, then add new capability without corrupting old knowledge—the host’s “Docker for language models.” Berman expects this to follow the RL S-curve: give a model new data, preserve everything it knows, and make it excellent at the new task. “We are not there yet.”

9. Reasoning must grow knowledge trees and learn invention

Berman framed reasoning as “the meta skill—the skill to create the skills.” Pre-training builds a web of associated facts, treating capitals, languages, physics, and relativity alike without guaranteeing causal structure; reasoning instead constructs a deductive tree from axioms. Intelligence is compression because deeper understanding lets a system regenerate more knowledge from fewer hints.
Reinforcement learning with verifiable rewards can, in his account, prune the associative web into environmentally coherent deductions. That yields his hot take: indiscriminate pre-training may slow reasoning by filling weights with answers rather than derivations. Post-training is “turning your consultant into Feynman,” and some knowledge may be better withheld for the model to derive.
Berman’s proposed invention environment would remove special relativity and its descendants from pre-training, then reward a model for rediscovering them. The host doubted a domain-general “invention circuit,” using editors unable to cut a machine-learning show as evidence that creativity needs domain knowledge; Berman replied that creativity is choosing which axioms advance a tree from level five to six.
They converged on a compact taxonomy: reasoning builds the tree, understanding is possession of it, and intelligence is the efficiency with which it can be acquired—even an ignorant child can therefore be highly intelligent. Berman’s image was “how wide and high is your garden of trees”; at the deepest level, he suggested, those trees may join in logic itself.

许高's First Interview: Plaud, the World's Fastest-Growing AI Hardware The Common Thread of All Technology: Monitoring the Situation, Ep.1