Recursive Language Models — Alex Zhang, MIT PhD
Summary
Alex Zhang argues that academia’s major advantage is the freedom to make “big bets” on problems industry overlooks, dismisses as trivial, or considers impossible. SWE-bench initially attracted little interest because the task looked infeasible; RLM, ReAct, Quiet-STaR, and similar work looked obvious only after someone articulated the design space. If a graduate student merely copies an industrial lab’s agenda without its compute, Zhang’s blunt conclusion is: “just go to an industrial laboratory.”
AI-generated GPU kernels are already competitive, but expert intuition remains the scarce input because benchmark wins do not guarantee system-level reliability. Recent GPU Mode leaderboards are dominated by AI-assisted submissions, yet longtime kernel developer Gorst produced nearly the only top-10 kernel that remained stable inside complex real systems. Expertise turns a practitioner into a strong verifier who can redirect a model before it wastes “a hundred billion or a trillion tokens.”
Smaller laboratories should compete through different model and output architectures, not by cloning the frontier labs’ autoregressive decoders. Zhang’s favorite example is JEPA: it reuses information learned about language but changes the model’s output space, enabling fast inference for tasks such as binary classification, where calling a full generative model could cost “400 times more.” He sees a broader opening for latency-sensitive systems that need intelligence without full text generation.
Zhang thinks Claude Code, Codex, Pi, and most current harnesses are variations of the same “trajectory as a prompt” loop. RLM is his counterexample: unload context into a persistent environment, let the model manipulate it through code, and recursively call subagents on locally tractable pieces. In experiments he describes, strategies learned on short tasks transferred to tasks 8–30 times longer and sometimes across superficially different domains because the high-level program stayed the same.
Prime Agent turns the RLM thesis into a framework architecture built around IPython, persistent state, and durable subagents. Every capability can be loaded as code, agents can communicate through explicit mechanisms, and the filesystem preserves context after the visible prompt is compressed. Harvey has already reported strong legal-document results from running RLM on its work, while other projects apply the abstraction to continual research, DSPy tooling, and ARC-AGI-3.
Agent swarms can now buy solutions to extraordinary problems, but their economics remain brutal and their orchestration is the actual challenge. The OpenAI effort discussed in the episode reportedly used 10,000 agents for 88 hours and produced 130 billion output tokens—about $40 million at public prices, before counting substantially larger inter-agent traffic. Zhang suspects “95% of the swarm is absolutely useless,” yet calls the ability to direct $40 million at one problem genuinely exciting.
The largest near-term opportunity may be making today’s models reliably perform mundane, month-long work rather than waiting for another intelligence leap. Zhang believes a leading model such as Astra may already be capable of much more with the right tools, including long, simple tasks and intern-like assistance, but language-model interfaces and continuous-learning-style workflows remain inadequate. His compact diagnosis is: “Your models are actually capable of more if you make an effort, so it’s a skills issue.”
Deep dive
1. GPU Mode turned kernel optimization into a learnable competitive sport
Zhang entered what was then CUDA Mode while trying, unsuccessfully, to write specialized kernels for Google’s “Infinite Attention” paper during a Snap internship. The community was started by Mark Saroufim with Andreas and Jeremy Howard, and combined accessible lectures with hands-on kernel work rather than restricting serious participation to industrial teams.
The project called Popcorn, later expressed through a leaderboard, began from an analogy: GPU programming resembles competitive programming. There is a surprisingly limited repertoire of optimizations and recurring kernel shapes, so Codeforces-style task volume might eventually let researchers automate kernel development.
KernelBench emerged from the same premise: can an LLM generate efficient GPU kernels? The practical motivation was visible in projects such as Mamba, where releasing an architecture without performant kernels could make it difficult for teams lacking the relevant expertise to use effectively.
2. Benchmark speed is not the same as production reliability
Recent GPU Mode submissions are overwhelmingly AI-generated or AI-assisted. Yet Gorst, an established kernel expert, produced almost the only top-10 solution that worked reliably inside complicated real systems—evidence that system hacks and incomplete verification remain central benchmark risks.
Zhang’s working method is revealing: “I always draw diagrams of my code,” and any part he cannot explain is unacceptable until he understands it. Domain knowledge supplies both the search direction and the verifier, allowing an expert to eliminate enormous branches before a model burns vast amounts of compute.
Matrix multiplication has a relatively clear physical ceiling, but even there the answer depends on whether data begins on the CPU or in GPU DRAM. Perfect overlap between compute and data movement may be unattainable; fortunately, many real kernels remain far enough below the ceiling that substantial gains are available.
3. Whole-model optimization will require fusion, compilers, and better objectives
A locally faster kernel can make an end-to-end model slower if it evicts data needed by the next operation. Zhang therefore distinguishes isolated kernel speed from system speed: sometimes one should sacrifice the first operation’s latency to preserve cache state for what follows.
The hosts framed this as operator fusion or “megakernels.” Zhang is cautious: individual kernels have constrained structures, while a megakernel introduces a more complicated design problem that may be better handled by a compiler assembling and selectively fusing well-understood components.
Zhang said he probably does not track energy consumption as a primary target. The important qualification is that neither speed nor efficiency can be assessed honestly without specifying the surrounding memory hierarchy and full execution path.
4. Academia earns its keep by attempting unfashionable ideas
Zhang attributes research taste partly to opportunity: Princeton exposed him to the SWE-bench group, and he later met adviser Omar Khattab at MIT. The recurring pattern among successful academic researchers is not merely intelligence but choosing questions most industrial laboratories are not already pursuing.
SWE-bench was initially ignored because people called it impossible; only after systems such as Devin appeared did the desired capability become obvious. RLM drew the inverse dismissal—“this is just subagents”—which Zhang treats as a promising sign that critics see the mechanism but not yet the destination.
ReAct, chain-of-thought, STaR, and Quiet-STaR can all look obvious in retrospect. Their value is the story they tell about how intelligent systems should work, not that any one paper ships a “GPT-6-level” product on publication day.
His sharp advice to graduate students is that limited compute should increase ambition rather than force imitation. Academia offers freedom from product bureaucracy; declining to make risky bets wastes its advantage, even though many of those bets will fail.
5. Alternative output spaces may matter more than another decoder
Zhang uses JEPA to challenge the assumption that a language model must be an autoregressive text-to-text decoder. Its proposal, as he describes it, is to use information learned about language while changing the output model space for a known task and latency budget.
Binary classification is the clean example: if only two outcomes matter, why invoke a large generative model and “pay 400 times more”? The interesting research questions concern the training objective, routing, partial-model execution, and whether harness behavior can eventually be reproduced inside the model.
Calibration is part of the product, not decoration. Ordinary next-token models can confidently produce a number such as “43%” when asked how certain they are; a useful fast classifier must align confidence with empirical correctness, potentially through synthetic labeled outcomes.
The Doom example made the latency thesis concrete. Zhang’s own game benchmark found that most vision-language models still could not meaningfully complete varied games, though he had seen Astra finish a Kirby game; fast language-conditioned control could unlock a class of low-latency tasks.
6. Smaller labs must choose a different axis of competition
Zhang’s strategic warning is categorical: a smaller laboratory copying OpenAI or Anthropic competes primarily on data and compute, the two dimensions it cannot control. Thinking Machines’ model innovations and JEPA’s altered output space are interesting precisely because they choose a different design axis.
Frontier labs have weak incentives to jeopardize an architecture whose scaling behavior already works. That leaves room for new laboratories willing to explore recurrent Transformers, routers, parallel decoding, specialized inference, or other architectures that trade generality for cost and speed.
The host’s pushback—worth keeping—is that JEPA might merely repackage familiar classification machinery. Zhang’s answer is that the simple implementation is not the point: the system opens a neglected design space and appears to rely on a non-trivial training recipe that open replications have not matched.
7. A harness is a program for adapting intelligence to a task
Zhang calls Claude Code, Codex, Pi, and related products “all the same” at the level of core logic: tools surround repeated model calls while the execution trajectory accumulates in context. Prime Agent differs, but most mainstream design choices remain strikingly uniform.
His reframing is that a harness is a specific program for how a language model should adapt to a problem. Since next-token prediction is awkward for repository navigation or long workflows, the harness decides what the model can inspect, remember, execute, delegate, and verify.
One host noted that neutral coding harnesses already generalize to presentations, design, and other domains. Zhang concedes their utility but argues that apparent universality should not end the search for environments with stronger inductive biases and better post-training scaling.
8. RLM converts diverse tasks into reusable high-level programs
In Zhang’s experiments, search and aggregation prompts produced very different trajectories in ordinary agents. An RLM instead delegated distinct local details to subagents while learning essentially the same root strategy, allowing training on one task to transfer immediately to another.
The length result is especially concrete: a policy learned on short instances generalized to problems 8–30 times longer because the procedure stayed fixed and only a length parameter changed. Zhang says similar transfer appeared between categories such as mathematics and writing, not merely within longer versions of one benchmark.
His GPU analogy makes the induction intuitive. Competitive programming and kernel optimization differ locally, but both can use a root program that enumerates candidate approaches, assigns subagents to promising options, iterates, and checks results with a verifier.
9. RLM keeps every local call closer to the model’s distribution
RLM stores the main context in a software environment—often a filesystem—rather than forcing the full state through every prompt. The model writes code, can call itself as a subagent, and accesses other tools as functions inside a Python or shell environment.
Zhang’s “locally in-distribution” argument is compositional: the overall assignment may be novel, but each delegated call can remain similar to work the base model has already seen. If each local task is familiar, the model is more likely to produce reliable components even when the full problem is out of distribution.
This also explains his objection to “trajectory as a prompt.” Conventional agents continually append actions and observations to one context; RLM gives them a central external state they can query selectively, compose through code, and revisit after conversational compaction.
10. Prime Agent operationalizes persistent, code-native delegation
Prime Agent layers RLM ideas over the minimalist Pi framework while restricting the available tool to IPython. Bash commands and other capabilities become modules or functions, trusting the model to write the intermediate tooling it needs.
Its continual-harness component lets the system modify selected parts of itself, including skills, subagents, system prompts, and the continual harness. Zhang likes the explicitness: self-improvement is constrained by design rather than treated as unrestricted prompt mutation.
Subagents can persist beyond the root agent’s original working window, receive follow-up prompts, and communicate through code-defined channels. Zhang contrasted that with Codex subagents that he found “very fleeting” and ill-suited to long-lived processes.
“The whole secret is to take everything out to the filesystem,” one host summarized. Zhang added the second half: make interaction occur through code, because today’s models are unusually capable programmers and can synthesize their own task-specific interfaces.
11. Early RLM adoption is moving beyond long-context demos
Harvey independently ran RLM on legal work involving review and extraction across many documents, reporting strong results. Zhang was surprised because the company had not coordinated with him; for him, that independence made the validation more meaningful.
He also highlighted a continually running research system apparently associated with Base10, Axe as a one-developer project resembling DSPy and RLM, and RLM support in DSPy. These projects often extend the abstraction rather than merely reproducing the paper.
ARC-AGI-3 is another natural setting because code, compositional search, and neurosymbolic operations can share a workspace. Zhang said he would assume OpenAI’s reported 99.9% result and its Navier–Stokes work did not use RLM specifically; they appeared closer to sophisticated shared-context swarms, though he was not certain about the details.
12. Swarms make compute a purchasable substitute for missing insight—at a price
The OpenAI run discussed by the hosts reportedly involved 10,000 agents over 88 hours, generating 130 billion output tokens and costing roughly $40 million at public prices. Alex added that total traffic was more than twice as high once messages passed among agents were counted.
Zhang thinks a sufficiently capable model may already be able to find a proof once the relevant information is assembled. The hard problem is obtaining that information through search, decomposition, researcher guidance, and coordination—not merely selecting a particular orchestration library.
His efficiency estimate is intentionally provocative: “about 95% of the swarm is absolutely useless.” Search produces dead branches, yet a domain expert can sometimes supply one hint that eliminates the trillion-token detour; this is why orchestration and verification remain more valuable than raw agent count.
Cursor’s multi-agent architecture reportedly converged on an ordinary software-team org chart: a lead coordinates workers, gathers their output, then redistributes work. Zhang called the gather-and-recoordinate step a bottleneck and said a genuine swarm should consist of more independent agents without drowning the user in internal traffic.
13. Open-ended research needs selection, while useful agents need endurance
Zhang compares open-endedness with unresolved mathematics: evolutionary search or agent swarms may explore indefinitely, but “how do you isolate the interesting things?” Systems such as Fugu suggest routing toward tasks where progress is possible, yet choosing the eventual objective remains the unresolved step.
Sakana impressed him as an eccentric, PhD-like laboratory rather than a frontier-model competitor. Its AI Scientist work invites criticism, but Zhang values a structure that funds unusual evolutionary research and has a model specialized for Japanese language and culturally adapted answers.
The discussion treated Kimi’s swarm idea as interesting, while a later discussion of dynamic workflows raised cost and reliability concerns. The transcript’s Speaker 1/2 labels make the precise attribution of those remarks uncertain.
The discussion praised AlphaGeometry-era work while also calling Gemini somewhat overrated and raising bureaucracy and weak uptake of Antigravity; the Speaker 1/2 labels make those individual attributions uncertain.
The conversation connected Zhang’s month-long reliability goal to continuous learning: make a leading model perform simple work consistently, or provide intern-like assistance while investigating, adapting, and remembering. Speculative tool execution—launching predictable calls before code generation finishes—is an “obvious” optimization, but durable state, feedback, and attention are the deeper constraints.
Zhang maintains 10–15 risky ideas, tests cheaply, discards most, then “drops everything” when one survives. He welcomes collaborators with specific disagreements or convictions, not emails that merely say “I love RLM”; in an agent-rich world, human attention and research taste remain the scarce resources.