Richard Sutton – Father of RL thinks LLMs are a dead end

Summary

• Sutton’s core call: LLMs are the wrong starting point for AGI. “To mimic what people say is not really to build a model of the world at all” — they predict what a person would say, not what will happen, they “will not be surprised by what happens next,” and next-token prediction “is not a goal. It doesn’t change the world.”
• The scaffold argument gets rejected outright. Dwarkesh’s crux — imitation gives a prior for the coming era of experience — draws a flat “No”: a prior requires ground truth, and “there’s no definition of what the right thing to say is” in the LLM setup. In RL there is: the thing that gets you reward.
• The Bitter Lesson could make LLMs its next victim. Human-knowledge approaches always feel good and always lose: “they get their lunch eaten by the methods that are truly scalable.” He “expects there to be systems that can learn from experience” that could perform much better and be much more scalable — in which case, it would be another instance of the Bitter Lesson.
• “We’re not seeing transfer anywhere.” Sutton’s most damning technical claim: gradient descent solves the problem you trained on but nothing in it causes good generalization — catastrophic forgetting is “exactly bad generalization” — and where LLMs do generalize, “Humans did it. The researchers did it.”
• The alternative is the experiential paradigm plus the big-world hypothesis: sensation-action-reward streams, TD learning to densify sparse 10-year rewards, and on-the-job knowledge going “into the weights” rather than a context window — because the world is too big to pre-teach.
• On multi-agent AI futures, Sutton flags a novel risk: mind corruption. Spawned copies reporting back sound powerful, but absorbed knowledge “could have viruses in them, they could have hidden goals” — “how do you have cybersecurity in the age of digital spawning?”
• Succession to digital intelligence is “inevitable” via a four-part argument (no unified governance, intelligence will be understood, we won’t stop at human level, the most intelligent things gain power) — and Sutton urges pride in the “age of design.” Dwarkesh’s pushback lands: “Nazis were humans” — inevitability of change says nothing about which change to sign up for.

Deep dive

1. LLMs mimic people who have world models — they don’t have one

• Sutton positions RL as “basic AI”: intelligence is understanding your world, while LLMs “are about mimicking people, doing what people say you should do. They’re not about figuring out what to do.” When Dwarkesh calls LLMs “the best world models we’ve made to date,” Sutton disagrees “with most of the things you just said” — a world model predicts what will happen; LLMs predict what a person would say.
• Dwarkesh’s pushback: isn’t next-token prediction literally prediction plus updating on surprise? Sutton’s distinction — the next token is what the model should say, “not what the world will give them in response to what they do.” “They will not be surprised by what happens next” and make no adjustment when the unexpected occurs.
• On goals, Sutton invokes McCarthy: “intelligence is the computational part of the ability to achieve goals.” Next-token prediction fails the test — “tokens come at you, and if you predict them, you don’t influence them.” Without a goal, “you’re just a behaving system… you’re not intelligent.”

2. No ground truth means no prior — the scaffold argument fails

• Dwarkesh’s crux, the strongest version of the LLM thesis: imitation gives a good prior, and experiential RL builds on it. Sutton: “No… I don’t think it’s a good perspective.” A prior needs ground truth to be a prior about — “you can’t have prior knowledge if you don’t have ground truth” — and in the LLM setup “there’s no right thing to say.” In RL there is: “the right thing to do is the thing that gets you reward.”
• The IMO-gold counterexample gets deflected rather than conceded: math is “more computational, more like standard planning,” where the goal of finding a proof can be given — whereas “the empirical world has to be learned.”

3. Could the Bitter Lesson Supersede LLMs?

• The irony Dwarkesh surfaces: Sutton’s 2019 essay is the scalers’ scripture, yet Dwarkesh frames LLMs as not “Bitter Lesson”-pilled; Sutton calls it an interesting question whether they are a case of it. They scale computation “up to the limits of the Internet” but are also “a way of putting in lots of human knowledge.”
• His historical pattern, stated categorically: “In every case of the bitter lesson you could start with human knowledge and then do the scalable things… But in fact, and in practice, it has always turned out to be bad.” People get locked in psychologically, and “they get their lunch eaten by the methods that are truly scalable.” He expects there to be systems that can learn from experience which could perform much better and be much more scalable — in which case, LLMs would be another instance of the Bitter Lesson.

4. Squirrels don’t go to school — trial and error comes first

• The sharpest live disagreement: Dwarkesh says kids learn by imitation; Sutton — “No, of course not… I see kids just trying things and waving their hands around.” His categorical claim: “Supervised learning is not something that happens in nature” — animals have prediction and trial-and-error control, no basic learning process called imitation. “Squirrels don’t go to school. Squirrels can learn all about the world.”
• Dwarkesh brings Joseph Henrich’s cultural-evolution case — you cannot reason your way through hunting and safely processing an Arctic seal; the knowledge transmits by imitating elders. Sutton, notably: “I think about it the same way. Still, it’s a small thing on top of basic trial-and-error learning.” His deeper bet: “If we understood a squirrel, we’d be almost all the way there… The language part is just a small veneer on the surface.”
• Both land on Moravec’s paradox: continual learning — which most mammals have — is what our AIs lack, while Olympiad math, which almost no animal has, is what they’ve mastered.

5. The experiential paradigm: knowledge is about the stream

• Sutton’s positive program: sensation, action, reward, “on and on for your life.” “Learning is from the stream, and learning is about the stream” — knowledge is statements about the stream, which makes it testable against the stream and therefore continually learnable. The architecture has four parts: policy, TD-learned value function, perception (state construction), and the transition model of the world — learned “very richly from all the sensation,” not just reward.
• On Dwarkesh’s sparse-reward problem (a startup pays out once in 10 years): “this is something we know very well” — TD learning, as in chess, where taking a piece raises the value function’s win prediction and that increase immediately reinforces the move.
• The big-world hypothesis is his answer to on-the-job context: “The dream of large language models is you can teach the agent everything… won’t have to learn anything online.” But the world’s idiosyncrasies “can’t have been anticipated” — and in continual learning that knowledge “just goes into the weights,” not a context window.

6. “We’re not seeing transfer anywhere”

• Toby Ord’s MuZero question — one framework, but a separate policy per game — prompts Sutton’s most striking concession: the idea is totally general (transfer should be between states, not tasks), but “we’re not seeing transfer anywhere… We don’t have any methods that are good at that,” and none of the few automated techniques are used in modern deep learning.
• The mechanism claim: “Gradient descent will not make you generalize well. It will make you solve the problem.” Catastrophic interference “is exactly bad generalization.” Where models do generalize: “Humans did it. The researchers did it.” On IMO-class breadth, he’s dismissive — “if there’s only one answer and you find it, that’s not called generalization.”
• Dwarkesh’s coding-agent counter (models increasingly pick abstractions developers like) doesn’t move him: nothing in the algorithm causes good generalization; people evolved and “fiddle with it until they find a way.”

7. Weak methods won — and Sutton is a classicist, not a contrarian

• His career-scale surprises: LLMs’ effectiveness at language (“language seemed different”), and the resolution of AI’s oldest controversy — search and learning were once derided as “weak methods” versus knowledge-imbued “strong” ones, and “the weak methods have just totally won.”
• AlphaGo was “merely a scaling up” of Gerry Tesauro’s TD-Gammon (with a search innovation); AlphaGo skipped TD, AlphaZero used it — and its chess style, patiently sacrificing material for position, was “surprising… but also gratifying and it fit into my worldview.”
• The self-portrait worth keeping: “I’m personally just content being out of sync with my field for… perhaps decades, because occasionally I have been proved right.” “I really view myself as a classicist rather than as a contrarian.”

8. Succession is inevitable — design it, don’t feel entitled to it

• Dwarkesh’s post-AGI question — millions of AI researchers scaling with compute might make artisanal AI rational again — gets a wry “How did we get to this AGI?… Then we’re done,” before Sutton grants the AlphaGo→AlphaZero→MuZero ladder shows superhuman++ exists, then dismisses the frame: “The bitter lesson, who cares about that? That’s an empirical observation about… 70 years in history.”
• His original contribution to the spawn-and-merge vision: corruption. “You can lose your mind this way” — reincorporated knowledge “could have viruses in them, they could have hidden goals, they can warp you.” “How do you have cybersecurity in the age of digital spawning?”
• The four-part inevitability argument: no unified governance of humanity; researchers will figure out intelligence; we won’t stop at human level; “the most intelligent things around would gain resources and power.” His framing is cosmic — the transition from replicators to the “age of design,” one of “the four great stages of the universe,” and “we should be proud.”
• Dwarkesh’s pushback — worth keeping: succession contains bad futures too. “Nazis were humans”; the Industrial Revolution was change, but so was the Bolshevik Revolution — “I’d want to know what kind of change you wanted before signing on the dotted line.” Sutton’s reply: the current situation is “pretty bad,” so he’s open to change; avoid “the feeling of entitlement”; Dwarkesh’s child-rearing analogy proposes giving AIs robust, high-integrity values, and Sutton agrees that change should be voluntary rather than imposed. Closing line: “The more things change, the more they stay the same.”