What does the next training paradigm look like?

Summary

• Dwarkesh’s core thesis: the labs’ big research bet — that scaling RLVR across “millions of verifiable tasks across thousands of diverse RL environments” will steamroll sample-inefficiency and may make continual learning unnecessary — is likely incomplete, because the next real breakthrough is AIs learning on the job, with weight updates from deployment experience.
• The underrated constraint on AI progress: verifiability isn’t enough — a domain must be “grindable”, meaning thousands of parallel rollouts against a deterministic, replayable simulator. That’s why computer use lags coding despite being clearly verifiable: “Andy Jassy will find your bots and shut your ass down.”
• Skills like building a business, winning court cases, or profitable trading can’t be containerized — outer-loop verification “may take months or even years of real-world actions,” making sample efficiency the binding bottleneck for most economically valuable domains.
• He reads a Dario quote on train-vs-serve context lengths as evidence that RLVR generalization is not infinitely strong: if short-horizon training doesn’t necessarily generalize to long horizons, why would white-collar task training produce an agent that could “build SpaceX for you” with $100M in 2002?
• Tradeable compute angle: 30-50% of lab compute goes to inference and “is currently not playing any productive role in helping improve the model” — a huge waste, since deployment is where the most valuable training signal (organization-specific tacit knowledge) actually lives.
• Two candidate recipes: on-policy self-distillation (OPSD) — distill session learnings into weights with dense supervision while preserving sparse updates — and speculative “dreaming,” where models train against self-built simulations, potentially a fourth axis of scaling alongside pretraining, RL, and inference-time compute.
• The 2027-28 scenario: RLVR yields an agent competent enough to deploy, week-long contexts plus a “work review” thumbs-up trigger distillation back into the base model, and capability expands adjacently far beyond verifiable domains — “every time that you interact with an AI, it’ll be smarter… that’s very scary and exciting.”

Deep dive

1. The labs’ RLVR bet

• Dwarkesh’s setup of the bull case he’s testing: train on millions of verifiable tasks and you get a general problem-solving agent — deficits like data inefficiency “can just be steamrolled if we scale training more, in the same way that all the fundamental research problems in natural language processing collapsed when we threw enough compute into LLMs.”
• The optimists’ answer to his sample-efficiency critique (models are one-millionth as sample-efficient as humans): training is a one-time cost amortized across billions of sessions; what matters is in-session competence, which RL is clearly improving.
• Continual learning “may simply not be necessary” if context windows get long enough: employees aren’t net productive until six months or more on the job — “what if you could just fit those six months into the context window?”

2. Verifiable isn’t enough — the domain must be “grindable”

• His resolution of a puzzle: why is computer use so much slower than coding, when “did my taxes get submitted?” is clearly verifiable? Beyond thin multimodal pretraining data, the underrated reason is you need deterministic, replayable simulators for parallel rollouts from identical starting points. Coding gets a thousand agents on identical containers; “you can’t just have a thousand agents go try the same checkout flow on Amazon… Andy Jassy will find your bots and shut your ass down.”
• Cloning Slack and Gmail works but is “very labor-intensive and unscalable” — until AIs code the clones themselves, which also doubles as a great coding RL objective. So computer use may soon be solved, but its lethargy reveals “the canyon walls against which this river of AI progress will only slowly chip away.”

3. Many valuable skills can’t be farmed — and Dario hints RLVR generalization has limits

• The hard cases: businesses, court cases, trading, elections — reset-free, non-stationary environments where verification “may take months or even years of real-world actions” and can’t be re-observed via perturbed parallel rollouts. “What is the RL environment to make an AI that is as good at politics as Lyndon Johnson?”
• Whether a trillion dollars of RL environments produces an agent that, dropped into Texas politics in 1948, out-advises LBJ is “an empirical question” — but Dwarkesh reads Dario’s train-vs-serve context-length degradation quote as a tell, hedged as “maybe I’m reading too much into this”: if short-horizon RL doesn’t necessarily generalize to long horizons, why would white-collar training generalize to building a business “as well as Sam Walton”?

4. Deployment is where the signal is — and it’s being thrown away

• The compute waste: 30-50% of lab compute goes to inference contributing nothing to model improvement, while deployment is precisely where the most valuable bits — “what kinds of mistakes do I tend to make in the real world?” — are revealed. His image: “some genius grad student who’s never been allowed to take a real internship” fed endless classroom case studies.
• Why context can’t substitute: ever-growing KV caches aren’t scalable and aren’t how humans work — no skull-expansion; savants who recall nonsense syllables with model-like fidelity are crippled at abstraction. Real continual learning is “chiseling the right intuitions… back into the weights.”
• But gradient updates are so sample-inefficient that shipped online learning needs identical objectives across millions of users — Cursor Tab predicts edit acceptance over 400M requests a day — whereas true continual learning needs per-deployment specificity that “simply can’t be stuffed into some shared training run.”

5. OPSD and “dreaming” — the candidate recipes

• Architecture isn’t the bottleneck (sparse attention, KV compaction abound); the loss function might be. OPSD — from his blackboard session with Sasha Rush — trains the base model to match the predictions of the context-laden “veteran teacher” model: no outer-loop verifiable reward needed, and per-token discrepancy gives far denser signal than a single projected reward.
• OPSD also beats SFT for this job: recalling transcripts “with perfect fidelity” is the wrong target, while RL-style sparse updates — “you only change the model as much as is absolutely necessary” — protect against overwriting the base model. He flips his own earlier post: RL learning less per sample “may be a good thing rather than a bad thing.”
• The speculative bet, “dreaming”: models build their own simulators and rehearse — EfficientZero would probably beat a novice human at an unseen Atari game after two hours, but only by playing “dozens of simulated games in its head” per real step. If it works, it’s a fourth scaling axis — instead of /compact’s “simulacrum of continual learning,” you hit /dream and “incinerate huge amounts of compute.”

6. The 2027-28 handoff: deployment becomes the training run

• The scenario: RLVR’s real gift is an agent “at least competent enough to start getting some real-world experience” — deploy it, let effective context stretch to a full week of co-working, then a thumbs-up “work review” triggers distillation of the session into the base model via OPSD, dreaming, or techniques “we aren’t even aware of.”
• The compounding loop: each round the AI improves at domains adjacent to its prior online learning, expanding “far beyond the verifiable domains” — until the main driver of improvement is broad economic deployment, not pre-release training. Closing line, verbatim: “Every time that you interact with an AI, it’ll be smarter, not only because it’s been learning from your previous sessions, but also because it’s been learning from all its interactions with all the other users in the world. And that’s very scary and exciting and different from the way that AI improves right now.”