The data black hole at the center of AI

Summary

• Dwarkesh’s core thesis: AI progress has come mainly from widening the data distribution, not clearly improving sample efficiency — RL is “basically a kind of synthetic data generation,” and models remain “a Frankenstein’s monster… built out of a billion grafts of carefully constructed examples, all sewn together.”
• The tradeable structural claim: Dwarkesh thinks data is the real driver of progress, rather than hyperparameters, training tricks, or architecture — which helps explain why open models can catch up to the frontier within months (Epoch reported a four-month lag). Data “can be easily distilled from public APIs,” suggesting training-trick moats may be thinner than assumed.
• The gap is staggering: humans see ~200M tokens by adulthood vs. tens-to-hundreds of trillions for frontier models — “close to a millionfold difference.” A teen learns to drive in 20 hours; even counting 16 years of growing up and building physical intuition, that is 3-4 orders of magnitude less data than Waymo/Tesla use. If AIs learned like human teleoperators, “robotics would be a deca-trillion-dollar industry.”
• Scaling can’t close the gap: per Chinchilla constants, infinite parameters only cuts data needs 10x, while humans are thousands-to-millions of times more efficient — “humans are on a different scaling curve altogether.”
• The bull case survives anyway: labs can be “ludicrously inefficient in training… and still be wildly in the green,” since skills amortize across billions of sessions — the data industry producing expert labels and RL environments is “earning billions a year in revenue, soon to be deca-billions.”
• Contrarian call: more demand for human software engineers in 2028 than now, “largely due to the complementary input of AI” — software engineering is probably an out-of-distribution job, even though it is the job AIs are supposed to take first.

Deep dive

1. Intelligence is sample efficiency — and progress is unclear

• Dwarkesh’s opening definition: intelligence is how much data you need to operate competently in a domain, and “it’s actually not clear that we’ve made that much progress in training sample efficiency” — gains have come from adding more and better data, with RL as “a kind of synthetic data generation” dumping compute against a verifier or rubric.
• Because the model needs some prior probability of the correct solution, you need “mind-stretching amounts of human expert trajectories” per skill — his intuition pump: Mercor/Surge listings for Word-file polishers, M&A diligence writers, consultant market-research templates. At least hundreds of experts correspond to each skill, and with GRPO, hundreds to thousands of rollouts per task to solve credit assignment.
• The signature image: not a human who learned skills, but “a Frankenstein’s monster… built out of a billion grafts of carefully constructed examples, all sewn together.”

2. Data may explain rapid frontier catch-up

• His read on Epoch’s finding that open models lag by four months: data is the real driver, and “data can be easily distilled from public APIs, whereas hyperparameters, training tricks, and architectural optimizations cannot.” If the latter mattered most, catching up would be far harder than observed.

3. The millionfold gap, quantified

• At ~2,000 words/hour, a human sees ~200M tokens by adulthood vs. tens-to-hundreds of trillions for frontier models — “close to a millionfold difference.” A human can learn to teleoperate a robot arm in hours; if AIs matched that, “robotics would be a deca-trillion-dollar industry” with “an endless army of Unitree G1s.”
• Driving: 20 hours of teen practice — even counting 16 years of physical intuition — is 3-4 orders of magnitude less data than Waymo and Tesla use.

4. Three objections, dismantled

• Evolution-as-pretraining (a point Dwarkesh thinks Karpathy made on his podcast): the genome is “only three gigabytes,” 1-2% protein-coding — “simply not enough space to store the parameters.” Evolution found hyperparameters and loss functions; the connectome is built from scratch. And even granting it, each marginal skill still takes enormous data — an educated human doesn’t need a hundred professors for a new programming language.
• Multimodal tokens: blind and deaf people still have general intelligence, so sensory tokens are “not really the thing that is making humans smart” — deaf sign-language users probably ingest far less than the 200M language tokens, so the millionfold gap “might be an understatement.”
• Just scale more (100T synapses vs. ~5T params): Chinchilla’s constants say infinite parameters only cuts data needs by 10x, while humans are thousands-to-millions of times more efficient — “humans are on a different scaling curve altogether.”

5. Why the labs can win anyway — and the OOD challenge

• The white-collar bet: common tasks are common, so labs can pull them into distribution — revenue curves suggest “an enormous amount of value” from this even without human-like learning. His analogy: a human who had to read through every public repository on GitHub before becoming a competent software engineer “would be on Social Security by the early stages of your education” — but AIs “firehosing gigawatts of training” amortize across billions of sessions, so labs “can be ludicrously inefficient… and still be wildly in the green.”
• The challenge is out-of-distribution work, and it depends on the job. Software engineering is probably one such job, even though it is the job AIs are supposed to take first — his bet: “more demand for human software engineers in 2028 than there is right now, largely due to the complementary input of AI.”
• The labs’ plan for this latter category is first to automate AI research, then have the automated AI researchers solve the sample-efficiency problem — teased for a future post; current intelligence-explosion discourse is “very clumsy”: people either dismiss AI speeding up AI or “assume that some kind of God pops out the other end,” without reasoning about fast progress “on top of LLMs and the particular kind of intelligence that LLMs are.”