Better Data is All You Need — Ari Morcos, Datology
Summary
Datology’s core thesis is that data quality is a compute multiplier: “models are what they eat.” Morcos defines curation broadly—filtering, rebalancing, sequencing, batching, source mixing and synthetic generation—and argues it can make models train faster, reach higher capability and use fewer parameters. For investors, the wedge is not merely lower pre-training spend but better performance per dollar and structurally lower inference costs.
Morcos’s conversion to data came from evidence that overturned six years of his own architecture research. ConViT’s soft convolutional bias helped below roughly 500,000 data points in image self-supervised learning, lost its advantage as data grew and became mildly harmful beyond about 1 million; several projects delivered the same “bitter lesson.” His conclusion was categorical: at sufficient scale, “all that really matters is the learned posterior from the data distribution.”
Datology claims gains large enough to challenge naive power-law scaling economics. Against DCLM, Morcos says its curated data reaches equivalent performance about 12x faster, using fewer than 10% of the tokens; under equal compute it adds roughly 4–5 absolute benchmark points, while models with fewer than half the parameters can outperform larger baselines. He sees “at least another 100x” of headroom beyond gains already approaching 10x.
Synthetic data works best when the model reformats knowledge rather than inventing it. Morcos is skeptical that net-new generation—“distillation in disguise”—can surpass its teacher without rejection sampling or another information source, and repeated generations risk collapsing from tails toward modes. Rephrasing is different: a relatively weak model can make source information more learnable, but Morcos says diversity is the key constraint because “textbooks are not all you need.”
Curriculum learning is returning because models moved from saturating scarce datasets to underfitting enormous ones. Ordering once offered limited value when ImageNet could simply run for 160 epochs; now it can determine whether a training run costs 10x more. Morcos wants pre-, mid- and post-training treated as one curriculum, including optimizing pre-training data for steeper RL and test-time-compute curves—or a flatter jailbreaking curve.
The enterprise prize may be “train smaller,” not simply “train cheaper.” Morcos’s example: a company spending $50 million annually on inference wastes $25 million in year one if its deployed model is twice as large as necessary, while a specialized replacement might cost only $2–3 million to train. He expects most models people use in three years to be single-digit billions of parameters or smaller, especially as test-time compute multiplies the cost of every reasoning step.
Datology’s moat depends on keeping some scientific know-how proprietary while proving that curation stacks. In the RC engagement, the host described 23 trillion tokens reduced to 6.6 trillion, while Morcos described roughly 25 trillion source tokens yielding about 7 trillion; the 4.5B model was already consistently beating Gemma before 1 trillion training tokens. The broader claim is that even DCLM, Nemotron and FineWeb remain far from exhausted: “We’re just scratching the surface.”
Deep dive
1. Datology treats the entire path from storage to training as an optimization problem
Morcos’s mission statement spans every decision between stored data and the model’s data loader: what to filter, synthesize, sequence, batch, upsample, downsample and mix. Each choice can materially alter the resulting model.
His signature framing is deliberately simple: “Models are what they eat.” Great inputs produce high-quality models; low-quality inputs produce low-quality ones, but identifying and assembling the useful inputs across trillions of tokens or billions of images is a frontier research problem.
The promised output has three dimensions: train much faster, train to better performance and train much smaller models that equal or beat larger ones. Automation is essential because expert-by-expert human review cannot operate at frontier scale.
2. An empirical-science background led Morcos away from architecture cleverness
Morcos began in neuroscience, trying to teach mice to count and analyzing thousands of neurons while they did it. Machine learning initially entered his work as a tool for understanding those neural datasets, not as his original field.
Starting his PhD in 2011 put AlexNet and DQN alongside that transition. He came to deep learning as an empirical scientist and wanted papers with two halves: first explain why a representation was good or bad, then use that understanding to improve the system.
That second half repeatedly failed. A property correlated with good representations would look actionable, but optimizing it often revealed that it was “a correlate,” not a causal variable; of perhaps 30 explanatory papers, only three or four successfully turned the explanation into an improvement.
The frustration carried a methodological lesson: deep learning’s rules may be designed, but the properties emerging after training on vast datasets are often unexpected. “Throwing spaghetti against the wall” was unsatisfying, yet clean mechanistic understanding rarely provided a reliable intervention.
3. ConViT delivered Morcos’s personal version of the bitter lesson
ConViT initialized a vision transformer exactly like a convolutional neural network: with a 3×3 kernel, nine attention heads could correspond to its nine positions. Unlike a CNN’s hard convolutional bias, the transformer could later undo the imposed weight tying.
The soft bias helped substantially in small-data image self-supervision—roughly below 500,000 examples—and found uses in scientific domains such as volcano prediction with around 1,500 data points. Its advantage decayed with scale and became mildly harmful beyond approximately 1 million examples.
Several papers around 2020 confronted Morcos with the same conclusion: once sufficient data arrives, built-in inductive biases cease to dominate and the learned data distribution determines the result. That was painful because he had spent about six years researching those biases.
Faced with the “bitter lesson,” he saw two rational directions: make “GPUs go burr,” or work on data. As a non-hardware engineer, he chose what he calls “the most underinvested-in area of research relative to its impact—and I don’t think it’s even close.”
4. Self-supervision changed data from a scarce asset into an unruly abundance
Research incentives once treated datasets as fixed: given ImageNet, maximize validation or test performance. Kaggle reinforced the same frame, and supervised datasets at least had a quality floor because a human had inspected every example.
Morcos regards transformers as one of many architectures that might have achieved comparable performance; the irreplaceable advance was self-supervised learning. Predicting one part of an example from another removed external labels and expanded training from roughly 1 million ImageNet examples to trillions of tokens—a “millionfold increase.”
That flipped models from repeatedly saturating scarce data—perhaps 160 ImageNet epochs—to underfitting immense scraped corpora. The new regime brought redundancy, low-information examples and no guaranteed quality floor, making data selection economically central rather than incidental plumbing.
His objection to classic Kaplan- and Chinchilla-style scaling analyses is that they assume IID data. “All data are not created equal”; treating each token as equivalent ignores the oldest warning in computing, “garbage in, garbage out.”
5. Human judgment cannot value examples because value is relational
Morcos cites DCLM’s sharpest finding: after roughly 30 graduate students spent two years designing filters, those nominal experts could not predict above chance whether the resulting classifier would keep or reject a given example.
Humans cannot scale to billions of examples, but his stronger claim is that they would be undesirable even if they could. A data point’s value depends on its relationship to everything already in the training set, not merely its standalone polish.
Ten thousand well-written summaries of Hamlet illustrate the trap: any one may look accurate, clear and valuable, yet the dataset does not need all 10,000. A reviewer cannot keep the entire corpus and its redundancies in mind.
Nor is zero redundancy optimal. Elephants are comparatively stereotyped, while dogs span hundreds of breeds, mixtures, sizes, textures and colors; understanding dogs therefore needs more repetition. A system must discover concepts unsupervised, estimate their complexity and adapt how aggressively it removes or retains examples.
6. “Concept” itself is a tunable quantity, not a fixed ontology
The hosts press Morcos on whether the relevant unit is elephant, dog, mammal or something finer. His answer is intentionally unsatisfying: concept granularity is a hyperparameter, an empirical knob controlling when clusters split and how much redundancy survives.
Datology has run “hundreds and hundreds of thousands of experiments” because no universal setting works for one corpus, much less every proprietary distribution. The scientific problem is paired with a deployment problem: generalize zero-shot to arbitrary new data.
This also explains why curation is not one silver bullet. Morcos describes perhaps 50 interventions, each producing a modest gain; the large improvement comes from finding them and making them compose, although “by default, they don’t play nice with each other.”
7. Better information gain could bend the scaling curve itself
Morcos says the host’s lawyer explanation for disappointing mega-models such as GPT-4.5 or Llama 4 is possible, but thinks scaling challenges and diminishing returns are probably the larger explanation.
Power-law scaling means every 10x increase in data buys a smaller performance increment. In “Beyond Neural Scaling Laws,” Ben Sorscher and Morcos linked that behavior, for perceptrons, to marginal information gain per example also decaying as a power law.
The proposed mechanism is direct: if curation can keep information gained from successive examples flatter instead of letting it decay, then performance need not follow the same curve. Datology’s purpose, Morcos says, is to turn that theoretical potential into a working system.
The investor-relevant implication stays in his own terms: better data changes “performance per dollar by orders of magnitude.” It is not merely another input cost; under persistent underfitting, it increases the value extracted from every unit of compute.
8. Datology reports 12x faster learning and 4–5 absolute points of lift
Against DCLM, Morcos says Datology can match convergence performance about 12x faster—fewer than 10% of the tokens. Evaluation averages accuracy across 15 standard tasks appropriate to model size, including MMLU, ARC and RACE.
The hosts challenge whether that invites benchmark overfitting. Morcos agrees the danger is acute, especially for synthetic-heavy models that score well but fail “vibe checks”; Datology keeps rarely viewed held-out tests and uses additional evaluations to guard against this.
Under equal compute, he reports beating leading open datasets by roughly 4–5 absolute points on average. Closing that gap by merely extending baseline training would generally require 5–10x more tokens because each successive accuracy point becomes harder.
Holding performance constant, Datology says models with fewer than half the parameters can train faster and still outperform larger models trained on less-curated data. Morcos credits unusually large headroom: gains already near 10x, with “at least another 100x” potentially remaining. He says most of these results came from a team of under 20, within a company now numbering about 30, and with far less compute than frontier labs.
9. Open data improves slowly because incentives favor secrecy and modeling demos
Morcos says frontier labs have invested heavily in better crawlers, yet crawling is different from asking which candidate example will teach a model the most given everything it has already seen. That conditional valuation remains largely secret and unsolved. He describes Nemotron, released about six months after DCLM, as having more unique tokens but roughly similar average quality.
Dataset folk wisdom often fails: GitHub stars are not a useful predictor of whether a repository improves coding capability. Work including StarCoder found no strongly discriminative proxy; simple properties such as length could outperform popularity signals.
On books, Morcos characterizes a recent appellate ruling as allowing training when the books were purchased but not when pirated, while emphasizing that the matter could proceed higher. At Meta, he saw legal approval around datasets become difficult enough that some risks could only be authorized by Mark Zuckerberg.
Datology therefore faces a genuine open-science tension. Its prospective moat can come from scientific know-how, engineering infrastructure and eventually brand; publishing enough detail for full reproducibility could erase the first, so its blogs aim to share mechanisms and intuition without giving away the complete recipe.
10. Synthetic generation divides into invention and transformation
Morcos separates net-new generation from rephrasing. When a generator supplies the knowledge itself, the process is “distillation in disguise,” and he doubts the student can reliably surpass its teacher without rejection sampling or another source of supervision.
Model-collapse arguments apply most directly there: a generator overrepresents modes and underrepresents tails, then repeated generations narrow the distribution toward “only mode,” eventually resembling a delta function. Filtering after each generation injects information and can interrupt that collapse; the hosts connect this to RL and verifiers.
Rephrasing instead conditions on existing source material. The generator need only restructure, clean or expose that information in a more useful or downstream-relevant format, so a comparatively weak model can create training data that teaches a substantially stronger model.
Morcos describes Datology’s forthcoming “Beyond Web” work as showing that rephrasing can outperform training on all raw tokens and potentially “break this data wall.” The crucial distinction is that the knowledge comes from the source data, not the model performing the rewrite.
11. Diversity, not textbook style, is the governing synthetic-data constraint
Asked whether “textbooks are all you need,” Morcos answers, “No, they are not all you need.” Textbooks contain excellent material but occupy a narrow distribution; if listeners retain one rule about data quality, he wants it to be “diversity.”
His changed mind is explicit: after researching out-of-distribution generalization, he bet scaling could not yield compositionality. Modern training effectively asked, “What if nothing was out of distribution?”—put everything into training—and it worked “shockingly well.”
The hosts point to Microsoft’s Phi series and question whether rewriting everything like textbooks became cargo cult. Morcos links narrow synthetic distributions to models whose benchmark results exceed their real-world appeal; Datology counters by rewriting into “many many many different styles and formats.”
One textbook lesson does survive: repeating high-quality tokens is almost always better than consuming new average- or unknown-quality tokens. Rephrasing those valuable examples can improve on literal repetition while retaining their information.
12. Curriculum learning matters once iteration—not saturation—is scarce
Morcos argued in mid-2023 that curricula would return while many researchers insisted they did not work. His conceptual argument uses a graph of concepts and prerequisite edges: curricula are useless only if the graph has no edges or every pair has an equally weighted edge, neither of which describes learning.
In supervised learning, reaching ImageNet performance in 80 rather than 160 epochs was useful but rarely decisive. With models underfitting huge corpora, ordering can instead decide whether training needs 10x the compute and potentially hundreds of millions of additional dollars.
He treats mid-training as a later phase of a discrete curriculum and wants pre-, mid- and post-training optimized as one process. Separate teams at major labs behave like customers in a chain, making it hard for downstream signals to reshape upstream data.
The hosts’ elicitation objection strengthens his case: if post-training exposes capabilities already present, pre-training should be optimized for a steep RL or test-time-compute curve. Conversely, he wants pre-training data optimized for a shallow jailbreaking curve; “if it’s easy to put it in, it’s easy to take it out.” He points to Qwen being easier to reinforce-train than Llama, likely because Qwen included synthetic reasoning traces—even incorrect ones—in pre-training, as evidence that the base model matters more than the reward signal.
13. Enterprise economics shift the prize from cheaper training to cheaper inference
Datology mainly serves organizations training from scratch or continuing pre-training with proprietary domain data, typically across at least tens of billions of tokens. After the host challenged the idea of literal country-owned models, Morcos agreed that current sovereign-AI examples are largely government grants or public-private partnerships.
“Train faster” rarely means spending $1 million for a model that previously cost $10 million. The host summarizes the customer ask as wanting a $100 million result for $10 million; Morcos agrees that customers generally want to train better. The operational value of faster training is compressing a ten-day experiment into an overnight run so the same team can iterate far more often.
“Train better” turns data into a compute multiplier, but advanced customers may value “train smaller” most. At $50 million of annual inference, deploying a model twice as large as necessary wastes $25 million in year one; training a better specialized model might cost only $2–3 million.
Morcos describes the enterprise target as “an inch wide and a mile deep”: a few tasks at “five nines” reliability and minimal serving cost. He believes specialized frontier-level training should soon cost $1 million or less for most organizations, while data remains the harder barrier after training infrastructure became largely commoditized.
14. Smaller models, stacked curation and task-relative valuation define the endgame
Parameter pruning still has a role, but Morcos no longer sees it as a panacea. His lottery-ticket work found that winning tickets were data-dependent; unstructured pruning removes many weights but incurs sparse-matrix overhead, while GPU-friendly structured pruning performs worse. Better-data training can stack with pruning and quantization.
He predicts that most models people use in three years will be single-digit billions of parameters or smaller. Test-time compute reinforces the direction: total cost equals per-step inference cost times reasoning steps, while stored factual knowledge consumes capacity that tool-using “cognitive core” models may not need. He likes the possible target of a model with near-zero GPQA performance but 100 on BrowseComp.
The RC engagement supplied a public demonstration of stacked techniques. From DCLM, Nemotron and FineWeb, Morcos describes roughly 25 trillion combined tokens curated to about 7 trillion; the 4.5B model beat Gemma consistently before 1 trillion tokens, and Datology now believes the same source corpus can yield 15 trillion tokens at similar quality.
The long-term scientific target is automatically valuing data for a specified downstream task, including proprietary distributions Datology cannot inspect freely. “There’s no golden curation”: optimal filtering, synthesis and weighting depend on the use case, making task-relative data valuation what Morcos calls “kind of the NP-complete problem of AI.”