Summary

Morris’s core thesis is that AI’s paradigm shifts are driven less by novel methods than by new data sources. AlexNet paired deep nets with ImageNet; transformers paired models with web-scale pre-training; instruction tuning added human-preference pairs; reasoning added verifiable feedback from math, code, calculators, and unit tests. For investors, the durable moat may be whoever owns the next unexploited data stream: “There Are No New Ideas in AI, Only New Datasets.”
Embeddings and released checkpoints may disclose far more proprietary data than their owners assume. Morris’s system recovered about 90% of a long sentence exactly from an embedding, while a later project used the difference between base and fine-tuned weights to select web data resembling the hidden fine-tuning set. The host highlighted the resulting extraction risk for vector databases and customer data; Morris describes model weights as a highly compressed version of training data.
A shared geometry across independently trained models could make capabilities swappable and reduce integration costs. Using an unpaired CycleGAN-style method, Morris’s team aligned GTR and GTE embeddings despite different architectures, objectives, and data; the representations “just perfectly snap to the same place.” The commercial promise is modular vision, audio, and speech adapters that can be stacked onto backbones without retraining everything, though the evidence remains research-stage.
Current transformers appear to exploit only about 3.6–3.9 bits of memorization capacity per parameter despite 32-bit training. Better architectures, optimization, or measurement might raise that ceiling, creating enormous compute leverage if more of the numerical capacity can be accessed. Morris cautions that this is not yet a universal constant—and the host’s essential pushback is that maximizing memorization could merely produce “the best actual compressor,” not better intelligence.
The AI talent market is pricing scarce research judgment and infrastructure fluency well ahead of product revenue. The host cited a recent Stanford graduate who had worked on reinforcement learning and a three-person company valued at $500 million with no product; Morris suggested the valuation might be paying for the ideas. Morris says becoming a GPU expert lets a researcher act as their own research engineer, while the host says deep knowledge of new GPU architectures is exceptionally hireable. Morris also points to Mojo, vLLM, and SGLang as promising areas.
The hoped-for tiny “cognitive core” remains unproven, supporting continued demand for large base models and frontier compute. Morris doubts that reasoning can yet be separated cleanly from factual knowledge: a model must at least know what search terms to use before tools can help it. The alternative vision—small, fast models plus retrieval, memory, and online learning—competes with the industry’s present course of building “500-billion-dollar data centers in the middle of Texas.”
For researchers and founders, the best timing advantage comes immediately after a new paradigm appears. Morris’s blunt advice is to wait for a shift, then “immediately start working as fast as you can to reimplement it”; after o1 arrived in September 2024, an entire stack of work opened around data, evaluations, and training schemes. His own embedding project illustrates the payoff from persistence: a metric stalled around 30, edged to 35, then jumped to roughly 97 after months of failed approaches.

Deep dive

1. ChatGPT helped move frontier science out of academia

Morris entered machine learning in 2017–2018, when BERT-class models and academic research still looked like the center of gravity. After a pandemic-era Google AI residency in 2020 and 2021, he began his PhD in 2021, after GPT-3’s 175 billion parameters but before InstructGPT, expecting professors and graduate students to remain major sources of foundational ideas.
ChatGPT’s November 2022 release marked the break. When Morris’s grandmother began asking about it, he realized AI had become a consumer category; simultaneously, many new boundary-pushing questions could no longer be answered with academic resources. “The whole meta has shifted”: power moved toward companies, formerly open work became closed, and the questions themselves changed.
His counterintuitive advice to a younger student is to watch for the next paradigm shift, then reimplement it immediately. o1’s September 2024 arrival created a field less than a year old: first reproduction, then questions about the right data, evaluations, and training schemes. In a new field, “no one has any big advantage.”
The talent-market anecdote captures the resulting distortion: the host cited a recent Stanford graduate who had worked on reinforcement learning and said the resulting three-person company was valued at $500 million without a product. The host called the situation “absolutely bonkers,” while Morris suggested the valuation might be buying the ideas rather than current output.

2. The compute gap temporarily stranded academic research below useful scale

Morris contrasts 125-million-parameter BERT models with 7B–8B systems. A 100-million-parameter model, however well trained, might answer almost every US-presidency question with George Washington; at 8 billion parameters, a model suddenly knows presidents and national capitals. That emergence changes which research questions are meaningful.
Academic labs lagged while deciding how many GPUs to buy and whether to purchase the current or previous generation. Morris’s harsh retrospective: for roughly two years, academics kept studying small models while industry moved to 7B–8B scale, and “none of it really mattered.” Access has since stabilized enough for university researchers to fine-tune and run inference at 7B.
Formal training did not close the skills gap. Morris says most graduate students work on one GPU, some use multiple GPUs, and almost none conduct multi-node training without company affiliations: “They don’t teach you anything.” Distributed FSDP, DeepSpeed, and cluster operations are generally learned online or from peers.
Morris recommends the GPU MODE Discord, the fast.ai team, and reaching out to Microsoft’s DeepSpeed team. The host qualifies his earlier downplaying of CUDA: hardware is only one slice of model knowledge and may be a poor starting point, but mastering modern GPU architecture puts someone in a very small, highly hireable group. Morris says the current sweet spot may instead be Mojo, and highlights vLLM and SGLang as important systems.

3. Deep learning needs a theory of usable, not merely present, information

Morris’s starting example compares plain English with the same content transformed into unreadable cryptographic text. Shannon information treats their bit content as equivalent, yet one is immediately interpretable and the other computationally inaccessible. V-information, from a 2020 paper on usable information under computational constraints, tries to measure what an observer can actually extract.
That lens explains why pre-training helps on unseen downstream data. Random and pre-trained weights may contain the same number of stored bits, but pre-training has performed computation that makes relevant information easier to extract. Morris describes weights as a nonlinear compression of training data plus the training algorithm, with activations adding the model’s current computation.
His historical analogy is telephony before information theory: engineers accumulated heuristics until the bit supplied a common unit for signal, error correction, and system design. Deep learning is still missing its corresponding primitive. “I don’t think we know what a bit is yet in terms of deep learning models,” he says, while inviting others to define it.
The host’s back-of-the-envelope example puts GPT-3 at 175 billion parameters and 350 GB using 16-bit storage, versus roughly 150 GB for Wikipedia—about 2.3 Wikipedias. Both speakers immediately qualify the comparison because text compresses heavily; tokenization and Kolmogorov complexity matter. Still, Morris values such arithmetic for grounding otherwise abstract questions.

4. Text embeddings are compressed records, not privacy-preserving substitutes

The motivating arithmetic was similarly simple: OpenAI embeddings had 1,536 dimensions, and Morris’s estimate using half-precision implied roughly 20 KB per vector—enough storage for many paragraphs of text. Meanwhile, heavily funded vector-database companies were exchanging customer vectors instead of raw documents, raising the practical question: “What data are they actually sending?”
Morris proposed reverse-engineering text from embeddings after substantial exploration and discarded research ideas. The resulting 2023 paper, Text Embeddings Reveal (Almost) As Much As Text, recovered about 90% of a long sentence exactly under the highlighted setting. He believes some vector-database providers subsequently changed privacy policies, which he found gratifying.
The research path mattered as much as the result. Months of architecture, decoding, and inference-time adaptation experiments left a metric near 30; Morris reached 35 and thought the work was finished. His advisor cautioned that proposing a new problem and showing only a move from 30 to 35 was confusing and not necessarily meaningful, after which the team found a method reaching roughly 97: “Who knew you could get this much information from an embedding?”
One follow-up turned inversion into an interpretability check. A debiasing method removed gender-correlated features from embeddings of “This woman is a doctor. She works at Weill Cornell”; the recovered text became “This person is a doctor. They work at Weill Cornell.” The output supplied human-readable evidence that the latent gender signal had actually changed.

5. Independent embedding models converge on a shared geometry

The first inversion attack assumed the adversary knew the encoder and could query it repeatedly. Critics correctly noted that a private or fine-tuned embedding model would break that assumption. Morris’s group nevertheless suspected that independently trained encoders learn similar structures because nearest neighbors often match and related fine-tunes can sometimes exchange representations.
The Platonic Representation Hypothesis sharpened that intuition: “All models are trained on data from the world. And there’s only one world.” As models scale in data and quality, they may converge toward the same underlying concepts even if their architectures, bases, and raw coordinates differ.
The methodological inspiration was CycleGAN. It can translate Monet paintings into photographs or horses into zebras without paired examples of the same scene, somehow preserving animals, legs, grass, and camera position. Morris’s team repurposed that unpaired distribution-matching idea for embedding spaces.
Applied to GTR, a T5-based retrieval model, and GTE, a BERT-based model, the method aligned representations despite different training data, architectures, objectives, and embeddings. The host found the result less profound than Morris did, saying it matched his intuition. Morris conceded that it does in hindsight, but distinguished having an intuition from demonstrating that an unseen mapping works.

6. Shared geometry points toward modular models, but compression sets hard limits

The practical prize is the ability to “glue together models.” Morris connects the result to systems in which a language-model backbone stays resident while separately trained vision, audio, speech, or text-to-speech parameters load only when required. If latent spaces can be aligned reliably, capabilities become “swappable, stackable” and less vulnerable to catastrophic forgetting.
Their open-source exchange preserves an important distinction. Gemma qualifies as open source if releasing weights on Hugging Face is sufficient; under a stricter definition it is open weights because neither the training data nor complete code is released. Both agree that relatively few groups build genuinely strong, efficient models.
Context length bounds inversion. Compress an entire book into a 500-dimensional vector, and two versions differing by one word may produce a collision; once lossy compression maps distinct inputs to the same code, exact recovery is impossible. Morris has no firm boundary for how much text fits, and floating-point precision further complicates the theoretical answer.
The host originally found Morris through contextual document embeddings, calling the two-stage CDE technique a potential “free lunch.” Morris groups that work with inversion and geometry under one question: what information and computation are represented in activations, as distinct from what is stored in weights?

7. Transformer memorization plateaus at a measurable capacity

Morris’s capacity experiments train language models on independent random strings while varying dataset size. Total memorization rises and then reaches a nearly flat plateau, even around one million examples, suggesting a stable model capacity. He speculates that the transformer may be dividing a tiny slice of its parameters or computation among every unrelated data point.
The internal implementation is unknown. A model might form independent computational columns for each example, or learn generic load-and-store operations that combine many random strings. Morris hopes to study this mechanism because the aggregate capacity curve is clear while “I have no idea how the mechanisms are actually implemented inside the models.”
For transformers trained in 32-bit precision, the measured capacity is approximately 3.6–3.9 bits per parameter. The host’s reaction—“Just store 32, bro”—captures the apparent waste. Practical value might come from predicting the exact model size and architecture a dataset requires, or finally explaining why LoRA training under 1% of parameters differs from full fine-tuning.
Morris does not call 3.6 a hard constant. A Llama-style or “GPT++” architecture, better optimization, or more suitable nonlinearities might push it upward; improved measurement could also reveal perhaps five reachable bits. The host’s caveat is decisive: memorization is not generalization, and optimizing the metric might produce “Gzip” rather than intelligence. Morris agrees the better move may be asking a different question.

8. A tiny reasoning core may be impossible without substantial world knowledge

The host introduces Andrej Karpathy’s hoped-for “cognitive core”: the smallest model smart enough to use tools while externalizing facts, ideally running quickly on-device. Gemma 3n, described as a 4B model and claimed on certain unnamed arenas to outperform Llama 4 and GPT-4.1, is offered as a candidate.
Morris doubts present methods can cleanly separate reasoning from knowledge. The host invokes a conversation with Noam Brown in which reasoning was described as emergent: placing the o1 harness over GPT-2 would yield nothing because GPT-2 lacks the necessary base competence; GPT-3- and GPT-4-level knowledge is needed before such reasoning can work. Even Google use requires knowing which search terms to enter.
A model could learn to search whenever uncertain, discouraging factual storage in weights, but Morris says that likely requires online learning, memory, or both. The aspiration is attractive; the deployed alternative remains ever-larger pre-training and “500-billion-dollar data centers in the middle of Texas.”
Humans are an imperfect existence proof for efficiency: the host estimates roughly 100 trillion neurons if treated as parameters, operating at about 20 watts, with connections that can drop through self-pruning. Morris’s open question is whether a 100-million-parameter model could ever reach today’s o3 Pro level through special data, architecture, or training—or whether current scaling curves already imply a plateau.

9. Weight differences can reveal the shape of hidden training sets

Open model weights are another dense information store. A downloaded model is a highly compressed transformation of its training corpus, yet Morris says he would be extremely surprised if providers did private training and suspects they rely on the absence of tools for “decrypting” the weights. His long-term ambition is to translate those weights back into text datasets.
His proof of concept simplifies the task by requiring two checkpoints. In the DeepSeek case, if one downloads the roughly 400-billion-parameter model weights, there are base and fine-tuned versions, leaving a parameter-space difference that can indicate what the model was fine-tuned on. Given the corporate and geopolitical debate over DeepSeek’s data, Morris calls that delta a “treasure trove of information” without an interpreter.
Instead of regenerating exact examples, the method scores candidate web documents by whether each example’s gradient points in the same weight-space direction as the observed fine-tuning delta. Selecting the highest-scoring data yields a corpus that trains a reasonably strong model, though it does not match the original model’s performance.
The proposed use case is that when a competitor publishes base and fine-tuned checkpoints, one could search a very large web corpus for examples that best reproduce their delta. DeepSeek, Mistral, and Llama releases make the setup plausible, but Morris presents the current method as an early approximation rather than completed extraction.

10. Every major AI paradigm paired a method with a new dataset

Borrowing Thomas Kuhn’s The Structure of Scientific Revolutions, Morris divides progress into rare paradigm shifts followed by rapid incremental work and eventual slowdown. Researchers prefer “cute new methods” that beat predecessors on fixed data, but his historical claim is that transformative systems consistently arrived with new data regimes.
The first shift paired deep neural networks with ImageNet’s millions of images. The second paired transformers, BERT, and the first GPT with web-scale pre-training. Morris praises transformers’ extraordinary durability but argues the essential leap was scraping, tokenizing, and learning from internet-scale text.
The third shift was instruction tuning, whose crucial asset was positive and negative human-preference data—not necessarily the original reinforcement-learning technique, since supervised learning and DPO can use similar signals. The fourth was reasoning with machine-verifiable feedback from mathematics, calculators, coding environments, and unit tests.
His deliberately provocative counterfactual is that the right datasets might have pushed sophisticated 2015-era methods, perhaps even RNNs, toward something resembling InstructGPT. He concedes RNN scaling curves are worse and will not claim they could reproduce reasoning models. The thesis elevates data without proving architecture irrelevant.

11. The next paradigm remains unknowable, but its scale is identifiable

The host’s rebuttal is that optimizer and architecture improvements translate directly into cheaper effective data and compute. Frontier systems may contain hundreds of small gains that stack; Muon, if adopted, could be an “atom killer,” while any better optimizer is a genuine free lunch in training time and cost. Morris agrees those gains are enormously valuable.
What the data thesis supplies is a filter: whatever follows reasoning should be comparable in magnitude to ImageNet, web text, human preferences, or symbolic verification—not another small benchmark improvement. Morris’s candidates remain tentative: video, embodied AI, or “some type of new data source that we’re not using yet.”
He refuses to manufacture a forecast: “Predicting the future is too damn hard.” Reasoning models began with o1 in September 2024 and may have several productive years left, so the fifth paradigm could look obvious only in hindsight. That uncertainty is preserved rather than converted into a confident product roadmap.
Morris’s research focus remains information contained in model weights and activations, including capacity, representation alignment, inversion, and underlying mechanisms. He describes his papers as occupying “a very small manifold of all possible problems” and invites collaborators working anywhere adjacent to that space.

The AI Village: Previewing the Giga-Agent Future with Adam Binksmith, Founder of AI Digest Market Overview, July 1, 2025 (Rebroadcast)