Summary

Google’s model strategy, in Jeff Dean’s telling, optimizes the whole stack rather than one flagship model. Frontier models expose new capabilities and supply the teacher logits needed to distill smaller models; Flash then converts those gains into lower-cost, lower-latency deployment across products such as Search. It is “not just one thing” but “a whole bunch of things up and down the stack.”
Flash catching yesterday’s Pro does not eliminate the frontier-model business because demand is not stationary. As coding models improve, users move from asking for a loop to requesting a software package; analytical requests similarly expand toward synthesizing worldwide renewable-energy deployments. Dean’s answer to commoditization is that capability creates its own harder workload, while frontier models provide the teacher for affordable tiers.
Useful long context ultimately requires the illusion of attending to trillions of tokens, not merely stretching quadratic attention. Gemini is pushing 1 million or 2 million tokens while simple needle-in-a-haystack tests are largely saturated around 128k; Dean instead envisions staged retrieval narrowing roughly 30,000 documents and 30 million interesting tokens to perhaps 117 documents for the strongest model. The product endpoint is permissioned access to “every email, every photo, every video I’ve watched.”
The decisive hardware cost is data movement: a multiply can cost sub-picojoule to roughly one picojoule, while moving data across the same chip can cost roughly 1,000 picojoules. That gap explains batching; together with sparsity and low precision, it motivates SRAM-resident models spread across 16 or 64 chips and speculative decoding. Google’s TPU-model co-design operates on an unusually long clock, predicting workloads two to six years ahead and targeting changes as far out as TPU N+2.
Dean expects general models to win in most cases, but sees retrieval and modular specialization as necessary answers to finite model capacity. A unified, roughly production-like model with extra inference budget replaced separate symbolic and geometry systems for IMO-level mathematics; smaller models, however, should not spend “precious parameter space” memorizing obscure facts. Healthcare, robotics, and low-resource languages may instead use enriched training, private data, retrieval, or installable modules attached to a strong base.
Coding agents move engineering leverage from typing code toward specifying, supervising, and organizing work. The discussion treats “50 interns” as plausible if they are good, with Dean suggesting they could be organized into subteams. Output quality will increasingly depend on being “pretty darn careful” about requirements, corner cases, and performance constraints. The hosts discuss 20x to 50x lower latency, while Dean says 10,000 tokens per second remains meaningful when 9,000 tokens can be reasoning behind 1,000 tokens of final code.
Gemini’s organizational origin reinforces the full-stack thesis: fragmented compute, talent, and ideas were themselves a bottleneck. Dean says he wrote a one-page memo arguing Google was “being stupid by fragmenting our resources” across language, multimodal, Brain, and legacy DeepMind efforts; the answer was one multimodal-from-the-start program. Its gains compound with earlier multipliers—sparsity delivered roughly 10x compute-to-quality improvement, transformers 10x to 100x, and an early 2-billion-parameter system achieved a 70% relative error improvement on ImageNet 22k.

Deep dive

1. Frontier capability and affordable inference are complements

Dean describes the stack behind Google’s model strategy as spanning highly capable large models, software that transfers those capabilities into smaller models, and hardware that serves both efficiently. Google wants a frontier model for deep reasoning and difficult mathematics alongside an affordable, low-latency model for broader workloads such as agentic coding.
The frontier model is not optional overhead: it reveals “what capabilities now exist that didn’t exist” in recent versions, then supplies the teacher for distillation. Dean says both tiers are useful and that “you need the frontier model” to obtain a highly capable smaller model.
Dean traces distillation to a 300-million-image, roughly 20,000-category data set. Around 50 specialists enriched on clusters such as mammals or indoor scenes formed a strong but impractical ensemble, prompting the team to “squish it into something that actually fits in a form factor that you can actually serve.”
Modern distillation replaces hard labels with logits from the larger model and supports repeated passes over a large training set, “coaxing” behavior the smaller model would not otherwise learn. The reported trend is explicit: across multiple Gemini generations, the next generation’s Flash has matched or substantially beaten the previous generation’s Pro.

2. Lower prices expand the workload instead of ending the frontier race

The hosts connected Flash’s economics to a changing estimate of “like 50 trillion” tokens and its presence across Gmail, YouTube, and other Google surfaces; Dean added that it is increasingly used in Search’s AI Mode and overviews. Its advantage is not only affordability but latency.
Lower latency matters because requests are expanding from “write me a for loop” toward “write me a whole software package.” More capable agents will generate many more tokens before returning, making inference speed part of effective capability rather than a cosmetic user-experience metric.
Asked whether Flash eventually saturates most user needs, Dean’s qualification was decisive: that holds only if “the distribution of what people are asking the models to do is stationary.” His own coding requests became more ambitious as models improved; analytical requests can now encompass worldwide renewable-energy deployments rather than isolated facts.
Dean prefers benchmarks whose first scores are roughly 10% to 30%, leaving room to climb toward 80% or 90%. Around 95%, returns diminish because the capability may be solved or public-data leakage may dominate, so Google relies on held-out internal evaluations known to be absent from training.

3. Long context must become hierarchical attention over the world

Gemini 1.5’s long-context work reflected a desired capability rather than loyalty to one architectural trick. Single-needle tests are largely saturated through approximately 128k tokens, with many systems around 128k or 256k, while Google is pushing 1 million or 2 million.
The useful tests now involve synthesizing a thousand pages or several hour-long videos, not retrieving one product number. Dean wants multi-needle and realistic synthesis evaluations that measure whether the model can turn a vast context into the answer a user actually needs.
Quadratic attention cannot simply scale from 1 million tokens to a billion or trillion. Dean’s target is algorithmic and system-level progress that can “give the illusion that you can attend to trillions of tokens,” enabling attention over the internet, YouTube’s pixels and representations, or a user’s permissioned personal state.
His proposed shape resembles Search: lightweight parallel models identify perhaps 30,000 documents containing 30 million interesting tokens; a stronger stage narrows those to roughly 117 documents; the most capable model performs the final analysis. “Can I attend to the internet while I answer my question?” is the capability goal.

4. Native multimodality extends far beyond text, images, and audio

Dean wants Gemini exposed to “hundreds of modalities,” including Waymo LiDAR, robot sensors, X-rays, MRIs, other health imaging, and genomics. Even when the main pre-training mix cannot absorb all available data, a small exposure can teach the model that the modality exists and carries meaning.
On whether vision is a “king modality,” Dean broadened the answer to vision plus motion. He cited eyes evolving independently in 23 ways because sensing the surrounding world is unusually useful, and highlighted native video understanding as a distinct capability.
His sharpest video example was an 18-event sports highlight reel spanning about 20 years. Given only the video, Gemini produced an 18-row table naming each event, its date, and a description—an operation Dean characterized as turning “video into SQL-like table.”

5. Google Search’s evolution supplies the systems blueprint

Even before LLMs, Search began with an enormous index, used lightweight methods to find perhaps 30,000 relevant documents, then applied progressively more sophisticated signals before showing around 10 results. LLM-based retrieval changes the representations, but Dean expects the funnel-shaped system to remain.
In 2001, one deployment used roughly 60 index shards with 20 replicas each: 1,200 disk-equipped machines. The team realized one complete index could fit across their combined memory, eliminating repeated disk seeks and allowing a three- or four-word query to expand toward 50 terms, including synonyms such as restaurant, restaurants, cafe, and bistro.
Moving the index into memory improved meaning-based retrieval years before LLMs. The most dramatic scaling dimension was eventually freshness: updates progressed from once per month to under one minute for any page, with recrawl schedules balancing the probability that a page changed against the value of keeping important pages current.
Dean’s design rule is to accommodate key parameters growing by factors of five or 10, “but probably not beyond that.” A 100x change often makes a different architecture rational; back-of-the-envelope latency, bandwidth, seek, and memory calculations help expose that new design point before code is written.

6. AI inference economics are governed by moving bits

An exchange about “numbers every AI programmer should know” centered on state movement: on-chip SRAM, accelerator-attached HBM, DRAM, and the network. The host’s analysis put a multiply at around or below one picojoule, while moving data from the other side of the same chip could approach 1,000 picojoules.
That disparity explains batching in the host’s analysis. Once a model parameter has incurred roughly 1,000 picojoules to reach a multiplier, “you better make use of that thing that you moved many, many times”; a batch of 256 or so amortizes the movement, while batch size one offers excellent latency but poor efficiency.
For a sufficiently small model, TPU model parallelism can stripe it across 16 or 64 chips so that it resides in SRAM rather than paying the higher cost of bringing it from HBM. The regular two- or three-dimensional TPU mesh and high-performance interconnect can then improve both throughput and latency.
Speculative decoding creates another small effective batch: predict eight tokens with a draft process, accept perhaps five or six, and obtain about a 5x improvement in amortizing weight movement. Dean evaluates such techniques through the joint lens of real energy, latency, and throughput.

7. TPU design is a two-way bet on future model architectures

TPU architects and model researchers co-design around “where we think the ML research puck is going.” A chip started today may take two years to reach a data center and remain relevant for another three, four, or five, forcing Google to forecast computations two to six years out.
Major features may target TPU N+2, though smaller changes can sometimes reach N+1. Cheap speculative circuitry is attractive when it consumes little area but could make a promising method 10x faster; large architectural bets require substantially stronger ML experiments before commitment.
Co-design runs backward too: model architectures are adapted to train and serve efficiently on the committed hardware. Researchers may train at a lower precision supported by a future TPU even when the current generation does not yet support it.
Dean is “a big fan of very low precision” because energy scales with bits moved; shared scaling vectors can apply across groups of low-bit weights. Analog substrates remain intriguing, but digital-to-analog and analog-to-digital boundary conversions can consume much of their theoretical power advantage.

8. Reliability and non-verifiable reinforcement learning remain open

Dean’s research agenda includes models that reliably execute much longer jobs containing many subtasks, plus orchestration in which one model calls other models as tools. The aim is to complete “much more significant pieces of work collectively” than a single model can handle alone.
The harder unresolved problem is getting RL gains beyond verifiable mathematics and coding. One path uses another model—or the same model prompted as a critic—to judge whether retrieved material is relevant or rank a large candidate set down to the 50 strongest results.
Dean does not claim the non-verifiable problem is solved. His optimism comes from the pace of demonstrated change: about two years earlier, models struggled with GSM8K-style rabbit arithmetic; roughly a year and a half later, they could handle dramatically more sophisticated mathematics in natural language.

9. General models win broadly, while modules handle scarce capacity

Dean was unsurprised that IMO work moved from AlphaProof, AlphaGeometry, and Lean-oriented machinery toward one unified model resembling the production model with additional inference budget. Humans manipulate symbols without obviously storing a discrete symbolic engine, so separate symbolic and neural systems “never made sense” to him.
Dean expects “general models” to win out over specialized ones “in most cases,” echoing the earlier transition from separate street-sign, speech, and vision models. The remaining question is how reliably a unified model generalizes to tasks it has never encountered.
The hosts’ capacity pushback still holds for smaller models: finite parameters cannot retain everything. Dean would rather spend “precious parameter space” on reusable reasoning than obscure facts available through retrieval, while preserving enough world knowledge—such as a representative bridge length—to anchor judgment.
Vertical models remain valuable as enriched continuations of a strong base. A robotics-heavy model might improve robotics while losing multilingual translation; modular healthcare, robotics, or 200-language capabilities could eventually be “knitted together,” with retrieval supplying some knowledge and training on 100 billion or a trillion domain tokens supplying the rest.

10. Different data constraints demand different adaptation mechanisms

Alessio’s Kalamang example illustrates another route: Kalamang is spoken by about 120 people and has no written text, making it small enough to place essentially the available data set in context. Languages such as Somali have more text and can improve when more is included during training.
Healthcare is unusually constrained because much useful data is private and unavailable to a general model. Dean sees scope for partnerships in which a large healthcare organization trains a bespoke model on its own data, potentially outperforming a model limited to public material for that organization’s use.
Dean recalled an early fused image-language model that generalized beyond its labeled image categories. Trained with telescope and binocular images but no microscope category, it could still label a novel microscope image correctly after its language and image representations were joined.

11. Scaling gains multiply across hardware, architecture, and organization

Dean’s scaling conviction dates to a 1990 undergraduate thesis on parallel neural-network training. He believed neural nets were the right abstraction but lacked compute; by 2008 or 2009, larger data sets and Moore’s law made real speech, vision, and language problems tractable.
Beginning at Google in late 2011, he revived model- and data-parallel ideas to train a 2-billion-parameter vision model—about 50x larger than prior neural nets—on 16,000 CPU cores for several weeks. It achieved a 70% relative error improvement on ImageNet 22k, reinforcing the mantra: “bigger model, more data, better results.”
Sparse models added another multiplier: trillions of parameters might be available while only 1%, 5%, or 10% activate, yielding roughly 10x better time- or compute-to-quality than dense models. Transformers supplied approximately 10x to 100x improvement over LSTMs; hardware, data, architecture, infrastructure, and RL gains then compound.
Dean partly accepts the critique that Google fragmented its scaling effort. His one-page memo called it “stupid” to split compute, people, and ideas across language, multimodal, Brain, Chinchilla, and Flamingo efforts; it proposed one multimodal-from-the-start program, becoming Gemini—“twins coming together,” with a secondary NASA pre-Apollo reference.

12. Coding agents make specifications and latency into capabilities

Dean says coding tools can now accept meaningfully complex delegated tasks, but interaction style shapes the result: developers can demand tests, brainstorm performance, iterate frequently, or send an agent away independently. No single supervision cadence fits every problem.
The hosts worried that 50 parallel agents could accumulate context that isolates human collaborators. Dean’s counterpoint: five people each managing 50 virtual agents might communicate with one another at higher bandwidth than five managers coordinating 50-person human teams, though he stressed that the eventual interface is unresolved.
Specifications, once taught but often ignored, become executable inputs. Developers must be “pretty darn careful” about corner cases, performance, and ambiguity; reusable guides covering techniques such as Paxos-like replication or duplicate requests could give agents the equivalent of seasoned distributed-systems instincts.
The hosts forecast systems that are 20x or 50x lower latency. At 10,000 tokens per second, a model could run parallel rollouts, generate code, and verify it—perhaps returning 1,000 tokens of code backed by 9,000 tokens of reasoning—alongside a Deep Think tier on the other side of the cost-and-latency tradeoff.

Technology Cycles, Traffic, and Artificial Intelligence — 张小珺 x 郑庆生 AI, R2 and the Future of Everyday Driving | Rivian CEO RJ Scaringe