Summary

Inference-time compute is the near-term explosion, and the economics are absurdly favorable. Noam Shazeer’s math: even a trillion-operations-per-token model costs ~$10^-18 per operation — “you’re getting a million tokens to the dollar,” making an LLM “100 times cheaper than reading a paperback” and a million-plus times cheaper than a software engineer or lawyer. “Taking advantage of ’think harder’ at inference time is just going to be an explosion” — a direct read-through to inference-optimized silicon and data-center demand.
The self-improvement feedback loop is treated as a live possibility, not a certainty. Jeff Dean wants to compress chip design from 18 months and 150 people to a few people running automated search, leaving TSMC’s 3-5 month fab time as the bottleneck — conveniently the length of a training run. Asked directly whether capability explodes at an accelerating rate, the answer was “Quite possibly,” and Dwarkesh’s tell: “I’ve stopped cleaning my garage because I’m waiting for the robots.”
Algorithmic progress rivals or exceeds hardware in driving each Gemini generation — architecture, data mix, and training changes make the model “better per flop,” and top labs now have “probably a million times as much compute as it took to train Transformer” to run automated idea search. The aspirational capability bar: from 80% reliability on 10-step task decomposition to 90% on 100-1,000 steps — “a major, major step up.”
Google is already eating its own dogfood at scale: a Gemini model further-trained on the internal monorepo means ~25% of characters checked into Google’s codebase are AI-generated (per Sundar), and the roadmap is context windows going from millions to trillions of tokens — “all the world’s source code in context for any open-source developer” — pending algorithmic approximations to quadratic attention.
Compute demand stacks multiplicatively toward 2030: inference scaling makes each request 50-1,000x heavier, only 10-20% of computer users have discovered chatbots, and models keep growing — two orders of magnitude from each, per Jeff. Noam goes further — world GDP up 100x, “the sun is what, 10 to the 26 watts” — and Jeff notably hedges: “I’m not sure I agree completely… even if you get partway there, it’s definitely going to be a lot of compute.” No comment on future capex, but “look at our past capital expenditures.”
The next architectural paradigm sketched here is the organic, modular “blob”: mixture-of-experts taken organic — experts varying 100-1,000x in compute cost, modules independently trained (per-language, per-data-domain, personal), surgically swapped, continually distilled. Pathways infrastructure already supports it but Gemini isn’t using those capabilities yet — “maybe we should.” If this wins, only companies that can serve a data-center-scale model with everything in HBM can play — a structural moat for TPU-pod owners.
Why Google was late despite inventing everything: the search-culture bar — “you want to get the right answer 100% of the time” — plus safety concerns made them undervalue chatbot usefulness for non-search tasks; Meena existed internally before ChatGPT. On publishing Transformer, Noam has no regrets: “it’s not a fixed pie” — though “these days we do publish a little less of what we’re doing.” Gemini 2 now tops LMSYS Chatbot Arena.
Jeff doubts data is running out. Jeff: “I don’t really believe that” — dropout plus 100 epochs over the world’s text, harder training objectives, video data barely touched, and action-taking models. The efficiency benchmark that haunts them: “a person has seen a billion tokens. And they’re pretty good at a lot of stuff.”

Deep dive

1. From 25 or 26 employees to “What the heck is Project Platypus?”

Noam joined in 2000 and was assigned a mentor who “knew everything” — it was Jeff, and he knew everything “because he had basically written everything.” Jeff’s phenomenology of company scale: first you know every name, then every engineer, then every project, and finally “you get an email that Project Platypus is launching on Friday” — the fix is a network where “with one level of indirection, you can usually find the right person.”
Noam’s origin story: skipped Google at a 1999 job fair assuming it was already huge, then applied on a whim in 2000, sold by a crayon chart of daily search queries on the wall — “It looked very exponential.” His plan: make startup money, then “just go work on AI for as long as I want after that.” His pattern since: “I seem to join Google every 12 years: 2000, 2012, and 2024.”
Jeff’s framing of why Google was always secretly an AI company: “organizing the world’s information and making it universally accessible and useful” is “a really broad mandate” that would require pretty advanced AI.

2. The algorithms follow the hardware — and quantization is a co-design problem

Jeff on the regime change: for a decade you could just wait 18 months for faster general-purpose chips; now process nodes take three years and multi-core gains have flattened, but specialized ML accelerators deliver efficiency for computation that isn’t “a twisty pile of C++ code trying to run Microsoft Office.” Noam’s compression of the whole era: “arithmetic is very, very cheap, and moving data around is comparatively much more expensive… pretty much all of deep learning has taken off roughly because of that.”
Dwarkesh invokes (possibly misquotes, he admits) Larry Page: “Our second biggest cost is taxes, and our biggest cost is opportunity costs” — the missed opportunity being chip area: “Fill the thing up with arithmetic units!” And asked the counterfactual — memory cheap, arithmetic dear — AI would look like “a lot more lookups into very large memories.”
On precision: TPUv1 bet the whole chip on uncertain INT8 serving; now INT4/FP4 and even one-bit quantization (“just a 0-1, and then you have a sign bit for a group of bits”) are in play — heresy to a supercomputing floats person of 20 years ago. The organizational lesson is co-design: ask the algorithm designer alone and they’ll refuse quantization as “irritating”; see the whole picture and “your model is going to be three times faster, so you’re going to have to deal.”

3. 1990 backprop thesis, 2007’s two-trillion-token N-gram model, 12 hours to 100 milliseconds

Jeff’s undergrad thesis implemented model and data parallelism for backpropagation on a 32-processor Hypercube — “four pages and then 30 pages of C code.” The naive hope that 32 processors could train real neural nets was off by six orders of magnitude: “we needed about a million times more compute,” which arrived around 2008-2010.
The 2007 story: Franz Och’s translation team won a DARPA contest by a large Bleu margin, but “we can’t launch this… it takes 12 hours to translate a sentence” — 100,000 disk seeks per word. Jeff spent 2-3 months building an in-memory compressed representation of five-gram statistics over 2 trillion words across 200 machines with a batched API, taking translation “from taking a night… to 100 milliseconds.” Dwarkesh notes the Chuck Norris-style Jeff Dean facts (“the speed of light was 35 miles an hour until Jeff Dean decided to optimize it over a weekend”) suddenly read as documentary.
Noam’s 2001 spelling corrector — one machine, in-memory — nailed every butchered query Jeff threw at it, including “scrumbled uggs Bundict.” “Yeah, I guess that was language modeling.”

4. “The best problem in the world” — and the cat neuron that justified scaling

Neither saw N-grams sweeping the world — “a lot of people were excited about Bayesian networks” at the time. But Noam’s conversion on neural language models is the episode’s cleanest thesis statement: “the best problem in the world” because it’s simple to state (a probability distribution over the next word), has roughly infinite self-supervised training data, “and then it’s AI complete. If you can do a great job of that, then you can pretty much do anything.”
On whether the Transformer was inevitable: Jeff’s view is ideas are “partially in the air” — neural Turing machines and attention were circulating — but “you need some group to go do it,” combining morphed existing ideas with the genuinely unsolved piece.
Jeff’s “holy shit” moment: early Brain, no GPUs, so 2,000 CPU machines and 16,000 cores doing unsupervised learning on 10 million YouTube frames — and one top-level neuron fired for cats without ever being told what a cat was. The model was ~50x bigger than anything trained before and advanced ImageNet’s 20,000-category state of the art by 60% relative — “scaling up neural nets… seems to be [a good idea], so we should keep pushing on that.”

5. Not an information-retrieval company — “what’s cool is a quadrillion dollars”

Jeff’s reframe of Google’s identity in the AGI era: “we’re an ‘organize the world’s information’ company, and that’s broader than information retrieval” — now including creating information (drafting the letter to your veterinarian), synthesizing across 100 web pages where no single source exists, and non-human modalities: “weird lidar sensors on autonomous vehicles, or genomic information, or health information.”
Noam, characteristically blunter on the size of the prize: “organizing information is clearly a trillion-dollar opportunity, but a trillion dollars is not cool anymore. What’s cool is a quadrillion dollars” — the value unlock being systems that “actually go and do something for you, write your code.”
Jeff’s unfinished-vision example: any video watchable in any of thousands of languages — “we’re not quite there yet, but that’s definitely things I see on the horizon.”

6. Attend to trillions of tokens — the monorepo in context, 25% of code AI-written

Noam’s diagnosis of hallucination: tens of trillions of training tokens get “stirred together” into parameters and go “squishy,” while in-context information is “really sharp and clear” thanks to attention. Current models handle millions of tokens — hundreds of PDF pages, 50 papers, hours of video — but the dream is trillions: “Could it attend to the entire internet?… all my emails, all my documents, and all my photos… all of the Google code base in context for every Google developer.”
The blocker is quadratic naive attention — “no hope of making that just naively go to trillions” — so it needs “a whole bunch of interesting algorithmic approximations.” Dwarkesh adds the memory asymmetry: parameters memorize roughly one fact each, while a context token costs kilobytes-to-a-megabyte of keys and values per layer — “you take a word and you blow it up to 10 kilobytes.”
Meanwhile the shipped version already matters: Gemini further-trained on the internal codebase means, per Sundar, ~25% of characters checked in are generated by AI coding models with human oversight.

7. A million researchers, a breakthrough a day

Jeff’s model of ML research: “if you have a brilliant idea… it has a 2% chance of working if you’re brilliant. Mostly these things fail, but if you try 100 things or 1,000 things or a million things, then you might hit on something amazing. We have plenty of compute” — modern top labs have “probably a million times as much compute as it took to train Transformer.” Dwarkesh runs the extrapolation — a 100,000-strong research community (15,000 at NeurIPS alone) scaled 1,000x — and Noam completes it: “A breakthrough a day? … Sounds potentially good.”
The discipline required: everyone wants to run at maximum scale, “but I think that’s a human problem” — the right move is “a 1/1000th scale problem and then vet 100,000 ideas on that.” Jeff’s vision is researchers “gently steering a more automated search process, rather than hand-babysitting lots of experiments” — 100x faster idea-vetting into production recipes.
The one thing that doesn’t parallelize: N=1 experiments at the largest scale, where you “put a bunch of brilliant people in the room” and stare. Jeff’s fix: “more hardware is a good solution. And better hardware… we’re counting on you.” Note Jeff’s discipline when Dwarkesh calls autonomous coding under-hyped: “I just don’t like to hype things that aren’t done yet.”

8. The feedback loop: chips in months, 1,000-step reliability, “quite possibly” an explosion

Jeff’s chip-cycle compression thesis: today ~18 months from “we should build a chip” to TSMC handoff, with fab (3-5 months on leading nodes, worsening with more metal layers) a small fraction. Automate the design search down to “a few people” and fab becomes dominant — and since fab time matches a training run, “you could potentially do both at the same time,” shrinking the ML-hardware forecasting horizon from 2.5 years to 6-9 months.
Asked point-blank whether this yields a rapid capability explosion past the tail of human intelligence: “Quite possibly.” Jeff’s preferred concrete framing: today’s models break problems into 5-10 steps at ~80% reliability; getting to “100 to 1,000 steps… 90% of the time” would be “an amazing improvement.” “We don’t need new hardware for that, but we’ll take it” — Noam: “Never look new hardware in the mouth.”
Dwarkesh’s positioning on the two-worlds question (slow two-decade grind vs. the most important two years in human history): “I’ve stopped cleaning my garage because I’m waiting for the robots. So probably I’m more in the second camp.”

9. Inference-time compute: 100x cheaper than a paperback, headroom everywhere

The load-bearing arithmetic, from Noam: at a trillion operations per token and ~$10^-18 per operation, “you’re getting a million tokens to the dollar” — versus 10,000 tokens per dollar reading a paperback book, 10,000x cheaper than a customer-support agent, a million-plus times cheaper than a doctor or lawyer. “Can we add computation and make it smarter? I think a lot of the takeoff that we’re going to see in the very near future is of this form… just going to be an explosion.”
Dwarkesh wants a dial: crank inference compute for hard problems, but “you probably don’t want to spend enormous amounts of compute to compute ‘what’s the answer to one plus one’” — the system “should decide to use a calculator tool.” Is linear inference scaling solved? “We’re working out the algorithms as we speak” — 10x more compute buys better answers today, but the goal is a bigger quality jump per 10x than currently achieved.
Noam’s compass is Rich Sutton’s Bitter Lesson: “the two techniques that are incredibly effective are learning and search” — search “has got to be part of the solution.” On efficiency mechanics: drafter models let a small model propose four tokens and the big model verify — “the big model’s being used as a verifier” — breaking the single-token decode bottleneck. And the deep research tool (launched ~a week prior) previews the async UI problem: 20 background tasks each interrupting for input (“I found your flights to Berlin…”).

10. Multi-datacenter training works — and the async-vs-sync argument between the co-leads

Google is “already doing it”: the Gemini 1.5 tech report disclosed training across multiple metro areas over long-latency, high-bandwidth links. Noam’s reasoning: with multi-second step times, 50ms latency is irrelevant — “as long as you can sync all of the parameters… in the time it takes to do one step, you’re pretty good.”
A genuine disagreement worth keeping: Jeff on going synchronous — “so pleasant… your experiments are now replicable, rather than your results depend on whether there was a web crawler running on the same machine.” Noam: “I love asynchrony. It just lets you scale so much more,” proposing replayable operation logs for async-but-replicable — though as scale grows “there may be a push to have a bit more asynchrony… than we have now.”
The debugging reality: neural nets are “so tolerant of noise” that misconfiguration just silently degrades — “You could have bugs in your code. Most of the time that does nothing. Some of the time it makes your model worse. Some of the time it makes your model better — then you discover something new because you never tried this bug at scale before.” Integration fails “50% of the time,” and the trade-off is codebase/algorithmic complexity versus every researcher wanting their invention shipped.

11. A million evil Jeff Deans: safety as an engineering problem

Jeff stakes out the middle between doomers and dismissers, citing his co-authored “Shaping AI” paper: both extremes are “laissez-faire,” whereas the real job is to “shape and steer” deployment toward education and healthcare and away from “the computer will take over” — “that’s an engineering problem: how do you engineer safe systems?”, analogous to airplane software’s safety record. Dwarkesh’s pushback: the 737 doesn’t sit in a box with compute for two years and emerge as version 1000.
Jeff’s technical bet on control: “analyzing text seems to be easier than generating text… the ability of language models to actually analyze language model output and figure out what is problematic will actually be the solution to a lot of these control issues” — plus a business argument: deployment is often limited by safety, “so it becomes very, very important to be really, really good at that.”
On the self-improvement loop specifically, Noam’s safeguard is a human gate: the system explores algorithmic ideas but “we’re going to make a decision” on what enters the core codebase — self-improvement “with human oversight,” never “full-on self-improving.” When Dwarkesh raises a million evil Jeff Deans as worse than anything short of nuclear war, Noam deadpans “Where do we get the training data?” — and Noam closes: “that doesn’t empower people… So I’m against that.” Dwarkesh: “Me too.”

12. The 2030 Fermi estimate: astronomical — with Jeff hedging Noam

Jeff stacks the multipliers: inference scaling makes one request 50-1,000x more intensive; only “10% of the computer users in the world have discovered [chat interfaces] today, or 20%,” so adoption adds “another order of magnitude or two”; bigger models add another one or two. Conclusion: “you want extremely efficient hardware for inference.”
Noam goes fully unbounded: a personal-assistant earpiece (“maybe it’s like you’re Joe Biden… the earpiece in the cabinet”), where 2x compute spend buys “5, 10 IQ points” — $10/day or $20/day for a smarter assistant that makes you “from a 10x engineer to a 100x or 10 million-x engineer”; world GDP up two orders of magnitude, energy solved, billions of robots building data centers, “the sun is what, 10 to the 26 watts?” Jeff’s on-record hedge: “I’m not sure I agree completely… even if you get partway there, it’s definitely going to be a lot of compute.”
On whether Google’s build-out is aggressive enough: “I’m not going to comment on our future capital spending because our CEO and CFO would prefer I probably not. But… you can look at our past capital expenditures over the last few years and see that we’re definitely investing.”

13. The organic blob: modular experts, surgical upgrades, Pathways already built for it

Jeff’s sketch of the next paradigm: today’s MoE is too regular — same-size experts, paths merging back immediately. He wants “a more organic structure” where teams independently train modules (“Southeast Asian languages or reasoning about Haskell code”) and hook them into a base model — “100 teams around Google… all collectively work on improving the model. That’s kind of a form of continual learning.” Modules also solve data governance: a personal module on your private data, YouTube-only modules, internal-only modules — restricted by surface.
Jeff’s caveat is science: during rapid progress, controlled from-scratch comparisons are what “helps us figure out what to build” — Dwarkesh’s answer is versioned modularity (frozen base, swap module N for N-prime), which could make research “much, much cheaper and faster.” Continuous distillation cycles per module — big version distills into small, delete the big, add fresh capacity, repeat “a thousand different places… in the background.”
Killer detail: “a lot of what Pathways, the system, can support is this kind of twisty, weird model with asynchronous updates to different pieces. And we’re using Pathways to train our Gemini models, but we’re not making use of some of its capabilities yet. But maybe we should.”
The structural implication Dwarkesh draws out: future experts varying “by factors of 100 or 1000” in compute cost, hot experts replicated, the Tahitian-dance expert paged to DRAM — but you need the whole model in HBM at large batch sizes (Jeff kills the misconception that unused experts cost nothing). Serving a data-center-scale blob could favor companies with TPU-pod-class infrastructure — and you can “always distill from that” onto a phone. “Anyone out there who invents amazing distillation techniques… that would be wonderful.”

14. “A person has seen a billion tokens” — Jeff doubts data is running out

Jeff pushes back on data-wall talk: “I don’t really believe that because I think we can get a lot more capable models out of the text data that does exist” — via dropout (invented for images, “we’re not really using it for text mostly”) and a possible “100 epochs over the world’s text data,” though he says that would be computationally expensive, plus harder objectives: “when you get to ’the answer is,’ maybe the model should, at training time, do a lot more work than when it gets to ’the’.” Vision-style tricks — hide half the bird, infer from partial information — have textual analogs.
The sample-efficiency benchmark: “a person has seen a billion tokens. And they’re pretty good at a lot of stuff.” Jeff’s ingredients for closing the gap: change the next-token objective (people read a chapter and answer questions at the back), train on barely-touched visual data, and above all action-taking — infants learn gravity by dropping things, “a much harder thing to learn when you’re not initiating the action.” Asked if Gato is the future: “something where the model can observe and take actions and observe the corresponding results seems pretty useful” — a hedge, not an endorsement.
The discussion adds learning without external data at all: Einstein’s thought experiments, mathematicians, chess self-play needing only the rules — “maybe we can just have this thing talk to itself and make itself smarter.”

15. Publishing the Transformer, being late anyway, and the humility that keeps you in the game

No regrets on giving away the Transformer, per Noam: “we did need to see the size of the opportunity, often reflected in what other companies are doing. And also it’s not a fixed pie.” But the policy has changed — “these days we do publish a little less” — with a tiered calculus Noam spells out: super-critical things stay internal, product-relevant things ship first and publish after (the Pixel night-sight model), and community-building work goes to NeurIPS.
Why competitors led despite Google’s head start: the search-culture factuality bar — “you want to get the right answer 100% of the time” — plus safety worries meant Google underweighted how useful chatbots are “for things you wouldn’t ask a search engine.” Meena existed internally pre-ChatGPT (pandemic Googlers used it as a lunch partner). Jeff’s honest reckoning: “Would we like to have released the chatbot earlier? Maybe.” Now Gemini 2 tops LMSYS — micro-kitchen chant: “such a good model, such a good model.”
On career longevity across search, systems, hardware, and AI: Jeff’s answer is working with “five or six colleagues who have different expertise than you do” so expertise rubs off both ways; Noam’s, delivered with a wink, is “humility, like I’d say I’m the most humble” — meaning the willingness “to drop an idea as soon as you see something better.” His organizational insight: Brain’s bottom-up “UBI” chip allocation (“everyone had one credit and you could pool them”) incentivized killing failing projects; Gemini’s top-down mode drives collaboration but tempts everyone to claim “what I’m doing is working great” — the future needs both. Jeff’s own bootstrap mechanism: an internal slide deck called “Go, Jeff, Wacky Ideas.”

Verification Notes

The speaker for the post-Gato discussion is ambiguous in the raw captions.

The ARC Prize 2024 Winning Algorithm [Daniel Franzen and Jan Disselhoff]Ray Dalio on AI, Job Loss & the Future of the Economy | EP #148