The Bottleneck Isn’t the Chip with Bubble boi | EP 166
Summary
Bubble boi’s investable thesis is that AI’s binding constraints are migrating from transistor shrinkage into packaging, interconnect, and memory. Moore’s Law once delivered roughly twice the transistor density every two years; now tens or hundreds of billions of dollars may buy only 15–20%. The alternative is expanding area and making multiple dies behave coherently: “We need more transistors by any means necessary.”
NVIDIA’s roadmap depends increasingly on suppliers and technologies it does not control. H100 used one large die, B200 moved to two, Reuben used two by default, and the Ultra design was intended to reach four—yet repeated packaging difficulties reportedly led to de-specification toward two dies per package across two packages. “Every time I look at NVIDIA’s roadmap, I just see: this is impossible today.”
AI hardware must now be designed as a rack-scale system, not an isolated accelerator. NVIDIA’s NVL72 connects 72 packages and uses roughly four NVLink switches, while the TPU team reportedly stretches its scale-up domain to 9,600 chips by embedding routing logic into the chips themselves. “These systems are not just chips anymore”; architecture now spans compute, networking, memory, prefill, and decode.
Inference memory demand may be driven more by context and concurrent users than by ever-larger model weights. Bubble boi cited rumored GPT-6 “looped transformers,” sparsity, quantization, pruning, and speculative decoding as evidence that raw weight growth may plateau, while context scales with the number of weights, heads, and users. His allocation view is that weights should probably be in HBM, while context not needed immediately should increasingly sit in flash.
High-bandwidth flash is technically intriguing but lacks a decisive data-center use case while HBM remains available. Its density is better, but Bubble boi put both HBF and high-end AI flash latency around five microseconds and argued HBF’s total product cost may be about the same as HBM’s. His “alpha leak” was that the main HBF customers he had heard about were Chinese customers unable to secure or produce high-end HBM.
The commercially important optimization is not maximum tokens per second in isolation, but more users per unit of hardware. Smarter caching, flash offload, prefill/decode disaggregation, CXL-attached memory, and context-aware controllers could, hypothetically, double or triple the users served by the same NVL72-class rack. “Agents are just context at the end of the day,” making agent harnesses and reusable prompts natural caching targets.
Bubble boi prefers the hardware “railroads” to trillion-dollar AI labs whose model alpha may diffuse through competition and employee movement. His durability test is: “If this company disappeared for six months, how screwed would everyone be?” TSMC and NVIDIA score highly; meanwhile he remains bullish because finding five or 10 drugs with AI, or another major efficiency breakthrough, could justify all the current AI capex.
Deep dive
1. Moore’s Law broke economically before engineers ran out of ideas
Bubble boi explains that, as a chip designer, he must simplify and abstract; he also says traders or “gamblers” need a high-level overview so they do not “get lost in the sauce.” Logan supplies the economic test: what must the system do at the end of the day, rather than getting stuck on marginal debates over bit-error rates, materials, or a few watts.
Moore’s Law roughly promised twice the transistor density every one-and-a-half to two years. More transistors meant more digital logic and performance, but its larger contribution was business predictability: a company spending $100 million could forecast better chips, premium pricing, market-share gains, and a return on that capital.
Bubble boi’s proposed “Bubble’s Law” tracks the exploding cost of density. Where improvement was relatively inexpensive from the 1960s through the 1990s, a modern node may consume tens or hundreds of billions of dollars for only 15–20% better density, leaving perhaps one or two companies able to remain competitive.
Clock speed hit a separate thermal and switching ceiling, pushing the industry toward multicore processors in the mid-2000s. Parallel workloads benefited, but sequential work did not: the Intel Core i9-14900KS reached roughly 6GHz, and today’s processors have not materially surpassed it for latency-bound, single-threaded tasks such as packet processing.
2. NVIDIA’s original moat was flexibility, not AI foresight
Graphics are largely transformations of shapes through highly parallel matrix math, making a dedicated processor unavoidable once CPUs could not render complex games at 60fps. AI uses related math at lower precision, allowing the graphics architecture to migrate naturally into scientific computing and then model training.
Bubble boi attributes NVIDIA’s survival to programmability. When a graphics API changed and Microsoft backed the new standard, NVIDIA reportedly came close to bankruptcy and had to tape out a replacement chip immediately; the company also added capabilities beyond the formal graphics specification rather than remaining rigid.
That flexibility let NVIDIA support scientific workloads that rival graphics vendors missed and still allows new models to run within “a day or two.” Bubble boi nevertheless keeps the conclusion conditional: programmability was clearly right historically, but whether the same advantage remains decisive as AI systems specialize is unresolved.
3. Packaging has become the new scaling law
Once shrinking transistors slowed, the practical alternative was expanding area: connect several dies closely enough that they behave like one coherent chip. Bubble boi said lithography may now deliver only roughly 10–12% improvement while requiring $20–40 billion, making advanced packaging a new scaling paradigm.
He contrasted the escalating tape-out curve—5-nanometer perhaps twice the cost of 7-nanometer, with 2-nanometer roughly doubling again—with the relative attractiveness of advanced packaging.
His Intel thesis rested on the company funding advanced nodes and packaging through the nonlinear investment cycle. Intel’s EMIB technology was, in his assessment, “miles better” than TSMC’s contemporary offering, while TSMC has since moved its packaging in a similar direction. The bet required confidence that Intel could actually ship.
NVIDIA exposes the dependency. Its roadmap went from one large die in H100 to two in B200; Reuben used two dies by default, while the transcript describes the Ultra design as intended to combine four. Bubble boi said repeated de-specification now points toward two dies per package and two packages linked through a PCB instead.
Longer PCB distances introduce throughput and signal-integrity problems as signals lose power or integrity and must move across different physical media. His hindsight trade was that Intel worked, but PCB manufacturers were the purer bottleneck play. Feynman, expected to use four dies by default and still more HBM, should intensify the same constraint.
4. Scale-up architecture now determines the chip itself
AI’s matrix workloads are “embarrassingly parallel,” so clusters can divide work across hundreds or thousands of accelerators. The physical limit is obvious: an NVL72 contains roughly 72 packages that cannot plausibly fit into one package, forcing system architects to preserve chip-like behavior across an entire rack.
Training and inference impose different interconnect requirements. Training moves activations and partial products and benefits from NVLink’s roughly 900-gigabit-per-second bandwidth; inference often moves less data but is highly sensitive to latency. Reusing a training-oriented fabric works, but requires engineering around that latency profile.
The TPU team impressed Bubble boi with a 9,600-chip scale-up domain despite less HBM per chip and no comparable multi-reticle package. TPU chips perform some routing themselves, whereas NVIDIA’s NVL72 reportedly dedicates substantial rack space to roughly four NVLink switches—space that otherwise could house accelerators.
Better interconnect can permit less on-chip memory because architects can add chips instead. It also enables disaggregation: Kimi’s reference design separates inference prefill from decode because they are fundamentally different operations. “You cannot just design a chip anymore”; the rack, workloads, controllers, and silicon must be co-designed.
5. Training remains compute-heavy, while inference exposes new levers
Bubble boi still considers NVIDIA’s chips excellent training products because they provide abundant FLOPs and bandwidth. But if the objective is simply more multiply-accumulate operations, the answer increasingly becomes a huge compute area—potentially a Cerebras-style design—rather than a clever new hardware optimization.
Training carries weights, activations, states, checkpoints, and the source dataset. Some material can move into cheaper memory or be recomputed later, but Bubble boi sees the remaining improvements as increasingly algorithmic: “We’ll hope that the people who do the algorithms and the machine learning can figure something out.”
Logan’s local-inference check challenged the idea that frontier models will soon run comfortably on consumer devices. His OpenRouter sample suggested even the weakest top-ten model needed roughly two to four H100-equivalent devices, while “GM 5.2” required roughly 40–50; an H100 was priced around $25,000 or $30,000.
Bubble boi added that even Llama 7B may require four GPUs “in practice” to run. Commodity local inference remains possible for older or heavily specialized models, but the larger models discussed still demand clustered accelerators primarily because their weights and serving state exceed local memory.
6. Context, not weights, may dominate inference memory
Bubble boi questioned indefinite parameter growth. He cited a common rumor that GPT-6 repeatedly loops over the same weights, while pruning, quantization, and speculative decoding work because weights are sparse and do not contribute equally. More capability therefore need not require proportionally more stored parameters.
Context scales with the number of weights, attention heads, and users. Because autoregressive generation repeatedly draws on the prior conversation to produce the next token, serving state can occupy more memory than the weights themselves. Logan’s “Her” ideal—an assistant that remembers a whole life—magnifies that requirement.
Today’s advertised one-million- or two-million-token windows are not fully effective in Bubble boi’s view: models must train at those lengths, and suitable long-form training data is scarce. Effective memory may instead combine retrieval, stored conversations, and post-training that burns durable facts—such as where a user lives—into weights.
The prefill/decode distinction creates the economic payoff. Prefill computes context; decode can reuse a cached representation. Bubble boi said “Kimmy” charges roughly $5 or $3 without a cache hit and much less with one, while agent harnesses are especially cacheable because they are largely repeated prompt injections.
7. Flash should absorb context, but HBF still needs a customer
Bubble boi jokes that changing the world requires either curing cancer or inventing “a new kind of memory,” which explains his attraction to high-bandwidth flash. HBF offers superior bit density and potentially useful bandwidth, but its case depends on whether accelerators truly need far more local capacity.
Reliability is not his primary objection. Flash writes cause wear, yet an application-specific controller could manage context differently from databases, games, or files; an old vacation conversation may not require permanent, bit-perfect preservation. The harder issue is that ordinary AI-oriented flash can already retrieve such context.
Bubble boi put both HBF and high-end conventional flash at roughly five microseconds of latency. HBF has less bandwidth than HBM and, despite lower cost per bit, may cost about the same in total. Its clearest customer is therefore someone who cannot obtain HBM, matching reports he heard from “two or three people” about Chinese demand.
NVIDIA’s network-attached flash is directionally better, though Bubble boi believes smarter orchestration can go further. CXL similarly lets additional memory appear conventional to software. The governing principle is simple: keep immediately used weights in HBM, offload colder context, and coordinate movement through controllers designed for AI.
8. Locality and scheduling may matter more than another memory headline
Bubble boi highlighted the Hot Chips presentation for “Jalapeño,” described in the conversation as an OpenAI chip, because it exploited memory locality. Different banks and channels are physically separated by millimeters; placing needed data close to the relevant compute engine improved performance despite the distances sounding trivial.
Three-dimensional DRAM bonded over logic could improve bandwidth, latency, and power through many short connections rather than one wide bus. Bubble boi did not recall the company’s name. The design challenge becomes spatial: engineers must know which bytes live near which compute block and organize work around those tiny local paths.
Timing matters alongside nominal latency. If an operation lasts ten milliseconds and data is not required until millisecond 11, it need not arrive instantly; it only needs to be prefetched correctly. Knowing what an inference workload will do allows memory layout and scheduling to substitute for more expensive raw bandwidth.
Bubble boi is far less enthusiastic about photonic compute. Analog optical matrix multiplication for present model dimensions could require a footprint approaching a football field—effectively free-space optics—so he sees nearer-term potential in embedded or IoT devices rather than data centers. “People are selling bridges there.”
9. The durable investment is infrastructure, unless labs create irreplaceability
Power efficiency matters, but the economic metric is users or tokens per watt, and batching more requests may beat exotic low-power circuitry. Bubble boi also corrected the emphasis: a data center’s largest cost is its hardware, not electricity. A neocloud first needs financing for land and GPUs; cheaper power does not erase that bill.
On “pausing the frontier,” Bubble boi questioned whether labs mean safety or vertical integration into valuable research. He contrasted valuable math and signal-processing work, such as Terrence Tao’s research contributing to cellular technology, with less consequential problems. Logan said solving Navier–Stokes might only save compute cycles in a CFD simulation.
Logan’s practical case is that powerful AI and high-compute resources should target drugs, new materials, and other important problems rather than merely websites or slop. Bubble boi agreed that current models automate grunt work but still do not replace thinking; users must check their context and sources.
Logan’s pushback—worth keeping—is that labs are not pausing their own multiyear gigawatt commitments. Bubble boi’s formulation: “They’re pacing the frontier for us. They’re not pacing it for themselves.” Both remain skeptical that current models replace thought, even as they see value in applying compute to difficult discoveries.
On open versus closed source, Bubble boi said open source has commoditized coding and that the frontier should move beyond computer tasks toward discoveries that change reality. He also said current models generally cannot run on commodity hardware without substantial know-how, except for older or specialized workloads.
Bubble boi would own “the hardware layer,” railroads, or proven operators rather than labs that may “compete to death.” His disappearance test favors TSMC, NVIDIA, and indispensable infrastructure. Potential expressions discussed included not-yet-public Etched and WKA’s flash-offload software; Groq and Cerebras arose in the surrounding discussion rather than as his public picks.
For agents, Bubble boi suggested that monetization may accrue through the companies adopting them or the workers they replace—possibly even an insurance business—rather than solely through an agent vendor.
His macro conclusion is bullish: finding five or 10 drugs with AI, or another major efficiency breakthrough, could justify all the current AI capex and trigger a “fast takeoff.”