(Preview) Inference in the Agentic Future, xAI Is Two Companies in One, Q&A on Elon’s Lawsuit, Intel, Apple

Summary

• Ben Thompson’s core call is that fast inference commands its strongest premium while a human remains in the loop. Enterprises historically bought faster computers because employee waiting time was expensive, and today’s coding agents still return work for oversight; Andrew recalled waiting 55 seconds, while Ben said “35 seconds if you’re lucky,” with some answers taking minutes. As autonomous runs lengthen and agents become “24-hour employees,” however, “they can sit around for compute”—latency need not destroy human productivity.
• The agentic bottleneck shifts from raw execution speed to memory capacity, especially the KV cache carrying context and state. Andrew’s example of a useful agent was one that could wake a week later knowing what happened and what to do, while the potential population of agents is “effectively infinite” compared with bounded human demand. That makes a tiered hierarchy—from HBM through ordinary RAM and SSDs—more valuable, because slower storage is acceptable when “there’s no human in the loop.”
• Cerebras and Groq show why ultra-fast inference remains valuable but specialized rather than universal. Cerebras stitches reticle-sized exposures across a 300-millimeter wafer into “one wafer, one chip,” delivering “orders of magnitude” speed on bandwidth-bound portions when the model and context fit; once memory spills off-chip or the KV cache swells, performance falls sharply. Voice and consumer interaction remain clear speed-sensitive markets, and Ben tentatively thought Cerebras might IPO that week into an unusually favorable window.
• Ben’s most direct beneficiary call was: “the biggest winner is China.” If agentic inference can use regular RAM, deep storage, and slower chips while waiting on memory, China can self-supply more of the stack and export lower-end components. The opening could widen while SK Hynix, Samsung, and Micron concentrate on HBM, raising the question of who will make ordinary DRAM.
• The shift challenges NVIDIA’s integrated inference economics without overturning its training position. Ben described Dynamo, mixed fast and standard compute, and SSD-heavy memory racks as evidence NVIDIA is adapting to KV-cache pressure, but expects inference components eventually to disaggregate into cheaper commodity markets. His hedge is explicit: this is “not necessarily totally bearish for NVIDIA,” whose approach remains “by far the best for training,” and training will not stop.
• The historical template is cloud computing’s move from premium Sun systems to commodity Intel fleets engineered for failure. Hotmail and Yahoo started the pattern, Google scaled it, and Amazon exposed it through AWS; software resilience made cheaper, flakier hardware economically superior. Ben expects the same integrated-to-modular arc in inference because “no one cares about how their inference stack is constructed as long as it does the job”—whether it works and, as Andrew noted, what it costs matter more than user-visible integration.

Deep dive

1. Human waiting time sustains the speed premium

• Andrew Sharp set up AI’s three shifts: ChatGPT proved token prediction useful; o1 made more test-time tokens yield better answers; Opus 4.5 plus Claude Code made agents usable through reasoning, tools, and verification. His question was whether fast coding inference is only a transitional prize.
• Ben’s qualification matters: “speed is always important,” but the economics depend on trade-offs. In the 1980s and 1990s, enterprises repeatedly upgraded expensive computers because humans cost more; any time an employee waited on a machine was lost productivity.
• Today’s agents have not escaped that logic. They finish a run, seek oversight, and force the developer to recover mental context; usage caps can produce the absurd workflow, “Well, I guess I’ve got to go home for the day.” Ben sees autonomous-task duration before a model “lose[s] the thread” as a particularly revealing benchmark.

2. Fast inference is a premium niche, not a universal architecture

• Reasoning makes latency more painful because more tokens can produce a smarter answer. Andrew remembered waiting 55 seconds; Ben corrected him with “35 seconds if you’re lucky,” with some jobs taking minutes—precisely the opening for Cerebras and Groq.
• Ben’s architecture tour centered on Cerebras: where ordinary chips are constrained by the lithography reticle limit, it wires across exposure boundaries so a 300-millimeter wafer functions as one chip. The payoff is huge on-chip compute and SRAM—“unbelievably fast,” even “orders of magnitude faster” on bandwidth-bound steps.
• The constraint is equally important: on-chip memory is finite, off-chip access makes performance “totally plummet,” and long conversations expand the KV cache. Ben called voice “the biggest” enduring speed use case; Andrew said consumer responsiveness still matters. Ben only tentatively expected Cerebras to IPO that week: “Maybe it’s today.”

3. Agents turn inference into a memory-hierarchy problem

• Andrew’s “24-hour employees” framing contains the inversion: agents never sleep, but therefore can wait. Ben’s formulation was blunter: “They can sit around for compute. It’s fine.” Once work proceeds independently of people, slower execution no longer wastes a costly human asset.
• State, not just model weights, becomes the load-bearing resource. Andrew described a useful agent that could wake a week later, recover the right context, execute, and sleep again; Ben contrasted that potentially unbounded computer demand with the human limit of how much people can come up with.
• Much of AI infrastructure has been optimized around training: fast GPUs, large pools of HBM, and networking that keeps tens of thousands—eventually hundreds of thousands—of processors fed. That flexible NVIDIA architecture also serves inference, but ever-growing KV caches are pressuring even GPU clusters.
• Ben pointed to NVIDIA’s Dynamo and a rack “that’s just memory…just SSDs” as the direction of travel. Registers and cache are fastest but tiny; RAM, SSDs, disks, and tape become progressively slower and larger. Inference moves from “one size fits all” GPU-plus-HBM toward deliberate tiering, acceptable because “computers are doing computing.”

4. Commodity memory makes China the clearest beneficiary

• Asked who wins, Ben answered categorically: “the biggest winner is China.” If ordinary RAM, bulk storage, and relatively slower processors can handle memory-bound agents, China can manufacture more of the relevant stack, self-supply more workloads than expected, and sell components abroad.
• The supply-side opening matters beyond domestic AI. With SK Hynix, Samsung, and Micron focused on HBM, Ben asked, “Who’s gonna make DRAM?” Lower-end Chinese supply could ease the broader memory shortage, while hyperscalers benefit from any cheaper mix of compute and storage.

5. Inference disaggregates, while NVIDIA retains the training moat

• Ben’s precedent was the dot-com stack: startups once spent their funding on integrated, reliable Sun systems; Hotmail and Yahoo experimented with commodity machines, Google built vast Intel-based fleets, and software supplied fault tolerance. Amazon then made that model broadly accessible through AWS.
• His forecast follows the same arc: dedicated high-end systems initially “do it all,” then “every piece of that system is going to get disaggregated” into commodity markets. Andrew summarized it as integrated-to-modular; unlike Apple’s user-facing integration, an inference stack has no user who values its construction—Ben’s point was that no one cares how it is built as long as it works, while Andrew added that cost will matter.
• The disagreement is more calibration than contradiction. Andrew asked whether cheaper, durable alternatives are bad news for NVIDIA; Ben conceded a long-run challenge but rejected a blanket bear case. NVIDIA is moving quickly on inference orchestration and memory, remains “by far the best for training,” and “it’s not like we’re gonna suddenly stop training.”