Ep. 034 - The Fight for Fast Tokens, TPU v7, Vera Rubin, and Engrams (AI Supply Chain, InferenceX)

Summary

• Engrams turn model architecture and memory hierarchy into one co-design problem. They learn 2-gram and 3-gram token embeddings in a hash table, letting the model retrieve familiar meaning instead of rebuilding it through later Transformer layers; the ablations suggest a hybrid of Engram and MoE layers beats either all-MoE/no-Engram or all-Engram/no-MoE. Because token IDs are known upfront, retrieval from DRAM or SSD can overlap with compute—effectively yielding “free layers”—although earlier placement helps the model while leaving less time to hide retrieval latency.
• The design reflects a structural Chinese constraint: less HBM forces more aggressive compression, tiering, and asynchronous offload. The speakers connect Engrams to DeepSeek’s earlier work on reducing KV cache and using HBM, and to domestic accelerators whose bandwidth trails US systems. Their broader call is that Chinese labs will keep moving KV cache and related model state down the hierarchy, while US labs have historically been less forced to optimize around that constraint.
• AgentX shows why KV-cache management becomes decisive in real coding-agent workloads. Replayed sessions captured through SemiAnalysis’s internal proxy have a P50 request near 100,000 input tokens and theoretical cache-hit rates above 98%, because each tool call appends to an already-long conversation. DRAM offload matters little for low-concurrency interactive use, but becomes material around 40–60 tokens/s with hundreds of concurrent agents, offline batches, or reinforcement-learning rollouts.
• Current inference economics can make the newest systems look like “money printers,” provided the workload and pricing assumptions hold. The team’s calculator combines hardware, power, datacenter cost, throughput, concurrency, and OpenRouter pricing; scenarios discussed included approximately a 40.7% margin on GB200 NVL72 and about 50% in a GB300/Moonshot scenario using 30% and 60% of list price. Results remain sensitive to utilization, node availability, software, and traffic.
• TPU V7’s external availability creates the clearest prospective third competitor beside NVIDIA and AMD. In the team’s early testing, TPU V7 occupied nearly the entire low-cost edge of the Pareto curve, with only one point beating it, using an external-TCO input around $1.21. Software is still immature; native Torch support, TPU vLLM, megakernels, V8, and GCP hosting beyond Google’s internal environment could improve the position.
• Vera Rubin looks less disruptive operationally than Blackwell but potentially much stronger economically. At a roughly 100-token-per-second operating point, the early discussion suggested an improvement approaching 3×, while high-throughput gains could be closer to about 1.4×; the roughly 3× HBM3-to-HBM4 memory-bandwidth progress helps explain the memory-bound middle. A hardware-accelerated lookup-table quantization path—8-bit table values, 3-bit keys, roughly 3.125 effective bits—remains an area to watch, with no major application identified yet.
• The debate over fast tokens is unresolved because agent workflows contain both latency-sensitive and latency-insensitive stages. Bryan, Cam, and Alec see limited ROI when several agents run in parallel or 30-second tool calls dominate, especially at prices approaching $300 per million output tokens; Alec also cautions that future workflows may change. TensorRT’s hundreds of tokens per second on NVIDIA GPUs strengthens the case for specialized inference software, but batch-size-1 megakernels, changing prefill/decode ratios, and difficult deployment economics complicate custom-silicon theses.

Deep dive

1. Engrams replace repeated semantic work with addressable memory

• Cam’s framing: ordinary token embeddings accumulate meaning as they traverse Transformer layers, but Engrams instead learn embeddings for two- and three-token sequences, hash them into a table, and inject the retrieved representation into the model.

• The motivating discussion referenced DeepSeek work comparing next-token prediction capacity across hidden-state depths and KL divergence relative to the final layer. Bryan’s explanation was that Engram moves word-meaning information earlier, so later layers do not need to spend as many parameters reconstructing it.

• The key ablation was not “replace everything with memory.” The loss curve improved around a midpoint combining Engram modules with conventional MoE layers, while the all-MoE/no-Engram and all-Engram/no-MoE endpoints were less effective. That is why the team describes the gain as receiving several “free layers,” not eliminating Transformer computation.

• Placement creates the central co-design trade-off. Earlier retrieval gives more downstream layers access to the memory and helped the model more in the reported ablations, but it leaves little computation behind which to hide a DRAM or SSD fetch; placing the module later provides more opportunity for asynchronous overlap.

2. Memory scarcity is shaping Chinese model architecture

• Token IDs are available before the forward pass begins, so an Engram lookup can start immediately through unified virtual addressing and pinned host memory on the CPU. Unlike retrieval dependent on an intermediate hidden state, it can overlap with other parts of the model, making DRAM—and potentially SSD—a practical memory tier.

• The speakers place Engrams alongside DeepSeek’s earlier work on reducing the KV cache and using HBM. Their broader point is that information that can be retrieved early and overlapped becomes a natural candidate for offload.

• Chinese accelerators intensify that incentive. Domestic systems may use a Chinese-built HBM-like technology with lower bandwidth, so Chinese labs are pushed toward smaller KV representations, compression, lower memory tiers, and architectural tricks; US labs have had more HBM and what Cam described as effectively unlimited compute relative to that constraint.

3. AgentX makes the KV-cache problem measurable

• Cam’s description of an agent trace: a user sends a task, the model emits text and tool calls, tool outputs are appended, and the growing conversation returns to a stateless model. The KV cache “grows at a crazy rate” across every turn.

• AgentX uses coding sessions captured through a proxy used by SemiAnalysis employees, removes unsuitable requests, and replays the remaining traces against open-source serving stacks. The team concedes this does not represent all coding traffic, but argues that the basic shape—task, tool loop, added context, repeated prefix—is common.

• The striking numbers are a P50 request around 100,000 input tokens and a theoretical cache-hit rate above 98%. Most turns reuse much of the prior context, so routing a request to the right prefill server, transferring KV between prefill and decode, and preserving cache residency all require careful system design.

• Offload’s value depends on the operating point. At low concurrency and high interactivity, there is little KV to spill and slower secondary-tier memory adds limited value; around 40–60 tokens/s with hundreds of concurrent sessions, offline batching, or reinforcement-learning rollouts, DRAM offload becomes relevant. Bryan separately said he would like to see cache-hit rates in the 20–30% range for HBM rather than VRAM.

4. Serving margins reward utilization more than benchmark supremacy

• Bryan explains that the calculator begins with hourly ownership cost—accelerators, datacenter costs, electricity, and associated infrastructure—then combines those figures with model throughput, concurrency, utilization assumptions, and OpenRouter pricing. It is intended to translate benchmark curves into the profit available from different chips under a specific workload.

• On the assumptions discussed, a GB200 NVL72 deployment could earn approximately a 40.7% profit margin. Other margin profiles were shown for Kimi K2 and DeepSeek V4.1 Flash. Jordan also cited a GB300/Moonshot scenario using 30% and 60% of list price that produced about a 50% margin.

• The pushback is embedded in the model: 60% utilization means the system is effectively saturated unless nodes are down, and providers may have poor utilization, different software, or different compute prices. Bryan said some providers can still be profitable at 30–40% utilization. Jordan’s point was that providers want to recover rack capital within months, not that every scenario guarantees that result.

5. TPU externalization turns Google into a platform competitor

• Alec’s early TPU V7 results put the public-cloud offering on the cheapest part of nearly the entire price-performance Pareto curve, with only one point beating it. The team used an external-TCO input around $1.21 and emphasizes that these are first-day results, before further software optimization.

• The comparison is imperfect: TPU’s silicon, programming model, accessibility, and community support differ materially from B200 or B300. Fixed-size systolic arrays can waste compute through padding, and Google’s internal models were historically designed around constraints that open-source models do not necessarily respect.

• Software is the obvious catch-up path. The speakers expect TPU vLLM, more megakernel work, and the forthcoming Torch-TPU backend to reduce programming-model friction; v8 is part of the next stage of TPU development, while v7 itself can improve as kernels and public-model support mature.

• Bryan’s larger call is “large-scale externalization”: Google will still use a large amount of TPU capacity internally, but will sell some of it. GCP hosting TPUs outside Google’s own environment could open the ecosystem to more customers, while Cam described AMD and NVIDIA as gaining a real third competitor.

6. Rubin, fast decoding, and custom silicon split the workload

• The move from H100/H200 HGX systems to GB200 and GB300 introduced an Arm CPU, scale-up NVLink switches, new networking, and liquid cooling; Blackwell’s SM100 also introduced a new FP4 data type. Vera Rubin appears operationally more incremental: no major new scale-up network or Arm CPU, and liquid cooling is already deployed, but with substantially more memory bandwidth.

• The performance curve is workload-dependent. Around 100 tokens/s, the early discussion suggested Rubin could approach 3× the performance or users at a given interactivity level; toward maximum throughput, the advantage could be closer to roughly 1.4×. The HBM3-to-HBM4 bandwidth progress of about 3× maps closely to the memory-bound middle of the curve.

• Rubin also exposes a new hardware-accelerated lookup-table quantization path using 8-bit values and 3-bit keys, around 3.125 effective bits. The team has not yet seen the specialized application and explicitly leaves open how much use the open-source ecosystem will make of it; comparisons with MXFP4-like scaling and accuracy evaluations remain areas to watch.

• Fast-token demand produced the sharpest disagreement. Bryan prefers many parallel agents; Cam notes that a 30-second tool call can nullify thousand-token-per-second decoding; Alec questions the ROI and says the price is high. Alec’s historical caution is that one or two years ago he would not have predicted his current AI workflow, so future demand is difficult to forecast. Cam’s prediction is that fast modes will be useful for some tasks, while batchable work can tolerate more latency.

• TensorRT demonstrates the software extreme: fuse much of the model into a giant dataflow-style megakernel and achieve hundreds of tokens per second, including a chart point near 500 on a reasonably large model approaching one trillion parameters, using NVIDIA GPUs alone. The cost is enormous engineering effort, slow model onboarding, and batch-size-1 behavior that favors interactivity over high-throughput serving.

• Jordan is therefore unworried about accelerator startups finding initial customers because “demand is completely ahead of supply,” but the group disputes whether that proves durable economics. Building working silicon is now plausible; achieving good yield, packaging servers, securing distribution, operating datacenters, adapting to shifting prefill/decode ratios, and competing with NVIDIA’s software and controlled GPU supply remain the harder business problems.