Ep. 030 - Long Live the Short King: Why 4-hi HBM Wins (Memory)

Summary

• Rubin Ultra is now understood to have shifted from the 1TB of HBM4E previewed at GTC to 192GB of 8-high HBM4—below Blackwell Ultra’s and vanilla Rubin’s 288GB. Myron Xie says the primary driver is supply rationing: NVIDIA and customers including Google and Broadcom have more TSMC logic secured than their HBM allocation can support in 12-high form, so using 8-high cubes lets them ship more accelerators. Jordan Nanos flags this as the first time, in his experience tracking NVIDIA, that a flagship has less memory capacity than its predecessor.
• The economic core: HBM bandwidth is essentially fixed per cube while suppliers charge for capacity, making 4-high the best dollar-per-bandwidth configuration when its capacity is sufficient. Four DRAM layers saturate the cube’s interface, so 8- or 12-high costs almost double or triple for the same bandwidth. “Paying for 4-high only, when all you really need is that bandwidth … is almost a free lunch.”
• Capacity pressure is much lower because parameter counts have not scaled as aggressively as expected. Xie compares Llama 3.1 405B at FP8, which used roughly 60% of a Hopper HGX node’s 640GB, with Kimi K3 at 2.88 trillion parameters, which he describes as a bit more than six times larger but which MXFP4 reduces to about 8% of a GB200 NVL72’s aggregate HBM. Deployment domains grew much faster, with Rubin Ultra’s NVL576 adding another 8x. Meanwhile post-training and standard inference, which are more bandwidth-sensitive, now dominate a larger share of frontier-lab compute, while classic pretraining has declined.
• Four-high can roughly double harvestable cubes versus 8-high and more than double them versus 12-high. Per-layer yield losses compound with stack height, and power delivery is easier in a 4-die stack. The bottleneck therefore shifts toward leading-edge logic, substrates and power. Lower HBM demand also frees DRAM wafers for conventional server DRAM, important while server sockets are being de-specced and CPUs handling tool calls may be poorly utilized without enough memory. Nanos frames 4-high as maximizing “tokens per HBM wafer,” analogous to tokens per watt and per dollar.
• No relief on the memory crunch is expected soon: Xie’s timeline is “not within this decade.” Cleanrooms are the main short-term blocker, followed by equipment such as EUV tools whose ASML supply chain—including suppliers making “very smooth mirrors”—cannot be expanded quickly. “You can’t just click your fingers and add wafer capacity,” even as SK hynix, Samsung and Micron benefit from high prices.
• The bear case is that exploding model sizes would make higher stacks worthwhile. SemiAnalysis modeled a model three times the size of Kimi K3, where batching makes the throughput gain from 8- and 12-high “probably worth the cost.” Xie’s response is that techniques such as looped transformers add compute depth without adding parameters; he says GPT-6 Astra is basically confirmed to use them. He also views the labs’ strong demand for 4-high as evidence that their road maps do not anticipate aggressive parameter growth.
• Expect more SKU segmentation. Meta already has a custom MI450 version with 8-high HBM instead of 12-high, and Xie says additional segmentation makes sense when supply-chain resources are tight. The current 288GB Rubin configurations are not the reset; the change is associated with Rubin Ultra, the next version after the current generation, arriving this time next year or into 2028.

Deep dive

1. Rubin Ultra: 1TB becomes 192GB, primarily as a supply decision

• Xie’s setup: every generation packed in more HBM—8-high in the Hopper/Ampere era, 12-high with Blackwell Ultra, and 16-high contemplated for HBM4E. NVIDIA previewed Rubin Ultra at GTC as 1TB per package via four compute dies and 16 stacks of 16-high HBM4E. A year and a half later: two compute dies, base HBM4 at 3GB per die versus 4GB for HBM4E, and 8-high—192GB.
• Nanos’s marker, worth keeping: 192GB sits below the 288GB in both Blackwell Ultra and vanilla Rubin—“the first time in the history of me keeping track of Nvidia that the next flagship GPU is going to have less capacity than the previous one.”
• The mechanism: NVIDIA and customers including Google and Broadcom have HBM secured alongside a larger amount of logic secured at TSMC. In 12-high form, the HBM supply cannot feed all that logic, so “let’s ration our HBM supply and ship it in 8-high cubes”; more cubes from the implied wafers help balance the equation. Beyond the threshold needed for model weights and KV cache to batch effectively, returns on extra HBM diminish, and with prices rising into the shortage, “the penalty for having too much HBM capacity becomes much worse.”

2. Why capacity gave way to bandwidth: the models did not get as big as expected

• Xie’s arithmetic across eras: Llama 3.1 405B quantized to FP8 consumed roughly 60% of a Hopper HGX node’s 640GB, which is why the H200’s capacity bump was genuinely valuable. Today’s biggest model, Kimi K3 at 2.88 trillion parameters—described by Xie as a bit more than six times the size of Llama 3.1—can be quantized to MXFP4 and occupies around 8% of a GB200 NVL72’s aggregate HBM. Scale-up domains grew much faster than parameter counts, and Rubin Ultra’s NVL576 adds another 8x.
• Nanos’s demand-side read: customers want faster versions of Kimi K3 and other existing models—and “speeding up tokens is all about bandwidth, basically. Capacity doesn’t really add anything.”
• The compute-mix shift reinforces it: post-training is “very inference-like,” and post-training plus standard inference now account for a dominant and growing share of frontier-lab compute, while classic pretraining’s share shrinks. That is visible in buildouts, where the marginal datacenter is going to research, post-training and inference, not “pre-training runs for that 20-trillion-parameter model that we were thinking about two or three years ago.”

3. The four-high free lunch: same bandwidth, a fraction of the bill

• The physics: bandwidth is essentially fixed per cube regardless of height, driven by the HBM-to-SoC interface. HBM4 exposes 2,048 I/Os between the cube and compute, and each DRAM die supports up to 512 signal I/Os, so four layers are needed to saturate the interface. You cannot go lower than 4-high and keep full bandwidth.
• The pricing asymmetry: suppliers charge effectively per gigabyte, so 8- or 12-high costs “almost double or triple” for identical bandwidth. “Paying for 4-high only, when all you really need is that bandwidth and 4-high capacity gives you enough—that’s almost a free lunch.”
• Xie’s categorical extension: other accelerator vendors will follow—“the supply chain challenges that NVIDIA faces are the same for everyone,” and “a lot of customers’ road maps have been reset to favor lower-height HBM.”

4. Supply-chain ripple: roughly double the cubes versus 8-high, and a bottleneck that migrates

• Why the yield improvement is so large, per Nanos’s challenge: per-layer yield loss compounds—an approximately 1% loss repeated across 8 or 12 layers produces much more loss than across four. Delivering power up a 4-die stack is also easier, and 4-high is a simpler product with a longer manufacturing history. Xie says the result is roughly double the cubes versus 8-high and more than double versus 12-high. The bottleneck then moves to leading-edge logic at TSMC and to substrates, “one of the tightest areas,” with power also a major constraint.
• A second-order benefit: relaxing HBM demand frees DRAM wafers currently being cannibalized from server DRAM, where per-socket de-speccing is widespread—an issue because GenAI needs “a lot of CPUs just to perform all the tool calls,” and those CPUs may be “poorly utilized” without enough DRAM. Nanos’s frame: in a DRAM-constrained world, 4-high maximizes “tokens per HBM wafer,” analogous to tokens per watt or per dollar.
• On why booming memory profits do not summon supply: long lead times—cleanrooms first, then equipment, with ASML EUV tools constrained by their own specialized suppliers, including those making “very smooth mirrors.” “You can’t just click your fingers and add wafer capacity.” Xie’s timeline for the crunch easing: “not within this decade.” Nanos replies, “2030, here we go.”

5. The bear case, the labs’ tell, and the coming SKU segmentation

• Xie volunteers the counterargument: if model sizes explode, batching economics flip—each user in a batch needs only one read of the weights, so larger weights make extra capacity more valuable. SemiAnalysis modeled a model three times the size of Kimi K3, where the gain from 8- and 12-high “is probably worth the cost.” The risk is whether everyone going 4-high would “shoot themselves in the foot.”
• His resolution: footprint-shrinking techniques keep winning—looped transformers, which he says GPT-6 Astra is basically confirmed to use, add compute depth without adding parameters. He also says “the loudest cries for 4-high are coming from the labs,” a tell that their research road maps do not show aggressive parameter scaling. Nanos’s supporting jab: “it would be very embarrassing for OpenAI and Anthropic if Kimi K3 was this close to their performance,” with Kimi at roughly 2.8 trillion parameters while they were at 10 trillion—implying frontier models may be of similar size.
• Nanos’s push on multi-SKU futures—V100 launched at 16GB then 32GB, A100 at 40GB then 80GB, and H100 at 80GB followed by H200 at 144GB—asks why a Meta flavor and an OpenAI flavor of the same die could not coexist. Xie says this is already happening, citing a custom Meta MI450 with 8-high HBM instead of 12-high; when overprovisioning gets costly, “more SKU segmentation is absolutely on the cards.”
• The closing timing caveat: the current 288GB Rubin configurations are not the change under discussion. The reset belongs to Rubin Ultra—“if there was an R200, this is the R300”—the next version after the current generation, arriving this time next year or into 2028.