How GPT, Claude, and Gemini are actually trained and served – Reiner Pope

Summary

• Batch size is the main effect behind “fast mode” pricing. Pope’s roofline analysis — time is lower-bounded by the max of compute time (batch × active params ÷ FLOPs) and memory time (weight fetch + KV fetch ÷ bandwidth) — shows a hard latency floor (reading all total params from HBM) and a hard cost floor (the compute line), with unbatched serving “a thousand times worse” economically. A hypothetical Claude Code Slow Mode barely helps: past the crossover batch, KV and compute are unique per user and can’t be amortized further.
• Optimal batch size ≳ 300 × sparsity ratio — a dimensionless hardware constant (FLOPs/bandwidth ≈ 300 on most GPUs, stable A100→B100) times the model’s expert sparsity (~8 for DeepSeek), giving ~2–3K concurrent sequences and ~128K tokens/sec per system — roughly one-thousandth of Gemini’s global traffic. Scale economies in inference exist but are smaller than commonly assumed.
• Scale-up domain size, especially aggregate bandwidth, is a key constraint on model scale — active parameters are limited by compute cost, while total parameters are limited by scale-up size; pipelining can solve the model-weight capacity problem. Hopper’s 8-GPU scale-up had 640GB in 2022; Blackwell’s 10–20TB scale-up is enough for a 5T model plus KV cache, and Rubin is discussed at ~500-something. Pope discusses larger scale-up domains as one possible explanation for why frontier parameter counts barely grew for three years after GPT-4, and they may have contributed to Google’s Gemini pre-training edge.
• The memory wall is real for context length: models have hovered at 100–200K for two years because KV-cache bandwidth, not compute, is the binding cost — and Pope sees “no very good path to solving that. The HBM is where it is.” Sparse attention’s square-root scaling is a get-out, which cuts against the “in-context learning is enough for AGI” thesis requiring 100M-token contexts.
• Frontier models may be ~100× over-trained vs Chinchilla, derivable from first principles: equalize pre-training, RL, and inference costs and you get roughly equal token counts in each (~150–200T), versus a Chinchilla-optimal ~2T for a ~100B-active model. Dwarkesh’s framing: “Each model should generate the sum of human knowledge on the output that it gets on the input.”
• API pricing leaks architecture. Gemini 3.1’s 50% surcharge above 200K context pins the compute/memory crossover and back-solves to ~2KB of KV cache per token; a 5× output-vs-input price gap suggests decode is “tremendously memory bandwidth bottlenecked”; and cache-storage tiers priced at 5 minutes vs 1 hour may correspond to the drain times of flash and spinning disk — “I’m kind of shocked to see spinning disk being used at all.”
• Dwarkesh’s paradox worth holding: hyperscalers reportedly spend ~50% of CapEx on memory (per Dylan), yet Pope’s math shows racks may have a capacity surplus for weights — pipelining means weights need not fit in one rack. For this scale-up question, the scarce resource is memory bandwidth, rather than weight capacity, a distinction that matters for anyone underwriting the HBM supercycle.

Deep dive

1. Two curves explain fast mode, slow mode, and everything in between

• Pope’s framework, applied to a Blackwell NVL72 rack: inference time is bounded below by the max of two terms — compute time (batch size × active parameters ÷ chip FLOPs, e.g. DeepSeek V3’s 37B active out of ~700B total) and memory time (fetching all total parameters plus, per batch element, an entire context’s KV cache, ÷ memory bandwidth). “Even with a simple model,” this gives “very strong predictive power.”
• Plotting latency vs batch size: compute is linear from zero, weight fetch is a constant floor, KV fetch is linear — so there is a lower bound on latency for a given hardware config: “I need to read all of my total parameters from memory into the chips… If I use all of my memory bandwidth, I can’t do any better than that.” That’s why paying 100× more can’t buy unbounded speed.
• Dividing everything by batch to get cost per token: weight fetch becomes a decaying parabola, KV and compute become constants. Cost “almost goes to infinity” at batch size one, then asymptotes to the compute line — hence unbatched serving can be “a thousand times worse.” And a Claude Code Slow Mode “wouldn’t help much because you’re not able to amortize the KV values over a much bigger batch. They’re unique per batch. The compute is also unique per batch.”

2. Batch ≈ 300 × sparsity, and the train departs every 20 milliseconds

• Equating weight-fetch time to weight-multiply time yields a clean result: batch size ≥ ~300 × sparsity, where ~300 is FLOPs ÷ memory bandwidth in FP4 terms — a dimensionless hardware constant that has “remained reasonably stable” from A100 through B100. With DeepSeek activating 32 of 256 experts (sparsity 8), that’s ~2,400; in practice “take this and maybe double or triple it.” Crucially it depends only on sparsity, “not on scale.”
• Queuing works like a train timetable: “A new train departs every 20 milliseconds. Any passengers who are ready board the train… If the train is not full, the train is going to go anyway.” Worst-case queuing latency: 40ms.
• The 20ms itself is HBM capacity ÷ bandwidth — the time to read all of memory once per forward pass (Rubin: 288GB ÷ 20TB/s ≈ 15ms), stable “on many different generations of HBM.” At around 30ms, Pope says, “I can read all of HBM twice… I don’t want to read the weight matrices twice.”
• Translated to throughput: batch × ~64 passes/sec ≈ 128K tokens/sec per system — versus Gemini’s remembered announcements of “hundreds of millions of tokens per second worldwide.” Pope’s bar: “To actually be competitive at scale, you need to be able to serve at least one-thousandth of Gemini.”

3. Sparsity looks inefficient on quality curves — and is still a pure systems win

• On quality, the returns are unimpressive: from the “Unified Scaling Laws for Routed Language Models” paper, a 64-expert, 370M-active model matches a dense 1.3B — “64x for 4x… a huge increase in parameter count for a modest increase in efficiency,” and results are “very sensitive to the actual choice of mixture of experts” (DeepSeek’s finer-grained MoE was “a big change”).
• Yet from the roofline view, doubling sparsity while batch grows to match is “a pure win. Keep doing it until you run out of available users” — the extra total-parameter fetches amortize across the bigger batch. The real costs: you need more concurrent demand, and more memory capacity — which pulls the whole thread toward rack architecture.

4. MoE layouts are shaped by the rack: all-to-all inside, 8× slower outside

• Standard practice is expert parallelism — DeepSeek’s 256 experts spread across (a rounded-down) 64 of the NVL72’s 72 GPUs — producing an all-to-all traffic pattern that is “a perfect fit” for how Blackwell racks are wired: every GPU reaches every other in two hops via central NVSwitches. Cross two racks and half your tokens hit the scale-out network, typically ~8× slower — “that becomes the bottleneck.”
• Why not one giant switch? Physical cabling. Doubling GPUs per rack means “literally twice the density of cables,” against connector density, backplane limits, and bend radius. Dwarkesh’s surprise stands in: “The rack is so big and we can’t just stuff more cables in there… I had no idea.” Racks push “space, weight… power and cooling… to very extreme physical limits.”
• The trajectory: Hopper 8 → Blackwell 72 was “mostly just the decision to switch from trays… to racks — a product decision”; Blackwell → Rubin’s ~500 has “a bit of Jensen math… but a genuine 4x increase” from a genuinely harder rack design. “One rack bounds the size of an expert layer you can do.”

5. One possible reason model sizes stalled after GPT-4

• Dwarkesh’s setup: GPT-4 (2023, rumored >1T params) wasn’t meaningfully out-scaled until roughly the last six months. Pope’s possible mechanism: eight Hoppers gave 640GB in 2022; only Blackwell’s rack-scale 10–20TB is enough for a 5T model plus KV cache. “Deploying in larger scale-up domains is a huge unlock.”
• On Gemini’s apparent pre-training lead — Google “has actually had very large scale-up domains for a long time” — Pope hedges honestly: “Not having been there at the time, I’m not sure how much is coming from successfully deploying higher sparsity ratios… It could also be a whole bunch of actual modeling things.” The broader takeaway is that larger scale-up domains may have contributed, but Pope says it is hard to disentangle them from modeling and data improvements. The clean claim: active params are limited by compute cost; total params by scale-up size.

6. Pipelining saves weights, not KVs — so inference lives inside one scale-up

• Pipeline parallelism (different layers on different racks) clears the scale-out bandwidth bar easily: scale-up traffic exceeds scale-out by (activated experts × layers per stage × 2), against only an 8× bandwidth penalty. Pope’s meta-point on why expert + pipeline won: “What are all the different dimensions in which a model is scaled up?… Every single one of those numbers you can choose to cut along” — and tensor parallelism became much less relevant because “the experts are so small now.”
• The catch Dwarkesh imported from Jane Street’s Axel: pipelining forces micro-batches. Working through the algebra, the per-GPU weight footprint shrinks with pipeline stages P, but the KV term’s P’s cancel — keeping all racks busy means more sequences in flight, and “those exactly cancel, and you end up not getting a saving per GPU.” KV cache “sucks from both points of view”: unamortizable across batch and unshardable across stages.
• Practical upshot, per the DeepSeek paper: max out expert parallelism to your scale-up size, then “very little pipelining. Maybe none at all, maybe two.” So, for typical models, frontier labs generally run inference within a single scale-up domain. In inference, pipelining is latency-neutral (~a few ms per rack hop, stacking to maybe 10ms/token across 4 stages — “actually a lot” when base latency is 20ms), and the memory it saves is already in surplus: “a trillion parameter model only needs one terabyte, so it already fits.”
• Historical color kept: Ilya’s “today we know not to do pipeline parallelism,” and Horace He’s objection that pipelining imposes architectural constraints (e.g. Kimi’s cross-layer attention residuals). Pope concedes “these complexities are real. Pipelining is a massive hassle” — the bubble diagrams show why training needs zero-bubble/1F1B tricks (“you can mine Bitcoin in that bubble. — Right. More usefully, you can do the weight gradient step”).

7. The memory paradox: 50% of CapEx on a resource racks have too much of

• Dwarkesh’s pushback — worth keeping in full: “Dylan said hyperscalers are spending 50% of their CapEx this year on memory… That’s why we’re not going to get new laptops and phones this year. But at the same time, we have too much memory? Why is Jensen shoving all this memory into these racks if you don’t need it?”
• Pope’s resolution: what big scale-up domains buy is not capacity but aggregate memory bandwidth — weight loads parallelize across every GPU in the domain, and that term “increased by a factor of 8 from Hopper” while per-GPU bandwidth grows only 1.5–2× per generation. “Pipelining totally solves the capacity problem, but scale-up size helps solve the bandwidth problem” — which is what enables low latency and longer, more agentic contexts.
• The bearish corollary for context scaling: model context lengths “shot up from about 8K to 100-200K” between earlier models like GPT-3 and GPT-4, though Pope doesn’t remember exactly when, and “for the last year or two, they’ve all been hovering around there… I think it’s because there is no solution to the memory wall here.” Sparse attention (DeepSeek’s published square-root variant) “gives you a get-out for sure… It’s not an infinite improvement because if you go too sparse, you lose too much quality.” Against Dario’s “in-context learning is enough” — which needs ~100M-token contexts — Pope: “I actually don’t see a very good path to solving that.”

8. First-principles the training budget: everything equalizes, ~100× over Chinchilla

• Pope’s heuristic: for power-law-ish curves, a sum of costs “tends to be minimized where the costs are equalized.” Set pre-training (6ND), RL (2–6× N×D_RL — 2 because rollouts you don’t train on cost only a forward pass, plus decode running at worse MFU), and inference (2ND) equal, and active params cancel: pre-training tokens ≈ RL tokens ≈ inference tokens, “within factors that we’re not able to reason about” — with somewhat fewer RL tokens since RL is machine-time-inefficient. (A live algebra mistake got caught on the board: “Billions of dollars worth of compute just flowed in the other direction… if you do it with a spreadsheet, you might notice when the money’s going down the drain.”)
• Plugging in: perhaps ~500M tokens/sec worldwide, reduced 5–10× per specific model, × two months of deployment ≈ ~200 trillion inference tokens — strikingly close to the reported-to-him ~150T pre-training tokens on a frontier model. Chinchilla-optimal for ~100B active params is ~2T, so this estimate implies models may be “a factor of a hundred over-trained.”
• Dwarkesh’s takeaway framing: “Even if it’s 50% off or something, it is wild that you can first-principles these kinds of numbers.” Pope adds the derating caveat: imperfect demand prediction and the risk of a discarded non-frontier model mean “you should derate the inference tokens by some amount.”

9. Reading the labs’ internals off their price sheets

• Context surcharge: Gemini 3.1 charging 50% more above 200K tokens than below it may mark the compute/memory crossover on the cost-vs-context curve — a two-tier price ensuring profitability on either side. Assuming ~100B active params and the 1/300 hardware constant, the crossover back-solves to ~1,667 bytes of KV per token — “almost two kilobytes… plausible, if maybe a little bit small,” consistent with e.g. 8 KV heads × d_head 128 with Character-AI/Gemma-style cross-layer sharing, or sparse attention with larger raw numbers. Dwarkesh: “It’s funny that they would leak so much information through their API pricing.”
• Input vs output: prefill amortizes weight fetches across the whole pass length while decode processes one token — so a 3–5× output premium suggests decode is priced ~5× compute cost, i.e. the serving stack “is, in fact, tremendously memory bandwidth bottlenecked.”
• Cache tiers: cache-write pricing at 5-minute and 1-hour durations may reflect a memory tier’s drain time (capacity ÷ bandwidth). HBM drains in 20ms — far too fast; DDR in seconds; flash ~1 minute; spinning disk ~1 hour. So the tiers are “probably flash and spinning disk” — not HBM/DDR as Dwarkesh first guessed. Pope: “It’s a really unattractive technology but it’s useful in some places.”
• The tiering logic, kept because it generalizes: rematerializing KV from scratch costs compute; storing costs a fraction of a device; the optimum holds an item “for so long such that the amount of time it’s in there is the time to get all your things in there and out.”

10. Ciphers and neural nets: same mixing machinery, opposite objectives

• From Pope’s blog post: both architectures must jumble information across all inputs, but “they’re actually trying to do the opposite things” — ciphers make structure indistinguishable from randomness; neural nets extract structure from apparent randomness. The differentiator is differentiability: residual connections and LayerNorm help keep derivatives contained and simple, whereas “the whole job of a well-designed cipher is to make the difference in output very large” — differential cryptanalysis being literally the attack of differentiating a cipher over the field of 2 elements. Adversarial image attacks exhibit the cipher’s avalanche property; in the backdoor example, the hidden input is into the backward pass.
• Using neural nets as ciphers: don’t — “if it doesn’t have 10 years of scrutiny, it’s probably broken.” But the reverse import worked: the Feistel cipher construction — turning a non-invertible f(x) into an invertible two-input layer (output x, y+f(x); invert via z − f(x)) — became RevNets, making whole transformer-like networks reversible so activations can be rematerialized during the backward pass instead of stored. Dwarkesh’s closing symmetry: RevNets spend compute to save memory — “the opposite of the KV cache.” Pope: “Spending more memory to save compute is generally profitable given where hardwares are.”