Developers
1 Curated Dialogues
Cam Quilici
Key Views & Dialogues
Ep. 002 - InferenceX 2.0 Release (Technical Staff) | Cam Quilici, Bryan Shan, Doug O’Laughlin, Jordan Nanos
InferenceX 2.0 shows GB200/GB300 delivering 20x DeepSeek-R1 throughput per GPU versus a fully tuned H100 at low interactivity, and 80–100x at 100 tokens/sec/user. NVLink’s 72-GPU domain, software optimization, and multi-token prediction drive the gap, while MI355’s roughly 25% TCO advantage over B200 highlights margin potential; composability, legacy fleets, and larger frontier models remain risks.
View summary (7)
- The headline result of InferenceX 2.0 is that GB200/GB300 delivers 20x the DeepSeek-R1 throughput per GPU of a fully-tuned H100 at low interactivity — and 80–100x at 100 tokens/sec/user, a speed the H100 essentially can’t serve at all. The mechanism is the NVLink domain: DeepSeek’s 256 experts dispersed across “72 GPUs acting as one” rather than over InfiniBand, which Cam Quilici says simply “frame mogs” everything else. Jordan Nanos stresses 100 tok/s is a real production requirement — “fast” tiers like Codex Spark (on Cerebras) already serve above it, versus ~40 for standard models.
- Software is compounding as fast as hardware: AMD’s MI355 multi-node DeepSeek recipe roughly tripled from ~1,000 to ~3,000 tokens/sec/GPU at 20 tok/s interactivity in about a month of ROCm optimization. Cam’s core critique of incumbent benchmarks is “benchmark maxing” — one point in time, one point on the Pareto frontier, submissions every three to six months, and vendors able to unsubmit losing results. InferenceX runs nightly, and “once they’re up, they’re not gonna be taken down.”
- Multi-token prediction is “basically a free optimization”: turning MTP on took B300 (Dynamo + TensorRT) serving DeepSeek at 68 tok/s/user from 25 cents to 5.7 cents per million tokens — an ~80% cost cut with no accuracy loss on GSM-8K. DeepSeek popularized pre-trained MTP heads and now “basically everyone is using it,” including GLM-5 and Qwen 3.5; it’s definitely in production today.
- On TCO rather than raw throughput, MI355 flips ahead of B200: the two are “for all intents and purposes equal” on FP8 SGLang throughput, but MI355’s ~50-cent price difference makes it ~20 cents per million tokens cheaper — roughly a 25% edge that matters “when margins are pretty thin.” The caveat is AMD’s composability problem: FP8 single-node matches B200, FP4 is significantly worse, and stacking disaggregation plus MTP degrades further — flags that work alone don’t work in combination.
- Doug O’Laughlin’s econ-brain takeaway is bullish inference margins: current benchmarks use random single-turn data with zero prefix caching, so “this is the worst it would be — it would only get better from here.” With MTP, multi-turn caching and the rest stacking toward ~50% price decreases, he now thinks Max “might not be as unprofitable as you think” versus SemiAnalysis’s first tokenomics estimates. Jordan’s offsets: frontier models are likely bigger than DeepSeek, and these gains apply to the newest GPUs, not the large legacy fleets still serving inference.
- Roadmap: Trainium and TPU coverage is a V3/V4 goal, plus planned day-zero support for DeepSeek V4; the team is currently trying to provide day-zero support for Qwen 3.5, Minimax and Kimi 2.5. They also plan EPD (encoder-prefill-decode) disaggregation for multimodal agentic workloads and a real-world multi-turn prefix-cache benchmark. The blocker is that agentic/multimodal datasets essentially don’t exist publicly — the best available is WildChat, “three years old… people asking GPT to be their therapist” — so the team plans to generate its own traces.
- The coda is an honest hedge on the whole AI trade: Cam admits a “fleeting thought — what if they just stop getting better? Then what? Then we’re fucked, man.” His resolution is the thesis InferenceX itself documents: even if “models aren’t getting smarter,” they are getting “cheaper, faster, and more performant,” so diffusion widens regardless. Cam’s experience with Codex 5.3 (“better for targeted issues,” Opus 4.6 “for general vibes”), which Doug agreed with, is offered as proof they update on evidence.