Ep. 002 - InferenceX 2.0 Release (Technical Staff) | Cam Quilici, Bryan Shan, Doug O'Laughlin, Jordan Nanos

Summary

• The headline result of InferenceX 2.0 is that GB200/GB300 delivers 20x the DeepSeek-R1 throughput per GPU of a fully-tuned H100 at low interactivity — and 80–100x at 100 tokens/sec/user, a speed the H100 essentially can’t serve at all. The mechanism is the NVLink domain: DeepSeek’s 256 experts dispersed across “72 GPUs acting as one” rather than over InfiniBand, which Cam Quilici says simply “frame mogs” everything else. Jordan Nanos stresses 100 tok/s is a real production requirement — “fast” tiers like Codex Spark (on Cerebras) already serve above it, versus ~40 for standard models.
• Software is compounding as fast as hardware: AMD’s MI355 multi-node DeepSeek recipe roughly tripled from ~1,000 to ~3,000 tokens/sec/GPU at 20 tok/s interactivity in about a month of ROCm optimization. Cam’s core critique of incumbent benchmarks is “benchmark maxing” — one point in time, one point on the Pareto frontier, submissions every three to six months, and vendors able to unsubmit losing results. InferenceX runs nightly, and “once they’re up, they’re not gonna be taken down.”
• Multi-token prediction is “basically a free optimization”: turning MTP on took B300 (Dynamo + TensorRT) serving DeepSeek at 68 tok/s/user from 25 cents to 5.7 cents per million tokens — an ~80% cost cut with no accuracy loss on GSM-8K. DeepSeek popularized pre-trained MTP heads and now “basically everyone is using it,” including GLM-5 and Qwen 3.5; it’s definitely in production today.
• On TCO rather than raw throughput, MI355 flips ahead of B200: the two are “for all intents and purposes equal” on FP8 SGLang throughput, but MI355’s ~50-cent price difference makes it ~20 cents per million tokens cheaper — roughly a 25% edge that matters “when margins are pretty thin.” The caveat is AMD’s composability problem: FP8 single-node matches B200, FP4 is significantly worse, and stacking disaggregation plus MTP degrades further — flags that work alone don’t work in combination.
• Doug O’Laughlin’s econ-brain takeaway is bullish inference margins: current benchmarks use random single-turn data with zero prefix caching, so “this is the worst it would be — it would only get better from here.” With MTP, multi-turn caching and the rest stacking toward ~50% price decreases, he now thinks Max “might not be as unprofitable as you think” versus SemiAnalysis’s first tokenomics estimates. Jordan’s offsets: frontier models are likely bigger than DeepSeek, and these gains apply to the newest GPUs, not the large legacy fleets still serving inference.
• Roadmap: Trainium and TPU coverage is a V3/V4 goal, plus planned day-zero support for DeepSeek V4; the team is currently trying to provide day-zero support for Qwen 3.5, Minimax and Kimi 2.5. They also plan EPD (encoder-prefill-decode) disaggregation for multimodal agentic workloads and a real-world multi-turn prefix-cache benchmark. The blocker is that agentic/multimodal datasets essentially don’t exist publicly — the best available is WildChat, “three years old… people asking GPT to be their therapist” — so the team plans to generate its own traces.
• The coda is an honest hedge on the whole AI trade: Cam admits a “fleeting thought — what if they just stop getting better? Then what? Then we’re fucked, man.” His resolution is the thesis InferenceX itself documents: even if “models aren’t getting smarter,” they are getting “cheaper, faster, and more performant,” so diffusion widens regardless. Cam’s experience with Codex 5.3 (“better for targeted issues,” Opus 4.6 “for general vibes”), which Doug agreed with, is offered as proof they update on evidence.

Deep dive

1. InferenceX 2.0 moves the benchmark from single-node deployments to frontier-grade serving

• Doug O’Laughlin joked about the rename (InferenceMax → InferenceX — “shout out HBO Max”). Cam Quilici said the 1.0 release spent three months benchmarking “just single node deployments” of DeepSeek, GPT-OSS and Llama 70B, while “big labs, neo clouds, and serverless providers are using more state-of-the-art techniques” — disaggregated prefill, wide expert parallelism, speculative decoding. 2.0 adds all of them, plus GB300 and B300 hardware, to show “a better picture of what frontier inference is like in production.”
• The technical crux is expert parallelism at NVLink scale: DeepSeek’s 256 experts dispersed across GPUs — EP8 on one node, but across a GB200 NVL72 domain it’s “72 GPUs acting as one,” versus eight B200 nodes scaling over InfiniBand. MI355 multi-node with its Pollara NICs is also now tested.
• Everything is public: the dashboard at inferencex.com and an open repo anyone can fork and run.

2. GB200/GB300 “frame mogs” H100 — 20x to 100x per-GPU throughput on DeepSeek

• Jordan Nanos’s read of the headline chart: versus an H100 with every accumulated tuning, GB200/GB300 supports 20x more throughput per GPU at low interactivity, and at 100 tokens/sec/user “it’s just doing things that H100 can’t even do” — 80 to 100x more throughput per GPU.
• Why 100 tok/s matters: quote-unquote “fast models” are already served above that level — Codex Spark on Cerebras, and Opus 4.6 Fast on suspected Trainium/TPU (Doug’s pushback: “they have GPUs too”) — while standard models sit around 40. “It’s a real requirement,” making the generational gain “such an obvious statement.”

3. A living benchmark: nightly runs, no unsubmitting, and MI355 tripling in a month

• Cam’s critique of prior consortiums (unnamed): “benchmark maxing” — a single point in time and a single point on the Pareto frontier, advertised as one throughput number, when “the problem of inference is so much more nuanced than that.” InferenceX runs nightly/weekly at the release cadence of vLLM, SGLang and TensorRT, and unlike supplier-centric benchmarks where a losing company “can unsubmit the results” — “once they’re up, they’re not gonna be taken down.”
• The proof point: one of AMD’s first public multi-node recipes, to Cam’s knowledge (January 8th), through mid-February shows MI355 FP8 climbing from ~1,000 to ~3,000 tokens/sec/GPU at 20 tok/s interactivity — roughly tripled in about a month of software-level optimization. Cam: hardware ships every 18–24 months; “these are just engineering challenges.”
• Both hosts pre-empt the favoritism accusation: recipes come from AMD and NVIDIA engineers directly — “all credit to them,” says Cam, naming the Dynamo, TensorRT-LLM and AMD Distributed Inference teams — but the results post regardless of who wins.

4. The flag-composability puzzle — and AMD’s specific weakness

• Cam’s picture of modern serving engines: vLLM, SGLang and TensorRT-LLM have “blown up in size,” leaving “1,000 different environment variables” whose right combination per deployment “requires a very deep understanding of all these frameworks.” Bryan Shan’s sharpening: “it’s not as simple as combining good flags — it’s about finding flags that work good in combination.”
• The AMD-specific finding, per Bryan: flags that individually deliver their advertised boost don’t replicate when stacked. Cam’s high-level shape of it: plain single-node FP8 is “on par with B200,” FP4 is “significantly worse,” and adding disaggregation and MTP is worse than even the last step — “composability of software is an issue for AMD at the moment, but I think they can do better.”

5. MTP is a free 80% cost-down — and on TCO, MI355 beats B200

• Bryan’s explanation of MTP: DeepSeek popularized pre-trained multi-token-prediction heads that exploit decode’s memory-bound nature, using logits already produced to verify speculated tokens — “almost a free optimization” with no accuracy drop, validated on GSM-8K. It’s “definitely in production,” and Jordan notes that GLM-5 and Qwen 3.5 now support it too.
• The number that lands: B300 with Dynamo/TensorRT serving DeepSeek at 68 tok/s/user drops from 25 cents to 5.7 cents per million tokens with MTP on — an ~80% decrease at a 3:1 input/output blend. Cam won’t call the pass-through: “I don’t know the economics of this… I’m assuming some of those savings will make it to the end user, but I’m not entirely sure.”
• The TCO layer: B200 and MI355 are throughput-equal on FP8 SGLang, but SemiAnalysis’s cost model — servers, power, everything — shows MI355, ~50 cents cheaper, undercutting by ~20 cents per million tokens at realistic interactivity. Jordan: that’s a 25% change, “when margins are pretty thin.”
• Doug’s tokenomics update: current runs use random single-turn tokens with zero prefix caching, so Cam frames them as “the worst it would be — it would only get better from here.” Doug’s conclusion: “pretty bullish in terms of inference margins… Max might not be as unprofitable as you think.” Jordan’s counterweight: DeepSeek is likely smaller than frontier models, the newest optimizations aren’t fleet-wide, and old GPUs still serve inference — “you need optimizations on all this stuff, not just the GB300 NVL72.”

6. The roadmap runs through datasets that don’t exist yet

• V3 ambitions: Trainium and TPU (“this is gonna be awesome when we do it” — V3, maybe V4), planned day-zero support for DeepSeek V4, current efforts toward day-zero support for Qwen 3.5, Minimax and Kimi 2.5, and Bryan’s EPD (encoder-prefill-decode) disaggregation to benchmark images, since Browser Use and Playwright-style agentic work needs vision.
• Cam’s honest blocker: real multi-turn, agentic, multimodal datasets “kinda don’t exist” publicly because they’re too valuable for training — “the best one online right now is WildChat, and that’s three years old… people asking GPT-4o to be their therapist.” The plan: spin up frontier models in OpenCode instances, simulate coding environments, and capture the traces — a repeatable internal process, not a one-off golden dataset.

7. Coda: the 180 on Codex, the human bottleneck, and Cam’s plateau fear

• The mind-change Doug wants on record (against Rune’s Twitter dragging): Cam hit an FFmpeg problem Opus couldn’t solve, switched to Codex 5.3, and “within 15 minutes I was like, dude, this is better” — “Codex is better for targeted issues and Opus is better for general vibes.” Doug’s meta-point: the jagged frontier — one-shot targeted asks versus build-the-whole-project — is “the most interesting problem” in agentic benchmarking, and “it’s a multipolar world.”
• On whether humans are now the bottleneck: Doug says that for most simple things he feels he is the bottleneck; he says the models now want to code and he just needs to prompt them, while Cam hedges: if a model “goes down the wrong path, bro, it’s over.”
• Cam’s bear moment: every CEO podcast assumes models improve “infinitely forever… what if that just isn’t the case? Then we’re fucked, man.” Doug’s answer: strong beliefs weakly held, and even a plateau is “enough to change our lives forever.” Cam’s close is the show’s thesis: “maybe models aren’t getting smarter… but they are getting cheaper, faster, and more performant” — so diffusion widens either way.