Ep. 033 - ClusterMAX 3.0 Is Here! Neoclouds Ranked (Neoclouds, GPUs)

Summary

• ClusterMAX 3.0 lands with a real shakeup: Nebius joins CoreWeave in the Platinum tier, Google Cloud joins Oracle in Gold, Azure slips to Silver, and AWS, Crusoe, and Together all fall into Bronze. SemiAnalysis ranked 77 providers this cycle, tracks 323 in its neocloud model, and has now interviewed over 200 end users — depth that makes the tier moves more than vibes.
• The GPU market is in “ridiculous backwardation”: buyers will pay almost anything for chips deliverable in weeks, while waiting a few months earns a big discount. Sam Harshe’s warning is that this flatters everyone — “just about everyone in the industry looks like a genius right now… even if your fivecoded Kubernetes is completely broken” — meaning low-quality providers who couldn’t sell early are now closing deals at better margins.
• Inference endpoints are quietly a cost minefield: in preliminary Endpoint X testing, one “very famous provider” delivered a 75% cache hit rate versus 99% for good peers — and that gap alone roughly doubled the total bill for the identical model and accuracy. Pratt Bhatt’s takeaway: you cannot “just go out, install vLLM… and assume to make 80% margins”; cache-aware routing and custom kernels are where the real endpoint business lives.
• Networking, not FLOPS, is what separates clouds — the ask is a “golden recipe” for NCCL that gives reasonable performance out of the box. Google won praise for baking its non-NVIDIA network into NGC containers, while AWS’s custom EFA fails to support DeepEP-style collectives built below NCCL: “the money that they’ve saved by building their own custom scale-out is being paid instead to engineers” fighting a fabric “nobody actually wants to use.”
• The financing backdrop is the reason ClusterMAX matters: NVIDIA’s off-balance-sheet backstop universe is over $588B at the end of FY27 and forecast to exceed $2T by end-2031, with Jordan cautiously citing ~$827B in FY31 for the supply chain (HBM etc.) alone. Sam Harshe frames the moment as “more money than good investment opportunities, except the money is still concerned about quality” — hence parametric insurance, standardized SLAs, and SemiAnalysis’s new credit and markets model.
• Old-GPU terminal value is resolving bullishly, against Dr. Burry’s depreciation criticism: four-year H100 contracts are being signed today, and “OpenAI has never given up an A100” — chips from 2020. The mechanism is physical: GB300 racks run 140–200kW versus 30–40kW legacy facilities that can’t host Vera Rubin, so those sites will likely keep running H100s “so long as they can be turned on.”
• Security is a “tinderbox”: two-year-old CVEs on cluster software, no tenant isolation, visible neighbor Grafana dashboards — and Pratt Bhatt rejects the sanitized framing of the incident: “It was the Hugging Face cyber attack by OpenAI,” built on “almost inexplicably mundane misconfigurations.” The lesson is that defense doesn’t require frontier AI, just basics: patched kernels, Kubernetes permissions, and alerting on agents’ internet access.
• Hosted RL training is the next product wave but Sam’s honest call is “I still don’t think there’s a market for it yet.” RL combines the hard parts of inference and training plus environments (literal “server farms just running environments,” including phone hardware), and hosted-training providers would need forward-deployed engineers and effectively guarantee model outcomes — a commitment CoreWeave and Nebius never had to make.

Deep dive

1. The 3.0 tier moves: Nebius ascends, hyperscalers scatter

• Headline changes as Jordan runs them: Nebius joins CoreWeave in Platinum; Gold is Oracle plus newly promoted Google Cloud; Silver has Azure moving down, Firmus and Lambda holding, GMI moving up, TensorWave staying; Bronze catches AWS, Crusoe, and Together, all moving down.
• The scale behind the rankings: 77 providers tested this round, market coverage expanded to 323 in the neocloud model, and “well over 200” end-user interviews — plus teased spin-offs Endpoint X, Post-Training X, Harness X, and Sandbox X. “We’re going to test everything.”

2. A backwardated market where bad providers look brilliant

• Sam’s read on what buyers care about: the price curve first. “People are willing to pay just about anything for any chips that they can get in the next few weeks,” with big discounts for waiting months — so “even if storage barely works, if you’ve got GPUs, you’re going to find people willing to pay a pretty good margin.” Jordan’s twist: providers who couldn’t sell early — usually a quality signal — are now closing at better margins.
• Pratt’s explanation for why clouds that can’t get Slurm right are launching four new products: compute margins are so high that everyone wants the labs’ “cake,” so they climb the stack into hosted training and managed inference. His through-line: “everything points to centralization of compute.”
• Sam’s counter-emphasis — there’s still money on the table in the boring layer: no fancy bells needed, “plenty of work to do just to get Slurm and Kubernetes bulletproof,” and a few extra cents per GPU-hour for “picking one lane and making it great.”

3. Health checks that are “worse than no health checks”

• Methodology in brief: active checks run an actual or dummy workload when the cluster is otherwise idle, normally preemptibly; passive checks watch in the background at near-zero overhead. SemiAnalysis injects synthetic XID failures — writing into the kernel ring buffer the exact message a real GPU hardware failure would emit — then times detection, drain, and hot-spare replacement for HGX machines (or reboot/repair on NVL72 racks).
• The best specimen of failure: on Amazon HyperPod Slurm’s first test, an auto-remediation health check “needed the node to be healthy before it could run” — yet was required to bring the node back into the fleet. “There was an obvious contradiction… the health check just never had a chance,” and Sam estimates probably five to ten such cases across testing.
• Why it matters commercially: customer recourse for paid-but-unusable GPUs depends on signed SLA credits that define downtime — which is why SemiAnalysis is circulating sample SLAs defining downtime, node vs. rack vs. site, and force majeure as an industry standard.

4. Endpoint economics: a 75% cache hit rate doubles your bill

• Pratt’s preliminary Endpoint X finding, ahead of a “hero run” testing concurrencies up to 24 or more: one very famous provider (unnamed) hit a 75% cache rate on roughly 800-token replay traces versus 99% for good providers — and that alone “almost doubles the total cost… if you’re paying $200, you’ll be paying $400 for the same exact model, for the same exact accuracy,” at probably slower speeds. “It’s quite shocking.”
• The deeper point, seconded by both: open-source vLLM/SGLang “runs pretty fine” until you scale to real customers, which is why kernel and megakernel startups exist. Neoclouds assuming an endpoint is “like a lambda function” with 80% margins are in for the same shock as their customers.

5. Networking is the differentiator — and custom fabrics are a false economy

• Sam’s simplest advice to providers: ship a “golden recipe” for NCCL that yields reasonable performance out of the box — sweep message sizes and node counts, expect a smooth logistic curve, no “pathological message sizes” dipping at 128KB. “I can tune that to my heart’s content… but something waiting for me on the cluster that I can hit enter” is the bar.
• The Google vs. AWS contrast: Google built its non-NVIDIA-silicon network into NGC containers so NCCL just works; EFA, meanwhile, is easier than last test but completely unsupported by DeepEP- or Mooncake-style collective libraries built on a lower-level layer than NCCL, NVSHMEM. Sam tentatively estimated that something like 75–80% of a large cluster’s BOM is for things other than networking, so saving a few percent while blocking “the DeepSeek communications recipe” is “absolutely not worth it.”
• The live case study: Firmus’s GB300 testing exposed communication time on a communications-bound TorchTitan expert-parallel run. Jordan cited 4,455 tokens/sec/GPU for Firmus and around 3,500 for others in some testing, while also characterizing Firmus as “way less performance… like 50% less”; the raw figures and that comparison are retained without resolving the discrepancy. Jordan’s caveat: “This is not a bad provider” — thousands of GB300s stood up, fix delivered after joint testing — but battle scars are the point: “Who are you going to trust for your Vera Rubin next year?”

6. Agentic coding: superficially useful, reliably foot-blowing

• Testing itself leads with the open-source CMAX CLI, with agents wrapped around it for debugging. The model banter is a time capsule: Sam is “a 5.6 guy,” finds Aider “just so badly undercooked,” gets more going with 5.6, and says “certainly not Claude” — the word on the street is that Claude’s writing was fixed, but he’ll believe it when he sees it.
• Sam’s signature framing: cluster orchestration is “the perfect sweet spot of close enough for the agents to be superficially useful and far enough that they’ll just blow your foot off every single time” — there are good Slurm patterns in the training data and plenty that don’t fit, so an agent can rearrange your orchestration “better than you could have done on your own… but it will do it incorrectly.” Sam’s example of false confidence: checking the computer’s projected endpoint cost returned $300,000–$400,000.

7. The best bug of the season: a dirty-inode deadlock that ate a node

• Sam’s story: an unprompted node-down email traced to an old Linux kernel (bugs fixed years ago upstream) with a quadratic-time algorithm reassigning dirty inodes as a Slurm job tore down. It locked every CPU core on one NUMA node, starving the Lustre client of CPU time, tripping a storage health check that itself couldn’t complete. Hours later, “as soon as it returned to healthy, it was like the cops were waiting outside its door” — the node was taken from the fleet for a filesystem health-check failure caused by a CPU problem, not a GPU one.
• The meta-joke that explains the job: ClusterMAX is the anti-performance-engineer role — “here, we want bugs. Otherwise we have nothing to do… if I met an actual perf engineer and shook hands with him, we would both disappear.”

8. Security: mundane misconfigurations, not sci-fi threats

• Pratt’s testing wrinkle: frontier models both refuse exploit work and manufacture false positives — one claimed host access to a cluster control plane, then after two hours of debugging admitted the PoC was fake. His distillation: “Are hallucinations gone? Most likely yes. But are false positives gone? Definitely not.”
• Pratt’s critique, delivered on the record: “I need to stop using the euphemism supplied by OpenAI’s lawyers… It was the Hugging Face cyber attack by OpenAI” — dozens of “almost inexplicably mundane misconfigurations”: an applicable kernel CVE used for privilege escalation, sloppy Kubernetes permissions, and no alerting on agents’ unmitigated internet access. Providers don’t get the “brave new world” excuse.
• The neocloud state of play matches: two-year-old CVEs on basic cluster software, tenant isolation gaps, neighbors’ Grafana visible, CUDA Toolkit 12.0 when 13.2 — “or whatever it is” — is the most recent version, and “a tinderbox for tools that are only getting more powerful.” Few neoclouds are participating in bug-bounty programs; Jordan cites OpenAI being hacked as part of a bug-bounty program, with the HackerOne AI researchers opening PR 1,000,001 on a monorepo to prove full access.

9. Financing, backstops, and why old GPUs won’t die

• Sam’s macro frame: “there is more money than good investment opportunities, except the money is still concerned about quality.” The two insurable risks are late delivery (failing acceptance) and SLA violations triggering cancellation rights — hence parametric insurers underwriting cluster risk with actuarial inputs (redundant power/cooling, parts depots, staffing, flood plains) and SemiAnalysis’s sample SLAs. The stakes: startups spending “literally 100% of their seed round” on neocloud capacity.
• The numbers behind the new credit and markets model: NVIDIA’s off-balance-sheet backstop universe exceeds $588B at the end of FY27, forecast above $2T by end-2031; Jordan cautiously adds ~$827B projected in FY31 for the supply chain alone — HBM and the like, excluding data-center buildout. “It’s quite insane.” And it’s not just NVIDIA — Amazon and Google are deploying balance sheets the same way.
• On terminal value, the tape disagrees with the depreciation bears: four-year H100 contracts signed today, and OpenAI has never given up an A100 it received in 2020. The physical lock-in: GB300 racks at 140kW, approaching 200kW peak, can’t fit 30–40kW legacy facilities, which can’t accept Vera Rubin — so they will likely keep running the H100s that are so valuable until those contracts expire.

10. Hosted RL training: harder than clusters, and maybe premature

• Sam’s hedged call, kept as hedged: “I still don’t think there’s a market for it yet. I said yet.” Pratt’s structural point on why RL is harder than its size suggests: it combines the hard parts of inference and training, synchronized, plus environments to deploy — one evaluation and RL-environment lab runs “server farms just running environments,” including iPhone and Android hardware.
• Sam’s hot take: hosted training only gets big if the provider handles literally everything — forward-deployed engineers (“FDRs, not FTEs”) building environments and hand-holding customers through training. That’s a commitment pre-training clouds never made: CoreWeave and Nebius never had to guarantee that the model being trained would work. Supporting evidence as told: “Anthropic bought the most environments last year — that’s why their coding model was so damn good.”
• On framework wars (Slime, MILES, veRL, PRIME-RL): MILES is “absolutely amazing… very agent-friendly,” PRIME-RL was pleasant from Pratt’s GPU-poor days, but trainer-inference mismatch can compound over 100 steps and collapse training — building these frameworks is “way harder than people think.” Sam’s closing wish is the whole thesis in one line: “I want nothing to write about in ClusterMAX 4.0… the better the cluster, the less there is to say.”

Verification Notes

• The Firmus passage cites 4,455 tokens/sec/GPU for Firmus versus roughly 3,500 for others, while also calling Firmus “like 50% less”; the transcript does not resolve that internal comparison, so no direction is inferred beyond Jordan’s stated characterization.