Nvidia CTO Michael Kagan: Scaling Beyond Moore's Law to Million-GPU Clusters

Summary

• Kagan’s central claim is that AI has pushed scaling beyond chip-level Moore’s Law toward system-level scaling: model size and capacity were rising 2X every three months, demanding roughly 10X–16X annual performance. NVIDIA therefore optimizes the “single unit of computing”—compute, networking, software and the full data-center-scale machine—not merely transistor density. Kagan says annual product waves now deliver roughly an order-of-magnitude gain at the machine level.
• Mellanox enabled NVIDIA’s scaling beyond a single node by extending the scale-up fabric and supplying the high-performance network needed to scale out. Today’s “GPU” can be a rack-size machine—72 GPUs across 36 computers, presented through CUDA as one programmable system—and “you need a forklift to lift it.” Cluster performance depends less on headline bandwidth than on tightly controlled latency: wide jitter forces every GPU to wait.
• At 100,000 GPUs, component reliability stops meaning the system is reliable because millions of parts make the chance that everything works “zero.” Hardware and software must assume something is always broken while preserving service, utilization and power efficiency. Kagan’s broader answer also includes BlueField DPUs, which separate infrastructure computing from applications and significantly reduce the attack surface.
• Kagan argues that inference compute demand is already “not less than training”—and is actually greater—as AI moves from one-shot perception to generation and multi-step reasoning. Training happens once, while inference happens repeatedly for every user and token. Prefill is compute-intensive, decode is memory-intensive, and NVIDIA’s bet is that specialized SKUs should retain a common CUDA programming model so capacity can shift between them.
• Energy and heat, more than an abstract networking law, are the practical limits on data-center size. Kagan cited large deployments around 100–150 megawatts, discussion moving toward gigawatt and even 10-gigawatt facilities, and liquid cooling as essential to higher density. Splitting one workload across distant sites then introduces speed-of-light latency and congestion, which Spectrum-X addresses with endpoint telemetry rather than jitter-inducing “huge buffers.”
• The Intel partnership reflects NVIDIA’s view that accelerated computing will expand rather than eliminate general-purpose computing. Kagan described a “culture of win-win”: NVIDIA is “after baking a bigger pie for everybody,” with x86 and accelerated computing opening channels and markets for both companies. The Mellanox precedent is striking—against the hosts’ framing of $7 billion paid, 45X value growth in six years and roughly 85%–90% employee retention.
• Kagan puts the current performance slope at “10X or a few orders of magnitude a year,” while explicitly admitting, “How long it will stay, I don’t know.” His long-term upside is broader than productivity: Earth-2 could make history “experimental science,” AI might infer physical laws humans have not imagined, and a technology that acts as the “spaceship of the mind” could expand desired work faster than it expands available resources.

Deep dive

1. Mellanox moved the unit of compute from chip to data center

• Kagan’s starting point is a mismatch of exponentials: Moore’s Law delivered roughly 2X every other year, while AI models began doubling in size and capacity every three months. That translates into a requirement for 10X–16X annual performance growth, which pushes optimization beyond a basic component.

• Scale-up now means using NVLink to turn 72 GPUs across 36 two-GPU computers into one system under the same software interface. What customers call a GPU is therefore “a rack-size machine” rather than a chip: “You need a forklift to lift it.”

• Scale-out divides an application among many such machines. Kagan’s example: split a one-second task across 1,000 GPUs and it might finish in one millisecond—but only if distributing the work and consolidating results do not expose communication time.

• Mellanox’s networking contribution is latency consistency, not merely “hero numbers” for peak bandwidth. If network jitter cannot hide behind computation, every worker waits and a job that could span 1,000 GPUs may efficiently use only 10. “Inherently, network determines the performance of this cluster.”

2. At 100,000 GPUs, failure, distance and heat become architectural inputs

• A component may work 99.999-whatever percent of the time, but a 100,000-GPU machine contains millions of components; Kagan’s blunt conclusion is that “the chance that everything works is zero.” Hardware and software must sustain service, performance and power efficiency while something is definitely broken.

• Ordinary data-center networks connect loosely coupled microservices; an AI compute network may run “one single application on 100,000 machines.” Schedulers consequently need hardware and low-level software hooks that place every part of a job efficiently.

• Crossing data centers adds speed-of-light delay and radically different latency distributions. Kagan rejects old telecom-style buffering—“bigger is not better,” because buffers create jitter—and says Spectrum-X supplies telemetry so endpoints can distinguish short from long communication and adjust around congestion.

• Pat’s pushback—if nuclear plants supplied abundant energy, would the data center itself still scale?—drew an honest partial answer: “I don’t know,” because incoming power still becomes outgoing heat. NVIDIA has moved largely to liquid cooling; current 100–150-megawatt-scale facilities sit alongside discussion of gigawatt and proposed 10-gigawatt sites, though deployment can still depend on “how fast concrete gets stable.”

3. Inference is becoming the larger and more varied compute market

• Training combines forward inference, backward propagation and, under data parallelism, consolidation of weight updates across model copies. It dominated when AI was mostly perceptual—identify a dog or recognize a person—but generative AI reruns inference for every token, and reasoning can involve comparing multiple possible paths.

• Inference itself splits into two unlike workloads. Prefill processes prompts and background material into context and is compute-intensive; decode generates the answer token by token and is more memory-intensive, even where techniques can produce more than one token per pass.

• Kagan’s conclusion: inference demand is “actually even more” than training because each response now requires much more work and “you train a model once, but you infer many times.” Inference can run on phones or smaller installations, while data centers can mix prefill- and decode-optimized GPU SKUs; common CUDA programmability lets either SKU absorb shifts in the other workload.

4. NVIDIA’s next expansion joins accelerated and general-purpose computing

• Kagan’s case for acceleration begins with the limits of explicit programming: a von Neumann machine can follow instructions, but “I can’t explain how to distinguish between a cat and a dog.” AI solves that different class of problem, yet general-purpose computing and dominant x86 workloads are not going away.

• The Intel partnership therefore “fuses accelerated computing with general-purpose computing.” Kagan frames the commercial logic as ecosystem expansion: NVIDIA is not seeking a larger slice of the existing market but “baking a bigger pie for everybody,” opening customers and channels that were harder for the companies to serve alone.

• Pat noted that NVIDIA and Mellanox were worth about $100 billion together in 2019 versus roughly $4.5 trillion six years later. Kagan had told Jensen that “one plus one will be 10”; he acknowledged that he had been “off by a factor of four.”

• The integration evidence was organizational as well as financial: Kagan estimates 85%–90% of the original Mellanox employees stayed, NVIDIA more than doubled its Israel headcount, and a new campus is planned. BlueField DPUs also broadened Mellanox’s role by moving the data-center operating system and infrastructure workload away from application processors, maximizing application computing and reducing the attack surface.

5. Kagan sees exponentials turning AI into scientific infrastructure

• Kagan’s science-fiction ambition is “how to make history experimental science.” Earth-2 could simulate how today’s actions affect global warming 50 years later; more broadly, AI’s ability to observe and generalize could help reveal laws of physics “that we don’t even imagine now.”

• Asked for a successor to Moore’s Law, Kagan offered “10X or a few orders of magnitude a year.” NVIDIA accelerated product introductions from every other year to yearly two or three years ago, delivering order-of-magnitude gains in the machine customers can build—not merely in an individual chip—but he explicitly hedged the duration: “I don’t know.”

• His optimistic analogy moves from Steve Jobs’s computer as the “bicycle of the mind” to AI as the “spaceship of the mind.” Greater efficiency will not satisfy fixed demand: give a project leader 2X the resources and they may do 4X more while wanting 10X. Like electricity, AI could become indispensable infrastructure whose applications cannot be predicted from the platform’s beginnings.