Agent Inference at the "Speed of Light" — How NVIDIA moves like a $4.3 Trillion Startup

Summary

• NVIDIA’s Dynamo thesis is that agent inference must be optimized as a data-center-scale system, not a collection of independent model replicas. Sitting above vLLM, SGLang, and TensorRT-LLM, Dynamo coordinates KV-cache-aware scaling, disaggregated prefill and decode, and scale-out scheduling. The payoff comes from economies of scale across many machines.

• The governing constraint is a three-way frontier among quality, cost, and latency, with the entire agent workflow—not merely the model—as the unit of optimization. More reasoning, retries, tools, and multi-turn context can lift quality, but they consume more tokens and extend latency. Operators therefore start with the required model and SLA, test parallelism and deployment configurations, and find “the lowest-cost version that hits all of these constraints.”

• Separating prefill from decode creates both an immediate efficiency lever and a path to specialized hardware. Prefill is generally compute-bound for sufficiently long sequences; decode is typically memory-bandwidth-bound, and mixing them can let long prefill steps block token generation. Dynamo puts them in independently scalable pools, while examples ranging from DGX Spark plus Mac experiments to the announced Rubin CPX suggest increasingly heterogeneous inference fleets.

• Model architecture is becoming inseparable from inference hardware and expected context. Kimi K2’s designers traded fewer attention heads for more, sparser experts, preserving activated-expert count while reducing attention work. The discussion framed this as hardware-model-context-code co-design. DeepSeek’s MLA was cited as shrinking a roughly 128,000-token context to about 8 GB, versus an estimated 40–80 GB for a similarly sized Llama 405B context in the same precision.

• Agents turn one model call into a structured system of models, caches, tools, and subagents. A “system as model” may hide many specialized models behind one API response. One participant expects an agent capable of running coherently for more than 24 hours before year-end, although current production autonomy discussed was closer to 20–45 minutes and another participant expects efficiency to improve before runtimes expand indiscriminately.

• The security rule offered is stark: an agent should normally receive only two of file access, internet access, and arbitrary code execution. NVIDIA’s internal teams are trying to enable widespread Codex, Claude Code, and OpenClaw experimentation while preserving explicit enforcement points; Brev’s isolated cloud VMs are one answer for running risky agents off the corporate network.

• Brev is becoming NVIDIA’s developer-facing bridge from local hardware to cloud GPUs and one-click software. Its original insight was to make “A100” the largest text on the page rather than burying it in a cloud instance code; Launchables apply the same logic to software deployment. DGX Spark can be registered with Brev in early access and remotely used from a laptop “just like any other cloud node,” turning the home box into a “tiny little data center.”

• NVIDIA’s execution culture combines “speed of light” deadlines with long-horizon investment in markets that may initially be worth zero. “SOL” asks for the physical lower bound before organizational constraints are layered back in, but stability and security remain real inputs—not excuses to “build everything and let it break.” At scale, the hardware economics can be dramatic: the discussion cited GB200 NVL72 serving as roughly 35 times cheaper per token than Hopper over much of one comparison curve.

Deep dive

1. Brev turned GPU provisioning into a developer product

• Nader Khalil’s original Brev premise was deliberately narrow: connect multiple GPU sources and minimize the time until a developer can SSH into one. Cloud consoles buried the requested “A100” inside obscure instance codes and multi-page forms; Brev made the GPU type the dominant object on the page.

• Even the animated GPU cards carried the product philosophy. Nader designed them in Figma, then hand-built the effect by changing SVG styles in React—“artisan” work that swyx initially questioned as a small investor, but later saw as evidence of the care running through Brev.

• The memorable GTC booth applied the same thinking to distribution. With the last booth in a distant corner, Brev installed surfboards and palm trees tall enough to rise above neighboring walls; the booth stood out and was remembered by the NVIDIA team.

• Launchables extended provisioning into one-click deployment of GPU software. After NVIDIA acquired Brev, Nader said the decisive fit was that “the soul of the products align”: Brev gained resources without abandoning its goal of making GPU access easier, and brev.nvidia.com became what the hosts called “a front page for GPUs.”

2. DGX Spark exposes the widening gap between GPU power and GPU usability

• The new audience includes OpenClaw hackers who may never have used a GPU. Nader’s broader claim is that AI is creating “a more technologically literate society,” but that expansion forces NVIDIA to build several new layers of developer UX.

• His first DGX Spark meeting clarified the problem. One proposed first use was buying two boxes and forming a Kubernetes cluster; Nader’s response was that the actual first task should be effortless SSH, because the Kubernetes user “has a higher propensity for pain” than someone who simply wants to run an agent.

• NVIDIA Sync wraps the SSH connection, while Brev’s early-access integration lets an owner register a Spark, leave it at home, and access it remotely from a café like any other Brev node. The home system becomes a “pre-provisioned data center,” with Brev spanning local and cloud compute through one interface.

3. “Speed of light” strips delivery dates down to their physical limits

• Nader’s favorite NVIDIA operating lesson is “SOL”: before accepting a three-week estimate, ask what physics makes three weeks unavoidable. “The speed of light moves at a certain speed, so if something’s moving slower, then you know something’s in the way.”

• His laptop analogy makes the doctrine concrete. If ordinary delivery takes five days, the SOL is someone walking into Best Buy and buying the machine immediately; only afterward should the team add the practical constraints of larger-scale procurement.

• A host’s pushback—worth keeping—was that perpetual SOL demands can become annoying when teams need stability and planning. Nader agreed: for CES, remote Spark registration could reach early access and appear in Jensen’s keynote, while unresolved networking and security work explicitly prevented treating it as finished.

• The phrase also came from hardware: an accelerator has a theoretical full-speed limit, followed by the practically achievable fraction, such as model FLOPS utilization. SOL is therefore not permission to ignore maintenance; it identifies “the minimum thing that we can get to,” then repeats the exercise for each incremental component.

4. NVIDIA treats organizational fluidity as a research advantage

• Kyle joined NVIDIA from college and moved from autonomous-vehicle object detection into GPU-accelerated recommender systems, then graph neural networks. Recommenders were becoming always-on infrastructure for exploding feeds, while graphs let him represent relationships among people, items, and concepts.

• NVIDIA encourages people to follow their interests: someone with an initiative can email people higher in the organization who may find it relevant and ask to work on it. A complementary rule is that “momentum is the only authority”—make something tangible, let colleagues try it, and use demonstrated progress to attract support.

• Communication mirrors that looseness. Email threads can become 60-person “mosh pits,” but they can preserve working context, let important threads resurface, add recipients freely, and fork as the work branches.

• Kyle described NVIDIA’s willingness to invest in a “zero billion dollar business”: an area may produce no revenue today yet still matter for understanding a future market. When a host noted that cars are hardly a zero-billion-dollar market, the distinction was clarified as an internal organization not needing immediate revenue to justify important research.

5. Dynamo treats the data center as the inference engine

• Kyle’s path eventually returned to recommenders from the opposite direction: Amazon Ads described using Dynamo for generative recommendation, so LLM inference was “supplanting what I was working on” five years earlier. That convergence illustrates why inference has moved from a niche systems topic to highway billboards.

• Individual engines such as vLLM, SGLang, and TensorRT-LLM primarily optimize a single copy or replica of a model. Dynamo sits above them as a modular, data-center-scale inference runtime, coordinating scale-out and exploiting shared-system opportunities such as KV-cache maximization and disaggregation.

• Scale-up eventually meets hardware and algorithmic boundaries. On most DGX H100 systems, the fastest NVLink domain spans eight GPUs; beyond it, communication moves to InfiniBand. Kyle estimated roughly 500 GB/s unidirectional for NVLink versus about 50 GB/s for InfiniBand, while cautioning that exact figures depend on generation.

• Scale-out is therefore unavoidable, but naïvely duplicating replicas leaves performance on the table. Dynamo’s claim is that the collection of machines contains unique cached state and shifting workloads that should be scheduled as one inference system rather than treated as interchangeable boxes.

6. Quality, cost, and latency define every deployment’s feasible frontier

• Kyle reduces inference viability to three axes: can the workflow complete the task at sufficient quality, can all its multi-turn calls be served cheaply enough, and can it meet the latency SLA? Dynamo supplies knobs for moving around that “Pareto surface,” but it cannot erase the trade-offs.

• The workflow now matters as much as the base model. A smaller model with more test-time scaling, retries, tools, or a stronger harness might match a larger model’s quality, yet consume more calls and tokens; faster model variants and adjustable thinking budgets make the same exchange explicit.

• Once the model and SLA are chosen, operators experiment with tensor-parallel size and other common configurations to find the cheapest feasible system. There is no universal recipe: expert counts, sparsity, serving layouts, and workload distributions often reflect extensive tinkering rather than a theoretically ordained answer.

• The hosts’ “just try again” example captures the demand side. One cited paper reportedly found roughly double the success rate when the model received the failed attempt as context and tried again. Self-distillation or reinforcement learning can exploit past failures more deeply; better inference may therefore mean intentionally buying multiple attempts rather than demanding perfection from one call.

7. Disaggregated prefill and decode remove a hidden scheduling conflict

• Conventional inference engines alternate between prefill—reading the prompt and producing KV cache—and decode, which uses that cache to generate output tokens. Because scheduling proceeds in steps, a long prefill operation can prevent the engine from starting the next decode step, hurting interactive generation.

• The phases also stress hardware differently. Prefill is generally compute-bound once the input is sufficiently long because it makes a full pass over the weights and has quadratic attention work; decode repeatedly retrieves a linear amount of memory and is typically constrained by memory bandwidth.

• Disaggregation assigns the phases to separate worker pools, eliminating their step-synchronous interference and letting each pool use a suitable configuration. The discussion noted that the idea appeared in academic work before Dynamo, while Dynamo was among the first frameworks to support it as a systems primitive.

• The hardware implications are already visible. Exo Labs demonstrated compute-heavy prefill on DGX Spark and decode on a Mac, while NVIDIA announced the prefill-specific Rubin CPX accelerator; the participants declined to speculate about using incoming Groq technology for very fast decode.

8. Grove lets prefill and decode capacity move independently

• Static pairing is inefficient because workloads do not preserve one prefill-to-decode ratio. A sudden influx of extremely long prompts pushes prefill sharply upward, while changes in output length, context, and request mix create a different scaling curve for decode.

• Dynamo includes a Kubernetes component called Grove for representing these multi-stage, multi-node deployments. The discussion contrasted it with LeaderWorkerSet: the latter serves many distributed jobs well but struggles when paired stages need a ratio that changes dynamically rather than a fixed replica shape.

• Dynamo can estimate how many prefill and decode workers a workload needs, then expose the scheduling machinery to realize that layout on the available infrastructure. Decode scales with both sides of the interaction: more output means more generation steps, while longer accumulated context raises the work performed at each step.

9. Kimi K2 shows architecture becoming a hardware allocation decision

• The discussion’s clearest co-design example was Kimi K2, which was described as DeepSeek-like, using MLA but scaling components differently. Its designers increased the number of experts while keeping activated experts constant, making the mixture sparser, and reduced attention heads to cut attention work.

• The trade reallocates resources rather than simply shrinking the model: more experts consume additional memory capacity, while fewer attention heads reduce computation that still scales quadratically with context. The expected workload and target hardware therefore influence which form of capacity the architecture buys.

• swyx’s framing was that recent labs are moving beyond a single “sparsity ratio.” Designers can vary total versus active experts, attention-head count, and the number of attention layers; the broader problem was called “hardware-model-context-code co-design,” because short- and long-context workloads favor different architectural bargains.

• Nemotron was offered as another co-design specimen: a hybrid state-space model benefited from extensive post-training, while NVIDIA released models, datasets, and recipes in layers. ServiceNow used the dataset for its own model, illustrating the stated goal of letting others reuse either the trained artifact or the process.

10. Harnesses are becoming part of the trained model

• The context produced by an agent harness should be treated as part of the model system and, where possible, “internalized” during training. If deployment always uses a particular structure and tool vocabulary, training against that environment should improve the resulting quality.

• A host’s counterargument was that broadly useful models must plug into many harnesses, search tools, and differently named functions. The discussion acknowledged the friction: users may have to mimic the original tool interface, undo some tool-specific post-training, or retrain around their own harness—“it’s really annoying.”

• The more optimistic endpoint is a model that can understand new tools without bespoke retraining, but the present limitation was not framed as AGI: “It’s a really stupid ‘learn my tool’” problem. Bash was described as a universal harness.

• More broadly, “system as model” means that a model call may conceal many models and components working together. Dynamo and other NVIDIA libraries are intended to help manage that complexity.

11. Context growth may depend on discontinuous “unhobblings”

• One host challenged the industry’s context trajectory directly: frontier models have hovered near one million tokens for roughly two years, and moving to two million does not lead plausibly to 100 million. “This doesn’t work” was the skeptical reading of the current scaling-law slope.

• The answer drew on Leopold Aschenbrenner’s “unhobbling”: scientific or architectural discoveries that release an otherwise blocked order of magnitude. Native multi-token prediction, which DeepSeek said improved training stability, is a smaller example; new attention mechanisms can create much larger changes in serving feasibility.

• MLA provides the numerical specimen. The discussion estimated that DeepSeek’s roughly 128,000-token context—possibly 256,000 with RoPE extension—fits in about 8 GB, whereas a similarly sized Llama 405B context previously required roughly 40–80 GB in the same precision.

• One speculative unhobbling would prefill unrelated documents locally in independent chunks, then decode with global attention across the full sequence. That could remove prefill’s global quadratic cost, but the idea was explicitly uncertain: it had not been demonstrated and “could be a tarpit” that simply fails.

12. NVIDIA’s agent rollout makes security an architecture problem

• NVIDIA has a large deployment of Codex, with “tens of thousands” of people, and employees also use tools including Cursor and Claude Code. Useful tools spread “like wildfire” through the company’s fluid email network, sometimes followed by Jensen directing teams to make them work company-wide immediately.

• An internal Outlook CLI became one participant’s proof point. Codex was asked to summarize every email, flag escalations, move likely replies into a folder, and archive everything else; after it worked, the tool was emailed to roughly 500 colleagues and others were helped to install it.

• The security rule offered is that agents have three powerful capabilities—file access, internet access, and arbitrary code execution—and “you should really only let an agent do two of those three things.” All three together expand the path for injected malware or unintended data movement.

• NVIDIA’s security team was described as progressive because it tries to enable the technology while defining enforcement points. For OpenClaw, the internal guidance was to run it in an isolated Brev cloud VM off the corporate network while the company works out stronger controls.

13. CLIs are emerging as the agent-native interface to enterprise software

• NVIDIA is turning business applications including Outlook and SAP into command-line tools, with an aspiration to open-source the work and perhaps seed an “Open CLI Foundation.” The updated developer-tool rule is no longer merely “make your docs LLM-accessible”; every tool should have strong CLI support.

• The historical irony is the best framing: computing began in the shell, then graphical interfaces made machines empathetic to humans; now LLMs are navigating those interfaces, and software is “not empathetic to the machine anymore.” Giving an agent a terminal restores a concise, composable control surface.

• The discussion cited portability, locality, discoverability, and the large volume of command-line examples in pretraining as reasons to use CLIs. It also argued that a CLI predefines allowable network calls and vulnerability boundaries instead of having a model improvise arbitrary Python requests.

• NVIDIA is already applying the pattern to inference engineering. Agents can request compute, launch Dynamo experiments, test configurations, and return points on the Pareto frontier; one agent “completely one-shot” a configuration problem that previously required specialists to conduct iterative experiments manually.

14. Agent systems create insatiable compute demand—but not uniformly

• Local agents expose a behavioral mismatch with compute economics. One participant’s 24/7 agent kept Runpod instances warm despite instructions to shut them down, treating 45 minutes of human time as a few minutes of agent time; a local dual-GPU server consequently became “a little space heater.”

• Local hardware offers control but not data-center economics. The discussion cited a roughly $8,000 Blackwell RTX PRO 6000 with 96 GB of VRAM as a capable local option, yet emphasized that centralized scale enables optimizations such as wide expert parallelism; GB200 NVL72 was about 35 times cheaper per token than Hopper across much of one cited curve.

• The discussion’s larger theme is “system as model”: one apparent API call may orchestrate many models, components, and subagents behind the scenes. NVIDIA’s DGX Spark model router chooses between a local model and a foundation model, replacing the either-or decision with “the best of everything that’s available.”

• Runtime expectations remain contested. One participant predicts coherent agents lasting more than 24 hours before year-end, while production autonomy discussed was around 20–45 minutes. Another participant expects the trade-off to be domain-specific: consumer users may not want 20-minute responses, while GPU-kernel, medical, and biological research may justify much longer runs; efficiency may improve before systems lengthen again.