The Future of AI Infra: from Kubernetes to Agent Sandboxes — Akshat Bubna, Modal CTO
Summary
Modal’s strategic reframe is that infrastructure UX should serve agents first. The team says the same benefits that apply to developer experience apply to agent experience: decorators let an agent change configuration beside the code and immediately test a “self-provisioning runtime,” instead of parsing hundreds of Kubernetes files and generating YAML. The emerging control surface is CLI-accessible logs and metrics: agents investigate, while humans interpret behavior and make judgment calls.
Modal is betting its moat lies above GPU ownership, as a capital-light “super cloud of all clouds” spanning 17 providers. That footprint supplies regions, GPU types, and specialized capacity without Modal building data centers; its software absorbs unreliable underlying infrastructure so “if the GPU falls off the bus,” workloads continue. The trade-off is explicit: remain cloud-only and concentrate differentiation in scheduling, elasticity, locality, and reliability.
The first product-market fit was elastic inference for custom models, not generic LLM APIs. Suno, Runway, robotics, and computational-biology companies brought models trained elsewhere, then relied on Modal as the “best black box for deployment” across unpredictable launches, multiple models, and offset regional demand. The same serverless problem now means scaling elastically from 1,000 to 1,500 GPUs quickly in a region, not merely from zero to one.
Burstiness is the unifying workload shape across inference, batch processing, training, and reinforcement learning. GPU snapshots preserve model and Torch-compiled model state to cut cold starts; jobs before training runs can require thousands of GPUs for encoding; and rollout workloads might demand 100,000 sandboxes because “RL is insanely bursty.” That makes orchestration and capacity utilization at least as thesis-relevant as raw accelerator access.
Modal is moving into the model layer because speculative decoding offers multiplicative gains that kernel tuning cannot match. A draft model predicts ahead and the larger model verifies tokens in a batch, preserving output quality while a longer acceptance length can deliver “something like 2× to 4×” speedups versus a few percentage points from faster kernels. Modal is open-sourcing the work, while differentiating through expertise, production reliability, and unusually elastic auto endpoints.
Agents are changing the balance of infrastructure demand from GPU-centric workloads toward constant GPU, CPU, storage, and network coordination. One host framed the shift as roughly 8:1 GPU-to-CPU becoming 1:1; Modal’s response includes regional collocation, multi-container sandboxes, controlled egress, persistent filesystems, private networking, and roughly 3 terabits/s of internal networking for RDMA training. AI infrastructure increasingly becomes “a systems problem of moving memory around, scheduling.”
Auto-research is currently closer to model-guided hyperparameter search than autonomous architecture discovery. Modal’s internal “AutoInference” agents already sweep configurations, run an NVIDIA Nsight profiler, and switch GPUs from H200 to B200; the discussion described the method as a hyperparameter sweep guided by model intuition. ModalBench similarly turns agent failures—especially around observability—into CLI features, skills, and measurable product work.
The business model depends on converting compute planning into both margin and customer savings. Modal’s compute-strategy team models one- versus three-year reservations, fungibility across regions and GPU types, and supply-chain bets; a batch tier could return latency-insensitive work in roughly 24 hours at lower prices. Erik described Modal as deliberately avoiding a hobbyist model-API market in favor of code-level, production workloads that require custom architectures and deeper control.
Deep dive
Not yet available upstream; scheduled sync will retry.