Why AI Compute Comes in Rectangles — Erik Bernhardsson, Modal

Summary

• Erik Bernhardsson estimates Modal is at least 50× larger than when he last appeared on Latent Space, in revenue, users, and usage. Revenue is now about two-thirds Modal Functions (mostly inference) and one-third Sandboxes—a product the team considered shutting down. Sandboxes saw little growth after a quick prototype, then took off as LLMs moved from snippets to full apps, with use cases from vibe-coding companies like Lovable, Ramp’s background agents, and neo-labs running reinforcement learning. Vibhu observed sandbox revenue growing 30% month over month; Erik called the growth “kind of nuts” and said it kept exceeding his expectations.
• The GPU procurement market changed sharply in the past year: Modal went from hyperscaler spot capacity and one-year reservations to working with almost 25 clouds, signing three-plus-year deals with meaningful upfront payments, and building a four-person procurement desk and a ten-person bare-metal engineering team. Capacity is often sold in “three-year rectangles” starting months ahead, while real demand is bursty. Modal’s proposed advantage is aggregating thousands of customers and flexibly slicing up capacity.
• Bernhardsson says Modal is, to some extent, “effectively long GPU prices already” through locked-in reservations, but worries more about a price collapse than further spikes. Hardware prices eventually tend to fall; today’s scarcity rent flows mostly to NVIDIA and partly to neoclouds, while buyers who locked up supply risk “buying at the top.” He is aggressively buying capacity, but not trying to corner the market: prices, utilization, and the resale market could fall together, as his Hunt brothers’ silver-market example illustrates.
• He explains the neocloud growth model: presell five years of chips to a large buyer, take the contract to Wall Street to raise money, then buy the hardware—“pre-sell things that you haven’t even bought.” He names Endscale as an example of a company that went from nowhere to a pipeline worth hundreds of billions of dollars despite having little capacity. Modal’s product-led-growth model can’t lock in long contracts in the same way. He’s glad startups are building GPU price indices and derivatives, but doubts GPUs will become a “Henry Hub… Brent crude”-style, highly commoditized tradable asset.
• “2027 might be the year of non-NVIDIA accelerators”—a call he admits he has gotten wrong for five to ten years. AMD is getting good scores on some tasks, and TPUs work well inside Google. He sees better software support as a potential unlock for alternatives, while noting TSMC and ASML may be deeper bottlenecks. Hybrid disaggregated inference—NVIDIA for prefill, Cerebras for decode—is already being tried.
• The future of sandboxes is more ephemeral: Modal’s median sandbox is about eight seconds long, and Bernhardsson envisions roughly 100-millisecond startup, running a few commands, then stopping and snapshotting it. That could enable branching agents and running every CI test in its own sandbox. The obstacle is overhead from starting and stopping; reducing that “transaction cost” could make highly parallel workloads practical.
• A shift in the fine-tuning value proposition: he says open-source models such as Kimi K3, DeepSeek 4.1, and GLM-5.3 are now almost as good as frontier models at lower cost, often without fine-tuning. He also argues there is a structural role for inference providers separate from model labs: demand shifts with new releases, and Modal can reallocate GPU capacity between models without being asked.
• Modal grew headcount about 4× annualized, a pace Bernhardsson calls unsustainable, and may reach about 700 people next year. FDEs grew from about four to about 24; he says Modal hasn’t aggressively built a product-management team. Code reviews are optional, backed by investment in CI and testing, and the company uses agents to probe its own systems—“escape the room, but for agents.” He expects models to keep improving in verifiable domains, while humans remain competitive for a long time in areas with diffuse targets and longer feedback loops.

Deep dive

1. Spotify’s recommender to Modal: “the infrastructure f*ing sucks”

• Bernhardsson joined Spotify out of school in 2008 or 2009 and spent seven years building its first music recommendation system—Related Artists, Discover Weekly, and “a lot of matrix factorization.” Modal’s genesis came from building AI applications and realizing “the infrastructure f*ing sucks. There’s got to be a better way”; it was a tool he had wanted throughout his Spotify years.
• The path ran through his open-source workflow orchestrator Luigi, in the same category as Airflow. Returning to workflow-orchestration ideas in 2021, he realized “a core puzzle piece isn’t just orchestrating the compute, it’s executing the compute”—a bigger problem—and shelved orchestration. Modal was the third idea he got excited about, after a web-scraping attempt that went nowhere.
• The Better.com detour: five years as CTO of the mortgage startup, growing the tech team from two to “300 or 400,” he thinks. He learned about management, hiring, and developer productivity before returning to infrastructure, AI, and machine learning. Interest rates rose quickly after the pandemic, causing issues at the company, but he had already left.

2. Bootstrapped, shitposted into existence, saved by Stable Diffusion

• He self-funded Modal through its first employees, taking no salary for a long time—“it made me really frugal about money”—and raised a seed round in early 2022. A co-founder joined as CTO around August 2021. Growth was founder-led marketing: “what I’m good at is, I guess, shitposting and being terminally online.”
• The killer app wasn’t ChatGPT but Stable Diffusion, which arrived that fall when Modal had four or five people. Revenue went from about $100/month in November 2022 to about $2,000 the next month, then $30,000 in January 2023, almost entirely from one New York Stable Diffusion company, Mage. Mage accounted for 80–90% of revenue in Q1 2023; Suno became a large customer in late 2023, and revenue later diversified.
• Since the last Latent Space appearance, he estimates Modal is “probably at least 50×” larger in revenue, users, and usage. Customers he names include Decagon, Suno, Runway, and Co-Cognition.

3. You can’t wrap Kubernetes: rebuilding the stack for GPU economics

• His thesis: every technology era runs up against the limitations of its tools. Kubernetes and Docker were built for steady backend workloads on relatively cheap CPUs; generative AI brings expensive GPUs, utilization pressure, and large models that need to load over the network. “You can’t build for this new generation of software by putting a wrapper on top of the old stuff. You need to start over a little bit from scratch.”
• Modal built its own file system and container runtime, along with CPU and GPU snapshotting, an autoscaler, a scheduler, and capacity-management tools. Developer-experience work also proved useful for production workloads, since scaling containers up and down quickly is a similar problem.
• One consequential early decision was going multi-region and treating capacity as “one big pool over the entire world,” rather than separate clusters in each region. Modal has had bugs and outages; recently, Bernhardsson says, it has basically hit four or five nines of uptime.

4. The capacity market flipped: 25 clouds and three-year prepaid rectangles

• Until the end of last year, most capacity came from hyperscaler spot capacity or one-year reservations. Spot capacity was preemptible, but worked in aggregate across hundreds of regions. Modal initially used AWS, later GCP and Azure, and rented Oracle capacity early for A10s and H100s.
• In the past year, Modal has had to work with almost 25 clouds. Getting capacity often means three-plus-year deals and meaningful upfront payments. The last six to nine months prompted a procurement desk of four people, full-time on calls, deals, and vetting, plus a ten-person engineering team managing bare-metal capacity worldwide and handling network issues, bad drives, and GPUs “falling off the bus.”
• Modal doesn’t own capacity; it rents it, typically through three-year deals. Locking up capacity for 2027 will require work across many avenues. Bernhardsson says capacity prepayments are currently the best use of cash; salaries and offices are “basically a rounding error” by comparison: “we have two hundred employees… the GPUs cost a lot more money.”

5. Rent vs. buy: already long GPUs, more afraid of collapse than spikes

• He isn’t dogmatic about buying. Equity can be “a very diluted way” to fund hardware; debt usually requires long-term revenue to cover interest. Owning hard assets adds operational and financial complexity. He would rather focus on being a lean software company, but “everything is on the table,” including joint ventures and complex SPVs.
• He pushes back on the idea that capacity is permanently unavailable: NVIDIA is delivering several hundred thousand Blackwells a month, and “if you’re willing to pay more than everyone else,” you can get capacity. He compares panic buying to the pandemic toilet-paper shortage. Prices could keep rising for another six months or another 36; it’s hard to know.
• He worries more about prices collapsing than rising further. That could hurt anyone locked into a lot of supply: the risk is “buying at the top.” Today’s scarcity rent goes mostly to NVIDIA and partly to neoclouds and other capacity owners. There may be a long-run premium for owning GPUs, but he questions whether it’s worth holding the assets on the balance sheet.

6. Don’t corner the market: Hunt brothers, Armenia, and an internal ClusterMAX

• Asked why not buy as much as possible and resell any surplus, Bernhardsson warns that in a downturn, “a lot of things are correlated”: if prices and utilization fall, the resale market could vanish at the same time. He is aggressive about buying capacity and raising money for more, but wants to build a company that lasts 20, 30, or 50 years. He invokes the Hunt brothers’ attempt to corner the silver market, which “failed spectacularly.”
• He speculates that NVIDIA prefers some diversification among buyers, while Anthropic and OpenAI tend to prefer established vendors and need very large blocks for training. Modal can pursue smaller, scattered capacity: “You can get 32 nodes in Armenia. You can get 128 nodes in Indonesia. We love that kind of shit.”
• Modal vets vendors with a spec sheet covering items such as IOPS, CPU PassMark score, north-south bandwidth, egress, and ingress. It often runs acceptance tests on one node, but larger commitments can still involve “a little bit of a leap of faith.” Internally, he says, they have something like their own version of SemiAnalysis’s ClusterMAX, rating vendors on reliability, responsiveness, and support.

7. The neocloud presell machine and GPUs as an asset class

• Neoclouds can scale quickly by offering a big buyer five years of chips at a good rate, taking the contract to Wall Street for financing, then getting hardware from companies such as Supermicro, Dell, or NVIDIA. “You can essentially pre-sell things you haven’t even bought.” His example is Endscale, which he says went from nowhere to hundreds of billions of dollars in its pipeline despite barely having capacity. Modal’s product-led-growth model lacks the same ability to lock in long-term contracts, making it harder to scale financially as quickly. He calls CoreWeave and other large neoclouds savvy about raising money and using GPUs as collateral for loans.
• On startups such as Silicon Data, Compute Exchange, and Ornn trying to create price indices and derivatives: compute is less homogeneous than other asset classes, given differences in networks, locations, disks, CPUs, and reliability. Still, “you can say the same thing about oil.” He sees a need for better risk hedging, risk-sharing, and insurance, but isn’t sure GPUs will become a highly commoditized tradable asset like Henry Hub or Brent crude.
• Asked about a tweet wishing IPO-ready companies in a CapEx arms race could coordinate and pace it: “No comment.”

8. “2027 might be the year of non-NVIDIA accelerators finally”

• He acknowledges his track record: “every year for the last five, maybe even ten years, I’ve been saying, ‘Next year is going to be the year of non-NVIDIA accelerators,’ and I’ve always been wrong. But I do think 2027 might finally be the year.” AMD is getting good scores on some tasks; TPUs are widely used internally at Google and seem to work well. He says he doesn’t see as much about Trainium, though its team is good and working on software.
• He sees software as a potential unlock: “if we fix the software layer for these accelerators, there’s going to be a lot of demand for them.” He also notes that TSMC and ASML may be deeper bottlenecks. He expects more alternatives over time, while emphasizing that NVIDIA is an excellent company with an excellent product.
• He’s excited by hybrid, disaggregated inference, such as running NVIDIA for prefill and Cerebras for decode. Some workloads are compute-bound and others memory-bound, so hardware can be mixed and matched, though the setups get complex in practice. He also mentions Groq.

9. Two products: Functions at two-thirds, Sandboxes the third they almost killed

• Modal Functions, launched in 2021, offers serverless deployment and scaling while Modal manages capacity. It reached product-market fit in 2023 and is now about two-thirds of revenue, mostly from inference. Bernhardsson resists becoming a neocloud that sells only rectangular, three-year GPU deals; Modal’s product is designed to deploy and scale cyclical or bursty workloads and aggregate demand across many thousands of customers.
• Sandboxes started with the observation that LLMs were generating code and might need a place to run it. The team already had container-management and isolation primitives, including gVisor; a co-founder built a prototype in a day or two. There was some early uptake, but it was an insignificant revenue source going into summer 2023. Two years later, the team even discussed shutting it down.
• He says LLMs became able to generate full apps in 2023, beyond refactoring functions and writing snippets. Growth came from vibe-coding companies such as Lovable, background agents at companies like Ramp, and neo-labs using Sandboxes for reinforcement learning. Sandboxes are now about a third of revenue.
• Bernhardsson says he didn’t foresee how large the product would become. In response to Vibhu’s observation that sandbox revenue was growing 30% every month, he called the growth “kind of nuts” and said that although he kept thinking it had plateaued, it kept growing. Multi-product was always part of the vision; Modal now has Functions and Sandboxes, with training and batch in the works. “We’re in the business of building building blocks.”

10. The ephemeral sandbox thesis: eight-second median, fork everything, thousand-test CI

• Today’s sandbox use is often “lift-and-shift”: put a dev box in the cloud and have an agent control it. Bernhardsson’s preferred model is long-running agents as durable workflows that can start a sandbox in about 100 milliseconds, run a few commands, shut it down, and snapshot it. The median Modal sandbox is “eight seconds long or something like that.”
• The CI example makes the case: rather than running a sequential install-and-test flow, run each test in its own sandbox and “parallelize the hell out of that”—potentially 1,000 sandboxes. Today, setup overhead can take minutes. If you do setup once, snapshot the container, and clone it, the parallelization benefits become practical. Otherwise, the friction is “the transaction cost.”
• He sees potential in forking, snapshotting, resuming, and live migration. Modal has added virtual machines alongside gVisor for workloads that need full Linux compatibility. He also describes Anthropic’s managed agents as having a bring-your-own-sandbox integration for Modal. His view is that Anthropic benefits from a broad ecosystem of tools, provided customers use Anthropic for inference.

11. Training’s underserved middle and the fine-tuning reversal

• Bernhardsson is bullish on training, especially for researchers iterating and for smaller and mid-sized workloads rather than 100,000-GPU hyperscaler or neocloud deals. Examples include text-to-speech, speech-to-text, recommendation systems, and computational biotech; some customers train relatively modest models on 64 GPUs. “It’s very hard to set up training clusters.”
• Batch is another opportunity: one customer needed to transcribe 5,000 years of audio from millions of files, and Modal had enough capacity to do it in a few days. He is also bullish on low-latency workloads as speech-to-speech models improve, and on building general-purpose, composable blocks for AI-powered software.
• His fine-tuning view has shifted. A few years ago, tuning a smaller open-source model could bring it closer to frontier quality at lower cost. Now, he says, models such as Kimi K3, DeepSeek 4.1, and GLM-5.3 are almost as good as frontier models at much lower cost and can often be used off the shelf. Demand for post-training remains, including supervised fine-tuning and reinforcement learning, but fine-tuning is optional rather than something users have to do.

12. The Hugging Face incident, agent pentesters, and why cyber incumbents sell fear

• The agents involved in the Hugging Face exploit were running inside Modal and found a vulnerability in an app deployed on the platform. As Bernhardsson understood it, they used that app as a springboard to get full internet access. He stresses that an app running on Modal was hacked; Modal itself was not.
• One lesson is whether Modal can make security more restricted by default without losing the product’s “magic.” He compares this with AWS making it harder to accidentally expose an S3 bucket publicly. Modal has also begun using agents internally to probe for vulnerabilities and sandbox escapes. “It’s like escape the room, but for agents.” They started a few months earlier, around the time Vibhu mentions Fable.
• He believes agents can be very good at finding vulnerabilities and can also be used defensively. Asked whether cybersecurity companies are “cooked,” he says that is outside his expertise and doesn’t jump to that conclusion: security vendors have enterprise relationships, and “a lot of what you’re selling is effectively fear.” Modal used to bring in human penetration testers roughly every six months, he thinks, and may now need them less often; he remains unsure. He’s optimistic that people and companies will use AI to work more efficiently.

13. Agent experience is developer experience: AEO, deprecation, the slop cannon

• Asked whether developer experience matters when agents can read the docs, he says almost everything that makes DX good also helps agent experience: good docs, descriptive errors, and fast feedback loops. Agents make some hard-to-use tools easier to work with, and the human toil of dealing with YAML has gone down. He expects infrastructure to become more programmatic, with SDKs and imperative operations rather than declarative DSLs.
• Agent training data makes deprecations harder: agents often use old Modal syntax, so the team has become more conservative about removing features and must keep examples online up to date.
• Modal runs internal benchmarks asking agents to deploy Stable Diffusion or rank sandbox vendors by reliability and price. Agents can misunderstand pricing, so the team works to make content clearer. Asked about generating GitHub repos full of Modal code to influence agent recommendations, he calls it a “slop cannon.” He thinks automated content could work, as with early SEO, but doesn’t like the idea.

14. FDEs, product management, the IC renaissance, and 4× headcount growth

• Modal scaled its forward deployed engineers from about four at the start of the year to about 24. They work with customers on performance tuning, including training draft models and adjusting parameters in DLM and SGLang; help customers work around problems; flag over-provisioned CPU or memory; and assist with migrations. They are not meant to make Modal a service-heavy business. A key goal is to identify product gaps: “Long term, how do we turn this into product so we don’t need an FDE for it?”
• Modal hasn’t aggressively built a product-management team. Bernhardsson says infrastructure product management requires deep strategic thinking about foundational building blocks and one-to-two-year horizons, and avoiding many similar but distinct features. Some people have moved into product roles from DevRel or engineering.
• He sees a shift back toward individual contributors, which GenAI may have accelerated. Modal makes clear there is no prestige difference between being a manager and an IC; former managers are returning to writing code, and Erik and Akshat still do so. Code reviews are optional. He sees them partly as a way to avoid accountability for merging code, so the alternative is investing in CI, deployment, and testing. Infrastructure companies need “one more nine” of uptime, with resilient systems, rate limits, decoupling, and strong test coverage. He says agents have become good at writing tests.
• Headcount grew about 4× annualized over the past year, which he calls unsustainable. He watches for cultural drift, islands of differing work styles, and engineers taking on speculative projects that may not be the right priority. Modal may reach 250 people by year-end and around 700 the following year, with most hires in systems engineering and more work on GPU performance and research, including post-training, distillation, and quantization. He says the story behind the Dunbar number may be “more fake than anything,” but that the underlying effect is “actually kind of real.”

15. The rectangle endgame: flexible capacity, Taylor Swift, and what’s not plateauing

• Bernhardsson’s tweet argues that the problem with GPU scarcity isn’t only getting GPUs; it’s that capacity is sold as a three-year rectangle starting three to six months in the future, unlike most users’ bursty demand. Modal can often provide 100 GPUs on demand, without a sales conversation. Its value proposition has two parts: developer experience and flexible capacity. By aggregating many thousands of customers, Modal smooths demand and can slice capacity up and down quickly.
• He sees a structural role for inference providers separate from model labs. Demand shifts when a new lab releases a model; Modal can move resources between models as demand changes. He says no one even has to tell Modal to do it; the system handles it.
• On whether LLMs are commodities, he’s unsure: “Is Taylor Swift a commodity?” If labs keep releasing new models and their margins cover training costs, they could still have a good business.
• He doesn’t think intelligence is plateauing, though he says it remains to be seen how well models handle tasks that are harder to train with reinforcement learning. Models have improved at coding, math, and cybersecurity—areas with more verifiable outcomes. Humans may remain competitive for a long time in tasks with diffuse targets and longer feedback loops.
• He names computational biology, where predictions can be checked with simulations of molecular dynamics, and materials science, where properties can be measured, as promising areas. On-prem remains a model he is skeptical of: he is a cloud maximalist and says BYOC and VPC peering reflect an outdated network-as-a-moat security model. He doesn’t know whether Modal will ever do on-prem, but would consider BYOC at a sufficiently high price and sees it as a possible way to use neocloud capacity while keeping Modal’s control plane in its own environment.