Summary

vLLM has become a widely used execution layer connecting open-weight models to major accelerators. Introduced as running on half a million GPUs at any moment, it supports more than 1,000 active model architectures while NVIDIA, AMD, Google, Amazon, Intel, and others ensure new chips can run it—and often benchmark against it. Simon Mo likens its role to “databases and operating systems” for AI.
Open weights shifted from enthusiast territory to strategic infrastructure when application companies needed differentiation beyond a proprietary-model wrapper. Matt Bornstein points to Cursor, Decagon, Harvey, and similar startups requiring their own mid-training, post-training, inference, and deployment techniques. Closed APIs do not provide that access, so open source became “deeply embedded,” even though Matt notes OpenAI and Anthropic models remain more widely used and generally more critical overall.
The economic case is increasingly about controllable performance, reliability, and data—not merely cheaper tokens. A voice-agent company can control its infrastructure and enforce a latency SLA. Kimi K2 bridges almost a 10x price gap without being as expensive as Claude or GPT-5, while bringing an Opus 4.1-level model onto infrastructure that users can run and fine-tune. Open-weight providers can potentially offer 10 speed tiers, including 400–500 tokens per second in some workloads, versus a proprietary provider’s regular and fast modes.
Open-weight licensing is moving away from unconditional gifts because frontier training cannot be sustained by donated developer time. Model labs face millions or billions of dollars of compute plus repeated failed runs, leading to usage thresholds, derivative-work provisions, and commercial agreements. Simon’s pharmaceutical analogy captures the requirement: released products must return enough revenue to fund the next risky R&D cycle.
Moderation failures may make open weights the default for trusted, specialized work. Simon argues that proprietary guardrails remain arbitrary and false-positive-prone; even GPU-kernel debugging can trigger restrictions and destroy a two-hour session. “If moderation is never solved,” users will prefer models whose guardrails they can control for trusted use cases.
Simon expects no meaningful open-versus-closed capability gap within one year because progress now depends more on environments and algorithms than distribution strategy. Moonshot’s front-end coding loop—generate, render, inspect, and iterate—is his key example of an environment that cannot simply be distilled. He leans against distillation as the main explanation for progress: the durable engine is “really smart people” combining compute, data, environments, and novel methods.

Deep dive

1. Inference became a systems problem before it became a market

Simon’s starting point: LLM serving is fundamentally unlike earlier ML workloads because inputs have variable lengths, outputs are nondeterministic, and every request needs a fast response despite intensive GPU or TPU computation. Batching and scheduling therefore sit “at the core of the inference engine.”
The hardware threshold arrived gradually. ResNet already needed an NVIDIA K80 for useful speed but could still run slowly on a commodity CPU; BERT made GPU acceleration important for efficient translation and related tasks, while Hugging Face accumulated “1,000 BERT variants” specialized for different jobs. This was before 2020.
Simon points to maybe 2023, when GitHub Copilot and ChatGPT became tools people felt they “cannot live without anymore.” Once model-backed applications entered daily work, accelerators and open inference software stopped being research conveniences.
Matt’s later threshold came about a year before the conversation: companies such as Cursor, Decagon, and Harvey concluded they could not build enduring products as wrappers. Their own mid-training, post-training, inference, and deployment techniques required open weights, making open source central but often invisible. He cautions that OpenAI and Anthropic models are still more widely used and generally more critical overall.

2. vLLM sits where new models meet every chip

Simon defines vLLM as the engine that turns available GPUs into “a running endpoint for intelligence,” supplying cost efficiency, reliability, and frontier compatibility. It supports more than 1,000 active architectures; when research prototypes become globally accessible open-weight models, they are live on vLLM immediately, a process called “day-zero model release.”
NVIDIA, AMD, Google, Amazon, Intel, and other hardware vendors work to ensure their newest chips run vLLM; in many cases, they use it as a benchmark. The project occupies the junction “where models run” and where hardware can meet them.
A release may coordinate the model lab, primary or secondary hardware vendors, Hugging Face, vLLM, and, depending on the lab’s appetite, 10–20 additional release partners such as inference clouds or public hyperscalers. Some labs already use vLLM during reinforcement learning and submit working code; others have internal engines but little understanding of the wider open-weight ecosystem.
The early Mistral release is Simon’s specimen: the lab initially posted a torrent link for Mistral 7B, leaving enthusiasts scrambling and struggling to run it. Mistral and the vLLM team worked behind the scenes over the weekend, then announced a working path that let everyone reuse the systems work.
Simon says open-source inference is currently the leading way to run models because inference clouds and APIs-as-a-service often leverage open-source engines under the hood, building on their battle-tested recipes and accumulated fixes.
Inferact signed NVIDIA’s Open Weights and American AI Leadership Letter because Simon says open weights matter and model development should not be blocked or banned by a world controlled only by proprietary APIs. Inference engines are downstream of pretraining and reinforcement learning, but are where the model meets the world.

3. Ownership buys a performance dial, not automatically a bargain

Simon says customer priorities fluctuate: control mattered throughout the past few years, while cost surged only recently as coding plans and token spending became expensive. Control includes the ability to manage cost, observe the entire system, and know what performance the bill actually purchases.
His voice-agent example makes the mechanism concrete. Controlling the model, hardware, and monitoring path lets a provider ensure that an agent answers within its SLA; depending on a proprietary API introduces outage and contract-performance risks outside its control.
Simon describes Kimi K2 as bridging “almost a 10x gap” rather than simply winning on price. It is not as expensive as Claude or GPT-5, but is more expensive than GLM-4.5 and Qwen 3.5. The stronger case is “bringing an Opus 4.1-level model to my own infrastructure” so operators can fine-tune it, understand token needs, profile performance, and govern security, compliance, and data retention.
Proprietary services commonly expose regular and fast modes—only two switches—whereas an open-weight provider could offer potentially 10 speed levels. vLLM’s own fast mode reaches 400–500 tokens per second in some workloads, typically 2–3x faster than the fast mode available from proprietary providers, making an agent feel like it is “executing” instead of remaining stuck thinking.
Simon specifically contrasts this control with Fireworks, which he says does not have a zero-data-retention policy. The point is not only lower cost but the ability to own the model’s performance and operating conditions.

4. Open weights require a new funding compact

Matt stresses that “open-source models” usually means open weights, not open-source software. Historically, models often arrived under Apache 2.0 terms: “Take it, modify it, do whatever you want with it.” Meta’s Llama licensing added a commercial threshold based on metrics such as daily active users or annual recurring revenue. Elena notes that the chosen threshold initially excluded all but roughly two companies worldwide.
Newer labs are testing more explicit economics: MiniMax’s M2.7 included usage-focused terms, while Kimi initially attached conditions to derivative works. Matt’s pushback—worth keeping—is that this is not necessarily greed: unlike software, a frontier model cannot be trained by volunteers donating nights and weekends. Labs need millions or billions of dollars of computing resources.
Simon compares model development with pharmaceuticals: expensive R&D must tolerate repeated failure, yet some released-product revenue must finance the next attempt. After release, a “whole village” still adapts one training configuration to different cluster topologies and use cases spanning edge devices, giant clusters, voice agents, and coding agents.
Matt’s thought experiment exposes the constraint: if GPU prices fell 99%, individuals might again explore frontier ideas collectively. AlexNet, the first neural network he identifies as running on GPUs in a consequential way, needed literally 2 GPUs; today’s frontier experiments require enormous compute.
Simon also frames scale as both fitting a gigantic model onto a data-center rack and optimizing it toward the maximum possible speed. Broad community and partner validation helps find bugs that appear only 0.0001% of the time.

5. Moderation failures make control a functional requirement

In Simon’s reading of the Hugging Face incident, proprietary APIs produced arbitrary, difficult-to-enforce guardrails with excessive false positives, and Hugging Face had to use an open-weight model for the trusted use case. He connects this to the longstanding difficulty of designing content filters and moderation systems.
His conditional claim is categorical only within its premise: “If moderation is never solved,” trusted users will default to open weights because they can lessen or control the guardrails for their use cases. Inferact developers already find that GPU-kernel work, including an invalid memory-access error, can trigger restrictions and cause them to lose a two-hour job.
Matt extends the social-media analogy but concedes that caution may be reasonable. AI centralizes not merely speech but code, spreadsheets, health advice, and action, without social media’s liability carve-out; providers also impose their own ethical boundaries, leaving difficult edge cases both inevitable and frustrating.

6. Environments, not distillation, may decide the frontier

Asked about 5 years, Simon initially jokes, and Elena resets the question to 1 year. Simon then says that capability-wise he sees little gap even today and projects that, one year out, there will be “not going to be any difference.” Open versus closed is primarily a distribution and go-to-market choice built from similar ingredients: a computer cluster, training data, brilliant researchers, and pretraining followed by post-training.
The differentiator is increasingly the learning environment. Moonshot’s front-end coding setup lets Kimi K2 code, inspect what it rendered, and continue looping. The advantage comes from constructing that feedback process and choosing algorithms that exploit it, not merely acquiring static source data.
Simon’s favorite technical example brings the field “full circle”: Jianlin Su, who introduced rotary positional embedding (RoPE), also wrote the explanation for why Kimi K2 did not need it. Matt’s reaction captures AI’s empirical character—once theory says a component is essential, deeper experimentation can reveal that “simpler actually is better.”
On whether Chinese labs depend critically on distillation, Simon leans against it. Proprietary learning environments and the process by which models improve inside them cannot simply be copied; dataset rewriting is possible with many models, but he says today’s progress is not primarily powered by distillation. It still comes from “really smart people” combining algorithms, data, environments, and compute.
Matt draws the policy implication: stopping distillation would not stop capable researchers elsewhere. Simon instead describes open weights as a “racetrack” where participants see one another’s position, learn openly, and stand on each other’s shoulders—accelerating innovation across countries rather than containing it.

178: Talking RSI with 田渊栋: How Will Model Self-Evolution Arrive?Chasing Trillion-Dollar Companies, Founder Ambition, Token Budgets, & Regulatory Capture