Why RL Won — Kyle Corbitt, OpenPipe (acq. CoreWeave)
Summary
OpenPipe found an early product wedge by distilling expensive GPT-4 workflows into smaller, cheaper models while open alternatives were not yet good enough. Its drop-in SDK captured production traces, trained a replacement, and changed only the inference URL; the first three customers arrived within a month. It reached $1 million ARR roughly eight months after launch, but repeated 3–5× frontier-token price cuts steadily consumed the wedge.
Fine-tuning still pays when deployment constraints force a smaller model, especially for real-time voice, single-GPU operation, or private-cloud inference. Kyle estimates that for “90% of use cases” without such a constraint, the ROI remains poor. Training runs cost only $5 to a couple hundred dollars; the real expense is weeks or months of engineering plus an ongoing loss of stack flexibility whenever prompts or context change.
OpenPipe’s task-specific RL strategy moved from a 25% long-shot in January 2025 to Kyle’s current 55–60% probability that scaled agents should learn through RL. Frontier labs had already shown RL’s value for general agentic and coding models after o1; the unresolved bet was whether enterprises should train each deployed agent on its own task and experience. “Not 100%… but the probabilities in my mind are going in the right direction.”
GRPO simplified RL by eliminating PPO’s value model and requiring only relative rankings, yet its dependence on parallel reproducible rollouts may make it “a dead end.” Relative scoring lets a judge choose the best of several trajectories without defining globally correct behavior. The cost is an environment that must reproduce production—including bugs, state, tool responses, and human variability—while PPO can, at least in principle, learn directly from real production traces.
RULER convinced Kyle that reward assignment is “fairly solved,” leaving environments as the dominant bottleneck. The library ranks a group of agent runs with an LLM judge; OpenPipe even trained Qwen 2.5 14B using Qwen 2.5 32B as judge and beat frontier models on its chosen task. “It basically just works,” although highly specialized domains may still benefit from task-specific reward models.
Prompt optimization was directionally useful but nowhere close to RL in OpenPipe’s tests: roughly 50% for its baseline, 56% with GEPA, and 96% with RL. Kyle preserves the caveat—“maybe we were holding it wrong”—and says customers care about final performance rather than whether prompts or weights changed. He agreed that online evals are more promising than static data, because feedback otherwise becomes stale and off-policy.
The CoreWeave acquisition gives OpenPipe a larger platform for serverless RL and continual learning rather than ending the product thesis. Inside the Weights & Biases organization, the team now handles GPU crashes, out-of-memory failures, and scaling while customers define environments and rewards. Its North Star is “an open world where every agent learns continually from its real-world experience,” potentially enabling what Kyle estimates could be 10 times as much AI inference as exists today from projects stranded at the proof-of-concept stage.
Open models could capture 15–20% of non-coding tokens by the end of 2026, but coding may remain proprietary while $200 plans heavily subsidize superior closed models. Enterprises want open models for cost and privacy if performance is sufficient. The discussion questioned how durable subsidies are, while Kyle noted that labs building their own infrastructure may keep prices low by monetizing otherwise-idle GPUs.
Deep dive
1. GPT-4’s price created OpenPipe’s first wedge
Before OpenPipe, Kyle spent about 4.5 years at YC leading Startup School. After leaving around the beginning of 2022 and trying several ideas, he and his younger brother formed OpenPipe around March 2023.
The opportunity was unusually legible: GPT-4 was “insanely expensive and extremely powerful,” while task-specific workflows could potentially be distilled into far smaller, cheaper models. At that point, open models were generally not good enough to replace the closed models directly.
The product wrapped that thesis in a managed flow rather than asking customers to become ML teams. A drop-in replacement for the OpenAI SDK captured GPT-4 questions and responses in production; when enough data accumulated, OpenPipe trained a model and exposed it behind another compatible API.
Production AI remained rare in late 2023, but the few companies operating at scale were sometimes paying OpenAI hundreds of thousands of dollars per month. That made the sale easy: OpenPipe found its first three customers within roughly a month and reached $1 million ARR over about eight months after launch.
The early result validated the pain, not the permanence of the solution. Frontier providers repeatedly cut token prices by 3–5× while releasing more capable small models, gradually erasing the savings that had made distillation such an obvious purchase.
2. Product experience mattered more than GPU-provider bundling
The hosts’ original market concern was a squeeze between frontier labs releasing cheaper “mini” models and GPU clouds offering fine-tuning to increase customer lock-in. Kyle felt the frontier squeeze acutely, but said competition from neocloud and GPU-provider fine-tuning products “never really materialize[d].”
Every infrastructure provider seemed to have an offering, yet OpenPipe’s prospective customers largely ignored them because they were difficult to use. Kyle’s explanation was prosaic but consequential: fine-tuning was not those companies’ product focus, and “developer experience matters.”
Mistral 7B and Mixtral created what the hosts called a “golden period of fine-tuning startups.” They were credible improvements over Llama 2, while Mistral’s Apache 2.0 license offered an unusually open commercial promise at a time when open-model licensing still carried substantial customer anxiety.
The strategic lesson from the original business was that a usable managed layer could outperform vertically bundled infrastructure—but it could not indefinitely defend against a frontier model whose quality rose while its price collapsed.
3. Fine-tuning earns its keep under hard deployment constraints
Kyle largely stands by his earlier provocation that most users “probably don’t need fine-tuning.” The clearest exception is a forced move to a smaller model, usually because real-time voice demands lower latency or because deployment must fit on one GPU or inside the customer’s cloud.
Under those constraints, tuning often becomes necessary rather than merely helpful: an untuned small model cannot reliably replace the larger model the application originally used. Outside them, Kyle estimates “90% of use cases” still do not offer a compelling return on fine-tuning effort.
The fixed cost starts with at least a couple of weeks from a competent engineer. A complex RL system requiring a bespoke environment can demand a couple of months, while every later prompt or context change may add hours of retraining and slow the product team’s iteration cycle.
Compute is almost irrelevant by comparison: individual runs cost between about $5 and a couple hundred dollars. “The dollar cost… is basically never a factor”; engineering time and the continuing loss of stack flexibility dominate the decision.
4. LoRAs remain an infrastructure advantage, not discount fine-tuning
LoRAs reduce training memory, but Kyle says their greater value appears at inference. Many adapters can be multiplexed over the same GPU deployment, enabling per-token economics instead of forcing each customized model into a dedicated GPU-hour contract.
The hosts’ marketing critique was sharp: LoRAs were presented as the “Walmart store-brand fine-tuning” for buyers unable to afford a full run. OpenPipe sometimes had to counter that perception by asking customers simply to test them against the lightweight task customizations they actually needed.
For those workloads, Kyle sees “really no downside” and considerable infrastructure simplicity. Thinking Machines’ later research comparing LoRAs with full fine-tuning, together with Kyle’s observation that big-lab researchers use LoRAs for experiments, felt like a significant vote of confidence.
Kyle’s diagnosis of LoRAs’ fall from fashion is therefore indirect: “LoRAs were uncool for a while… mostly just ’cause fine-tuning was uncool.” If fine-tuning returns as task-specific RL grows, their deployment properties become newly valuable.
5. o1 turned task-specific RL into a calculated company bet
Leaks around Strawberry and the release of o1 told OpenPipe that someone had made RL work meaningfully for LLMs. By late 2024, Kyle considered the frontier case increasingly settled: RL investment was producing materially better agentic behavior, most visibly in coding models.
The unproven extension was task-specific customization. Could individual companies teach agents their particular jobs, or would frontier labs eventually train on every important task and eliminate the need? OpenPipe chose in January 2025 to spend at least several months finding out.
The team had already RL-trained a playful Hacker News headline generator before the email-agent work. Kyle assigned only a 25% probability that task-specific RL would become something “everyone doing inference should be doing,” but the upside of being early justified the bet.
Customer work has since raised his estimate to 55–60% that agents deployed at scale should receive RL either before deployment or continuously afterward. His hedge remains load-bearing: “It’s still a bet,” not an established universal architecture.
6. GRPO’s relative rewards trade simplicity for an environment trap
GRPO removes PPO’s separate value model, along with the operational work and hyperparameters required to train it. It generates multiple trajectories under identical conditions, scores them, and increases the probability of the better-performing paths relative to the weaker ones.
That group-relative structure lowers the burden on the scorer. A human or LLM need not answer whether one output is absolutely good; it only needs to choose which of a handful is better—a comparison Kyle says is much easier and can work without globally calibrated rewards.
The “huge downside” is that parallel rollouts demand a reproducible environment. For agents operating real applications and codebases, creating a sandbox with production-equivalent state, responses, failure modes, and data is now the hardest part of getting RL to work.
Kyle therefore suspects GRPO “is likely to be a dead end” for many deployments. PPO is more operationally involved, but at least in principle it can learn from actual production traces without requiring the simulated world that GRPO’s group-relative training requires.
7. High-fidelity environments are the scarce RL input
Kyle’s Airbnb analogy exposes the workload: training a booking agent requires a replica that responds like the real site, including its bugs. Omit a production failure mode and the trained agent may simply “fall over” the first time that bug appears outside the sandbox.
Cooperative agents add another simulation problem—the human. An LLM with a user-simulator prompt produces much narrower behavior than real customers, whose phrasing, corrections, and reactions vary in ways the training agent may never encounter.
In theory, companies already possess such systems for end-to-end testing. In practice, Kyle says enterprises “almost universally” lack realistic, fully populated replicas; even existing test environments exclude odd bugs and production-grade data distributions.
Kyle thinks environment startups may still have value, but said the current customer pool is only roughly four, perhaps six, major labs. The host estimated roughly 20 such startups and at least seven-figure deals; Kyle characterized the current model as a services business in which specialists build integrated environments ad hoc rather than shipping a simple CSV-like data product.
Varas, one of OpenPipe’s portfolio companies, works with agent builders on internal tool-call loops and turns their traces into data for OpenPipe’s RL. Kyle cited financial-services use cases such as balance and transaction questions, where poorly documented or named tools make agent behavior difficult to improve and test.
8. GEPA lost to RL, while online evals survived the comparison
The hosts questioned whether prompt optimization and weight updates were apples and oranges. Kyle’s answer was commercial: a company seeking the best agent does not care which layer changed—“If you get better performance on my agent, I’m happy”—and maximum performance may ultimately use both.
OpenPipe’s empirical result was lopsided: its naive prompt with its model scored about 50%, GEPA reached roughly 56%, and RL reached about 96%. Kyle also said they had tried the previously hyped MIPROv2 approach. He repeatedly qualified the conclusion: “Maybe we were holding it wrong.”
Kyle’s strongest argument for GEPA was that it philosophically resembles the iterative, genetic evolution of the large labs’ long system prompts, but the hosts pointed out that human prompt authors can inject taste, intuition, and information absent from a fixed evaluation. They may reject a change that scores well because it is “not really” what they want.
Kyle agreed more strongly with the online-eval thesis. Static data becomes stale as prompts or weights change, just as old RL trajectories become off-policy; useful improvement loops need current production feedback, bringing analytics, UX, experimentation, and model observability into one system.
9. RULER made weak judges surprisingly effective
RULER—Relative Universal LLM-Elicited Rewards—packages the GRPO insight into an approachable reward system. Given a task and, for example, four agent runs, an LLM ranks which performed best instead of assigning independently calibrated absolute scores.
Kyle says the judge can “self-ground” against the group, and the method worked far better than expected across internal experiments and client projects. Once added to a training run, “it will probably just work,” making reward design much less bespoke.
The striking specimen used Qwen 2.5 14B as the model being trained and Qwen 2.5 32B as judge. Despite the judge being much weaker than a frontier model, the trained agent surpassed frontier models on the specific task OpenPipe evaluated.
That pushed Kyle from roughly 25% to 50%-plus confidence in task-specific RL and left him feeling “the reward assignment problem is fairly solved.” He is bearish on generic specialist judge models because LLM-as-judge is common enough that frontier labs are likely training on many such tasks. He still allows that unusual tasks with abundant task-specific data could benefit from a dedicated reward model or judge.
10. World models might replace hand-built sandboxes
Kyle sees world models as one plausible answer to the remaining environment problem. An agent would issue a tool call, while the world model generates the likely response and maintains enough internal state to reflect earlier actions and downstream consequences.
The proposed workflow is to condition that simulator on production traces so it learns a particular system’s behavior and failure modes, then train the agent inside the imagined environment. Success requires the resulting policy to transfer back to the real system—a condition Kyle treats as promising research, not a solved fact.
The hosts connected this with Meta’s code world-model work, but Kyle drew a distinction: that project appeared closer to pre-training a model that understands program execution. Code rarely needs a synthetic deployment environment because trainers can simply execute it; enterprise tools and human workflows cannot be reproduced so cheaply.
11. Open-model economics hinge on coding and subsidized capacity
An Andreessen Horowitz estimate put open models at about 5% of tokens and falling; the hosts expect the share to rise as enterprises seek cost, privacy, and control. One host’s end-2026 estimate was 15–20% after excluding coding, rather than anything close to 50%.
Coding changes the denominator because proprietary models remain better for the relevant workloads and $200 “max” subscriptions can generate thousands of dollars’ worth of apparent usage. Kyle described paying $200 and accidentally spending $100 in an hour, while the host characterized the plans as heavily subsidized.
Kyle countered that subsidies may persist: Anthropic and OpenAI are building their own infrastructure and may have idle GPUs they own, creating an incentive to keep utilization high and subsidize some usage. He compared this with cheaper compute offerings around $1.40 for an H100 versus AWS’s listed $2.20.
The financing tangent captured the industry’s scale: Kyle argued that roughly $200–300 billion could probably be raised and committed toward a much larger plan, while the host joked that there is “no other industry in the history of the world” where another couple hundred billion can sound incidental.
12. CoreWeave gives continual RL a larger operating base
RULER was released on July 25, and the acquisition followed in September. Weights & Biases founders Lucas and Shawn initiated the conversation after W&B had recently been acquired by CoreWeave, which wanted to move further up the stack.
The process was “long” and “pretty painful”; as late as the week before signing, Kyle remained unsure it would close. OpenPipe now sits within the Weights & Biases organization and ships under that brand, although the combined structure is still being worked out.
Kyle had feared that selling would make the work environment “really, really suck”; after about one month, he said it had been far better than expected. The newly launched serverless reinforcement-learning product lets customers define an environment and reward, then send trajectories and scores for model updates. OpenPipe’s team absorbs GPU management, crashes, out-of-memory errors, and scaling.
The North Star is “an open world where every agent learns continually from its real-world experience.” Kyle estimates that 10 times as much AI inference as currently exists could eventually come online from projects stranded at the proof-of-concept stage; he separately says 90% of today’s addressable inference market could come online if reliability improves.
Reward hacking does occur, but Kyle considers it manageable: once a model finds a loophole, it repeats the trick so consistently that operators quickly notice. Adding an extra negative term to the reward prompt has made the behavior “just go away” in customer work.
His YC lesson explains OpenPipe’s evolution: “Hold your problem tight and your solution loosely.” Yet he now questions relentless gradient-following; with enough vision and taste, his next startup might stay heads-down longer and pursue a more ambitious idea before demanding immediate market proof.