The Dawn of Dynamic AI: RFT Comes Online, w/ Predibase CEO Dev Rishi, from Inference by Turing Post
Summary
RFT is a path from static enterprise models toward continuously improving systems. Dev Rishi says “train once and learn forever” is beginning today: customers often use a model on which someone else did “99% of the heavy lifting,” then add the final 1% of customization. Today, RFT is one of several useful tuning techniques and remains a one-off process; its larger promise is becoming part of an online feedback loop. It can use “a dozen examples or so” plus reward functions that grade behavior, rather than the large labeled datasets required by supervised fine-tuning. For code, rewards might assign five points for formatting, 10 if it compiles, and 20 if unit tests pass: “If you can measure it, you can improve it.”
The key infrastructure is the feedback pipeline connecting deployment, evaluation, and post-training. Predibase has launched automatic collection of prompts and responses from production deployments and is researching how RFT can learn from small amounts of user feedback alongside techniques such as DPO. Early healthcare companies combine LLM judges with clinician labels from patient conversations, though Dev stresses that only “the most cutting-edge companies” are doing this today.
Agentic AI’s commercial bottleneck is reliability, not demo capability. Dev describes an agentic workflow as multiple LLM calls, likely with tool use, making errors compound across a chain; in his stated example, five calls that are each 90% accurate produce a “sub-50%” user experience. Many demos work on the “golden path,” but production demands robustness when users depart from it.
Enterprise AI is fragmenting into specialized systems rather than converging on one universal model. Dev’s favorite customer formulation is, “General intelligence is great, but I don’t need my point-of-sale system to recite French poetry.” He expects narrow, automation-oriented agents to dominate many enterprise use cases, while broad versatility may remain more valuable for consumers. Enterprises will mix open and commercial models across parameter sizes, choosing the best tool for each task.
Inference becomes defensible when it includes production reliability, efficiency, and learning—not merely a model endpoint. A large model may require eight or 16 H100s for one replica, while business-critical deployment can require 99.9% or 99.999% SLAs, fault tolerance, blue-green updates, and multi-region replication. Dev expects base-model endpoints to commoditize; Predibase’s “intelligent inference” concept connects serving directly to post-training and continuous improvement.
Open-source models have closed the performance gap faster than Dev expected. He says DeepSeek R1 or V3, Qwen 3, and Llama 4 are on par with—and on some benchmarks ahead of—leading commercial models, a milestone he considers roughly six months early. He contrasts this with 2023, when GPT-J was the best open-source model and far behind GPT-3.5. Open reasoning models also expose reasoning tokens, enabling evaluation of the steps behind an answer rather than only its final output.
Dev sees compelling enterprise unit economics but worries that buyers may aim at the wrong applications. He says Fortune 200 CIOs working with generative AI rarely question the ROI relative to “50 cents or a dollar” per million tokens. The danger is attaching expectations to flashy multi-agent demos, then entering a “crash of disillusionment” when brittle systems fail to deliver practical business value.
The winning development cadence may be to ship a 60% solution, test demand, and use production data to improve it. Dev sees larger companies adopting a startup-style loop: validate product-market fit before perfecting the system, then collect the feedback needed for continuous learning. His mentor’s rule captures the operating philosophy: “If you ship something that you’re not at least a little embarrassed by, you’ve waited too long.”
Deep dive
1. RFT turns post-training into a live feedback loop
Dev argues that “train once and learn forever” is beginning now. Customers typically take a model whose creator performed “99% of the heavy lifting,” add the final 1% of customization, and are starting to replace that static artifact with a pipeline that improves while deployed.
RFT reduces dependence on large labeled datasets by combining perhaps “a dozen examples or so” with a grading rubric. For code, Dev’s example awards five points for formatting, 10 if the output compiles, and 20 when unit tests pass—the model learns toward explicitly measurable outcomes. Today, he describes RFT as one of several useful tuning techniques and a one-off training process; its larger shift would be becoming part of a continuous feedback loop.
Dev says he thinks Predibase was the first end-to-end platform to offer RFT when it released the capability a couple of months earlier. Predibase has also launched automatic collection of prompts and responses from production deployments, addressing the data bottleneck, and supports feedback approaches including DPO while researching RFT with user-feedback data.
Healthcare companies are beginning to feed LLM-judge assessments and clinician labels from patient conversations into early RFT loops; forthcoming work will show how small feedback samples affect performance, then how those gains scale with additional feedback. Dev stresses that only “the most cutting-edge companies” are doing this today.
2. Agents expose compounding errors and weak evaluations
Dev describes an “agentic workflow” as two components: a chain of multiple LLM calls—one-shot document classification does not qualify—and likely tool calling, allowing the agent to invoke functions on a user’s behalf. His example is a medical assistant that helps a user understand a diagnosis and schedule a follow-up.
His pushback on current excitement is operational: agents are “quite brittle,” and polished demos often survive only on the “golden path.” Dev says five successive calls at 90% accuracy can already mean a “sub-50%” user experience, making the final increments of quality disproportionately important—a pattern he had also seen while working on Google Assistant.
Evaluation remains an open problem because outputs range from objectively classifiable to subjectively “good.” Dev sees three working methods: historical holdout data where available, stronger LLMs acting as judges, and shipping the product to collect directional feedback—an approach that means “we should not underestimate how much of it is vibes today.”
The host’s objection is that closed-model releases introduce a new persona and force teams to develop a new evaluative “vibe.” Dev’s partial answer is open reasoning models: DeepSeek R1 exposes reasoning tokens, letting teams assess the steps used to reach an answer rather than grading only the answer itself.
3. Predibase bet that enterprise intelligence would specialize
Predibase began by democratizing deep learning, with fine-tuning pretrained models such as BERT as its most popular workflow in early 2022. When OpenAI “democratized deep learning more than any of us” at the end of 2022, the user journey and persona shifted from specialist NLP engineers toward newer AI builders.
The company focused its product in early 2023 on LLMs, betting that production models would be specialized and customized; later that year, it went heavily into inference. Dev characterizes this as either a complete pivot or a sharp focusing of the original mission.
Checkr illustrates the enterprise thesis: its model extracts specific criminal codes, violations, and background-check details rather than writing Python or French poetry. Dev expects narrow automation-oriented applications to dominate enterprise value, while conceding that broad versatility may remain more useful in consumer products.
By 2025, his conclusion is categorical: “We aren’t going to live in a world where one model rules it all.” Enterprises will mix open and commercial models across parameter sizes, choosing the best tool for each task; he expects narrow-AI use cases to grow even faster than the broader AI market.
4. Production inference is an operations and economics problem
Initial inference is accessible to a capable engineer, especially through open frameworks such as Predibase’s LoRAX. The hardware still bites: one replica of the largest models can require eight or 16 H100s, followed by decisions about procurement, autoscaling, and the serving framework.
Production is “another ball game.” Availability must move beyond 95% or 99% toward 99.9% or 99.999% SLAs, with fault tolerance, blue-green deployment updates, multi-region replication, monitoring, and enough throughput to support a business-critical application.
GPU expense makes total cost of ownership central. Dev points to smaller models and TurboLoRA, a software-defined technique he says can increase throughput by 2×; optimization determines how much useful output each expensive deployment produces.
Dev agrees that inference for the same base model—such as DeepSeek—will increasingly commoditize. His alternative is “intelligent inference”: deployment connected to post-training so models improve over time, combining serving and customization in one system rather than competing on an interchangeable endpoint.
5. Weekly breakthroughs reward adaptable plans and practical ambition
Predibase plans through 2026, but Dev says AI produces a breakthrough in expectations “about every week,” making an RFT launch vulnerable to OpenAI, Anthropic, Mistral, DeepSeek, Google, Amazon, or Meta absorbing attention. The fixed north star is specialized-AI tuning and serving; techniques and modalities, including vision and voice, remain adaptable.
AGI is “far from the world that I see in honesty,” even though Dev thinks models may already be near a Turing-test definition. His focus is “practical specialized intelligence” that creates step-function productivity gains in real business processes, including at organizations such as Fortune 200 company Marsh McLennan. He emphasizes that the industry has struggled to predict even 18 months ahead, let alone five, 10, or 20 years.
Dev’s concern is not whether useful ROI exists: CIOs deploying applications rarely ask whether paying 50 cents or a dollar per million tokens can create value. It is that organizations may chase “French-poetry-style” or elaborate multi-agent demos, miss today’s high-impact applications, and revive the disappointment created when AI overpromised outside the top 1% of organizations from 2012 through 2022.
His excitement is the reversal of “perfect and then ship.” Teams increasingly release “60% solutions” to test product-market fit and collect the data needed to improve models; continuous learning turns an initially imperfect launch from a concession into part of the development strategy.