Summary

With one 5090, a total outlay of a few hundred dollars, and 2-3 weeks, 逯雨鑫 took a Llama 4 2B fine-tuning project on agentic TAU-bench from roughly 15 to 55-60. 曲凯 also noted at the top of the episode that 逯 had previously built a model by fine-tuning Gemma and Fable that reached No. 1 in Hugging Face’s open-source model rankings. His recipe: QLoRA, a $200 Claude Max subscription to synthesize targeted data, and more than 10 hours of actual training; hardware costs “absolutely would not exceed $100.” V1 took five days, V2 took 2-3 weeks—and he is not a PhD: “I taught myself the most basic things.”
He spent 95% of his time on data, which he sees as the real challenge. Synthetic data is “still very poor” and is mainly useful for targeted fixes, such as a small model’s overconfidence—declaring a task complete when it has not finished. Real data should account for 60%-70%; out of 5,000 samples, perhaps 500 need to be reviewed manually, with only a few thousand ultimately making it into the training set. “The real problem is having insight into the data.”
For individuals and many application companies, SFT is the most practical industrial-grade path, and 逯 believes RL performs worse than SFT. After extensive experiments on several 8-GPU H200 setups, he said on-policy distillation was the strongest approach, while off-policy methods and RL were more icing on the cake; good distillation means the anchor matches the target, because “a benchmark is just an exam.” Teams also need to watch for the capacity gap: Phi-4’s real-data scale, in his estimation, exceeds 3T, and when distilling into a small model, a teacher that is too strong and a student that is too weak can leave some capabilities “impossible to learn.”
Costs have fallen far enough for mid-sized and large application companies to build small in-house training teams: small models can cost a few thousand yuan to no more than RMB10K, large models RMB100K-RMB200K, and one H200 costing tens of thousands of yuan can support more than a dozen employees. Training is being productized: overseas, a company called Ans Off already offers an end-to-end fine-tuning product, and Hugging Face Jobs is working on something similar. 逯 expects costs to decline like mobile data as GPU supply improves—possibly becoming cheaper or even unlimited.
The two speakers’ reversal is that application companies could become the next new lab. AI labs will move toward higher-level research and pretraining, which may require 128-264 B300s running for months, while application companies with first-party data could take over post-training. 曲凯 said he had been pessimistic about application companies just a few weeks earlier; he now believes “model value is gradually shifting toward application companies,” and that AI labs buying data resembles ByteDance buying users early on—“the beginning is actually when it is lowest.”
The biggest tail risk is an open-source cutoff: 逯 pointed out that Qwen 3.7 released no small models and worries that, if this continues, application companies may have to do their own post-training, “closing the road completely.” His conclusion: anyone who wants to build in AI might as well start now. 逯 still joined an AI Lab because “there will definitely be AI Labs that win”; 曲凯 believes open source should persist for a long time as long as competition remains.
Local AI is 逯’s “big trend,” though the discussion also acknowledged that it is not yet something everyone needs. The drivers are refusals, cost, and privacy: users in cybersecurity and biological research say Phi-4 often refuses requests, while 逯’s own local-AI usage rate is “not high at present.” He cited the possibility that Apple could launch a 2T unified-memory computer in 2027 or 2028; if a future phone or computer can run a satisfactory local model, users may choose local AI for privacy and cost reasons. 曲凯’s cooler pushback: “Doubao will still have more users in the end”; privacy is more likely to be pushed by application companies than chosen proactively by users.

Deep dive

1. A Nontraditional Background Can Still Get You to Fine-Tuning: The Barrier to Training Models Is Systematically Overstated

逯雨鑫 started as a data engineer. He began using GPT when it first became publicly available in 2022-2023, later pursued graduate study in AI, and only joined an AI lab after gaining attention through personal projects. “I taught myself the most basic things … Graduate school is more about giving you a direction. As long as you are headed in the right direction, self-study is entirely possible.”
His framework divides AI into academic-grade and industrial-grade work. The two standard industrial paths are SFT and RL. “Once you fill in the basics, anyone can gradually do this.” For most people, however, starting with fundamentals such as MLPs is “not necessary.”
曲凯 asked whether this was something everyone was already doing and the outside world simply did not understand. 逯’s answer: people genuinely are not doing it, mainly because they make the task far more complicated than it is.

2. The Project Ledger: A Few Hundred Dollars, 2-3 Weeks, and TAU-bench From 15 to 55-60

曲凯 said at the start of the episode that 逯 had previously built a model by fine-tuning Gemma and Fable that reached No. 1 in Hugging Face’s open-source model rankings. The personal project discussed in detail this time, however, was explicitly a fine-tuning project on Llama 4 2B. When discussing model selection, 逯 also said Llama 4 was very poor at Chinese in its “12B model” configuration, so this project should not be rewritten as a 12B model.
The setup: one 5090 with QLoRA, plus a $200 Claude Max subscription for synthesizing targeted data. Renting a 5090 cost roughly $1-$2 an hour, in his estimate; actual training took more than 10 hours, and hardware costs “absolutely would not exceed $100.” V1 took five days, V2 took 2-3 weeks. V2 also incorporated valuable real-trace data from Phi-4 supplied by Hugging Face followers or community users.
On the agentic TAU-bench benchmark, the original model scored about 15 and the fine-tuned version reached 55-60. Other benchmarks declined to varying degrees: the gains were specific to the target domain. 曲凯 summarized the approach as using a stronger teacher to generate data, then distilling a small domain-specific model through SFT so that it could outperform the base model in that domain. 逯 agreed.

3. The Full Loop: Define the Goal, Choose the Base, Build the Data, Pick the Method, Iterate

逯’s process starts by defining the objective around the business rather than chasing whatever is hot, then selecting a base model. One major problem he encountered with Llama 4 was its weak Chinese capability: when discussing the “12B model,” he said it spoke Cantonese most of the time. For Chinese-language work, Qwen may be a better choice.
The data pipeline can involve playing the user yourself and having an AI generate the interactions, or using two models—for example, GPT-5.6 as the user and another model as the model actually being used. Public data can also be sourced from Hugging Face, ModelScope, and similar platforms.
The standard industrial methods are SFT and RL. He ultimately chose SFT for his own project, arguing that RL is “overhyped” and performs worse than SFT in practice. Other methods such as OPD are relatively expensive for individuals and many application companies.
Training scripts, public datasets, and benchmarks are generally available in open-source form; they can be downloaded and adapted with an agent. He completed the entire personal project using Claude. He does not like Codex, arguing that GPT is better suited to execution and coding than to planning.

4. 95% of the Time Goes Into Data: Synthetic Data Can Only “Treat Specific Illnesses”

逯’s core lesson is that he spent 95% of his time on data. The real challenge is “having insight into the data,” which he sees as the biggest bottleneck across AI today. V1 took roughly three days for data and one day for training; during V2’s two-week cycle, about 12 days went into data.
Synthetic data is currently not very good and is mainly useful for targeted fixes. One problem with Llama 4 2B was “overconfidence”: on TAU-bench, it would announce “I completed it” despite not actually finishing the task. Public datasets and ordinary real-world usage rarely contain this kind of targeted example, so synthetic data is needed to address it directly.
Most of the data should come from real usage, including the process of using the model oneself and publicly available real-use data from Phi-4. Real data should account for roughly 60%-70% of the total.
Manual review cannot be skipped. A 5,000-sample dataset may require personally reviewing 500 entries. Telling an AI “help me create data to fix overconfidence” can produce results that are “completely wrong”; the user has to keep pointing out problems, confirming the target, and regenerating the data. It may be possible to generate nearly 10,000 samples, but after filtering, the actual training set is typically on the order of a few thousand.

5. The Capacity Gap and the Definition of “Good Distillation”: The Anchor Must Match the Target

The result of training often cannot be judged from the data alone; each version has to be tested. Even when the data has been checked by AI and spot-checked personally, training may still produce no improvement. One possible explanation is a capacity gap: the teacher is too strong and the student too weak.
逯 used Phi-4 as an example, saying its real-data scale “should be above 3T.” Distilling those real traces and answers into a 12B small model creates a large capability gap, leaving some content the student simply cannot learn.
After extensive experiments on several 8-GPU H200 systems, he said on-policy distillation was clearly the strongest approach. Off-policy methods and RL are more icing on the cake than tools for actually solving the core problem. SFT, targeted data, and high-quality public data are the practical distillation methods he favors.
The quality of a distillation run is not determined by its label but by whether the anchor matches the objective. If the target is a safety boundary but the benchmark is used as the anchor, that is bad distillation. If the model’s distribution is genuinely pulled toward the target, it can still be good distillation even with bugs or a lower score. For many application companies, a benchmark is just an exam; real-world usage matters more than anchoring to a benchmark alone. His project was anchored to agentic and coding performance, which helped it attract more downloads and use.

6. The Cost Ledger and the Traps to Avoid: Qwen Has the Fewest

逯 estimates that an application company can train the small model it wants for a few thousand yuan to no more than RMB10K; a large model may cost RMB100K-RMB200K. The serving side brings separate architecture questions involving SGLang, vLLM, and others. For just a few or a dozen-plus concurrent users, one H200 costing tens of thousands of yuan should generally cover usage by more than a dozen employees.
On model selection, he considers Qwen the least troublesome option, with training, infra, scripts, and deployment all relatively strong among small models. Llama 4, along with a 30B model he recalls may have been called Muse, uses a less mainstream architecture and is more complicated to train.
The most important mindset is not to give up. Do not trust AI too much; check the source material promptly and study real examples of successful projects.

7. Model Training Is Becoming a Product

曲凯 asked why there was still no one-stop service covering the entire pipeline. 逯 said that, apart from RSI, almost every company is considering the direction. Overseas, a company called Ans Off has already built a product that lets users choose a model, upload data, and fine-tune directly. It is relatively well known on Hugging Face, though expensive.
Hugging Face has also offered Hugging Face Jobs specifically for this kind of work. Vendors in China and overseas are moving to turn training into a product; infra, particularly on the serving side, remains difficult.
逯 believes a mid-sized or large company could handle the work with a 3-4-person team. GPUs, memory, and RAM are currently expensive and in short supply. Once supply improves, he expects costs to fall like mobile data prices—from an initial RMB30 toward something cheaper, or even unlimited.

8. The Reversal: Application Companies Could Become the New Lab as AI Labs Move Up the Stack

曲凯 cited signs including sovereign AI, the data-labeling company Mokao, Harvey, and Fireworks, as well as a US Sequoia article about AI application companies becoming the new lab. 逯 expects the number of AI labs and the division of value between them to change. Small application companies can rely on AI labs early on; once they reach several million or tens of millions of users, they can build their own AI teams.
曲凯 compared the shift with the recommendation-algorithm era: companies that own the application, use cases, commercial loop, and user data ultimately develop stronger algorithmic capabilities than a neutral third party providing recommendation algorithms. He said that, whereas he had been pessimistic about application companies just weeks earlier, he now believes the value of AI labs and application companies will rise together, with model value gradually shifting toward application companies.
逯’s reversal scenario is that the AI labs that survive will move toward research, frontier development, and pretraining, while application companies take over post-training and retain their own models because they do not want to share their data. Pretraining will remain primarily the domain of AI labs and may require 128-264 B300s running for months. In his view, the core value of Kimi and 智谱 lies in pretraining.

9. Data Is Like Early User Acquisition: Why He Still Joined a Lab—and the Open-Source Cutoff Risk

曲凯 sees AI labs paying for data as similar to ByteDance paying for users in its early days. 逯 agrees that real-use traces are highly valuable: when a user asks GPT-5.6 to write a podcast script or revise content, the process by which the AI actually solves the problem is valuable data in itself.
When companies offer free access to their models, 逯 believes they may need to deploy a large number of B300s, with rental or purchase costs reaching hundreds of thousands or several million yuan. Free access is both marketing and a way to collect real-world data for the next generation of models.
He still chose to join an AI Lab because he believes “there will definitely be AI Labs that win,” and wants to join a company with a chance to move toward research, RSI, high research level, and pretraining.
His biggest long-term concern is that if AI labs stop open-sourcing, it could be “a fairly devastating blow” to application companies. He cited Qwen 3.7 releasing no small models and warned that, if this continues, application companies may have to do their own post-training, “closing the road completely.” Application companies should therefore start iterating on their own models now. 曲凯 believes open source should continue for a long time, at least as long as competition remains.

10. The Local AI Debate: Refusals and Cost Are Real Drivers; Privacy May Be Pushed by Companies

逯 believes local AI “will definitely” develop, arguing that data sent through APIs may be visible to the upstream provider. He also criticized some proxy services as operating in a gray area and potentially selling data, and mentioned cases around him involving hacker threats and data leaks. As Nvidia and Chinese chipmakers advance, he believes everyone may eventually have their own local AI.
He cited a forecast that Apple could launch a 2T unified-memory computer in 2027 or 2028, with 2T of memory potentially enough to run the GPT-5.2 of that era. If a future phone can run a satisfactory 27B model, users may simply use a local model rather than call an upstream API.
陈皮 asked about the gap between local models and SOTA. 逯 acknowledged that his own local-AI usage rate is “not high at present,” but said he had received extensive feedback from cybersecurity and biological research, where Phi-4 frequently refuses requests and users can only consider locally deployed models of middling capability. He also said that after Hugging Face was attacked by OpenAI, the issue was ultimately resolved with a Chinese domestic model deployed locally. These were all statements made by 逯 on the program.
Novel writing is another cost-sensitive use case: producing a draft can cost several thousand or even tens of thousands of yuan, while local AI assistance can reduce the cost substantially. 曲凯 argued that users may not actively care about privacy and that Doubao may still end up with more users; local AI is more likely to be pushed by application companies such as Apple than chosen proactively by individual users. 逯 agreed, concluding that convenience and solving the problem are what matter.

NVIDIA's $96.2B Quarter, China's 200,000 Fake Accounts, & OpenAI's New Chip | EP #284 Factory's Reyes: Anthropic's $2T coding bet; 80–90% of neo-labs die