Summary

Reinforcement learning is not about one-step classification, but long-horizon decisions with no standard path and rewards judged only at the end. Hence Wu Yi’s line that “life is a reinforcement-learning process,” though Qu Kai pressed on the key limitation: in life, the reward function itself is often unknown. Wu’s answer was that much of the time, people must discover their goals through action and feedback rather than clarify the goal before they begin.
RLHF’s historical role was mainly to turn GPT from a model that could continue text into one that could follow instructions; it was not the same as making intelligence scale continuously. InstructGPT treated the prompt as the task and each token as an action, then trained a reward model on human rankings. Wu’s analogy: it was like taking “a smart Tsinghua or Peking University student and, after an internship, turning them into a formidable company employee”—without fundamentally making that person smarter.
The reasoning RL introduced with o1 is what many see as a second intelligence curve beyond pretraining: add inference-time compute, have the model generate a large number of thinking tokens, then submit the answer. The model can explore freely in the middle, while training rewards only the final result, making mathematics and other verifiable tasks the main proving ground. But there is still no satisfying theory for “why emitting more tokens makes it more accurate,” and OpenAI may have taken roughly 1.5 to 2 years to turn the slow-thinking concept into a working system.
Checking only the final outcome opens up exploration, but also creates problems around thinking length, cost, and generalization. DeepSeek-R1 may spend a long time thinking about “1+1,” while Anthropic can make simple questions consume little thought and complex ones more—but Wu explicitly said he did not know whether that came from routing among multiple models or better training. RL learned on science problems can generalize to the humanities, but SFT, RLHF, and a strong base model are still needed to pull an “extreme nerd model” back into a usable range.
Pretraining and reinforcement learning are multiplicative: the former supplies understanding, memory, and representations; the latter converts those capabilities into reasoning, decision-making, and action. OpenAI hit a wall trying to buy plane tickets on the web using RL alone in 2016. In robotics, a dexterous hand solving a Rubik’s Cube in 2019 was remarkable, but a 2020 attempt to tidy a table from visual input still failed, and the robotics team was disbanded in 2021. Language control only began to work clearly once pretrained multimodal representations such as CLIP entered robot training. “A primary-school-level person won’t get anywhere by thinking harder”; a strong base model remains the foundation of an Agent.
DeepSeek-R1’s most important industry contribution was not declaring a single winning route, but sharply reducing the technical uncertainty around reasoning RL. Wu’s lighthouse analogy: OpenAI pointed east in the dark, while DeepSeek effectively said, “Walk for a month and you’ll get there.” The real barriers still include a strong base model, ultra-long rollouts, parallel generation and grading, stable gradient return, and high-throughput training systems, while ByteDance, DeepSeek, OpenAI, and others continue exploring different routes.
The next competitive phase will move from single-round problem solving to Agents that interact with environments over multiple rounds, with differentiation along generalization, coding, and tool use. OpenAI’s Deep Research and Operator have already shifted the paradigm from “one instruction—one answer” to repeated observation, thinking, and tool calls. Wu believes the US remains clearly ahead in large-scale Agent RL; domestic reasoning and point-solution coding are approaching parity, but public products have not yet demonstrated equally fine-grained interaction training.
For investors and startup teams, near-term moats are more likely to lie in infrastructure, data, and iteration speed than in any isolated algorithm. Wu said bluntly that “infrastructure is far greater than everything else.” AReaL-Boba cut 7B training from roughly a week in its early days to 2 days and reported an AIME 2024 score above 60. Application companies should not assume the end state in advance: like Manus, they can use fine-tuning, APIs, and engineering to ship useful products while capabilities remain uncommoditized—“startups cannot think in terms of the end state.”

Deep dive

1. Reinforcement Learning Optimizes the Outcome, but Does Not Define Life’s Goal

Wu Yi first drew the line between RL and traditional machine learning: every cat-or-dog sample has a standard answer, while a game requires many consecutive actions and reveals only at the end whether you won or lost. Reinforcement learning handles problems where “many decisions have to be made, and there is no standard answer for exactly how to make them.”
Ping-pong, Tetris, and DOTA all allow many viable paths: whether you turn left or right when leaving home may not matter; winning at the end does. Precisely because it covers a broader class of problems, RL is more general and more difficult than the supervised learning of the face-recognition era. It did not truly break into the mainstream until AlphaGo.
When Qu Kai pressed him on the fact that “life has no objective solution,” Wu acknowledged that RL still presupposes some reward. Reality is harder because “a large part of life is spent exploring what the reward function actually is”; people may reach the finish line only to discover they optimized the wrong goal.

2. RLHF Made Large Models Useful First; It Did Not Suddenly Make Them Smarter

A pretrained large model performs next-token prediction. Wu explained the source of its generalization with the saying: “Read 300 Tang poems and you may still not write poetry, but you can recite it.” Yet in the GPT-3 era, if a user asked, “Can you explain the moon landing for me?”, the model might simply repeat the question over and over.
InstructGPT treated the instruction as the task and every generated token as an action, with the final answer’s compliance serving as the reward. Because “following instructions” has no gold-standard answer, OpenAI had humans write demonstrations, compare and rank multiple answers, and train those preferences into a reward model, forming RLHF.
Wu stressed that the main value of early RLHF was alignment and instruction following. It turned a smart student into “a formidable employee at a company” after an internship, but there was no clear intelligence scaling law: more compute and preference data did not mean base intelligence would rise at the same rate.

3. o1 Turned “Think a Little Longer” into a Second Intelligence Curve Beyond Pretraining

As pretraining gains slowed, OpenAI began looking for a second curve. Wu credited the initial insight to Ilya: people think before tackling complex problems, while traditional large models answer immediately. The key was to give models the capacity for slow thinking.
The successful paradigm did not require a complicated new architecture. It was simply to “make the model emit more tokens”: generate 10,000 thinking tokens or more, then submit the answer. Once trained, the model might become more accurate as inference-time compute increased from 10 seconds to 30 or 100 seconds.
This route materialized in o1. DeepSeek-R1 adopted a similar approach, and the AReaL system also supports this class of reasoning models. But Wu’s candid answer was: “Why does it emit more tokens? I don’t know.” Treating the process as a rough draft is an intuitive analogy, not yet a theory of the mechanism.
OpenAI may have taken roughly 1.5 to 2 years to go from the idea of slow thinking to a working implementation. In hindsight, “write more, then answer” looks extremely simple. At the time, however, there was no established path from the need for thought to the expansion of tokens; the difficulty was persisting with the exploration.

4. Verifiable Answers Freed the Chain of Thought—and Created Runaway Length

Reasoning RL returns to the logic of games: “Explore whatever you want to emit in the middle; I only care whether the final result is correct.” Each token is an action, with no human-specified standard path for the entire chain of thought; the final answer supplies the reward.
Humans cannot reliably compare 10,000-token and 20,000-token internal reasoning. If both chains conclude that National Day is October 1, it is unclear which is better. That is why o1, R1, and similar models focus primarily on mathematics and other tasks whose answers can be checked objectively, allowing the process to diverge freely.
Qu Kai pointed out that cost and efficiency matter just as much. If the only reward is correctness, a model might spend half an hour pondering “1+1.” Wu thought Anthropic had done an excellent job controlling thinking length: simple questions get no extended thought, while complex ones get 10 seconds. Whether that comes from routing among multiple models or from training, however, “is unknowable” from the outside.
Capabilities learned on verifiable tasks do generalize: DeepSeek-R1 can also reason through humanities questions. But generalization alone is not enough. The “extreme nerd model” must ultimately be combined with a base model, followed by SFT and RLHF to preserve both reasoning ability and normal conversation.

5. DeepSeek-R1 Turned an Unknown Route into an Engineering Project Others Could Chase

Wu believes DeepSeek’s biggest contribution was proving that reasoning RL “really is doable.” His lighthouse analogy: OpenAI pointed east through 365 degrees of darkness; DeepSeek came back and told the industry, “The lighthouse is in this direction—walk for a month and you’ll get there.”
OpenAI may have taken 2 years from project launch to o1, while DeepSeek’s highly focused team may have validated the route in “a little over a month.” That does not mean the work was easy. It turned an unknown distance, cliff risk, and question of feasibility into an engineering problem whose parameters could be estimated.
2 prerequisites remain unavoidable: the base model must be at least at “high-school student” level before deep thinking can solve university-level problems; and the training engine must support ultra-long rollouts, massive parallel exploration, automatic grading, gradient return, and stable, efficient operation as outputs grow longer.
When Qu Kai asked whether this was already the optimal route, Wu did not give a definitive answer. Teams at DeepSeek, ByteDance, and elsewhere are still exploring in parallel, while DeepSeek’s 3.2.4 model has also highlighted coding. Most teams are still in catch-up mode because public models have not comprehensively surpassed R1.

6. Agent Training Upgraded Single-Round Answering into Repeated Interaction with an Environment

OpenAI’s RL is no longer limited to reasoning. Operator and Deep Research repeatedly go through “think for a while—act—observe the feedback—think again” within a single task, moving from one-shot instruction-answering to multi-round decisions under environmental state.
Multi-round interaction is not just single-round training repeated. Mathematical training can be split into generation, grading, and updating; a web Agent must at minimum add an interactive virtual webpage, while search results and page state feed back into the rollout. The more modules involved, the less stable the system becomes.
Wu’s working definition of an Agent is deliberately simple: if a model’s output affects software, hardware, or a virtual world beyond text, it is an Agent. Even outputting “on” and actually switching on a light crosses the boundary of a pure language model.

7. Real Model Capability Is Eating into Prompt-Driven Workflows

Systems from the AutoGPT and LangChain era also qualified as Agents, but their ability to act came mainly from prompt engineering: humans specified the role, steps, and output format in advance, then used scripts to connect function calls or prescribed text to real-world operations.
The change with Operator and Deep Research is that humans provide only the objective, while the model decides how to proceed. Qu Kai made the crucial point that users often do not know how a task should be decomposed, so a fixed workflow cannot cover genuinely unknown exploration.
Wu compared ComfyUI with GPT-4o. In the past, image editing required chaining multiple Diffusion Models and workflow nodes. A strong multimodal model can directly understand several images, interact continuously, and generate the result, causing large amounts of manual workflow to disappear as decision-making improves.
It is like teaching someone to cook: a beginner needs step-by-step instructions such as “turn on the heat” and “pour in the oil,” while a chef needs only the dish name and a few constraints. RL delivers more than a higher reasoning score; it gives the model the ability to unfold long-horizon steps on its own when the goal is ambiguous.

8. RL Hit the Wall Twice on Its Own; What Was Missing Was Always Pretrained Understanding

In 2016, OpenAI launched Universe and used work such as World of Bits to explore buying airline tickets with RL on the web. The team built webpages that could run locally, demonstrated entering a San Francisco-to-New York itinerary and navigating United’s site, and pursued a goal that was essentially computer use 9 years early.
The large-scale effort ultimately failed, and Wu said it triggered OpenAI’s first major round of layoffs. The problem was that RL alone struggled to extract understanding from pixels and clicks that could generalize across websites, while the available data lacked sufficient breadth.
The second collision came in robotics. The dexterous hand solving a Rubik’s Cube in 2019 was impressive, but the team’s 2020 attempt to tidy a table from visual input failed to produce a general capability. OpenAI’s robotics team was disbanded in 2021.
The turning point was pretrained representations. OpenAI’s CLIP first gave language and images strong representations through pretraining. The University of Washington team then incorporated it into RL or data-collection workflows and demonstrated language-controlled tasks such as folding clothes. Wu’s own failed embodied-agent work at Facebook in 2017 likewise taught him that “all the bottlenecks were in vision; they had nothing to do with RL.”

9. Pretraining and RL Are Multiplicative, Not Substitutes

Wu’s core formula is that pretraining supplies understanding, memory, and visual and linguistic representations, while RL supplies decisions, long-horizon thinking, and action. “Pretraining and reinforcement learning have a multiplicative relationship”: if either component approaches zero, overall intelligence cannot get off the ground.
Pure RL can learn to win on a fixed Go board, but it cannot naturally transfer to chess. Even if it memorizes the webpages of dozens of airlines, it may still merely remember click sequences when it encounters a new site rather than actually understand the page.
Is this the complete end state? Wu’s answer was, “I don’t know.” The best Agents today do come from strong base models combined with RL, but the algorithms and paradigms are still in their infancy. At the time, he expected the industry to see stronger decision models and general-purpose Agents by at least the end of that year.

10. Pretraining Has Not Failed; It Has Moved from Shallow Water into Synthetic-Data Deep Water

When Qu Kai repeated the claim that “pretraining no longer works,” Wu immediately corrected him: it has not stopped working; marginal returns have fallen. Once surface oil is exhausted, shale oil remains, and better data cleaning and more difficult data extraction can still improve models.
Multimodality depends especially heavily on synthetic data. The internet contains abundant text, but far less data in which images and text are combined with strong relationships between them. To train better multimodal understanding, teams must construct data deliberately and “move from shallow water into deep water.”
Model scale also needs to be reconsidered. Adding a complex RL system to a 1T or 10T model could make training costs prohibitive. Wu rejects treating distillation as inherently “shameful”: producing a small model like o3-mini is difficult in its own right.
Deep water may produce model personalities. Claude may not top every leaderboard, but it stands out in interactive coding; GPT may be better suited to Deep Research and web Agents; Grok may be more comprehensive. The divergence comes from data, training objectives, and RL methods—not from a single overall ranking.

11. RL Training Looks More Like Alchemy, and Iteration Speed Is an Algorithmic Advantage

Asked what it means to train better, Wu began with the simplest answer: look at the results. Hard metrics in mathematics and coding must be high enough, but leaderboard performance does not prove a model is genuinely good; real-world use and “feel” still provide the final test.
The process is an alchemical coupling of data, algorithms, parameters, and infrastructure. Like an olympiad teacher, matching 2 problems to a specific capability gap may immediately fix a weakness; repeatedly feeding the model elementary-school problems will not create a new capability no matter how much data is used.
Infrastructure directly determines the number of experiments a team can run. Wu’s team initially needed about a week to train a 7B model, later reducing that to roughly 2 days. Whether a team can iterate multiple times within the same week translates into a radically different pace of algorithmic progress.
Training curves are also deceptive. A model may rise steadily and then suddenly plateau, or show no improvement for a long time before producing an “aha moment.” It is therefore difficult to draw conclusions from an intermediate state; the only reliable approach is to let training run to completion, then inspect the hardest metrics and actual user experience.

12. Post-Training Cannot Be Separated from Pretraining; Understanding Remains the Foundation

Pretraining and post-training use different paradigms, so teams can divide the work but should not build walls between them. RL needs SFT for cold start, and the post-training team must participate in data selection. If it exposes a gap in the base model, it must push pretraining to add data and representations in return.
Wu believes AI teams most need a “break-out-of-the-silo mindset.” Algorithm researchers need to understand data and engineering, while infrastructure teams need to understand algorithmic requirements. An RL system cannot first be fully standardized and then handed to an independent engineering department to implement from a specification.
In multimodality, his view is that “generation is easy; understanding is hard.” Someone who has never learned Russian cannot understand a Russian instruction. Brute-forcing results with RL may eventually teach the model something, but at extremely low efficiency. Pretraining first makes the instruction intelligible; RL can then teach controlled generation much faster.

13. Paths That Look Inevitable Today Were Hidden in the Fog of War

Qu Kai’s hindsight question was: if the logic is so straightforward—train understanding first, then use RL with standard answers to train decision-making—why did it take so many years? Wu said the industry first had to spend years testing and retesting before confirming that RL alone could not produce general understanding; he himself did not fully accept this until around 2018.
The next step was not inherently generative pretraining either. BERT-style supervised learning, contrastive learning, and next-token prediction all competed as possible routes. Yann LeCun advocated the contrastive direction for years, while GPT-3’s scaling law required Ilya’s conviction, Alex Radford’s execution, and multiple coincidences before it became visible.
Once every step is known, the derivation looks obvious. At the time, however, it was “fog of war.” Wu’s summary is cutting: “The answers are often simple. If an answer isn’t simple, it means the answer is wrong.”

14. Talent Scarcity and the Startup Window Both Come from Capabilities That Have Yet to Be Standardized

RL researchers were scarce to begin with. Building environments, reproducing results, and clearing the mathematical bar are all harder, while paper citations are often “an order” below those in CV or NLP. Industrial use cases were also limited, preventing universities from building a sufficiently large pool of industry-ready talent.
Engineering is the third funnel. Many people have studied algorithms but never had the chance to scale them on serious infrastructure. That is why Wu moved from multi-agent work such as MAPPO to AReaL, hoping open source would lower the barrier to real training.
His own experience shows how cold the field once was. When he applied for a PhD in 2014, he did not even know what reinforcement learning was. After arriving at Berkeley, he caught Pieter Abbeel’s team going all in on deep RL, knocked on doors to find projects, and later helped produce the Best Paper-winning Value Iteration Network. The Turing Prize eventually recognized Sutton and other RL researchers; Sutton had studied psychology as an undergraduate.
It now makes sense for an application company to have someone who understands RL, because simple fine-tuning will eventually become widespread. But startups should not wait for the end state. Wu admires Manus precisely because, before the complete model had arrived, it used fine-tuning, APIs, and engineering to assemble a useful product as quickly as possible: “Startups cannot think in terms of the end state.”

15. Decision Models Will Split into Generalization, Coding, and Agent Branches

Wu sees one main scaling track for continued gains in overall intelligence, alongside 3 branches that could each grow into large markets: generalization, professional coding, and Agents/function calling. DeepSeek, Claude, and OpenAI each illustrate a different emphasis.
What users perceive is not only final accuracy. Mathematical answers, research reports, and website operations can be verified, while the quality of frontend code also depends on interaction experience. Benchmarks and continuous human use are 2 forms of evaluation that cannot substitute for each other.
A representative Claude detail is its willingness to ask follow-up questions. When debugging code, it may first ask about the environment, installed packages, and version instead of guessing at the bug. Understanding what happened in the past is a base-model capability; deciding when to ask, what to ask, and when to continue acting is closer to an RL decision problem.

16. AReaL Turned “It Runs, It’s Stable, It’s Fast” into an Open-Source Foundation

If RL is to become a post-training scaling law, the system must actually scale. Wu describes the value of a framework this way: “If you want to butcher an ox, you need an ox-cleaver.” The goal is to keep training running across tens of thousands of GPUs, ultra-long rollouts, and multiple modules.
Agent RL adds environmental interaction to pure mathematical training, and behavior is unpredictable. Departmental bottlenecks in a complex system can produce tenfold efficiency differences. Training 10x faster means nearly 10x more experiments in the same period, so infrastructure directly changes algorithmic progress.
The new AReaL release, AReaL-Boba—short for Ant Reasoning Reinforcement Learning—was developed in collaboration with Ant Research. It includes some integration with inference engines such as SGLang, with a focus on speed and stability. The 7B model could already be trained stably; the 32B model still fell short of the team’s expectations at the time.
The team released the source code, data, models, and evaluation scripts, describing the framework as among the “very fast” open-source options. Its 7B RL model scored above 60 on AIME 2024, which Wu believed might be the first public report of a 7B model reaching that level.

17. The US-China Gap Has Shifted from Problem Solving to Training Agents inside Environments

Wu sees the US lead most clearly in large-scale Agent RL within environments. He asserted that Claude’s Sonnet was produced through reinforcement learning and noted that OpenAI had said Deep Research uses RL. Public domestic models perform well in reasoning and point-solution coding, but still lag in fine-grained, multi-round Agents.
DeepSeek demonstrates both the value and the limits of focus. A small team pushed a text base model and reasoning RL to the extreme, but never built multimodality. After broadening its scope, OpenAI became “very much like Google” and may therefore have slowed down, but once the goal is clear it can still deliver strong products.
Qu Kai asked whether writing performance as pretraining × post-training implied that domestic teams lagged on both sides. Wu only emphasized that DeepSeek may have gone further in post-training, while its base model is also strong. He refused to quantify the gap in years and offered only “cautious optimism.”

18. A Maximum-Entropy Life Is Closer to Wu Yi’s RL Philosophy than a Local Optimum

Wu’s favorite but rarely cited body of work is diversity-driven reinforcement learning. Traditional RL finds a stable winning route in soccer and repeats it; humans get bored and try short passing, crosses, and headers. “That is what life looks like.”
Adding a constraint to the algorithm—that reward must be high, while each new solution must differ from every previous one—pushes it to discover strategies whose use is not yet clear. Qu Kai connected this to Texas Hold’em: once all top players learn from AI, their styles converge and the game becomes less interesting.
Wu turns the idea into life advice: do not choose only the lowest-risk local optimum at age 20; pursue “a maximum-entropy life.” Life cannot run infinite rollouts, but “you can choose 3 or 4 times, at least.” Choices that raise expected value also increase variance, but “go out and take a leap; at worst, you can leap back.”
When the reward is unknown, action itself is information-seeking. In his work on human-machine collaboration and multi-agent environments such as Overcooked, AI must probe and interact to infer what its partner is trying to do. Qu Kai added that the future can be assessed only about 1.5 years out at most; goals can be discovered gradually through the act of pursuing them.

Scaling "Thinking": Gemini 2.5 Tech Lead Jack Rae on Reasoning, Long Context, & the Path to AGI David Capital's Adam Patinkin Updates the Vistry Thesis $VTY