吴翼

Reinforcement learning is not about one-step classification, but long-horizon decisions with no standard path and rewards judged only at the end. Hence Wu Yi’s line that “life is a reinforcement-learning process,” though Qu Kai pressed on the key limitation: in life, the reward function itself is often unknown. Wu’s answer was that much of the time, people must discover their goals through action and feedback rather than clarify the goal before they begin.

RLHF’s historical role was mainly to turn GPT from a model that could continue text into one that could follow instructions; it was not the same as making intelligence scale continuously. InstructGPT treated the prompt as the task and each token as an action, then trained a reward model on human rankings. Wu’s analogy: it was like taking “a smart Tsinghua or Peking University student and, after an internship, turning them into a formidable company employee”—without fundamentally making that person smarter.

The reasoning RL introduced with o1 is what many see as a second intelligence curve beyond pretraining: add inference-time compute, have the model generate a large number of thinking tokens, then submit the answer. The model can explore freely in the middle, while training rewards only the final result, making mathematics and other verifiable tasks the main proving ground. But there is still no satisfying theory for “why emitting more tokens makes it more accurate,” and OpenAI may have taken roughly 1.5 to 2 years to turn the slow-thinking concept into a working system.

Checking only the final outcome opens up exploration, but also creates problems around thinking length, cost, and generalization. DeepSeek-R1 may spend a long time thinking about “1+1,” while Anthropic can make simple questions consume little thought and complex ones more—but Wu explicitly said he did not know whether that came from routing among multiple models or better training. RL learned on science problems can generalize to the humanities, but SFT, RLHF, and a strong base model are still needed to pull an “extreme nerd model” back into a usable range.

Pretraining and reinforcement learning are multiplicative: the former supplies understanding, memory, and representations; the latter converts those capabilities into reasoning, decision-making, and action. OpenAI hit a wall trying to buy plane tickets on the web using RL alone in 2016. In robotics, a dexterous hand solving a Rubik’s Cube in 2019 was remarkable, but a 2020 attempt to tidy a table from visual input still failed, and the robotics team was disbanded in 2021. Language control only began to work clearly once pretrained multimodal representations such as CLIP entered robot training. “A primary-school-level person won’t get anywhere by thinking harder”; a strong base model remains the foundation of an Agent.

DeepSeek-R1’s most important industry contribution was not declaring a single winning route, but sharply reducing the technical uncertainty around reasoning RL. Wu’s lighthouse analogy: OpenAI pointed east in the dark, while DeepSeek effectively said, “Walk for a month and you’ll get there.” The real barriers still include a strong base model, ultra-long rollouts, parallel generation and grading, stable gradient return, and high-throughput training systems, while ByteDance, DeepSeek, OpenAI, and others continue exploring different routes.

The next competitive phase will move from single-round problem solving to Agents that interact with environments over multiple rounds, with differentiation along generalization, coding, and tool use. OpenAI’s Deep Research and Operator have already shifted the paradigm from “one instruction—one answer” to repeated observation, thinking, and tool calls. Wu believes the US remains clearly ahead in large-scale Agent RL; domestic reasoning and point-solution coding are approaching parity, but public products have not yet demonstrated equally fine-grained interaction training.

For investors and startup teams, near-term moats are more likely to lie in infrastructure, data, and iteration speed than in any isolated algorithm. Wu said bluntly that “infrastructure is far greater than everything else.” AReaL-Boba cut 7B training from roughly a week in its early days to 2 days and reported an AIME 2024 score above 60. Application companies should not assume the end state in advance: like Manus, they can use fine-tuning, APIs, and engineering to ship useful products while capabilities remain uncommoditized—“startups cannot think in terms of the end state.”

🔗 Original source & video: A Masterclass in Reinforcement Learning | A Conversation with Wu Yi, Assistant Professor at Tsinghua’s Institute for Interdisciplinary Information Sciences

Operator’s key leap is not that it can click through websites, but that it extends o1 and o3’s single-shot deep reasoning into multimodal, closed-loop interaction with the environment. The model observes, acts, receives new feedback, and acts again; it can close pop-ups, backtrack after a failed click, and switch to a calculator after browser-based calculations fail. 吴翼 describes this as expanding from vertical depth of thought into horizontal generality—“backtracking happens in the action, not in its chain of thought.” That pushes AI’s value frontier from generating answers to executing tasks, but makes reliability a hard gate for commercial conversion.

Operator has no mysterious technical secret; its moat is the ability to perfect 3 capabilities simultaneously. First, a high-quality native multimodal foundation model; second, demonstration data and tasks that need not be massive in volume but must exhibit strong behavior; third, a reinforcement-learning system that can efficiently drive browser or operating-system interaction across thousands or even 10,000 GPUs. OpenAI’s first large project in 2016 was already a Web Agent, but it failed and led to roughly 20-30 layoffs because the team had reinforcement learning but no foundation model, labeled data, or even enough compute. The decisive formula 9 years later is the combination of foundation models and reinforcement learning.

Operator is more like “a multimodal, closed-loop o1” than o3 with a pair of hands. Its chain of thought is relatively short, relying mainly on trial and error through actions; it likely uses a foundation model different from o1 and o3, so the separate product entry reflects both model differences and a transitional product state. OSWorld is currently only in the low 30s, yet 吴翼 predicts that, at OpenAI’s pace, “this year it will definitely get you to 70-80.” The capability curve may still be steep, but the launch signals a technical breakthrough, not a commercially mature product.

Operator’s commercial model is fundamentally about selling time, not chat turns. If 3 reliable AIs could handle returns, procurement, tax filing, and other tasks in parallel, 吴翼 conservatively estimates that people’s effective time could be “multiplied by 2”; for enterprises, that means an “AI employee” cheaper than adding headcount. But OpenAI’s product revenue first provides cash and financing leverage—“users provide money, not data”—and while user data is a moat, it will not automatically become an intelligence flywheel the way recommendation systems do.

General-purpose model companies will occupy the consensus battleground; startups’ opportunity lies in the 1% of the long tail they have not covered. The number of software products and websites globally is enormous, so even if Operator covers 99%, the remaining 1% is still enough to support vertical B2B Agents for customer service, logistics, tax filing, orders, and more. 吴翼 says the US already has more than 10 such companies, while China has relatively few. GUI Agents are general-purpose but inefficient, and may only be an intermediate state: if Agents initiate a large share of requests 2-3 years from now, websites may open 2 doors—one for people and one for Agents.

Operator corresponds to L3 Agents in OpenAI’s capability ladder; a structural gap remains before L4 Innovators. L1 through L3 are all verifiable forms of instruction following: chat, deeper reasoning, and interaction with the external world. L4 must produce something “never seen before and better” without clear instructions or a standard verifier. 吴翼 says “the jump from 3 to 4 is a particularly large change” and may take more than 2-3 years. Lower-level L5 organizations may emerge first through passive coordination among multiple Agents, but autonomous organization will still depend on creativity.

2025 looks more like an inflection point for Agent capabilities than the end state of commercialization. Multimodal foundation models, reinforcement learning, and interaction infrastructure have all entered a new phase, and the industry will turn open-loop text reasoning into closed-loop multimodal reasoning. But the field has already become highly consensus-driven: giants will advance “with tanks,” while resource-constrained teams must look for non-consensus opportunities along “country roads.” Operator is merely a signal that the pointer is shifting from 90 degrees toward the abstract world to 80 degrees toward the physical world; genuine physical intelligence, personalized memory, and inter-Agent collaboration remain open questions.

🔗 Original source & video: 88. A Technical Breakdown of OpenAI Operator with 吴翼: The Beginning of Reasoning’s Move from the Abstract World to the Physical World