Pioneers Insight Method Research Author
吴翼 Breaks Down OpenAI Operator’s Move Toward the Physical World
Back to Episodes

吴翼 Breaks Down OpenAI Operator’s Move Toward the Physical World

Summary

  • Operator’s key leap is not that it can click through websites, but that it extends o1 and o3’s single-shot deep reasoning into multimodal, closed-loop interaction with the environment. The model observes, acts, receives new feedback, and acts again; it can close pop-ups, backtrack after a failed click, and switch to a calculator after browser-based calculations fail. 吴翼 describes this as expanding from vertical depth of thought into horizontal generality—“backtracking happens in the action, not in its chain of thought.” That pushes AI’s value frontier from generating answers to executing tasks, but makes reliability a hard gate for commercial conversion.

  • Operator has no mysterious technical secret; its moat is the ability to perfect 3 capabilities simultaneously. First, a high-quality native multimodal foundation model; second, demonstration data and tasks that need not be massive in volume but must exhibit strong behavior; third, a reinforcement-learning system that can efficiently drive browser or operating-system interaction across thousands or even 10,000 GPUs. OpenAI’s first large project in 2016 was already a Web Agent, but it failed and led to roughly 20-30 layoffs because the team had reinforcement learning but no foundation model, labeled data, or even enough compute. The decisive formula 9 years later is the combination of foundation models and reinforcement learning.

  • Operator is more like “a multimodal, closed-loop o1” than o3 with a pair of hands. Its chain of thought is relatively short, relying mainly on trial and error through actions; it likely uses a foundation model different from o1 and o3, so the separate product entry reflects both model differences and a transitional product state. OSWorld is currently only in the low 30s, yet 吴翼 predicts that, at OpenAI’s pace, “this year it will definitely get you to 70-80.” The capability curve may still be steep, but the launch signals a technical breakthrough, not a commercially mature product.

  • Operator’s commercial model is fundamentally about selling time, not chat turns. If 3 reliable AIs could handle returns, procurement, tax filing, and other tasks in parallel, 吴翼 conservatively estimates that people’s effective time could be “multiplied by 2”; for enterprises, that means an “AI employee” cheaper than adding headcount. But OpenAI’s product revenue first provides cash and financing leverage—“users provide money, not data”—and while user data is a moat, it will not automatically become an intelligence flywheel the way recommendation systems do.

  • General-purpose model companies will occupy the consensus battleground; startups’ opportunity lies in the 1% of the long tail they have not covered. The number of software products and websites globally is enormous, so even if Operator covers 99%, the remaining 1% is still enough to support vertical B2B Agents for customer service, logistics, tax filing, orders, and more. 吴翼 says the US already has more than 10 such companies, while China has relatively few. GUI Agents are general-purpose but inefficient, and may only be an intermediate state: if Agents initiate a large share of requests 2-3 years from now, websites may open 2 doors—one for people and one for Agents.

  • Operator corresponds to L3 Agents in OpenAI’s capability ladder; a structural gap remains before L4 Innovators. L1 through L3 are all verifiable forms of instruction following: chat, deeper reasoning, and interaction with the external world. L4 must produce something “never seen before and better” without clear instructions or a standard verifier. 吴翼 says “the jump from 3 to 4 is a particularly large change” and may take more than 2-3 years. Lower-level L5 organizations may emerge first through passive coordination among multiple Agents, but autonomous organization will still depend on creativity.

  • 2025 looks more like an inflection point for Agent capabilities than the end state of commercialization. Multimodal foundation models, reinforcement learning, and interaction infrastructure have all entered a new phase, and the industry will turn open-loop text reasoning into closed-loop multimodal reasoning. But the field has already become highly consensus-driven: giants will advance “with tanks,” while resource-constrained teams must look for non-consensus opportunities along “country roads.” Operator is merely a signal that the pointer is shifting from 90 degrees toward the abstract world to 80 degrees toward the physical world; genuine physical intelligence, personalized memory, and inter-Agent collaboration remain open questions.

Deep dive

1. Operator’s Impact Comes from the Details, Not from Being First with the Concept

  • 吴翼’s first reaction was “as expected,” because OpenAI was bound to launch a GUI Agent in 2025. But when he watched the model act continuously, he was still “pretty shaken,” because the behavioral details were already strikingly human.

  • The clearest evidence was not that the model completed a particular web task, but that it proactively closed pop-up ads. When calculating a refund, it tried several times in the browser, failed, then switched to pressing keys on a calculator—changing tools to solve the problem just as a person would.

  • What 吴翼 values is that this behavior can be scaled and generalized through reinforcement learning: the model has not memorized the fixed steps for a particular website, but displays trial-and-error, backtracking, and alternative paths that “are exactly what a person would do.”

2. Closed-Loop Interaction Fills in the Horizontal Axis of General Intelligence

  • o1 and o3 demonstrated depth on focused problems: the user gives an instruction, the model thinks internally for 20 seconds, then returns an answer. There is no environmental feedback along the way and no genuine multimodal interaction; the system remains open-loop overall.

  • Operator changes the process to “observe—think—act—observe again”: the model clicks on a webpage, reads the resulting change in the external world, and decides what to do next. It does not just process its own tokens; it brings the constantly changing internet or software interface into context.

  • 吴翼 summarizes the trajectory through 2 axes: the vertical axis is depth of thought, while the horizontal axis is multimodality, environmental interaction, and breadth of generality. Operator does not reset the ceiling for pure logical intelligence, but it clearly expands OpenAI’s capabilities along the horizontal axis.

3. Operator Is an o1 Moving Broadward, Not a Direct Extension of o3

  • Asked whether it was a continuation of o1 and o3, 吴翼 qualified the answer: “not completely.” All 3 share the post-training and reinforcement-learning trajectory, but o1 and o3 push long chains of thought to the limit, while Operator emphasizes task breadth and continuous action.

  • Publicly visible chains of thought show that Operator’s internal reasoning is relatively short. Unlike o1 and o3, it does not contain long sequences with extensive backtracking; when it clicks the wrong place, cannot open something, or reaches a dead end, it backs up. Hence, “the backtracking is mainly in the actions, not in the chain of thought.”

  • 吴翼 therefore characterizes it as “a multimodal, closed-loop o1,” or an earlier-stage Agent o1, rather than a multimodal o3. If stronger chains of thought, visual understanding, and generalization are added later, the model still has substantial room to improve.

  • The separate product entry may also indicate that it uses a foundation model different from o1 and o3. The 2 capability sets can eventually be fused, but keeping them separate for now reduces product-comprehension costs and fits the role of an “intermediate transitional model.”

4. Multimodality Makes Generalization a Day-One Requirement for Operator

  • o1 and o3 could start narrow in mathematics, scientific understanding, coding, and pure text reasoning, then gradually address generalization. Operator has to deal with a wide variety of websites, software, ads, buttons, and tasks from day 1; a narrow model simply cannot function.

  • 吴翼’s most direct explanation is that “it has eyes” (“它有了眼睛”): visual input naturally brings stronger generalization, and Operator must therefore handle a broader range of tasks than o1 and o3.

  • Accordingly, Operator needs a more complex Reward Model, because it must handle diverse web operations and determine whether each one succeeded, rather than dealing with relatively concentrated pure-text or coding tasks.

5. 3 Engineering Capabilities Form the Real Moat

  • The first is a native multimodal foundation model that is good enough. Drawing on his own training experience, DeepSeek’s reports, and OpenAI’s model evolution, 吴翼 emphasizes that how much post-training can unlock is heavily constrained by how much language and visual potential the foundation model already possesses.

  • The second is high-quality human demonstrations and task data, such as closing ads, returning after failure, and switching to another tool. Post-training does not need the massive data volumes of pretraining, but the data and tasks themselves must be prepared in advance and accurately demonstrate behaviors the foundation model did not naturally possess.

  • The third is a scalable Agent reinforcement-learning system. Mathematical reasoning only requires the model to keep generating tokens; a web task requires the computer to execute mouse or keyboard actions after every segment of tokens, then send screen feedback back to the model.

  • Once training scales to thousands or 10,000 GPUs, efficiently keeping browser or operating-system environments waiting, executing, and returning feedback makes the training system far more complex. Supporting efficient training is itself a research problem.

6. CUA’s Novelty Lies in Specialized Training, Not in the Word “Agent”

  • OpenAI calls the new model Computer-Using Agent, or CUA. 吴翼 points out that Anthropic’s Claude had publicly demonstrated similar capabilities earlier, while “closed-loop interaction between a model and its environment” is itself a basic framework dating back to the reinforcement-learning era.

  • Traditional large models receive an instruction and produce a one-shot output, with context largely unchanged by external feedback. CUA outputs both thinking and action, waits for the environment to return a new state, and then continues reasoning from there; context therefore keeps changing throughout execution.

  • Earlier workflow Agents typically relied on Agent Prompting or other engineering techniques to wrap a general-purpose model not designed for interaction. CUA is a model trained on a strong foundation model and reinforcement learning, specifically customized for interaction and Agent use; the quality gap will be “very large.”

  • There is therefore no mysterious secret to replicating Operator domestically: the path is clear, and much of the infrastructure is already “half there.” The real uncertainty is whether every component can be pushed to the limit, and how far ahead OpenAI’s undisclosed “underwater” models are.

7. OpenAI Spent 9 Years Completing the Formula That Failed in 2016

  • OpenAI’s first large project after its founding was already a Web Agent: teaching a general-purpose visual agent to click through webpages. In 2016, when reinforcement learning was at its peak, the team tried to use a large LSTM, CommNet, to interact directly with the internet environment.

  • That attempt lacked a foundation model, human labels, and compute. Reinforcement learning alone could not learn general web capabilities from scratch, while engineering a complex environment required enormous investment. After the project failed, roughly 20-30 people were laid off.

  • 吴翼 attributes the reversal 9 years later to completing the formula: “Without a good foundation model, reinforcement learning alone doesn’t work; but a good foundation model alone doesn’t really work either.” Combining a multimodal foundation model with the reinforcement learning validated by o1 and o3 is what makes 2025 a breakthrough year for Agents.

8. L1 to L3 Expand the Scope of Execution; L3 to L4 Rewrites the Task Definition

  • 吴翼 describes the L1 Chatbot as a reactive system: a person says one thing, and the machine answers one thing. L2 Reasoners still involve only 2 parties, but the model plans and reasons through many steps internally before answering.

  • L3 Agents introduce a third party—the external world. The model must both think and observe changes on the internet or in software, completing actions without being directed step by step by a person. But the task is still defined by a human, and a verifier can still determine whether it has been completed.

  • The difficulty of L4 Innovators is that humans can provide only direction, not a complete instruction. The result must not merely be correct; it must be good and exceed the existing knowledge framework. 吴翼 believes “the jump from 3 to 4 is a particularly large change” and may not be achievable within 2-3 years.

  • The sequence between L5 Organizations and L4 is not necessarily fixed: when multiple pieces of software each have an Agent, a lower-level organization may form passively. But getting AI to propose an organizational structure and create one spontaneously still requires L4-style creativity.

9. Agent Economics Come from Reducing Human Intervention

  • Moving from Chatbots to Agents, the amount of human intervention decreases step by step while the scope of automation expands. Tasks such as tax filing, returns, and procurement used to consume one person’s attention; reliable Agents allow multiple processes to run in parallel.

  • 吴翼 does not exaggerate the efficiency gain as linear addition. Even with 3 reliable AIs, a person’s time may not be multiplied by 3, but “multiplying my time by 2” would already be highly valuable because it frees cognitive and operational time that previously could not be parallelized.

  • For enterprises, the logic translates directly into AI employees: if an Agent can truly deliver independently, its cost advantage over adding a secretary or employee should be easy to recoup. The business model is not selling the concept of a model; it is getting customers to “spend money to buy time.”

10. The More Autonomous the System, the More Important Product-Level Brakes Become

  • In public demos, Operator pauses and asks a human to take over before sensitive actions such as placing an order or making a payment. 吴翼 leans toward the view that this does not mean the model solved every safety issue during training; instead, product and safety teams continue monitoring at the outer layer and block actions by category.

  • The understanding side may come from a native multimodal version related to GPT-4o, but 吴翼 explicitly says he “doesn’t know which version of 4o.” The action side is simpler, consisting mainly of clicking and typing, and can be constrained through structured thinking, action, and special tokens.

  • Long-horizon planning does not change the principles of reinforcement learning. Some human data first demonstrates behaviors such as dragging pages, closing pop-ups, and correcting after failure; reinforcement learning in complex environments then elicits self-exploration and self-correction. The behavior only emerges once every detail has been trained correctly.

  • On the risk of loss of control, 吴翼’s judgment is explicitly conditional: if the training method is correct and safety is adequately ensured during training, the system has not yet entered a state of “complete uncontrollability.” He jokes about his mentor Stuart Russell: the fact that his mentor is still alive shows that an evil AI has not traveled back through time to stop him.

11. Fixed Tasks Do Not Need Multiple Agents for Now

  • 吴翼 is not bullish on using multiple Agents to complete a single web task in the near term. A large model with native multimodality, long context, memory, and strong reasoning can already handle a fixed objective on its own; multiple Agents may not add value.

  • Genuine multi-Agent systems will emerge passively when external services are also taken over by Agents—for example, when the website Operator visits has an Agent of its own. The collaboration will come from system boundaries rather than human-designed teams.

  • Over the next 1-2 years, he still expects a single Agent operating a human GUI to dominate. In 2-3 years, if Agents initiate 80% of human-work queries, websites may sell dedicated entry points that Agents can call directly, bypassing inefficient mouse clicks.

12. Operator Data Is More Valuable Than Chat Data, but Still Not an Intelligence Engine

  • 吴翼 agrees that “Chatbots are not the most effective form for extracting intelligence.” Everyday small talk does not, by default, require deep thought, and much of the user feedback improves tone and comfort rather than breaking through the model’s hardest capabilities.

  • Operator’s query distribution is different. Tasks people are willing to delegate to an AI for web interaction are usually more demanding, such as complex itineraries, tax filing, or repeated searches. Instructions, execution traces, and final success or failure naturally form task-outcome data better suited to reinforcement learning.

  • The outside joke that OpenAI is “collecting data again” through the Reinforcement Fine-Tuning API is not entirely unfounded. Reinforcement learning does not require huge amounts of data; 1 genuinely difficult task with an evaluable outcome may help the model fill in an entire capability gap.

  • But 吴翼 immediately adds a limitation: higher-value user data does not mean the product automatically generates the next generation of intelligence. “It may still be worse than labeling it yourself.” His causal direction is better intelligence first, followed by new products—not product usage forcing intelligence to improve.

13. OpenAI Separates Intelligence, Post-Training, and Delivery into 3 Layers

  • The organizational structure 吴翼 describes is as follows: a relatively small core research group advances pretraining and reinforcement training; a larger post-training team continues training and adjustment based on product needs and user data; the product team handles final delivery.

  • This division explains why researchers can focus on “the hardest problems” while the post-training team remains indispensable. A mathematics or science pipeline running successfully does not mean the model is ready for ordinary users, nor that it matches product requirements.

  • OpenAI’s continuous product launches are not simply “flexing.” It is a profit-seeking company operating in intense competition. Products get users to pay, and that money can then be used to improve intelligence—or leveraged to raise more money. The host’s summary was pointed: “Users provide money, not data” (“用户提供的不是数据,用户提供的是钱”).

  • AI data is therefore a moat, but not an automatic recommendation-system-style flywheel. Taobao or Douyin can optimize naturally as clicks accumulate, while general-purpose AI data is noisy and requires researchers to identify key gaps, clean the data, and process it. The loop is at most “semi-automatic.”

14. 2025 Will First Be About Raising Capability Scores; Commercialization Still Requires Reliability

  • 吴翼 calls 2025 “the year we see breakthroughs,” not the year commercial transformation fully lands. o1 is powerful, yet users still complain about its output, showing that there is still distance between a capability becoming visible for the first time and a product running stably and economically.

  • What we will see this year is pure text reasoning moving toward multimodal reasoning, open-loop answers moving toward closed-loop environmental interaction, and a large number of models trained specifically for Agents. He expects public leaderboard scores to rise sharply.

  • Operator currently scores only in the low 30s on OSWorld, which “doesn’t look like OpenAI’s style.” 吴翼’s explicit prediction is that “this year it will definitely get you to 70-80,” and he does not rule out the possibility that internal scores are already higher but have not been published.

  • On resources, he believes the GPT-4o-derived Operator model itself should be relatively small, with training unlikely to reach the 100,000-GPU scale. But outsiders cannot infer from the final training run how many rounds of research and experimentation came before it, so the total investment remains unknowable.

15. The Product Roadmap Will Expand from Screen-Clicking to Interfaces and Embedding

  • GUI Agents are only the first form. 吴翼 speculates that if Operator has already been integrated into Apple’s systems, it may call operating-system APIs; it may also work with enterprises to obtain code or dedicated interfaces, rather than imitate human clicks forever.

  • OpenAI does not currently offer an Operator API, but 吴翼 infers from its usual practice that some early partners may already have begun integration. The next versions could be Operator v2 and v3, or the product could receive a new name after switching to a different foundation model.

  • GUI’s advantage is that it is the most general interface in a “world designed for humans”; its drawback is inefficiency, making it more like an unavoidable intermediate state. If people eventually agree to delegate roughly 80% of their work to Agents, software interfaces and calling methods may adapt in the opposite direction.

16. The Physical World, Vertical Startups, and Non-Consensus Research Are Still at the Starting Line

  • 吴翼 divides intelligence into abstract text reasoning and visual-signal reasoning, then extends the discussion to reasoning in the physical world. Operator still only operates webpages, and webpages are essentially “text turned into images.” It has neither the spatial reasoning required to find socks or pack luggage nor any increase in o3’s pure logical ceiling.

  • His metaphor is that OpenAI previously advanced toward abstract intelligence along a 90-degree path; now the pointer has shifted toward the physical world at 80 degrees. “The hand seems to have just reached out, but has not really touched anything yet.” Commercially, it will first monetize the digital world; physical intelligence may also be pursued through investments in companies such as Physical Intelligence—“just build the brain.”

  • A foundation model is not the entry ticket to Agent entrepreneurship. Even if general-purpose models cover 99%, the remaining 1% of customer service, logistics, tax filing, and order workflows still represents a huge market. The US already has more than 10 vertical Agent companies, while China has relatively few, but opportunities remain in vertical areas not yet covered by large models.

  • Multimodality, reasoning, and Agents have become highly consensus-driven in 2025. Giants will occupy the main road “with tanks,” while entrepreneurs can take motorcycles down country roads. That has also prompted 吴翼 to reconsider his academic positioning: before ICML, he abandoned a submission after DeepSeek and Operator made him feel that some conclusions had already been stated by others, and pointed to personalized memory, human feedback, and genuine Agent-to-Agent interaction as open questions that may become important in 2-3 years.