E201｜The Next AI Frontier: Reinforcement-Learning Pretraining and AGI’s Inflection Point (Part II)

Summary

• The key divide in Agent training is not just data scale, but whether a task can be closed-looped through an objective and a Verifier. Tasks with existing text, image, and video collaboration data can first learn via SFT, then use RLHF to align preferences; but if a task requires 50 tool calls, no such complete trajectory exists online. Goal-driven tasks in code, math, physics, and supply chains need counterfactual generation of new solutions, followed by ground-truth verification—“If you can find a Verifier, you can consider the problem solved.”
• Within OpenAI’s five levels of AGI, the real technical chasm is not chat to reasoning, but Level 3 Agent to Level 4 Innovator. 朱哲清 sees the framework as a product-capability taxonomy: Level 1 depends on the definition, Levels 2 to 3 may not represent a huge leap, and the gap between Levels 4 and 5 may not be that large either. The hard part is whether a Verifier can generalize from known knowledge to judge new knowledge discovered by AI itself. “The gap between 3 and 4 is far greater than the gap separating 1, 2, 3 from 4 and 5.”
• RL pretraining could push AI toward knowledge unknown to humans, but it could also produce strategies that run while remaining unintelligible to people. Current attempts may still carry prior knowledge unavailable through pretraining; if developed further, code could have completely correct inputs and outputs while its internal variables, operations, and compilation logic look like gibberish. “Human readability” itself is difficult to encode as a rule-based check. 朱哲清 therefore sees reward design as a potential regulatory target, because the incentives assigned to an Agent during training determine its eventual behavior.
• RL’s cost disadvantage is real, but complex tasks may eventually leave companies no choice. 泓君 cited the industry view that RL delivers roughly 2x better results at roughly 10x the token consumption. 朱哲清 did not confirm the ratio, but explained that RL must rely solely on a reward function to reach its objective, while SFT already has standard answers; “RL fine-tuning will cost more.” Once tasks become complex enough to have no standard answer, the cost-effectiveness comparison breaks down: “Problems will become more complex in the future… and then there will be no choice.”
• Meta’s rationale for acquiring Scale AI may not be a return to a pure-text data race, but filling the medium-term annotation gap in multimodal and robotics. 朱哲清 summarized the resource cycle as compute in the short term, data in the medium term, and RL optimization over the long term. Image aesthetics are hard to standardize, video details are difficult to parse, and annotators may not even understand a robot’s intermediate actions. Multimodality may first require better parsing and reward-model capabilities before a Judge or Verifier can be built and RL fine-tuning can begin.
• Pokee AI is betting on a proprietary “reasoning + tool sequence” model that delivers the result through a single API and one line of prompt. A browser or virtual machine can compress thousands of tools into one generic interface, but at high cost, low speed, and with weak generalization in professional scenarios. Pokee instead accesses tools directly and hopes eventually to combine semantic understanding, planning, tool selection, and result generation into a single model. But 朱哲清 stressed: “Model capability determines your product’s floor, while product details determine its ceiling.”
• Scarce RL talent and negative-gross-margin Agents will jointly drive the next round of capital bifurcation. 朱哲清 estimates that “the entire industry has only a few dozen people” who can truly optimize RL well. Meanwhile, Agent companies may grow faster and lose money faster, with gross margins potentially reaching negative 50%. The technical route determines whether a company survives; the founder and team determine product differentiation. Over the next 1 to 2 years, the market may see a “big fish eating small fish” wave in which small teams are acquired.

Deep dive

1. As Agent Chains Lengthen, SFT Runs Out of Training Data

• 朱哲清 first distinguishes two RL architectures: one uses language models and tokens as the decision units, while the other treats the Agent’s actions as the decision points. “Neither approach is inherently better or worse”; the difference lies in the use case, while both train against an ultimate objective.

• For text, image, and video tasks with abundant collaboration data, SFT is usually enough to establish a high level of capability. RLHF can then fine-tune the model to human preferences, because internet data is “mixed with good and bad”: pretraining learns the overall distribution, which does not inherently equal the answers humans prefer.

• Tool chains break this data logic. A single tool call can be handled through supervised learning, but the complete trajectory corresponding to “these 50 tool calls” exists neither online nor in the output of almost any human producer. If SFT is insisted upon, the only option is manual annotation.

• Math and code still have relatively large sample pools, but goal-driven professional tasks in physics, finance, urban planning, research execution, and supply chains generally lack data. 朱哲清 advocates generating counterfactual solutions that have never appeared before, then having a ground-truth Verifier judge them. This is “when reinforcement learning shines brightest.”

2. The Real AGI Chasm Lies Between Levels 3 and 4

• 泓君 used OpenAI’s five-level framework to locate the industry’s progress: Level 1 is a ChatGPT-style chatbot, Level 2 is reasoning AI, Level 3 is an Agent capable of completing multistep tasks such as booking flights and hotels, Level 4 Innovator can discover new molecules or create new tools, and Level 5 can independently take responsibility for an entire organization.

• 朱哲清’s objection is that this is primarily a product-capability classification, not a technical-capability classification. Level 1 depends on the definition, there is no necessarily huge jump from 2 to 3, and the capability distance between 4 and 5 may not be large either. “The main thing is that there is a huge gap between 3 and 4,” because verification cannot smoothly extrapolate beyond the domains known to humans.

• His arithmetic example makes the problem clear: an Agent that can verify 20-digit addition still cannot judge a subtraction answer if it has never seen subtraction. The next step is both to derive a new Verifier directly from the human-provided relationship between subtraction and addition, and to use existing knowledge as an anchor to explore and extend the boundary of verifying future knowledge.

• If a Verifier can adapt to new outputs that depart from human experience, AI may follow paths humans never imagined, like AlphaZero, and even discover new physical theorems. This is not the same problem as hallucination. 朱哲清 also believes conflicts between the objectives of multiple Agents could create deadlocks resembling office politics or race conditions; 泓君 further raised the “paperclip problem.”

3. RL Pretraining Can Discover New Knowledge—and Produce Answers Humans Cannot Understand

• 朱哲清 has not yet seen a “major breakthrough” in Verifier generalization; the main path remains distillation of human knowledge. Although RL pretraining is being discussed by more people, it is “completely counterfactual learning”: the system may find correct solutions humans cannot understand, such as code that runs while its variables and operational logic look like gibberish. Similar attempts they are making still contain prior knowledge that cannot be obtained through pretraining.

• This makes reward definition a fatal constraint. If the reward checks only inputs and outputs, the model may sacrifice readability; but “human readability cannot be solved with a rule,” which returns the problem to something unverifiable. 泓君 saw the risk rising. 朱哲清 distinguished this from Hinton’s neural networks, which are mainly used to represent human knowledge, whereas counterfactual knowledge discovery or strategy discovery requires reinforcement learning.

• 泓君 cited the industry view that RL delivers roughly 2x better results at roughly 10x the token consumption. 朱哲清 explained that the cost gap reflects the scarcity of information: RL has only a reward function and must find its own path to the objective, while SFT already has standard answers. When complex tasks are still limited, the two remain alternatives; once standard answers disappear, “it is no longer a matter of choice.”

4. Multimodality Turns Data Annotation Back into the Medium-Term Bottleneck

• Asked why Meta would acquire Scale AI if RL can reduce annotation, 朱哲清 speculated that Meta may be filling gaps in multimodality and robotics. The declining importance of text data does not mean image, video, and action data can skip annotation, because verification first depends on the model accurately parsing the input.

• Current multimodal models still skew toward text capabilities and struggle to parse many details in images and video. Images also lack natural ground-truth answers. One possible path is to use data to train the base model and reward model, raise multimodal capability, convert input-output capability into a Judge or Verifier, and only then move into RL fine-tuning.

• The difficulty of image annotation is not simply a question of how many people to hire: for the same product image, “100 people may give 100 different answers.” Complex rubrics must be designed and annotators trained. Robotics is harder still: people may not understand why an intermediate action occurred, while the robot’s plan may be to complete that step before reaching a later objective.

• 朱哲清’s resource timeline is: compute was the short-term bottleneck; multimodal data is the medium-term bottleneck; RL optimization remains the long-term bottleneck. He believes that after GPT-4o, the iteration path seen in code and text is moving sideways into multimodality, while aesthetic, video, and action annotation create new alignment problems.

5. Pokee Uses a Proprietary Model to Orchestrate the Tool Chain

• Pokee AI wants enterprises, developers, and professionals to have a ChatGPT-like simple experience when calling an Agent: “one simple API, one line of prompt,” and a final deliverable directly, without managing browser environments, virtual machines, or extensive infrastructure handoffs.

• In terms of technical trade-offs, a browser is equivalent to “using one tool to replace thousands of tools,” reducing the number of tools. Pokee instead tries to compress more capability into the model and expose the internet’s tools directly. Its target is not a single Agent or a single tool, but the complete tool sequence of “reasoning plus tool selection.”

• 朱哲清 said Pokee has already used its proprietary model to complete substantial model calls and tool selection. In the future, it may also merge in the language-model component, allowing the same Agent to handle user input, semantic understanding, planning, tool selection, and final output, with fewer handoffs between infrastructure layers.

• The PostHog case shows the limits of the browser route: the Agent must read user behavior and metrics, import analytics scripts, generate multiple segmentation charts, write them into a report, and output a PDF. A browser or virtual machine is both expensive and slow, and the model has not seen this kind of specialized webpage during training. Direct tool access is the only way to obtain precise data and complete the workflow.

6. The Model Sets the Floor; Product Details Set the Ceiling

• 朱哲清 divides himself in two. As a researcher, he favors end-to-end training in increasingly general environments, because that may produce a stronger model. As a product builder, he prefers more determinism and control over details, because “user experience is not proportional to model capability.”

• Pokee once completed a cross-platform posting workflow end to end but failed to return the published link to the user. Tool selection was correct throughout, yet the user still had no idea where the content had been posted. The failure was not in model capability, but in the product’s failure to make the process and result visible.

• When generating Google Slides, Pokee also moved from returning only an external link to embedding the page directly, allowing users to view and edit it in place. 朱哲清’s product conclusion is: “Model capability determines your product’s floor, while product details determine its ceiling.”

7. Model Plasticity and Reward Design Are Long-Term Constraints on RL

• 泓君 introduced Richard Sutton as the 2025 Turing Award winner. 朱哲清 said Sutton and his own advisor were close friends and that he met Sutton many times while pursuing his PhD at Stanford. He especially admired Sutton for not giving up while battling cancer, spending several years out of work, and during the period when RL was dismissed as “mysticism.” His lesson was: “If you believe the first principles are right, do not give up.”

• Sutton repeatedly emphasized model plasticity to him: a model cannot be trained indefinitely. After reaching a certain point, it can collapse and may suffer catastrophic forgetting. 朱哲清 compared it to filling a sponge with water: what comes out when more water is poured in is not necessarily new water; it may be critical knowledge already inside. If the model forgets addition, subtraction, and multiplication, the rest of its knowledge system may collapse with it.

• 朱哲清 believes models are growing linearly in scale while data is growing exponentially, so the industry will inevitably hit a wall. Pokee has already seen performance suddenly fall sharply when training smaller models; only moving to a larger model allowed training to continue. Continual learning is therefore not some distant abstract problem.

• Another line of Sutton’s work concerns rewards: a single-step incentive may appear reasonable, yet diverge from the designer’s intent once rewards accumulate over a sequence. Balancing becomes even harder when optimizing 4 or 5 objectives at once. The generalized value function attempts to have the model estimate the value of multiple objectives in the same state, then decide how to trade them off.

8. Only a Few Dozen People Can Optimize RL Well; Research Is Moving from Dedicated Verification to General Agents

• 朱哲清 estimates that “the entire industry has only a few dozen people” who can truly optimize RL well. The talent base mainly comes from early OpenAI, Pieter Abbeel’s students, Sergey Levine’s students, Richard Sutton’s students, and the David Silver-centered DeepMind lineage. Microsoft’s John Langford was also an early theoretical pioneer.

• From 2015 to 2018, AlphaGo, AlphaZero, MuZero, and StarCraft pushed RL to a peak. David Silver’s camp leaned toward formal verification: when rules are explicit and outcomes are knowable, train a dedicated Agent to solve a single problem. The Bay Area, with its concentration of OpenAI- and DeepMind-linked talent, is more likely to be the current center.

• OpenAI’s early Gym environments tried to demonstrate goal-directed capability through simple games, but also led the industry to view RL as something that “stops at games.” 朱哲清 acknowledges that RL today is still mainly an optimizer layered on top of large language models. If RL pretraining can truly train a general decision-making system from scratch, that would constitute a substantive breakthrough.

9. Agent Companies Must Survive on Gross Margin Before Differentiating and Consolidating

• Discussing technology startup fundraising at an ICML panel, 朱哲清’s strategy was not to hard-sell a technology outside market consensus, but to “use the market’s consensus, then add a logical leap on top of that consensus”: first explain the problem the market already recognizes, then prove that your implementation and unfair advantage are different.

• Looking at the different approaches of Genspark, Manus, OpenAI’s ChatGPT, and Fellou, he expects the sector eventually to differentiate in the same way as Mistral, OpenAI, and Anthropic. Technology determines whether a company can survive; founders and teams then steer the product toward different areas of focus.

• The first survival constraint for Agents is cost. A company may raise a great deal of money yet “the faster it grows, the faster it dies.” If spending continues to exceed revenue, gross margin could reach levels such as negative 50%. The next round of investors will ask whether technical upgrades can bring gross margin back into positive territory.

• Over the next 1 to 2 years, 朱哲清 expects the market may see “a very crazy wave of big fish eating small fish.” Companies valued near $1B are relatively difficult to acquire, but smaller teams that are easier to sell may have a high probability of being acquired.