111: 朱哲清’s Pokee.ai Agent Playbook: RL as the Backend, Language Models as the Frontend | Agent #3
Summary
Poke AI’s core bet is to use language models as the frontend for understanding intent and interacting with users, while putting RL-trained non-language models in the backend to handle planning, reasoning, and tool calls. Bill argues that if 50 tools each require 1,000 tokens of documentation, tool context alone reaches 50,000 tokens; once memory, web results, and a dozen-plus execution steps are added, a single task can run into more than 1 million tokens, making language models prone to hallucination. Over time, the LM may look more like a UI than the Agent’s decision-making core.
The product wedge is not generating more content, but letting Agents write into real work accounts across platforms. Poke AI plans to launch with roughly 1,000 callable sub-tools spanning dozens of platforms; it does not currently use MCP, but plans to support MCP and Agent-to-Agent, while offering a simplified JSON-based integration path. It prioritizes official APIs and code, using browser automation only where necessary.
Bill’s opening summary of a strong general-purpose Agent included four traits: faster than humans, minimal human intervention, able to both read and write, and low cost. Poke’s demos take about 60 seconds when approval is required, and potentially 10–20 seconds without it; Bill says the per-task cost is roughly one-fortieth of comparable products. Because tasks write into user accounts, the product supports both controlled and autonomous modes.
The commercial entry point is developers and prosumers, moving bottom-up into the enterprise; the moat is defined as workflow attachment, not point-model technology. For users in social media, marketing, legal, and finance, the pain point has shifted from content generation to cross-platform distribution and operations; enterprise workflows spanning dozens or hundreds of steps are the larger end market. Bill believes integrations, historical files, account relationships, and usage habits can create deeper user lock-in.
The technical barrier is effective exploration across a large tool universe, not enduring overwhelmingly negative signals through random sampling. When only one of 10,000 tools produces the right answer, traditional supervised or contrastive-learning methods may repeatedly receive the wrong signal. Poke AI wants to use smarter exploration to skip known-bad tools and paths, reducing the data required for training.
Market education shaped when Poke AI could move back from a vertical application to a horizontal platform. When it raised money in fall 2024, investors said Bill was “six months ahead of the curve”; after DeepSeek and related work from other companies created broader market consensus, the company says it heard from more than 100 investors and dozens of large prospective customers. After releasing its demo in March, it generated more than 800 waitlist sign-ups in a week; after launching in March, another 800-plus people signed up, with online-post-to-waitlist conversion at roughly 8%–9%.
There is a competitive window, but Bill does not believe speed alone will secure the market. He expects at least 10 horizontal Agent companies to emerge over the next year, with perhaps 3–4 or 4–5 ultimately remaining, differentiated by vertical focus, capability emphasis, integrations, and user lock-in. North America’s open APIs, SDKs, and third-party integrations make it the first major battlefield; China’s relatively closed commercial ecosystem may require Poke AI to first partner with large incumbents around platform capabilities.
The founder-market fit comes from nearly a decade of deploying RL, while whether a travel demo could generalize to the real world was once a “big question mark.” At Meta, Bill led Applied RL and estimated that the algorithms contributed nearly $500M in annual revenue, while he completed a Stanford PhD in parallel; his method for evaluating technical judgment is to first find a minimal viable example with very low compute requirements and broad applicability, test whether the technology has a first-principles advantage, and only then attempt to scale it into a large system.
Deep dive
1. Bill put research, production deployment, and management on the same RL track
朱哲清 Bill moved to the US in 2014, completed a computer science degree at Duke, and joined Meta. For his first 3-plus years, he built B2B recommendation systems from scratch, covering business growth and ad growth, just as deep-learning recommendation systems were beginning to take off.
For the next 3-plus years, he led the Applied Reinforcement Learning team, bringing RL into advertising, recommendations, Reels short video, and data infrastructure. The team also open-sourced a reinforcement-learning framework called Pro; Bill roughly estimates that RL algorithms contributed close to $500M in annual revenue to Meta, with the biggest impact on the advertising side.
Running in parallel with the full-time job was a Stanford PhD in reinforcement learning. The arrangement worked only because his manager at the company, his doctoral advisor, and his research direction were closely aligned; the cost was more than 6 years of roughly 110 hours a week, with nearly all waking time spent working and doing research.
2. Extreme time pressure taught him to divide tasks into 0, 80, and 100 points
The first lesson Bill drew from running on two tracks was priority: some tasks should receive 20% effort and get to 80 points, critical tasks deserve 100% effort and 100 points, and others should receive 0% and be abandoned outright. “If I wanted to do everything well, I probably wouldn’t have been able to get through this PhD.”
That also explains why he refused to keep changing direction. Meta’s RL team once shrank from more than 20 people to just 3, and a director was preparing to shut it down; after Bill took it over, it recovered to more than 10 people, while the path centered on deploying RL continued for nearly a decade.
Early language models, chatbots, 3D vision models, and safety models all tempted him to switch tracks, but he only tried them briefly before stopping. When the host asked whether he had missed the language-model opportunity, Bill said he could not reason backward and assume he would have built InstructGPT at the time; RL was the field he understood best, so he should first deploy it in the real world and build a reputation there.
3. RL’s long period in the cold reinforced his belief in being “a little stubborn”
Bill took from Rich Sutton’s experience that today’s Turing Award winners can spend nearly 4 years being ignored early in their careers, with no university willing to hire them; Hinton’s early push for deep neural networks was also dismissed by many as having no future. Whether a direction is right cannot be determined solely by the prevailing consensus at the time.
He is not arguing that people should stop evaluating their choices, but that they should not keep second-guessing themselves with excessive noise. If the underlying framework still holds, the direction should be pursued until it is proven that someone else has already done it and there is no way to take over. “A lot of the time, you probably still have to be a little stubborn.”
4. Sample efficiency is the condition for RL to move from “community self-congratulation” to production systems
Early on, Bill studied planning and Monte Carlo Tree Search with Ron Parr; Deep RL began to gain momentum in 2016 with results such as AlphaGo, AlphaGo Zero, and AlphaZero. During his PhD at Stanford with Ben Van Roy, he shifted his focus to sample efficiency, or reducing RL’s dependence on interaction data.
The difficulty comes from multi-step planning: data requirements scale with the number of actions, states, and planning steps. Bill’s example is that if 10,000 tools each require 100 data points, a naive approach needs 1 million data points.
The first layer of compression comes from generalization—for example, reducing 10,000 tools that must be learned individually to 5,000 or even 1,000 representative tools. The second is to move from linear scaling toward square-root scaling, so understanding 1,000 or 10,000 actions no longer scales linearly with tool count.
The key move is intelligent exploration: known-bad tools should no longer be called, and branches with similar properties whose existing paths have already been shown infeasible can also be skipped. By exploring only what is genuinely unknown, the data required for the same result could fall to one-tenth, or even one-hundredth, of the original amount.
5. Incremental gains inside Meta revealed an RL opportunity larger than recommendation systems
RL has already generated measurable business impact in Meta’s advertising, recommendations, and short video, but Bill believes its potential should not stop inside one company. If RL can provide reasoning, planning, and long-term decision-making, it could become the engine behind Agents and even superintelligence.
That led to his startup question: could an Agent be built around RL, with strong planning, reasoning, and tool-calling capabilities, without relying on a language model for decisions? In September or October 2024, when he discussed the idea with advisors, friends, and investors, neither Agents nor RL was a hot phrase, and most VCs could not understand the connection between the two.
Academia and industry broadly viewed the idea as promising, and some people asked to join before the financing had even been disclosed. Bill therefore stresses that the company was not chasing RL after DeepSeek became popular; it had already made the bet before external consensus emerged.
6. DeepSeek made RL hot while o1 still lacked a public explanation
The host asked why Bill said RL was popularized by DeepSeek rather than o1. His distinction was that although o1 was described as RL-driven, the underlying training logic was not disclosed; the market could only speculate that it might use inference-time reinforcement learning, along with Chain of Thought and Monte Carlo Tree Search-style methods.
Bill explicitly limited this view to speculation: “There is no conclusion at this point,” and OpenAI has not confirmed it. He was inferring from phenomena such as reasoning speed that o1 may not have done much optimization on the training side, instead putting more effort into inference.
DeepSeek was closer to a point somewhere between AlphaGo and AlphaZero: rather than manually labeling every point in a sequential action-taking process, it could use a reward close to a rule or ground truth to judge the overall result, verify the outcome immediately after each round of actions, and continue training, while reducing the large volume of human reward-model labeling required by RLHF.
DeepSeek-R1-Zero also exposed the limits: a rule-based reward can keep scores rising while outputs turn into gibberish that humans cannot understand; human heuristics and RLHF were still needed later. Bill also stressed that the AlphaZero analogy is not 100% accurate.
7. Poke AI pulls the decision process out of natural language
Bill’s definition of an RL Agent is not a language model producing a paragraph with a function call embedded inside. Planning, reasoning, and tool calls should become an abstract sequential decision process, planned as “one concept followed by another,” where one step may be a tool call and another may be information retrieval.
He repeatedly clarifies that RL and language models are not in conflict: RL can train a language model, or it can train a non-language model. Robot actions or a planning policy in a given domain can be decided by RL; Poke AI has chosen the latter implementation.
The overall layering is: “The frontend is still done by the LM, but the backend uses no LM at all.” The language model handles natural-language understanding and user interaction, while the RL model handles abstract tool planning and invocation.
The grocery-shopping example makes the long-term vision clearest: the user’s intent is understood through the language layer and passed to RL, which sends the request to the service provider’s B-end Agent; the provider retrieves data from its database, dispatches workers offline, and eventually returns the completion status to the language layer. The information flow in between does not need to remain text throughout.
8. The real dividing line is not whether an Agent can search, but whether it can write
From the user’s perspective, Deep Research and Manus still mainly “surf the internet”: scraping, summarizing, analyzing, and generating content. Bill wants to add the ability to post to, update, or trigger actions on accounts and platforms such as Facebook, LinkedIn, personal websites, and Shopify.
His product comparison is that Operator may have stronger execution than Manus, but its success rate remains very low; Manus is stronger at Deep Research and information synthesis, while its write capabilities are weaker. Bill considers Manus “a very good engineering product,” but says it has not solved the long-term backend problem Poke AI is pursuing.
The end-state assumption is that the importance of the internet frontend will decline: UI and the large amount of JavaScript around it exist to help humans understand information flows, while an Agent only needs to understand raw text and images. “If the future world is all Agent-to-Agent,” polished webpages will look more like an intermediate state.
9. Tool counts and long tasks push language-model context toward a breaking point
Bill’s example is that if 50 tools each require 1,000 tokens of description, tool documentation alone consumes 50,000 tokens; once the user prompt, Agent memory, and historical context are added, the model carries a high information cost before execution even begins.
Multi-step tasks keep stuffing tool results back into the prompt: if a single scrape produces an article with 10,000 tokens, a task can exceed 1 million tokens after a dozen-plus steps. Bill’s view is that all LLMs may hallucinate under these conditions.
Asked whether this limit will eventually be solved, Bill acknowledged that unlimited compute and continually larger models could theoretically improve attention and context. In reality, tools may multiply explosively while context length can expand only relatively linearly; the world’s entire universe of vertical tools cannot be placed into one prompt forever.
10. “Let the model build its own tools” still runs into documentation and access rights
The host proposed another route: the model would not need to know 100 tools, only 10 “tool-building tools,” such as being highly proficient in Python. Bill’s counterexample was scheduling Zoom or Tencent Meetings: if the model has never seen the relevant documentation, it cannot write the integration code correctly using common sense.
Switching to a browser returns to the problems of excessive tokens, complex workflows, and slower-than-human speed; retraining a language model for every new batch of interfaces is equally uneconomical. Poke AI’s goal is to keep the language model as the interaction foundation while allowing thousands of newly added tools to be called directly.
11. Write operations force approval to become a first-class product feature
Poke AI’s interface still looks like chat on the left and results on the right, but before execution it displays the overall plan and asks whether the user is satisfied. The reason is that tasks genuinely change the user’s social-media, work, and personal accounts.
In step-by-step mode, many non-search steps display their inputs, allowing the user to approve or manually edit them. Bill does not want users to press a button, have “nothing to do,” and then discover half an hour later that the Agent got stuck or failed.
Feedback is sharply divided: business users cannot accept a system writing to Facebook, Instagram, or LinkedIn without consent, while developers, researchers, and AI enthusiasts find approval buttons too slow and want the Agent to run autonomously from start to finish. The product therefore keeps both controlled and automatic modes.
Bill remembers that Manus showed users execution highlights written by an LLM, but did not let them pause, edit the entire plan, and then continue; Operator and Anthropic’s products often do not provide a plan at all. Poke AI does not show a virtual machine because most tasks run through platform APIs, and continuously reading and displaying internal state could raise platform-privacy issues; future Deep Research tasks will tell users what each step is doing.
12. Official interfaces, code, and protocols make up the execution stack, with the browser last
Bill says the primary point of differentiation is connecting to dozens of internet platforms and to the roughly 1,000 APIs that may sit behind them. The challenge is not just volume: many interfaces look almost identical, so the Agent must reliably distinguish among similar inputs, outputs, and endpoints.
Tool selection follows a clear hierarchy: use an official API instead of a webpage, write code when code can solve the problem, and add MCP and Agent-to-Agent support later. The browser is reserved for situations where search or web access is unavoidable, because “in our eyes, the browser is not the future.”
The current system uses no MCP at all, but plans to be compatible with the existing MCP ecosystem; it will also provide a lighter integration path in which developers only need to provide a JSON file specifying the input, output, and endpoint, without hosting anything themselves.
The first release is expected to include roughly 1,000 sub-tools across dozens of platforms. The company will later build a developer community where SaaS vendors can submit tools; Bill does not view this as an MCP competitor, because the same tool can be integrated through MCP or through its simpler protocol.
13. Prosumers are the first to feel that generation is easy but execution is hard
Poke AI’s current core users are developers and professionals: individual developers inside companies, social-media marketers, ad buyers, and legal or finance professionals. Bill describes it as “a product whose scenarios are To B, but whose users may be individuals on the C side.”
In early interviews, advertising and social-media professionals said they did not lack content generation; after content was produced, they still had to spend 3–4 hours distributing, promoting, and operating across platforms. Once a post generated dozens of replies, reading and responding to them one by one took even longer; Poke AI can find the post, generate a personalized response to each comment, and write the responses back.
Moving into enterprises will require solving issues such as private cloud, model size, and scalability, because sensitive data is exactly why many companies do not want to connect directly to OpenAI or Anthropic. For now, the company is serving individual developers and professionals before gradually moving into internal enterprise infrastructure.
14. The enterprise opportunity is deleting steps, not copying workflows
Asked why the company does not continue expanding toward ordinary consumers, Bill said overly general products make users overestimate their capability boundaries. He and friends used Operator to schedule meetings and send emails, and Manus to post to Facebook; success rates on those execution tasks were close to 0.
The densest workflows are still inside enterprises, often running dozens or even hundreds of steps. Bill wants an Agent to link all the steps automatically and delegate to humans when needed, rather than having people work for a long time and then find an LLM to generate a small piece of content at every step.
His criticism of Salesforce-style drag-and-drop workflows is rigidity: a slight change in the process can make the workflow fail. He sees Agents, tools, drag-and-drop workflow products, and LLM tools as lower-level components; Poke AI is trying to become a higher-level planner that decides when and in what order to call them.
The host pointed out that large-company workflows may be more fixed and less variable. Bill’s answer is to go bottom-up: let individuals and small and midsize companies first prove that a process originally requiring 20–30 steps can be reduced to 7–8, then influence large-company adoption. He also acknowledges that the sales cycle is difficult and depends on trust.
15. Failure tolerance is closer to real workflows than making every step succeed
If a subtask fails, the RL Agent uses the preceding context to determine whether subsequent steps depend on it; if not, it skips the failed branch and completes the rest. “Complete as much as you can,” then let the user start another interaction around the unfinished steps instead of freezing the entire task.
Automatic debugging is not yet available because retrying may change the user’s input. Bill uses LinkedIn as an example: publishing 2 identical posts in succession will be blocked, and a “smart enough, sneaky enough Agent” might slightly change the wording and post again; that may bypass the technical restriction without matching the user’s intent.
The team therefore needs to distinguish errors that can be bypassed automatically from those that must fail immediately and be reported to the user. The hard question is not whether the system can retry, but what permissions the Agent has to modify things while resuming execution.
16. Speed, human intervention, reading and writing, and cost are the four requirements for a general-purpose Agent
At the start of the show, Bill summarized 4 traits of a strong general-purpose Agent: completing tasks faster than humans, requiring no human intervention, being able to read and write information, and costing little. Poke AI still controls write operations through approval and similar mechanisms; that does not mean every task is already fully autonomous.
Bill’s demos for social-media operations, meeting scheduling, and analysis were not accelerated. Including approval and back-and-forth clicks, they could take around 60 seconds; if approval were removed and the Agent executed autonomously, completion could take 10–20 seconds.
Bill says Poke AI’s current per-task cost is roughly one-fortieth that of comparable products on the market. He does not present low price as a permanent advantage, instead expecting other companies to keep driving down computational cost.
17. The challenge in scalable training is exploration and negative signals
Bill says some approaches related to contrastive learning performed poorly when the team tested them. These methods may involve strong supervision, but when only one of thousands of tools is correct, random calls almost always produce a negative signal.
A tool may be sampled and manually labeled many times without finding the right answer, making training extremely difficult. A smarter training and exploration mechanism is therefore required, which Bill calls Poke AI’s secret sauce.
The first Poke AI release is expected to include roughly 1,000 callable sub-tools across dozens of platforms; Bill also emphasizes that the low price comes from advantages in the current architecture, not from a moat that will remain unchanged over the long term.
18. A four-person team covers research, production, and product end to end
The only principle that was clear at the company’s start was that the Agent had to complete complex tasks across platforms and tools, because that is how human work is structured. Low cost was also emphasized as an architectural advantage during fundraising, but Bill does not want to sell price as a permanent marketing message.
The full-time team has 4 people: Bill handles overall direction; a research scientist who previously worked on his Meta team leads reinforcement-learning research; a former subordinate from the B2B recommendation-systems team handles ML production; and a longtime friend and former Meta colleague leads product engineering.
The setup covers fundamental research, production engineering, and experience engineering. The team builds its own AI tools for much of the scaling and internal work, while contractors in Asia handle miscellaneous tasks; a large share of the work is AI-assisted.
19. Travel and Shopify were stress tests for the general architecture, not the final direction
The original travel-planning product was not a pivot, but a demo built in 1–2 weeks: a model with only a few million parameters could call Google Maps across cities and connect to tools such as Google Docs and Calendar, showing investors how planning and tool calls could scale quickly.
When the market still doubted that general-purpose Agents could make money, the team chose Shopify as a vertical landing point, integrating the command line, GraphQL, APIs, and SDKs into a full-function Agent for merchants and customers. Bill says the traditional approach might take 1–2 years, while they completed the underlying logic in 2 months.
The project validated development speed and robustness and was briefly considered for continued commercial sale. After DeepSeek took off and the market reached broader consensus around the direction, the team returned to its original horizontal platform vision starting in December, leaving the vertical product in place to sell if appropriate.
20. From “six months early” to 8%–9% conversion, the market was educated within months
Fundraising in fall 2024 was the low point: investors later said Bill was “six months ahead of the curve,” because nobody understood RL plus Agents at the time. He does not consider the decision to narrow the product to Shopify a mistake; if the market believed only in generation and not in planning and tool calls, the grand vision would have faced sustained resistance.
In December, January, and February, attention from investors, customers, and developers was said to grow geometrically: more than 100 investors and dozens of large customers reached out proactively, while the number of smaller developers was “uncountable.” The team did not bear the full cost of market education, which was an important condition for returning to the platform route.
Poke AI released its demo on March 3, and Manus launched 2 days later. The team initially worried about colliding with a direct competitor, but after trying it concluded that Manus still centered on browser-based and generative tasks; its marketing, product design, and multi-tool orchestration were worth learning from, while the execution layer could be complementary.
The demo generated more than 800 waitlist sign-ups in its first week; after the March launch, another 800-plus people signed up. Conversion from online posts to the waitlist was close to 8%–9%, with roughly 10,000 views. The beta would open directly rather than use invite-only scarcity, but the site and tools would have rate limits to prevent concentrated scraping from being treated as a DDoS attack.
21. The moat has to grow out of workflows and market migration
Bill’s concern about pure-generation products is that quality depends entirely on backend technology, without integration, user stickiness, or workflow attachment. Poke AI wants to accumulate users’ historical files, videos, images, and account relationships, so a single instruction can retrieve old assets and publish them to Instagram or TikTok, creating deeper lock-in.
The host asked whether this turns the market into a speed game in which whoever binds users fastest wins. Bill countered with MySpace and Facebook: Facebook was not the first entrant, but survived market changes; what mattered was the ability to adjust strategy, capture user groups, and deepen stickiness through services built in-house or through partnerships.
He acknowledges that technology will not remain exclusive forever: “Eventually, someone will definitely be able to build something similar.” The job of a first-mover advantage is to complete the first wave of integrations, scale, and user attachment before other teams catch up, not to assume the algorithm will lead forever.
Geography directly affects integration speed. Bill sees North America as having the most open commercial ecosystem, Europe in the middle, and China as open on the model side but relatively closed commercially. Facebook offers APIs, SDKs, and third-party tools for posting, while WeChat Moments, WeCom, and Channels lack equivalent user-posting interfaces, making a unified Agent spanning Tencent, Alibaba, and Baidu ecosystems more difficult.
22. Horizontal Agents will not eliminate vertical companies; they may become their infrastructure
Bill expects at least 10 general-purpose Agent companies to emerge over the next year, with perhaps 4–5, or even 3–4, ultimately remaining. Like Claude and ChatGPT, they will start from similar positions and gradually tilt toward coding, To C, or specific verticals to maintain share through differentiation.
The host noted that “general-purpose” and “differentiated” appear to conflict. Bill answered with operating systems: Android, iOS, Windows, macOS, and Linux are all general-purpose enough, yet have coexisted for years because of differences in architecture, ecosystems, and open-source strategy; the complexity of Agents is closer to an OS than to a single language model.
Poke AI’s long-term role is to power vertical AI: social-media scraping, file writing, slide creation, and webpage building can be shared, while vertical companies only need to add their most important proprietary tools and can then call 1,000-plus tools to complete workflows.
Regarding open-source projects such as OWL and Open Malice, Bill believes Deep Research and generative Agents are relatively easy to open-source, while versions capable of writing across platforms have the hardest integration burden and the longest lifecycle. Open-source versions of the latter will eventually appear in some form, but the specific shape remains uncertain.
23. The minimal viable example is Bill’s core method for judging technical potential
The approach he learned from Rich Sutton, his manager, and his doctoral advisor is to first build a toy example with minimal compute requirements that is sufficiently intuitive and general: other approaches should be unable to solve it in principle, while his technology should solve it systematically. Hacking a corner case would not prove real-world value.
Once the first version works, he scales up the problem and checks deployment, system integration, and bottlenecks. Bill believes this can provide roughly 80% confidence in technical feasibility, but “it is not 100% guaranteed to succeed”; running a large experiment directly introduces too many variables, often making it impossible to know what actually worked.
By the GPT-2 era, he already believed language models would inevitably become important, but felt that “the boat was already full”: he should learn the technology, but becoming one of the earliest deployers was no longer realistic. Rather than jumping ship to chase the fashionable field, he chose to keep going deeper in RL, where he had accumulated expertise.
His reasoning at the time was that if RL could handle complex, long-horizon planning at the token level, it should also work for RL planning at a more abstract level. Recommendation systems offered a precedent by treating articles as actions and recommendation sequences as plans.
Bill began thinking about these questions around the time he graduated, and it took roughly 6 months to arrive at the current plan. Although the travel toy example worked, whether it could scale across countless scenarios and genuinely generalize remained a “big question mark”; he ultimately chose to leave Meta because golden handcuffs do not disappear on their own, and starting a company requires a clear conviction signal.