How I Use RL + LLMs to Build Agents | A Conversation with Pokee AI Founder 朱哲清 Bill
Summary
Bill’s core view is that RL is not an alternative to LLMs, but a learning and planning method layered on top of Transformers. LLMs provide language, embeddings, and a basic understanding of the world; RL searches unknown paths and makes decisions based on long-term reward. The two are not parallel competitors, but “may be vertical” layers. That is the premise behind his view that RL can be “reborn” in the era of large models.
The bottleneck for complex Agents is not understanding instructions, but finding viable solutions across an exponentially expanding sequence of steps. LLMs tend to loop along the highest-probability path, while RL can prioritize the most uncertain direction and backtrack or switch routes based on expected future value. Bill summarizes it this way: “You don’t need brute-force enumeration anymore; you need search.” Travel planning is the classic example: the system only discovers on step 10 that cities or dates conflict, then has to know where to backtrack.
Pokee AI is betting on an architecture where “RL makes decisions and LLMs handle I/O,” rather than putting a larger LLM in charge of everything. User requests are abstracted into goals, states, and actions; RL determines API call order and parameters, and the final result is converted back into human-readable form by an LLM. Its public Shopify system reportedly uses a 15M-parameter RL Agent and a language model under 10B parameters to master dozens of APIs in less than a month, with no special personalization training required when moving across stores.
The commercial leverage of this route comes from reliability and cost, not model-scale narratives. Bill says the team spends less than $10,000 a month on training, and the RL model can run on CPUs. If GPT-4o were used for comparable capabilities, the cost could be “1,000x” higher. His response is: “If it solves the problem, that’s enough.” The company does not position itself as a large-model company, but as a “tool-using Agent with strong reasoning and planning.”
E-commerce is merely the first closed-loop setting for a general-purpose API Agent; the value proposition is replacing fragmented plugins and human coordination. A Christmas promotion that once required sales, engineers, and multiple vendors to separately modify discounts, shipping, the homepage, search, and recommendations could eventually be executed by an Agent through a single text instruction. Bill says some merchants spend more than $1M per quarter on this kind of work. If the product delivers, he estimates that “45% to 60%” of this purely engineering-oriented software could lose its value.
Pokee and coding Agents target different markets: the former combines existing business capabilities, while the latter creates capabilities from scratch. Bill believes most code in real-world work is simply composing APIs. Treating one API call as an action is far cheaper than generating, compiling, and validating code token by token. The roadmap is therefore to expand to thousands and eventually tens of thousands of APIs, then move into advertising, travel, education, fitness, legal, and finance—not to chase general-purpose coding first.
For investors, the key questions are accuracy, reliability, and unit economics in the public beta—not the slogan “the first year of Agents.” Bill sees no shortage of US capital and continued interest in vertical Agents, but also a crowded field and a confusing target set. Less crowded areas such as science and supply chain may attract more attention. He rejects the first-year narrative, saying only that the next year may become “the year when Agents truly help you take action and produce results.”
Deep dive
1. RL’s Revival Comes from Large Models Filling in Real-World Representation
朱哲清 Bill spent 7.5 years at Meta. During his first 3 years, he focused mainly on applying ML, leading the ML recommendation system for Meta Ads’ enterprise business and the company’s first reinforcement-learning models, including an RL application for ad discounts. Later, he led the Applied Reinforcement Learning team, bringing RL into ad recommendations and infrastructure. He also spent more than 6 years pursuing a PhD in reinforcement learning at Stanford. Pokee AI extends this same technical line: combining RL and LLMs to build Agents with stronger reasoning, planning, and personalization.
Bill recalls that after AlphaGo and breakthroughs in games in 2016, RL remained constrained by its environments. Games allow unlimited interaction and immediate feedback; the real world does not permit repeated trial and error. Around 2020, as GPT, NLP, and Decision Transformer gained momentum, the industry briefly believed that multi-step decision-making could be handed entirely to Transformers and imitation learning.
The turning point was the realization that more complex reasoning and planning could not be achieved through existing data and imitation alone. Systems had to search paths beyond the training set. Bill cited a researcher who recently worked on OpenAI o3 and o1 saying that RL was so powerful that “the way we think about problems recently has all been RL-ified.”
2. The LLM Is the Interface to the World; RL Is the Learning and Decision-Making Method
Bill rejects treating the two as mutually exclusive architectures: “RL and this whole series of large-model approaches aren’t parallel operating modes. RL may be a vertical layer.” Large models can be compared with traditional neural networks, while RL is a training and decision-making method that can fully leverage Transformers to enable stronger planning.
In the past, open-world RL required teams to explicitly design the environment, actions, and representations. With LLMs, much of this can now be reconstructed through language and embeddings. The LLM provides a basic understanding of the world; RL then learns how to interact with it, eliminating the need for many bespoke architectures.
This is also why Bill chose to start a company now. At Meta, many people handled the conversion from concrete business requirements into abstract actions for the algorithm. Outside a closed system, that step once prevented RL from being deployed. LLMs can now generate offline data and evaluate Agent outcomes, finally completing the training loop he had envisioned for years.
3. RL’s Advantage Is Knowing Where to Explore, Not Simply Trying More Times
Pokee’s intended workflow is to first decompose a request into a goal, the current state, and available actions, then have RL search for the order and combination of those actions. Once the path is executed, a language model rewrites the result into something a person can read. When the decision space is defined, the system calls APIs directly; when it is not, the LLM helps generate candidate actions.
Mathematical reasoning falls into the latter category: the model generates one step, evaluates it, and backtracks when it gets stuck. 曲凯 asked why an LLM could not simply “generate, verify, and switch paths” on its own. Bill’s answer was that language models are bounded by their datasets and often have “only one track,” while random exploration is also highly inefficient.
Bill uses 3 restaurants to explain uncertainty-driven exploration: one where you eat 5 times a week, one you tried once and found terrible, and one you have never visited. If the goal is to find your favorite restaurant, you would test the most uncertain third option first rather than sample randomly among the 3. That ordering of exploration is precisely what RL must learn.
In a maze, an LLM may repeatedly return to the highest-probability path without knowing how much uncertainty lies beyond it. Bill calls the deeper capability counterfactual reasoning, or “the parallel-universe problem.” Under a “big world” hypothesis, the system does not need to learn every possible variation of the world. It only needs to estimate the conditional probability, probability of success, and final reward associated with different decisions based on the paths already taken and the current state. It is not actually testing every path in the background; it is calculating value under conditional probabilities.
4. Fast and Slow Thinking Can Be Split, but the Industry Has Yet to Back One Route
The alternative theoretical route represented by Rich Sutton uses RL and continual learning to challenge Transformers: data is no longer a closed dataset, but an infinite stream arriving continuously. Bill uses a spring to explain plasticity: once a model is stretched to its limit and approaches saturation, adding more information may cause it to “break.” He also acknowledges that this route is further from deployment.
The fast- and slow-thinking systems discussed by Google, Meta, and others are closer to a composite architecture: fast thinking answers Q&A directly, while slow thinking executes chain of thought or other planning. Pokee takes the division further: “All decisions are made by RL; all external I/O interfaces are handled by the LM.” It is like using dopamine rewards to train the decision-making region while leaving the knowledge region to handle expression.
This is not a consensus in the US technology industry. Bill noted that the latest AI Agent paper from 李飞飞’s team at the time was centered more on imitation learning. He expects the industry to converge on a particular route only after the first Agent reliably completes complex multi-step decisions appears.
o1 and o3 also show why architecture and algorithms should be considered separately. Bill believes they did not change the underlying LLM architecture, but changed the final training algorithm. He also said they performed optimizations resembling RL mainly at the inference layer. Outside observers speculate that o1 uses MCTS to repeatedly traverse paths, backtrack, and switch routes, but the cost is high inference time and pricing. World models instead try to predict the next state after each action in advance. Bill said he had spoken twice with Yann LeCun, who also wants to build systems capable of long thinking and planning, but approaches the problem more from the world-model angle.
5. RL Is Best Suited to Global Optimization, Unknown Users, and Interdependent Actions
In Meta’s ad auctions, RL optimizes expected company-wide revenue rather than ensuring that every individual bid improves. A local action may perform worse, but if several users or auctions deliver a 20%—30% improvement, overall ad efficiency can still rise significantly. That is the value of an expected-value objective over point prediction.
Facebook Reels’ bandit project illustrates another suitable setting: for new users whose interests are unknown and who are easy to lose, exploratory recommendations reportedly generated an engagement lift of dozens of percentage points, according to Bill. Standard recommendation systems keep betting on known high-probability content but cannot quickly identify what a user actually prefers.
Multi-step dependencies are equally important. Spending $10 at the first point may mean having less to spend at the second; spending $5 first leaves a larger future budget. When the current action changes the future state, RL has a clear advantage. If there is only one action and the outcome is known, the problem is essentially prediction, and “RL won’t do much there.”
6. Real Agents Reveal Their Difficulty Through Complex Travel Planning
Bill says bluntly that “the Agent concept has been overplayed.” Buying a plane ticket is a one-step task that can be handled through conditional logic and traditional engineering, without an LLM. The real difficulty is planning a trip involving multiple locations in Shanghai, a stop in Hangzhou, and a return to Beijing within 2 days, all under multiple constraints.
Pokee’s first demo for investors was exactly this kind of multi-day plan across multiple hotels and locations within a single city. 曲凯 noted that travel Agents have long been hackathon staples. Bill’s explanation is that an LLM can stitch together an answer for a short trip, but the longer the path, the more likely late-stage conflicts are to force the system to backtrack.
“Every additional step increases the complexity exponentially.” If the system discovers at step 11 that it cannot continue, the challenge is not generating another plan. It is deciding which layer to return to and which earlier change will create the most value for the future.
RL estimates how the current action affects future expected outcomes. If one of 3 candidate actions reduces the expected probability of downstream success to zero, it should be discarded immediately rather than enumerated further. Bill’s conclusion is direct: “You don’t need brute-force enumeration anymore; you need search”（你需要的不是一个穷举的能力了，而是一个搜索的能力）.
7. Pokee Starts by Teaching a Small Model Shopify’s Underlying APIs
Pokee’s long-term goal is to autonomously call thousands or even tens of thousands of APIs to complete complex planning and reasoning. Its first beta product is built on Shopify. According to Bill, a 15M-parameter RL Agent paired with a language model under 10B parameters mastered dozens of APIs, along with basic search, recommendation, and customer-service capabilities, in less than a month.
This Agent does not learn the higher-level plugins installed by stores; it learns Shopify’s underlying APIs. Once APIs are converted from concrete operations into abstract actions, the Agent uses self-play to learn what result each action produces and whether the user is satisfied, including when to call an API and which parameters to pass. Bill did not disclose the specific training details.
Bill says this training has “not much to do” with training the large model itself. The direct connection is mainly the embedding layer, which converts concrete text into abstract embeddings. The LLM still participates in the training loop by helping assess execution quality, generate requirements, and provide high-quality embeddings. RL is what determines API order, parameters, and multi-step combinations.
For merchants, the product is like “hiring a website lead.” Search, recommendations, and customer service share the same data and decision system, while multiple UIs other than product-display pages can be generated by the Agent from templates. Merchants no longer need to install dozens of disconnected plugins or hire engineers to maintain them one by one.
8. The E-Commerce Use Case Compresses a Promotion into a Single Prompt
During the early public beta, Pokee invited 50 e-commerce merchants to install the product and extended the free trial from 14 days to 30 days in exchange for feedback. Planned capabilities also include merchant-consumer negotiation through the Agent, automatic discount creation, and using a prompt to change site-wide recommendation and search strategies simultaneously.
Bill’s mature use case is a Christmas promotion: discount relevant products, modify shipping fees and delivery times, then switch the homepage, search, and recommendations to a unified Christmas theme. Today, these actions are split among sales teams, engineers, recommendation vendors, and database operations. In the future, users would specify the products, promotion code, and ranking preferences in plain text.
曲凯 added that marketing emails to existing customers could also become an action in the same workflow. Bill says some e-commerce companies he has spoken with spend more than $1M per quarter on similar operational and engineering work. If the Agent matures, a process spanning teams and vendors could be completed “within 5 minutes.”
9. API Agents and Coding Agents Belong to Business Composition and Creation from Scratch
Bill draws a clear boundary between Pokee and coding Agents such as Devin. Pokee arranges existing business capabilities to solve specific customer problems; coding Agents face situations where “the business does not exist at all,” attempting to replace 20 employees with 20 developers and build a complete system from scratch.
He observes that most of the 100 lines of code written by engineers at large companies are also combinations of existing APIs. If an Agent knows how to make multi-step decisions across APIs, it covers a large share of real-world tasks. Only when the required capability does not exist anywhere does the system need to write code token by token, compile it, validate it, and establish a new codebase.
The decision cost of calling an API directly is also lower: one action produces one result. Generating code, by contrast, requires “every token to be guaranteed precise.” Pokee therefore focuses on treating API calls as actions and composing existing business capabilities across multiple steps, rather than generating complete code from scratch.
The long-term roadmap includes opening product, discount, and other APIs to merchants beyond Shopify, then entering advertising, travel, education, and fitness, followed by legal and finance. Agents could also hand off inventory and orders directly between companies. If the other party only has a phone or email, the Agent can call the relevant service, but a reliable closed loop remains difficult without standard interfaces.
10. Small-Model Economics Are the Moat Hypothesis; Deployment Reliability Is the Test
Bill says the team’s training costs have remained below $10,000 per month, and the RL model can run on CPUs. He rejects the criticism that Pokee is a “fake large model”: “You don’t necessarily need to build a hammer so large that it eventually becomes a nuclear weapon.” Pokee has never called itself a large-model company; it is an Agent company.
An early planning demo used a 10M-parameter RL model plus Llama 8B. In Bill’s tests, it outperformed GPT-4 on planning within a single city at the time. As for vertical competitors, he believes that even if they compress chain of thought and model size, they still bear the cost of the underlying LLM. If they rely directly on GPT-4o, comparable capabilities could cost “1,000x” more than Pokee’s.
If general-purpose Agents mature, Bill estimates that “45% to 60%” of the thousands of Shopify plugins could be replaced, mainly products that add engineering wrappers on top of underlying APIs. Non-engineering services would remain. The next goal is for the system to understand requirements and accurately call thousands or tens of thousands of APIs without special prompting.
His market view is that “money is not scarce.” US investors remain interested in vertical Agents, and AI investment rounds are rapidly emerging at tens of millions and hundreds of millions of dollars. Accuracy, reliability, and business models remain common challenges. Rather than declare another “first year of Agents,” Bill would define the next year as the year when many companies begin truly enabling Agents to “take action and produce results.”