Tsinghua's 徐梦迪 on Scaling Law Signals, Embodied AI & World Models

Summary

• 徐梦迪 says he is seeing early signals of the Scaling Law in embodied intelligence, but stresses that loss does not equal success rate. He cites results from other companies showing that scaling pretraining data from 100K hours to 1M hours produced strong performance on unseen test sets. But he uses the example of a 20-step cup-grasping task—where the robot may not even touch the cup in the first 17 steps—to show how temporal imbalance can decouple loss from success rate. What he really wants to see is “data gradually increasing, models gradually getting larger, and success rates on unseen tasks gradually rising.”
• He sees embodied intelligence as being at the GPT-1 stage today—pretraining plus fine-tuning for a target domain—with the key inflection point being a GPT-3-style in-context learning moment. The end state is for end users to define robot behavior with prompts and demos: clothes could be folded into squares, rolled up, or simply hung rather than folded. He thinks scaling could produce a clear paradigm shift or inflection point over the next 3-5 years, but the marathon has only “run 10 kilometers” so far.
• In the route debate, he clearly favors world models because the number of tasks makes brute-force enumeration impossible to scale. “There are simply too many tasks; it’s hard to solve them through enumeration. But if a world model can learn the rules governing how the world changes, that is a more fundamental capability.” His watchlist includes Physical Intelligence, Google, Nvidia, Generalist, and Roda (phonetic) in the US. In China, he highlights 林波’s newly released LingBot VLA 2, calling its ICL approach “highly meaningful.”
• His overall view of China’s embodied-intelligence startups: “There has been a great deal of real progress, but there will also be some bubbles; the two are not mutually exclusive.” Chinese companies are working across the full stack—materials, sensors, robot bodies, data, and models—with resources continuing to flow into the sector, making him “very optimistic” over the long term. But the industry’s focus has swung from data to world models and back to data, like an underdamped control system oscillating through overshoot; the field is far from convergence.
• The riskiest part of commercializing embodied intelligence is the chicken-and-egg problem in the data flywheel. The base model has to be good enough for an end user to see “obvious progress” after 30 minutes of teaching before the loop can start turning, yet the model needs to be used in advanced real-world scenarios to improve. His team is experimenting with a real dual-arm robot equipped with dexterous hands on tabletop tasks, collecting human correction instructions to change the robot’s behavior, with the long-term goal of full-body dexterous manipulation.
• On data, he calls himself a “fusionist” and sees very high value in human data. Teleoperation data is closest to the robot’s embodiment and suited to post-training; simulation data is clean and controllable and suited to pre- and mid-training; UMI sits in the middle; and human data is the cheapest way to cover a broad range of scenarios. “Humans are indeed a very mature embodiment,” while morphology gaps can be addressed through transfer because what the model learns is task information that is not tied to any particular shape.
• The conversation was threaded through with methodological one-liners. 李飞飞 first pressed him on which benchmark metric was “actually important”; 徐梦迪’s answer was final success rate. She also quoted Einstein: “Making things simple is not simpler.” 姚期智’s advice was that “there are actually very few truly important questions.” 扣姐 closed with: “It is easy to be correctly pessimistic; only optimism can lead to success.”

Deep dive

1. The 30.5-Year-Old Assistant Professor: From Vehicle Engineering to Embodied Intelligence

• 徐梦迪 describes himself as “30.5 years old,” an INTJ and a Capricorn, and is now an assistant professor at Tsinghua’s IIIS. His research agenda, in one line, is “to enable robots to continuously improve themselves from real-world environments.” The thread on generalization began during his time at CMU and continues today, tracing back to Charles Finn—Stanford professor and one of the founders of Physical Intelligence—and MAML: “It enables robots to learn tasks they have never seen.” He decided then that “this was truly one of the directions for the future of robotics.”
• His career switch was unconventional. He majored in vehicle engineering at Tsinghua; in his sophomore year, he worked with 严少泽 in the mechanical engineering department on bio-robotics, studying the hyperelastic recovery of bee wings after collisions. The summer after his junior year, he joined Hobby Cho Set’s (Howie’s) group, designing a snake-like robot that could climb through pipes. The work gradually pulled him from mechanical design toward algorithms.
• His breakdown of generalization ran through the entire conversation. Humans generalize well for 2 reasons: “people have a lot of prior knowledge,” and “people can learn knowledge very quickly.” True generalization, he argues, is about giving robots a similar ability to learn quickly.

2. The Hengshui Competition Class: Comrades, Not Rivals

• A childhood detail: from 衡水 in Hebei, he saved up for a four-wheel-drive car by picking up small pieces of trash on the way home from primary school and selling the steel parts. 扣姐 called it the seed of his future in vehicle engineering, and he agreed: mechanical structures have always seemed “very cool” to him—precision systems that crystallize “many generations of craftsmanship.”
• His experience of the controversial Hengshui model was in a physics competition class. Morning runs and schedule management were as strict as in the regular college-entrance-exam classes, but the atmosphere was “more like everyone banding together to solve the problem,” with little rivalry and strong camaraderie: “The friends around you all share the same vision… you don’t feel that it’s苦.” Competing felt like starting a company with no way back. Only after it became clear that a recommendation-based university placement might not be available did he transfer into the regular exam class in the second half of his senior year.

3. CMU’s Best Dissertation: Pretraining Is Not Everything

• His PhD dissertation, Building Adaptable Generalist Robots, won one of the department’s 2 best-dissertation awards that year. Its central proposition has 2 equally important sides: “Pretraining itself is certainly very useful, and I strongly believe in scaling, but pretraining is not everything.” Robots cannot be expected to achieve “100% success” through pretraining alone; after deployment, they must adapt to changes in the specific environment.
• His evidence is the “open world.” A home may contain 20 objects today and 30 different ones a week later; human preferences also keep changing. The task itself is moving, and the robot must “improve itself within this changing environment.”

4. 2 Routes to the Problem: ICL and Changing Just 0.5% of the Parameters

• The first route was Prompt Decision Transformer in 2021, directly inspired by GPT-3’s in-context learning: “You don’t need to retrain in the deployment environment; it is a more efficient and faster form of adaptation.” The model receives a demonstration and adapts to different locomotion and manipulation tasks.
• The second route, Hyper Design Transformer, addresses the double-edged sword of changing parameters at test time: change too much and the model forgets its pretrained knowledge. The approach first trains a hypernetwork, which then generates an adapter containing only around 0.5% of the parameters in the full Transformer and adapts that small component.
• His conclusion favors the first route. With small-scale pretraining, hypernetworks are more likely to work. But once data and pretraining scale up, “the real future direction is still in-context learning”—either changing no parameters at all or waiting until enough multi-task data has accumulated for a batch update.

5. Model Parameters Should Be Equivalent to an RL Algorithm

• The line of work he is most excited about is DeepMind’s algorithm distillation. Mainstream VLA models use expert demonstrations for imitation learning, effectively treating model parameters as a task-memory module. AD’s claim is that “your model parameters should be equivalent to an RL algorithm”: put suboptimal trajectories into the context window and have the model continually revise future actions based on past bad actions and history. What is distilled into the parameters is the ability to improve itself.
• He is direct about the limitation. AD has mostly been tested in simulators on maze navigation and locomotion; manipulation “is still relatively difficult.” Related work includes a paper from depark’s group called Local Former, which deploys ICL-style adaptation on quadruped robots, and Nvidia’s recently published Robot TTT, which has robots watch long-horizon human manipulation videos and use TTT to change their behavior.

6. Why Adaptation Has Only Now Become a Consensus

• 扣姐 asked why this has only recently become a topic when it sounds like something that “should naturally be important.” His answer: the field previously focused on scaling data and models to build up priors, corresponding to the first layer of human generalization. More recently, researchers have realized that robots “really cannot solve every application through pretraining alone or through expert-demonstration-based imitation learning alone.” The second layer—rapid learning—has now moved to center stage.
• The sequence is logical because adaptation places hard demands on the base model. “We don’t want robots learning in real-world environments in a terrifying way—for example, they might smash a cup.” So the field should build the prior first and then add adaptation. “I think it was reasonable for everyone to start with foundation models.”

7. The Stanford Postdoc Trilogy: BEHAVIOR, Steerability, and Creative Tool Use

• He worked on 3 projects in 吴佳俊 and 李飞飞’s group. The BEHAVIOR benchmark asks what tasks robots should actually solve, covering homes, schools, and a large number of open-source assets. He helped use a method called momentum to generate simulated-environment data from human operation. The benchmark is still actively maintained and is becoming easier to use.
• The second project made robots more steerable: following more diverse language instructions and adapting to different users’ preferences. It was directly connected to his core research agenda.
• The third had the clearest real-world flavor: a creative tool-use benchmark. Humans use tools based on function and affordance rather than names—“If I want to cut a cake but have no knife, I can use a fork or chopsticks.” If there is no broom for sweeping a table, a book or even a forearm might work. The team collected large amounts of aligned human data from first- and third-person views and evaluated 3 dimensions: a multimodal model’s ability to select tools and generate trajectories; 3D/4D reconstruction of deformable objects such as cake and water; and whether dexterous human tool use can transfer to robots.

8. 李飞飞’s Lesson: Only Final Success Rate Really Matters

• 李飞飞 is vision-driven and pushed him to step outside any single project to ask what would truly matter for the field over the next 10 years and what role he should play. The most concrete example came when his team had specified benchmark metrics in detail—progress, success rate, time taken, and so on. She asked which one really mattered. 徐梦迪’s answer was: “There is actually only one: your final success rate.” She then quoted Einstein: “Making things simple is not simpler.”
• His takeaway was that the transition from PhD to PI means moving from debugging detailed metrics to “getting everyone to focus on the one point that truly matters.” 扣姐 summarized it as moving “from problem solver to problem setter.” 吴佳俊 was more understated, discussing specific designs with him and pushing him to build his own research brand.
• His understanding of a research brand is surprisingly simple: “If you study a problem for more than 3 or 5 years, that itself has already become your research brand.” The essence is to start from genuine interest, not to force yourself to be “different from everyone else.” 扣姐 compared it with Paul Graham’s How to Do Great Work: the ball of knowledge looks smooth from afar, but up close it is full of bumps and black holes; those gaps are where the opportunities lie.

9. The Decision to Return to China: Freedom Beats Compute

• In 2023, during the fifth year of his PhD, he faced the fork between industry and academia. His Google internship was excellent and offered “the resources to work at larger scale,” but “there would also be a loss of freedom.” He wanted “to maximize the share of my time allocated to the problems I consider important,” making academia his optimal choice. He described it using an ML concept: “There is no free lunch—you have to accept the relatively weaker side of whichever decision you make.”
• 吴毅 connected him with the IIIS interview. He chose the institute for 2 reasons: its students are exceptionally strong—“Many people in robotics and AI in the US know graduates of Yao Class”—and robotics is a system. “Doing only the brain or only the hardware would both feel incomplete to me.” The institute would let him build robotics into a genuinely deployable system.

10. 姚期智’s Advice and the Interview Question He Bombed

• 姚期智’s advice continued to benefit him during his postdoc: “There are actually very few truly important questions, and a person’s time is extremely limited. If you spread your time across too many things, the result may not be very good.” Focus is essential. He admits to being highly curious and needing to remind himself repeatedly to stay focused.
• Another professor asked a question he feels he answered badly: “Why do you think you will definitely become a top researcher in your field 5 years from now?” His answer at the time was pure confidence: “I think I definitely can.” His revised answer today is: “If I find a topic I’m interested in—getting robots into millions of homes one day and enabling them to keep improving in real-world environments—and spend 5 years researching along the right path, I don’t think I’ll turn out badly.”

11. Choosing an Environment Like Clearing a Game Level: Your Ability Should Be Able to Double

• He rejects the label of having had a “smooth ride” and chooses “lucky” instead. None of his transitions were “completely smooth”; friends, teachers, and mentors supported him along the way. But luck has a methodology: “What a person becomes is partly determined by themselves and partly by the environment they choose.”
• His criterion for choosing an environment is surprisingly not to pick “the best,” but to ask whether, after joining, “your ability can double… like clearing a level in a game.” Execution requires information gathering: connect with friends first, and if that fails, send an email. “Many researchers are very warm and willing to help.”
• His cold-email advice for introverts: write the first draft yourself, then have AI polish it. He says he usually ignores “emails written entirely by AI.” The standard for sincerity is whether “you have translated your interest into actual work and whether that chain can be completed.” Interest without execution is the common trap.

12. China’s Embodied-Intelligence Market: Progress and Bubbles Are Not Mutually Exclusive

• His overall view after more than a year back in China: “There has been a great deal of real progress, but there will also be some bubbles; the 2 are not mutually exclusive.” The positive side is the full stack—materials, sensors, robot bodies, data, and models—all of which have Chinese companies working on them while resources continue to flow into the sector. “Over the long term, I remain very optimistic.”
• The bubble comes from the lack of route convergence. The focus moved from data to world models and has recently swung back to data. He uses control theory as an analogy: “It’s like tuning the parameters of a PID badly; it starts oscillating—overshooting.” But these are not mutually exclusive routes; they operate at different layers. Data determines “what the model is able to see,” while world models determine “what your representation is and how you understand the world.”

13. His Belief: World Models, Because Tasks Resist Enumeration

• He is explicit about his position: “I personally do believe more in the world-model route.” The reason is that tasks themselves are difficult to scale: “Tasks are an extremely easy way to become unscalable because there are simply too many of them. You can’t solve them through enumeration. But if a world model can learn the rules governing how the world changes, that is a more fundamental capability.”
• His watchlist is filtered by that belief. In the US, he names Physical Intelligence, Google, and Nvidia; among startups, Generalist and Roda (phonetic), which “has also been trying some in-context learning recently.” In China, he praises 林波’s new LingBot VLA 2, “which is also following the ICL route and is very meaningful work.”

14. Scaling Law Signals Are Emerging, but Loss Is Not Success Rate

• Asked about the title of the episode, he was measured: “I think we have seen some, but they’re not from us.” Results from other companies show that scaling pretraining data from 100K hours to 1M hours produced good results on holdout, unseen test sets. But most current evaluations measure loss, and “there is a tricky point in robotics: your loss is not directly linked to your success rate.”
• His example is worth quoting in full. A robot takes 20 steps to grasp a cup, and in the first 17 steps it may not touch the cup at all. “Even if you fit those first 17 steps extremely well, if the error is relatively large when the robot actually contacts the cup on step 18, the task can fail very easily.” Temporal imbalance means loss and success rate are often not directly correlated.
• The scaling law he actually wants to see is this: “As data gradually increases and models gradually get larger, the success rate on tasks you have never seen should gradually rise.” 扣姐 confirmed that there are signals today, but “they are not that strong.” He did not disagree.

15. Embodied Intelligence Is Still at GPT-1; He Wants the GPT-3 Moment

• Will robotics have a GPT moment? “I think it will. I still strongly believe in scaling.” The current mainstream embodied model is “pretraining plus fine-tuning for a target domain,” which corresponds to GPT-1. What he really wants is the GPT-3 moment: use ICL and prompting to give the model task information without fine-tuning. 扣姐 added: ordinary people should be able to use it too.
• He broadly agrees with Nvidia’s 金范’s presentation that embodied intelligence maps closely to the stages of language models, but adds 2 differences. Robot safety must be moved earlier: before deployment, robots need something like stress tests, which are “more important and more urgent for robots than for language models.” And robots have bodies, making their contexts more diverse. 扣姐 added the real-time constraint: waiting 20 minutes is “completely OK” for a language model, but with a robot, “if it takes even 1 second, I start wondering whether it has crashed and feel like tapping it.”
• He thinks benchmarks are even more necessary for embodied intelligence than for language. Developers and users need to know in advance “where the capability boundary lies,” including failure and risky scenarios. Among open-source benchmarks, he recommends BEHAVIOR for its many long-horizon tasks and rich household environments. The drawback is that it can be criticized for its sim2real gap, “but from the perspective of model capability, simulation is eventually just one kind of environment.”

16. 3 Ways to Fold Clothes: ICL’s End State Is User-Defined Robots

• His most important work over the past 1-2 months remains on the ICL route. He explains what it enables through folding clothes. Many robot demos fold clothes into squares, implicitly assuming that “everyone folds their clothes into squares.” But he uses a capsule wardrobe and wants to roll his clothes up; some people arrange clothes by color, while others hang everything without folding. “What in-context learning itself is trying to solve is giving the model the ability to be defined by the end user.”
• The deeper point is learning the speed of learning itself. A robot learning to cook a dish might initially need 30 minutes to adapt. After seeing 1,000 similar tasks, it might need only 10 minutes or less. “What we are training is the model’s ability to quickly learn new tasks.” As the system is deployed across more homes and service scenarios, “it will actually learn faster and faster.” That is why he believes ICL can ultimately allow models to scale up.
• Asked where the route is most likely to fail, he points to the data flywheel and its chicken-and-egg problem: “The base model needs to be good enough for the learning speed to be fast enough, but it also needs to enter real scenarios first so users can start using it.” Users have to see “obvious progress” after 30 minutes of teaching for the loop to turn. “Whether we can get this data flywheel running is actually the riskiest point.” His team’s initial loop uses a real dual-arm robot with dexterous hands on tabletop tasks, where the robot changes its behavior based on human corrections and demonstrations. The end state is full-body dexterous manipulation.

17. Offline Imitation Is Only Interpolation Generalization; Data Requirements Are Defined by Model Capability

• The fundamental difference from offline imitation leads back to algorithm distillation. Imitation learning treats the model as a task-memory module, with generalization relying on “interpolation generalization”: if the model has seen 5 objects and 10 positions on a tabletop, it can handle a new position within the same range. “But it has not actually learned the ability to learn a new task.” He wants models to see rich offline data—suboptimal trajectories, body gestures, verbal correction instructions, and human and robot demonstrations—to unlock capabilities “far beyond what merely imitating expert demonstrations can provide.”
• The difficulty with data lies in defining the requirement: “The definition of data does not come from nowhere; it is determined by the model’s capabilities.” Smooth actions require smooth data; failure recovery requires data containing failures and corrections. The problem is that “people’s definition of model capabilities is still relatively vague.” Should the model have to perform segmentation, tracking, depth and normal reconstruction, and other computer-vision tasks? ICL itself is also a new capability that still needs to be defined.

18. A “Fusionist” on Data Sources: The Value of Human Data

• He calls himself a fusionist, with each type of data assigned to its proper role. Teleoperation data is closest to the robot’s embodiment and suits later-stage fine-tuning and post-training. Simulation data is clean, can generate positional and texture generalization, and can vary factors such as color, making it suitable for pretraining and even mid-training. UMI data sits in the middle: it resembles robot data but costs less to collect.
• His view of human data is unequivocal: “Humans are indeed a very mature embodiment.” It is the lowest-cost option and can be scaled across different scenarios, providing robustness and understanding of the environment. To the objection that morphology differences are too large, he responds that human morphology is itself a morphology; effective morphology transfer can therefore be extremely valuable. What the model learns from human data is “how the environment changes, how objects change during a task, and task information that is not so tied to a specific shape.” His conclusion: “Whatever data you use, use it in the right place and build the link between data and model capabilities correctly.”
• His colleagues at IIIS—许华哲, 高阳, 陈建宇, and others—are all busy but stay in touch, sharing the broad vision of bringing robots into real-world use through scaling. Their detailed routes differ: some focus more on vision models, while others want to pursue a combined VLA-and-vision route. “In the short term, I think this is a good thing, because the field has not yet converged.”

19. Advice for Students, the Flood of AI Papers, and a Marathon at Kilometer 10

• He gives aspiring researchers 2 pieces of advice. First, read broadly: “The book gets thicker and then thinner again.” When pressed, many students’ interest in fancy concepts turns vague, which usually means “they simply don’t know enough yet.” Second, get hands-on early: “Real-robot deployment consumes a tremendous amount of experience.” When interviewing PhD students, he often asks, “What is the paper you are most interested in?” The best answer he heard came from a student who cited not a diffusion policy paper but the original diffusion-model paper—showing the ability to “trace back and find the true source of an idea.”
• Robotics papers are doubling in number, and many show heavy signs of AI authorship. “They look detailed, but in reality many of the arguments are full of holes.” Once he realizes a paper was written by AI, “my interest drops somewhat.” His filter is to check the work shared and promoted daily by researchers he follows; those papers have already passed a first screen in which the authors decided they were “worth being proud of and worth promoting.”
• His end-state assessment: the 42-kilometer embodied-intelligence marathon has “run 10 kilometers.” Robots emphasizing generalization and general-purpose capability are “still not actually deployed,” but “over the next 3-5 years, we may see a clear paradigm shift or a clear inflection point through scaling.” 扣姐’s closing line is worth keeping: “It is easy to be correctly pessimistic; only optimism can lead to success.”