Talking Agents, Multimodal & Embodied AI with Former Hunyuan Lead 淏楠

Summary

• In 淏楠’s view, the next leg of opportunity in agents lies in broader scenario coverage and a better user experience. He thinks models already outperform the average AI engineer at coding in most cases, while areas such as game agents remain underdeveloped. Greater intelligence only becomes useful through GUI, CUA, and product interaction. “Agents are clearly the main interaction paradigm right now”; multimodal creation will be organized around that same idea.
• Video generation still depends more on pretraining, data, and capital, and 淏楠’s team never saw the “R1 moment” it expected. He began the year thinking RL could deliver an efficient leap, but changed his view after six months. His caveat: post-training scaling has not been efficient; that does not mean more data cannot improve results. Around 10 experienced engineers can build reasonable infrastructure in 2 months. What is hard to replicate is accumulated data and product feedback.
• Video apps still have a wide gap in controllable creation and interaction. Asked by 曲凯 about Kling 2.5, 淏楠 rated it around 60 out of 100 and explicitly predicted better studios or harnesses would emerge within the next six months. Rerolling will persist, but may shift from filtering out distortions, face swaps, and other errors to “they’re all good; you and I just have different preferences.”
• Real-time video is not mature, and embodied AI may not need to wait for a complete world simulation. Speeding generation up can break consistency, while improving quality often requires larger, slower models; 淏楠 cannot predict when genuinely real-time interaction will arrive. He is more persuaded by an approach that understands the current environment, takes one action, then updates its understanding: “You don’t need to understand the whole world before doing one thing.”
• 淏楠 favors an embodied-AI split between a large-model brain and proprietary hardware with smaller models. He says embodied-AI companies have “two orders of magnitude” fewer GPUs than large-model companies. A more realistic startup moat is to co-design hardware and System, use a 7B–16B model to control the robot, and connect it to an external large model. Real-robot data still matters, especially for dexterous-hand movements that crude simulation cannot reproduce.
• Astra’s most important potential value is taking over failed steps and closing the data loop. 淏楠 imagines that once a small model reaches a 50%–60% success rate in unfamiliar settings, GPT can take over failures and generate error-recovery data, eventually lifting success from 60% to 80% and allowing one operator to monitor 10 machines. He endorses “soft distillation” of the takeover steps only; having a large model output mechanical joint angles throughout is “too slow and too expensive.”
• Embodied AI’s high valuations still need to be earned in real-world settings. Asked by 曲凯 about a peak valuation of around $20B and roughly RMB5B–RMB6B on the balance sheet, 淏楠 pointed to hotel bed-making: flat floors make wheeled robots viable, and the work sits between standardized and non-standardized. He moved into physical AI as a long-term career choice, but his bar for the industry is direct: “Real deployment, in real scenarios.”

Deep dive

1. Agents have become the main interaction paradigm; next comes getting them into more use cases

• 王淏楠’s early experience, from July 2025 through February or March 2026, was in language agents, mainly SFT and web-search data construction. By the end of 2025, he thought that staying on would probably mean “spending a whole year making data,” which did not fit his interests or his desire to build up his skills.
• Having worked on both sides, he felt caught in a classic grass-is-greener dynamic: agent teams dislike cleaning data, while video teams worry their work does not draw on enough intelligence. But agents are the main interaction paradigm, and if one company fails to execute, you can move to another and keep working on them; multimodal roles depend more on the company’s direction. The understanding track has also picked up again with GUI, including WorkBuddy and GPT-6 Extra.
• 曲凯 suggested personal AI may be the end state for agents, but 淏楠 still sees gaps in scenario coverage, including game agents. He thinks coding ability now exceeds the average AI engineer’s in most cases, making incremental gains harder to describe. The next question is “how to make it useful to everyone”; as for the next breakout, he admitted he has no idea.

2. The RL gap is mostly about compute and accumulated experimentation; engineering is only one part

• 淏楠 joined Hunyuan Multimodal at 庞天宇’s invitation to lead RL infrastructure, with UniRL as the main focus. He stressed that RL for images and video requires a closed loop across infrastructure, algorithms, and rewards; a well-trained reward model is especially important, and all three have to advance together.
• On the gap between China and overseas teams, his qualification was blunt: “The biggest gap, I think, may simply be that we don’t have the GPUs.” Engineering practice is not that far behind, but a compute shortage means fewer experiments, shorter training runs, and less accumulated experience. If you rent cloud compute for a month and just want to push scores up quickly, SFT is indeed more efficient and better at brute-force scoring.
• Multimodal work has not sidestepped agents. The team wants to build “the Cursor for multimodal”: generate 3 video options, let a user choose one or suggest a fourth version, then iterate from the selected direction. Post-training would center on this human-computer interaction, rather than only improving a single generation.

3. Video did not get the expected R1 moment; competition is back to pretraining and data

• At the start of the year, 淏楠 thought video generation might have an R1 moment like language reasoning. After six months, he changed his view: “There isn’t an efficient scaling law.” He was referring to the post-training path his team explored; even that, along with rewriting captions, did not deliver the expected leap.
• Asked by 曲凯 why leading video models still perform so well, 淏楠 first pointed to data quality. He allowed for an exception: as he understands it, ByteDance’s post-training is working, while RL has not worked for many other teams. Even where it does work, he said, the most important capability gains still come from the pretraining that came before it.
• He did not reject scaling itself: more data can still move performance forward; the question is efficiency. The bar for pretraining infrastructure is relatively low: a little over 10 experienced people can reach a “reasonably good level” in 2 months. Open source has also narrowed the engineering gap, but there is no equivalent shortcut for data.
• Data either costs money or comes back through a product. He recalled that teams could still scrape YouTube in late 2024 and early 2025; anti-scraping measures have since made video much harder to obtain. This path therefore depends more on capital, large teams, and company-level accumulation. His team tried to break through with a group of strong people, but “six months went by without success.”

4. Video is only at 60 points; rerolling will shift from error-filtering to aesthetic choice

• 曲凯 cited the Bilibili short drama “Invincible Superman” as evidence of quality gains, while wondering whether it involved a lot of professional creative work. 淏楠 agreed that human input still matters and said getting “close to movie quality” is achievable. His score for Kling 2.5 was still only 60; the technology is nowhere near its capability ceiling.
• The value of human input is more about using taste to steer the work. 淏楠 thinks coding has widely shared standards of quality, while film involves differences in preferences around camera language and expression. Continuity and editability are common requirements; creators still need to influence how a scene is shot.
• “Rerolling will definitely remain,” but its purpose will change. Today, people reject clips with distortions or poor consistency; in the future, every clip may be good, with the difference coming down to taste. 曲凯 compared it to a director shooting multiple takes and choosing one, and 淏楠 agreed.
• Rich captions can already control shots in detail, but asking users to type out long descriptions is still exhausting. A better approach would be for the machine to propose several shots based on the preceding context and let the user choose. He also cautioned that making a short drama remains “extremely painful”; context management and other parts of the workflow are still inadequate.

5. Once understanding and generation converge, platforms still have to clear the creation hurdle

• Asked whether one-off capabilities such as copying a popular video or swapping its subject would be absorbed into models, 淏楠 said yes. His core view is that “understanding will definitely help generation”: you need to understand the people and visual details before you can edit them well. He was less certain that generation would in turn help understanding.
• There are far more video consumers than creators. The barrier is not that people cannot learn, but that creating is cumbersome and costly. He and others in the field hope that over the next six months to a year, lower barriers and generation costs will let more people become both producers and consumers, creating a new platform.
• On the product side, he was pessimistic about images—“there’s not much left to do; it’s mainly about video”—and made a clear prediction: “Within the next six months, a better interactive studio or harness will definitely emerge,” putting model capabilities in more people’s hands.

6. Real-time video is still trading consistency for speed, with no clear timeline to maturity

• 曲凯 distinguished between 2 approaches: one that generates faster than people consume and only looks real-time, and another that can continuously respond to dialogue and movement. 淏楠 thinks almost everyone is still on the first path, which is itself immature; the longer generation continues, the more likely consistency is to break.
• He thinks their product is at H3.0, while model capability is roughly at the 2.0 level, with failed clips still occurring. The stronger part of 2.5 also includes controllability: it can take in many reference images of a person from the front and back, hairstyles, storylines, as well as video and audio, enabling more usable commercial products.
• Schedulers, layer skipping, distillation, and one-step generation can all speed things up, but often make results more likely to break. His example: when the camera turns back, objects or faces in the scene may already have changed. Personally, he would prioritize quality.
• 曲凯 noted that this is a trade-off at the current stage and that both paths can advance in parallel; a future approach might also sacrifice some capability for speed and lower cost. 淏楠 accepted that possibility but could not predict when it would arrive. Teams are still tackling capability limits—“we haven’t reached the point of surplus capability”—and larger models are inherently slower.

7. A world model need not reconstruct the whole world before it can serve embodied AI

• On video world models that take commands such as up, down, left, and right, then continue the scene, 淏楠 cited demos resembling running around in Zelda. People are working on them, but their quality and usefulness remain uncertain. Rather than simulating an entire world with a huge model, he thinks “a VLM like Astra paired with a smaller Diffusion model for rendering” looks more promising.
• 曲凯 asked why the market had shifted from excitement to disappointment when researchers should already have known how difficult this path was. 淏楠 thinks researchers may never have expected to build it quickly; market sentiment “put them on the hook,” demanding immediate delivery without necessarily having thought through what a completed system would be used for.
• He stressed that 李飞飞’s 3D approach and 杨立昆’s latent approach are not the same kind of world model. Embodied AI does not need to model grass, leaves, and lighting before acting: “I just need to understand the current situation, take one action,” then observe and act again. That may be enough.

8. Astra’s breakthrough is turning large models’ visual understanding into physical action

• Moving from video to action is technically natural: action can also be a modality. The same diffusion approach can move from video plus audio to video plus action. Embodied systems take video, or video from multiple angles, and output vectors with perhaps 20-plus dimensions, each representing a physical quantity such as a joint angle—not a 4096-dimensional latent.
• In the past, asking a language model to write “0.96,” then a comma, then the next string of numbers token by token, before parsing them into motor angles, was awkward and did not work. 淏楠 thinks Astra’s change is stronger reasoning: it can follow instructions to produce sequences that are genuinely meaningful and controllable.
• Non-general-purpose hardware remains a challenge, especially dexterous hands. The notable capability, he said, is that the model can move a joint, inspect the video, and connect changes in variables to their visual effects, learning over multiple rounds how its own outputs affect the hardware. “That’s really powerful.”
• 淏楠 stressed that this time, the model was trained on more embodied data. Asked by 曲凯 whether OpenAI’s dedicated embodied-AI effort or its main large-model line contributed more, he chose the latter. From what he has seen, “folding it into the main line is what works best”: generalization from a stronger flagship model.

9. Simulation can provide a training environment, but real-robot data still determines operational quality

• 曲凯 asked whether the reward comes from a visual judgment that the robot has “picked up the apple.” 淏楠 did not claim to know the mechanism; he speculated that the capability may transfer from other domains. For example, a model might use Blender and inspect rendered results to learn the relationship between actions and changes in 3D visuals, then transfer that knowledge to embodied AI.
• The value of simulation depends on who is using it. A large-model company can treat it like a game environment; an embodied-AI company ultimately cares whether its own robots work in the field, so it needs traces from the exact hardware, including successful and failed runs and trajectories after a human takes over.
• The dexterous-hand example shows the gap: picking up a bottle with 2 fingers may count as a successful grasp, but the real goal is to wrap all 5 fingers around it. A simulated success may not meet the product’s actual requirements, so models still need to be trained and adjusted on data from the corresponding real robot.
• He left room for the direction to reverse: if simulation becomes sufficiently good, real-to-sim-to-real could indeed scale, and both embodied-AI and large-model companies could move that way. The hard part is getting details such as physics and friction coefficients right. Today, “the simulation isn’t good enough.”

10. Brains are moving toward platform competition; smaller companies’ moat is hardware-System co-design

• 淏楠 recalled the old System 0, 1, and 2 framework and thinks things may now converge into 2 systems: one layer that directly controls the body, and a large model above it with strong language and reasoning. That is also why he thinks embodied-AI companies should embrace Astra.
• 曲凯 cited a peak valuation of around $20B, roughly RMB5B–RMB6B on the balance sheet—less than $1B—and a market with only 5 or 6 large-model companies splitting the GPUs versus more than 100 embodied-AI companies. 淏楠 described the compute gap between the two sides as “two orders of magnitude.” Startups lack accumulated experience, GPUs, and funding, making it hard to compete by retraining enormous models.
• 曲凯 noted that embodied-AI companies generally have their own robot bodies. 淏楠 emphasized that the moat is “whose hardware is better” and how tightly it is integrated with the System. He favors co-design: pair a company’s own product with a 7B–16B small model, connect it to an external large model, and get the full system working and deployed quickly.
• He thinks people who want to do embodied AI are better off at an embodied-AI company. The supply chain is long, like in cars or phones, and margins are not like software; large companies also have less control over the chain. Big tech is a good place for model research, but departmental boundaries make it harder for model trainers to collect data or teleoperate robots themselves and understand problems from the ground up.

11. The long-term draw is physical productivity; valuations need real-world use cases to justify them

• 曲凯 asked why 淏楠 joined the field when valuations were already high and Astra had just shaken up the landscape. 淏楠 said video still has opportunities, but he personally wants to work on productivity and physical AI over the next 10 or 20 years. If he had optimized for short-term returns, he would have stayed in agents; he had also received a quite substantial package from TikTok in the US.
• On AI for science, he distinguished between using AI to solve ODEs and PDEs—for example, planning Wi-Fi placement in obstructed settings—and using AI to help with chemistry experiments. The latter needs facilities capable of wet-lab experiments, much like the offline data-collection system required for embodied AI. He thinks that area may still be early, and progress in embodied AI could help it.
• He sees hotels as a promising use case: flat floors make wheeled robots viable, while bed-making has a fixed process but varying degrees of mess, putting it between standardized and non-standardized work. Whether it can lower costs and replace some hotel housekeeping is the practical test for whether high valuations can rise further.

12. Having a large model take over failed steps could close the data loop

• Asked whether Astra was an embodied-AI GPT-3 moment, 淏楠 said “somewhat.” His scenario: a model that can achieve a 50% or 60% success rate in a setting it has not been trained on is already doing well. Let it operate first, then have GPT take over when something goes wrong, potentially replacing some teleoperators.
• The system could read the video in real time to decide when to intervene, route the rest of the operation to GPT, and keep the resulting high-value trajectory. What happens after a failure—error recovery—is especially important training data in its own right.
• His hypothetical iteration: feed the data back into the company’s own 7B or 8B model and lift its success rate from 60% to 80%. The small model keeps operating, failures go to GPT, and only problems GPT cannot handle reach a human. That could take the ratio from 1 teleoperator watching 1 machine to 1 person watching 10.
• He compared it to speculative decoding: the small model drafts, and the large model approves or takes over. As the success rate rises, both GPT usage and the burden on people fall. Having GPT routinely output mechanical joint angles directly is “not worth it,” because “it’s too slow and too expensive.”

13. Local soft distillation is more viable for embodied AI than having a large model demonstrate every step

• 曲凯 asked whether Astra’s capabilities could be distilled into open-source embodied models. 淏楠 first distinguished “hard distillation”: having a large model roll out traces and then training on them. He thinks that is too costly in embodied AI—“there’s no need”—and does not make economic sense.
• He prefers “soft distillation”: a small model fails partway through a task, the large model takes over for just 1 or 2 steps, provides a normal trajectory, and then hands control back. “You only distilled those 1 or 2 steps.” The data volume is small, the value is high, and the cost can be covered. He thinks embodied-AI companies are the likeliest to do it because they can deploy hardware and build data collection into the workflow more easily.
• 曲凯 said the market and valuations over the next few years remain uncertain, but new possibilities have clearly opened up. 淏楠 called Astra a game changer: for the first time, he felt, “Damn, this thing can really be deployed; it can really work” (“真的落地，真的能够有场景”). His final expectation still carried the original scale of surprise: “With generalization this strong, it’s not even 3.5.”