55. Behind the Making of a “Robot Brain”: VLM, VLA—What Doesn’t Change Is Better Perception

Summary

• The Beijing Humanoid Robot Innovation Center—the core team behind Tiangong Robot—has open-sourced Pelican VL (Chinese name: “天狐”), an embodied-brain model whose 72B version the team calls “probably the largest open-source embodied-brain model by parameter count currently available.” Joy compares the division of labor to “VLM as the brain and VLA as the cerebellum”: build the VLM first, then add an action head; 鞠笑竹 stresses that whatever direction the VLA architecture takes, visual-language understanding remains the core, and that the VLA route has yet to converge.
• The gap between general-purpose multimodal models and embodied applications is far wider than most people realize: in the team’s tests, once connected to a robot, models including ChatGPT could not even assemble three or four blocks, much less identify “the biggest one.” Joy called this “an extremely painful point”: large models that can write poetry and solve math and physics problems still cannot do things a 2- or 3-year-old can handle. The team began focusing on spatial understanding at almost exactly the same time as 李飞飞.
• The team does not believe that collecting as much data as possible is necessarily the best way to train its own models, but it acknowledges that scaling law exists. 鞠笑竹 uses an elephant that eats tens of kilograms of food a day while gaining only marginally in weight and intelligence as a metaphor for data efficiency; he noted that a Sequoia report’s “$200B question” became “the $600B question” a year later. The team developed DPPO (Deliberate Practice Policy Optimization), using RL for autonomous exploration and weakness discovery, followed by targeted SFT reinforcement, to improve data and training efficiency. Pelican uses roughly 1.8M data points—“medium-low” by industry standards—but has delivered better results on relevant benchmarks.
• The product lineup is tiered by compute and use case: the 72B cloud-based brain targets commercial and household applications with effectively unlimited demand, the 7B model targets industrial settings with fixed workflows, and the team plans to open-source a 3B model as a base for foundational VLA research. Joy says smaller models have limited capacity; injecting a large amount of new data can pull them away from their original distribution and cause them to forget existing capabilities. That is one reason the team released the 72B model.
• The benchmark suite covers Where2Place, RoboSpatial, long-video understanding and Nvidia Cosmos-related tasks; the signature example is the “sushi test,” in which only Pelican recognized that the task was complete and output “stop,” while the base model and GPT-5 did not. The video shows a robotic hand already removing the sushi, and 卫诗婕 stresses that a robot must first determine whether a task is finished; otherwise, it may continue making incorrect moves after the objective has already been achieved.
• The key divide between the VLA and VLM approaches is their data-collection paradigm: as 卫诗婕 puts it, VLA relies on humans repeatedly completing tasks through teleoperation to collect end-to-end data, while VLM+DPPO connects the model to a robot and lets the robot practice autonomously. Joy believes VLA may overfit to existing action trajectories and fail to transfer when an object’s color or shape changes; VLM, by contrast, could evolve into a “24-hour workhorse,” continuously collecting data as long as the robotic arm does not need to rest. The two approaches are complementary: VLA emphasizes action and uses smaller models suited to real-time edge deployment, while VLM emphasizes understanding and generalization.
• The reasons for open-sourcing include sharing the team’s results, gathering user feedback and offering a strong base model while embodied technology remains unsettled. The team wants to recreate the open-source ecosystems that drove the internet, computing and AI, and bring more people into the research process. Joy hopes to iterate quickly through V1, V2 and V3—“by V3, we might have our DeepSeek moment.” 卫诗婕 also compares embodied data collection with autonomous driving’s long-term road-data accumulation, noting that the physical world is even more complex and that embodied data and capability-building remain at an early stage.

Deep dive

1. From Motion Control to Building a Brain: Robot Backflips Are Routine; the Question Is “What For?”

• During his time at UBTECH, 鞠笑竹 mainly worked on motion control for large humanoid robots. At the time, robots had not even solved the problem of walking, let alone operating in real-world settings. The turning point came at the 2019 DARPA challenge, where reinforcement learning beat traditional analytical modeling in a specific scenario and took first place. As algorithms improved and sim-to-real challenges were gradually resolved, reinforcement learning began moving into humanoid robotics.
• Today, “robots doing backflips and dancing” is, in his words, routine. The question has shifted to what robots are actually for, and when they will be able to do housework or enter the workplace. The flood of videos showing remotely operated robots also highlights that the public increasingly wants to see robots complete tasks proactively and autonomously.
• Joy’s observation after moving from autonomous driving into robotics is that both fields must first perceive the environment and determine the next action, and they share some of the same sensors: autonomous vehicles commonly use cameras and lidar, while embodied-intelligence systems also frequently rely on both.

2. Imitation Learning Could Produce “A Task-Execution Machine With No Feelings”

• In his post-UBTECH work, 鞠笑竹 experimented with imitation learning in the RT series. His conclusion is that imitation learning could ultimately turn a robot into “a task-execution machine with no feelings”: no matter how much a person disrupts or sabotages it, the robot keeps pushing toward task completion without stopping to consider why the person is interfering or whether the task should continue.
• Once the mechanics of operation are largely solved, real intelligence still depends on the brain’s ability to think, plan and understand context. That is why the team is building an embodied foundation model.
• 卫诗婕 offered an intuitive definition of generalization: a rule-based robot breaks as soon as a task changes slightly, whereas generalization means that after learning task A, the model can build on that data, gradually transfer its capabilities to task B and create a flywheel. Joy agreed that this is why people are focused on VLA and on combining VLM with VLA—for example, to have robots cook, sweep floors and collect deliveries at home.

3. VLM Is the Brain, VLA the Cerebellum; Visual-Language Understanding Is the Core

• Joy compares VLM and VLA to the brain and cerebellum. VLM perceives the environment, understands changes and issues instructions; a VLA or another model downstream can carry out the actual operation. If it sees that a cup has tipped over, for example, VLM understands that the cup needs to be righted and passes down an instruction executable by the robot or downstream model.
• VLA may fail on a complex instruction such as “make breakfast” because a general-purpose vision-language model has not seen enough embodied data. It does not know how to convert semantic and visual information into robot-executable features and pass them to the action module. The team therefore wants to use VLM to turn visual semantics into representations a robot can understand, then attach an action head to create a more generalizable VLA.
• “Action” does not necessarily mean highly specific joint movements. It may instead be a high-level feature or instruction understood only by the robot and the model.
• 鞠笑竹 frames this as a judgment about the technology path: VLA has evolved from end-to-end RT architectures to fast-and-slow system decomposition and then to function-call-based approaches. But however VLA changes, the core remains VL—visual-language understanding. Close the gap between general intelligence and embodied intelligence first, and future action architectures can build on a better base.

4. “An Apple Is Just a Pile of Numbers”: The Distance From a General Brain to a Robot Brain

• A general-purpose VLM can converse and understand images and videos, but to become a robot’s brain it must also understand time and space. Joy says spatial intelligence is one of the next major problems in embodied intelligence: a model must know where objects are and how they change over time before it can walk, jump, grasp and pour in the real world.
• The model also needs function-calling capabilities to translate natural-language instructions or high-level plans into tools and actions that a robot can invoke.
• 鞠笑竹 explains that without contact with the real physical world, an apple is just a pile of numbers to a robot; it does not truly understand the apple. An embodied brain must use the robot’s body to touch the physical world and perhaps even “slowly develop feelings.” 卫诗婕 immediately jokes that this would produce an electronic boyfriend who is forever 18.
• 卫诗婕 also cites the three capabilities 李飞飞 discusses for world models—multimodal understanding, interaction and generation—and asks whether the first 2 are also central to VLM. Selina agrees.

5. The Block Experiment: Even Models at the Top of the Field Fall Short of a 3-Year-Old

• Joy describes one “extremely painful point” from the team’s block experiment: the robot was asked to reproduce exactly the structure formed by several randomly arranged blocks. After testing strong general-purpose multimodal models, including ChatGPT, the team found that even with a simple arrangement of 3 or 4 blocks, a 2- or 3-year-old could copy it while the large models could not.
• Ask the model to identify “the biggest block,” and it may still fail—even “completely miss the point.” The models can write poetry and solve math and physics problems, yet cannot build a single layer of blocks. To the team, that result was deeply counterintuitive.
• 鞠笑竹 and Joy break down the failure modes: the models confuse color and shape, have weak understanding of distances and dependencies between objects, do not know what capabilities they have for manipulating objects, and cannot plan a sensible assembly sequence. The team therefore needs to strengthen affordance understanding, chain-of-thought reasoning, planning and inference at the brain level.
• The team happened to see 李飞飞’s work on spatial perception and the first related dataset while running its own experiments, so the 2 groups began focusing on the problem at almost the same time.
• 鞠笑竹 adds that VLMs biased toward language can sometimes hallucinate large amounts of content from internal linguistic information even when their eyes are covered. 卫诗婕 summarizes the broader issue: general-purpose models learn primarily from 2D internet data, while the physical world is 3D. Watching a video of someone picking up a cup many times does not mean the model can perform the operation in reality.

6. DPPO: Making the Model Practice Deliberately Like a Student Preparing for Exams

• Joy compares the process with preparing for China’s gaokao: a student first builds a foundation, then works through 100 problems and finds that 50 remain unsolved. After repeated review, 30 are still out of reach, so the student asks a teacher for targeted instruction, masters those problems and moves on to harder ones.
• In model training, the RL stage lets the model solve problems and explore repeatedly. When it occasionally succeeds, it also reviews why that attempt worked and uses the process to identify problems it still cannot solve at all. An SFT teacher then explains those weaknesses in a targeted way and gradually brings the model into embodied scenarios.
• DPPO stands for Deliberate Practice Policy Optimization. 卫诗婕 summarizes it as “autonomous practice under video guidance (RL) → identify weaknesses → supervised fine-tuning,” while Joy calls it “a process of turning a model into a specialist.”
• The training approach also aims to prevent capability forgetting: people do not forget how to ride a bicycle after learning to drive, and a model should not lose its video-understanding ability simply because it learned left-right relationships.

7. Elephants and Human Brains: Acknowledging Scaling Law While Emphasizing Data and Training Efficiency

• 卫诗婕 relays a view she heard in Silicon Valley: pool the large sample databases of several companies, then discover new intelligence in massive, diverse datasets. The more diverse the datasets and tasks, the more intelligence they may generate.
• 鞠笑竹 says scaling law definitely exists, but that collecting as much data as possible is not necessarily the best way for the team to train a strong model. Joy adds that video contains temporal information and some spatial information; as the camera moves, a model can extract part of the spatial and physical structure from 2D video, but never 100% of it.
• When 卫诗婕 asks whether data such as Open X—which includes robot manipulation, physical interaction and proprioceptive motion information—is more “nutritious,” 鞠笑竹 uses an elephant as an analogy: it eats tens of kilograms of food every day, yet gains only a little in weight and intelligence. Companies with sufficient capital, compute and data can afford that approach; startups may not.
• DPPO was designed as a finely tuned training recipe to maximize the efficiency of data and compute utilization. Even with a small team, the goal is to train a model with solid performance that can compete with models currently on the market.
• 鞠笑竹 broadens the issue from startup constraints to the industry as a whole. A 2023 Sequoia report called it “AI’s $200B question”; a year later, the title had become “the $600B question.” If training and data efficiency do not improve, continued investment will become a heavy burden for companies and nations. 卫诗婕 adds, half-jokingly, that the team faces staggering GPU bills every day and is afraid to go to sleep at night.

8. 7B, 72B and the Planned 3B: Parameter Count Maps to Use Case

• Joy explains that a model’s capacity is its parameter count. A 7B model is like a small storage space: inject a large amount of new data and it may be pulled away from the distribution it originally learned, forgetting existing capabilities. The smaller the capacity, the more likely this loss of memory is to occur. That is one reason the team released the 72B model.
• 鞠笑竹 divides the use cases this way: commercial and household environments have effectively unlimited user demands and require stronger general capabilities and generalization, making the 72B cloud-based brain appropriate; industrial workflows are relatively fixed and may not need a 72B model, with 7B potentially sufficient.
• 7B is also suitable for students and research institutions that cannot train or run 72B, and can serve as a research scaffold. The team plans to open-source an even smaller 3B model as a stronger base for foundational VLA research.

9. Benchmark Results and the “Sushi Test”: Only Pelican Said Stop

• In the comparison chart Selina presented, the white bars represent the Qwen2.5-VL-72B base model and the red bars represent Pelican. The team says the model delivered marked gains across multiple benchmarks under the DPPO framework.
• The evaluation covers standard embodied tasks. Where2Place determines free space—for example, when placing a teapot on a table, the robot cannot choose a location occupied by a cup, flowerpot or another object. RoboSpatial detects object locations, and another task evaluates long-video understanding from a robot’s first-person view. Selina says Pelican outperformed several current open- and closed-source models on these tasks.
• The more intuitive example is the “sushi test”: the model receives a video and is asked what the robot should do next. The robotic hand has already removed the sushi from the pan, and only Pelican recognizes that the task is complete and outputs “stop”; neither the base model nor GPT-5 makes the same judgment.
• 卫诗婕 stresses that recognizing completion is essential for a robot. Once the sushi has been removed, the robot should not continue with irrelevant or incorrect actions and risk undoing a task that was already completed.

10. Long-Horizon Task Demo: Dividing Work Between Arms, Retrying Failures and Function Calls

• The instruction in the demo is: “I’m heading to work. Please put the shoes on the shoe rack, put the garbage in the trash can and put the clothes in the washing machine.” Pelican receives the natural-language instruction, remembers where the shoes and clothes are, decomposes the task autonomously and generates subgoals.
• This planning is not “what you see is what you get.” The model must recall what it saw earlier and where each item is, while also knowing which skills it has. If one arm is holding the clothes, it must plan for the other arm to open the door; if the washing-machine door does not close on the first attempt, it must recognize the failure and retry.
• The text shown in the demo is the model’s chain of thought and reasoning, while the terminal in the lower right displays calls to action tools and environment-perception tools. Function calls convert the large model’s understanding into specific instructions that downstream operation and motion modules can execute.
• The setting is not limited to the home: the team has also tested generalization in supermarkets and laboratories. Selina wants intelligence to move beyond the screen and into the real world. 卫诗婕 says supermarkets and shelf-moving were among the most frequently demonstrated scenarios at this year’s World Robot Conference, while 鞠笑竹 sees relatively strong commercialization potential in business services.

11. Physics Is Everywhere: From High-School Physics Problems to Dodging a Duckling

• The team also compared Pelican with other models on a physical-common-sense reasoning benchmark, showing an upper-secondary physics problem involving heat and input in a complete cycle. Both 卫诗婕 and Selina said the problem was difficult; Selina said Pelican performed better than the other models.
• Joy explains that physical understanding means understanding the laws of the physical world. If a model does not know that tipping over a cup will spill the water, it cannot make the corresponding judgment.
• In a real-world setting, if a duckling suddenly runs into the robot’s path, the robot must use the duckling’s speed to estimate how long it will take to stop or evade it. Autonomous driving likewise requires an understanding of speed, acceleration and time to collision. Joy adds that daily life also involves phenomena such as electricity and heat—for example, when to bring in clothes after they have dried or how to repair a failed light bulb—so the number of physical laws a model must understand is enormous.

12. Affordance: A 1977 Psychology Concept Becomes Central to Embodied Intelligence

• Selina explains that affordance, first proposed by a psychologist in 1977, refers to a person’s ability to infer from an object’s form and function what actions it may support.
• An empty garbage bag can be lifted with 2 fingers; once it is full of fruit, the robot must find a different support point and carry it from underneath. A long door handle must be pressed down before pushing, while a round knob must be turned before pushing. A person picks up a teapot by its handle, but a robot may mistakenly grab the spout.
• Perception, grasping and placement, flexible manipulation and long-horizon planning all ultimately depend on understanding affordances, spatial relationships and temporal order. Together, they form the critical loop that takes an agent from understanding the cognitive world to acting in the physical world.
• In a multi-view task that progressively generates affordances, 3 cameras film the same scene from different angles. Pelican must infer which objects are interactive and which locations are suitable for grasping or placement, while maintaining consistency across views: the same banana looks different from different angles, but the model must still recognize it as the same object. It then uses triangulation from 2D to 3D to convert that understanding into concrete action coordinates.
• In the grasp-and-place demo, the model first plans a trajectory and decomposes the task, identifies the locations of the bread and plate, then uses bounding boxes and grasp points to narrow the target area before executing the action.
• Joy distinguishes between 2 layers of capability. VLM first needs open-vocabulary understanding—recognizing phrases such as “a small apple” or “a cute apple.” Affordance is then added so the model knows how to manipulate the object. Combined with Where2Place to identify the most appropriate placement location, the system can complete a full pick-and-place task.

13. Force Control: Estimate First, Then Correct Through Tactile Feedback

• For gripping hard and soft objects differently, Pelican uses a humanoid-style closed loop: it first observes the object’s shape and material, the brain estimates the required force based on experience, and a force-control algorithm adjusts that estimate using tactile feedback from the fingers before storing the information in the memory module.
• When it encounters the same object again, Pelican can use its prior gripping experience to apply a more appropriate force, making the grasp more accurate, faster and more efficient. 鞠笑竹 adds that the model must understand an object’s size and material and estimate how much force is enough to lift it without crushing or deforming it.
• If 2 objects look identical but are made of different materials, the model may initially make the wrong judgment. Real-time feedback can detect the deviation after the grasp, correct it promptly and update the information in the model.
• Joy believes VLA may need to collect more force-contact data, often by repeatedly gathering data on the same type of object after seeing it once. If the object’s color or shape changes, VLA may fail to transfer the skill because some current VLA systems are overfit to existing action trajectories and do not fully exploit the controllability and world knowledge of large models.

14. Teleoperation and the “24-Hour Workhorse”: 2 Data-Collection Paradigms

• Joy describes the team’s experience operating teleoperation equipment firsthand: the device had no force sensing, so whatever the object, it might simply squeeze all the way down and treat picking it up as success. A VLM-based model, by contrast, can use its understanding of materials, friction and related knowledge to grasp objects less brutally.
• 卫诗婕 summarizes the data difference between the 2 approaches: VLA typically has a human repeatedly complete tasks through a teleoperation system, collecting end-to-end data to train the model; VLM+DPPO connects an embodied brain to a robot and lets the robot attempt the tasks on its own.
• Joy says VLM could strive to become a “24-hour workhorse”: as long as the robotic arm does not need to rest, it can continuously collect data, and that data may in turn be used to train VLA. VLM aims to understand how to grasp a class of objects, rather than collecting an equivalent amount of data from scratch every time the object changes.
• The 2 approaches coexist because their roles are complementary. VLA teams focus more on action, use relatively smaller models and are suited to real-time task execution at the edge. VLM teams focus more on the upstream understanding layer, with the goal of feeding knowledge into VLA to improve generalization, success rates and overall performance.

15. 1.8M Data Points, Manual Cleaning and “Interns as the First-Order Production Force”

• Pelican uses roughly 1.8M data points. Joy considers that volume “medium-low” for the industry, but the dataset contains a relatively high share of video and therefore carries more information, helping the model perform well on long-video understanding.
• The dataset also includes images, object locations, bounding-box detection, question-and-answer pairs, multiple-choice questions and other formats. Embodied-intelligence datasets have many dimensions because the model itself needs to master a wide range of capabilities.
• Yoyo says the data is fairly messy when it first arrives. Humans must inspect it item by item, determine which samples are useful for the model and then combine that process with model-based filtering. Fully reviewing data at the million-item scale takes enormous effort and is highly error-prone.
• 卫诗婕 introduces Yoyo, a second-year master’s student, through an industry joke sparked by MiniMax’s training-work announcement that “interns are the first-order production force.” Yoyo studied Information and Computing Science at Zhejiang University as an undergraduate before entering Peking University’s School of Software and Microelectronics. She has competed in academic competitions since childhood and enjoys independently researching and solving problems, which gives her a strong sense of accomplishment from coding work.
• The livestream also deliberately highlighted the work of young female engineers. 鞠笑竹 says women in technology are making important contributions at the frontier and in the newest fields. 卫诗婕 cites research showing that women face a relatively high probability of leaving STEM careers midway through their working lives and says she hopes more people will see women’s contributions to hard tech.

16. The Open-Source Bet Is a DeepSeek Moment; the Name Points to Absorbing Wisdom

• Joy says open-sourcing is a way to share the team’s research and help more people discover it, while also allowing more users to try the model and offer feedback. The team remains an early-stage startup and needs outside input.
• 鞠笑竹 adds an industry rationale: embodied-intelligence technology has not converged, especially with VLA still under exploration. The team wants to provide a relatively strong base so researchers do not have to take a general-purpose model and retrain every dimension of capability from scratch. It also hopes to recreate the open-source ecosystems that powered the internet, computing and AI.
• He believes VLM is currently in a relatively stable phase, but the model will later incorporate more modalities such as touch and depth—capabilities that general-purpose models do not naturally possess.
• Joy hopes open-sourcing will accelerate iteration through V1, V2 and V3: “By V3, we might have our DeepSeek moment.”
• The closing Easter egg explains Pelican’s Chinese name, “天狐.” 鞠笑竹 says the character “狐” is taken from the “hu” in 鹈鹕, and recounts online memes involving Pelican, capybaras, little goldfish and Doubao Niang. The team’s base model is Qwen, whose cartoon mascot is a capybara. Pelican can “eat” these characters, but in English the point is not to say eat; it is to absorb the essence of their wisdom and stand on the shoulders of giants. The Tiangong robot also described a video of a Pelican interacting with a capybara.