E211 | At the Intersection of Content Creators and Robots: The Evolution of 3D Digital Humans
Summary
- Sora 2 proves that text-to-video can now give characters rich movement, but it still fails the production constraints of real-time digital humans. It remains limited to 10 seconds, mangles text in the background, and cannot finely edit character movements, expressions, or cross-frame consistency; customer-service, sales, and showroom digital humans must respond within roughly 1.5–2 seconds and “cannot have any flaws; they must be physically accurate.” The core divide is not whether the image looks realistic, but 2D pixel generation versus controllable, interactive, structured 3D.
- The cost advantage of 3D digital humans comes from compressing a person’s expression into hundreds of parameters, then replacing traditional rendering and physics simulation with AI. 魔珐 says its real-time latency is now 500–600 milliseconds, with generation costs roughly a few dozen times lower than LLM speech synthesis; even a low-cost RK3566 chip costing a few hundred yuan can run it. Pixel-by-pixel video generation, by contrast, lacks structured information, and at high concurrency every additional user adds another inference stream.
- 魔珐’s truly hard-to-replicate asset is not a single algorithm, but roughly 1,000 hours of high-quality 3D animation data and the capture capabilities behind it. 柴金祥 puts the industry cost of high-quality facial, hand, and expression animation at “at least RMB1,000 per second,” while qualified production teams are scarce worldwide; content companies have data but lack AI capabilities, and foundation-model companies have algorithms but lack 3D data. “Data is the core. Without data, no other R&D is possible.”
- The Nebula platform is taking 魔珐 from a project-based 3D production house to a scalable model and API provider, with the commercial inflection point coming from eliminating one GPU per stream. Six months ago, each real-time stream still required a graphics card costing roughly RMB20K–30K, and many customers abandoned the project after asking for a quote; after switching to AI rendering and physics solving, the platform has hundreds of B2B companies testing it, some already paying, and plans to launch in October, roughly 2 weeks after the interview. “If you can’t remove the GPU used for rendering and physics solving, you can’t really talk about applications.”
- AI rendering could rewrite the value chain of game engines, but for now it looks more like a runtime replacement than the disappearance of training-data sources. 魔珐 says that in its digital-human scenarios, no one can tell the difference in a side-by-side comparison between AI output and the highest-quality game-engine rendering; game companies could benefit as edge-compute and cloud-GPU requirements fall, Unreal may not, while NVIDIA is in a “left hand to right hand” relationship. Traditional engines still produce high-quality training data; models simply bypass them at runtime.
- The broadest entry point for digital humans is not one-off video, but becoming the visual interaction layer for every kind of foundation-model endpoint. 柴金祥 expects phones, PCs, in-car systems, TVs, showroom displays, and mini holographic screens eventually to need interfaces that integrate voice, movement, expression, and posture; companies will have a unified brand representative, while individuals “may have a digital twin.” That is why domain-specific voice, persona, lip-sync, and synchronized movement must be handled as one coordinated expressive capability rather than through a generic voice interface alone.
- Digital-human technology can feed back into robotics, but the robotics investment narrative remains constrained by real-world generalization and dynamics. 柴金祥 believes certain actions, such as walking in defined settings, are largely solved, but no company may yet reliably handle arbitrary staircases without falling or flexibly grasp objects in a home; the industry only believes VLA, reinforcement learning, and Scaling Laws might eventually work. His honest estimate for a robotic “GPT-3 moment” is that “2 or 3 years” is impossible to call, “10 years may offer hope,” and commercially “white-collar applications may arrive before blue-collar ones.”
Deep dive
1. Sora 2 Raises the Ceiling for Video—and Exposes the Limits of Real-Time Interaction
泓君 opened with Sora 2, released on the last day of September: it turns a single sentence into a 10-second short video in a format resembling social media. Compared with Sora 1’s focus on scene generation, the new version clearly shifts toward people and animals as protagonists, letting characters dance, eat, or converse.
柴金祥 acknowledged the progress in image quality and movement richness, but held to two judgments: the duration remains capped at 10 seconds, and although many clips seen while scrolling are attractive, “there are still too many flaws” when he generates them himself.
When 泓君 generated “a Silicon Valley 101 debate about AGI,” the word “Alignment” in the background turned into gibberish. More importantly, creators cannot correct only the text or a particular movement; they cannot make fine-grained edits as they would to media footage.
The most important signal from Sora 2 is that foundation models can now make the same character perform multiple behaviors. But 柴金祥 believes that if the goal is a digital human who can converse, dance, and entertain, the eventual training stack “may need to combine 2D and 3D training techniques.”
2. 3D’s Value Is Not Depth, but Real-Time Performance, Accuracy, and Control
泓君 summarized the distinction as Sora 2’s “text-to-2D video” versus 魔珐’s “text-to-3D.” The latter can enter VR or AI environments and, more importantly, carries directly controllable parameters for movement, posture, and expression rather than manipulating pixels alone.
A showroom digital human serves as the communication layer between people and machines. 柴金祥 puts the experience threshold at end-to-end latency below roughly 2 seconds, ideally 1.5 seconds; he says current real-time latency is about 500–600 milliseconds. Users cannot wait 5 or 10 minutes as they might for video generation.
Service and sales digital humans also cannot tolerate fingers multiplying, arms breaking, or body parts intersecting across frames. “There can be no flaws; it must be physically accurate”（不能有瑕疵，物理上是准确的）, because a generated video can be selected from multiple takes, while a live conversation has no window for retries.
Real time also means high concurrency: if 1,000 users interact simultaneously with the same persona, each receives different content, and inference costs could multiply by 1,000. Latency and unit economics are therefore the same deployment problem.
3. Hundreds of Structured Parameters Reduce Video Generation to Token-Level Economics
柴金祥 explains that describing a person’s movement and expression requires only hundreds of parameters, roughly corresponding to the control of hundreds of muscles. Hair and clothing then go through physics solving before the result is rendered as video on the screen. This follows a fundamentally different cost curve from generating every frame directly from unstructured pixels.
魔珐’s first model turns text into speech, facial expressions, body movements, gestures, and posture parameters. The second uses AI for 3D rendering and physics solving. Once the parameters are transmitted to the endpoint, there is no need to keep an expensive game engine running.
柴金祥 says the system can run on a Rockchip RK3566 costing a few hundred yuan, with costs “possibly only a few dozen times lower than speech synthesis.” His analogy is that generating only hundreds of parameters per frame makes the economics closer to a foundation model generating tokens.
The screen itself may sell for only about RMB10K. If a digital human continuously consumes an expensive GPU during every 20-minute interaction, the annual operating bill will not work. Low cost is not an experience optimization; it is a prerequisite for the endpoint product to exist.
4. Two Models in Series Form the Complete Human–Machine Interaction Loop
柴金祥 divides interaction between people and digital humans into two halves: the eyes and ears handle perception, while a multimodal foundation model understands, decides, and outputs text. 魔珐’s text-to-3D multimodal model then turns that text into emotional speech, expressions, movements, and gestures.
The first half can use models such as 千问, DeepSeek, or 豆包; the second is 魔珐’s own domain-specific model. The goal is not merely for a digital human to “say” an answer, but for language, lip movement, expression, posture, and gestures to convey it in sync like a real person.
At the time of the interview, the Nebula platform was still in testing, with hundreds of B2B companies using it and some already paying. The team planned to launch its text-to-3D multimodal model in October, roughly 2 weeks later, and let developers integrate the capability into their own applications.
泓君 therefore judged that 魔珐 was shifting from a 3D digital-human company toward a platform company. 柴金祥’s response was, “More or less.” The platform strategy is to package more than 20 years of accumulated know-how so developers no longer have to reinvent the wheel.
5. Professional-Grade “Human Creation” Remains an Expensive Industrial Pipeline
The traditional process starts with multi-camera scans of a real person, followed by reconstruction of geometry, textures, and materials, then modeling and rigging the muscle system. The actor subsequently wears a motion-capture suit, cameras record the movements, and a game engine or offline engine renders the final video.
柴金祥 cites the digital double of Jensen Huang shown at an NVIDIA event: “The person you see isn’t really me.” Achieving comparable realism requires joint work by R&D and art teams; by his estimate of industry costs, creating one person costs about $100K, with video production charged separately by the second.
This cost structure has kept 3D digital humans concentrated in professional content production such as Hollywood and AAA games. Ordinary people cannot create assets at the same quality level. The opportunity for generative models is to reduce the costs of modeling, animation, and final output simultaneously.
6. The Scarce Commodity in 3D Is High-Quality Data, Not Another Algorithm Team
柴金祥 sees an industrial divide: film, animation, and game companies are good at producing high-quality 3D content but mostly lack AI capabilities; foundation-model companies have strong algorithms but not enough trainable 3D assets. There is “very little overlap” between the two lines, making it difficult for either side to build a 3D foundation model independently.
Since 2018, 魔珐 first provided 3D content-production services for game, film, animation, enterprise, and virtual-idol customers while experimenting with combining AI and art, then shifted toward building its own data. The team has accumulated roughly 1,000 hours of high-quality 3D animation data covering faces, gestures, expressions, and body movements.
Those 1,000 hours cannot be compared directly with low-cost text or public video. 柴金祥 puts the domestic production cost of high-quality facial animation, gestures, and expressions at at least RMB1,000 per second, and teams capable of sustaining that standard must still be found. “Data is the core. Without data, no other R&D is possible”（数据是最核心的。没有数据，其他任何研发都没法做）.
魔珐 is also absorbing behavioral information such as walking and conversation from public video, but video has no native 3D structure. The current training path combines video data with scarce but precise 3D data rather than relying exclusively on either.
7. Digital Animation and Robotics Are the Two Ends of the Same Technical Lineage
柴金祥 entered Carnegie Mellon’s robotics institute for a PhD in 2000, researching interactive 3D digital humans and AI animation. He says the team was among the earliest in the world to use AI for animation. After joining Texas A&M in 2006 and founding a company in 2018, he has continued working on the same core problem.
His advisor, Jessica Hodgins, was studying humanoid robots when the robot “had only one leg,” because bipedal balance was so difficult. She later brought physical motion control from the real world into computer graphics, becoming one of the earliest researchers to create digital-human animation through physics simulation and control.
Around 2000, motion capture began to emerge and spread, giving researchers the conditions to introduce AI. 柴金祥 recalls that some of the earliest papers using reinforcement learning to create animation appeared around 2004 or 2005. Data moved animation from hand-built controllers toward learning systems.
That talent pipeline later flowed back into robotics: Sergey Levine studied animation during his PhD before becoming a co-founder of Physical Intelligence and researching robotic brains; Karen Liu has long worked on both Animation and Robotics. Virtual humans and robots share motion planning and control, with the real world adding hardware and physical constraints.
8. Hollywood, Games, and Real-Time Companionship Need Three Different Optima
柴金祥 breaks the product constraints into quality, cost, and use case. Hollywood can wait 100 or 200 hours and pay more for maximum fidelity; real-time services must respond immediately and can accept a lower visual ceiling, but not lower stability.
泓君 noted that Tilly Norwood had gained 65K followers within 6.5 months of her debut, with selfies, coffee drinking, and endorsements all approaching the look of a real person. 柴金祥, however, believes its production method remains PGC-style 2D content. When publishing daily, producers can generate 100 clips and select one, masking errors in the process; that cannot transfer directly to real-time interaction.
He cites the earlier Lil Miquela as a reference: she has roughly 1–2 million followers on Instagram, showing that virtual-human success also depends on social-media operations, persona, and professional content, not generation technology alone. For live concerts or real-time interaction, 魔珐’s structured 3D places greater emphasis on interactivity.
魔珐’s rollout sequence is daily interaction, services, and companionship first, followed by games, and Hollywood last. It can enter IP-derived content earlier, but film-grade core assets require higher-quality data, and there “may be only a few people in the entire world” capable of producing it.
9. The Platform Model Works Only After Eliminating One GPU per Stream
The Nebula API was technically more or less complete 6 months ago, but each user stream still required a graphics card costing roughly RMB20K–30K. Many B2B customers initially said they wanted to try it, then dropped out after hearing the price, forcing the team to solve unit economics first.
The main cost center is not text-to-motion, but real-time 3D rendering and physics solving; traditional solutions typically depend on Unreal and high-end GPUs. 柴金祥 once thought the problem could not be solved, but the team eventually used AI to replace rendering and physics solving simultaneously, moving computation to edge chips costing RMB100–200 or RMB200–300.
When 泓君 asked whether the API could make money, 柴金祥 replied, “Definitely.” His logic is not to subsidize growth, but that an AI startup in China that is not ByteDance, Alibaba, or Tencent must make the commercial math work first.
In 魔珐’s side-by-side tests for digital humans, the left side used a game engine and the right used AI rendering; “not a single person could tell” the difference. For the goal of deploying digital humans to every screen, that matters more than simply raising model image quality.
10. AI Rendering Could Weaken Game Engines’ Runtime Position
柴金祥 believes game companies could be direct beneficiaries: AAA games currently depend on cloud GPUs or high-performance endpoints, while phones heat up during runtime. If specific games can use edge-side AI rendering, distribution costs for games and future virtual worlds could fall substantially.
泓君 raised the risk that Unreal and Epic Games could face if they fail to update. 柴金祥’s qualification is that traditional engines will still produce the highest-quality training data, much as robots first collect data in simulation and only run the model at deployment. The phase most likely to be replaced is real-time rendering.
For NVIDIA, he describes the dynamic as “left hand to right hand”: cards lost in real-time rendering may be replaced by another source of demand. For Unreal, “it may not be particularly good news”; for game developers, it is more of an opportunity.
Genie 3 points in a similar direction: its training content can be produced by a game engine, but when generating interactive, 3D-like video, “you actually don’t need a new game engine.” 柴金祥 limits the claim to an AI-rendering model for “a particular game, a particular application scenario,” rather than declaring that general-purpose engines have been replaced.
11. The Next-Generation Interface for Foundation Models May Be an Expressive Face
柴金祥 cites a very famous psychologist’s 1970 claim that in human communication, visual signals may account for 50%–60%, voice for 30%–40%, and language itself for about 7%. His product judgment is that relying only on a text box or “speaking into the air” discards most of the information in communication.
Foundation models will eventually enter phones, tablets, PCs, in-car systems, TVs, showroom displays, and desktop holographic screens. Digital humans can restore voice, lip movements, expressions, posture, and gestures, making human–machine interaction more like human–human communication.
ChatGPT-style systems already cover multimodal input to text and text to images or video, but 柴金祥 believes they still lack “text to multimodal expression.” Connect the two segments, and a machine can see the user’s expression and understand speech while responding with complete body language.
Companies may have a unified customer-service or brand-representative persona, while individuals develop their own digital twins. 柴金祥’s vision for 泓君 is that while the person is resting, the digital twin can continue using their expertise to communicate with others and provide feedback.
12. Quality, Latency, and Cost Are the Three Mountains Scale Must Climb
柴金祥 summarizes the three metrics customers repeatedly ask about: whether voice, movement, expression, and lip-sync feel natural; whether an answer takes 5 seconds; and whether the improvement in experience justifies the investment. The system must also support different screens, operating systems, chip capabilities, and concurrency levels.
Quality starts with synchronized data. 魔珐 operates its own studio and records the highest-quality voice, lip movements, expressions, and movements together, because data in which “voice is voice and movement is movement” cannot teach the coordination present in real human expression.
The model must also extract emotion and intent from text: laughing, greeting someone, explaining something rigorously, or offering intimate companionship should all change TTS, facial expression, and posture together. The team therefore handles text-to-speech and the full 3D expression stack itself rather than simply connecting a generic voice API to animation.
A vertical persona is another reason to build in-house. Customer service, sales training, doctors, and interviewers require professional or serious voices, while consumer companionship may call for a cartoon child or a “cute puppy-like boy.” Customers with their own voice or text models can call the corresponding capabilities directly; customers with nothing can use the full platform stack.
13. Digital Humans’ Motion Models Can Naturally Become Training Data for Robots
魔珐 has already begun using the same capabilities to drive robots. A digital human can generate complete motion from face and hands through the legs, along with voice and posture; a robot can use the movements and gestures suited to its own body. Most robots currently lack facial muscles, so they cannot yet reproduce expressions.
柴金祥 divides execution into Kinematics and Dynamics. Kinematics first answers what posture and position the hand should use to pick up a cup; dynamics then determines how much force to apply for the hardware to reproduce that path and posture.
魔珐’s movements can enter simulation and then be adapted to a specific embodiment through imitation learning or reinforcement learning. Upper-body movements already show some generalization; full-body movements such as climbing stairs depend more on the robotics company’s balance control and hardware.
The team plans to release a 3D motion foundation model this year. A user could say, “Walk forward 5 steps, lie on the floor, get back up, then run,” and the model would generate the motion data directly. 柴金祥 believes this can reduce motion capture and train robots, because capture is fundamentally the collection of motion samples.
14. Robots Have Moved Beyond the Controller Era, but Not the Generalization Chasm
Early ASIMO-style robots relied on large numbers of engineers manually tuning controllers; simply walking on a fixed surface was a breakthrough. Today, vision, reinforcement learning, and NVIDIA-style simulation mean that any reasonably capable team can generally produce walking and running, but “whether it still avoids falling in another setting” is a different question.
泓君 noted that Boston Dynamics’ earlier stair-climbing demonstrations often used the same staircase. 柴金祥 said that once height, number of steps, speed, and friction coefficient change, the system may no longer be stable. He does not believe any humanoid robot today can reliably handle every unfamiliar staircase, adding, “If there is one, I’d like to learn from it.”
Grasping also remains a carefully staged demo. At an event, 泓君 asked a robot to open a can of Coke; staff required the pull tab to be pre-positioned toward its fingers, otherwise it could not rotate the can before opening it. A home with changing lighting, object placement, and obstacles would be harder still.
柴金祥 declines to name the “best” robotics company in the world: the embodiment, cerebellar control, and VLA brain are different tracks, and a path that leads today may ultimately prove unworkable. On a GPT-3 moment, he cannot call it in 2 or 3 years, but believes “10 years may offer hope.” Commercially, the digital world is more likely to create value first, while physical robots may see “white-collar applications arrive before blue-collar ones.”
泓君 characterizes the shift as moving from white boxes to black boxes: the past involved calculating forces and controllers one by one, while today data, reward functions, and reinforcement learning let systems adjust through experience. 柴金祥 agrees that this path has opened, but does not guarantee that Scaling Laws will converge; the industry is excited to have “just found a new path,” while downturns and unexpected challenges may still recur.