Moonlake: Interactive, Multimodal World Models — with Chris Manning and Fan-yun Sun
Summary
Moonlake draws a hard line between video generation and world modeling: a real world model must be action-conditioned, interactive, and consistent over long horizons. Photorealistic output from Veo 3, Genie, Marble, or Sora may look convincing without representing 3D state or predicting what an action changes minutes later. The decisive test is: “Can I interact with the objects in the world and see the right consequences of actions?”
The efficiency thesis is “structure not scale,” tempered by the explicit concession that “scale is good, too.” Internet video supplies observations but rarely the actions that caused them, while language already packages abstractions such as “professor” or “condescending” into meaningful tokens. Sun argues that semantic structure could let Moonlake work with “five orders of magnitude less data” than a pixels-only approach while still benefiting from scale.
Manning’s contrarian bet is that language and symbols are cognitive infrastructure, not low-bandwidth exhaust. Against Yann LeCun’s more vision-first worldview, he points to language, mathematics, and programming as “cognitive tools” that enable abstraction and extended causal reasoning. He concedes JEPA’s joint-embedding premise, but argues transformer weights themselves might hold a joint representation even when outputs are generated autoregressively.
Moonlake separates causal state from visual fidelity through two models: a multimodal reasoning system and the Reverie diffusion renderer. The first handles causality, persistence, logic, and game state, including geometry, physics, affordances, and audio triggers; Reverie restyles that persistent representation without sacrificing interactivity. A bowling demo makes the architecture concrete: picking up the ball, knocking down pins, changing the score, resetting the game, and reproducing consequences all matter more than a plausible-looking clip.
Gaming is the commercial wedge, with an unusually ambitious rendering claim attached. Sun believes Reverie could become “the next paradigm of rendering,” replacing rasterizers and DLSS while enabling “skins for worlds” and programmable effects—after collecting 10 apples, for example, bullets could become apples as part of the game loop. The products are also intended to create a data flywheel in which creators reveal which capabilities need improvement.
Embodied AI is the adjacent market: generate a distribution of environments tailored to a goal, then train and evaluate policies inside them. Examples range from rescue drones to a vacuum robot navigating one office robustly; the end product might be the simulation itself or evidence showing where a policy fails. Sun does not claim this solves reward hacking—his answer is “not necessarily” when the reward remains misspecified.
Evaluation remains use-case-specific and resistant to a single leaderboard. Proposed end metrics include time spent in a generated game, whether a designer can produce the imagined experience in reasonable time, and real-world robustness after simulation training. Manning expects substantial “voting with their feet,” because beautiful graphics can conceal broken gameplay just as static benchmarks can miss whether an assistant gives useful shopping advice.
Execution is early and technically constrained, but the build plan is specific. Moonlake has about 18 people, is based in San Mateo, and is planning to move to San Francisco; it acknowledges both a data constraint and difficulty reaching photorealism through structured generation alone. Hiring targets the intersection of code generation, computer vision, graphics, game engines, reinforcement learning, and multimodal latent-space alignment to build a self-improving loop across text, audio, image, and video.
Deep dive
1. Interactive data, not prettier footage, created the opening
Fan-yun Sun traces Moonlake’s origin to PhD work with NVIDIA Research generating interactive environments for reinforcement-learning and embodied-AI agents. NVIDIA and academic groups were paying substantial sums for manually curated worlds used in training and evaluation, revealing a supply problem rather than merely a graphics problem. Sun also says co-founder Sharon’s work with Fei-Fei Li and Andrej Karpathy, followed by work with Ron and Chris Manning, connected him to Chris.
In related foundation-model work, Sun’s team generated synthetic data that proved “as useful as real-world data when it comes to multimodal pre-training.” His conclusion was that embodied general intelligence requires models to learn the consequences of their actions, while demand for the interactive data needed to teach those consequences is “growing exponentially.”
Manning’s diagnosis is broader: computer vision has historically been far larger than language research, yet visual understanding “sort of stalled out” after object recognition. In contemporary vision-language models, he argues, “the language is doing 90% of the work and the vision barely works,” because mainstream systems remain too close to surface-level pixels.
His own shift began with visual question answering, where models exposed their lack of semantic grounding by answering “two” whenever asked how many people sat at a table, regardless of the image. Students’ enthusiasm and creativity, together with work across language, vision, and generative imagery, pulled him further toward the world-model problem.
2. A world model must predict consequences over minutes, not frames
Manning’s operational definition is an “action-conditioned world model”: given an action, it must predict what changes in the world because of it. Next-frame prediction can be comparatively easy; spatial intelligence requires likely consequences minutes into the future, which in turn demands an abstract semantic model and persistent state.
The first scaling obstacle is that easily mined video is observational. It records what happened without labeling the actions that caused the transition, forcing a model to infer hidden actions from outcomes—a path Manning calls possible but very hard and “not really established” at any scale. Simulation attracts interest precisely because it provides known actions.
Text has a structural advantage before training begins. Every token already carries meaning, while descriptions such as “professor” and “condescending” encode abstractions that would require enormous processing to recover from pixels. Sun says that if explicit structure permits progress with “five orders of magnitude less data,” Moonlake can iterate faster and more cheaply without rejecting the benefits of scale.
Human perception supplies the analogy: people process the focus of attention in detail while representing the rest of a scene top-down as rough semantics. That abstraction supports Moonlake’s other requirements—real-time operation, long-term planning, and consistency—and resembles Physical Intelligence’s reported use of text to preserve long-horizon memory rather than retaining everything as pixels.
3. Symbolic cognition is Moonlake’s deepest break with Yann LeCun
Sun rejects the claim that abstraction violates the bitter lesson. A maximally literal multimodal system could encode images, video, and audio as bytes and train next-byte prediction, but the required compute would be prohibitive. The live research question is therefore “what is the right abstraction level today?”—not whether scale matters.
Manning’s disagreement with Yann LeCun is philosophical. LeCun treats visual experience as intelligence’s high-bandwidth substrate and language as low-bit-rate human communication; Manning counters that chimps already possess strong vision, memory, planning, and primitive tool use, while human language produced a “vaulting” increase in knowledge, construction, and long-range reasoning.
Borrowing Daniel Dennett’s phrase, Manning calls language a “cognitive tool,” alongside mathematics and programming languages. He accepts JEPA’s goal of a consistent joint embedding, but disputes that autoregressive models are disqualified: although tokens emerge sequentially, the transformer’s internal weights can plausibly constitute a joint model of the world.
4. The bowling demo turns causal reasoning into visible product behavior
Moonlake’s published traces show an agent reasoning through geometry, physics, affordances, symbolic logic, perceptual mappings, assets, and state before constructing a world. A bowling prompt therefore expands into far more than graphics: the system must decide what matters for learning and encode the mechanics that make repeated practice meaningful.
When the ball hits a pin, pins must fall, audio must trigger, the score must increment, a timer may advance, and resetting must begin a genuinely new game. The score must also correspond to the number of fallen pins. Without those linked consequences, Sun argues, the user cannot learn “what it takes to actually get a high score.”
swyx’s pointed comparison is that Google’s Genie demos and World Labs’ Marble do not provide interactive worlds in this sense. The test is not merely whether something happens after one prescribed interaction, but whether a player can choose among “50 options” and infer the plausible result of each.
The hosts’ Unity pushback—worth keeping—is that this might simply be prompt-to-code generation over a conventional engine. Sun’s answer is that physics engines, code, and software are “cognitive tools” selected as means to an end; another task might instead require fluid dynamics or a different representation. The same model can already configure multiplayer and a persistence database when prompted.
5. Reverie makes appearance a programmable layer over persistent state
Moonlake divides the problem between a multimodal reasoning model, responsible for causality, persistence, logic, and determinism, and Reverie, a diffusion model responsible for pixel distribution. Reverie takes the persistent representation produced underneath and restyles it into photorealistic or arbitrary aesthetics while respecting the world’s existing interactivity.
Sun believes this “is going to be the next paradigm of rendering,” potentially replacing rasterizers and DLSS. Rather than having diffusion invent an entire scene without spatial understanding, Moonlake uses diffusion atop semantic state to offer customizable “skins for worlds”—including the photorealistic game treatments sought by modders.
More radically, rendering can enter the gameplay loop instead of remaining a passive derivative of state. Collecting 10 apples might turn a player’s bullets into apples because a state transition changes the renderer’s conditions. The discussion points to combining text with reference images and assets, preserving a layer where human intent can be expressed explicitly; as Sun puts it, “we’re not going to be more creative than our users.”
6. Utility, not visual polish, is the only credible evaluation target
Sun’s evaluation framework begins with purpose. For games, the direct metric might be time spent in the generated world; for embodied deployment, it is robustness in the target environment after training in generated worlds. Current benchmarks are proxies for these end metrics, and the appropriate proxy changes with every use case.
Manning sees the same crisis across AI. Question answering, logic, math, and object recognition are measurable component tasks, but they do not capture whether an assistant conducts a useful conversation about the right backpack for next month’s European trip. Likewise, “can a game designer produce what they are imagining in a reasonable amount of time?” is meaningful but difficult to benchmark.
Manning’s likely resolution is “people voting with their feet,” much as users choose among GPT-5, Gemini, and Claude based on experienced utility. swyx emphasizes that visual appeal is particularly misleading: primitive-looking games can succeed through concept and mechanics, while teams can spend millions on photorealism and still produce a game that “sucks.”
The hosts probe alternative worlds through Baba Is You and Ted Chiang: change one rule—gravity, time, or the logic of objects—then preserve consistency everywhere else. Sun says code-based execution makes such changes more controllable than a video model trained mostly on conventional reality, though he also notes that code-based approaches can be narrow, overtrained, and limited by how creatively the rule is expressed.
7. The symbolic-pixel boundary will move as customer value becomes clearer
The discussion revisits Sora 1’s 2024 framing of video models as world simulators. Sun accepts that pixel-coherent simulators are valuable for games, marketing, and other visual work, but says they are less useful than assumed for causal reasoning and embodied AI. Moonlake’s bet is that a “disproportionately large share of value” lies in tasks where high-resolution pixels are unnecessary.
A host’s three-body-problem analogy exposes the limit: symbolic simulation may eventually have to solve the world deterministically while video merely approximates it convincingly. Sun reframes this as a fluid boundary between diffusion priors and symbolic priors. Moonlake reviews that boundary continually, moving behavior toward whichever representation handles a customer requirement more economically; his bias is that physics belongs with symbols.
Commercialization begins with a gaming beta and a data flywheel: creators use the system, reveal missing capabilities, and guide model improvement. The three-year platform vision is broader—users specify an outcome, from teaching children humility to training rescue drones, and Moonlake generates a distribution of environments for creation, training, evaluation, and failure analysis.
The limitations are explicit. Sun cites data as a constraint and says more data is better; structured generation does not immediately match video models’ photorealism; and symbolic worlds do not inherently prevent reward hacking. Manning also argues that Sora cannot produce compelling mechanics or gameplay history, while a host counters that AAA games increasingly blur into “a 2-hour movie.”
8. Spatial audio reveals why multimodality must share one world state
The audio challenge is not TTS layered over video. Games require effects, background music, echoes, reflections, and spatial changes tied to simulated events. Moonlake gets part of that structure from the underlying engine and code, then lets the model combine those tools with other audio systems—the kind of capability Sun describes as emerging from interoperable “LEGO blocks.”
The discussion contrasts this with generated videos where music or soundscape is attached without a shared causal model. swyx’s concrete Sora 2 test is simple: have a dog move away from the camera and ask whether its sound diminishes. Sun answers that Sora 2 has no spatial audio; lip-sync or overlaid voice is not spatial audio.
Moonlake ultimately wants one combined representation across audio, text, image, and video. Sun’s intuitive example is hearing a car skid away with closed eyes and mentally extrapolating its trajectory. Agents could act in generated worlds, collect long-horizon video, encode experience back into the model, and drive the self-improvement loop suggested by the company’s “reflection” imagery.
Sun says the name Moonlake was partly chosen for a DreamWorks-like creative vibe, while the moon’s reflection evokes the self-improvement loop. Building that loop still requires people. Moonlake has about 18 employees, is based in San Mateo, and is planning to move to San Francisco; it is seeking researchers spanning code generation, computer vision, graphics, game engines, reinforcement learning, and multimodal latent-space alignment. The sharper recruiting test: “if you’ve written a game engine before,” Moonlake wants to talk.