After LLMs: Spatial Intelligence and World Models — Fei-Fei Li & Justin Johnson, World Labs
Summary
World Labs is betting that spatial intelligence will complement language as the next foundation-model frontier, with compute finally large enough to attack it. Justin Johnson estimates roughly 1,000x more performance per card since AlexNet and training across hundreds to tens of thousands of GPUs, producing “a millionfold more” compute per model. Visual, spatial, and world data require far more processing, making world models a plausible next scaling frontier.
Marble is a deliberately two-sided wedge: a useful 3D product now and the first public step toward general world models. It accepts text, one or multiple images, generates editable 3D worlds, and supports precise camera placement and export; emerging use cases include gaming, VFX, film, interior design, and potentially robotic simulation. World Labs intentionally tried not to make it a pure “science project,” while Fei-Fei Li calls Marble “the first glimpse” of a larger spatial-intelligence stack.
The current technical choice—Gaussian splats—turns generation into navigable geometry rather than a sequence of video frames. Splats render in real time on mobile and VR, although targeting 30–60 fps at high resolution on a four-year-old iPhone caps density and fidelity. Other systems already use frame-based generation, and future systems could attach mass or springs to particles or use token-based representations; the data structure is not treated as permanent.
Physics is the largest unresolved capability gap and the boundary between a creative tool and trusted engineering software. Pattern fitting might predict plausible orbits without deriving force vectors or “F equals MA”; Fei-Fei says there is “no indication” latent modeling yields causal law, while Justin hopes emergent physics appears at scale. Plausibility is enough for a film backdrop, but not for a building that must stand.
The commercial expansion path is horizontal, but its ordering remains intentionally unsettled. Creative industries are Marble’s beachhead; interior-design beta users already reconstruct and edit rooms, while robotics could use generated worlds as the “important middle ground” between scarce real-world data and uncontrollable internet video. Fei-Fei says whether to move directly into embodied use cases “is to be decided.”
World Labs’ discussion does not call for throwing out transformers; it points toward multimodality and a richer learning loop. Attention remains, and Justin notes that transformers natively model sets—the 1D order comes from positional embeddings—so spatial tokens need not require architectural demolition. The deeper missing ingredient may be hypothesis, action, falsification, and online updating, not a change of modality alone.
The talent and research bottleneck extends beyond capital: academia is under-resourced and too tempted to imitate frontier-lab scaling. Fei-Fei defends open benchmarks such as BEHAVIOR; Justin wants academia pursuing “wacky ideas,” including distributed primitives beyond matrix multiplication as clusters replace single GPUs. His hardware warning is concrete: Hopper-to-Blackwell performance per watt is “about the same,” leaving room for 10–20-year architectural bets.
Deep dive
1. A millionfold compute jump opened the world-model window
Their connection began in 2012, when Justin joined Fei-Fei’s lab in the quarter AlexNet appeared. After his work at the University of Michigan and Meta, Justin’s interests had shifted toward 3D vision, graphics, and generative modeling. When they began talking again more than two years ago, they found they had independently reached the same “what’s beyond language models” question and chose to “put all the eggs in one basket” by forming World Labs.
Asked for world models’ AlexNet-equivalent breakthrough, Justin named enabling conditions rather than one model: about 1,000x more performance per card since AlexNet, plus the move from one GPU to “hundreds or thousands or tens of thousands or even more.” A single model can consequently marshal roughly “a millionfold more” compute than at the start of his PhD.
Language was the first large-model domain to work well, but Justin’s next-step logic is that visual, spatial, and world data simply require much more processing. Fei-Fei’s destination is correspondingly broader: models that can “understand, reason, generate” across multimodal worlds and eventually support interaction approaching the complexity of human interaction.
2. Academia’s problem is resources, not an open-versus-closed binary
The hosts’ pushback—worth keeping—was whether ImageNet could be recreated when a lab holding the best dataset must earn a return and PhD students are recruited earlier into industry. Fei-Fei said she has concerns, but more about the “imbalanced resourcing of academia” than commercial pressure. Justin framed the ecosystem as a mixture of open, closed, proprietary, and public models, comparing different business models to iOS and Android.
Her counterexample is Stanford’s open BEHAVIOR dataset and benchmark for robotic learning in simulated environments. Through Stanford HAI, Fei-Fei has also advocated public-sector compute and data infrastructure, including work during the first Trump administration on a bill called the National AI Research Resource, or NAIRR, encompassing a national compute cloud and data repository.
Justin’s reframing: academics could train state-of-the-art systems on a few GPUs five or ten years ago, but losing that ability means “the technology actually worked.” Academia should stop pretending it can win the largest-model race—or becoming vocational preparation for access to industrial GPU clusters—and instead pursue “wacky ideas,” new algorithms, architectures, and systems. Fei-Fei added theoretical underpinnings, interdisciplinary work, and blue-sky science.
His specimen is hardware-aware architecture research. Today’s networks center matrix multiplication because it fits GPUs, yet the unit of compute is becoming a node or cluster while models are still conceptualized as monoliths. Asked whether NVIDIA had simply won the “hardware lottery,” Justin answered “yes and no”: Hopper-to-Blackwell performance per watt is about flat, making alternative distributed primitives a plausible 10–20-year academic project, not a three-month startup cycle.
3. Image captioning foreshadowed both multimodality and product instinct
Fei-Fei once expected to spend “the rest of my entire career” solving one problem: given a scene, tell its story in natural language. Combining convolutional-network image representations with an LSTM produced their CVPR 2015 captioning system, generating a sentence from an image—a result she had thought might take 100 years.
The team believed it was first, then learned Google had reached a similar result simultaneously. New York Times reporter John Markoff discovered both efforts and covered them together, preserving a useful lesson from that era: major model ideas can emerge independently once representations, compute, and data cross the same threshold.
Justin then worked with Andrej Karpathy on an ICLR 2015 language-modeling paper—“I should have stuck with language modeling”—training small RNNs on data including Linux source code. Their empirical interpretability work found LSTM units whose activity correlated with structures such as an opening parenthesis and switched off at its close.
Dense captioning at CVPR 2016 joined three representations: the whole-image gestalt, proposed regions, and text for each region. The joint network drew boxes and emitted descriptions in one forward pass; Justin also built a webcam demo served from Stanford. At ICCV 2015 in Santiago, predictions traveled from California at a terrible “1 FPS,” but “the fact that it worked at all” mattered.
4. Pixels preserve structure, but pattern fitting does not guarantee physics
Fei-Fei distinguished the modalities: generative architectures may share components, but a “deeply 3D, 4D spatial world” has structure fundamentally different from a one-dimensional signal. Justin nevertheless defended “pixel maximalism”: text is physically seen with the eyes, and tokenization discards font, line breaks, and 2D layout, while pixels offer a more lossless—if less efficient—representation. Fei-Fei also noted that sound can be encoded as a signal.
Fei-Fei cited a Harvard paper in which a model produced plausible planetary orbits but “wacky” force vectors. Justin laid out the two paradigms: explicitly train on forces generated by a physics simulator, or train end to end and hope an internal physics-like representation emerges. Fei-Fei’s warning was sharper: “There’s no indication” latent modeling reaches causal laws; contemporary deep learning still fits patterns.
Marble’s arches made the distinction concrete. A model can render a structure that looks right without knowing how its central stones bear load. For virtual film production, Justin said plausibility may be sufficient; for an architect producing a building that will exist physically, modeling forces matters because “you don’t want the thing to break.”
Fei-Fei said she did not think anything fundamental separated visual and architectural generation, framing the issue as one of scaling data and improving the model. Justin made the emergence claim explicitly conditional: other large models have acquired unexpected capabilities at scale, and he hopes that transfers, “but that’ll be a process that we need to play out.” His caution is that model cognition, if it exists, may be an “alien form of intelligence” unlike human introspection.
5. Marble couples a foundation-model program to a usable 3D product
Marble takes text, an image, or multiple images and generates a matching 3D world. Users can edit counterfactuals—make a bottle blue, remove a table, rearrange objects—and export several formats. Fei-Fei calls it the first publicly accessible model generating 3D worlds at this fidelity, but also “the first part of that journey,” not the finished spatial-intelligence system.
Justin described a deliberate balancing act: build capabilities needed for general world models while avoiding a “science project.” Marble already hints at understanding varied inputs, modeling different worlds, and representing how they could change; meanwhile, gaming, VFX, and film users can extract value today. The product supplies both an immediate commercial wedge and a foundation for larger models.
Precise camera control is a revealing capability rather than a cosmetic feature. Video generators often force users to describe movements such as pans without exact geometry; Marble lets users position and record a camera because the system has a 3D scene beneath it. Justin stressed: “You’re not generating frame by frame.”
Today Marble natively outputs Gaussian splats: semitransparent particles with positions and orientations whose composition produces the scene. They render efficiently on client devices, including phones. Justin emphasized that this atomic unit is contingent: World Labs’ RTFM model generates frames one at a time during interaction, while future architectures might generate tokens representing chunks of a 3D world.
6. Dynamics, robotics, and design extend the same horizontal stack
Justin offered two routes from static splats to dynamics. World Labs could predict mass, springs, or other physical properties for particles and pass them into classical or learned simulation; alternatively, a model could regenerate the entire scene after every action. Regeneration is more general because it is not limited to predefined properties, but substantially more computationally demanding. Existing physics engines can also generate training data—their behavior “distilled” into network weights.
Fidelity is partly a product constraint. World Labs wants scenes to run cleanly on mobile and VR, including 30–60 fps at high resolution on an iPhone from four years ago, which limits splat count. A newer phone, MacBook, local GPU, lower frame rate, or lower resolution relaxes that constraint and permits denser, higher-resolution worlds.
Fei-Fei positioned robotic training between two constrained extremes: high-fidelity real-world data is critical but scarce, while internet video is plentiful but lacks controllability. Simulated worlds are therefore an “important middle ground,” supplying diverse states in which embodied agents can act. Marble is more creative-industry-focused today, and whether robotics arrives immediately remains undecided, although Justin said it “may be going to happen sooner than later.”
Horizontal capability is already creating unplanned design uses. Fei-Fei said users can reconstruct a kitchen from two images and edit countertops, floors, and cabinets; Justin said he had used Marble for his garage, and beta API users are building interior-design applications. Yet the launch exposed ordinary execution risk too: Justin could not find advanced editing, prompting Fei-Fei’s concession—“We can improve on our UI”—and a jokingly direct hiring case for product talent.
7. Spatial intelligence restores the embodied substrate language abstracts away
Fei-Fei questioned the phrase “a data center full of Einsteins.” Drawing on the idea of multiple human intelligences, she defined spatial intelligence as complementary to linguistic intelligence: the capacity to “reason, understand, move and interact in space.” World Labs is not proposing language’s replacement, but filling a capability class language alone does not capture.
Her high and low examples carry the same mechanism. In Fei-Fei’s simplified account, conjecturing DNA’s double helix involved reasoning about molecules and bonds in 3D; grasping a mug requires perceiving the mug, hand, context, geometry, and affordance point. Language can narrate either process, but “that narrated language itself cannot get you to pick up a mug.”
Justin calculated that speaking continuously at about 150 words per minute produces roughly 215,000 tokens per day—tiny beside the bandwidth of lived experience. Fei-Fei added the evolutionary asymmetry: nature spent about 540 million years optimizing perception and spatial intelligence, versus a generous estimate of roughly half a million years for language. Vision feels easy because it is deeply optimized, not because it is simple.
Newtonian theory illustrates the hierarchy. Justin argued that laws are written only after extensive embodied experience observing the world; language then abstracts above native spatial understanding. LLMs have “jumped all the way” to highly abstract reasoning, while spatial intelligence reopens the black box to recover what may have been lost in that jump.
8. Causal discovery needs experimentation, while transformers remain usable
Justin proposed feeding a model celestial-motion data to see whether Newtonian law emerges; Fei-Fei offered her “honest guess”: it probably would not. A sufficiently trained model might predict a fictional planet’s day-by-day trajectory accurately, yet abstractions such as “F equals MA” or “action equals reaction” occupy a different level. Justin sharpened the issue by asking how a model trained on visual data would avoid a geocentric account.
Justin located the missing capability in the learning paradigm, not strictly the modality. Humans form theories, act, compare outcomes with expectations, and update online—the mundane example was going downstairs for keys, failing to find them, and revising the hypothesis. Scaled up, that loop of experiment and falsification produces physics. A host framed the same process as efficiently eliminating possible worlds inconsistent with evidence.
Neither guest wants language removed. Marble already accepts language, future systems may combine specialized models, and Fei-Fei allowed that “maybe one day we’ll have a universal model.” A host separately linked theory of mind with emotional intelligence and warned that people may already depend “probably too much” on systems that do not possess those capabilities.
Fei-Fei expects world models to move beyond sequence-to-sequence architectures, but Justin clarified that this need not discard transformers or attention: “You stick with stuff that works.” A transformer natively models a set of tokens; 1D order enters through positional embeddings, while its token-wise operations and permutation-equivariant attention can support other arrangements. Spatial modeling therefore changes the represented structure without demanding that every proven component be rebuilt.