102. A Conversation with 张祥雨 on the Struggle of Multimodal Research and 2 “GPT-4 Moments” in the Next 2 Years
Summary
- 张祥雨 sees 2 technical inflection points in the next 2 years as potential “GPT-4 moments”: multimodal reasoning within 1 year, and autonomous plus online learning within 2 years. The former would connect image understanding, controllable generation, and interleaved image-text CoT into a single chain; the latter would let models set their own goals, absorb environmental feedback, and update continuously instead of freezing after training. His view is optimistic but conditional: “the prerequisite technologies are all ready”; how many problems remain afterward is unclear.
- A trillion parameters do not guarantee better reasoning: Step-2 had a trillion total parameters and more than 200B active parameters, with very strong humanities and writing performance but mathematics at one point weaker than 7B. In controlled experiments across 1B, 7B, 30B, and 70B models, knowledge, dialogue ability, and “EQ” continued to rise with scale, while mathematical reasoning first improved, then plateaued, and finally declined. Larger models are more prone to “skipping steps,” and one local error can ruin a long chain. For foundation-model investment, parameter count can no longer serve as a standalone proxy for intelligence quality.
- This counterintuitive curve points to a target mismatch in next token prediction: it rewards more faithful compression of human corpora, but does not guarantee getting the problem right. If half the training samples state the answer directly and half show step-by-step derivations, a large model can fit both peaks and report complex local results “in one shot” with more than 90% probability; a small model, constrained by its capacity, may instead learn more reliable step-by-step calculation. “A higher compression ratio does not necessarily correspond to higher computational accuracy.”
- O1’s decisive breakthrough was not a new RL algorithm, but putting patterns such as reflection, backtracking, and verification back into the action space. PPO, GRPO, REINFORCE++, and even vanilla policy gradient can all work; the real bottleneck is that pretraining compresses a solution space spanning thousands of tokens into fewer than 10 critical decisions, while over-compressing the rare reflection paths in human data. 张祥雨 calls this “CoT’s CoT”: the model does not merely follow one chain of thought, but switches among multiple chains.
- Multimodal fusion stalled because the core problem was not merely feeding visual tokens into a language model; visual generation was still stuck in a “no take-backs” one-shot regime. Step-1 mixed image and text from pretraining onward and achieved strong image understanding and text alignment, but removing its attached diffusion generation branch had almost no effect on understanding. More than 6 months of alternating training produced 2 separately stronger models, but no “1+1>2” effect. The generator created images that violated perspective and physical common sense, while the understanding model knew they were wrong but could not control it.
- The scarce resource in visual reasoning is action patterns connected to natural pretraining data, not simply more synthetic problems. Artificially generated dots, counting marks, and maze annotations can solve their corresponding tasks but barely generalize. O3’s seemingly primitive crop and resize operations are more robust because natural data such as repair websites contains abundant “full image—local enlargement—text explanation” structures. A synthetic pattern that “can represent only itself” struggles to activate the broad, interlocking capabilities embedded in pretraining.
- The near-term breakthrough is to constrain generation to domains that do not require complex reasoning, first acquiring highly controllable visual actions and then letting CoT repeatedly correct them. The latest Sora and Gemini results convince 张祥雨 that instructions such as connecting points A and B, removing a person, changing an object’s position, or generating charts by structure are already within reach of current architectures. The final output in text is understanding; the final output in images is generation; the middle is unified into interleaved image-text CoT. Near-term competition will therefore depend more on data cleaning, difficulty stratification, and closed-loop training than on naming a new architecture.
- The long-context arms race may confuse “fitting it all in” with “using it well”; next-generation systems are more likely to rely on hierarchical memory and multi-model collaboration. Transformer keeps 100K, 1M, or even 10M tokens in the same context, creating little incentive to compress and generating attention interference. 张祥雨 envisions short-context models handling local execution while a global model handles overall perception and planning, with RL teaching the system end to end when to clear and switch contexts. At the Agent level described by OpenAI, the key is not today’s workflow assembly but “finding its own goals, exploring on its own, and learning its own value.”
Deep dive
1. Scaling, not architectural tricks, has defined 张祥雨’s research for more than a decade
张祥雨 attributes AlexNet’s 2012 success to its historical timing: ImageNet had already supplied more than 1.2M images in 2009, CUDA compute was becoming mature, and the only missing piece was model scaling. Earlier networks had only a few million parameters; AlexNet suddenly reached “more than 700M,” filling that model-scaling gap.
During his PhD from 2012 to 2016, he continued expanding width, depth, and resolution. MSRA init eased gradient explosion and vanishing during width expansion, while ResNet pushed networks from the teens of layers to dozens, hundreds, or even more than 1,000 if desired. The host also noted at the opening that his papers have accumulated more than 370K citations.
After joining Megvii in 2016 or 2017, he instead turned toward small models for edge devices. As long as a large model is pointed in the right direction, the details of interlayer connections often matter little; once compute is constrained, hardware execution characteristics and architectural details determine final efficiency.
This led to NAS and SPOS in 2019: searching for the best model for a given hardware target and dataset at relatively low cost. His long-term conclusion was that “architecture is not important” does not mean architecture is unnecessary; architecture always serves the objectives of algorithms, data, and systems.
2. NLP’s real leap came from a scalable objective, not Transformer itself
张祥雨 believes the main driver of NLP’s progress around 2019 was not Transformer, but escaping manually labeled training. As long as every example required annotation, the data could “never scale up.”
BERT used a cloze-style task to learn representations from unlimited unlabeled text before transferring them to downstream tasks. The upstream task appeared unrelated to hundreds or thousands of downstream tasks, yet more data produced stronger associations and knowledge.
GPT’s decoder-only design and next token prediction went one step further: it required no labels and could unify many downstream tasks through in-context modeling. NLP’s route was not truly complete until at least GPT-3.
3. Self-supervised CV never reproduced the language model scale law
In 2019, Google’s semi-supervised work such as UDA pushed contrastive learning rapidly toward ImageNet-level supervised representations. In 2020, ViT brought pure Transformer into CV, followed by iGPT, BEiT, and the later strong-performing MAE.
张祥雨 remained “cautiously optimistic”: these methods looked impressive on small models, but their gains decayed rapidly when scaled up, unlike NLP’s curve in which more data produced more capability.
His diagnosis of contrastive learning is that its core invariance comes from artificial augmentation rather than being data driven. Rotations, color perturbations, and multi-crop rules can be learned from limited data; adding more data does not automatically generate new invariances, so the information gain does not continue.
MIM learns invariance to occlusion: even with part of a cup hidden, people still know it is a cup. That is useful, but only a necessary condition, not a sufficient one. 张祥雨 publicly argued in early 2022 that MIM had no obvious scale law and was criticized at the time; relatively few people now pursue that route.
4. Static images do not form a closed loop of generation, understanding, and human alignment
In 2022, 2 papers by Jason Wei on chain of thought and emergence deeply shocked 张祥雨. Vision research was still stuck at the representation layer, while NLP had begun studying reasoning and higher-order intelligence, prompting him to reassess the pure-vision route.
In the GPT framework, natural-language generation, understanding, and human alignment can be unified. Modeling the joint distribution enables generation; changing the preceding text changes the conditional probability of what follows, which itself constitutes understanding; and because the corpus is produced by humans, it naturally provides distribution-level alignment.
Static images are different. Even fully modeling the joint distribution of every pixel guarantees only that the system can generate images, not that its internal representations will match human understanding. “Images are created by nature”; human semantics such as occlusion invariance and object concepts are not naturally encoded in pixels.
Video may provide some of the missing information through multiple views, motion, and occlusion over time, but it is harder to exploit and has lower effective information density. By 2022, 张祥雨 had largely stopped working on static-image representations and shifted toward language, multimodality, and video.
5. Multimodality became the escape route, but the industry badly underestimated data requirements
张祥雨’s route had 2 layers. In the short term, web pages, papers, journals, and public-account posts with interleaved images and text could let images borrow language’s closed loop for understanding and alignment. In the long term, video and larger systems could provide detailed processes, physical motion, and visual intelligence.
This idea was not trusted in 2022. China’s 2021 boom in 100B- and trillion-parameter models generally produced limited results: data quality and instruction learning remained unsolved, and the industry had badly underestimated data scaling.
He uses GPT-3 as an example: pairing 100B parameters with only several hundred billion tokens now looks “completely upside down.” A model of that size should have been paired with more than 10T or even several dozen T tokens. Many failed models were severely under-trained, rather than evidence that the large-model route itself had been disproved.
ChatGPT rapidly reversed that skepticism in late 2022. 张祥雨 admits that he also underestimated multimodal difficulty when starting his company, initially believing that mapping images into language space and then copying next token prediction would be enough to reproduce the GPT route.
6. Step-1 showed that native multimodality strengthens understanding, but did not unify generation
When training Step-1 in 2023, the team did not first build a pure language model and align it afterward. It organized the pretraining data as interleaved image and text from the start: both were tokenized, with text predicted after text and images predicted after images; a pretrained diffusion model was attached on the generation side.
Text performance was comparable to pure-text training, while image understanding was especially strong. Asking about text written directly inside an image performed almost identically to running OCR first and then asking about the extracted text, suggesting the model could freely access information from both modalities.
Generation alone was “particularly bad.” The language component had weak control over visual generation, and removing the generation branch halfway through training barely affected understanding performance. On the surface, the architecture supported both image-text generation and understanding; internally, it had not formed a true generation-understanding integration.
7. More than 6 months of alternating training produced 2 strong models with no mutual gain
Gemini and other overseas efforts mentioned during the discussion, along with GPT-4V, also failed to unify image understanding and image generation. Understanding gradually absorbed language, audio, vision, and video, while generation continued for years along separate DALL-E and Sora lines.
The team tried to build a closed loop: use the latest understanding model to label visual data, train a stronger generation model, then feed that generator back into the understanding side and use its gradients to improve understanding, repeating the cycle to narrow the gap.
After more than 6 months, both models became increasingly strong, but there was still no “1+1>2.” Removing either branch made the other neither stronger nor weaker; the supposed joint training was functionally still 2 models placed side by side.
More strikingly, the understanding model could accurately identify limb distortions, perspective errors, and physical violations in generated images, yet could not stop the generator from making the same mistakes. “It knew,” but that knowledge had not become control over the generation process.
8. Step-2 drew the back half of the curve where reasoning declines with scale
After completing Step-1 at the end of 2023, the team launched Step-2 in early 2024: a trillion total parameters and more than 200B active parameters, a “truly gigantic” model unlike many of today’s low-activation MoE models.
Scaling parameters and data together made compute requirements grow approximately quadratically. The project ran for more than 9 months before reaching a state the team considered satisfactory. Its initial estimate of the data required at this scale was still too low; the investment seemed like a “bottomless pit.”
The puzzling result was that Step-2 was very strong in humanities, writing, knowledge, and general dialogue, while mathematics and local logic were at one point weaker than 7B and difficult to repair through conventional LM training.
The team then rigorously retested 1B, 7B, 30B, and 70B models. EQ and knowledge kept rising with scale, while mathematical reasoning first improved, then plateaued, and finally declined. At that point last year, very few people had actually drawn the back half of this curve.
9. Large models are not incapable of reasoning; they over-trust their ability to skip steps
Looking at problems one by one, 张祥雨 found that large models were more inclined to omit local calculations. They might simply state the sum of 3 numbers, while small models, despite weaker capabilities, would still calculate step by step.
Internet corpora contain abundant omissions of intermediate steps. The team’s synthetic data could keep skipped steps rare, but not erase that distribution. Large models could fit humans’ habit of omitting steps; small models, constrained by capacity, could not learn it as easily.
The industry had long criticized next token prediction as behavior cloning that can hallucinate after an OOD fork. The new issue here is that even perfect fitting of the training distribution may make better compression actively induce unreliable computation.
10. A bimodal thought experiment explains why compression and mathematical accuracy diverge
Suppose that in a corpus involving the addition of more than 10 numbers, 50% of examples give the final answer directly and the other 50% show the steps. Under maximum likelihood, the optimal model should answer directly half the time and calculate step by step the other half.
A small model cannot fit the complex function that maps a string of numbers directly to their sum, so it learns the simpler step-by-step branch. This resembles feature collapse in generative models: with insufficient capacity or training, it first captures the easiest peak.
A large model like Step-2 really can directly report the sum of more than 10 two-digit numbers, with local accuracy sometimes exceeding 90%. From the perspective of distribution fitting and lossless compression, it is clearly better than a small model that drops one of the peaks.
But mathematics requires getting the answer right, not resembling the corpus. If every step in a long chain carries risk, each instance of “I can handle this” skipping accumulates into a high failure rate; a small model doing simple calculations step by step may ultimately be more reliable.
11. Rule-based RL targeted the right objective, but only O1 crossed the return plateau
The solution is straightforward: if compression and task accuracy have diverged, optimize accuracy directly. Rule-based RL does not care whether the path resembles a human one; it rewards only the final answer, penalizing a large model’s unreliable shortcuts.
Many teams were already trying this approach in early 2024. Large models typically benefited more than small models because their baseline was lower and they skipped more steps, but the overall gains remained limited, and many problems they originally got wrong stayed wrong.
O1 changed the picture dramatically. 张祥雨 believes calling it another “GPT moment” is not an exaggeration, but the breakthrough was not the data scale or RL algorithm upgrade that people first suspected.
12. “Patterns are all you need” gets closer to O1’s essence than PPO versus GRPO
O1-like training can use PPO or GRPO, as well as REINFORCE++, or even more primitive policy gradient. The algorithms differ operationally, but not in a way that explains the capability leap. What truly determines performance is the pattern used to organize the chain of thought.
RL in traditional games and control faces extremely sparse rewards. Random actions almost never complete a game; in chess or Go, directly challenging a mature agent yields a near-zero probability of positive reward, requiring MCTS, exploration bonuses, and reward propagation.
Language models have heavy pretraining. A mathematical answer may contain thousands or even tens of thousands of tokens, and each action’s apparent space may cover the whole vocabulary, but pretraining has already turned most subsequent expressions into near-automatic continuations.
This is why MCTS showed no decisive value in many O1-like experiments, and PRM is not universally necessary. 张祥雨’s explanation is that these problems may still be relatively simple; more importantly, pretraining has already performed most of the search-space pruning for RL.
13. In a solution spanning thousands of tokens, fewer than 10 steps may require real search
The team analyzed output sequences of roughly 4K or 5K tokens and found that the critical decisions that actually changed the method and affected the final answer were often no more than 10. Once a branch was entered, many tokens merely deterministically expanded its form.
This also explains why random rollouts work. Before training starts, a simple problem may already be answered correctly in more than half of samples; for a hard problem, 100 rolls can often yield several complete correct paths, something nearly inconceivable in traditional RL.
RL’s practical task is therefore not to search every token, but to make the model consistently choose the correct branch at a small number of critical tokens. The question then becomes: why did older rule-based RL, despite seeing the correct path, fail to push a fork with 60% accuracy to 100%?
14. The judgment required at some critical forks exceeds the compute budget of a single token
At an intermediate node, the model must decide whether to go left, go right, or use a different construction. But whether a complex mathematical construction works often becomes clear only much later, making it impossible to decide reliably from the preceding context in a single token.
张祥雨 uses large-number multiplication to illustrate the complexity boundary. Ordinary multiplication requires at least approximately (n^2), while better implementations still take roughly (n\log n); a single Transformer dot product is (O(n)). A calculation beyond the single-step budget cannot be completed reliably by reporting the answer “in one shot.”
2 problems that look almost identical but differ in only a few digits may require opposite branches at the same point. If one class of problems should go left 60% of the time and another should go right 40% of the time, assigning all probability to the left makes every problem in the latter class fail.
That is why reward rises to a plateau and stops. The model has seen the correct path, but lacks enough computational basis to make a certain judgment at the fork. Reinforcing the same token cannot conjure additional compute for that step.
15. Reflection upgrades single-path CoT into a backtracking meta-CoT
The solution is to allow both branches to be explored: follow one path to the end, discover that it is wrong, reflect and backtrack, then try the other. O1’s most impressive feature is that it genuinely adds this “reverse edge” to the computation.
Ordinary RL could theoretically trigger reflection on its own, but with low probability, high cost, and unstable results. The reason lies in pretraining: human answers usually preserve only the cleaned-up correct process, deleting failed attempts and detours, so the model comes to believe every problem should proceed smoothly through CoT to the end.
Cold starts and other pattern-injection methods re-expand the action space that had been over-compressed, after which RL reinforces it. 张祥雨 calls this “CoT’s CoT”: the first layer decomposes individual calculations, while the second selects, switches among, and even overturns different CoTs.
16. Visual generation remains at the language model’s original one-shot stage
The biggest limitation of autoregressive image generation is that it has “no take-backs”: once a patch is generated, it cannot be revised, so the model must decide that region’s structure, semantics, and relationship to the global image in a single step.
If asked to draw a blackboard and write out the complete solution to the chickens-and-rabbits problem on it, the surface task is image generation but the substance includes multistep mathematical reasoning. If the answer itself cannot be reported in one shot, reliable pixel-level generation is even less feasible.
Diffusion is a multistep ODE or stochastic denoising process, but 张祥雨 does not believe those denoising steps constitute semantic reasoning. A painter sketches first, fixes outlines and objects, then colors and renders; denoising has no clear object-level sequence, and its semantics may already be fixed very early.
Visual generation therefore lacks not only O1-style meta-CoT, but even ordinary CoT; it has been “left 2 generations behind” language models. The root of the generation-understanding gap is first the absence of appropriate computational complexity and process representations.
17. Artificially assigned visual actions can solve problems, but do not reproduce language reasoning’s generalization
At the end of last year, the team began genuine visual-space slow thinking. For counting, it marked points one by one on the image; for mazes, it drew lines and erased them after reaching dead ends; for tables, coordinates, and clocks, it moved visual attention rather than thinking only in text.
After adding these actions, the team used simple rule-based RL to reward correct answers. In theory, the system now had marking, search, and reflection, and it could indeed solve specially constructed counting and line-drawing tasks.
But after 6 months, the result was: “Whatever data I create, it can solve only that kind of data.” Compared with O1-like language models’ ability to transfer a way of thinking across domains, this visual system showed almost no strong generalization.
18. O1 generalizes a way of working, not just mathematical knowledge
张祥雨 once trained an O1-like model only on pure mathematics data, then asked it to write classical Chinese poems that strictly satisfied meter, rhyme, and meaning. Poetry was absent from the training set, yet the model naturally generated the same workflow it used for mathematics.
The model first drafted, then checked character counts, rhyme, and requirements one by one. If a local substitution damaged the meaning, it overturned the preceding and following lines; if that still failed, it rewrote the entire poem. Reflection, verification, trying another method, “large-loop” revision, and rereading the prompt all transferred.
Pure mathematics training could not transfer to an endgame in Dou Dizhu. The model miscounted cards and even discussed how to suppress the opponent after the opponent had already played out all cards. 张祥雨 believes games require a minimax pattern that mathematical training did not activate.
Once enough synthetic game data is added, the minimax approach also emerges easily. The model is not incapable forever; each broad problem class requires a training signal that activates its corresponding thinking actions from latent space.
19. Sparse reflection in natural data determines whether cold start can generalize
Highly upvoted answers on forums such as MathOverflow often preserve genuine struggle: an author writes “wait,” notices a missing factor, admits that the original method does not work, then rethinks or changes the formulation. These are the natural prototypes of the patterns used in O1-like cold starts.
张祥雨 instead criticizes many Chinese solutions for overusing “notice that,” stripping away all attempts and scaffolding and leaving only a magical construction. If a model learns heavily from this kind of corpus, it becomes more jumpy and more inclined to omit critical calculations.
Natural reflection data is sparse, but it comes from different authors, fields, and knowledge systems and is interwoven with the broader pretraining corpus. Amplifying these nodes during cold start can activate large amounts of connected capability, whereas fixed-program visual marking “can represent only itself.”
20. O3’s crop and resize look primitive, but win because pretraining contains them
When the team tested long visual reasoning chains with O3’s image inputs, it found that O3 mainly performed simple edits such as crop and resize. Its action ceiling was limited; without being allowed to write programs, it certainly could not solve maze-like tasks.
Yet on problems within its reach, O3’s success rate was far higher than that of the artificial-marking system. The reason is that interleaved image-text corpora contain abundant patterns of “show the full image, enlarge a local region, then explain it.”
张祥雨 cites electronics repair websites: a user uploads a radio photo, and an answerer enlarges different regions to identify a burned capacitor or faulty component. Crop is primitive, but tightly connected to natural data; more “scientific” synthetic actions lack that connection and are harder to generalize.
21. Before complexity is solved, multimodal pretraining can indeed damage text ability
李广密 asked whether multimodal data could drag down text performance. 张祥雨’s answer was that, ideally, deciding when to generate an image and how to combine distant image-text information should provide rich semantic supervision.
In reality, diffusion samples only 1 denoising step per training iteration. At most time points, the semantics have already been determined and the loss concerns only local detail. Backpropagating that gradient into the language component contributes nothing to semantics and may even damage it.
Autoregressive generation is somewhat better because training and inference are more consistent: each step at least genuinely produces a patch. But “no take-backs” still leaves the supervision without a clear reasoning process, so the improvement is not fundamental.
The analogy is to pretraining on hard problems that provide only answers and no process. A strong model may memorize “choose C,” while a weak model learns an incorrect bias such as “the longest option is C.” If image-generation signals are equally uninterpretable, the best outcome is uselessness; the worse outcome is corrupted gradients.
22. Video can supplement visual CoT, but cleaning and language alignment are hard bottlenecks
The first unlocking route is to expand the pretraining corpus. Instructional videos naturally contain step-by-step processes: a teacher points with a laser, sketches, and draws auxiliary lines, with actions synchronized to language and therefore closer to visual CoT than static image-text data.
The problem is that, from a training perspective, most videos are “garbage”: they are very long, contain sparse useful segments, and have low information density. The useful actions still need to be accurately segmented and cleaned.
Image-text webpages usually contain rich textual explanations, while videos may have only ASR, subtitles, brief narration, or no sound at all, making language-vision alignment harder. 张祥雨 believes many organizations are pursuing this route and specifically mentions Google as particularly strong at using video data.
23. Highly controllable generation need not wait for complete visual CoT; lower the difficulty first
The second route is to expand the visual action space so the model can not only edit an original image but also freely generate sketches and intermediate representations. Text descriptions or imagined scenes can first be made concrete and then used for further reasoning.
This once created a deadlock: generation needs visual reasoning to improve controllability, while visual reasoning needs generation to provide actions. The latest Gemini and Sora results make 张祥雨 believe that one side may already be able to move first.
The key is not to solve all generation immediately, but to strictly constrain the domain and difficulty and train tasks that can be completed in one shot: connect A and B in a geometric diagram, remove a person, adjust an object’s position, or generate a structure based on what appears “on the left, in the middle, and above.”
These tasks rely mainly on perception, localization, and simple generation rather than complex reasoning. In the past, news illustrations, memes, and tasks of every difficulty were mixed together, leaving the model struggling for years. With better data cleaning and difficulty stratification, current architectures may already achieve high instruction fidelity.
24. Interleaved image-text CoT will turn understanding and generation into one task
StepFun has already trained an editing model. The data remains dirty and the system still has a “building-block” quality, but structural control and instruction fidelity are far better than in early generation-understanding integration attempts.
The next step is to use that model to synthesize large amounts of highly relevant data and then train a unified model. The team is not prioritizing facial aesthetics or fine-grained visual polish; for reasoning, whether the semantic action is completed matters more than whether the image is beautiful.
The starting point only needs basic visual understanding without CoT and the ability to execute simple instructions. Once it enters interleaved image-text CoT, the intermediate image can be checked, modified, or redrawn; the final output in text is understanding, and the final output in images is generation.
张祥雨 calls this the next “multimodal GPT-4 moment” and puts the timeline at “no more than 1 year.” Current architectures are broadly sufficient; the main variables are data cleaning and training stability. Video-related compute will not be small, but he explicitly remains uncertain about how many unknown problems remain.
25. Long context matters for applications, but may not be the right direction for intelligence
张祥雨 first distinguishes applications from intelligence. Enterprise applications need to fit domain-specific material into the model and use reference materials to reduce hallucinations, so long context is genuinely important as an information container.
Human memory, however, is highly hierarchical. Working memory is generally thought to last only about 2 to 4 seconds, enough to retain a phone number without loss; medium-term memory extracts key points, reinforces them repeatedly, and gradually forgets; long-term memory is closer to being fixed into parameters for life.
2 to 4 seconds of visual input alone amount to roughly 30K or 40K tokens by his estimate. Today’s models instead treat 100K, 1M, or even 10M tokens as the same kind of short-term memory. They solve “fitting it all in,” but not how to select, compress, and use it.
26. A Transformer that never forgets is continuously interfered with by its own history
After a meeting or film ends, people remember the key events rather than “how many cups were on the table at minute 37.” A standard Transformer’s memory grows in proportion to its input, with no capacity pressure, loss, or mechanism forcing it to extract the important parts.
In experiments, clearing the context after solving each math problem preserved a stable performance level. Solving 100 or 200 problems continuously made performance deteriorate toward the end. Attention kept being pulled toward similar earlier problems, forcing the model to spend substantial capacity resisting interference.
Retrieval benchmarks such as needle-in-a-haystack further reward never forgetting anything, creating a bias toward lossless storage. 张祥雨 believes this is not intelligent progress and may even obstruct intelligence, because it turns compression, abstraction, and forgetting into errors.
He does not regard the quadratic complexity of attention as the central issue. The deeper problem is that the modeling approach is wrong. “If information is not processed and compressed,” it cannot produce a more refined representation and is unlikely to generate a qualitative capability shift.
27. Next-generation memory will look more like turning pages than stuffing a whole book into one brain region
RNNs, linear Transformers, and other infinite-sequence structures are worth exploring, but 张祥雨 believes the key is still not a different attention formula. The focus should be dividing labor between global information and precise local computation instead of putting everything into one context.
He envisions a short-context ordinary Transformer handling precise local computation while another infinite-sequence module forms global awareness. After reading a book, if asked to extract a memorable line, the global memory would point to “somewhere around 20%,” and the local model would revisit the passage like turning pages.
李广密 summarizes this as a prototype of multi-agent collaboration. 张祥雨 qualifies that it is more like cooperation among different regions of one brain than multiple independent individuals. Each module might begin as an LM or VLM and then differentiate during training into memory, planning, and action functions.
28. Even 6 days and 6 nights of reasoning may not require millions of tokens resident in one context
OpenAI has described O3 as reasoning continuously for 6 days and 6 nights and generating more than 10M tokens. 张祥雨 does not know how it is actually implemented, but sees no need for a single architecture to carry the entire history. That would be slow, expensive, and poor at isolating contexts.
One model could handle planning and retain only explored branches and high-level conclusions; another could receive summaries, perform concrete calculations, and report back either an answer or the reason for failure. When switching to a new path, the executor could completely clear the previous one.
If the search space unfolds like a binary tree, flattening every path produces a CoT with tens of millions of tokens. The planner needs to retain only the tree’s high-level state, while the executor needs only the current path, reducing resident information to approximately log scale.
This need not be a hand-built “Agent” assembly. By constraining the 2 models’ contexts, allowing them to communicate and clear state, and using final accuracy as rule-based RL, the collaboration protocol could be learned end to end. Architecture remains a tool for algorithms; if a no-backpropagation approach such as the one proposed by Hinton ultimately works, an ordinary RNN might even support online learning directly.
29. The next wall for current RL is environment scaling and impoverished feedback
张祥雨 plots large-model evolution along 2 axes: optimization moving from next token prediction toward RL, and modalities expanding from language into vision, audio, and video. The 2 axes rise in a spiral, but current rule-based RL is beginning to expose a new boundary.
Programming RL requires building a Docker environment, inputs and outputs, and tests for every project before producing verifiable data. After model and data scaling, the bottleneck becomes “environment scaling.” Large companies can hire engineers to build environments one by one, but this is far less efficient than humans reading documentation, setting up environments, and learning from feedback themselves.
Natural-language feedback is also heavily wasted. When a teacher evaluates an essay, they separately discuss the opening, rhetoric, transitions, typos, overall style, and length. Existing RL often compresses these dimensions into a single “3-point” score, leaving the model unable to tell how the score was composed or which paragraph to change.
Next-generation learning must extract direction from unstructured feedback and form intrinsic rewards rather than relying forever on external KPIs or reward models. It must also model real environments that are indefinitely long, dynamic, and non-stationary, allowing the model to keep updating its own weights.
30. A true Agent is autonomous evolution, not a longer workflow chain
张祥雨 expects autonomous and online learning to arrive after multimodal reasoning, but says it “might not even take 2 years,” because the field has become a major academic focus and investment is rising rapidly.
Once implemented, an internal company Agent could continuously absorb proprietary knowledge and know-how, becoming a highly customized “employee.” The new problem is how to copy, transfer, or merge the memories of 2 individuals, much as human experience cannot be directly copied into another person.
He believes OpenAI’s 5-level intelligence framework has a clear algorithmic logic: Chatbot corresponds to next token prediction, Reason to strong CoT plus RL, and Agent should correspond to autonomous, online learning that can work independently and continue to improve. He calls this the A+ era.
Systems that proactively make function calls today, or use RL to decide whether to call a tool, still belong to the Reason era. Earlier LangChain systems were mostly handcrafted pipelines and still belonged to the Chatbot era. A true Agent must “find its own goals, explore on its own, and learn its own value.”
31. World models, robotics, and autonomous driving will ultimately converge on the same AGI track
On the world models discussed by 李飞飞 and LeCun, 张祥雨 believes the goal converges with autonomous learning, but the methodologies differ. Generation is not necessarily an external organ an agent must possess: humans have no visual generator, yet can imagine the future in their minds.
At the current stage of technology, however, generative training remains the most convenient supervision for learning a world model. More complex spatial reasoning may ultimately require a world model; for now, generative approaches may be easier.
Autonomous driving and robotics are “getting a head start.” They use constrained scenarios, lower-dimensional control, rule-based methods, and modular decomposition to deliver applications first. Autonomous driving has moved toward end-to-end systems, while robots can now fold clothes and run, but neither comes close to general visual intelligence.
As these systems further integrate end-to-end control, multimodal reasoning, and autonomous learning, their paths will merge with AGI’s end state. The 2 moments 张祥雨 is willing to bet on remain multimodal reasoning in roughly 1 year and autonomous, online learning that may emerge within 2 years. Beyond that, he declines to make a definite prediction.