Scaling "Thinking": Gemini 2.5 Tech Lead Jack Rae on Reasoning, Long Context, & the Path to AGI
Summary
Gemini 2.5 Pro’s clearest differentiation is not a nominal million-token window but reliable command of what fills it. Nathan Labenz’s decisive test was a messy 400,000-token research codebase: earlier Gemini models could accept that scale, but 2.5 Pro appeared able to identify what mattered without the user already knowing where to look. Jack Rae describes the intended experience as having “read and studied all that information,” turning long context from a specification into a practical workflow unlock.
The reasoning-model surge looks less like a sudden algorithmic breakthrough than accumulated reinforcement-learning work crossing a visible capability threshold. Within Gemini, Google had leaned increasingly on discrete, verifiable rewards for at least a year, including before its explicit “thinking” effort, while internal metrics improved with an “almost like scarily predictable” regularity. Individual recipe changes remain stochastic, but the aggregate curve is smooth; discrete releases make the downstream gains look emergent.
Frontier labs converged on inference-time scaling because the direction was obvious, the vein proved rich, and talent plus compute let everyone move unusually fast. Gemini organized dedicated thinking groups around September or October, found what Rae calls modeling breakthroughs within roughly a month, and shipped an experimental Flash-based thinking model in December. The implication is continued competitive intensity around a scalable paradigm with a substantial “low barrier to entry” for organizations already comfortable with language-model RL.
Rae’s Gemini team prefers the simplest training recipe that generalizes, but it is pragmatic about mixing human data, synthetic distillation, and end-to-end RL. He wants to impose as few priors as possible on the model’s cognitive behavior, letting useful self-correction and exploration emerge rather than scripting a humanlike procedure. Crucially, he argues post-training must eventually create new skills, not merely rearrange knowledge acquired during pre-training, if these systems are to surpass humans on important tasks.
Raw chain of thought is simultaneously a capability interface, an interpretability opportunity, and a safety liability that Google has not resolved. Gemini 2.5 Pro exposes raw thought tokens in AI Studio and the Gemini app, but Google is exploring summaries and graph-like interfaces instead. Labenz presses the obfuscated-reward-hacking risk; Rae’s line is narrower but important: thoughts should remain faithful to the underlying computation, and developers “probably don’t want to add training objectives” that encourage deceit.
Rae’s AGI road map is an integration thesis: better reasoning, agents, memory, tools, and reliability compound rather than substitute for one another. Contexts of 1 million or even 10 million tokens begin to approach “lifelong scales,” yet he does not consider memory solved and expects richer read-write systems may prevail. Current models still need to become far more general and trustworthy on open-ended tasks, but he says the constituent technologies are actively advancing and AGI is “coming quite fast.”
Deep multimodal co-training could extend Gemini’s integrated-model approach, but only where shared representations deliver positive transfer. Rae credits Gemini’s multimodal-first design for native image understanding, generation, and editing that feel richer than static tool calls; text came first because it is highly compressed, knowledge-dense, and abundant. Whether genomics or other domains belong inside one world model depends on data scale and transfer—otherwise a specialized tool remains the better architecture.
Gemini 2.5 Pro’s experimental label accelerates market feedback but leaves disclosure behind deployment. Rae says experimental models receive extensive policy review, safety testing, red teaming, and external red-team support, while the full system-card treatment arrives at general availability. Google would not name current external partners; Rae says past technical reports have acknowledged such partners, though he was unsure whether the eventual report would do so.
Deep dive
1. Gemini 2.5 Pro turns long context into working knowledge
Labenz calls Gemini 2.5 Pro Experimental 03-25 the first Google model he would rank number one across several important dimensions. The decisive moment was not a leaderboard: it was asking the model to extend a shared research codebase while reusing existing work and making only a “really light touch.”
That codebase occupied 400,000 tokens and was organized like research code—individual contributors’ folders rather than polished production architecture. The model’s command of it felt “qualitatively different”: it could navigate information Labenz himself had not fully mastered, rather than merely retrieving details he already knew to request.
Rae says this distinction reflects substantial work beyond extending the nominal window to 1 million or 2 million tokens. With 2.5 Pro, long context increasingly feels as though the model has “read and studied all that information,” retaining key details instead of silently dropping them.
Rae spreads credit across pre-training, thinking, post-training, and the broader Gemini organization. Internally, the team liked the model but did not know how users would receive it; the strongest validation has been people reporting “noticeable deltas” on real tasks and, in Rae’s phrase, “feeling the AGI.”
2. Reasoning improved smoothly before the public saw a step change
Rae rejects the premise that correctness-based RL suddenly began working. Within Gemini, Google had increased its use of reinforcement learning on accuracy tasks for at least a year, using “a very discrete verifiable reward signal” to improve reasoning even before the dedicated thinking program began.
His model of progress separates leading indicators from product thresholds. Internally, capabilities can rise with “almost like scarily predictable improvement,” even though any individual change to the RL or model recipe may fail; enough improvements accumulate until a discrete release crosses the point where users feel a step change.
Labenz’s summary—public emergence may be a mirage over a smooth underlying curve—largely lands. Rae compares the trend to “a Moore’s law style improvement,” with excitement arriving discontinuously because practical tasks only become useful after capability passes particular thresholds.
The conclusion is not that the recipe is easy. Training a 100-billion-plus-parameter model once resembled alchemy, and modern language-model RL has its own collection of failure points: with five things broken, fixing one produces no visible gain, encouraging teams to abandon an approach that may only need several more iterations.
3. Early self-correction showed that the difficult RL stack could work
Rae recalls “initial sparks” in which reinforcement learning caused the model to use its thoughts for self-correction and exploration. Those were precisely the cognitive behaviors researchers hoped would emerge, but they did not know the result was achievable until it appeared in their own experiments.
That experience contextualizes reports that DeepSeek-R1 struggled to apply RL to smaller models. Rae considers such failures entirely plausible, but does not identify model scale as the explanation: he emphasizes numerous implementation details whose interactions can conceal progress until the full system is aligned.
The team’s governing preference is an Occam’s-razor-like one: if fewer imposed priors still yield a powerful, general model, prefer that recipe. In practice, however, Gemini remains “outcome driven,” testing human data, model-generated synthetic distillation, and end-to-end reinforcement learning, then using the mixture that produces the strongest final result.
4. Frontier labs found the same vein because it was visibly rich
Rae dismisses the need for a San Francisco-party explanation for simultaneous invention. Once even limited evidence shows a new paradigm improving models, an unprecedented concentration of researchers and compute can react immediately: “people will flood into it very fast.”
Inside Gemini, dedicated reasoning groups assembled around September or October to focus on thinking and test-time compute. Within roughly a month, the team found results it regarded as modeling breakthroughs; by December, Google had shipped its first experimental thinking model based on Flash.
Google’s existing comfort with applying RL to language models lowered the cost of exploring the direction. Once experiments revealed substantial capability, scaling the program was a natural extension—not evidence of a single secret trick shared across laboratories.
Rae’s own research habits reflect the same shift from idea scarcity to execution bottlenecks. He reads fewer papers than five or ten years ago because many known problems have no published solution; the answer will emerge among colleagues running experiments. For literature triage, he uses Gemini to ingest papers or cited collections and cut directly to “the critical idea and the critical results.”
5. Useful thoughts need not resemble polished human reasoning
Rae’s primary criterion for thinking tokens is whether they improve “accuracy, safety, factuality” and overall capability. The thought stream is a scratch space, so cyclic or apparently unhelpful text may be acceptable if it enables a materially stronger final answer.
Gemini commonly reasons in English even when solving non-English tasks, yet Rae says it remains strong on those tasks. The product question is unresolved: localized thoughts might look more natural, but forcing them could remove an internal behavior that helps the model reason effectively.
Early Flash thinking models also adopted conspicuously templated routines for decomposing problems and constructing responses. The team viewed that as potentially inefficient: if a model always follows one formula, it may not be using its inference-time compute as flexibly as a more natural, task-specific process would.
This makes human likeness a secondary objective, not the training target. Gemini researchers want thought tokens to be efficient and maximally capability-enhancing, while remaining open to changes when a repeated behavior appears to waste compute or undermine the product experience.
6. Google exposes raw thoughts while debating how much to shape them
Rae confirms that Gemini 2.5 Pro shows raw, unmodified chain-of-thought tokens in both AI Studio and the Gemini app. Google has not settled on that interface: raw thoughts can be verbose, summaries may be more useful, and NotebookLM’s graph-based thought explorer suggests a third path for following branching ideas.
Decisions about disclosure combine research, safety, and leadership input, rather than belonging to Rae’s thinking team alone. He will not reduce OpenAI’s choice to show summaries to distillation risk; interpretability, user experience, and the danger of optimizing thoughts to look pleasing could all matter. Rae says he does not personally weigh in on the release decision.
Labenz’s pushback cites OpenAI’s obfuscated reward-hacking result: suppressing explicit reward-hacking reasoning initially reduced the behavior, but later the hacking returned without appearing in the visible chain of thought. That suggests optimizing the transcript itself can destroy its value as evidence without fixing the underlying objective.
Rae does not endorse an absolute taboo, but he accepts the core hazard. If thoughts are displayed, they should be interpretable and faithful to the computation producing the answer; objectives that encourage deception would be counterproductive. Experimental releases are partly intended to reveal how real users consume these traces before Google fixes a long-term policy.
7. Human process data matters more than forced inner monologues
Labenz describes asking domain experts to “staple your pants to the chair” and record not only inputs and outputs but every incremental judgment behind their work. In his own experiments, including example reasoning in fine-tuning data often improved performance on one or a very small number of tasks, yet eliciting faithful explanations from people proved remarkably difficult.
Rae separates observable process from literal chain of thought. Inner reasoning is “a pretty latent thing,” and people asked to transcribe it artificially often produce less valuable data than expected; by contrast, essays, technical discussions, and worked solutions already contain organic examples of humans decomposing problems.
Those naturally occurring traces help explain why pre-trained models could respond to “let’s think about this step by step” before dedicated reasoning training. The model had already learned what reasoned exposition looks like, even if that text was not a faithful recording of anyone’s private cognition.
Process data still becomes valuable for open-ended tasks involving intermediate calculations, possible actions, or interaction with an environment. Rae sees reasoning and agency as tightly coupled, while preserving a research division of labor: agent teams specialize in environments and actions; thinking teams focus on the reasoning behind those actions.
8. Post-training must create skills, not merely select behaviors
Rae characterizes pre-training as learning a “massive bag of function approximators” spanning good reasoning, bad reasoning, correct answers, and mistakes. Better next-token prediction compresses and models the full distribution; post-training then discards many modes and concentrates behavior around a smaller set of useful responses and actions.
Reinforcement learning should do more than select from that bag. Ideally it composes primitive abilities into stronger skills, and Rae considers learning genuinely new capabilities during RL “absolutely crucial” if models are eventually to surpass humans on critical tasks; behavioral cloning of pre-training knowledge cannot be enough.
On the Platonic model hypothesis, Rae grants a mathematically grounded convergence story for pre-training: a Bayes-optimal text compressor approaching the entropy floor would best model the world process that generated the text. But understanding that distribution is not equivalent to following instructions, acting usefully, or generalizing to tasks never performed before.
At RL scale, he sees no convergence toward one inevitable system. Existing models already have distinct strengths and weaknesses, while Gemini researchers deliberately cultivate particular capabilities. He allows that models may converge toward a better representation of the process generating their training data, but does not see one inevitable world model governing behavior; systems remain “very steerable,” creating substantial responsibility for researchers.
9. AGI requires memory, agents, interpretability, and more modalities to compound
Rae’s early DeepMind work focused on episodic memory, and his PhD studied lifelong reasoning with sparse, compressive memories. Contexts of 1 million or even 10 million tokens now begin to “verge on lifelong scales,” but he does not call memory solved; richer read-write mechanisms resembling the Neural Turing Machine or Differentiable Neural Computer may ultimately provide extremely long or lifelong memory.
Long context has already compounded with inference-time reasoning: Rae says the ability to think deeply while using millions of tokens has unblocked additional problems through their coupling. The remaining road map includes more reliable and general reasoning, stronger agents, open-ended tool and action spaces, and memory that preserves understanding across long spans of time.
Labenz worries that latent-space reasoning could make powerful systems opaque. Rae argues against tabooing it before investigation, citing MuZero’s search over latent vectors that could still be decoded into states; Labenz counters that general AI lacks the bounded, confidently decodable ground truth of a game board.
Rae expects capability advances to accelerate mechanistic interpretability as well, rather than capability growing exponentially while safety stays linear. Yet both agree the problem remains open: whether reasoning uses tokens or latent vectors, researchers need tools that establish how faithfully visible thoughts track computation and outcomes.
Multimodality extends the same integration logic. Gemini’s joint treatment of text, image, video, and audio enables native image editing and interaction beyond a static text-to-image tool call; other domains should join the shared world model when co-training produces positive transfer, while weakly related expert systems may remain tools.
Rae’s bottom line is deliberately mixed: today’s systems already “feel like AGI” when they one-shot complex codebases, yet they remain unreliable on open-ended work. Better memory, reasoning, and agency would make that judgment much more broadly compelling, and because every component is under active development, “it’s coming quite fast.”