Summary

Erik Torenberg’s investment thesis is that persistent, evolving memory—not raw world knowledge—may be the last major unlock separating today’s copilots from “drop-in knowledge workers.” Enterprise context remains scattered across Slack, email, documents, GitHub, meetings and task systems, making Tyler Cowen’s “Context is that which is scarce” literal for AI. Torenberg estimates company-specific models might cost millions or tens of millions for century-old enterprises, versus tens to low hundreds of thousands for smaller businesses—potentially cheap once amortized across many AI workers.
Titans changes the memory substrate from a vector or matrix of numbers into a neural network that learns during inference. Unlike RAG, which stores searchable records, or Mamba-style recurrent systems, which update numerical state, Titans uses an MLP whose weights are updated through gradient descent at runtime. Ali Behrouz sees the contribution less as a finished model than a new design axis: “No architecture is end game.”
The memory MLP learns an associative map from attention keys to their corresponding values, allowing future queries to retrieve an approximation of historical payloads without retaining every token. Attention provides the nonparametric solution by explicitly comparing a query with all stored keys; Titans compresses those relationships into fixed-size parameters. This sacrifices exact recall for efficiency and a more human-like, fading long-term memory.
Titans makes memory management highly input-dependent: prediction error supplies “surprise,” momentum extends an important update across the surrounding episode, and decay makes room for new information. Behrouz’s intuition is that “everything that is surprising probably is worth memorizing,” yet the explanatory tokens after a surprise may matter even when they are unsurprising themselves. Learned, token-dependent controls determine memory decay, current updating and whether previous surprise should carry forward.
Behrouz rejects the premise that recurrent memory should replace attention; Titans is explicitly a hybrid of accurate short-term attention and compressed long-term memory. The team tested Memory as Context, Memory as Gate and Memory as Layer, with the more principled context and gating designs generally beating simple layer interleaving. In Nathan Labenz’s rough count, the conventional layer approach won only about two of roughly 30 scale-and-task comparisons.
The headline benchmark is long-context performance, but Behrouz repeatedly cautions that the reported tests are synthetic rather than proof of equivalent gains on general workloads. Small Titans models reportedly scaled to 2 million tokens and even 10 million tokens at roughly 70% accuracy, while GPT-4’s benchmark performance dropped quickly. The implementation was already faster than Mamba in the reported comparison, though some modern linear models remained faster and Titans had not received custom-kernel optimization.
Titans opens a credible route toward long-running agents, but it does not solve lifelong learning or enterprise knowledge acquisition by itself. Retrofitting an existing Llama- or R1-style model appears possible but was not demonstrated, and repeatedly updating a finite memory risks catastrophic forgetting. Behrouz’s next test is broader: whether neural memory works across decision-making, reinforcement learning and other modalities, as transformers did—not merely on language modeling.

Deep dive

1. Memory is the missing middle between context windows and economic value

Torenberg’s framing begins with two capabilities that LLMs already possess: immense but crystallized world knowledge in model weights, and attention-based working memory inside a finite context window. What remains missing is sustained coherence—an agent that can “know who it is,” “know what it’s done” and “know where it’s trying to go” across a long-running episode.
The operational constraint is not merely context-window size. Relevant organizational knowledge is fragmented across Slack, email, documents, GitHub, meeting transcripts and task systems, making assembly and maintenance tedious. Torenberg argues models only began properly rewarding that work “truly with Gemini 2.5 Pro”; ChatGPT’s newer memory features offer a glimpse, but he knows of no productized equivalent that works well at enterprise scale.
His GE-or-3M thought experiment: train a model on a century-old company’s products, personnel, history, finances, processes and debates until it knows the institution better than any single person. It could still search official records for grounding, but would recognize when it had found the right material and pick up the tacit layer—“how we do things around here”—that makes human onboarding costly.
Torenberg estimates startup costs could reach millions or tens of millions of dollars, roughly where OpenAI had publicly priced custom models, yet remain a bargain when amortized across as many copies as needed. Smaller companies might spend tens of thousands to low hundreds of thousands. His conditional call is stark: a long-term-memory breakthrough might be the main remaining barrier to dramatic labor-market disruption.

2. Behrouz reached sequence architecture through neuroscience

Behrouz describes his route as “very unusual”: he began with graph algorithms, then used graph neural networks for anomaly detection. Connections between graph anomalies, disease detection and brain disorders pulled him toward neuroscience and, eventually, architectures inspired by how human memory learns.
His foundational bet is that humans learn effectively from few samples because their machinery reflects “millions of years of evolution.” It is hard to believe researchers will leap directly past that process with one superior architecture; a more plausible path is to imitate useful brain mechanisms first, then improve beyond them toward whatever “superintelligence” becomes.
Starting his PhD alongside S5, RetNet and other alternative sequence models made Behrouz interested in moving beyond transformer monoculture. But he parts company with researchers seeking to eliminate attention: accurate modeling of dependencies among current-context tokens remains necessary, so he expects hybrids—not pure recurrent replacements—to win.

3. Titans replaces a memory state with a memory network

In Behrouz’s high-level account, a conventional RNN projects incoming data and a vector-valued hidden state, combines them, and writes the result back. Reading is another projection from that state. Causal linear attention admits a recurrent formulation too, but its hidden memory is matrix-valued rather than the vector used by traditional RNNs and models such as Mamba.
The conceptual break is asking what physical shape memory should have. Behrouz does not believe a few isolated numbers resemble biological memory: “Is it a vector? Is it a matrix or is it a neural network?” His answer is the interconnected network, making Titans’ memory itself an MLP rather than a passive container.
That shift creates a second scaling axis. Sequence models progressed from vectors to linear layers or matrices; Titans permits two-, three- or four-layer MLPs, with future possibilities including deeper networks or convolutional structures. The initial MLP is deliberately simple, not a claim that the architecture search is finished.
Titans also embraces continual test-time learning. Behrouz argues memory cannot be cleanly divided into pre-training and post-training because human memory is “constantly learning.” The TTT perspective already interprets recurrent models as performing a form of test-time training; Titans makes the trainable object richer and the update rule more expressive.

4. Short-term accuracy and long-term compression demand a hybrid

Behrouz’s human-memory analogy distinguishes attention’s high-fidelity short term from an RNN’s fading long term. People rarely invent details about information received 20 or 30 seconds ago, but may reconstruct or hallucinate an event from 20 years earlier. Recurrent decay is therefore not simply a defect—it resembles the unavoidable selectivity of finite memory.
Nathan sharpens the engineering constraint: brains do not grow with every life event, whereas full attention’s all-token interactions become progressively more expensive and, in his formulation, produce a memory footprint growing with the square of sequence length. A bounded system must decide what deserves retention and what can safely disappear.
Attention remains valuable precisely because it preserves recent tokens explicitly and models their pairwise dependencies accurately. Long-term neural memory accepts fuzzier retrieval in return for fixed size, recurrent updating and a horizon beyond the sliding attention window. Behrouz’s proposed destination is the combination, not victory by either mechanism.
The unresolved problem is transfer between the two stores. Behrouz notes that neuroscience itself contains disagreement about how short-term memories become long-term ones, leaving no settled biological mechanism to copy. That uncertainty motivates Titans’ multiple integration designs and makes their relative performance a substantive architectural result.

5. Mathematical equivalence does not erase architectural direction

Nathan asks whether Titans is fundamentally new or merely another reformulation within a family encompassing TTT, DeltaNet, Gated DeltaNet and Longhorn. Behrouz concedes the family connections: a common test-time-training formulation can unify many models, and seemingly small mathematical changes often separate them.
His measure of contribution is therefore prospective: not whether a model is “the end of the world,” but whether it changes what researchers try next. Transformers themselves kept improving through FlashAttention and additional components. Titans’ purpose is to “deliver a new perspective” that produces future studies, rather than declare an endpoint.
Some Titans configurations collapse toward existing systems. Choose a matrix-valued memory, remove some components such as internal layer normalization, and the result can resemble DeltaNet or Gated DeltaNet; comparable links exist to TTT and Longhorn. Yet the perspectives still generate different experiments even when portions of their equations coincide.
Behrouz separates two research paths. One improves recurrent memory management—deciding what to retain through better update and decay rules. The other improves the memory architecture itself, moving from vector to matrix to MLP and beyond. Future systems can combine both, but they answer different questions and should not be treated as one optimization problem.

6. Persistent memory completes the three-store architecture

In the full design, a sliding window supplies short-term context, the neural module returns relevant long-term information, and persistent learnable parameters encode task-level knowledge that is independent of the current data. These pieces enter the attention process alongside one another, after which the larger network proceeds in a broadly transformer-like fashion.
Behrouz admits persistent memory “might not be very necessary.” It completes the human-inspired taxonomy and produced a “very, very slight improvement,” with no observed harm. Implementers seeking a leaner Titans variant could reasonably omit it to save parameters and simplify the design.
There is also a technical rationale: causal attention can over-focus on initial tokens, particularly when retrieved long-term material is concatenated before the current context. Placing learnable, data-independent tokens first can help mitigate that effect and improve performance.
The idea predates the attention-sinks paper Behrouz discusses. An earlier derivation interprets a two-layer MLP as attention-like multiplication, replacing its GELU-style nonlinearity with softmax; concatenated learned parameters can then perform a comparable role inside attention. For Torenberg, this is the episode’s recurring lesson: equivalent mathematics can carry very different and productive conceptual meanings.

7. Associative memory turns key-value attention into a learned map

Writing to ordinary recurrent memory means projecting new data and combining it with the hidden state; reading means applying a query to that state. Once memory is a neural network, the read becomes a forward pass: supply a query, and the network outputs the information it associates with that input.
Behrouz explains attention itself as associative memory. Stored keys are linked with values; a new query need not exactly equal any key, so its dot products with the keys measure similarity. Those scores express the query as a mixture of known keys, and the same mixture over associated values returns the relevant payload.
Titans distills that operation. During runtime, it trains the MLP so that feeding a token’s key produces that token’s value, minimizing the discrepancy between predicted and target value. A later token’s query, being similar to relevant historical keys, should elicit an approximation of the values full attention would have recovered explicitly.
Torenberg repeatedly tests this interpretation, and Behrouz confirms it: the model learns “what each token has” in anticipation of a future query asking for it. Attention is the nonparametric solution that retains and compares all pairs; the recurrent neural memory uses optimization to compress those associations, trading some accuracy for bounded storage and computational efficiency.

8. Surprise and momentum decide what survives

A fixed-size MLP cannot absorb information indefinitely, so Titans needs a retention policy. Behrouz’s intuitive rule is that “everything that is surprising probably is worth memorizing.” Technically, poor key-to-value prediction produces an associative-loss gradient, providing the momentary surprise signal that drives a larger parameter update.
Surprise alone would isolate one token when the useful unit may be an episode. Behrouz’s example, “I’m leaving,” is mundane at the end of a work shift but consequential if it means leaving the company. The surrounding words reveal which interpretation applies, while subsequent explanatory tokens may be predictable—and thus locally unsurprising—despite being essential to the event.
Momentum carries the update state forward, preserving heightened sensitivity across those surrounding tokens. Torenberg compares it with remembering not only the shock of 9/11 but the classroom, subsequent conversation, girlfriend and father’s return home. The surprising event creates a period whose details become disproportionately salient.
Both memory and surprise decay. Old parameters must yield to new information, while an old surprise should stop amplifying updates as context changes. Behrouz confirms that the decay, momentary-update and momentum controls are learned and input-dependent: each token influences how much state survives, whether its own gradient matters and whether previous surprise should continue.

9. Runtime gradient descent can become GPU-friendly multiplication

A literal gradient update after every token would create a sequential recurrence and train slowly. Titans is practical because the gradient calculation and parameter updates can be rewritten in a dual, matrix-multiplication form, extending the technique described in TTT and exposing parallel work to GPUs or CPUs.
Weight decay adds elementwise multiplication by a scalar or channel-wise vector, but can still be represented with Hadamard products or diagonal matrices. Momentum looks harder, yet its surprise-state equation is itself a linear recurrence, so it can be computed chunkwise with a parallel-scan algorithm similar to those used in Mamba.
Another option is to expand the momentum recurrence and express both gradient terms and decay coefficients as matrix operations. The larger point is that “learning at inference” does not require an irreducibly token-by-token optimizer loop; the same update semantics can admit hardware-efficient batching.
In the paper’s timing experiments, attention’s training-time efficiency degraded as context length increased, Titans ran faster than Mamba, and some newer linear models remained faster than Titans. Behrouz stresses that the team focused on architecture rather than low-level kernels; he expects dedicated optimization could make efficiency comparable with very fast, simple linear RNNs.

10. Context and gating beat naïve layer interleaving

Memory as Context creates a feedback loop. Long-term memory first retrieves information relevant to the current input and places it in attention’s context; attention then decides how to combine retrieved history with recent tokens. Its output returns to the long-term module, helping determine what should be written for future use.
Memory as Gate treats short- and long-term memory as parallel branches. The same data enters both; attention models exact pairwise relations in the local window while the neural module supplies historically relevant information. Their outputs are then concatenated or multiplied, allowing the network to combine the two sources.
Memory as Layer alternates the modules sequentially, much like established transformer–RNN hybrids. One short-term layer filters information for the following long-term layer, whose output then informs the next attention layer. Behrouz included it partly because this configuration is common in the literature, not because neuroscience or first principles singled it out.
The comparison challenges the default hybrid recipe. Nathan’s rough reading across nine categories and three scales found Memory as Layer winning perhaps two of roughly 30 cells, while Context and Gate divided most of the rest. His takeaway—accepted by Behrouz—is that principled interaction between memory systems generally beat simply shuffling attention and recurrent layers together.

11. Nonlinear memory expands expressivity, but theory needs caveats

Considered alone, Titans’ neural memory remains an RNN-like mechanism, but Behrouz calls it more expressive because both its memory architecture and recurrent update are richer. The complete Context or Gate architecture can theoretically ignore one branch when a task favors the other, offering the “best of both worlds,” though he concedes a hybrid will not beat every pure RNN or transformer on every task.
His concrete micro-skill is state tracking: begin at zero, follow commands such as “go left” and “go right,” then report the final position. He says diagonal state-space models and transformers cannot guarantee this class of computation under the cited theoretical framework, while a one-layer nonlinear RNN can handle it straightforwardly.
Behrouz places Mamba, Mamba-2, RetNet, S4 and S5 within the relevant diagonal-model limitation, associated with the TC⁰ complexity class. Titans’ nonlinear memory can handle the state-tracking task; a non-diagonal linear transition, as in DeltaNet, can also solve it, so the advantage is not unique to Titans.
Nathan’s pushback is practical: a reasoning model such as o1 would seemingly “talk its way through” the task. Behrouz agrees theory necessarily simplifies real deployments and may not capture extra inference tokens, prompting or model combinations. The claim concerns guarantees under assumptions, not an assertion that production transformers invariably fail simple tracking prompts.

12. The standout result is millions of tokens, on synthetic tests

Behrouz identifies long context as Titans’ most exciting result, immediately adding that the benchmarks are synthetic and may overstate gains on general tasks. Within that controlled comparison, small Titans models handled 2 million tokens and reached 10 million tokens with accuracy around 70%, while GPT-4’s benchmark performance dropped rapidly.
Memory as Context was particularly important at that horizon. Separating short- and long-term branches lets one retrieve compact historical information while the other reasons over a bounded local window; Memory as Gate and Memory as Layer achieved much shorter context lengths, with higher accuracy in those experiments.
Scaling order may change with the training-token budget. Behrouz cites a “mixture of Mamba” plot where Mamba-like recurrence performs better with less training data, but transformers eventually scale better and overtake it as token count rises. Architecture rankings from any fixed academic budget therefore should not be mistaken for immutable scaling laws.
Titans remained consistently strong compared with the tested transformer and linear-RNN alternatives as the team increased training tokens, without showing that crossover inside the explored range. Behrouz treats this as encouraging convergence behavior, not a final verdict: academic experiments cannot feasibly sweep ever-larger models and token budgets far enough to settle the scaling question.

13. Length generalization matters more than million-token training data

Nathan observes that the internet contains few coherent episodes spanning millions of tokens, seemingly leaving long-memory models without suitable training material. Behrouz’s answer is length generalization: a model need not train directly on every target horizon if its architecture and training techniques let it extrapolate from shorter sequences without a severe performance drop.
Pure recurrent or neural-memory models can struggle as length grows, with degradation appearing around 16K in Behrouz’s experience; unmodified transformers face related problems. Positional encodings, normalization techniques and architectural choices all matter, so more long data helps but “it’s not the end of the story.”
Memory as Context appears well suited to extrapolation because responsibilities remain bounded. The memory module summarizes a long stream into a fixed number of retrieved tokens, while local attention never has to attend over the full history.
Nathan proposes grafting Titans memory onto an off-the-shelf Llama or R1 model, followed by continued training so attention learns to use retrieved context. Behrouz says it is “potentially” possible and hopes such work happens, but offers no retrofit result; the episode’s implication is a research opportunity, not a deployment-ready recipe.

14. Catastrophic forgetting remains the roadblock to lifelong agents

Nathan asks whether memory constrained to a pretrained model’s latent space can truly learn an unfamiliar domain such as GE’s or 3M’s history. He suggests continued pre-training might enrich the base representations before runtime memory begins operating, combining durable domain knowledge with evolving episodic experience.
Behrouz redirects the issue toward catastrophic forgetting. Updating parameters across long periods and successive tasks can make a model adapt to the present context by erasing earlier ones. A finite neural memory may extend context dramatically, but that is not equivalent to indefinitely accumulating skills and knowledge without interference.
The concession matters for the agent thesis: Titans may help a system remember mistakes and maintain continuity, but it does not yet yield a lifelong worker that “doesn’t make the same mistakes twice.” Behrouz says overcoming forgetting at that level would also bear on robotics and reinforcement learning—evidence of how large the unsolved problem remains.
Behrouz’s preferred next experiment is to move beyond text into agents, decision-making, reinforcement learning and other modalities where long histories are intrinsic. Different integrations may suit different domains: MAC might be suitable for language modeling, while MAG might be effective for decision-making. The decisive test is whether Titans becomes broadly useful across deep learning, as transformers did; for now, “we don’t know.”

E191 | Small, Beautiful Opportunities Are Here: On the New Paradigm in This Wave of AI Agent Evolution Ex-Neuralink Founder: AI Enhanced Bodies Are Nearly Here w/ Max Hodak | EP #171