Nested Learning

The episode frames continual learning—not another incremental gain in static pre-training—as the missing capability between today’s models and durable AI collaborators. Behrouz identifies two linked gaps: current LLMs cannot efficiently absorb new knowledge into billions of parameters without risking catastrophic forgetting, while token-space memory eventually exceeds context limits. The target is a model that “adapt[s] to the environment and the context” while compressing experience into increasingly general abstractions.

Nested Learning replaces a single training clock with modules that update at different frequencies, potentially shifting scaling from stacking more layers to nesting more learning timescales. Fast modules adapt to high-resolution recent context; slow modules preserve durable knowledge and extract higher-level patterns, provided there is effective knowledge transfer between them. The architectural thesis is that “frequency of update” and “knowledge transfer between the levels” become new scaling dimensions.

HOPE operationalizes the framework with multiple MLP memories and, in its fuller form, a self-modifying Titans module whose update rule evolves with context. HOPE Attention retains attention but replaces one fixed MLP memory with several MLPs updating on roughly 128-, 512-, and 2,048-token schedules in the reported configuration, as Behrouz recalls. Full HOPE replaces attention with a sequential associative memory that generates its own values, letting “the model itself” modify how it learns from every token.

The most diagnostic result is not a marginal perplexity win but HOPE’s ability to learn two previously unseen languages simultaneously in context. A conventional transformer can translate one unfamiliar language after receiving its grammar and dictionary, but “almost collapse[s]” when Manchu and MTOB are supplied together; adding HOPE levels progressively restores performance toward the single-language baseline. That is early evidence of a qualitatively different memory-management capability, not merely a better next-token predictor.

Behrouz argues that architecture and optimization are closely related because both are associative memories compressing different contexts. An architecture learns from tokens; an optimizer learns from gradients; momentum is itself a memory compressing gradient history. M3 applies the same multi-frequency idea to optimization with two memories and outperforms Adam and Muon in the reported setup, though Behrouz stresses that optimizer rankings are task-dependent.

“Language Models Need Sleep” adds an offline consolidation phase in which recent knowledge is distilled from fast memories into slower ones without unbounded model growth. Temporary parameters create room at the receiving level, then are removed and recycled after consolidation; compression pressure forces the slower memory to represent broader rules rather than copy examples. During “dreaming,” the model generates synthetic text from its own recent knowledge and trains on continuations, combining memory transfer with self-improvement.

The upside of persistent learning is extreme personalization, but the same mechanism creates unresolved privacy, alignment, evaluation, and product-versioning risks. Behrouz’s honest answer is, “I don’t have a very concrete idea” for fully controlling drift: a model could internalize everything about a user, adversarial inputs could become durable beliefs, and no operator can rerun a complete safety suite after every update. He points to knowledge transfer as a possible control point, augmented by input-dependent learning rates that may gate surprising but irrelevant data.

Continual learning could reinforce a winner-take-all platform, yet Behrouz expects differentiated intelligences to provide a countervailing ecology. A universally deployed model might compound experience into an unbeatable advantage, while the host speculates that personalized learners could instead specialize and forget unused competencies. Behrouz’s narrower claim is that varied systems—with different strengths and weaknesses—could be “better than having one single form of intelligence in the world.”

Nested Learning “is not a solution to continual learning”; it is a tool for discovering one.

🔗 Original source & video: Nested Learning: Ali Behrouz on the Quest for Continual Learning & Illusion of AI Architectures