103: The Evolution of Attention Optimization: DeepSeek and Kimi
Summary
NSA and MOBA’s shared breakthrough is not another sparse Attention pattern, but pushing sparsity into pretraining and bringing theoretical speedups to the GPU. Most prior approaches trained densely and only introduced sparsity at inference, creating a gap between training and deployment; these two works offer positive evidence that, after sufficient training, sparse models can match Full Attention and in some cases even outperform it. “You can be smart and fast at the same time.”
Long chains of thought have upgraded Attention optimization from a “long-input product feature” into an infrastructure problem spanning training, RL, and inference. Kimi-like use cases once focused on quickly pre-filling an entire book; after R1, RL must repeatedly generate, process, and learn from increasingly long COTs, lifting both training and decoding costs. 肖朝军’s view is that “long COT has already become a very important problem,” with optimization likely to shift especially toward the RL phase and long outputs.
NSA and MOBA converge on a hybrid of block sparsity, local windows, and dynamic block selection—not purely dynamic or purely static designs. Each query first selects relevant historical blocks, reads the details within those blocks, and retains nearby context by default; dynamic selection preserves quality, while static windows preserve efficiency. Block-level computation also enables contiguous memory access and high parallelism: “Either I want the whole block, or I want none of it.”
The technical moat is shifting from good ideas themselves to co-design across algorithms, operators, and hardware. GPUs are naturally good at regular, dense, contiguous computation; without low-level implementations in Triton/CUDA and similar tools, lower theoretical compute may never become real throughput. Papers, model weights, and training/deployment code are separate layers of open source. 肖朝军’s reflection on his early InfLLM work is blunt: “the speedup never materialized,” while NSA’s impact lies in translating theoretical pre-filling and decoding gains into practice and extending sparse computation into training.
The most informative way to evaluate sparse Attention is not a single leaderboard point, but training curves and the performance-efficiency Pareto frontier. 傅天予’s standard is: at the same speed, is it the smartest; at the same intelligence, is it the fastest? Reducing sparsity can always bring performance progressively closer to dense models, so the real unknown is whether sufficient training resources can bring sparse models to the same ceiling. The two are most interested in NSA’s long-reasoning math tasks, training loss curves, and long-COT decoding—not the already relatively mature “needle in a haystack” tests.
Sparse Attention solves which history to compute, but not why all history must be stored; memory capacity will be the next harder wall. Compute may already have increased 10x while memory capacity may not yet have doubled; offloading data to system memory or disk expands capacity at the cost of access speed. For research tasks lasting months or even 1-2 years, models need to learn “what should be stored and what should not,” which is closer to a memory architecture than a pure Attention speedup.
Multimodality and long-term memory will push long-sequence demand even higher, forcing sparse patterns to vary by modality. 傅天予 estimates that people read about 18K tokens of text per hour, while an hour of audio can become 90K model inputs, and an hour of video—even at just 1 frame per second—can reach 1 million tokens, approaching the length of the complete Harry Potter series. Over the long run, sparse Attention is the path closest to today’s architectures, while linear Attention/RNNs promise better scaling; the destination is not context length itself, but modality, knowledge, emotion, reasoning, and even autonomous research capability.
Deep dive
1. Attention Gives Every Word Meaning Through Context
肖朝军 starts with the model’s input: a large language model receives a sequence of tokens, but an individual token often cannot be interpreted on its own. The pronoun “it,” in particular, requires knowing what it refers to. Attention calculates the relevance between the current word and all preceding words, then weights historical information accordingly.
傅天予 adds that the mechanism is not merely asking “what does this word mean?” but “what does it mean here?” Every historical token retains its own vector representation, and the current token extracts historical content according to relevance. Semantic understanding and contextual association therefore happen in the same computation.
肖朝军’s intuitive translation is that Attention expands the fixed-size memory of traditional models into “all previous tokens.” When processing the current word, it calculates the relationship with each historical token and extracts the relevant information.
2. Transformer Replaced RNN’s Fixed Memory and Forgetting by Preserving All History
RNN processes input one word at a time in a loop, compressing prior information into a fixed-size matrix. 肖朝军 uses “I am 肖朝军” as an example: by the time the model reads “肖,” “I am” has already been pushed into memory. That may work for a short sequence, but after a 2-hour podcast, the name at the beginning may already be forgotten.
Transformer no longer continually squeezes history into the same fixed container; it preserves representations of every historical word. When “I” appears again 2 hours later, it can still establish a strong connection with the opening “肖朝军,” avoiding the natural forgetting problem RNN faces on long sequences.
程曼祺 summarizes the leap as a move from “long text” to “long long long long long text.” Hundreds of tokens once counted as long; early BERT could handle only 512 tokens, sparse Attention pushed that to 4K, and open-source models later moved from 8K and 32K to 128K and even 1 million tokens.
3. Full Attention’s Cost Scales Quadratically with Length
Full Attention is the standard mechanism in the original Transformer: all historical representations are retained, and every current token calculates relevance against all preceding tokens. Doubling the length does not merely double the data stored; total Attention compute scales as n².
A single token may require thousands of numerical values. At an assumed 2 bytes per value, multiplied across the layers and millions of tokens, the stored data can reach “several hundred GB.” In reality, GPUs typically have only tens of GB of memory, so long text may hit the capacity ceiling first.
The compute problem compounds as well: in a 2-hour conversation, the current word must scan 2 hours of history; after 4 hours or a full day, it must scan even more. “As length increases, both time and storage costs become extremely large”—the reason Full Attention cannot scale sustainably to ultra-long sequences.
4. Sparsity Is Not an Artificial Assumption; It Is Already Present in Attention
傅天予 explains mathematically that softmax in Attention is like a “softer version of max”: the largest term might be 0.9, the second 0.09, and the third 0.009. Among a large number of relevance scores, only a small number are naturally large, creating room to eliminate low-weight computations.
Language relationships are sparse as well. In a 2-hour, million-word podcast, a given occurrence of “I” may be highly relevant only to the opening “肖朝军,” rather than equally relevant to every word. Sparse Attention aims to exploit precisely these few but critical long-distance connections.
傅天予 also points to inspiration from neuroscience: the human brain does not fully connect every neuron to every other neuron. Its connections are sparse and may become sparser with age. From mathematical, linguistic, and neuroscientific perspectives, sparse Attention therefore offers a basis for improving efficiency.
5. The Architecture Race Has Split into Sparse Attention and Linear Attention
肖朝军 divides the mainstream approaches into two categories. One retains Transformer but reduces the amount of history Attention actually examines; the other, represented by Mamba, replaces Attention with a mechanism closer to RNN and accepts a degree of forgetting through fixed-size memory.
傅天予 adds that many current projects avoid calling themselves RNNs and instead use the term “linear Attention.” Some do process history sequentially, while others compress a long input into a smaller representation in one pass. They are not all traditional RNNs, but they all aim to make cost grow linearly with length.
Before Mamba, sparse Attention was more mainstream; after Mamba, it is difficult to say which path will dominate. Sparse approaches are compatible with existing Transformer architectures and require smaller changes, while linear approaches promise better asymptotic complexity. At this stage, however, purely linear systems may still face capability losses.
6. Flash Attention Showed the Industry That Real Costs Can Change Dramatically Without Changing the Model
FlashAttention does not change Full Attention’s mathematical result. It reorders computation and data access at the systems level so that the same Attention runs faster and uses less memory. Its distinction from sparse Attention is that it is “fundamentally identical to Full Attention at the computational level.”
傅天予 divides the optimizations users encounter into three layers: faster GPUs are hardware; pushing the utilization of peak compute and memory bandwidth as close to their limits as possible is systems work; sparse Attention and MoE are model algorithms. According to 肖朝军, FlashAttention took Attention efficiency “to another level” and reduced memory usage by at least an order of magnitude.
Even after the model is fixed, the generation paradigm can still be optimized. Generating 2 tokens at a time instead of one, generating a paragraph at once, or having 10 models write 10 passages in parallel can all reduce total latency. Long-text efficiency is not controlled by a single knob; it spans the full stack from chips to service scheduling.
7. MLA, Mooncake, and Sparse Attention Address Different Bottlenecks
肖朝军 uses DeepSeek V2’s MLA to illustrate another algorithmic direction. Transformer must preserve all historical representations; MLA tries to compress the several thousand values originally required for each token to, for example, 500, reducing storage and the cost of reading them step by step.
Both speakers understand Kimi’s Mooncake as serving long-text use cases while operating mainly at the systems layer. The underlying algorithm still relies substantially on Full Attention, with the emphasis on systems-level optimization.
傅天予 stresses that the systems objective also depends on the scenario. Optimizing a single component is a different problem from serving large numbers of users while balancing each user’s latency against overall throughput.
8. Papers, Model Weights, and GitHub Code Represent Different Levels of Openness
程曼祺 notes that NSA initially released a preprint without a GitHub implementation, while MOBA already had a project page and engineering code. 傅天予 explains that a paper is mainly a technical summary and method description; low-cost replication of training and inference also requires hardware-adapted code.
“DeepSeek is open source” also needs to be unpacked. Releasing model weights does not mean outsiders can reproduce its low-cost training and inference process. Without hardware-co-designed operators, training frameworks, and deployment code, the community receives the model’s result—not the complete engineering process.
GitHub itself does not automatically mean fully open source. A repository may contain only the model or a demo while omitting training and low-level implementation. For outside developers, the paper answers “what is the idea,” the weights answer “can the model be used,” and the engineering code comes closer to answering “can the cost be reproduced.”
9. NSA and MOBA’s Most Important Common Feature Is Sparsity from the Start of Training
傅天予’s first notable observation is that “both sides conducted sparse pretraining.” Most earlier work trained with dense Attention and removed many connections only at inference; even though Attention during dense training is itself somewhat sparse, the mismatch between training and deployment inevitably introduces error.
The industry initially prioritized making models smart enough, leaving efficiency for post-processing; dense training was the safer choice for maximizing capability. Now that models must serve large user populations, efficiency itself has become a core constraint, making the industry more willing to invest in validating sparsity during training.
NSA at least shows that sufficiently trained sparse Attention can match Full Attention’s performance and may even outperform it in some cases. 傅天予 calls this “a shot of adrenaline” for the field: in response to the objection that users must “become a little dumber to get faster,” the answer becomes, “you can be smart and fast at the same time.”
10. Long COT Shifts Long-Text Demand from Pre-Filling Toward RL and Decoding
肖朝军 believes NSA’s timing is closely related to R1. Long text once mainly meant feeding a book into the model, making pre-filling the systems focus; reasoning models can output extremely long chains of thought, extending long-sequence costs from the input side to the output side.
In RL, the model first generates large numbers of long reasoning trajectories, then updates repeatedly based on rewards. The trend shown by R1 is that as RL training steps increase, output length also grows. If training continues to scale, sparse mechanisms must enter the training phase rather than being added as a deployment fix.
Kimi/Moonshot’s product scenario is more focused on long inputs and pre-filling. NSA can also accelerate long inputs, but 肖朝军 is more interested in its performance on long outputs and decoding. “Making this chain of thought extremely, extremely long” has become an important direction for further improving reasoning ability.
11. Trainable Sparsity Simultaneously Removes the Training-Inference Gap and May Lower Training Costs
The focus of “trainable sparsity” is not merely whether the sparse pattern can be learned. 肖朝军 gives it two meanings: training and testing/deployment use the same Attention, avoiding a mechanism switch; and training itself can also accelerate because of sparsity.
The difficulty is that GPUs are naturally better suited to regular dense matrix operations, and sparsity does not automatically produce speedups. If dynamic filtering, fragmented reads, and control overhead exceed the computation saved, theoretical compute may fall while actual runtime does not improve.
Moving from inference-time sparsity to pretraining-time sparsity therefore is not as simple as flipping the switch earlier. The algorithm and low-level operators must be jointly adapted to the training setting.
12. Dynamic Attention Preserves Capability; Static Attention Preserves Efficiency
A static mechanism specifies in advance where the current token can look—for example, a sliding window that sees only the previous 512 tokens. The system can retrieve earlier data without interpreting the current content, keeping control overhead low and execution fast. The cost is that a critical name far away may never enter the window.
A dynamic mechanism decides where to look based on the actual token content. It can capture the connection between “I” and “肖朝军” 2 hours earlier while retaining position-dependent patterns. But the system must first read the query, identify relevant history, and then perform irregular accesses: “the results are good, but it is slow.”
傅天予 sees a clear tradeoff: position is static, content is dynamic. Purely static mechanisms mainly retain positional patterns, while dynamic mechanisms account for both. Once deployed on a GPU, dynamic decisions add control overhead and fragmented access, making high efficiency inherently harder.
13. NSA and MOBA Both Mix Dynamic Block Selection with a Fixed Local Window
Both mechanisms first divide history into blocks, construct representations or calculate relevance for each block, select relevant blocks at a higher level, and finally read the details within those blocks. Both also retain local context around the current token by default, because natural language is usually most sensitive to recent content.
The KV blocks selected by each query can differ: as the model generates different words, it dynamically calls on different parts of history. The shared framework is “coarse filtering first, fine inspection second.” The main differences lie in how block representations are built, how relevance is calculated, and how the underlying implementation is organized.
程曼祺 once obtained the explanation from DeepSeek that “MOBA is block-level, while NSA is finer-grained.” The two speakers then clarified that NSA is also sparse at the block level. MOBA is not purely dynamic routing either: it always selects the block containing the current token, giving it a static component.
14. Block-Level Sparsity Is the Key Hardware-Aligned Interface
GPUs derive their computing advantage from single instruction multiple data: parallelism is highest when many data points execute the same instruction simultaneously. If only scattered positions perform different operations, the compute units are difficult to use efficiently.
Memory access also favors contiguity. A GPU retrieves a large block of data at once; if only a few values are useful, most of the bandwidth is wasted. Block-level sparsity makes the choice binary: “Either I want the whole block, or I want none of it.” Contiguous memory access and batched computation then work together.
肖朝军 explains that one of the key meanings of NSA being hardware-aligned is block-level processing. But writing the algorithm in blocks is not enough; real speed depends on extensive operator optimization. His InfLLM had already argued that sparsity must operate at the block level to be hardware-friendly, while NSA goes further in optimizing the execution layer.
15. Triton Lowers the Barrier to Writing Sparse Operators, but It Is Not the End of Performance Optimization
NSA’s low-level implementation uses Triton. 傅天予 describes it as an OpenAI-open-sourced programming interface built above CUDA, particularly suited to block-sparse computation; researchers can use it to write custom operators.
The guests mention a story explicitly labeled “an unreliable piece of industry gossip”: during the GPT-2 or GPT-3 era, OpenAI may have experimented with training using sparse Attention, and the related systems work may have helped give rise to Triton. The rumor’s accuracy is unknown, but Triton later became a tool researchers were willing to use for implementing block sparsity.
肖朝军’s view is that Triton first makes operator development more approachable. To push performance further toward the limit, DeepSeek may still need to work deeper in the stack. He also speculates that OpenAI must have worked on long-text optimization internally, although the outside world is “not entirely clear” on the specific technology it used.
16. As Attention Patterns Converge, Real Speedups Matter More Than Architecture Diagrams
傅天予 believes that the design question of “which historical tokens to calculate relevance against” is becoming less decisive. As long as the pattern is learnable, the gap between algorithms may not be especially large. NSA, MOBA, and earlier academic work have converged substantially on the ideas of chunking, selecting blocks, and retaining a local window.
肖朝军 cares more about whether theoretical savings translate into wall-clock time for pre-filling and decoding, with particular attention to NSA’s decoding speedup. Long COT simultaneously determines RL training cost and the serving cost of reasoning models after deployment.
This is also the basis of his self-criticism about InfLLM: “I also wanted to take it into the training phase,” but because he did not understand hardware and low-level operators, he assumed sparsity was inherently unsuitable for GPUs and eventually abandoned the effort. After seeing NSA, he realized the constraint lay not only in resources but also in the limits of his own understanding.
17. Academia Chose Training-Free Approaches Because of Both Resource and Product Constraints
Pretraining a sparse model requires substantial compute, but 傅天予 points to a deeper challenge: academia lacks industry’s data, training experience, and accumulated tricks. Even if researchers train a weaker dense model and a sparse model separately for comparison, the result may still carry limited weight with industry.
When building MOA, they deliberately insisted on “no training,” hoping that any dense model could be used as a plug-and-play base. Besides saving cost, this avoided changing the original model’s preferences through training: a model that was originally polite might suddenly become “very clever” after modification.
肖朝军’s InfLLM likewise put “Training-Free” in its title. At the time, open-source models were mostly around 8K, long-text interest was far lower than today, and long COT did not yet exist as a requirement. Adding the capability at test time was therefore the more realistic research choice.
18. The Most Important Experiments Are Training Curves and the Performance-Efficiency Frontier
肖朝军 is most interested in NSA’s small table on long-reasoning math problems and how training loss declines as training progresses, including NSA’s Figure 4 and MOBA’s Figure 3. Traditional pre-filling and “needle in a haystack” long-context tests are generally expected to perform reasonably well.
Performance alone is not enough, because lowering sparsity and moving closer to dense computation can always bring performance progressively closer to Full Attention. 傅天予 proposes the Pareto-frontier standard: “Among models as fast as me, am I the smartest? Among models as smart as me, am I the fastest?”
Training curves answer the question that was genuinely unknown: after receiving sufficient resources, can a sparse model reach the same ceiling as a dense model? The positive signal from both papers is that the two will “most likely eventually converge,” and sparse models may even perform better under certain settings.
The goal of these papers is therefore not to decisively beat Full Attention on capability, but to run as fast as possible with essentially no performance loss. Once comparable capability is achieved, the next question is how much training and deployment cost can be reduced.
19. Long-Text Evaluation Has Passed Through Three Stages: Speaking Like a Human, Speaking Truthfully, and Thinking Toward the Right Answer
傅天予 calls the earliest tasks “speaking like a human”: once the input exceeds the training length, the model should not mix languages, output strange characters, or produce broken sentences. Early work such as StreamingLLM mainly addressed this layer, commonly using perplexity or training loss to measure how closely the output matched human text.
The next stage was “speaking truthfully”: given the family relationships among A, B, C, and D, identify A’s great-grandfather. At this point, capitalization, given name, or surname does not matter; the entity simply has to be identified correctly. The metric shifts from language matching to accuracy.
Reasoning models have now entered the stage of “thinking toward the right answer.” Not every sentence in a chain of thought is necessarily useful, and R1-Zero may even produce unreadable content or mixed language. If the only concern is the improvement in final problem-solving, existing reasoning benchmarks remain sufficient.
傅天予 points out that sparse Attention may change the pattern of chains of thought and even make them less readable. Relevant data may already exist, but there is no mature metric for COT readability. Whether to construct a new benchmark depends on whether researchers consider readability an objective.
20. Ablations and Reproduction Still Need Validation in Real Use Cases
Ablation experiments remove components one by one and observe the change in performance to determine each mechanism’s contribution. If a proposal is complex, this decomposition is important; if removing the core leaves only Full Attention, directly comparing performance and speed is sufficient, and there is no need to force the result into an ablation framework.
Both speakers believe NSA describes its algorithmic process and design ideas in considerable detail, and members of the community had already quickly reproduced the underlying operators. But reproducing the implementation and reproducing all of the paper’s results are two different things; real-world performance still requires further validation.
肖朝军’s position is “do not trust blindly.” Whether the result comes from an individual reproduction or from DeepSeek itself, it must ultimately be tested in actual use. Sparse approaches are particularly prone to a gap between attractive theoretical compute and disappointing real-system speedups.
21. Sparse Attention Does Not Reduce Storage Complexity; Storage May Become the Next Hard Constraint
Sparse Attention eliminates a large amount of historical computation but still has to store all the content. As context extends to several months or even 1-2 years, storage will eventually become a problem. RNN sits at the opposite extreme, retaining only fixed-size memory but potentially sacrificing capability; a satisfactory middle ground remains elusive.
傅天予 notes that GPU compute has advanced far faster than memory. Across several chip generations, compute capability may increase 10x while storage capacity may not yet have doubled. High-speed memory consumes significant chip area, and expanding it further would quickly reduce yields and raise costs.
Moving data to system memory or even disk can expand capacity, but introduces transfer and access latency. This can relieve the problem of “not being able to store it,” but cannot also solve “being able to read it quickly.” 肖朝军 distinguishes the capacity ceiling from data-transfer speed: both matter, but they are not the same constraint.
22. A True Memory System Must Decide What to Store, Not Merely Compute Less from All History
肖朝军 uses research tasks to illustrate the need. Solving a math problem may take only 10K or 20K tokens, but selecting a topic, running experiments, and writing a paper may take months or even 1-2 years. If a model is to become a doctoral student or scientist, storing the entire process word for word would clearly create problems.
A higher-level solution is hierarchical summarization: read 100 papers, write a 20-word summary for each, end up with roughly 2,000 words, and then decide which original paper to reread. This is not a change inside Attention; it is a way for the generation paradigm and general algorithms to manage long contexts.
The two speakers therefore distinguish among long sequences, Attention, and memory. A long sequence is an input that keeps growing; Attention concerns how that input is processed once it arrives; memory decides “what should be remembered and what should not,” then passes the retained content to the model together with the new input.
傅天予 closes with 汪玉’s familiar formulation, “Y=F(X)”: multimodality makes X longer, while chains of thought make Y longer. F must compute quickly and remain correct with both long inputs and long outputs. Improving Attention is only one way to achieve that goal.
23. Multimodality Turns an Hour of Content into Millions of Tokens and Changes Sparse Patterns
傅天予 gives an order-of-magnitude comparison: people read roughly 18K tokens of text per hour; an hour of audio converted into model input may contain 90K tokens; and an hour of video, even at just 1 frame per second, may reach 1 million tokens. “An hour of video is half a movie to us,” while its text equivalent approaches the length of the complete Harry Potter series.
At roughly 100 tokens per frame and 24 frames per second, 1 second produces about 2,400 tokens and 10 seconds about 24,000. Current systems often sample 1 frame per second or even 1 frame every 10 seconds, but this can lose the continuous streaming information in video.
Video has a different relevance structure from text. When tracking a ball, the model needs to focus on the same spatial position across different frames rather than always looking at adjacent tokens, potentially producing a jumping pattern of “looking once every roughly 220 tokens.” Sparse Attention must adapt to the structure of each modality.
傅天予’s FrameFusion starts from redundancy rather than importance alone. Some objects in a video are important but appear repeatedly in every frame, so they do not need to be retained multiple times. “Keep only what is important and unique”; both repeated and unimportant content can be discarded.
24. Sparse Attention Is the Near-Term Engineering Fix; Linear Attention and Long-Term Memory Point to the Further Horizon
傅天予 believes sparse Attention is compatible with existing architectures and does not require radical changes, but it only reduces compute by a constant factor. As sequences grow without bound, it will still encounter problems. Linear Attention offers a blueprint in which growth is slower; even if it starts slower on short sequences, it becomes more advantageous once sequences are sufficiently long.
肖朝军 summarizes future capabilities as modality, knowledge, emotion, and ability: receiving rich modalities; retaining knowledge approaching the scope of the internet; remembering shared experiences across a user’s “lifetime”; and using long COT for deeper reasoning. All 4 depend on long sequences, but simply expanding the window does not automatically provide them.
Both speakers view autonomous research as a concentrated expression of higher intelligence because it requires reasoning, memory, efficient learning, and exploration of new knowledge to work together. Current models still derive their knowledge and supervision from humans; if a model could conduct research, it might “iterate and upgrade itself,” whereas having models repeatedly train on self-generated text today can still cause them to collapse outright.
On the OpenAI roadmap’s claim that “organization comes after innovators,” 肖朝军 remains skeptical: ants can also organize and collaborate, and a multi-agent organization may not require the highest possible intelligence from any individual agent. 傅天予 sees the two as parallel paths—individual capability and collective cooperation—saying, “which one develops first is not necessarily clear.”