Summary

Data walls are making algorithms valuable again, and China leads in architecture innovation. 杨松林 puts it plainly: “Domestic algorithm innovation is definitely stronger,” especially in architecture—a product of positioning. China has fewer GPUs and faces greater pressure to improve efficiency; “some Silicon Valley companies basically have too many GPUs, so they can’t be bothered,” while “of the three horses, at least one has to run faster.” He only has an intuition that the US is investing much more heavily in optimizers, with China catching up. Kimi was “the earliest to eat the Mew crab.”
Attention is the next architecture-level breakthrough, and Chinese companies are placing separate bets. The episode opens by calling DeepSeek’s MoE the biggest architectural breakthrough of the past few years. 杨松林 thinks MoE has already become a global “显学”: “Transformer has only two modules.” The FFN has been thoroughly refined, and “I think everyone can take a crack at attention too—why not.”
Kimi Linear/KDA is built around fine-grained decay and a 3:1 hybrid mix. DeepSeek R1 and Kimi 1.5 made tens of thousands of tokens of long chain-of-thought common, making full-quadratic attention “too expensive” during decoding. KDA builds on his Gated DeltaNet, replacing Mamba-2-style coarse gating with an independent decay rate for every dimension, delivering “fairly substantial” gains in multi-hop reasoning, coding, and math. The 3:1 linear-to-global ratio has become “a consensus within the non-consensus of Hybrid Linear”—ByteDance’s from-scratch experiments and Qwen3-Next both converged on it.
Pure linear attention has been written off. “The current consensus is that pure linear attention doesn’t work”: an RNN’s state stays fixed, so in theory it cannot retain arbitrarily long text. The only known weakness of hybrid architectures is long-context multi-hop reasoning, but he thinks the gap “could potentially close directly, or even reverse,” because “hybrid linear attention is only a beginning.”
MiniMax M2’s return to full attention may be the price of incomplete evaluation. M1’s Lightning Attention “felt like work from two years ago”; its evaluation focused mainly on MMLU-style metrics and missed multi-hop reasoning, so the large drop in agent and coding tasks prompted a retreat. She calls this “history moving in a spiral,” and says M3 may return to hybrid attention. The sharpest line came when he was asked whether he had worked with MiniMax: “If I had been involved, they probably wouldn’t have used this approach. I would have felt it was going backwards.”
The end state is a fusion of sparse and linear attention, not an either-or choice. The true rival to linear attention is sliding-window attention—GPT-3 and, he says, OpenAI’s open-source models should both use global plus sliding-window attention. His ideal architecture would “replace the global attention in hybrid attention with sparse attention”: sparse handles retrieval while linear attention saves KV cache. The bottleneck is that sparse attention “doesn’t select accurately enough,” which is why DeepSeek uses a distilled indexer. The ceiling for Transformer is clear: “First let’s see if we can eliminate global attention.”
Hardware is the final arbiter. Hardware is co-evolving with Transformer—Tensor Cores and Blackwell—and FA4’s matrix multiplications are now so fast that even softmax’s exponential has become the bottleneck. Algorithms must be matrix-multiplication-friendly: “Is your algorithm made of gold or silver, that hardware companies should optimize it for you every day?” On hardware affinity, DeepSeek beats Kimi by an “Absolutely”; DSA’s FP8 indexer could be a candidate for the next generation of architectures.
The bottleneck in deployment is the surrounding infra, and the positive feedback loop has just begun. He said the field’s bottleneck was “infrastructure support not keeping up” at a MiniMax roundtable six months ago. FLA’s Triton kernels were merely “good enough,” but the open-sourcing of Qwen3-Next and Kimi Linear is pushing inference engines such as SGLang to add hybrid support. His advice to young researchers: “Architecture requires compute. Without compute, you can’t do architecture”—start by finding a lab internship.

Deep dive

1. After the data wall, algorithms are valuable again—and China leads in architecture

小珺’s opening framework: data, compute, and algorithms are the three horses driving AI. Data walls are forcing companies to “sculpt model architectures again, in the hope that the magic of scaling laws can continue.” 杨松林 agrees: “You need to compress more intelligence into a finite amount of data.” When data could still scale indefinitely, data efficiency was not very meaningful—“everyone could just close their eyes and add data… all they needed to do was buy GPUs.” Once the wall arrives, “you ultimately have to return to the fundamentals, which are algorithms.” He recalls OpenAI’s CTO making a similar point: at this stage, algorithm research will regain importance.
On China versus the US: “I think domestic algorithm innovation is definitely stronger—if we’re talking about architecture, domestic architecture is definitely stronger.” China has fewer GPUs and therefore greater pressure to improve efficiency, creating more incentive to try efficient variants. “Some Silicon Valley companies basically have too many GPUs, so they’re too lazy to bother.” 小珺 jokes: “It doesn’t matter if your brain isn’t that great; just pile on compute first.” 杨 replies: “Of the three horses, at least one has to run faster.” But he feels the US invests much more heavily in optimizers, while China is gradually catching up. Kimi was “the earliest to eat the Mew crab.”
The counterexample defines the boundary: in fields where data remains scarce, there is no need to sculpt the architecture. In robotics, “if adding data can produce a significant improvement, then there’s no need to innovate on model algorithms yet… solve the data problem first, then come back to efficiency later.”

2. The next architecture-level breakthrough is attention—the MoE playbook will repeat

The essence of architecture optimization is: “Given the same number of flops, how do you use them better to achieve a lower loss?” The biggest breakthrough in recent years was MoE. When GPT-4 was rumored to use MoE in 2023, “many people were hesitant to follow.” Then one paper made MoE work, and everyone came back to it. MoE is now a “显学,” with every company building fine-grained MoE. The FFN has been refined into fine-grained MoE, while “Transformer has only two modules… I think everyone can take a crack at attention too—why not.”
The two modules scale different things. Reducing FFN flops is about fitting a larger parameter count; attention scales the context window. Lowering attention flops on long inputs benefits long generation and agents processing large amounts of workflow through larger context windows.
小珺’s preferred brain analogy: attention is working memory—when a new scenario appears, it dynamically processes information inside the context window. The FFN is closer to the hippocampus, “basically treated as an associative network of key-value pairs” that stores world knowledge.
Transformer’s ceiling is first an efficiency problem: “First let’s see if we can eliminate global attention.” Global attention is the main bottleneck to scaling the context window further and “will have to be removed sooner or later.” Once that is solved, “we won’t need the whole RAG stack anymore; we can just do in-context RAG directly inside the context.” The second frontier is what he calls “anti-bullshit capability,” potentially even “turning pre-training into starting directly from RL, letting the model learn from scratch without feeding it pre-training data.”

3. Linear attention 101: remove softmax, write it as an RNN, and let its fortunes follow the context wall

The guest is an MIT PhD student researching linear attention. He is known as the “mother of Linear Attention” for the Flash Linear Attention (FLA) open-source library and several influential papers; much of the industry uses FLA to explore linear attention. The path began with Hazy Research at Stanford, whose blog drew him into sequence modeling. He started by improving a Microsoft Research Asia paper whose original name he could not clearly recall and heard as RetNet, then combined gating and the Delta Rule into a unified RNN update rule while preserving hardware-efficient training.
The mechanism in one line: softmax attention generates an L×L matrix through QK^T, creating quadratic complexity. Linear attention removes softmax’s nonlinear operator and, after algebraic rearrangement, becomes an RNN-like recurrence—O(1) per step and O(L) for the full sequence. It is foundational architecture research in the pre-training stack. The hottest direction today is hybrid attention: retain softmax in a small number of layers and replace most layers with linear attention.
His historical summary: “Every time people become interested in linear attention, it’s definitely because they’ve hit some kind of context wall.” In the 2020 wave, BERT trained at length 512, while 8192 was already considered long context. The attention matrix had to be materialized in global memory, creating huge read-write overhead and frequent OOMs. Flash Attention broke through that wall: “If we can compute softmax attention exactly, then there’s no need to find an approximate attention mechanism to approximate it.” Linear-attention research cooled accordingly. His verdict on the kernel methods of that era for approximating softmax attention: “Looking at them today, I think this was a completely wrong direction.”
This revival is being driven by the explosion in long-context decoding. The cost of large models generating huge numbers of tokens “naturally led people to revisit this entire line of technology.” After Flash Attention, academia also recognized that hardware efficiency is essential for adoption—the original motivation behind his FLA Triton kernels. “Whenever softmax efficiency becomes a bottleneck, people come back to linear attention.”

4. Kimi Linear and KDA: long chain-of-thought pushed decoding costs into a wall

The trigger came at the start of the year. DeepSeek R1 and Kimi 1.5 had just launched, and RL was producing tens of thousands of tokens of chain-of-thought for test-time scaling. Full-quadratic attention made decoding “far too expensive”: every layer had to store a large KV cache, and decoding L tokens was also quadratic. Against the backdrop of long CoT and agentic AI, hybrid attention could cut inference costs substantially. The target was to match full attention without “losing performance while making inference many times faster.”
The engineering lead was 张宇, another core FLA author who joined Kimi after completing his PhD. Kimi uses a “scaling ladder” in which a model must perform well at one scale before moving to the next and continuing its comparison against full attention. 张宇 tried many mixing schemes and found that Gated DeltaNet worked best, but still lagged softmax in some areas. After adding fine-grained decay, “he found the improvement was actually quite large.”
KDA, or Kimi Delta Attention, is named as a counterpart to DeepSeek Sparse Attention. Technically, it builds on his Gated DeltaNet from last year. It replaces Mamba-2-style coarse gating—one decay rate shared by every dimension within an attention head—with an independent decay rate for each dimension. Every hidden-state dimension gets its own update frequency, which intuitively allows the RNN to make better use of its finite hidden state. Gated DeltaNet used coarse-grained decay at the time because it was “forced” to: neither the algorithm nor the kernel optimization was ready.
He describes his own contribution modestly. After seeing a paper—heard as “Comba”—that reduced Gated DeltaNet’s matrix inversion by one operation, “I immediately derived an algorithm applicable to KDA” and told 张宇. 张宇 wrote the kernel implementation and found that it scaled better. “That may be my only contribution to this work… the credit is basically with him, not me.” 张宇 conducted the massive ablation campaign. Positional encoding remains unsettled: “In these hybrid architectures, everyone is cutting RoPE; the question is how much.” Kimi removed 100% of it with NoPE, while Qwen3-Next removed 75%, retaining 25% partial RoPE. RoPE may impede length extrapolation in hybrid models, but there is no consensus yet.

5. Archaeology of the variants: advanced techniques were buried along the way

The earliest linear attention, in 2020, “really didn’t work”—even short texts were difficult. The first major improvement was decay. A representative paper introduced input-independent decay and enabled linear attention to scale effectively in language modeling. This evolved into input-dependent decay, as in his Gated Linear Attention and Mamba/Mamba-2: the model dynamically decides how much to forget for each token. If it decides the prior content should not be forgotten, it sets the value to 1 and retains it completely; if the information is no longer useful, it sets the value to 0 and clears it.
The second major improvement was the Delta Rule. Traditional updates use a Hebbian rule, simply adding the key-value outer product to the hidden state. The Delta Rule first uses the key to retrieve the old value from memory, linearly combines it with the input value using a learnable coefficient beta, then “subtracts the outer product of the old value and key from memory and adds the new outer product.” Addition is memory; subtraction is targeted deletion, making it much more precise than blanket decay. DeltaNet, Gated DeltaNet, and the work heard as RWKV-7 all use it.
The buried history is the episode’s through-line. DeltaNet was proposed by the “father of LSTM” in 2021, but “nobody took it seriously for the next few years—there was basically no follow-up work.” Fine-grained decay can be traced back to 2016, with related work existing by at least 2022, yet a 2023 paper used a coarser-grained version instead. “The earlier technology wasn’t passed down properly.” His self-assessment is stark: “If I hadn’t followed up on it, this entire technical path might have remained buried in the sea of literature.”

6. Pure linear attention is dead; 3:1 is the “consensus within non-consensus”

The current verdict: “The consensus is that pure linear attention doesn’t work.” An RNN’s state size is fixed, so as context grows it will “eventually run out of storage”; in theory, it cannot solve long-context tasks. Hybrid architectures preserve a floor by retaining global-attention layers, and Kimi Linear and Qwen3-Next show no degradation on long-context tasks such as RULER. He is less willing to call hybrid attention itself a consensus: “People are still trying different approaches in different places.”
The convergence around 3:1 linear-to-global attention is clear. MiniMax previously used 7:1, but “there weren’t enough softmax attention layers, so the long-context guarantee may not have been as strong.” ByteDance ran extensive pre-training-from-scratch experiments and also found 3:1 best, with Gated DeltaNet outperforming other candidates. Qwen3-Next likewise uses 3:1 with Gated DeltaNet. “3:1 should be a consensus within the non-consensus of Hybrid Linear.”
Why hybrid attention may not be worse than global attention: after training, “you may find that the vast majority of attention layers are probably not doing anything… only attention in a few key layers is useful.” The network’s inherent redundancy creates room to replace layers with linear attention. Hybrid attention “could very well be an all-around better replacement.”
He sees long-context multi-hop reasoning as the main current weakness. These tasks depend heavily on relationships between tokens and on having enough softmax layers. But KDA’s replacement of coarse decay with fine-grained decay produced “fairly substantial” gains in multi-hop reasoning, coding, and math. He thinks the gap “could potentially close directly, or even reverse.” “Hybrid linear attention is only a beginning.”

7. MiniMax M2 returned to full attention—the price of incomplete evaluation

He praises the MiniMax team for sharing its experience so openly. M1 moved to Lightning Attention because it performed well on monitored metrics and was more efficient, but the evaluation pipeline may not have tested multi-hop reasoning. It “mainly looked at capabilities such as MMLU.” Once the team moved to agent tasks and coding, the drop in multi-hop scenarios was “very large,” so M2 used neither hybrid nor sparse attention and “simply reverted to full attention.” His technical assessment is blunt: Lightning Attention “is actually a relatively weak form of linear attention… it feels like a linear-attention model from two years ago.” The technology was still at a two-year-old stage.
His view of the reversal: “If you look at history, it moves in a spiral.” Any technical approach has to go through many rounds of validation before settling. M1 was adopted somewhat hastily without sufficient validation, so reverting was normal. MiniMax says it is still exploring hybrid architectures; “perhaps the next version, M3, will become a hybrid-attention architecture again.”
The sharpest line came when 小珺 asked whether he had worked with MiniMax: “No… If I had been involved, they probably wouldn’t have used this approach. I would have felt this approach was going backwards.” His involvement map: he was one of the authors of the Kimi Linear paper, is “quite familiar” with the Qwen3-Next training team and helped with academic discussions, and had no involvement with MiniMax.

8. Sparse versus linear: not rivals—the end state is a fusion

The two paths solve the same problem: efficient long-context decoding. DSA, in his view, should use sparse attention at every layer with no full-attention layers, but it does not save KV cache; it uses a distilled indexer from a trained full-attention teacher to select top-k tokens. Hybrid linear attention uses RNN-like layers for most of the network, sharply reducing KV cache and allowing larger batch sizes during decoding. As sequence length increases, each hits a different wall: hybrid attention remains constrained by global-attention efficiency, while fully sparse attention may be bottlenecked by KV-cache management.
His end-state idea, which he previously outlined in a Zhihu answer, is: “The best combination is to replace the global attention inside hybrid attention with sparse attention.” That removes global complexity, while linear layers reduce KV-cache size. “This is the efficient architecture I currently consider ideal.” The condition and bottleneck are straightforward: if sparse attention can select accurately, it can completely replace the full-attention layer in theory—but “its problem right now may be that it doesn’t select accurately enough.” The root issue is training difficulty: from scratch, “the gradients aren’t very accurate… learning to select blocks is quite hard,” with various sparse-gradient problems. That is why DeepSeek uses distillation. As far as he knows, no industrial system combines both yet, though academia is exploring it.
The real competitive relationship is different: “The main competitor to linear attention may actually be sliding-window attention.” The GPT-3 technical report explicitly describes a combination of global and sliding-window attention, and he believes OpenAI’s recent open-source models still use this setup. The fair comparison is to control state size. Sliding-window attention has a bounded KV cache; make that limit comparable to the number of RNN state variables. “Decoding is basically a memory-bound process.” If the states are roughly the same size, efficiency should be roughly the same.
There is still no true apple-to-apple comparison. 小珺 notes that the Kimi Linear paper calls itself “the first hybrid linear architecture verified to outperform full attention,” but it compares only against full attention, not sparse attention. “Perhaps resources were limited… there weren’t enough GPUs to run comparisons across all these different approaches.” Silicon Valley’s systems are closed-source, so “you also don’t know whether they ran them.” “If there were somewhere willing to do charity and run an apple-to-apple comparison, it would be better for everyone to understand the trade-offs.” He does not know which approach is better either.

9. Hardware is the final arbiter: everything converges on matrix multiplication

Responding to 小珺’s description of Transformer as “the chosen architecture of this generation’s hardware,” he says: “It definitely is chosen.” The FFN is a large matrix multiply. Although attention has one higher order of complexity than RNNs, it is far more hardware-friendly. “People would rather use a Transformer with higher theoretical complexity than an LSTM with lower theoretical complexity.” The historical parallel is structural: without Flash Attention, softmax attention would never have reached its current position. The parallel algorithm he designed for DeltaNet plays the same role for the Delta Rule.
The lock-in comes from co-evolution. Hardware is taking on “the shape Transformer prefers”: Tensor Cores, TMA, and dedicated memory for matrix multiplication on Blackwell. FA4’s matrix multiplications are now so fast that “even softmax’s exponential has become the bottleneck,” requiring approximation methods for exp—“which is pretty funny.” Algorithms therefore have to support matrix multiplication or they will never keep up with hardware efficiency. Chunk-wise parallelism in linear attention happens to be “basically a series of matrix multiplications.” Its lower training efficiency than FA4 on Blackwell is acceptable because many applications care only about inference efficiency.
His principle is that algorithms should be mathematically grounded while satisfying hardware’s universal principles, such as memory hierarchy and the superiority of matrix multiplication. They do not need to target H100 specifically, but algorithms that violate general principles “basically have no practical value in a scalability-is-all-you-need setting; they become self-entertainment.” Expecting hardware companies to accommodate an algorithm is fantasy: “Is your algorithm made of gold or silver, that hardware companies should optimize it for you every day?”
His ranking is unambiguous. Who has the best hardware affinity? “DeepSeek. Absolutely.” DSA’s indexer uses FP8 to calculate attention logits, requires no softmax, and removes the expensive exponential. It is “basically a huge number of matrix multiplications, extremely fast,” and could become a candidate for the next generation of architectures. “DeepSeek is a company that cares very, very deeply about hardware-algorithm co-design; infra has a relatively strong voice in their algorithm iteration.” Kimi is “not as relentless as DeepSeek.” As far as he knows, Kimi Linear is still trained with the Triton kernel written by 张宇—“good enough for now.” Once the approach is validated, a CUDA kernel can be added later.

10. Research method and ecosystem: archaeology, open source, and the positive feedback loop

His research philosophy is the episode’s most personal thread. Before starting his PhD, he had six months of free time after submitting his applications and spent all of it surveying architecture papers, going as far back as 2017. “I can say I’ve basically read every paper worth reading in this field… I call this archaeology.” A one-year-old paper is old; a five-year-old paper is “definitely an ancient paper.” His method is to search the literature for work that makes sense from a machine-learning perspective and can support a parallel algorithm. Gated DeltaNet came from “if you can’t beat them, join them”: DeltaNet could not beat Mamba-2, so he borrowed Mamba-2’s gating and combined A with B. His PhD went smoothly because “the bigger challenge is not knowing what you want to work on… when you’re clear about what you want to do, you don’t really encounter setbacks.”
Infra is the real deployment bottleneck. At a MiniMax roundtable six months ago, the host 俊贤 asked about the field’s bottleneck, and he answered: “The supporting infrastructure hasn’t caught up.” That surprised the host, but it was accurate: “Algorithmically, the developments of the past two years are already ready for large-scale experimentation.” FLA provides Triton kernels that are “basically good enough to use, but definitely not optimal.” Now Qwen3-Next and Kimi Linear are open-sourced, pushing engines such as SG—which previously lacked hybrid inference support—to add it. “A modeling company releases a promising open-source result → inference engines have an incentive to support it → the infra ecosystem develops → more companies dare to use it.” That is a positive feedback loop.
小珺 notices that he repeatedly talks about “playing with” and “sculpting” architectures, and cites the saying heard as 杨志令: “Don’t embroider flowers on top of things.” His response: “There’s no alternative right now. Compute is insufficient and data is limited, so we can only sculpt. But I think sculpting architectures is genuinely useful.” Once M1’s approach was refined, “it basically became a consensus.” “Sculpting” is not derogatory; it means polishing a module until it works better. One aside: DeepSeek’s OCR paper—washing books and PDF data for pre-training while also compressing context—is “an interesting idea, but I’m not sure how well the approach works.”
His advice to young researchers entering the field is singular and almost brutally practical: “Find a company and get an internship.” Architecture requires compute; “without compute, you can’t do architecture.” So start by finding a lab internship.

The Playbook on Buying and Running Companies Forever Triple H on How WWE Evolved: Trump, The Rock, and the Rise of the Antihero