A Guided Reading of Kimi K3's Technical Report: Architecture to MoE

Summary

• K3’s core achievement is not any single trick, but making “effective scaling” work for a fully open-source MoE with 2.8T total parameters, roughly 100B active parameters and a 1M context window. 孙宇涛’s first-principles view runs through the entire discussion: “Bigger models have a higher ceiling for intelligence.” There are countless ways to make benchmark results look good, but parameter count remains the most effective variable when it comes to the underlying capability of a model. He argues that pushing active parameters all the way to “a full 100 billion” does not depend on any special technology; it is a non-technical decision that nevertheless requires conviction.
• K3’s architecture is a composite of the best ideas from across the field, with every innovation grounded in a clear historical lineage. KDA evolved through RetNet, decay mechanisms, Mamba, DeltaNet and Gated DeltaNet, while 苏剑林 developed a GPU-friendly chunk recurrent form on top of DeltaNet. KDA then adopted channel-wise decay and co-designed its kernels around the dynamic range of BF16. Latent MoE came from the Nvidia team, while Quantile Balancing originated in 苏剑林’s blog. 孙宇涛 credits Kimi for citing prior work without diminishing the contributions of its predecessors.
• K3 made several counter-consensus choices, driven mainly by engineering and training logic. It dropped the industry-standard WSD schedule in favor of the easier-to-tune cosine decay; it avoided sparse attention because the cost of calculating sparse indices is high on Blackwell, and because sparse attention is not sufficiently compatible with the hybrid architecture or from-scratch pretraining. The interface can be preserved and revisited later.
• On-Policy Distillation (OPD) in post-training is increasingly a tool for simplifying team structure and training strategy. The approach runs from MiniLLM’s reverse KL to what Thinking Machines Lab calls OPD. Today, the more common use case is not distilling a large model into a small one, but “distilling the model into itself.” RL strategies and datasets are highly heterogeneous; OPD converts them into homogeneous teacher models that can then be combined through multi-teacher distillation. 孙宇涛 says teams including Xiaomi and JOM are using similar approaches.
• His historical framing of both DeepSeek V3 and K3 centers on size and effective scaling rather than isolated breakthroughs. V3 was among the earlier domestic models to scale to a meaningful size while completing the full pipeline. K3 has more than 3x the active parameters and nearly 3x the total scale, putting it on another level. But simply “turning up the numbers” does not solve the problem; the real innovation lies in making a larger model work. He also argues that science does not advance in discrete leaps—some points along a process of incremental improvement simply get labeled milestones.
• Kimi and DeepSeek are diverging strategically: one is prioritizing the ceiling of open-source capability, while the other places more weight on cost-performance. Kimi K3’s API is very expensive, and cost-performance was not the primary objective. 孙宇涛’s closing view is that frontier language models may have little fundamental innovation left; progress from here will be more evolutionary. Model scale will continue to increase, but not without limit. Once a capability and its task are clearly defined, sustained optimization can achieve them. He has turned to world models because, in his view, the large-model field offers individuals limited room for new credit.

Deep dive

1. The guest’s baseline: Architecture research has shifted from performance to efficiency

• 孙宇涛 is a PhD candidate in the Department of Computer Science at Tsinghua University and a Purui Scholar at Shanghai Innovation Institute. His doctoral work has focused primarily on large-model architecture and pretraining.
• His PhD research, which began in 2023, has centered on the inefficiencies of large-model inference. In the Vision and ImageNet era, elegant architecture could directly lift model performance. In the large-model era, parameter count remains the dominant driver of performance, while architectural changes deliver relatively modest gains. The inference gap between architectures, however, can be enormous—and it largely determines deployment cost. Architecture research has therefore shifted from optimizing performance to optimizing efficiency.

2. The starting point: Adding decay to linear attention and introducing chunkwise recurrence

• His first major work in the area did 2 things: it introduced a decay mechanism relatively early, allowing a finite state space to retain the information most likely to matter for the final result; and it found a trade-off between fully recurrent and fully parallel computation by proposing a chunkwise recurrent formulation of linear attention.
• Chunkwise recurrence preserves the benefits of linear complexity while allowing kernels to make fuller use of Tensor Cores and increase local compute density. Many subsequent linear-attention efforts—including Mamba-2, Gated DeltaNet and Kimi Linear KDA—have adopted the same chunkwise computation pattern.

3. Pure linear attention concedes; hybrid becomes the “trade-off that isn’t”

• 孙宇涛 says candidly that pure linear attention was “unquestionably a fairly failed attempt” as a model architecture: a finite state space makes it difficult to match full attention exactly on long contexts.
• Research consequently shifted from pure linear attention to hybrid attention, combining linear and full attention. Experiments show that hybrid attention is a trade-off at the architectural level, but not necessarily in model performance. With a sufficient share of full-attention layers, it can deliver lossless—or even better—long-context performance.
• A common mix is 1:3 full attention to linear attention, with full attention accounting for one-quarter of the layers. The speedup, however, is constrained roughly by that mix and tops out at around 4x. For a PhD student, the approach was highly effective but not especially novel, prompting him to look for a larger acceleration opportunity.

4. YOCO: One shared KV Cache across all layers, with full attention skipped during Prefill

• He started from first principles: the model has to access all preceding information, so eliminating KV Cache entirely along the token dimension is impossible. If savings cannot come from context length, they can come from the layer dimension.
• YOCO lets all layers share a single KV Cache while decoupling model computation from storage. It stores only one KV Cache but retains the computation structure of Full Attention across multiple layers, producing results broadly equivalent to full or hybrid attention.
• It can also skip Cross Attention during Prefill. The primary purpose of Prefill is to generate the KV Cache used during Decode; it does not need to decode a next-token prediction at every position. Linear Attention is therefore sufficient to generate the KV Cache.
• From a token-wise perspective, if information completeness is required, one KV Cache is already the minimum. It cannot be reduced further.

5. Loop LM: 2x the compute for almost 2x the performance, with little change in storage overhead

• 孙宇涛 breaks inference costs into 3 components: Prefill latency, Decode cost and KV Cache storage. These are the 3 main bottlenecks.
• YOCO primarily addresses Prefill and KV Cache. Decode can still be accelerated through sparse attention, although Kimi K3 does not currently use it.
• Traditional Loop Language Models have high Decode costs: they save parameters but not computation. At the same compute budget, unrolling the parameters or using a larger non-loop model may perform better. The KV Cache in a traditional loop also grows with inference depth.
• YOCO Universal concentrates the loop in the linear-attention component. Because linear attention’s KV Cache is nearly negligible and computation is saved as well, the incremental cost is small. With roughly 2x the original model’s compute intensity, it can deliver close to 2x the performance improvement, while model storage and KV Cache remain roughly unchanged.

6. Why pursue architecture—and why give the credit back to history

• He likes architectural innovation partly because it is “interesting and fun,” and partly because it can have a “force-multiplier” effect, making it a strong research direction for a PhD student. He also believes that by 2026, the room for architectural innovation may be narrowing.
• Around 2023, full attention was a major bottleneck in large-model inference and deployment. If large models were to be deployed at scale, attention had to be solved, giving architectural innovation substantial industry value at the time.
• His talks do not simply recite the K3 paper. He starts from the paper’s contributions and traces the history of the relevant work. A paper can tell you “how” something was done, but not necessarily “why,” what alternatives had been tried, or why the field ultimately converged on the current solution.

7. K3 at a glance: Effective scaling at 2.8T

• K3’s core selling points are model scale, context length and overall results. 孙宇涛 uses an extreme example to explain “ineffective scaling”: initializing a 2.8T model, training it on 100B tokens and releasing it would obviously not qualify as effective scaling.
• K3 achieves effective scaling at 2.8T total parameters and roughly 100B active parameters. At launch, that scale was clearly larger than that of other domestic open-source models; its context length also reached the 1M range.
• He insists that bigger models have a higher ceiling for intelligence. Benchmarks can be optimized in countless ways, but parameter count remains the most effective variable when measuring a model’s fundamental capability.
• Kimi K2 largely carried forward the DeepSeek-V3 architecture, with the emphasis on scale and non-architectural improvements. K3 combines work from the Kimi team, other industry teams and multiple architectural innovations and implementation refinements from the past year, creating a substantially different overall profile.

8. The linear-attention lineage: RetNet → Mamba → DeltaNet → Gated DeltaNet

• KDA’s complex equations were not designed in one shot. The starting point was simple RetNet-style linear attention, which accumulates the outer products of q, k and v. Position-dependent decay was then introduced incrementally, making linear attention no longer entirely position-agnostic.
• Mamba turned position-independent decay into position-dependent decay. DeltaNet sought to increase the context capacity of linear attention at the same KV Cache size, making it better suited to long contexts than RetNet and Mamba.
• When DeltaNet first appeared, it had no positional decay, so its perplexity and benchmark results were not particularly strong. Gated DeltaNet combined DeltaNet’s higher computational capacity with decay, producing a more expressive form of linear attention.
• 孙宇涛 at one point leaned toward conceding the pure-linear-attention path, believing it would struggle to match full attention. 苏剑林 chose instead to keep improving linear attention’s long-context capacity. Both directions ultimately produced results.

9. KDA: Channel-wise decay and kernel co-design

• In Gated DeltaNet and earlier work, each head typically used a single scalar decay coefficient. KDA replaces that scalar with finer-grained channel-wise decay, giving each channel its own coefficient.
• Channel-wise decay can degenerate into uniform decay, so it is strictly more expressive. The cost is a more difficult kernel: scalar decay requires handling only one variable per tile, while channel-wise decay is considerably more complex.
• K3’s lower-bound decay serves both the algorithm and the implementation. To represent relative position through absolute positions, q and k must undergo decay and inverse-decay transformations. For example, “subtract 2” can be represented as “subtract 3, then add 1”; through the associativity of multiplication, this produces a result equivalent to the recurrent form.
• Dividing by a very small number, however, creates numerical problems, so decay must be kept within BF16’s dynamic range. Kimi Linear uses 16-token tiles and derives the maximum allowable decay within each tile from the BF16 range.
• This is closer to code design: constrain the algorithm so that numerical problems do not occur within a tile, allowing a more efficient kernelized implementation. KDA ultimately adds finer-grained computational capacity to Gated DeltaNet while limiting the resulting difficulties in kernel and inference implementation.

10. The residual lineage: From ResNet to Hyper-Connections

• Attention Residual has deep ties to ResNet. ResNet addressed the problem of ensuring that capability and stability do not deteriorate as model parameters increase. 孙宇涛 notes that ResNet had both CVPR and ICCV versions, and that 何恺明 also discussed the relationship between Pre-LayerNorm, Post-LayerNorm and training stability.
• Post-LayerNorm was more common in the BERT era. As models grew, gradient vanishing and training instability became increasingly visible, and the industry shifted to Pre-LayerNorm.
• 苏剑林 stresses that Attention Residual cannot simply be a rearrangement of Pre-LayerNorm and Post-LayerNorm. It should be a superset of Pre-LayerNorm—at minimum, it must be able to reduce to Pre-LayerNorm—and should be strictly more powerful, otherwise large-scale training would not be safe.
• 孙宇涛 particularly likes Hyper-Connections. Its core idea is to use more capacity on the residual branch than in the model’s hidden state, representing the model’s state across inference depth. The idea is simple, but the paper’s formulation is abstract and difficult to read, so it did not gain broad attention at the time.
• MHC later attracted more interest, and he believes one important reason was that MHC came from DeepSeek. Hyper-Connections and Attention Residual are nearly free at inference, with no meaningful increase in inference cost, making them easier to adopt than approaches that trade slower inference for better results.

11. DenseNet’s echo and the adoption rule of “free at inference”

• DenseNet is an unavoidable predecessor to Hyper-Connections and Attention Residual. It allows deep layers to see all shallow-layer states directly and then aggregates them. ResNet focuses mainly on connections between adjacent layers; DenseNet aggregates information from all shallow layers.
• The DenseNet era did not yet fully exploit attention’s expressive power, relying primarily on a relatively large linear layer to aggregate hidden states. In the attention era, researchers found that this implementation imposed substantial additional compute, leading to lighter efforts such as DenseFormer.
• DenseFormer reduces overhead through infrastructure design, including block attention, and decouples the model depth. Strictly speaking, it is less expressive than full Attention Residual, but it claims limited capability loss while delivering a clear improvement over the baseline.
• 孙宇涛’s adoption rule is straightforward: if a design requires slower inference in exchange for better performance, teams will ask why they should not simply scale up the model; if inference is nearly free, there is little organizational resistance to adoption.
• Asked which works he likes most, he chooses DenseNet and Hyper-Connections. Later work’s stability improvements and co-design are important, but the larger conceptual framework was established by the earlier papers.

12. Latent MoE: Communications slimming that approaches a free lunch

• Under Expert Parallelism, one of MoE’s biggest challenges is all-to-all dispatch. Tokens must be sent to different expert cards, and communication volume rises with both the number of active experts and hidden-state size. Training stability, training methods and low-level infrastructure are consequently complicated.
• Latent MoE first compresses the hidden state used for dispatch into a lower-dimensional representation. Communication volume can fall by 2x or 4x, while the lost parameter count is offset by increasing the MLP intermediate dimension or using more experts and activating more of them.
• With the right configuration, Latent MoE can fully preserve standard MoE performance and may even improve on it. During training, overlap can partly hide communication. During inference, communication sits on the critical path and cannot be removed, so the inference benefit is more visible than the training benefit.
• If communication can be reduced without sacrificing expressive power, the design approaches a free lunch. Kimi K3 adopts this structure and adds stability measures on top of Latent MoE.

13. Two matrix multiplications and C2-GLU: Giving activations a mathematical upper bound

• Two consecutive matrix multiplications can be merged into one in terms of expressive power, but their optimization properties are not equivalent, and training can become unstable. If an architecture must split 2 linear layers apart, normalization is usually needed between them.
• MLA has a similar issue when a low-dimensional projection is projected back up. Latent MoE first compresses the hidden state, while the first FFN operation is itself a linear projection, so RMSNorm is inserted in between to control the hidden state.
• C2-GLU targets the tendency of intermediate MLP activations to explode. The problem is partly related to the optimizer and partly to numerical precision; outliers are more likely to emerge at low precision.
• GPT-OSS has used direct clipping, for example setting a clip value between 5 and 10 to impose a hard upper bound. 孙宇涛 considers hard clipping relatively crude. Replacing it with soft clipping naturally produces tanh, which approaches the upper and lower bounds smoothly.
• C2-GLU uses tanh to constrain the portion of the intermediate computation that can otherwise produce unbounded activations, giving MLP intermediate activations a strict mathematical bound and improving training stability. Its impact on expressive power still requires empirical validation, but it is a relatively safe design.

14. Muon and outliers: Architectural constraints are more direct than optimizer debates

• 孙宇涛 observes that outliers may be more likely when using Muon, but he also expects 苏剑林 to reject that attribution: any optimizer can produce outliers. If model behavior must be strictly constrained, the direct solution is to impose the constraint in the architecture rather than trying to manage it purely through the optimizer.
• He cites discussions around MuonClip: Adam is not completely stable in a strict sense, and DeepSeek-V3 may also have experienced instability that did not affect the final result. The difference may simply be when a phenomenon appears. If it must be tightly bounded, the model architecture should impose the bound.
• Moonlight introduced VDK relatively early in large-scale Muon training to preserve stability over longer runs. Kimi K2 added QK clip on top of Muon. Because QK Norm is inconvenient in an MLA setting, QK clip can instead constrain QK logits.
• In his view, DeepSeek-V4 and StepFun also use Muon but rely on GQA rather than MLA, allowing them to use the simpler QK Norm directly. The QK component can be constrained through clipping or normalization, while the MLP component can be controlled through activation functions such as C2-GLU.

15. Quantile Balancing: Replacing the heuristic in loss-free routing with a more principled solution

• Quantile Balancing was not published as a standalone paper; it was discussed by 苏剑林 in a blog post. It follows the Loss-Free Routing idea of controlling expert load through bias, while attempting to replace the relatively heuristic bias update.
• Loss-Free Routing’s bias directly affects which experts tokens select, but its update rule is ad hoc, lacks a strict mathematical convergence criterion and has an unclear coupling with the main-model update. Its performance is weaker in the bottom layers, so some implementations turn the bottom 1 or 3 layers into dense layers.
• Quantile Balancing attempts to derive the bias directly through linear programming, solving for a load-balancing bias from the aggregate activation pattern within a single step. To prevent information leakage between tokens, the β calculated in the current step is not used immediately; it is applied in the next step.
• Its advantages include eliminating one learning-rate-like hyperparameter and improving load balancing. The first layer can also use MoE directly, rather than being converted to dense because Loss-Free Routing fails there. This suggests that dense bottom layers are not inherently necessary; they are a workaround for a particular routing scheme.
• The K3 paper adds an engineering implementation. A large-model training batch can contain tens of millions of tokens, making exact quantile calculation expensive or even infeasible because all token activations would need to be stored. K3 buckets the value range—for example, dividing the 0-to-1 sigmoid output into multiple buckets—and then uses a histogram to approximate the calculation with constant storage, while supporting large-scale DP and EP.

16. ViT from scratch and optimizer compatibility

• Initializing with a pretrained ViT Encoder such as SigLIP 2 can accelerate convergence, but K3 argues that direct initialization may create greater training instability. “Instability” is relative and can be assessed through the gradient-norm distribution, the number of intermediate outliers and norm magnitude.
• Training from scratch may consume somewhat more compute, but with appropriate training the final result need not suffer and may be similar or better, while gradient norms are more stable.
• 孙宇涛 links this to training compatibility between Adam and Muon: whether an Adam-trained model can continue training with Muon, or a Muon-trained model can switch to Adam, the native and internally consistent training setup is generally best. On that basis, he speculates that Kimi may want to use an all-Muon training process for greater stability and compatibility.

17. Why K3 does not use sparse attention

• DeepSeek Sparse is used in DeepSeek-V3.2, and GLM-5 uses a similar form of DeepSeek Sparse Attention. But in Decode, calculating the sparse index itself is expensive, so the actual speedup is small. On Blackwell, more advanced hardware may make this overhead more visible; in some cases, sparse attention may not be meaningfully faster than full attention.
• The GLM team proposed Index Cache, allowing different layers to share an index—for example, sharing once every 4 or 8 layers. 孙宇涛’s own proposal goes further by calculating the sparse index only once and amortizing its cost across all layers, much like KV Cache.
• After K3 adopts hybrid attention, several linear-attention layers sit between adjacent full-attention layers. It remains unclear whether sharing indices across layers would work effectively or deliver enough acceleration.
• Sparse attention also generally cannot be trained from scratch directly. The usual path is to train full attention first and convert it to sparse attention through post-training. K3 could eventually convert its existing full-attention layers to sparse attention, but that would require additional work; implementation would be even harder with MLA.
• His conclusion is that K3 is postponing sparse attention for 2 reasons: it wants real, measurable speedups, and sparse attention is not sufficiently compatible with pretraining. The interface can be kept for later.

18. The Ship of Theseus: Is this still a Transformer?

• 张小珺 observes that model training today appears to lack a major paradigm innovation; instead, it combines many existing ideas and searches for better outcomes. 孙宇涛 agrees, using the Ship of Theseus as an analogy.
• The original 2017 Transformer consisted of attention, residual connections and stacked layers, among other components. Some of those concepts were not entirely new even before Transformer. Over time, the components have been replaced one by one, until the resulting model bears little resemblance to the original Transformer.
• Yet the field continues to call it a Transformer. Whether that label still holds is both a technical and a philosophical question.

19. K3’s character: In the ResNet era, each individual piece might have been an architecture paper

• 孙宇涛 breaks K3’s integrated effort into several major components: attention design, MoE design, the optimizer and the structure of intermediate connections.
• In the ResNet era, any one of these pieces might have been enough to constitute an architecture. In the Transformer era, a single component may not qualify as an independent milestone. Building a working model now requires validating scaling, solving engineering problems, handling data and completing full training before the architecture earns broad recognition.
• From a research perspective, he is not especially interested in validating every component only as part of one integrated package. Architecture validation has strict standards and does not necessarily require bundling every component together. From an industry and project perspective, however, this kind of integrated effort is now necessary.

20. A contrarian LR schedule: WSD’s flexibility may be overstated, so K3 returned to cosine

• WSD was initially proposed by MiniCPM and later explored by work such as LLaSD. The observation was that the exact decay path from a high learning rate to a low one has limited impact on the final result, while loss falls faster during the cooldown phase. Better data can therefore be reserved for cooldown, linking the data schedule to the learning-rate schedule.
• Another advantage of WSD is that the learning rate stays flat in the early phase, giving training teams more flexibility to adjust the final token count. They can bring forward or delay a release without fixing the full training length from the outset.
• In practice, however, the optimal learning rate still depends on total token count: 6×10⁻⁴ may be better for a 10T-token run, while 3×10⁻⁴ may be more appropriate for 20T tokens. Arbitrarily extending the token count may therefore be less effective than expected, and WSD may be no easier to tune than cosine.
• K3 chose cosine decay because it has mainly 2 variables: training token count and maximum learning rate. WSD requires an additional decay-ratio parameter. From a tuning standpoint, cosine makes it easier to find a workable configuration.

21. Scaling efficiency: K3 is 2.5x K2, but that is an integrated result

• Relative to K2, K3 increases total parameters from roughly 1T to 2.8T, less than 3x, while active parameters rise from 32.6B to 104B, more than 3x.
• Kimi’s reported scaling-law results show K3 achieving 2.5x the scaling efficiency of K2. The comparison can be understood within the lineage of classic scaling-law work from OpenAI, Chinchilla and others.
• 孙宇涛 notes that K2 and K3 cannot have used exactly the same data, so their scaling behavior was probably not measured under fully fixed-data conditions. The final 2.5x reflects the combined effect of model architecture, training recipe and data strategy.

22. NoPE: A long-context solution for hybrid attention

• Traditional long-context models generally need to adjust RoPE parameters. A paper from the Cohere team proposed that, in hybrid attention, the linear-attention component already introduces positional information, allowing the full-attention component to remove RoPE and use NoPE.
• NoPE offers 2 benefits: better model performance and no need to retune architectural parameters when extending context length. 孙宇涛 reproduced the result soon after seeing it and found it both effective and elegant, giving the work a strong accept recommendation during review.
• He also argues that RoPE’s fundamental effect is a recency bias. It is effective for modeling short contexts, but does not help with long contexts and may even be harmful. RoPE does not create long-context capability; it may instead impair it.
• K3 removes RoPE directly during long-context training, so extrapolation does not require retuning parameters. Long-context performance remains sound and may even improve.

23. Why low-precision QAT is introduced during SFT

• DeepSeek-V3 used native FP8 training from the outset, while DeepSeek-V4 uses native W4A8 training. Kimi instead completes training at higher precision and introduces low-precision QAT during SFT.
• From a project-management perspective, higher precision is safer for large-scale training. Low precision can create additional issues at larger scale, and those issues are difficult to validate fully and precisely in advance.
• Technically, 孙宇涛’s own experiments suggest there is no need to introduce low precision from the scratch stage. As long as W4A8 is introduced after a sufficient amount of training, the final result may be broadly similar.
• Kimi has emphasized RL consistently from K1.5 through K2.5, and K3 inherits those relatively mature approaches, so this part was not discussed in much detail.

24. OPD: From white-box distillation to multi-teacher merging and “distilling yourself”

• OPD’s early lineage can be traced to work from the team led by 董立 at Microsoft Research Asia. MiniLLM took the opposite direction from traditional distillation: rather than having the Teacher generate an answer for the Student to learn through SFT or forward KL, the Student generates its own answer and the Teacher corrects it step by step.
• Intuitively, traditional distillation resembles a teacher lecturing while a student learns. Reverse KL is closer to a student solving a problem and writing out its reasoning, with the teacher pointing out where the intermediate steps go wrong. Thinking Machines Lab later named this paradigm On-Policy Distillation.
• Early OPD could distill a larger model into a smaller, faster one. Today, companies generally train a flagship model directly and serve it in production, so they do not necessarily train a large model first and then a small model. The more common use is therefore “distilling the model into itself.”
• From a non-technical perspective, OPD can simplify post-training team management. RL strategies and rewards for different specialties are highly heterogeneous; jointly training verifiable rewards, preference rewards and reward models is difficult. OPD converts different strategies into different teacher models and then merges them through multi-teacher distillation.
• RL strategies and data are highly heterogeneous, but the models used for different tasks generally have similar architectures and are easier to combine. 孙宇涛 says K3 was not the first to use this approach; teams including Xiaomi and JOM should also be using similar methods.

25. Draft models and DFlash: The MTP interface must be built into pretraining

• Xiaomi has had a setup that reached 1,000 TPS. Part of the gain came from work by Tianyi Liang’s team on fusing operators across different stages, which may deliver 300 to 400 TPS; TensorRT and DFlash speculative decoding then added further acceleration.
• This approach suits applications willing to pay more for higher per-user inference speed. The benefit is particularly visible at small batch sizes when the goal is high single-user throughput.
• Early speculative decoding used a relatively independent small model to predict the large model, which failed to make full use of the large model’s intermediate hidden states. MTP and Eagle-3 both seek to use those hidden states, allowing a smaller draft model or a higher acceptance rate at the same parameter count.
• DFlash connects the traditional autoregressive language-model approach with diffusion language models. Diffusion language models can infer faster for single users or small batches, but are less efficient to train from scratch and do not necessarily lead on aggregate multi-user throughput. DFlash uses the same inference pattern to further optimize MTP or the draft model and improve single-user throughput.
• MTP remains tightly linked to pretraining. Pretraining must provide MTP with an interface useful to the downstream draft model, followed by fine-tuning and specialization. One example is replacing the conventional next-token prediction loss with KL loss so the draft model aligns more closely when the main model verifies it. 孙宇涛 does not view this as the endpoint for MTP.

26. KDA’s CP: Two-level recurrence enabled by arbitrarily divisible chunks

• For sliding-window or full attention, CP can be implemented through ring attention or zigzag attention, and the computation pattern is relatively intuitive. CP for Linear Attention requires returning to chunkwise recurrence.
• In linear attention, computation is parallel within a chunk and recurrent across chunks. At the single-machine kernel level, computation can be divided into smaller tiles of 16 or 64 tokens. KDA’s decay constraint is primarily intended to improve parallelism within a chunk.
• A key feature of linear attention is that chunks can be divided arbitrarily, with different chunk sizes remaining mathematically equivalent—for example, a chunk of 512 tokens or one of 8K tokens. This differs from concatenated approaches that use Linear Attention between chunks and Full Attention inside each chunk, where chunk size is part of the architecture.
• This allows a 2-level chunk structure: GPUs can perform recurrent computation across large 8K-token chunks, while each machine further divides them into 16-token tiles for parallel computation. Larger chunks can continue to use a chunk-within-chunk structure. The flexibility comes from the chunkwise recurrent nature of pure Linear Attention.

27. MoE EP: Card-level balancing through limited expert redundancy and online planning

• Dropless EP has 2 problems: different GPUs receive different numbers of tokens, leading to different execution times and idle waiting; and the unknown token count requires extra communication to tell each card how many tokens it will receive before dispatching them.
• From a pure infrastructure perspective, EP with token dropping can use global allocation to give each expert an even token count. The impact of dropped tokens on model loss and capability, however, remains significant, so this approach is generally not used in practice.
• Kimi proposes dynamic EP: introduce a small number of redundant experts and use online planning to determine in advance which redundant experts each card should host, making total tokens per card exactly equal. 孙宇涛 believes it is mathematically possible to show that a small proportion of redundant experts is sufficient for card-level balance.
• Card-level token balance is not the same as expert-level balance. A card can host multiple experts, and there is no guarantee that each expert receives the same number of tokens. Achieving perfect expert-level balance would require a lossy strategy. The trade-off between model capability and infrastructure remains an open area for further work.

28. Memory and overlap: Replacing complex communication optimization with simpler paths

• From a software-engineering perspective, K3 makes a more granular decision for each component about whether to retain activations or offload them, and offloads a large amount of intermediate computation to the CPU. To prevent offloading from slowing training, CPU-GPU communication must overlap fully with computation; the exact boundary depends on the cluster environment and model configuration.
• Offloaded data can be stored in FP8, cutting storage requirements by half and potentially crossing critical memory-capacity or communication-bandwidth thresholds.
• DeepSeek-V3 uses a more complex form of MoE overlap: it reorders the MoE forward and backward passes for one batch alongside the next batch’s MoE forward pass, hiding dispatch and combine communication. DualPipe is also used to mitigate PP bubbles and activation imbalance.
• Because Latent MoE sharply reduces communication volume, K3 does not need to bring another batch into the overlap. It only overlaps MoE forward and backward communication with shared-expert computation within the same batch. Once communication is small enough, the shared expert itself may cover the communication time.
• On PP, K3 does not use a fully symmetric DualPipe path. Instead, it moves memory-intensive content from high-memory-pressure PP ranks to ranks with more available memory, addressing imbalance through a lighter-weight path.
• Muon also creates a problem for optimizer sharding: Newton-Schulz iterations require the full matrix. Adam is element-wise, so sharding flattened parameters generally does not affect results. Muon requires special handling to shard effectively at the optimizer level.

29. Multimodal infrastructure: Each added capability raises complexity sharply

• Vision Encoders typically compress visual tokens before passing them into the language model—for example, from 32K input tokens to 8K tokens in the backbone. The Encoder still has to process the longer uncompressed sequence, however, so it may require CP even when the language-model backbone does not.
• Native VL training also creates inter-card imbalance. Some cards may handle almost entirely text tokens, while others have a high share of visual tokens, affecting both compute distribution and PP.
• Placing the Vision Encoder at the beginning of PP would substantially increase the workload on the first PP rank and disrupt pipeline balance. K3 places the Vision Encoder in the middle or at the end, allowing idle PP capacity to precompute later Vision Encoder steps while the earlier ranks process shallow language-model layers, reducing bubbles.

30. Why the paper is so detailed: Reducing dependence on rumor

• 张小珺 notes that the paper is unusually detailed. 孙宇涛 explains that when an organization lacks sufficient execution density and cannot solve these problems itself, people become fixated on rumors and privately seek out engineering details. Publishing those details removes the need to rely on such information.
• He believes infrastructure is fundamentally deterministic and does not contain many mysterious difficulties. But proposing solutions independently requires analytical and profiling ability, as well as sufficient organizational density and team culture. Teams that cannot develop their own solution can still follow strong engineering practice; if they cannot even follow, the problem is more serious.

31. RL infrastructure and inference-engine integration: Shared gradient buffers and linear-attention caching

• RL requires large-scale sandbox management, and each trajectory may have its own Docker container, making container management difficult in its own right.
• A complete RL system generally includes a policy model, reference model, the policy’s gradients and optimizer, and potentially a reward model. K3 discusses ways to share memory across these models.
• The reference model has no gradients and does not need gradients when calculating the objective. Once gradients are available, the reference model may no longer be used. The gradient buffer can therefore be repurposed for forward passes by non-policy models, saving some GPU memory.
• Linear Attention on the inference side faces a prefix-cache problem. Full Attention has an incrementally updatable cache at every position, while Linear Attention overwrites the previous key-value state at every step, so the prefix state differs at every step.
• Saving all of these states would erase linear attention’s advantage; saving none would forfeit prefix caching. The current compromise is to divide the sequence into blocks and cache the prefix once per block.
• VLM paged KV also has to support different attention patterns. Full Attention, Linear Attention and sliding-window attention cannot all use the same token-by-token KV Cache management, requiring additional infrastructure and kernel configurations.

32. Overall assessment: 100B active parameters required conviction; science does not advance in leaps

• What impresses 孙宇涛 most is not any single technical point, but that K3 pushed active parameters all the way to 100B—larger than he expected. He views the exact active-parameter target as a non-technical question: what stage does the team want to reach, and does it have the conviction to push an open-source model into a new scale tier?
• On 杨植麟’s notion of “probabilistic non-consensus,” his view is that genuinely valuable technical innovations have already been written up as individual papers. What surprised him most when K3 launched was the size; the rest was broadly within expectations.
• He believes science does not produce step-function improvements. Technology advances incrementally through the work of companies, projects and the industry as a whole; the field simply labels certain points along that trajectory milestones.
• He approves of Kimi’s research process, seeing its scientific character as rooted largely in internal management. The team emphasizes model interpretability, seeks more reliable behavior traces and assigns explicit causes to instability or model collapse.
• He also believes Kimi’s culture does not depend on individual heroism. When 张小珺 said that people at Kimi did not seem afraid of 杨植麟, his response was that whoever has the better argument should be heard and whoever is right gets to decide. Team culture, he says, depends heavily on the leader’s taste and personality.

33. K3 and DeepSeek: Strategic divergence and V3’s historical position

• DeepSeek has released 2 tiers of models, including a 1.2T model and smaller models, with a greater emphasis overall on cost-performance. Its Flash model has also performed well.
• Kimi is currently more focused on raising the capability ceiling of open-source models, with cost-performance a secondary consideration. K3’s API is therefore quite expensive—much more expensive than competing models.
• On whether the 2 companies will continue to diverge, 孙宇涛 says there is not much technical debate: good is good and bad is bad. The direction of each organization, however, depends on the choices made by its people.
• He sees DeepSeek-V3 and R1 as icing on the cake in one sense. V3’s biggest historical significance is that it was among the earlier domestic models to scale to a meaningful size and complete the full pipeline. Before V3, the biggest domestic leap in model scale may have been Qwen-72B.
• K3 increases active parameters by more than 3x and brings total scale close to 3x, putting it on another level. Increasing model size is not itself an innovation; turning up the numbers does not solve the problem. The real difficulty and innovation lie in making a larger model work.

34. The outlook: Defining capability, model scale and world models

• 孙宇涛 believes model size will unquestionably continue to increase, but cannot grow without limit. The amount of information humanity can accumulate on the internet is finite, so models have no need to expand indefinitely. Larger models may offer a broader capability bound and a higher ceiling, but they remain constrained by data volume.
• Whether a capability can be achieved depends on whether the task is clearly defined. Mathematics problems and AIME can be described precisely and optimized against directly. If a commercial use case defines a capability clearly, it can likewise be reached through sustained optimization.
• AGI’s biggest problem is that it is not well-defined. If AGI means embodied intelligence that must interact with the physical world, the gap remains enormous. If the discussion stays within the scope of language models, the problem is comparatively less difficult.
• A new team can theoretically rebuild a model if it has a strong enough team, culture and individual execution capability, but those conditions are unlikely to emerge simultaneously. The simultaneous formation of strong people and a strong organizational culture is historically contingent; it cannot be achieved simply by deciding to do so.
• He has begun exploring world models because he believes large models may have limited room for further improvement and little new personal credit left to claim. World models pose a larger problem, but at least they represent a new field.