Silicon Valley Coordinates x TensorMesh 江鋆晨: AI’s Memory—A Three-Layer Understanding of KV Cache

Summary

• 江鋆晨 lays out the episode’s core framework: KV Cache has three layers of understanding, and most of the industry is still stuck at the first. The first treats it as reusable “black-box data” that can be stored; the second treats it as a white box containing semantic (attention) information, enabling lossy compression, non-prefix reuse, and cross-model reuse; the third directly changes the semantics—“pay more attention to this, pay less attention to that”—and can even improve output accuracy. “Very few people in industry understand this layer,” which is precisely why TensorMesh was founded.
• He thinks the “KV Cache moment” may arrive sooner than expected—“possibly before the end of this year,” with a wave of companies entering the space. He is not intimidated by Big Tech: the advantage is cross-disciplinary. Infra engineers who can work with GPU data cannot handle layers 2 and 3, while ML researchers who understand semantics lack the engineering capability required for layer 1. He also cites Sam Altman’s view that large companies have “people’s attention spread across too many places,” meaning a team might work on something for 2 months before moving on.
• He dismantles the pricing illusion that prefill is worthless because input tokens are cheap: the true compute cost per input token is roughly the same as for an output token, while pricing does not reflect cost. Prefill is highly parallel and, in his example, accounts for roughly one-tenth of the user’s wait time. But in real use cases, inputs running to tens of thousands of tokens are often more than 10x longer than outputs of a few thousand tokens. Because agent applications are stateless, input will only keep getting longer. That is the foundation of TensorMesh’s cost-reduction logic.
• The economic logic is “trade storage for compute,” but TensorMesh is building software on top of hardware, not buying hardware. When KV Cache sits on CPU, local SSD, or GDS, retrieving it is both cheaper than recomputing it and faster; only colder storage layers create a cost-versus-latency trade-off. He also stresses that the company is decoupled from the storage-hardware cycle: price spikes reflect a supply-demand imbalance, with orders booked 2 years out, and should ease over the long term. The calculation customers should make is how much they spend on hardware versus software.
• He rejects the TAM question outright: “total addressable market is a misleading word,” because most people do not yet realize this market exists. With the model and prompt fixed, helping the model understand the prompt better is a “third path” to improving agent quality. The best use case is enterprise shared knowledge—codebases, policy and legal documents—reused across coding, chatbot, and RAG applications.
• Differentiation rests on ecosystem and technology: LMCache has the best ecosystem support among comparable projects, while CacheBlend solves non-prefix reuse—KV Cache storage “is not merely Storage; it is a Service” with compute inside. Asked about similar projects mentioned by investors, he simply says, “They really are similar, good luck.” His response to architecture risk is that KV Cache is only a temporary name; fundamentally it is model-native data, and it will remain relevant in the Mamba era—Qwen 3.5, for example, still retains roughly one-quarter Transformer layers.
• Two analogies are worth remembering: KV Cache is “big data for the AI era,” and TensorMesh wants to build the Databricks for “data only models can read,” with Ion Stoica as an adviser; in distribution terms, it is the CDN of its era—“Akamai was OpenAI back then.” The company is currently building a mini-CDN inside the data center. Once edge computing and distributed inference take off, it could become an internet-scale knowledge delivery network: “We may have been looking at 6 years into the future 3 years ago.”
• He bets the foundation-model endgame looks more like online video than search: a handful of companies consolidate, but sovereign AI and enterprise private deployments will support many independent services. His open-source view is blunt: OpenAI and Google’s open-source models “will find it difficult to compete with these Chinese open-source models,” and the open-source market may still be China’s to watch over the next few years. xAI’s acquisition of Cursor is one example of the consolidation wave, though he says he does not know much about that specific case.

Deep dive

1. The Bottleneck Has Moved: Agentic Long Context Puts Resources Out of Balance with Workloads

• 江鋆晨 opens with a clear diagnosis: “AI is no longer the AI of 2 years ago.” Beyond the sharp increase in workload, prompt patterns have changed. Agentic use cases create long context; the longer the input, the greater the demands on GPU compute and internal storage, and the “memory” a model produces after processing that context also has to be retained. The fundamental bottleneck is the mismatch between resources and workload. Waiting for better GPUs is not enough; the software layer needs to innovate.
• TensorMesh’s position is memory management for large models. KV Cache is “a large pile of floating-point numbers” that stores the model’s understanding of context, but the model is stateless: “After seeing something once and producing an output once, it just throws those memories away.” TensorMesh and its open-source projects retain that memory and reuse it when similar inputs appear. The main target is the prefill phase, although excessively long decode sequences may also require remote or separate devices for storage.

2. Cheap Input Is a Pricing Illusion, Not a Cost Fact

• 曹卿云’s challenge is pointed: input tokens cost only a fraction of output tokens, which “sounds like this isn’t a startup that can create much value.” 江鋆晨’s answer separates cost from pricing. The FLOPS cost per input token and output token is “about the same,” and total prefill cost may even be higher.
• The illusion of cheap input comes from the user experience. Prefill runs with massive parallelism; in his example, it takes up only about one-tenth of the time the user spends staring at the screen, while “90% of the time is spent on output.” Output is generated sequentially and consumes more GPU time, giving providers a better rationale for charging more. Pricing reflects how providers balance their costs; it “doesn’t actually reflect the real cost.”
• The volume is also concentrated on the input side. In the real-world scenarios he cites, inputs run to tens of thousands of tokens while outputs are only a few thousand, making the former more than 10x longer. In agents and long conversations, the model is stateless and must be fed more context every time: “Input will only get longer, but output will not grow correspondingly.”

3. Three Layers of KV Cache Understanding—the Intellectual Core of the Episode

• The first layer treats KV Cache as cacheable compute state: a black box stored as is and reused. “A lot of companies in industry are doing this.” Large-model inference providers offering low-priced cached tokens validate the direction, but he says current prompt-caching use cases remain limited. In the open-source project and its users’ deployments, cached tokens are kept only on CPU; moving them to cheaper remote storage causes a substantial performance hit.
• The second layer recognizes that KV Cache carries semantics: it represents the model’s attention. “As long as you don’t change its semantics,” its values can be changed through lossy compression, non-prefix reuse, reuse across text that is different but semantically similar, and even cross-model reuse. “Academia has done a lot of work here; industry is doing very little.”
• The third layer is the most interesting. Model output depends on two things—weights and KV Cache—so changing the cache can change the output. “You can tell the model, pay more attention to this, pay less attention to that,” and the operation “can even make the model’s output more accurate.” Academia has begun exploring this; very few people in industry understand it.
• Why is the frontier still in academia? When engineers see data in a system, their first instinct is not to modify it, but to store it as a black box. Researchers ask what semantic information the data contains. “Something similar happened during the big-data era.” In the AI era, the most important and interesting data may be KV Cache.

4. Why Big Tech Is Slow to Replicate: Cross-Layer Talent and Scarce Attention

• He acknowledges that competition may arrive early: he originally thought it would happen next year or the year after, but now it “could happen before the end of this year.” Covering all 3 layers is extremely difficult. Layer 1 requires an infra or backend engineer who can reach the GPU, but may not understand the meaning of the data. Layers 2 and 3 require researchers, who lack the engineering insight and access to the real KV Cache system. Large companies have all 3 types of people, but struggle to put them together into one complete system.
• He closes with Sam Altman’s view: even with the best vision, a large-company CEO “may not be able to do it.” The issue is not capability, but that “people’s attention is spread across too many places.” A Big Tech group might work on the problem, then do something else 2 months later. TensorMesh, by contrast, is focused on just this one thing—even to the point that an internal Big Tech project was justified as: “TensorMesh raised so much money and is only doing this? We should do it too.”

5. Customer Profile: Shared-Knowledge Reuse Plus the “Third Path” to Better Quality

• One revealing admission: the team cannot see who is using its open-source project. They ask many companies whether they want to try it, only to hear: “Oh, we’ve been using it all along.” Based on those conversations, the best use case is enterprise shared knowledge: a shared codebase for every coding agent, policy documents for an internal chatbot, and documents in legal-services applications reused across products. “Having the LLM read the same thing over and over is obviously wasteful.” Best practices should be stored in a model-native format rather than as human-readable text, so the model can read them directly.
• The more agents iterate, the better the setup works for TensorMesh. An agent is fundamentally “a logic that interacts with its environment iteratively.” More interaction creates more history, and the context keeps getting longer.
• The value is not limited to cost reduction; it can also improve quality. KV Cache is generated unidirectionally and read once. Long text creates the “lost in the middle” problem: information at the beginning or end is retrieved relatively well, while information in the middle is not—“like a person reading a long book only once.” One vision is to optimize the cache at the storage layer, letting the model reread the important parts of the text and raise their attention, so subsequent reuse produces a more accurate output.
• His answer on TAM preserves the original bluntness: “Total addressable market is a misleading word, because the premise of ‘addressable’ is that people are thinking about it, and most people are not.” Once the model and prompt are fixed, helping the model understand the prompt better is a “third path” to improving quality. Very few people are focused on it today.

6. Competitive Differentiation: LMCache’s Ecosystem and CacheBlend

• His self-assessment is candidly tiered. Pure KV Cache storage technology “really isn’t that different from others.” But when it comes to having a real artifact that people can use smoothly, with multiple companies maintaining it in open source, “I’m confident that no other company can do ecosystem support as well.” TensorMesh was first to research KV Cache system optimization, first to open-source the work, and among the first to launch a commercial product. Asked about similar projects cited by investors, he says: “They really are similar, good luck.”
• CacheBlend is the technology he believes may be unique. Traditional KV Cache supports only prefix reuse; if the reusable text appears in the middle of the input, the cache must be updated before reuse, and updating requires compute. “This KV Cache Storage is not merely Storage; it is a Service. There is smart intelligence inside.” The storage system must therefore include compute—it genuinely requires understanding both systems and ML.

7. Multimodality Pushes Token Volumes into Another Order of Magnitude

• Images and video are still tokens to a model, just a “much longer sequence of tokens.” A video lasting several tens of seconds may generate as many tokens as the text in a hefty book. Multimedia also contains substantial semantic redundancy, leaving enormous room for compression. One research idea is to convert the media into text first and then feed it to the model.
• The definition of reuse will also have to be rebuilt. The same video in 480p and 1080p “should be one input,” but after tokenization the two inputs may look completely different. Optimization techniques will continue to emerge, preventing storage-hardware supply constraints from materially determining outcomes, while demand and resource requirements keep rising.

8. The Economics: Storage for Compute, Storage Cycles, and Hot/Cold Tiers

• The classic trade-off has a counterintuitive feature in LLMs. When KV Cache is stored on CPU, local SSD, or GDS (GPUDirect Storage), retrieving it is faster than recomputing it—“it lowers your cost and makes you faster at the same time.” Colder, more remote storage sacrifices latency, but “the cost-reduction benefit is still there.” Compression also reduces KV Cache size, so as long as network speed is not too slow, storing KV Cache is generally beneficial.
• He attributes higher storage prices to a supply-demand imbalance. Orders are booked out 2 years, so prices “won’t come down that quickly,” but they will decline over the long term. The real calculation is the split between hardware and software spending: “Rather than spending more money buying more GPU memory, ask whether software can do something similar.” He also leaves the outcome open. AI usage patterns differ from the cloud and big-data eras; RL training, for example, can turn one model into 30 versions and potentially drive GPU usage higher. “These are things we have to see.”
• Hot-versus-cold classification cannot be handled as a black box. Whether a KV Cache will be used in the next second or 10 minutes, or whether it is used today, tomorrow, and every day at 8 a.m., is difficult for the product to infer correctly on its own. Operators need an interface to express their domain knowledge. “We do not want to build a transparent KV Cache system.”

9. The Best Analogy: A CDN for KV Cache, and a Databricks Mirror

• His historical reference point is the CDN around 2000: “Akamai was OpenAI back then.” Without it, loading a webpage took 10 seconds; with it, half a second. KV Cache needs a content delivery network, but models currently run in the same data center. “So we’ll first build a mini-CDN inside the data center”—today’s KV Cache Storage. As agents become more latency-sensitive and inference hardware moves closer to end users at the edge, the distance between models will grow, creating the need for an internet-scale distribution system—a knowledge delivery network. “We may have been looking at 6 years into the future 3 years ago. The fact that this doesn’t happen in 3 years doesn’t mean it won’t happen in 6.”
• The advisers are 张辉, a former CMU professor and Conviva CEO, and Ion Stoica, a Berkeley professor, Databricks/Anyscale founder, and whose group built SGLang and vLLM. Stoica initially “didn’t understand what I was talking about,” then realized it was a data problem, distinct from the compute problem addressed by vLLM and SGLang. The dividing line is simple: “Databricks works on enterprise-grade human-readable data; we work on data that only models can see. If there were a world made entirely of AI models, maybe we would be doing Databricks’s job.”

10. Token Maxing, the Model Landscape, and the Gasoline Close

• His criticism of token maxing is unsparing. Agent development is currently in a “wild-growth phase”: everything the model has seen or generated is stuffed back into the context and read again. “This is an extremely, extremely inefficient method.” Many things cannot be compressed and can only be reused, and reuse does not have to be limited to prefixes—which is why CacheBlend has attracted particular interest from many companies. Efficiency will not shrink the market. Citing Jevons’ paradox, he argues that higher efficiency will only increase demand, and token consumption will continue rising.
• The foundation-model endgame has 2 possible shapes: search, where Google dominates, or online video, where YouTube and Netflix coexist with many other healthy services such as HBO and Hulu. He bets on the latter. A handful of companies will consolidate, but sovereign AI and enterprise users with strict commercial-secrecy requirements “will always need a software system they can control themselves.” His open-source view is blunt: from DeepSeek in 2025 to 智谱 GLM in 2026, OpenAI and Google’s open-source models “will find it difficult, more or less, to compete with these Chinese open-source models.” Unless something changes significantly, open source “may still be a China story” over the next few years.
• The architecture risk is manageable because KV Cache “is actually a temporary name.” Fundamentally, it is model-native data: intermediate states generated during inference, which also exist in Mamba, diffusion models, and CNNs. Mamba’s intermediate state is a linear state not directly tied to text length, but the industry consensus is a hybrid architecture. In Qwen 3.5, for example, most layers are Mamba, while roughly one-quarter remain Transformer layers with full attention.
• The closing analogy picks up the hosts’ image of KV Cache as gasoline, an oil-refining byproduct treated as waste before the internal-combustion engine arrived. 江鋆晨’s response: “People who understand KV Cache cannot access it, and people who can access KV Cache do not understand KV Cache. The day a company truly extracts this value, waste will become treasure.” When will the internal-combustion-engine moment arrive? “That depends on the development of our company.”