AI Inference Cost Optimization

Oct 27, 2026

SemiAnalysis Ep. 041 - EMERGENCY: Inside the Anthropic’s S-1 (Tokenomics/ Markets)

• SemiAnalysis Anthropic’s disclosed economics challenge the idea that frontier inference depends on subsidies. Joey cites two profitable quarters, margins above 10%, 85% inference gross margins and 41% compute margins. Jeremie highlights 90% gross …
• Compute efficiency strengthens Anthropic’s economics, although the panel disputes how much credit belongs to the company. Joey cites inference revenue per megawatt rising from $16 million to over $60 million. Doug asks how much reflects Anthropic versus broader …

TBPN Gavin Baker on Anthropic's S-1, GPU Prices, and SpaceX

• Gavin Baker Training demand creates a potential inference-supply risk within Baker’s bullish framework. A bearish argument he considers compelling is that Anthropic dedicates 74% of its compute to training. That suggests room for more inference …

Oct 26, 2026

Delphi Digital Why OpenAI Should Be Scared of Open Source

• Tarun Chitra A low advertised inference price can conceal a higher total agent bill. In Tarun’s GLM 5.2 illustration, a nominal 95% discount becomes $1.05 over 21 queries when caching savings disappear, versus just over $1 …

Oct 24, 2026

20VC 20VC: Unconventional's Naveen Rao on robotics and $10TRN inference

• Naveen Rao Rao forecasts a $10 trillion inference market within 10 years, while expecting pressure on inference operators’ margins. Falling token prices squeeze the layer mapping models onto existing hardware; improving energy-to-intelligence conversion offers more room …

Sourcery AI Agents Could Drive 10,000x More Web Activity | Former Twitter CEO

• Parag Agrawal Parallel sells search quality, model independence and lower inference costs. The introduction reports a $100 million Series B at a $2 billion valuation, while Agrawal says headcount grew from close to 30 to about 120 …

Oct 23, 2026

The a16z Show Amazon CEO: $220 Billion Still Can’t Meet AI Demand

• Andy Jassy Jassy sees enterprise production applications becoming AI’s largest absolute usage segment, with inference cost a major adoption hurdle. He thinks quite a few, if not most, existing enterprise applications will use inference pervasively when viewed five to seven years out, …

Oct 22, 2026

20VC 20VC: Trillion or Bust, CoreWeave’s Sell-Off and the $90B Anduril Test

• 20VC Modal’s $750M raise at $15.75B exposes an inference opportunity the panel admits it underestimated. Its run rate exceeds $1B, against Rory’s recollection of less than $100M around mid-to-late last year. He contrasts roughly $5–6B across …

Latent Space Why AI Compute Comes in Rectangles — Erik Bernhardsson, Modal

• Latent Space A shift in the fine-tuning value proposition: he says open-source models such as Kimi K3, DeepSeek 4.1, and GLM-5.3 are now almost as good as frontier models at lower cost, often without fine-tuning. He also argues there is a structural role for inference providers separate from model labs: demand shifts with new releases, and Modal can …

Oct 21, 2026

168X 168X in Conversation with Jason Kam: Crypto & AI—How to Read This Cycle?

• Jason Kam The hardest AI inflection point to call is not token pricing, but how much revenue each GPU ultimately earns. Lower token prices could be offset by model efficiency, chip optimization and higher output. He admits that “nobody knows” whether demand …

Oct 20, 2026

Delphi Digital The Secret Supplier Feeding Frontier AI

• Andrej Radonjic Inference makes regional capacity and latency increasingly important. Andrej previously thought Live Context Retrieval needed ten times as many nodes, but infrastructure improvements enabled an earlier launch …

Oct 19, 2026

硅谷坐标 Silicon Valley Vector AI's Next Phase: 11 of Silicon Valley's Top AI Practitioners

• 贾扬清 Value migration is the episode’s central trade: Ed Chi says model capabilities are starting to commoditize, shifting value toward applications and token factories; long-term optimization should focus not just on capability but on capability over cost. 贾扬清 compares AI infrastructure to “masks during the pandemic”: demand surges squeeze the supply chain, creating an opportunity to improve …

硅谷坐标 Silicon Valley Vector From Compute to Scale: The Future of AI Infrastructure | A Conversation with Google VP of Engineering Bill Jia

• Bill Jia In response to the host’s concern that Google’s balance sheet is negative and CapEx is out of step with revenue, Bill pointed to 3 ways to cut costs: smarter models that use fewer FLOPs, software optimization across networking, memory caching and compute, and cheaper hardware. He believes continued optimization can ultimately deliver positive cash flow; as long as optimization continues, there is no need to fear …
• On startup opportunities, he highlighted devices and agents that can work efficiently at scale, and said he would specifically invest in agents that optimize inference. There are many N×M combinations of models and hardware; optimizing them one by one with people is “very, very unscalable,” while agents can …
• Bill expects inference to account for a larger share of compute over time; whether the split is 50/50 today is “hard to say,” and inference may already exceed training. If the split is 50/50 now, it could move to 40/60, 30/70, or 20/80 over time. Memory will remain scarce in the agentic era, while demand for …

硅谷坐标 Silicon Valley Vector Fireworks CEO乔琳：有PMF也可能破产，AI不是SaaS

• 乔琳 Amber One cuts costs by shortening chains of thought; the platform jointly optimizes training and inference. 曹卿云 says chains of thought can be cut in half, halving token usage and cost. He also says customized models can deliver bitwise-equivalent …

硅谷101 E256 | Agents, Robots, and Pharma: Which Will GPT-6 Astra Reshape First?

• 硅谷101 Astra 每百万 Token 的输入价格为 $10、输出价格为 $50，约为 GPT-6-4 的 4 到 5 倍，给消费级 Agent 带来很大的成本压力。 Bill 认为竞争者可能以开源模型打底，再做 Routing；他的愿景是让至少 60% 到 70% 的 Token 在本地或私有算力上产生，减少对 Frontier Model 的依赖。如果行业能烧上一两年，小模型成熟后或能逐渐形成平衡；而另一端的传统企业“还在纠结怎么脱离 …

The a16z Show Why a $1M Startup Wasn’t Worth Building in the Age of AI

• Jean-Denis Greze Cheap tokens expand the market into work nobody previously found economical to perform. Greze’s “token distribution machine” could monitor restaurant suppliers for about $1 a month, finding nothing useful in 11 months but saving …

Oct 18, 2026

硅谷坐标 Silicon Valley Vector DeepMind's Ed Chi: Future of Personalized General-Purpose Assistants

• Ed H. Chi（纪怀新） He argues that economics ultimately comes down to capability over cost, and that cost control could become the most important moat if model capabilities become widespread. Google’s advantage is its full stack—from models and applications to hyperscaling, token factories, chips, and energy—and its ability to …

Oct 17, 2026

20VC LangChain's Chase: vertical agents win, routing lives in the harness

• Harrison Chase His biggest change of mind: "the market for inference is massive" — passing on building a gateway a year and a half ago was "100% the wrong decision." He invokes Eric Vishria's broader observation that "everything just works" and the market is big, and recalls Thomas Tunks's point about …

Oct 16, 2026

Pitch The PM EP55 Token Prices Dropping on Model Shifts with Carmen Li, CEO of Silicon Data & Compute Exchange

• Doug Li attributes falling token prices to customers shifting their model mix toward cheaper alternatives. Silicon Data tracks consumption across 300 open-source/open-weight models plus closed models, observing “people shifting from premium models …

Oct 15, 2026

20VC 20VC: Valar’s Pro-Rata Fight, Meta’s Muse and NVIDIA’s Bet

• 20VC The first practical AI deflation may have arrived with Haiku 5.5, Sonnet 5.5 and Jev—but a 90% price cut requires at least 10 times the usage merely to preserve revenue. Jason described Haiku 5.5 as nearly Sonnet-quality for many jobs at one-tenth or one-twentieth the cost, while Sonnet 5.5 is about 10% …

Oct 14, 2026

SemiAnalysis Ep. 038 - Is GLM 5.3 a Mythos-Class Cyber Model From China? (InferenceX)

• Wenyao Gao AMD’s MI355 challenged NVIDIA’s serving economics within weeks of GLM-5.3’s launch. At 150 tokens per second per user, Wenyao reports costs roughly matching B200; Jordan also highlights MI355 throughput at 100 tokens per …
• SemiAnalysis Ultra-low latency has different economics from maximum aggregate throughput. TailRT reaches up to 470 tokens per second at batch size 1 through a decode megakernel, but cannot serve large batches. Kimbo calls memory …

Oct 12, 2026

20VC Atlassian’s Cannon-Brookes: Agents Won’t Replace Interfaces

• Mike Cannon-Brookes Cannon-Brookes has revised downward his estimate of future inference prices, although lower unit costs will drive much more usage. Capital, talent, chips, data-center design and distillation should push price-performance down “at a precipitous rate,” but he rejects …

Sourcery Perplexity CEO: SMBs Could Drive the Next Economic Boom

• Aravind Srinivas Perplexity is betting that long-context agents will make local compute economically and strategically important alongside the cloud. Chat can cost 30X more locally, but around 100K context the data-center advantage may narrow to 3X or nearly 1X; meanwhile, 100 million …

The a16z Show Recursive Self-Improvement and the Future of AI

• The a16z Show Their commercial wedge is inference efficiency, an AI problem they believe can improve both performance and cost. Faster inference also lets a system do more inference; the founders say their multi-LLM agent can choose what to optimize, and even its …

Oct 11, 2026

十字路口Crossing “With AI, I Feel Terrifyingly Powerful!” | A Conversation with 鸭哥

• 鸭哥 Cost declines are unlocking use cases that once seemed uneconomical: his tour-guide project burns about 4B tokens a day, which would cost an estimated $12,000 on Sonnet versus just $300 on DeepSeek Flash. An open-source setup using Qwen reads logits without fine-tuning or training, scoring 85% against GEV’s 88%. 鸭哥 sees plenty of potential for …

Oct 9, 2026

No Priors Beam: The Great American Open Model with ReflectionAI Co-Founder and CEO Misha Laskin

• Misha Laskin Beam’s economic pitch is reasoning efficiency: comparable work in fewer tokens, less time, and therefore lower inference cost. The 500-billion-parameter model has 23B active parameters; pre-training used 6,000 GB300s and can now run in about 12 days, while …

SemiAnalysis Ep. 037 - Who's Funding the $11 Trillion AI Buildout? (Capital Markets)

• Dan Nishball The willingness to finance AI rests on unusually strong economics throughout much of the stack. SemiAnalysis models up to $100 million of annual frontier-token revenue per megawatt against roughly $40 million to install GB300 capacity, …

Oct 7, 2026

SemiAnalysis Ep. 036 - $200 Buys $12,000 of Opus Tokens, We Bought Every Plan (Tokenomics)

• Max Kan Anthropic’s $200 subscription can deliver roughly $12,000 of usage at API prices without necessarily losing money. Max cites a consensus forecast of more than 90% API gross margins; if the average subscriber consumes only about 10% of the maximum …
• SemiAnalysis A subscription’s value cannot be reduced to one dollar figure because each model and token type consumes plan credits differently. Fresh input, cache writes, cache reads, and output carry different weights, so the same $200 plan can produce radically different …
• Caching is the decisive economic lever in agentic inference. The tested agent workload was roughly 96% cache reads; those reads can cost 0.1x or less than fresh input because they retrieve an existing …
• Andrew Megalaa Anthropic offered materially more usable tokens than OpenAI in the speakers’ like-for-like $200 comparison. They measured about 1.9 billion Astra tokens versus 3.1 billion Fable 5.1 tokens, with Fable consuming only 50% of the Anthropic allowance …
• Max Kan Realized subscription margins depend on ordinary users consuming nowhere near the published maximum. Max believes Anthropic utilization “cannot be much higher than 10%” if leaked financial figures are to reconcile, implying approximately …

Oct 3, 2026

20VC Crusoe CEO: Why Everyone Gets GPU Depreciation & AI Energy Costs Wrong

• Chase Lochmiller The real inference race is not simply dollars per token, because tokens differ in value and customers also optimize latency, throughput, and utilization. Since GPUs are “the most valuable thing in the entire data center,” idle time is burning money; KV-cache orchestration across HBM, DRAM, …

Frictionless Why Memory Is AI's Biggest Bottleneck with Vikram Sekar | EP 170

• Vikram Sekar Consumer agents should expand inference demand, but not on a permanently dedicated-processor curve implied by the Mac mini craze. Most personal tasks can run on Sonnet-level or open-source models, while processors and memory are shared or spun up just in time. Demand …
• The memory hierarchy should be optimized for useful work per token, not maximum tokens per second everywhere. A coding task where time-to-market matters may justify 1,000–2,000 tokens per second and expensive bandwidth, while an agent reading PDFs in …

Oct 2, 2026

SemiAnalysis Ep. 034 - The Fight for Fast Tokens, TPU v7, Vera Rubin, and Engrams (AI Supply Chain, InferenceX)

• SemiAnalysis AgentX shows why KV-cache management becomes decisive in real coding-agent workloads. Replayed sessions captured through SemiAnalysis’s internal proxy have a P50 request near 100,000 input tokens and theoretical cache-hit …
• Current inference economics can make the newest systems look like “money printers,” provided the workload and pricing assumptions hold. The team’s calculator combines hardware, power, datacenter cost, throughput, concurrency, and OpenRouter pricing; scenarios discussed …
• The debate over fast tokens is unresolved because agent workflows contain both latency-sensitive and latency-insensitive stages. Bryan, Cam, and Alec see limited ROI when several agents run in parallel or 30-second tool calls dominate, especially at prices approaching …

Empire Everyone Is Watching AI While Crypto’s Bull Market Builds | Weekly Roundup

• Santiago Roel Santos Falling inference costs could strand capacity—or unlock enough usage to validate Jevons’ paradox. Santiago was unsure whether the model discussed was Jev or Type Safe; for Jev, he said it was 238 times cheaper than Fable, 70 times cheaper …

晚点聊 LateTalk 183: AI Quarterly 26Q3 with Henry: Muse Ignites Personal Assistants, Astra Enters Robotics, OpenAI Revenue Surges

• Henry Yin The collapse in intelligence costs has moved from model leaderboards into product P&Ls, with comparable capability tiers seeing roughly 22x price cuts within months. DeepSeek V4.1 Flash, GLM-5.3 Flash, GPT-6 Luna, and MiMo 2.6 are moving high-frequency agent calls from uneconomic to viable; Henry’s own …

Oct 1, 2026

高能量 Vol.236 Macro Talk 111 | China-U.S. Summit Talks, AI's Limits, and the Global “Reservoir” Logic of Capital

• 高能量 Three to 4 years after ChatGPT’s launch, there is still no second mass-market AI application, suggesting the real constraint may be cost rather than model capability. Every conversation, inference, and long context window adds cost, while the value of ordinary search, summarization, and document generation …

Sep 30, 2026

The a16z Show AI, Infrastructure, and the Next Investment Cycle

• The a16z Show Falling inference costs are making multi-step agents economically viable and potentially more profitable for application companies. Agent-token usage on OpenRouter has grown 14-fold, Hebia’s financial-chat workloads became 10 times cheaper, Databricks’ router solved more …

Sep 25, 2026

Empire Markets Sending, Just Buy Consensus, Meta's AI Moment & Who Wins Tokenization? | Weekly Roundup

• Empire AI is collapsing application costs, but the blockchain analogy has limits. One side sees the same progression from scarce, expensive block space to abundant application capacity, citing models becoming roughly 50% …

Sep 23, 2026

SemiAnalysis Ep. 033 - ClusterMAX 3.0 Is Here! Neoclouds Ranked (Neoclouds, GPUs)

• Pratt Bhatt Inference endpoints are quietly a cost minefield: in preliminary Endpoint X testing, one "very famous provider" delivered a 75% cache hit rate versus 99% for good peers — and that gap alone roughly doubled the total bill for the identical model and accuracy. Pratt Bhatt's takeaway: you cannot "just go out, install vLLM... and assume to make 80% margins"; cache-aware routing and custom kernels are …

Sep 21, 2026

Frictionless The Bottleneck Isn’t the Chip with Bubble boi | EP 166

• Bubble boi The commercially important optimization is not maximum tokens per second in isolation, but more users per unit of hardware. Smarter caching, flash offload, prefill/decode disaggregation, CXL-attached memory, and context-aware controllers could, hypothetically, …