AI After Entering Production: Where Will Intelligence, Compute, and Chips Go?
Summary
Agents are shifting AI’s unit of delivery from an “answer” to a completed task. The industry consensus has moved from looking at the Model alone to “Agent = Model + Harness”: the Model sets the ceiling for intelligence, while the Harness handles context, routing, tools, memory, execution, security, and evaluation. Enterprise competition is also shifting from whether to deploy AI to whether Agents can be operated at scale, securely, and under governance.
Token demand is not linear; it is the product of three multipliers: task count, tokens consumed per task, and multi-agent coordination. IDC expects task executions to rise from 280B in 2026 to 415T in 2030, while complex tasks push per-task consumption from dozens of tokens to hundreds of thousands; its forecast puts the compound annual growth rate of global token consumption at 4822.6% over the same period. “What everyone can see is Agent applications; what they cannot see is the massive expansion in demand for underlying compute infrastructure.”
The compute supply-demand gap may widen for years, so it is a mistake to assume the shortage will disappear once infrastructure comes online in 2028. IDC’s compute-fulfillment rate falls from 74% in 2026 to 71% in 2027, then recovers to roughly 75%-77%, but the absolute shortfall could still reach $380.9B in 2030. Expansion plans, power constraints, higher memory prices, and shorter repricing cycles all point to the “limits of the physical world becoming the norm.”
The next generation of intelligence cannot be built by scaling data and compute alone; it must also solve few-shot reasoning, continual learning, and catastrophic forgetting. Zheng Qinghua uses the human brain—approximately 86B neurons, 20-23 watts of power consumption, and roughly 4% of neurons activated synchronously—as a reference point, and proposes moving from large models toward “machine-memory intelligence.” The goal is to balance timeliness and open-ended generalization within limited compute, storage, data, and time, approaching the impossible trinity of finiteness, timeliness, and openness.
The core benchmark for production-grade AI will shift from static scores to end-to-end reliability and cost. Jiang Peihang proposes expanding the evaluation target from individual models to system-level Agents, replacing static tests with dynamic continuous testing, and extending result scores to execution traces, permission calls, exception recovery, inference cost, and TCO. “Whether each token earns its cost” will be assessed alongside accuracy, hallucinations, and robustness.
Intelligent-computing systems will split into two tracks: pushing the frontier of intelligence and expanding the supply of intelligence. Capability compute targets the low latency, long-duration stability, and larger effective compute domains required by 10T-parameter models; capacity compute targets commercial inference, using heterogeneous coordination, dynamic scheduling, and “token factories” to raise output per dollar invested. Inspur Information cites a token-generation latency of 5.85 milliseconds and a 10x increase in token output at the same investment.
Chip value will depend not only on process node and peak compute, but also on data movement and software fit—the two bottlenecks between silicon and production capacity. Wu Huaqiang’s compute-in-memory approach lowers bandwidth and power consumption by putting computation where the data is; a 28nm trial lifted compute from 150 TOPS to 600 TOPS. FlagOS aims to become a “universal socket” for multi-chip systems, allowing a single model to support 12 chips within a day. In production, the real question is: “Who can get every task to completion?”
Deep dive
1. An Agent’s ceiling is set by the model, but its production capability is set by the Harness
庄明浩 distilled the conference consensus into “Agent = Model + Harness.” The Model’s variables include reasoning capability and context length, multimodality, tool use, cost, and task specialization, making model routing a dynamic choice based on task type, user intent, and workload.
The Harness organizes model output into an executable workflow through context management, model routing, tool connections, state and memory, execution control, security, observability, and evaluation. It makes intelligence runnable, governable, and recoverable rather than leaving it as a one-off answer.
IDC divides Agents into five stages: functional AI, personal Agents, single-workflow Agents, multi-Agent collaboration, and supervised closed loops. 庄明浩 believes the industry is moving from stage 2 to stage 3, while frontier vendors are already testing cross-role collaboration and continuous operation.
Professional-services Work Agents and software and IT Coding Agents remain the most mature categories. Education, research, healthcare, finance, energy, manufacturing, and retail have not reached the same level. His shorthand for the industry’s 2026 consensus is “Work Agent plus Coding Agent.”
2. Three demand multipliers turn compute shortages into a structural problem
IDC expects AI to create 22.5T in cumulative global economic value by 2031, with global active Agents reaching 2.216B by 2030. 庄明浩 uses global GDP of roughly 126T as a reference point and, with the caveat “if I remember correctly,” says 22.5T would be close to one-sixth of it.
The first multiplier is task count, rising from 280B in 2026 to 415T in 2030. The second is complexity: per-task consumption can rise from dozens of tokens to hundreds of thousands, taking workloads well beyond the scale of simple question-and-answer interactions.
The third multiplier is multi-Agent coordination. A single request is broken into multiple parallel subtasks, while Agents must pass context, consolidate results, and maintain long-running tasks. Multiplied together, these drivers produce IDC’s forecast of 4822.6% compound annual growth in global token consumption from 2026 to 2030.
The share of demand met by compute is expected to fall from 74% in 2026 to 71% in 2027, recover to 75% in 2028, and reach roughly 76%-77% in 2029-2030. The improvement in percentage terms does not mean the absolute gap is narrowing: the 2030 shortfall could still reach $380.9B.
3. Investment expansion, price signals, and power constraints all point to a prolonged tight balance
An IDC survey found that 97.3% of enterprises intend to increase investment, with 37.7% planning to raise inference spending. Pricing cycles have already shortened to under 6 months, and 庄明浩 observes that the most constrained memory chips are being repriced quarterly or even monthly.
He believes annual increases of 30%-50% may only represent the average, with some core segments rising far more. The bottleneck is not just capacity but also energy and power. “The limits of the physical world will become the norm,” making it unsafe to extrapolate a supply-demand inflection point simply from new data centers or higher chip output.
The real demand-side shift is from one-off calls to continuous coordination. AI is moving from a content-generation tool to an operating unit capable of independently taking on production tasks, elevating compute to the level of critical national infrastructure. IDC also identifies China’s lead in Agent deployment penetration as a major trend.
4. Machine-memory intelligence seeks to move beyond pure Scaling
Zheng Qinghua’s core argument is that the next breakthrough in intelligence “cannot rely only on expanding data and compute.” Systems must continue learning and reasoning with limited knowledge and resources in constantly changing environments, combining data-driven methods with knowledge guidance.
Existing large models face high compute consumption, heavy data dependence, limited human-like abstraction, black-box opacity, catastrophic forgetting when learning new tasks, and difficulty performing complex reasoning from few examples. These constraints directly limit reliable deployment in critical and fast-changing settings.
The human brain offers a different reference point: approximately 86B neurons, power consumption of just 20-23 watts, and an average of roughly 4% of neurons activated synchronously to solve many problems. It can also correct errors amid noise, recover memories through association, and switch flexibly between states.
“Machine-memory intelligence” aims to structure and compress knowledge, using sparsity, coordination, and association to produce stronger reasoning. It must simultaneously handle finite resources, time-bounded output, and the dynamic absorption and generalization of new knowledge rather than simply increasing parameter count.
5. Reliability evaluation must expand from model scores to the full execution trace
Jiang Peihang argues that strong point metrics do not prove a system is usable. Government, industrial, and consumer applications have different requirements for latency, energy consumption, stability, and security, while completing a multi-step task once in a pilot does not mean the system can keep doing so under high concurrency in production.
The new framework requires five shifts: the evaluation target moves from a single model to a system-level Agent; the method moves from static tests to dynamic continuous testing; the boundary expands from the model to the full technology stack; the focus moves from outcomes alone to the entire execution trace; and the yardstick adds value, cost, and ROI.
Evaluations must cover tool calls, permission use, task iteration, and exception recovery, while incorporating inference cost and TCO. 庄明浩 summarizes the requirements as “provable, controllable, affordable, and reproducible,” with predictable behavior, detectable anomalies, and actionable remediation.
The framework spans chip compute, bandwidth, and energy efficiency; system interconnects; cluster fault recovery; framework operators; model hallucinations and robustness; data ownership and provenance; and application-level security exits. At the conference, the Fifth Electronics Research Institute of the Ministry of Industry and Information Technology, together with BAAI, Inspur, Moore Threads, and more than 20 other industry-chain organizations, formally released a reliability evaluation framework for the Agent era.
6. Capability compute asks how powerful intelligence can become
刘军 of Inspur Information defines the shift this way: “Agents make intelligence scalable for the first time, like compute and electricity.” Intelligent computing therefore splits into capability and capacity tracks: the former pushes the cognitive frontier, while the latter expands commercial inference and broad access.
Capability compute serves frontier-model development. 庄明浩 notes that open-source models such as Kimi K3 have already reached 2.8T, while the rumored Cloud “should” be around 10T. Once Agent runtimes extend from hours to days, any compute, communications, or software failure can bring down the entire long-running task.
The system requirements consequently become tight alignment with model evolution, ultra-low latency, and stable long-duration operation. In engineering terms, the system must “fit, run fast, and run steadily.” The critical metric is no longer just single-chip performance, but how large an “effective compute domain” a single Agent instance can use efficiently.
The corresponding architecture is the supernode: organizing more servers into a single, larger computer that provides a unified and stable compute domain for 10T-parameter models. This track addresses how powerful intelligence can ultimately run.
7. Capacity compute asks how much intelligence each dollar can produce
Capacity compute serves massive numbers of high-concurrency endpoints, with a strong focus on dynamic scheduling and minimizing idle capacity. Its governing principle is “compute defined by workload”: inference is not a fixed type of computation, and different chips should take on the loads they handle best.
Heterogeneous compute coordination, high-density workload structures, and inference clusters turn the system from “a computer” into a factory for producing intelligence—a token factory. This track is no longer about peak capability, but how much intelligence can be continuously delivered per unit of resource.
Inspur Information launched 元脑 SD200 Ultra and 元脑 HC2000 for the two tracks. The former claims to reduce token-generation latency to 5.85 milliseconds while supporting 10T-parameter models; the latter targets heterogeneous inference and can raise token output 10x at the same level of investment.
8. Compute-in-memory and an open software ecosystem determine whether chips become production capacity
Wu Huaqiang identifies the main bottleneck in traditional architectures as the repeated movement of data between processors and memory. High-frequency transfers consume both bandwidth and power. Compute-in-memory allows memory arrays to participate directly in computation: “where the data is, the computation is.”
The approach uses physical simulation to reduce data movement and seeks performance gains from mature process nodes. His team has moved from 130nm to 28nm and 22nm while exploring 7nm; in a 28nm trial, compute rose from 150 TOPS to 600 TOPS, roughly 4x the performance of a conventional architecture on the same node.
The architecture could also support low-power always-on wake-up, millisecond-level local processing in vehicles, and ultra-long standby for IoT devices. Wu Huaqiang frames it as a path to “overtaking by switching lanes”: reducing dependence on leading-edge process nodes and challenging silicon’s energy-efficiency limits through architectural innovation.
林勇华 addresses the final mile on the software side. Roughly 90% of new AI chip architectures are shut out of existing ecosystems, while compatibility pressure forces innovative architectures to compromise with legacy standards. FlagOS coordinates framework plugins, open-source operators, and low-level compilation, provides more than 8,800 operators, and aims to become a “universal socket.”
FlagOS says it can adapt one model to 12 different AI chips within a day and reach dozens of types of compute within 72 hours. Its ecosystem already includes more than 100 partners, with training spanning more than 50 countries. Its aim is not to build another closed wall, but to lay down “shared soil.”
庄明浩 ultimately reduces production-grade AI to coordination across three layers: demand, trust, and supply. Agents create demand, reliable evaluation constrains delivery, and systems, components, and ecosystems absorb scale. “AI entering production is about who can get every task to completion” (“AI进入生产阶段，比的是谁能让每一个任务跑完。”).
He also uses a pyramid to describe the ecosystem: the performance layer delivers trusted operation; the efficiency layer builds the software ecosystem and evaluation standards; stability is implemented in compute, systems, and data centers; and security is anchored in the underlying layer, chips, and hardware.