What Kills Agent Projects Is Often Not Lack of Intelligence
Deep thoughts on AI and aspirations —— ByteDance Deep Thinking Circle
In YC’s Spring 2025 batch, nearly half of the 144 projects claimed to be building Agents. At the June YC AI Startup School that year, Musk came to predict digital superintelligence, Altman talked workflows, and Karpathy talked evaluations. On the surface, it was a rally for Agent startups, but the colder question circulating in the room was: why do so many Agent projects have beautiful demos but never make it to production?
My assessment is straightforward: treating probabilistic systems as deterministic functions is the cause of death for most Agent projects.
Most Teams Are Building Agents on a False Assumption
The common approach to building Agents is: tune a large model, add some conditional logic, and string together several operations into a workflow. This hides an implicit assumption—that the model’s output is stable, like calling a function where the same input yields the same output.
But LLM outputs are probabilistic modeling, not deterministic computation. The same task that runs smoothly today might drift tomorrow; what’s consistent in one context becomes inconsistent in another. Built on such assumptions, every Agent run is a gamble. Without automated evaluation at each step, these systems can neither be tested nor reproduced.
Karpathy made an apt analogy at that conference: LLMs are like eccentric geniuses living in the cloud—they can write code that exceeds human capability, but occasionally misspell your name. His warning was blunt—without rigorous evaluation, continuous integration, and exception handling, using it in production isn’t engineering, it’s faith.
Many projects showcase precisely this kind of faith. A working demo doesn’t equal a working product. A demo is one sample under best-case conditions; production is a long-term endurance test under worst-case conditions.
Call Sequence Is Not Process; Process Is Responsibility Structure
Altman repeatedly emphasized at the conference that the future of Agents lies in running complete workflows. Many teams heard this as “arrange the calls in the right order.” This is the easiest mistake to make.
Call sequence addresses “what to do first and what to do next”; process addresses “who’s responsible when things go wrong.” A truly production-ready process is a chain of actions with strong constraints on time, causality, and accountability: when a step fails, who’s responsible? How do you roll back? Is the user notified? Can the result be explained?
Take an Agent handling customer complaints. The prompt-chaining approach strings together reception, appeasement, and compensation sequentially. The responsibility-chain approach answers questions like: which keywords trigger escalation to humans? How do you correct misclassifications? What’s the default action when emotion recognition fails? After failed appeasement, do you stay silent or transfer to a human? Is there a second verification? These decision points determine whether an Agent “runs smoothly” or “blows up fast,” and they depend not on model parameter count but on business abstraction and fault-tolerance design.
Most so-called “workflow Agents” on the market are essentially prompt chains—logically coherent but structurally absent. Such systems instantly fail when encountering edge cases, business changes, or regulatory intervention.
Looking at projects that do make it to production, the difference often lies in this structure. They place a human gate before high-risk actions, archive input and output from each step, can precisely roll back to the previous checkpoint on error, and define permission boundaries for each Agent—what data it can touch, how much money it can move, whether it can send external messages, all hard-coded. These things are invisible in demos, but they determine whether an Agent can enter real business.
Customers Don’t Pay for Intelligence; They Pay for Peace of Mind
There’s an even more practical layer: when it comes time to sell Agents, customers don’t care how smart they are. They care about: can this thing integrate with my existing systems? Does it have security audit interfaces? When something goes wrong, can the decision path be traced? Can it collaborate with humans in sharing responsibility?
In other words, enterprise procurement isn’t buying intelligence; it’s buying peace of mind. Intelligence can be purchased from model APIs; peace of mind can only grow from your own process design.
This is also why regulation becomes a barrier to entry. Most teams treat compliance as reactive—go live first, deal with problems later. But when audit mechanisms, responsibility tracing, and behavior logs are embedded in product architecture, they become insurmountable obstacles for competitors when policies tighten. Nadella’s repeated mentions of GDP at that conference carry the same logic: AI must prove it creates new value that is explainable and accountable to continue receiving social license.
Industry understanding of value falls in the same place. Whether an Agent can be deployed often depends not on what AI does best, but on the parts where humans must intervene: when to respond, when to hand off, when to stay silent. These judgments can’t be learned from papers; they can only be gained through industry immersion. YC increasingly values whether founders come from the vertical domain rather than whether they understand LLMs, for this very reason.
Counterargument: Isn’t This Just Old-School Software Engineering?
The strongest counterargument is: testing, staged rollout, rollback, CI/CD—these are standard software engineering practices from the past few decades. What’s new?
What’s new is the magnitude of difficulty. Traditional software embeds deterministic logic where test cases can essentially be exhausted; once non-deterministic models are embedded, you can’t enumerate all outputs—you can only design evaluation distributions, failure tolerance, and rollback mechanisms. The same engineering discipline, moved to probabilistic systems, doubles in difficulty.
Additionally, the commoditization of demos has changed supply-demand dynamics. Making a decent demo used to require effort, so the demo itself proved capability. Now anyone can quickly build a 90-point demo, so a 90-point demo proves nothing. When intelligence becomes universally demonstrable, only reliability remains worth paying for.
Some will ask: model capabilities are still improving—won’t these problems automatically disappear when models get stronger? My assessment is negative. The stronger the model, the larger its potential blast radius, and the higher—not lower—the requirements for verification and fallback mechanisms. Reliability has never been a free accessory the model provides; it’s engineering the product must build itself.
So I summarize it in one sentence: in Agent entrepreneurship, verifiability is the moat. Whoever can prove their system is reliable and continuously improving has a product; everyone else only has experiments.
What This Means for Those Building Agents
Agent entrepreneurship appears to be about AI technology, but the real competition is “industry process + uncertainty engineering.” Model capabilities are converging, APIs are commodities—what can truly be retained are only two things: your understanding of a particular industry process, and your ability to engineer uncertainty into a system.
Before going live, self-check with three questions.
First, does the system have automated evaluation mechanisms? Can you use data to prove this version is better than the last, rather than relying on manually reviewing a few outputs?
Second, when each step fails, who’s responsible, how do you roll back, does the user know? Write down the answers—wherever you can’t write them down is where things will blow up later.
Third, beyond the demo, do you have a paying customer running it daily in real business? Customer renewals and complaints are the only honest evaluation set.
If you can answer these three questions, you’re building a system; if you can’t, no matter how beautiful the demo, it’s just fireworks.