Your Agent's Self-Improving Swiss Army Knife: Composio CTO Karan Vaidya on Building Smart Tools
Summary
Karan Vaidya’s moat thesis is usage-driven tool improvement, not merely a catalog of 50,000-plus tools across 1,000-plus apps. He says runtime failures can trigger a new tool version in real time, while zigzag traces become reusable skills that take a “straight path” next time. He presents accumulated experience across agents and apps as a moat; Nathan reports using Composio over the last couple of weeks.
The scarce resource in agent systems is attention, so exposing more tools can make the product worse. Karan warns that an agent may “use the wrong blade and suicide via context overload”; Composio instead presents a few meta-tools, discovers the relevant action or skill just in time, and uses code sandboxes for workloads involving 10,000 or even 1 million items. “Harness is nothing but context.”
The go-to-market splits between prosumers buying setup simplicity and developers buying governance. One MCP server can connect Claude Code or similar runtimes to managed authentication, while production teams can adopt discovery, sandboxing, auth, or the full execution layer modularly. Karan cites AWS, Zoom, Glean, and Airtable as customers, with action-level scopes, hooks, human approval, SOC 2, and VPC self-hosting supplying enterprise trust.
The strongest use cases already resemble complete jobs, but autonomy increases the value of permission design. Karan’s agent finds open-source contributors, enriches their contact data, reaches out from its own email, and generated 30–40 calls over one or two weeks; a sales agent prepares similarly targeted drafts. His control model separates read-rich, non-acting research agents from write-capable agents with limited data and human-in-the-loop checks.
Skills could re-commoditize frontier models by moving judgment into reusable instructions. Karan routinely has Opus create a skill and Sonnet run it, says Haiku has not worked well in his experience, and reports that roughly 90% of tested GPT skill cases work, with 90–95% generally achievable across current models. Anthropic’s better polling behavior versus GPT stopping for input is the key caveat. Composio is developing metrics and benchmarks and already does some provider-aware adaptation, making reduced model lock-in a core value proposition.
AI should strengthen core infrastructure while pressuring packaged SaaS at the interface and customization layers. Karan expects AWS and Cloudflare to benefit as software creation expands, while Salesforce and Slack can defend themselves if they ship agentic interfaces quickly. Nathan’s sharper test is Intercom: he cited Fin as resolving close to 70% of tickets at $0.99 each when they spoke, but 133 Composio tools could let some companies build a tailored skill and potentially save 80–90%; Karan expects selective, not wholesale, insourcing.
Composio’s internal cost structure is an early signal of labor shifting from execution to agent supervision. A three-person team runs the pipeline that builds and improves tools, the company has roughly 15 people overall, and Karan says they spent about $100,000 on the pipeline last month, with token costs higher than human costs for that work. His formulation is “we need more humans to spend more tokens,” alongside plans for one-wallet premium toolkits and a universal CLI.
Deep dive
1. A useful Swiss Army knife cannot expose all 50,000 blades
Nathan’s starting point: Composio gives agents more than 50,000 tools across 1,000-plus apps, covering common integrations that would be wasteful to build bespoke. Even granting Google Drive or Slack access can require enough console work and permission clicks to stop a casual user.
Karan accepts the Swiss Army knife description but rejects breadth as the final product: hand every blade to a model and it may “use the wrong blade and suicide via context overload.” He defines Composio instead as an “agentic tool execution layer.”
The harness combines just-in-time discovery, dynamic tool injection, authentication, authorization, scoped access, sandboxes, and triggers such as a new email, Slack message, or pull request. Governance, observability, and auditability are also available through Composio’s dashboard. Its job is to give the agent only the execution context it needs.
Nathan says he has used Composio over the last couple of weeks, including wrestling with Google Drive and Slack permissions. That grounds the setup problem in his own testing rather than merely describing a theoretical integration burden.
2. Individuals buy convenience; production teams buy control
For a Claude Code or similar prosumer, the proposition is one MCP server instead of separately installing and learning integrations for Google Drive, Zoom, Datadog, and everything else. Authentication can be initiated conversationally or managed through Composio’s dashboard.
Production developers can take the full harness through MCP, API, or SDK, or select only components such as discovery, the Workbench sandbox, authentication, or actions. Their own tools can sit beside Composio’s tools rather than forcing an all-or-nothing architecture.
Karan says governance, observability, and auditability matter at scale. He cites AWS, Zoom, Glean, and Airtable as customers whose evaluations provide reassurance to smaller buyers handling sensitive company data.
The security stack begins with action-level least privilege: an agent may read email without being allowed to send it. Pre- and post-execution hooks add policy checks or human approval, while SOC 2 and VPC self-hosting address enterprise requirements; Composio is self-hosted inside AWS for AWS’s use case.
3. Sandboxed code turns massive workloads into tractable jobs
Nathan’s question is architectural: Claude Code executes Bash locally, some provider tools run remotely, and results cross between environments—so where should tool execution live? Karan says Composio’s sandbox primarily removes infrastructure the model should not have to recreate.
Agents receive utilities and abstractions that minimize code, especially around authentication and translating between programmatic execution and function calls. Karan says the same Docker-image-like environment will be available locally very soon for internal or machine-local tools.
Direct function calling breaks down when an agent must process 10,000 emails because every result cannot occupy its context. In the sandbox, it can write code—and even write code that itself invokes an LLM—to process 10,000 or potentially 1 million records. Karan calls this “kind of like inception.”
Mounted folders solve another mundane failure point: files placed there are uploaded to S3 and receive shareable links. An agent can analyze email or Stripe activity, generate a report, copy it into the designated folder, and share the result without inventing a file-sharing pipeline.
4. Smart MCP means progressive disclosure plus learned execution
Nathan distinguishes smart MCPs from the first wave of API wrappers: he is looking for a layer that accepts higher-order intent, composes several calls, and progressively reveals capabilities. He sees MCP-versus-CLI arguments as less fundamental because either interface can eventually support intelligent disclosure.
Karan’s governing constraint is that “attention is definitely not free.” Even with million-token contexts, showing 50,000 definitions can overwhelm performance, so the model initially sees only a few tools and receives relevant actions dynamically.
Discovery extends beyond tools into just-in-time skills. A retrieved skill can specify which actions to call, what code to run in Workbench, and which trajectory previously reached the outcome; the agent then adapts that recipe to the current request.
Repeated failures become advance warnings—pitfalls, do’s, and don’ts placed into context before execution. Karan’s reduction of the entire product philosophy: “Harness is nothing but context. You have to engineer the context.”
5. User requests are shifting from instructions to complete outcomes
Composio usually does not receive the user’s raw language because Claude Code or another agentic runtime acts as an “intelligent mediator.” If someone asks to connect Google Drive, the runtime already knows to call connection management instead of sending an ambiguous discovery query.
Karan identifies December as a trust inflection point: users concluded that current models could handle materially more autonomy. Software engineering moved first, but knowledge workers increasingly state the desired outcome and let agents determine tools and intermediate steps.
One direct example is granting an agent Gmail access and asking it to review the prior month, judge which messages are no longer useful, and archive them. The agent writes code and uses an LLM within that code to classify the emails.
Karan’s personal hiring workflow is closer to a full recruiter job: it scans agentic, Python, and TypeScript repositories, identifies strong contributors, enriches location, email, LinkedIn, and social data, then contacts candidates from its own email account. After emailing thousands of people, he says it created roughly 30–40 calls in one or two weeks.
6. Long-context autonomy makes permissions an architectural primitive
Nathan describes a local, gigabyte-scale database containing five years of email, Slack, direct messages, recorded calls, and podcast transcripts. It gives an assistant rich relationship history, but may also contain emailed passwords, recovery codes, or credit-card details he cannot inventory safely.
Karan’s answer is not one universally trusted assistant but “multitudes of agents” with distinct profiles. A research agent might read broadly while lacking permission to send messages or execute external actions, keeping its large context self-contained.
A write-capable agent gets the inverse profile: permission to act, but little sensitive personal or company information. Human-in-the-loop checks can inspect proposed messages or operations before execution, reducing the chance that a secret token is emailed externally.
The principle is capability separation: broad knowledge and broad agency should not automatically coexist. Composio’s scopes and hooks are meant to express that separation for different Claude Code, Codex, or other agent instances.
7. Every failure can improve a tool—or become a reusable skill
Composio’s integrations are themselves built through an internal agentic pipeline: agents obtain developer-app credentials, create actions, identify dependencies, and test real-world scenarios and edge cases. The same machinery can be invoked when a customer’s agent cannot understand or successfully use a tool.
A runtime failure may produce a new tool version immediately and insert it into the active context. Karan says the same tool could accumulate tens of thousands of versions, allowing an improvement to remain personalized when it serves only one customer’s unusual workflow.
Generic corrections propagate broadly. Agents use APIs in “insane, different ways,” uncovering cases documentation does not describe correctly; Karan says autonomous discoveries recently made Composio’s Google Calendar tools “much better than what docs propose.”
Nathan worries that upgrades may break a workflow already dialed in. Karan’s answer is to let tools improve broadly while saying that learned skills do not change as often and preserve the desired trajectory and repeatable behavior.
8. Skills make model switching practical, though not yet seamless
Detailed skills specify enough of the path that less judgment is required during repetition. Karan’s regular workflow is to let Opus discover and encode a successful procedure, then switch to faster, cheaper Sonnet for subsequent runs; he says this works “phenomenally well,” while Haiku has not worked well in his experience.
Cross-provider portability is weaker but still high. Karan says that, in his tests with GPT models, 90% of skills just work; elsewhere he characterizes 90–95% as readily achievable across current models, with the remainder exposing provider-specific behavior embedded in the original skill.
His sharpest counterexample is polling: Anthropic models tend to wait and continue polling until a job completes, while GPT may stop and ask for user input. Those default behaviors can alter execution even when both models understand the written instructions.
Asked about Gemini Flash, Karan hedges: he has not tested the exact skill-transfer case himself, though production experience feels smooth and he expects either the current or next iteration to reach the required level.
9. Translation skills could become an anti-lock-in layer
Nathan’s reframing is that skills may be a “re-commoditization layer.” Frontier models can diverge qualitatively and create stickiness, but exhaustive instructions move more behavior out of the model’s defaults and into a portable artifact.
Composio therefore positions its harness as one way to avoid model lock-in: authentication, tools, and skills remain in one layer while the model underneath changes from Anthropic to OpenAI or eventually to open-source models that Karan says could be roughly 10 times cheaper.
Karan sometimes describes a model switch as retaining 99% reliability, but separately acknowledges that raw cross-provider skill portability is closer to 90–95%. The remaining work is provider-aware transformation that compensates for known behavioral differences.
Composio is developing metrics and benchmarks and already does some of this provider-to-provider adaptation. Karan says getting from the readily achievable 90–95% to 100% is genuinely hard because skills are unstructured and can silently encode assumptions about the model that wrote them.
10. Agent enablers expand, but incumbent systems still own the data
Karan is bullish on tools created specifically for agents: memory products such as Mem0, Supermemory, and Zep; payment infrastructure such as Skyfire; commerce through Shopify; and search or retrieval through Exa, Firecrawl, Tavily, and similar services.
Composio partners across categories rather than selecting a single winner, letting builders choose combinations appropriate to their use case. Memory sees obvious demand, while background agents associated with the OpenClaw movement are increasing interest in payments and commerce.
Asked what category is still missing, Karan gives an honest non-answer: builders appear to be attacking nearly every activity a human performs, including agent-to-human delegation. He would need more thought to name a conspicuous empty space.
Despite the agent-native wave, most usage still touches Slack, Salesforce, and similar traditional software because “that’s the system of records.” At the infrastructure layer, Karan expects AWS and Cloudflare to strengthen as easier software creation increases dependence on underlying platforms.
11. SaaS survives through interfaces, while customization shifts build-versus-buy
Karan expects the interface to existing SaaS to change, while the underlying systems of record remain important. Startups will build new agentic interfaces, but Salesforce, Slack, and other incumbents are moving quickly enough that the outcome depends on execution speed rather than incumbency alone.
Nathan’s pushback favors established vendors in large accounts: an AI-first CRM may be compelling, but enterprise sales cycles give Salesforce time to copy its strongest features. The more disruptive threat may be customers bringing narrow workflows in-house rather than adopting a competing suite.
Intercom makes the tradeoff concrete. Nathan cited Fin as resolving close to 70% of customer-service tickets across many thousands of customers when they spoke, at $0.99 per resolution, offering instant 24/7 responses; yet Composio exposes 133 Intercom tools, leading him to speculate that a company could encode its own support skill.
Nathan’s hypothetical was that the token cost might be around $0.10 per case, potentially reducing costs by 80–90%. Karan argues customization matters more than savings: an internal agent can use several apps, restrict particular actions, and implement company-specific governance. He does not expect a widespread exodus from Fin within a year, but agrees that falling friction means “people will inch towards build compared to buy.”
12. Agent delegation, token economics, and interfaces remain bipolar
Karan’s delegation heuristic is contextual weight. If a task consumes only “1–2%” of context, the main agent should keep control and receive the right tool; handing appointment booking to a context-poor sub-agent could create a collision with an important board meeting already visible to the main agent.
Deep research belongs with parallel sub-agents because exploration consumes substantial context that can be condensed before returning. He highlights Claude’s shared task-list pattern and Composio’s agentic tools, which preserve a session ID for repeated natural-language exchanges.
Internally, one orchestrator can supervise 20–30 Claude Code agents. A three-person team manages the pipeline that builds and improves Composio’s tools; against roughly 15 employees company-wide, the company spent about $100,000 on that pipeline last month. Karan says token costs are already higher than human costs for the work and summarizes the model as: “We need more humans to spend more tokens.”
Product expansion follows the same consolidation logic: planned premium toolkits use one Composio wallet for services that currently require separate API keys, while a universal CLI accesses many apps from one interface. Karan expects CLI and MCP to coexist in a “bipolar world,” with token allocation and improving traceability helping determine the balance.