The Agent Cloud: Databricks’ Bet on the Future of AI — Matei Zaharia and Reynold Xin
Summary
Databricks’ open OmniGen agent layer is designed to make model and harness churn an interface problem, not a customer migration. Matei Zaharia traced it to engineers building workflows around Isaac, an internal wrapper for Claude Code and Codex, while Genie and other agents needed portable sessions, collaboration, history, search, security and spend controls. The common API accepts messages or files, streams text and tool calls, and can cancel turns; Matei’s open-source rationale is that an open layer with many integrations should outcompete a closed equivalent.
The core platform thesis is that useful agents need proprietary data in the right place, not merely another general-purpose model. Swyx articulated the view that traditional software may be rewritten by putting the data in place and adding capable agents; Matei agreed that “without the right data, you can’t really do that.” Databricks’ closing strategy is therefore to make data, governance and execution safely accessible to agents.
LTAP attacks the entrenched tax of moving transactional data into analytical systems through brittle CDC pipelines. Matei argued that Databricks can obtain roughly “99%” of HTAP’s benefit by unifying storage rather than query engines: idle storage-fleet CPUs transcode row-oriented Postgres pages into compressed columnar data that analytics can read immediately. The approach aims to avoid an intervening CDC pipeline while keeping investigations from overloading the production database.
Agent adoption creates a control-plane market because binary permissions cannot express enterprise risk. Zaharia’s contextual policies remember whether an agent installed a one-day-old NPM package or read 1,000 confidential documents, then allow, block or escalate later actions. The same session state can cap a task at $5 after a debugging job burned $500. Security is paramount, while cost exposure also compounds for consulting companies with 100,000 employees coding for customers.
Databricks cited operating scale both as infrastructure evidence and as a source of systems-design data. Matei said it launches 50–60 million virtual machines daily across three clouds and processes exabytes of data, while Neon launches roughly 13 million databases a day, with agents and branching contributing to that activity. Separately, Databricks described a new analytical engine whose sampled trace table contains about a quadrillion data points and whose rollout targets workloads finishing in tens of milliseconds.
Databricks’ competitive argument against Snowflake is path-dependent: “start open and start large.” Ion Stoica’s account emphasized Databricks’ origins in bulk ingest, machine learning and open formats, followed by faster serving, versus Snowflake’s narrower proprietary warehouse focus. For enterprises wary of decades of lock-in, an open foundation and a unified data-management plane can outweigh an individually optimized engine.
Mosaic’s role has shifted from general open models toward specialized models and compound agent systems. Ion Stoica said Databricks chose not to focus primarily on another frontier model. Zaharia’s document-understanding model returns structured JSON while being roughly 100× cheaper—and still better—than frontier-model parsing; Databricks also described open-model pipelines that generate training environments and beat Opus and GPT-5.5 on a task. The timing of mainstream self-customization remains uncertain, but stronger bases, better RL traces and synthetic data should keep lowering the barrier.
Deep dive
Not yet available upstream; scheduled sync will retry.