Retrieval After RAG: Hybrid Search, Agents, and Database Design — Simon Eskildsen of Turbopuffer
Summary
Turbopuffer is betting that AI creates a once-in-15-years database opening because models can reason over knowledge but cannot store it all in full fidelity. Simon Hørup Eskildsen’s category thesis is that every company will connect large datasets to AI, creating demand for an external source of truth: “We can’t compress all of that into a few terabytes of weights.”
The company began with a brutally specific cost gap: Readwise spent about $5,000 monthly on its entire infrastructure, while one useful recommendation feature projected at $30,000. Eskildsen inferred that it would have shipped at one-tenth the price, then designed an object-storage-first search engine from napkin math rather than broad customer research. “That haunted me.”
Turbopuffer’s architectural window opened only after cloud NVMe SSDs arrived around 2017, S3 became consistent in December 2020, and compare-and-swap reached S3 in late 2024. Durable state lives in object storage, hot data rises into NVMe and DRAM, and there is no separate consensus system; Eskildsen’s operating principle is blunt: “I don’t want state in two systems.”
Cursor and Notion supplied the early commercial proof, with Cursor migrating in one or two weeks and cutting its cost by 95%. When Notion needed lower latency across Oregon’s roughly 14-millisecond public-exchange path, Turbopuffer bought about $5,000 of dark fiber, absorbed egress and tuned TCP. The buy-versus-build shift, Eskildsen argues, is now “not really about can we build it? It’s about do we have time to build it?”
Agentic retrieval is expanding query volume from one RAG lookup into many concurrent searches by one agent, forcing database economics to adjust. Turbopuffer is reducing query pricing roughly 5× as customers parallelize semantic, full-text and regex searches against warm datasets. The hosts’ synthesis was that “all workloads are hybrid,” while Eskildsen relays Sualeh’s framing of retrieval as “cache compute.”
The business reached profitability partly because early infrastructure bills sat on Eskildsen’s credit card and forced first-principles optimization before institutional funding. Pricing remains storage-plus-writes-plus-queries with “duct tape and spit,” while deployment spans SaaS, single-tenant and customer-VPC configurations. His fundraising compact with Lachy Groom was equally unusual: if product-market fit failed to appear by year-end, “we’ll just return all the money to you.”
Near-term upside rests on moving from vector search into full-text search and 100-billion-item datasets without losing startup focus. ANN v3 searches 100 billion vectors at roughly 40-millisecond P50 and 200-millisecond P99; ANN v4 is underway, ANN v5 is being planned, and FTS v3 features are rolling out incrementally. Longer-term query plans may include OLAP, logging, time series and graphs, but Eskildsen says the likeliest regret is “having tried to do too much.”
Turbopuffer’s execution model depends on unusually selective hiring rather than headcount accumulation. Every candidate begins as a rejection unless an interviewer is prepared to “have both fists up” and fight for the hire; the sought-after P99 engineer can identify a 10× gap between napkin math and reality, then bend the software toward the physical limit.
Deep dive
1. AI’s missing layer is a searchable external memory
Eskildsen defines Turbopuffer narrowly today as a search engine: it provides vector search and full-text search, while workloads requiring substantially more may belong elsewhere. The larger ambition is to become the search engine for unstructured data rather than another generic database wearing an AI label.
His premise starts with compression limits: models can absorb “exabytes and exabytes” of training data and encode ways to reason about the world, but “we can’t compress all of that into a few terabytes of weights.” AI therefore needs an external system holding knowledge “in full fidelity and truth.”
Eskildsen sees three prerequisites for a major database company: a workload that eventually touches every company, a storage architecture incumbents cannot easily retrofit, and a path toward implementing nearly every query plan customers might ask of stored data. Oracle captured one era; Snowflake and Databricks captured another roughly 15 years later—or more; connecting large datasets to AI could define the next.
2. A $30,000 feature exposed the market before customers did
Nearly a decade at Shopify taught Eskildsen to scale databases under extreme traffic, including events approaching one million requests per second. The most aggravating system to operate was self-hosted Elasticsearch circa 2015: projects were constrained by it, and exposing the Lucene behavior Shopify needed proved difficult.
After leaving, he practiced what he called “angel engineering,” taking roughly three-month assignments at friends’ companies including Readwise, Replicate and Causal. At Readwise, his ostensible job was improving Postgres—“basically boils down to tuning autovacuum”—when the ChatGPT moment suggested embedding articles for recommendations.
The prototype worked almost uncomfortably well: recommendations for one Readwise co-founder surfaced articles about having a child before Eskildsen knew the news. Yet Readwise’s entire infrastructure cost around $5,000 monthly, while embedding and indexing its articles for this single feature penciled out near $30,000.
Readwise shelved the feature until costs fell, but Eskildsen could not: “That haunted me.” His only market datum was that the company likely would have shipped at one-tenth the cost, so he began learning vector indexes and cloud primitives instead of manufacturing a broad macro thesis.
3. Object storage became the database, not merely its backup
Eskildsen’s napkin math suggested putting durable data in object storage, pulling active portions into NVMe SSDs and promoting the hottest subset into DRAM. In his simplified example, a terabyte in S3 cost about $200 monthly, perhaps 5%–10% needed NVMe residency, and still less needed DRAM—dramatically reducing the cost of “inflating” stored data.
The tradeoff is explicit: every write may take a couple hundred milliseconds, and a first query might take half a second. Turbopuffer is therefore unsuitable for high-transaction workloads; later Eskildsen states its write latency is around 100 milliseconds. He never assumed search was purely read-heavy, noting that Readwise could create more writes through content churn than actual searches.
His first vector design was almost deliberately primitive: store cluster metadata in a
clusters.jsonfile, keep each cluster in its own object, fetch the nearest clusters, then calculate neighbors locally. That creates roughly two storage round trips rather than a long sequence of dependent reads.The deeper systems principle is massive concurrency with few decisions between rounds: issue perhaps 1,000 S3 requests together, process the result, then repeat no more than about three times. Used this way, Eskildsen argues, NVMe can approach DRAM bandwidth within a low multiple, while object storage can saturate the network card.
4. Three cloud upgrades made the architecture newly possible
The enabling chronology matters: cloud NVMe SSDs appeared around 2017; S3 became consistent in December 2020; and S3 gained compare-and-swap only in late 2024. Together they made it possible to build a database around object storage without maintaining a separate foundation database, ZooKeeper or similar consensus layer.
Compare-and-swap lets many nodes download
metadata.json, modify it, and write it back only if nobody changed the original meanwhile; conflicts simply retry. Google Cloud Storage already offered the primitive when Turbopuffer began, largely by luck: Eskildsen chose GCP because Shopify used it and he knew its Canadian team.Turbopuffer consequently went “all in”: even turning off every server would not lose data. Eskildsen and co-founder Justine preferred this to dual-state operations because their worst on-call experiences involved systems falling out of sync. Asked why he would choose fiber over ZooKeeper, he answered, “Way rather. I don’t want state in two systems.”
That conviction became painful when Notion, an AWS customer, wanted lower latency. Oregon traffic took a roughly 14-millisecond route through Seattle because the providers’ regions were geographically separated, so Turbopuffer bought dark fiber between the AWS and GCP regions in Oregon, routing through the Portland exchange for about $5,000, absorbed egress and accepted a single line when multiple redundant circuits were customary.
5. Cursor and Notion converted architecture into product-market fit
The launch was intentionally skeletal: after working alone through the summer, Eskildsen shipped a Rust binary on one eight-core machine inside tmux. Deployment meant watching request logs and pressing Control-C during a quiet moment—a Shopify-derived rule that infrastructure should earn sophistication only after showing “at least the inkling of PMF.”
Cursor co-founder Arvid initiated a terse exchange of QPS, costs and growth projections. When Sualeh later proposed a call around 4:00 a.m. Pacific, Eskildsen accepted from the East Coast, sensed he needed to meet the team, and arrived in San Francisco while Cursor’s Postgres was down—prompting an impromptu recommendation to tune autovacuum.
Cursor migrated over the following week or two, and Turbopuffer reduced its cost by 95%, which Eskildsen believes repaired its per-user economics. He recruited Justine, “the best engineer” he had worked with at Shopify, and the pair spent the next month or two ensuring the database never became Cursor’s problem.
Notion’s internal engineer had independently sketched essentially the same storage architecture, then discovered Turbopuffer had built it. Eskildsen’s explanation of the purchase is revealing: AI has changed buy versus build from “can we build it?” to “do we have time to build it?” A vendor that behaves like an extension of the team buys speed.
6. Hybrid retrieval survives because different queries reveal different truths
Cursor chunks and embeds entire codebases using its own embedding model, reportedly producing a 25% improvement on one specific evaluation and working especially well on larger repositories. Its agent uses semantic searches to find similar or functionally related code, but it also uses grep; neither mechanism eliminates the other.
The hosts press the “is RAG dead because grep?” argument, then land on the broader lesson that workloads are hybrid: semantic search, lexical search and regex serve different questions. Eskildsen avoids predicting the macro future—“That has turned out to be a giant waste of time”—and instead collects concrete customer case studies.
Cursor also treats the external database as a security boundary: its private embedding model makes reversal harder, file paths are obfuscated, and customer data in Turbopuffer’s bucket is encrypted with Cursor’s own encryption keys. Eskildsen agrees these are sound practices for any external database, not concessions unique to his company.
Sualeh’s framing, as Eskildsen recalls it, is that retrieval is “cache compute”: at a particular moment, the model is focused on a particular context, and search supplies an intermediate layer tailored to that state. Eskildsen will not predict how its value changes over time, but current workloads show it matters for specific queries.
7. Agents turn one retrieval into a burst of concurrent searches
Eskildsen associates classic RAG with an 8,000-token context window and one retrieval that had to count. Agents instead treat search as a tool call, repeatedly querying and changing their working state while the model handles reasoning.
The architectural shift is concurrency within one user session, not merely batching across users: “One agent driving multiple.” Notion launches what Eskildsen calls a ridiculous number of queries per round trip, Cursor’s agent is increasingly parallel, and the objective mirrors Turbopuffer’s internals—hit a warm dataset with many searches while minimizing sequential turns.
The hosts cite Cognition doing eight fast-context searches in parallel and ask how an agent avoids issuing the same request eight times. Their answer is query diversity, with hybrid retrieval supplying fundamentally different search modes rather than cosmetic variations on one semantic request.
More searches change unit economics. Turbopuffer is reducing query pricing about 5×, with further reductions possible, to support these bursts. Eskildsen says write volume remains extremely high relative to reads, but expects the ratio could shift if customers lean fully into agentic parallelism.
8. Open-card financing reinforced first-principles economics
Initial pricing was “very vibe priced”: Eskildsen estimated physical costs and added a little margin. When Cursor’s usage accelerated, its invoice remained below Turbopuffer’s GCP bill, so he and Justine optimized relentlessly to reach even a roughly 5% margin while the cloud liability expanded on his personal credit card.
That pressure helped make Turbopuffer profitable, “to the chagrin” of its VCs. Current pricing still decomposes into storage, writes and queries, but Eskildsen calls it the original structure held together with “duct tape and spit”; more changes are planned. Customers can choose SaaS, a dedicated single-tenant cluster, or BYOC inside their own VPC.
While raising amid a competing launch, Eskildsen chose Lachy Groom over database-specialist investors because he could call without preparation and speak plainly: if PMF did not arrive by year-end, “we’ll just return all the money to you.” His rule when unfamiliar with a game is simple: “I just play with open cards.”
Groom’s lack of database expertise became useful rather than disqualifying: the founders and employees supplied that depth, while Groom helped with candidates and customers without pretending otherwise. Accepting the check also marked Eskildsen’s deliberate commitment to make the company “part of my life’s journey” and give it everything once employees and investors depended on him.
9. Turbopuffer will broaden only after search earns the next act
Act One was vector search; Act Two is full-text search. Turbopuffer claims to beat Lucene on some unusually long, LLM-generated queries over Common Crawl-scale datasets, while adding the large feature surface users expect from mature lexical engines and attracting migrations from traditional search products.
Full text remains valuable even for tiny human queries: typing “si” into Command-K might lead embeddings toward the Spanish or Italian word for “yes,” while literal prefix search could surface a document beginning “These are all the reasons I hate Simon.” Hybrid search maps both meaning and exact user intent.
Scale is the other near-term priority. ANN v3 searches 100 billion vectors at roughly 40-millisecond P50 and 200-millisecond P99; ANN v4 is underway and ANN v5 is being planned, while full-text improvements will roll incrementally toward FTS v3. Eskildsen also wants a database console with the practical usefulness of phpMyAdmin rather than the startup dashboard accumulated over two years.
Long term, a major database must support aggregation, joins and nearly every query plan. Possible next acts include simpler OLAP, traces, logging, time series and graphs atop Turbopuffer’s underlying key-value system; Simon cites a report that Cursor moved roughly 20 terabytes from Postgres to defer sharding. Yet search must remain the primary reason to adopt it today: “What we’re most likely to regret at the end of the year is having tried to do too much.”