Summary

The investable shift is bigger than AI search: Robert McCloy argues that “AI’s replacing web browsing.” Consumers increasingly prefer ChatGPT, Claude, Perplexity, and agents to navigating websites, making the emerging control point the complete “agent experience,” not merely link ranking. The durable winners will make their products intelligible and usable inside these interfaces while retaining direct audience relationships.
AI search still rests largely on traditional search infrastructure, but an LLM-mediated re-ranking layer changes what earns visibility. Systems generate one or more queries, retrieve with mechanisms such as BM25, TF-IDF, or PageRank, re-rank results, read selected pages, and synthesize an answer. Traditional SEO therefore still matters, but clickbait metadata can be discarded in favor of pages whose titles and descriptions clearly explain their relevance.
ChatGPT traffic can carry unusually high commercial intent because users arrive while solving a problem, not merely researching one. Robert says Clerk saw roughly 6x growth in ChatGPT traffic and 9x growth in conversions after targeted content work; companies actively working to improve their sites typically achieve double-digit traffic improvements within one or two months, though Robert cautions that developer tools are especially well suited. In B2B, users are already outsourcing three-vendor procurement “bake-offs” to ChatGPT and deep research.
Most near-term optimization is positive-sum content engineering, not an adversarial ranking game. Robert says prompt injection “often works right now,” but expects black-hat techniques to be penalized as systems mature from perhaps “1/100th of the sophistication they’ll eventually have.” Clear prose, structured facts, FAQs, server-rendered HTML, and explicit explanations of who a product serves remain the larger opportunity.
Several fashionable GEO tactics presently contribute little to discoverability. llms.txt is not retrieved by default for discoverability, embeddings are not the key to understanding how current platforms consume pages, and most retrievers do not execute JavaScript. The brutally simple diagnostic is: “Turn JavaScript off, check your page”; if important content disappears, AI search probably cannot use it.
The publisher-platform bargain remains unresolved because restricting AI access can also choke off distribution. Robert sympathizes with Cloudflare’s attempt to compensate publishers but says “the bridge doesn’t connect on both sides yet”; AI browsers such as Dia, Comet, and the discussed OpenAI browser make model-wide exclusion even harder. Publishers without strong direct audiences still depend on Google and increasingly ChatGPT for traffic.
ChatGPT has the strongest durable consumer relationship, but valuable niches make aggregate market share an incomplete guide. Perplexity is Robert’s second-largest AI-native search platform and has unusually committed users; Claude is smaller but its users can be valuable and passionate, while Meta AI is a “silent monster” with a reported 700 million active users. Personal memories and preferences may eventually produce “no single ChatGPT,” turning persona-level monitoring into essential infrastructure and a measurement headache.

Deep dive

1. Scrunch began by rejecting the website-chatbot future

Alessio disclosed that Decibel invested in the Scrunch round announced with the episode. McCloy and co-founder Chris left Hearsay, where they had served as CTO and chief product officer, convinced that starting anything without being LLM-native would leave them “already behind.”
Their enterprise contacts initially wanted branded ChatGPT widgets. Robert’s reaction was blunt: “Nobody wants to use a chatbot on your website”; pop-ups send him into “a visceral rage.” Swyx challenged whether technologists were representative, citing Intercom, but Robert distinguished support tools such as Fin—used within an existing relationship—from unsolicited widgets greeting cold visitors.
Robert saw stronger evidence in ordinary consumers’ attachment to ChatGPT: they prefer a familiar, productive environment to “Google 10 blue links,” poorly designed sites, or each company’s search bar. “They want to use the tools they like.”
The underlying customer need was discovery, not a widget. When Scrunch showed CMOs how ChatGPT surfaced their content and described their brands, the response was: “That’s really scary.” Scrunch formed around measuring this first major change in the consumer internet experience in years.

2. Commercial queries increasingly invoke live search

Scrunch simulates consumer prompts across the platforms its customers’ audiences actually use: ChatGPT, AI Overviews, AI Mode, Gemini, Perplexity, Claude, and Meta AI. Claude matters despite smaller volume because its users can be disproportionately valuable; Meta’s distribution makes its assistant a “silent monster.”
Trained-in model knowledge is less actionable than search-backed answers, but Robert’s consolation for brands is that commercially meaningful queries increasingly use live search for timeliness and hallucination control. Scrunch therefore emphasizes ChatGPT with search enabled and comparable retrieval products.
Pre-training exposure still merits deliberate policy. GPTBot, CCBot, and other identifiable collectors crawl business sites, so publishers should consider information half-life: putting a seasonal e-commerce sale into long-lived model knowledge may be counterproductive even if exposing durable product information to crawlers may be useful.

3. AI browsers weaken the case for treating OpenAI as one gatekeeper

Robert supports the sentiment behind Cloudflare’s bot-paywall initiative: publishing is getting harder, and some “grand bargain” for compensating content producers may be necessary. Yet the technical marketplace is immature—“the bridge doesn’t connect on both sides yet”—and earlier mechanisms have not become core internet infrastructure.
The harder issue is power. Publishers care how content is used but still need traffic from Google and increasingly ChatGPT; unless they possess an audience that already “know you, love you,” aggressive restrictions can erase the distribution needed to acquire users.
Swyx presented the opposite creator strategy: make everything free because reproduction costs approach zero, invite models to “train on me,” and use citations to build authority. He noted that ChatGPT sent Lenny’s newsletter more traffic than Twitter during the cited period despite Lenny having 250,000 Twitter followers and receiving only about 9,000 views there.
Dia, Comet, and the discussed OpenAI browser broaden the contest beyond centralized search boxes. Once models operate inside browsers—or through local tools—blocking one crawler cannot prevent users from applying LLMs to accessible pages. Robert’s “medium spicy take”: replacing search is the less interesting story; “AI’s replacing web browsing.”

4. Re-ranking rewards descriptive pages over clickbait

Robert’s technical model begins with an LLM generating one or multiple queries, sometimes sequentially with a reasoning model. Traditional mechanisms—BM25, TF-IDF, PageRank, or similar indexed search—retrieve candidates; a more intelligent relevance step then reorders them before selected pages are read, summarized, and used to generate an answer.
That intermediate step explains why a high-ranking page can still disappear. Metadata optimized for human urgency or click-through may say little about the content, so ChatGPT can reject it and choose a lower traditional result whose title clearly signals, “This page actually has something that’s relevant to your question on it.”
For commerce, Robert’s default is “disclose more.” A visual leggings page should still explain who buys the product and for what use; those facts help the model match prompts such as finding a hiking gift for a friend in Los Angeles, then send the buyer to a checkout—as native agentic shopping develops.
Websites cannot currently receive the originating ChatGPT prompt, and Robert expects privacy to prevent it: users put health information and intimate details into conversations that must not leak to retrieved sites. Companies can nevertheless infer demand from destination pages, then publish more material around the problems attracting ChatGPT referrals.

5. The new optimization target is the agent’s complete customer journey

Robert accepts “GEO” because “you can only fight so many battles,” but regards SEO, AEO, and GEO as names for only the entry point. An assistant increasingly discovers, interprets, compares, and may transact with a business without requiring the user to browse its site.
His preferred frame is “agent experience,” by analogy with customer experience: companies should measure how AI systems interact with their content and infrastructure because those systems increasingly serve the ultimate customer. Visibility alone is insufficient if the resulting answer misrepresents the product or cannot complete the user’s task.
Personalization makes this harder. ChatGPT memories, explicit preferences, and instructions can alter queries, preferred sources, and presentation; Robert’s senior-engineer configuration produces different search results from an incognito consumer session. “Maybe where this ends up is like there’s no single ChatGPT”—a nightmare for marketers accustomed to one ranking.
Scrunch therefore groups monitored prompts around customer personas rather than treating every answer as universal. The relevant question becomes how a senior engineer, product manager, or other ideal customer experiences the entire interaction, with their likely goals and preferences modeled explicitly.

6. Problem-solving traffic is closer to conversion than conventional search

Studies can make Google look safe by classifying ChatGPT activity as generation rather than search. Robert’s counterargument is that users often skip “How should I think about solving this?” and ask the model to solve the problem—write the proposal, build the financial model, or produce runnable code. “You get a solution. You can run the solution. Problem solved.”
That behavior represents unusually high intent: the user is already executing rather than browsing possible approaches. A striking B2B cluster is software procurement, where employees ask ChatGPT or deep research to produce the required three-vendor bake-off and comparison table, with checkboxes reflecting the buyer’s criteria rather than vendors’ messaging.
Robert qualified the evidence. Public usage studies generally rely on opt-in clickstream panels whose members may not represent high-value users such as staff engineers at Stripe. Scrunch supplements those panels with observation, interviews, and follow-ups asking people who reported finding a product through ChatGPT what they were doing at the time.

7. Clear server-rendered content beats GEO tricks

Prompt injection embedded in a feature page “often works right now,” Robert conceded, but his forecast is “it works until it stops working.” Today’s AI-search glue lacks Google’s 25-plus years of abuse defenses; eventual sophistication should push out black-hat tactics and attach penalties, even if bans are not yet common.
The larger opportunity is still positive-sum. Many companies are not insufficiently flattering themselves; they simply fail to describe what they do. Better facts, examples, FAQs, and structure improve the model’s comparison, the user’s decision, and the platform’s answer. Startups with beautiful parallax homepages that never explain the product are a recurring failure case.
llms.txt is useful for loading prose-heavy documentation into a context window, not current default discovery; oversized llms-full.txt files can exceed common context windows anyway. Embedding-similarity optimization is similarly peripheral and may hurt. Ordinary clean HTML organized for humans remains the stronger substrate.
Most AI retrievers do not execute JavaScript. A stray client-side effect that breaks server rendering can make an entire Next.js page unavailable, so static or server-rendered output is typically necessary. Programmatic SEO is durable when it exposes proprietary insight—support-ticket-derived how-tos are Robert’s best example—not when it endlessly remixes public text “using LLMs in order to be read by LLMs.”

8. Deep research magnifies both useful facts and stale contradictions

Robert separated three modes: regular search retrieves and summarizes one search; multi-search, increasingly visible in regular ChatGPT 4, AI Mode, and more recent o3 integrations, runs several searches; deep research follows up sequentially with a reasoning model—“These sources didn’t answer the question. Let me try something else.”
The ingestion fundamentals remain similar. Its larger appetite can produce worse answers when extra pages add noise: old pricing pages are especially dangerous because conflicting figures increase the chance that ChatGPT presents the wrong price.
Swyx proposed guiding the next search with related links; Robert replied, “Why not just include the content in the first place?” He conceded the branching and context-window limits, favoring focused pages that completely answer a query for a particular persona over a sprawling “choose your own adventure.”

9. Conversion data validates the thesis, while platform loyalty sets priorities

Clerk provides the clearest operating proof. Its strong developer documentation gave it a favorable starting point, but targeted experimentation produced approximately 6x growth in ChatGPT traffic and 9x growth in conversions. A query such as “How do I implement enterprise SSO in my app?” comes from someone with an immediate problem and readiness to implement.
Robert carefully limited Scrunch’s credit: Clerk knows its developers and creates the content; Scrunch supplies monitoring, feedback, and experimentation. The roadmap is to put parallel testing and delivery “more on rails,” while investigating mechanisms such as MCPs and NLWeb under the broader agent-experience thesis.
Outcomes vary by vertical, and Robert would not promise Clerk’s lift broadly. Still, companies actively updating sites and publishing material typically see double-digit traffic improvements within one or two months; businesses staying in wait-and-see mode should not expect the same result.
Platform priority starts with ChatGPT “by far and away,” followed by enormous imposed reach from AI Overviews, AI Mode, and Meta AI. Perplexity is the second-largest AI-native search platform and unusually sticky; Claude has fewer but passionate, valuable users. Gemini, DeepSeek, and Grok have shown more peaks and troughs, while ChatGPT’s relationship is durable—though “not infinitely so.”

Demis Hassabis: Future of AI, Simulating Reality, Physics and Video Games | Lex Fridman Podcast #475 Random Ramblings July 2025