40% of Major Websites Are Blocking AI, But No One Knows How to Charge the Machines

Deep thoughts on AI and aspirations —— ByteDance Deep Thinking Circle

In July 2024, only 9% of the world’s top 10,000 websites blocked AI crawlers. A year later, that number is 37%, and it’s still rising.

Website owners aren’t blocking a specific bot—they’re responding to a new reality: nearly half of all internet traffic no longer comes from humans. These machine visitors consume content to feed models or perform tasks on behalf of people, paying nothing while websites bear the bandwidth and server costs.

Most discussions frame this as an arms race: crawlers getting better at hiding, blocks getting more aggressive. But the arms race is just the surface. The real problem is that the internet was designed from the ground up for human users. Pricing, payment, identity—all assume there’s a person on the other end. When that assumption breaks, the entire commercial logic fails, more thoroughly than most imagine.

Three Patches the Internet Built for Humans That Machines Can’t Use

Start with advertising. The internet’s dominant business model is “you consume content, I sell your attention.” This model is completely ineffective for machines: crawlers don’t view ads, don’t generate attention, and advertisers won’t pay for machine visits. Websites whose content gets scraped are essentially moving bricks for free.

Then payment. Human users have bank cards and subscription accounts—the internet’s commercial loop is built on these. AI agents have no bank accounts, can’t open them, and can’t pass KYC. Getting an agent to pay per page scraped has no corresponding interface in today’s financial system: transactions are too small, frequency too high, and the entity isn’t human.

Finally, identity. Bots can replicate infinitely—one server can “become” ten thousand users overnight. Distinguishing human from machine currently relies on old methods like CAPTCHAs, which new-generation models can already solve. Deepfakes and social media bots make it worse: not only can’t you tell who’s on the other end, you can’t even tell if they’re human.

When all three systems fail simultaneously, you have the root of the past two years’ conflict. Law moved first: multiple US newspapers sued model companies for copyright infringement, education sites saw traffic plummet as students switched to AI tools and simply put up walls. The rise from 9% to 37% blocking is the direct consequence of this litigation and blockade war.

But litigation can’t settle the accounts. A lawsuit can determine a lump-sum settlement but can’t calculate what any single scrape is worth or which layer in the chain should get paid. If a dataset has been referenced by thousands of crawlers, how do you attribute contribution and calculate splits? Courts can’t provide that granularity. Pricing ultimately needs a technical solution.

Machine Users Need Machine-Native Infrastructure

This is where blockchain has an opportunity. Its role here isn’t about token prices—it’s about rebuilding three things for the machine world: machine payment, machine identity, and machine-readable property records.

On payment, digital cash systems happen to fill the gap in the financial system: near-zero-cost micropayments, programmable revenue splits, and the ability to hold value without a bank account. A crawler agent could carry an on-chain balance, negotiate in real-time with each website’s access protocol, and pay per page scraped; smart contracts could automatically split a single payment among multiple upstream content sources. Protocols like x402 are already moving in this direction. Human users could take a different path: use decentralized identity tools like “proof of personhood” to prove they’re human and continue accessing for free. For the first time, websites could price machine traffic and human traffic separately.

Identity works similarly. AI agents need a cross-platform “digital passport”: who issued it, what version, what capabilities, what reputation history—something any interface can parse uniformly. Without this public identity layer, agents must negotiate from scratch with each system, reputation resets with each channel switch, and discovery mechanisms remain ad hoc patchwork. Property records are the same: ownership and licensing of training data, model weights, and generated content—if machines are to execute this automatically, it must be written in a machine-readable, tamper-proof registry format. Early attempts like Story Protocol and on-chain IP registration are betting on this layer.

But let’s throw cold water on this. The list of Crypto + AI combinations can run long, from decentralized compute to on-chain AI companions, each sounding like the future. Most entries can’t withstand one question: Is this a new problem brought by machines?

That’s my only filter. Machine payment, machine identity, machine-readable rights—these contradictions weren’t acute before machines became the internet’s primary users, traditional solutions genuinely can’t handle them, and blockchain has a chance. Decentralized compute, privacy advertising, on-chain AI companions—these problems either existed long ago or blockchain is optional in the solution, added more for narrative completeness. To judge whether a cross-domain scenario is real, drop it into a pure Web2 environment and see if it breaks: what breaks is a real need, what can be worked around is decoration.

There’s another easily overlooked layer: blockchain’s real selling point in these scenarios isn’t performance. For user experience, vertically integrated big platforms always do it better—Amazon and Facebook’s closed identity systems remain the optimal experience today. What blockchain sells is “no one’s in charge.” When the internet’s negotiating table has hundreds of thousands of small and medium websites on one side and a handful of model giants on the other, a neutral middle layer has value. This is essentially a political antitrust problem, just solved incidentally through technical means.

Three Real Problems and Their Respective Tipping Point Conditions

When will the tipping point come? Watch three signals. First, whether mainstream hosting services (CDN layer like Cloudflare) start building in paid crawler protocols—robots.Txt is a gentleman’s agreement from the nineties that won’t evolve into a billing system on its own. Second, whether proof-of-personhood identity user volume crosses critical mass—this infrastructure has network effects, slow early but suddenly accelerates past the threshold. Third, whether the first killer scenario emerges where “high-quality content can’t be scraped without payment.”

Until then, most of these attempts will remain experimental, measured in years, not quarters.

For people building agents and people investing in agents, this assessment has practical implications. Machine-native payment and identity will likely be a late-arriving but unavoidable piece of agent infrastructure: teams treating it as a supporting role now will find the price higher than expected when the rails are built. Conversely, if you’re just adding a blockchain layer to your product whitepaper for AI narrative purposes, save yourself the effort. Machines don’t need narratives. Machines only recognize interfaces.