Cloudflare launches ‘Pay per Crawl’ to help sites monetise AI access
Cloudflare is testing a new way for websites to charge AI companies that use their content to train models, run inference, or power search. The new feature, called Pay per Crawl, lets website owners set their own terms for how and when AI bots can access their sites.
By default, Cloudflare now blocks AI crawlers from accessing websites without permission. Site owners can choose whether to allow bots and decide if access should be free or paid. AI companies, in turn, must disclose whether their crawlers are being used for training, inference, or search, giving publishers more context to make decisions.
The idea is to help publishers regain some control over how their content is used in the AI economy. Many websites today are being scraped by AI models without compensation, even as those same models pull traffic away from traditional search engines.
For the past year, Cloudflare has rolled out tools to block or monitor AI crawlers. That includes a one-click setting to stop AI bots entirely and a dashboard that shows who is scraping what. Pay per Crawl builds on that work and puts a price on access — even if it’s just a few cents per visit.
In the private beta, publishers can choose to allow individual crawlers, block them, or set rates for each crawl. Cloudflare serves as the go-between, handling payments and forwarding revenue to website owners. Both the publisher and the AI company need Cloudflare accounts to take part.
The company says the system could be useful in a future where AI agents — not people — visit websites, gather data, and deliver summaries or answers directly to users. In that scenario, users might give their AI agents a budget to pull high-quality content from trusted sources. Cloudflare believes a system like Pay per Crawl could support that kind of use.
There’s growing tension around how AI firms use online content. Some large publishers like The New York Times have sued tech firms for scraping articles without permission. Others have signed licensing deals to supply training data and let their content show up in chatbot responses. But those agreements are limited to major players, and it’s unclear how much revenue they actually generate.
Smaller publishers haven’t had many options. Pay per Crawl could give them a chance to charge for their content directly, rather than relying on middlemen or hoping for referral traffic.
But convincing AI firms to pay won’t be easy. Many of them are already scraping content without permission. And the economics may not work in publishers’ favour — at least not yet.
Cloudflare shared new numbers that show how rarely publishers get traffic back from AI bots. According to the company, Google’s crawler scraped sites 14 times for every one click it sent back. OpenAI’s crawler scraped sites 1,700 times per referral, and Anthropic scraped sites 73,000 times per referral. For publishers relying on traffic to support ad revenue, that’s a bad deal.
For years, publishers have let search engines like Google crawl their sites for free in exchange for visibility and clicks. That model is starting to break down as AI tools replace search — pulling answers directly from crawled content without sending users to the original source.
Cloudflare’s default for new websites is now to block AI crawlers. Site owners will have to manually allow specific bots. The company calls this a “default of control.”
A long list of media and content organisations have expressed support for stronger protections over how their work is used, including BuzzFeed, TIME, Reddit, Condé Nast, and Stack Overflow.
Whether Cloudflare’s model will work at scale is still uncertain. It depends on publishers setting prices that make sense and on AI companies being willing to pay. Cloudflare CEO Matthew Prince also said the company is exploring the use of stablecoins to support these transactions, either by creating its own or partnering with other providers.
AI agents are likely to play a bigger role in how users access information online. If that’s the case, Cloudflare’s system could be an early test of whether it’s possible — or practical — to charge for the content that powers them.
0 Comments