As AI tools rapidly reshape how people access information online, Cloudflare is proposing a new deal for content creators: get paid when AI crawlers use your content. On Tuesday, the cloud infrastructure giant — which serves around 20% of the entire web — launched Pay per Crawl, a first-of-its-kind marketplace designed to give publishers more control and compensation in the age of AI.
This experimental platform, now in private beta, allows website owners to decide how AI crawlers interact with their content. They can choose to charge per crawl, allow free access, or block AI bots altogether. It’s a flexible framework aimed at solving a growing problem: AI companies are scraping vast amounts of web content to train models and serve chatbot responses — often without permission or payment.
With Pay per Crawl, Cloudflare hopes to rewire that relationship by giving publishers the tools to see who’s crawling their site, understand why it’s being scraped, and decide whether (and how) to monetize it.
From Blocking AI Bots to Building a New Business Model
Cloudflare has spent the last year rolling out tools that help publishers track and block AI crawlers. Features like one-click AI bot blocking and detailed analytics have already been adopted by major outlets such as TIME, The Atlantic, ADWEEK, Fortune, and The Associated Press. Now, these tools form the foundation of a much bigger idea.
Pay per Crawl acts as a real-time marketplace between AI companies and publishers. Once both sides set up their Cloudflare accounts, publishers can set micropayment rates for their content. AI firms, in turn, can choose which sites to crawl — and pay accordingly. Cloudflare sits in the middle, managing payments and access.
This move comes as publishers are grappling with existential shifts in their traffic sources. The Google Search model, where content is scraped in exchange for referral traffic and ad revenue, is breaking down. New AI search and chatbot tools, from OpenAI to Anthropic, often extract answers directly from content — without sending users back to the source.
Cloudflare’s data paints a stark picture. In June, it found that:
- Google scraped websites 14 times for every one referral
- OpenAI scraped content 1,700 times per referral
- Anthropic performed an astonishing 73,000 scrapes for each referral
That imbalance signals a troubling trend: publishers are providing the fuel for AI without any compensation or return traffic.
Turning the Tables on AI Crawling
By default, new websites using Cloudflare will now block all AI crawlers unless given explicit permission. Cloudflare is calling this the “default of control,” a shift that puts the power in the hands of site owners rather than AI companies.
The broader vision is to turn crawling into a permission-based economy. Much like how ad platforms evolved to respect user preferences and privacy laws, Cloudflare believes the AI ecosystem needs to mature — with publishers deciding who gets to use their content and at what cost.
CEO Matthew Prince hinted at this shift as early as 2024, calling for a marketplace where content owners could license data directly to AI firms. The launch of Pay per Crawl marks the first step in making that concept a reality.
But this new model won’t be without hurdles. Convincing AI companies to pay for data they’ve previously accessed for free could be difficult. At the same time, it’s unclear whether micropayments will generate meaningful revenue for smaller publishers.
Still, Prince believes the market could thrive in an “agentic” future — one where AI agents autonomously browse the web on users’ behalf, pulling together research or recommendations. In that future, automated microtransactions might power everything from deep medical queries to travel planning — rewarding the content creators along the way.
Cloudflare is even considering launching its own stablecoin to support seamless payments in this marketplace, though for now, no cryptocurrency is involved in the beta.
A Bold Play in the AI Content Economy
While major publishers like The New York Times have opted to sue AI firms over unauthorized training, others have quietly signed multi-year licensing deals. But those agreements are limited to the largest players. For the rest of the web — independent newsrooms, educators, niche bloggers — Pay per Crawl could be the first real chance to monetize AI access at scale.
Cloudflare may be uniquely positioned to build this marketplace. With millions of websites already using its tools, it has the infrastructure and trust to broker a new kind of deal — one where publishers don’t just get scraped, but get paid.
Whether this becomes a new revenue stream or a short-lived experiment remains to be seen. But Cloudflare’s effort to rethink content value in the AI age is a bold move — and one publishers can’t afford to ignore.
