Cloudflare Launches Pay Per Crawl to Help Publishers Monetise AI Access

In a significant move to reshape how online content is accessed and monetized in the age of artificial intelligence (AI), Cloudflare has announced the private beta of Pay Per Crawl, a system that allows publishers to charge AI crawlers for accessing their content. According to the cloud platform, traditionally, website owners faced a binary choice: allow AI systems unrestricted access to their content, or block them entirely. Cloudflare’s new initiative introduces a third option — enabling content owners to charge for access, using standard web infrastructure and a revitalized HTTP status code: 402 Payment Required.

• Make Telecom Talk My Trusted Source
• 
• 

Also Read: AI: Vodafone Deploys AI Agents, Cloudflare AI Labyrinth, HCLTech–Western Union AI Partnership

How Pay Per Crawl Works

“We wanted content creators to have control over who accesses their work. If a creator wants to block all AI crawlers from their content, they should be able to do so. If a creator wants to allow some or all AI crawlers full access to their content for free, they should be able to do that, too,” Cloudflare shared in a blog post on July 1, 2025.

Pay Per Crawl integrates into existing HTTP protocols and uses a combination of cryptographic authentication and structured pricing headers to establish a secure, verifiable mechanism for payment-based content access. Publishers can configure a flat per-request fee, and then define policies to allow, charge, or block specific crawlers. Cloudflare acts as the Merchant of Record for pay-per-crawl, managing transactions and revenue distribution.

Crucially, the system prevents crawler spoofing by requiring authenticated requests signed with Ed25519 keys. AI companies must register with Cloudflare and implement signed headers to identify themselves before accessing paid content.

Addressing Publisher Concerns About AI Scraping

According to Cloudflare, the initiative responds to growing concerns from publishers, news organizations, and content creators who want fair compensation when AI models use their content. Current licensing models are fragmented and depend on private negotiations. Cloudflare’s model introduces a scalable alternative, allowing even smaller publishers to participate in monetizing their digital assets.