Support Telecomtalk
Andrew Bonwick
Vice President of Product Development at Relm Insurance
Madhav Sheth
CEO of Ai+ Smartphone
Stephen Rose
CEO Render Networks
In a significant move to reshape how online content is accessed and monetized in the age of artificial intelligence (AI), Cloudflare has announced the private beta of Pay Per Crawl, a system that allows publishers to charge AI crawlers for accessing their content. According to the cloud platform, traditionally, website owners faced a binary choice: allow AI systems unrestricted access to their content, or block them entirely. Cloudflare’s new initiative introduces a third option — enabling content owners to charge for access, using standard web infrastructure and a revitalized HTTP status code: 402 Payment Required.
- Make Telecom Talk My Trusted Source
Also Read: AI: Vodafone Deploys AI Agents, Cloudflare AI Labyrinth, HCLTech–Western Union AI Partnership
How Pay Per Crawl Works
“We wanted content creators to have control over who accesses their work. If a creator wants to block all AI crawlers from their content, they should be able to do so. If a creator wants to allow some or all AI crawlers full access to their content for free, they should be able to do that, too,” Cloudflare shared in a blog post on July 1, 2025.
Pay Per Crawl integrates into existing HTTP protocols and uses a combination of cryptographic authentication and structured pricing headers to establish a secure, verifiable mechanism for payment-based content access. Publishers can configure a flat per-request fee, and then define policies to allow, charge, or block specific crawlers. Cloudflare acts as the Merchant of Record for pay-per-crawl, managing transactions and revenue distribution.
Crucially, the system prevents crawler spoofing by requiring authenticated requests signed with Ed25519 keys. AI companies must register with Cloudflare and implement signed headers to identify themselves before accessing paid content.
