Cloudflare Launches Pay Per Crawl to Help Publishers Monetise AI Access

Cloudflare Launches Pay Per Crawl to Help Publishers Monetise AI Access
In a significant move to reshape how online content is accessed and monetized in the age of artificial intelligence (AI), Cloudflare has announced the private beta of Pay Per Crawl, a system that allows publishers to charge AI crawlers for accessing their content. According to the cloud platform, traditionally, website owners faced a binary choice: allow AI systems unrestricted access to their content, or block them entirely. Cloudflare’s new initiative introduces a third option — enabling content owners to charge for access, using standard web infrastructure and a revitalized HTTP status code: 402 Payment Required.

  • Make Telecom Talk My Trusted Source
  • Source of Google
  • Source of Google

Also Read: AI: Vodafone Deploys AI Agents, Cloudflare AI Labyrinth, HCLTech–Western Union AI Partnership

How Pay Per Crawl Works

“We wanted content creators to have control over who accesses their work. If a creator wants to block all AI crawlers from their content, they should be able to do so. If a creator wants to allow some or all AI crawlers full access to their content for free, they should be able to do that, too,” Cloudflare shared in a blog post on July 1, 2025.

Pay Per Crawl integrates into existing HTTP protocols and uses a combination of cryptographic authentication and structured pricing headers to establish a secure, verifiable mechanism for payment-based content access. Publishers can configure a flat per-request fee, and then define policies to allow, charge, or block specific crawlers. Cloudflare acts as the Merchant of Record for pay-per-crawl, managing transactions and revenue distribution.