In a landmark move aimed at shifting power back to digital content owners, Cloudflare has announced it will block all AI crawlers by default across its customer websites, while introducing a new “pay per crawl” model that enables creators to monetize access to their web content. The announcement marks a significant turning point in the relationship between content creators and artificial intelligence systems, many of which have been freely scraping the internet to train their models. Cloudflare’s approach gives website owners more granular control including allowing, denying, or charging AI companies for content access.
“If a creator wants to block all AI crawlers from their content, they should be able to do so. If a creator wants to allow some or all AI crawlers full access to their content for free, they should be able to do that, too. Creators should be in the driver’s seat,” Cloudflare said in its official report.
The “pay per crawl” model is designed to integrate smoothly with existing web infrastructure. It uses HTTP status codes and familiar authentication methods to facilitate paid access. When an AI crawler attempts to access content, it must include a payment intent in the request header; otherwise, it receives a 402 Payment Required response, indicating pricing information and access terms.
The move has received a strong endorsement from the cybersecurity community. Ray Canzanese, Director of Netskope Threat Labs, applauded the move, likening AI’s voracious appetite for content to a “Pac-Man” effect: “The rapid development of AI has seen it behaving a little like Pac-Man – consuming everything in its path. We are now finally starting to see a correction, where organisations, who are realising exactly how much value lies within their data, are regaining control over it. Whether it be publicly viewable, such as pricing intelligence on online retailer sites, copyrighted but ungated, for instance a digital newsletter, or privately held, which can include internal documentation, source code, and more, data has value to both the company that owns it and to AI systems that may scrape from and train on it.”
“Blocking AI crawlers from accessing website content without the owners’ permission is the latest in efforts by the industry to place the power back where it belongs, with the data owners. Defaulting to block is important, as it allows organisations to reassert their control and ownership without having to take any explicit actions, providing some breathing room while they work out a mutually beneficial relationship with AI. We hope that this is just one of many changes we will see in the coming months to help organisations reassert ownership and control over their data,” he added.
As a cybersecurity leader, Netskope says that it supports initiatives that prioritize data ownership, especially amid the growing influence of generative AI and autonomous AI agents across enterprises. The company called for broader adoption of similar standards that build healthier, permission-based relationships between data creators and AI systems.
