Cloudflare makes AI crawlers opt-in, giving power to creators

Cloudflare has introduced a default setting to block AI crawlers from accessing web content without explicit permission, making it the first internet infrastructure provider to take this step.

With this new measure, website owners using Cloudflare's services will have the choice to allow or block AI crawlers, moving from a previous opt-out system to an opt-in approach. This change is designed to address issues concerning the unauthorised scraping and usage of web content by AI companies for purposes such as training and inference, often without the knowledge or compensation of the content creators.

Permission-based controls

Under the new system, AI companies are now required to disclose the purpose of their crawlers, specifying whether they are used for training, inference, or search. This allows website owners to make more informed decisions about which bots may access their data. Cloudflare is also developing a "Pay Per Crawl" feature that will give content creators the ability to request payment from AI companies for access to their content, which could generate new revenue streams for publishers.

Cloudflare's Chief Executive Officer and Co-founder, Matthew Prince, stated:

"If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, tomorrow's AI founders, and the future of the web itself. Original content is what makes the Internet one of the greatest inventions in the last century, and it's essential that creators continue making it. AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone."

This revised approach follows previous Cloudflare initiatives to block AI crawlers, which began with a one-click option introduced in September 2024. Since then, more than one million customers have chosen to restrict AI crawlers from their websites. Now, blocking occurs by default for all new customers, eliminating the need for domain owners to adjust settings to prevent unauthorised crawling.

Support from publishers

Prominent media organisations and publishers have expressed support for Cloudflare's move, including ADWEEK, SkyNews, Fortune, The Associated Press, BuzzFeed, The Atlantic, TIME, Reddit, and Pinterest. These companies have advocated for fair compensation frameworks and greater transparency around how content is accessed and used by AI platforms.

Roger Lynch, Chief Executive Officer of Condé Nast, commented:

"Cloudflare's innovative approach to block AI crawlers is a game-changer for publishers and sets a new standard for how content is respected online. When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation built on permission and partnership. This is a critical step toward creating a fair value exchange on the Internet that protects creators, supports quality journalism and holds AI companies accountable."

Neil Vogel, Chief Executive Officer of Dotdash Meredith, added:

"We have long said that AI platforms must fairly compensate publishers and creators to use our content. We can now limit access to our content to those AI partners willing to engage in fair arrangements. We're proud to support Cloudflare and look forward to using their tools to protect our content and the open web."

Renn Turiano, Chief Consumer and Product Officer at Gannett Media, also noted:

"As the largest publisher in the country, comprised of USA TODAY and over 200 local publications throughout the USA TODAY Network, blocking unauthorised scraping and the use of our original content without fair compensation is critically important. As our industry faces these challenges, we are optimistic the Cloudflare technology will help combat the theft of valuable IP."

Bill Ready, Chief Executive Officer of Pinterest, said:

"Creators and publishers around the world leverage Pinterest to expand their businesses, reach new audiences and directly measure their success. As AI continues to reshape the digital landscape, we are committed to building a healthy Internet infrastructure where content is used for its intended purpose, so creators and publishers can thrive."

Steve Huffman, Co-founder and Chief Executive Officer of Reddit, stated:

"AI companies, search engines, researchers, and anyone else crawling sites have to be who they say they are. And any platform on the web should have a say in who is taking their content for what. The whole ecosystem of creators, platforms, web users and crawlers will be better when crawling is more transparent and controlled, and Cloudflare's efforts are a step in the right direction for everyone."

Vivek Shah, Chief Executive Officer of Ziff Davis, commented:

"We applaud Cloudflare for advocating for a sustainable digital ecosystem that benefits all stakeholders — the consumers who rely on credible information, the publishers who invest in its creation, and the advertisers who support its dissemination."

Industry consortia and authentication

Cloudflare is also participating in the development of new technical protocols to allow AI bots to authenticate themselves and for website owners to reliably determine the identity and intent of incoming requests. This aims to improve overall transparency and control over the use of web content by automated agents.

Additional media and technology companies have added their support, indicating a broad industry move towards permission-based AI access to digital content. The list includes companies such as The Arena Group, Atlas Obscura, Quora, Stack Overflow, Universal Music Group, O'Reilly Media, and others.

This change comes as publishers report reduced website traffic and declining advertising revenues linked to AI platforms generating answers directly to user queries without referencing or referring traffic to the original sources. Cloudflare's new default blocking of AI crawlers aims to restore a value exchange between content creators, consumers, and technology companies as artificial intelligence continues to shape the internet landscape.