Cloudflare’s new policy pushes AI companies to pay for publishers’ content
Cloudflare is giving AI companies until September 15 to separate web crawlers used for search from those used for AI training and agents, or risk being blocked by default on many publisher sites.
Cloudflare has just issued the AI industry a new deadline to separate the web crawlers used for traditional search purposes, like Google Search, from those used for AI agents and training. Starting on September 15, 2026, Cloudflare’s default settings will block “mixed-use” crawlers from any pages that host ads, the company announced on Wednesday.
That means that the crawlers that blend search, agent use, and training will be blocked from crawling these sites by default, unless the site owner adjusts the settings otherwise. These changes to the defaults will apply to new Cloudflare customers, new sites set up by existing customers, and all existing free customers, the company says.
The move could impact how AI model providers are able to access web content for training purposes and to help power their agentic services.
Cloudflare points out that most website owners want their content to be discoverable via search and often through AI services as well, but they want protections against having their intellectual property given away for free.
Cloudflare specifically calls out the “world’s largest search engine” (clearly a Google reference!) as having access to about “2x more information” than other AI companies because the search giant makes it difficult for customers to remain discoverable without being used for AI.
Google has pushed back against this generalization in the past, noting that it provides a bot called Google Extended that lets site owners opt out of having their content used for training and AI products and services like Gemini Apps and Vertex API. Its use doesn’t impact a site’s inclusion in Google Search. However, the tech giant’s flagship Googlebot crawls for Search, including AI features like AI Overviews and AI Mode.
“Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” said Cloudflare co-founder and CEO Matthew Prince in his announcement of the news, referring to the recent milestone where bots surpassed human traffic online for the first time. That shift was not expected to occur until next year.
“Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed-use crawlers to separate out search from agent use and training,” Prince said.
While Cloudflare offers a number of products to help users launch their own AI systems , the company has also released a range of tools to give publishers more control over their content in the AI era. In recent years, Cloudflare launched tools to combat AI bots , including a marketplace that lets websites charge AI bots for scraping , dubbed Pay Per Crawl.
The latter is now also evolving into “Pay Per Use,” the company said, which will allow publishers to charge AI companies when their content creates value, not just when it’s fetched.
The change could also help conserve publishers’ bandwidth and compute resources for AI model providers, as Cloudflare’s data suggested that over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages.
To put this into action, Cloudflare is initially working with two partners, Ceramic.ai and You.com. When a publisher opts in, they’re paid when their content appears in Ceramic’s AI search results or when You.com accesses a piece of their premium content.
Other AI companies can customize this model for how they work, Cloudflare says.
When you purchase through links in our articles, we may earn a small commission . This doesn’t affect our editorial independence.
Last chance to save up to $190 on TechCrunch Founder Summit. Join 1,000+ founders and VCs at all stages for real-world scaling insights and connections that move the needle. Savings end June 26, 11:59 p.m. PT .
Flipper Device’s new Busy Bar is a customizable display for productivity Ivan Mehta
Flipper Device’s new Busy Bar is a customizable display for productivity
Flipper Device’s new Busy Bar is a customizable display for productivity
Ford rehires ‘gray beard’ engineers after AI falls short Anthony Ha
Ford rehires ‘gray beard’ engineers after AI falls short
Ford rehires ‘gray beard’ engineers after AI falls short
Govee’s smart nugget ice maker makes every iced drink feel like a luxury Aisha Malik
Govee’s smart nugget ice maker makes every iced drink feel like a luxury
Govee’s smart nugget ice maker makes every iced drink feel like a luxury
Instagram is testing more ways to customize ‘Your Algorithm’ Anthony Ha
Instagram is testing more ways to customize ‘Your Algorithm’
Instagram is testing more ways to customize ‘Your Algorithm’
Asian AI startups launch Mythos-like models as Anthropic’s export ban drags on Kate Park
Asian AI startups launch Mythos-like models as Anthropic’s export ban drags on
Asian AI startups launch Mythos-like models as Anthropic’s export ban drags on
FTC gives Musk the OK to acquire SpaceX alumni startup Mesh Marina Temkin
FTC gives Musk the OK to acquire SpaceX alumni startup Mesh
FTC gives Musk the OK to acquire SpaceX alumni startup Mesh
Corgi, the buzzy Y Combinator-backed insurance tech startup, says it didn’t steal an open source product Julie Bort
Corgi, the buzzy Y Combinator-backed insurance tech startup, says it didn’t steal an open source product
Corgi, the buzzy Y Combinator-backed insurance tech startup, says it didn’t steal an open source product
Key takeaways
- Cloudflare mandates separation between search and AI crawlers, impacting data access.
- The change may stimulate a debate on digital content monetization in Brazil.
- AI companies will need to adapt their practices to avoid being blocked on publisher sites.
Editorial analysis
Cloudflare's new policy marks a significant turning point in the relationship between AI companies and content publishers, especially in a landscape where intellectual property protection is becoming increasingly critical. For the Brazilian tech sector, this change could stimulate a broader debate on digital content monetization and the role of intermediary platforms. The need to separate search crawlers from AI training crawlers may lead Brazilian companies to reconsider their online visibility strategies, particularly those that rely on traffic generated by search engines and AI services.
Moreover, Cloudflare's decision could have direct implications for how AI companies access data for training. With the potential blocking of mixed-use crawlers, companies that do not adapt to this new reality may struggle to obtain the necessary information for developing AI models. This could create a scenario where only companies that invest in transparent and ethical data collection practices can thrive, which in turn may benefit the quality and diversity of AI services available in the market.
Finally, it is important to observe how this policy will be received by other platforms and tech companies. By highlighting the issue of non-human traffic, Cloudflare is signaling a broader movement towards regulating data use in the AI era. Brazil, with its growing tech and innovation industry, should closely monitor these changes, as they may influence how startups and established companies operate in the digital ecosystem. What is now expected is a market response and how Brazilian companies will position themselves in light of these new demands and challenges.
What this coverage includes
- Clear source attribution and link to the original publication.
- Editorial framing about relevance, impact, and likely next developments.
- Review for readability, context, and duplication before publication.
Original source:
TechCrunch AIAbout this article
This article was curated and published by AIDaily as part of our editorial coverage of artificial intelligence developments. The content is based on the original source cited below, enriched with editorial context and analysis. Automated tools may assist with translation and initial structuring, but publication decisions, factual review, and contextual framing remain editorial responsibilities.
Learn more about our editorial process