The Free Ride for AI Crawlers Just Ended

Up until now, if your site ran ads and an AI company’s bot came crawling through to gather training data, there wasn’t much stopping it by default. Cloudflare just changed that, and the change goes live today.

Starting now, Cloudflare is defaulting to blocking what it calls mixed-use crawlers, bots that quietly blend regular search indexing with AI training and agent data collection, on any page that carries ads. Search crawlers still get through. The ones scraping your content to feed a model or an autonomous agent do not, unless a site owner opts back in.

This matters because most publishers never had the technical know-how, or honestly the time, to sort out which bots were doing what. A lot of AI crawlers used to disguise themselves as regular search traffic. Now the default protects the side that was getting the short end of that deal.

It applies to new Cloudflare customers, any new site set up by an existing customer, and every free-tier customer already on the platform. If you’re a paid customer with your settings already dialed in, nothing changes for you automatically.

Cloudflare is pairing this with a bigger idea too, turning its old pay-per-crawl marketplace into something closer to pay-per-use, so publishers can actually get paid when their work trains someone else’s model. Whether that catches on is the real question. But defaulting to protect creators first, instead of asking them to opt in after the fact, is the right instinct, and I’d like to see more of the internet built that way.

None of this means AI training grinds to a halt, and it shouldn’t. Data still moves, deals still get made, agents still get built. What changes is who gets asked first. A small blog owner who never touched a robots.txt file in their life just got a layer of protection they didn’t have to go looking for, and that’s the kind of default I want to see more platforms lean toward as this whole industry keeps growing up.

——

Follow: @Ali Demi
Book your free AI clarity call, NOW!
https://buff.ly/TpWy277

——

Sources:
https://blog.cloudflare.com/content-independence-day-ai-options/
https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/
https://www.mitsloanme.com/article/cloudflare-puts-publishers-in-control-with-default-block-on-ai-web-crawlers/

Repost this. Thanks.