On September 15 Cloudflare starts blocking AI agents by default on any page that runs ads. That covers about one in five websites. Here are the two ways through, and the checklist to test both before the switch flips.
You have an agent that reads the web for you. It pulls research. It checks a competitor's page. It reads an article and hands you the summary. Today it works.
On September 15 a lot of those pages stop answering it.
Cloudflare sits in front of roughly one in five websites. On that day it starts blocking AI agents by default on any page that runs display ads. That is not one site opting out. It is an entire fifth of the web going quiet for your agents at once. If you run agents for your own operations or for clients, this is the week to find out which of your workflows break and how to keep them alive.
What Is Changing, And When
September 15, 2026. Put it on the calendar.
Cloudflare is flipping a default. From that date, any web page behind Cloudflare that runs display advertising blocks AI agents unless the site owner says otherwise. The owner can still let specific agents through. The point is that the starting position moves from allow to block.
One in five websites covers a lot of ground. Competitor pages, industry news, research sources, SaaS marketing sites. Most of them carry ads, so most of them fall inside the new default.
The important word is default. Nobody has to lift a finger for the block to happen. It just turns on. That is what makes September 15 a real deadline instead of a headline.
Why Your Agents Are Getting Blocked
This is about ad money, not a fear of robots.
When a person opens a page, they see the ads. The owner gets paid. When your agent opens the same page, it lifts the text and skips the ads. The owner gets nothing back. Multiply that across every agent reading their content and you can see why they want a gate.
Cloudflare is giving site owners an automated way to enforce something they always wanted: traffic that pays its way. An ad-supported page was built for human attention. An agent that takes the content without generating any ad revenue looks like a free rider, and the block treats it as one.
You do not have to agree with the logic to plan around it. The logic is set. Your job is to keep your agents working inside it.
What Actually Breaks
Think through what your agents do all day, then mark the ones that read live web pages.
- Research agents. They pull from news sites, blogs, and reports. Most of those run ads.
- Competitor monitors. Competitor marketing pages and pricing pages sit behind Cloudflare more often than not.
- Read-and-summarize workflows. The classic "go read this and tell me what matters" task. It reads an ad page, so it hits the wall.
- Price trackers. Retail and marketplace pages with ad slots stop returning clean data.
- News aggregators. Ad-supported by design. These are among the first to go dark.
If you run these for clients, assume close to every web-reading task you deliver is affected. Anything that quietly fetches a page in the background is in scope. The failure is not loud. The agent asks for the page and gets a blocked response instead of content, and the workflow either stalls or returns thin, wrong answers.
There are two ways through. One trades on identity. One trades on payment. You will likely use both.
Path One: Web Bot Auth (Verified Identity)
Give your agent a credential the site trusts.
Web Bot Auth is Cloudflare's verification layer for legitimate automated agents. Instead of looking like an anonymous scraper, your agent presents a cryptographically signed identity on every request. The site owner can choose to allow verified agents through even while anonymous ones stay blocked. You stop being a stranger at the door.
How it works
- Your agent registers and receives a signed credential tied to an identity, a token or certificate.
- On every request to a Cloudflare-protected page, the agent presents that credential in its request headers.
- Cloudflare validates the credential and routes the agent through, as long as the site owner has opted in to allow verified bots.
| Consideration | Detail |
|---|---|
| Cost model | Access is granted on identity. No per-page charge once you are registered. |
| Site coverage | Only pages where the owner has opted in to allow verified agents. |
| Best for | Agents running at volume against a small, known set of target sites. |
| Setup effort | One-time registration, then inject the credential into your agent's headers. |
Your first move. Pick the one agent you would miss most if it went blind tomorrow. List the sites it reads. If that list is short and stable, register it for Web Bot Auth first. This is the path that pays off when the same agent hits the same handful of sources over and over.
The mistake that makes it fail. Assuming a credential opens every door. It does not. Web Bot Auth only works where the site owner has switched on "allow verified agents". A site that blocks all agents, verified or not, stays blocked. Treat this path as your answer for proprietary or industry sources where the owner has a business reason to let real tools in, not as a universal key.
Path Two: Pay-Per-Use Crawling
If identity will not open the door, pay to read.
Cloudflare is launching a pay-per-use crawling service. Your agent routes its read requests through Cloudflare's crawling API and pays a small fee per page. Cloudflare returns the content, and the site owner takes a cut of that payment. The ad problem gets solved by replacing the ad with a direct payment. Nobody is skipping revenue anymore, so the block lifts.
How it works
- Integrate Cloudflare's crawling API endpoint into your agent's HTTP layer.
- On requests to blocked pages, the agent routes through the paid service instead of hitting the site directly.
- Cloudflare bills per successful page fetch and returns the content as structured data or raw HTML.
| Consideration | Detail |
|---|---|
| Cost model | Pay per page read. The bill scales with how much your agent reads. |
| Site coverage | All Cloudflare-protected pages, whether or not the owner opts in. |
| Best for | Broad research agents and market monitors that read many different domains. |
| Setup effort | Get an API key, then update the routing logic in your agent's code. |
Your first move. Take a broad agent, the kind that reads across the open web rather than a fixed list. Get an API key and route one sample workflow through the paid endpoint today. Confirm the content comes back clean. You want to see a real bill and a real page while your agents are still working, not on the day the default flips.
The mistake that makes it fail. Turning it on with no budget in view. This path charges per page, so an agent that reads thousands of pages a day carries a real running cost. If you run agents for clients, that cost belongs in your delivery price as a line item, not as a surprise on your own card. Decide the ceiling before you wire it in.
Which Path Fits Your Setup
Choose on three things: how many pages you read, how varied the sources are, and what you are willing to spend.
| Your situation | Recommended path |
|---|---|
| Agent reads a fixed set of known sources, roughly 10 to 50 sites | Web Bot Auth. Register once, carry no per-page cost. |
| Agent reads broadly across the web for news, research, and competitors | Pay-per-use crawling. It covers every site with no per-site setup. |
| You run agents for clients at volume | Pay-per-use crawling, budgeted as a line item inside your delivery cost. |
| Agent reads a mix of known and unknown sites | Hybrid. Web Bot Auth for your core sources, pay-per-use for everything else. |
Most real setups land on the hybrid answer. Your agent has a handful of sources it hits constantly, and a long tail it reads once. Verify the handful. Pay for the tail. You keep the recurring cost low where you can and only pay per page where you have to.
Do This Before September 15
Seven steps. None of them take long on their own. Doing them now means you catch the breaks in a test window, not in a live client workflow.
- Audit your agent workflows. List every agent task that reads an external web page. Mark which sources are likely ad-supported. Most are.
- Check which sources sit behind Cloudflare. Run
dig +short TXT <domain>or open browser dev tools to spot Cloudflare-protected domains in your list. - Register for Web Bot Auth if you have a fixed set of high-value sources. Get the credentials before the deadline, not after.
- Set up a Cloudflare crawling API account if your agent reads broadly. Get an API key and test the integration on one sample workflow now.
- Update agent HTTP headers to carry your Web Bot Auth credential on the requests that need it.
- Route broad ad-supported reads through the paid endpoint and confirm the content comes back correct.
- Test in staging before September 15. Run your agents against a sample of real target URLs and check that they return content, not blocked responses.
Done. If every task on that list returns a page in your test window, you are ready. If one returns a block, you found it with time to fix it.
Honest: What Nobody Can Promise You Yet
A few things are still moving, and you should plan with that in mind.
Web Bot Auth depends on the site owner. You can hold a perfect credential and still be blocked by a site that has chosen to allow no agents at all. There is no way around a hard no. For those sources, pay-per-use is your only path, and if it is not offered there either, that source is closed to your agents for now.
Pay-per-use pricing scales with volume. A cheap fee per page stops feeling cheap when an agent reads at scale every hour. Watch the meter for the first week and set a hard cap before you point production traffic at it.
And this is a default, not a wall built once and forgotten. Cloudflare can tune it, and site owners will make their own calls over the following weeks. Treat the fix you build now as version one. Re-test after September 15 to confirm your agents still read what they read the day before.
None of this makes the change smaller. It makes it plannable. The builders who set this up early keep their agents working while everyone else spends the week after the deadline finding out what quietly broke.
Close: If You Only Do One Thing This Week
Audit your agents and find the ones that read the web.
Not all of them. Just the ones that fetch a live page. Run step one and step two from the checklist above, so you know which of your workflows sit in the blast radius and which sources hide behind Cloudflare. That single pass turns September 15 from a surprise into a scheduled task.
You can wire the two paths in afterward, on your own time, before the deadline. But you cannot fix what you have not found. Spend twenty minutes on the audit this week. If it turns out none of your agents read ad-supported pages, you spent twenty minutes buying certainty. If it turns out half of them do, you just gave yourself the runway to fix it calmly instead of in a fire.