AI Crawler Access: Which Bots to Allow, Block, or Charge

The Permission Flip

For thirty years the web operated on a simple bargain: crawlers could take your content, and in exchange they sent you traffic. Search engines indexed pages and returned visitors. The exchange was implicit, unmetered, and broadly fair. Robots.txt existed as a courtesy protocol for the small number of cases where a site owner wanted out.

Generative AI broke the bargain on one side. A model can ingest a page, absorb the expertise, and answer the user directly. The content is consumed; the visit never happens. That asymmetry has been building since 2023, and 2026 is the year the infrastructure layer responded to it.

On 15 September 2026, Cloudflare changes its defaults. Newly onboarded domains, new sites created by existing customers, and all existing free-tier customers will have training crawlers and agent bots blocked by default on pages that display ads. Search crawlers remain allowed. Cloudflare has also been extending its pay-per-crawl marketplace toward a pay-per-use model, where a publisher is compensated when content generates value inside an AI product rather than merely when a bot fetches it.

The strategic point is not the specific policy. It is that a meaningful share of the web now has an AI access posture set by a CDN default rather than by a marketing decision. If your site sits behind Cloudflare — and a large share of commercial sites do — your AI visibility strategy may be quietly overridden by an infrastructure setting nobody on your team has looked at. That is a governance problem before it is a technical one.

Three Bots, Three Bargains

The core analytical error in this whole debate is treating “AI crawler” as one category. It is three, and they run in opposite directions commercially. We call the resulting framework the Crawler Exchange Rate: for each bot type, what do you give and what do you get back?

Read in that order, the standard advice to “block AI bots to protect your content” is close to backwards. The bot most people are angry about — the training crawler — is the one whose exchange rate is genuinely poor. The other two are the mechanism by which AI search returns traffic and revenue. A blanket block sacrifices the two that pay in order to stop the one that does not.

The Crawler Reference Table

Directional recommended posture for a typical B2B SaaS, DTC, or professional services site whose content exists to generate demand rather than to be sold. Publishers and data businesses should read the training column differently — see the section below.

Crawler names and behaviours change. Verify against each operator’s published documentation before writing rules, and confirm identity by reverse DNS or published IP ranges — user agent strings are trivially spoofed.

The Google-Extended Trap

One control causes more expensive mistakes than the rest of the category combined. Google-Extended is a robots.txt token that governs whether your content may be used for training and grounding in Gemini apps and the Vertex AI API. It is not a crawler. It does not control Google Search, and it does not control AI Overviews.

AI Overviews and AI Mode are generated from the live Google Search index — the index Googlebot builds. Blocking Google-Extended has no bearing on whether your pages appear in an AI Overview. The only way to remove yourself from AI Overviews is to block Googlebot or suppress snippets, and both cost you organic search in its entirety.

The practical result is that a team that blocks Google-Extended in order to “stay out of Google AI” has surrendered Gemini grounding — a genuine citation surface — and achieved nothing on the objective they were actually pursuing. Before writing any rule in this category, be precise about which system a given control governs. Most of the cost here comes from acting on the label rather than the mechanism.

Access Is the Precondition for GEO

Generative Engine Optimization is the discipline of earning citations inside AI answers. Every technique in it — direct-answer structure, entity consistency, schema markup, citable statistics, third-party corroboration — assumes one thing: that the retrieval layer holds a current copy of your page.

If a retrieval crawler is blocked at the CDN or in robots.txt, none of that work can produce a citation. We have seen teams spend a full quarter restructuring content for AI citation while their edge configuration returned 403s to the exact crawlers that would have indexed it. The content was excellent. The retrieval layer never saw it.

Sequence the work accordingly. Audit and fix crawler access first, confirm retrieval crawlers are getting 200s on your priority pages, then invest in the content layer. The reverse order wastes a quarter, and the failure is invisible in every dashboard a marketing team normally looks at.

A Five-Step Access Audit

This is the sequence we run for clients before any AI visibility engagement. It takes a competent technical marketer under a day.

The output should be a one-page policy document naming each bot type, the posture, the rationale, and the person accountable. Crawler policy drifts because nobody owns it — it sits between marketing, SEO, and infrastructure, and each assumes another has it. Assign it explicitly.

Posture by Business Model

The training-crawler decision is genuinely different depending on whether your content is the product or the marketing for the product. The retrieval and agent decision is close to universal.

Note the consistency of the third column. Almost no commercial business benefits from blocking retrieval and agent traffic. The genuine strategic debate is confined to training, and it turns on a single question: would a competitor pay for this content? If yes, meter it. If it exists to make people aware you exist, giving it away is the point.

Measuring What Access Buys You

Access policy is only defensible if you can see what it produces. That requires the measurement layer described in our guide to AI search attribution — most assistant-referred traffic lands in analytics as Direct, so a team that changes crawler policy without attribution instrumentation cannot tell whether the change helped or hurt.

The minimum instrumentation before changing posture: crawler hit rates by bot and status code from server logs, citation monitoring across the major assistants, referrer grouping that separates assistant traffic from Direct, and a self-reported attribution question on your primary conversion form. Establish a baseline for at least four weeks before making changes, or you will be reading noise.

Judge the result on unit economics rather than sessions. AI referral traffic tends to arrive later in the research cycle and convert well above blended organic, so a channel that looks small in volume can be material in pipeline. Run it through channel-level CAC benchmarking and the LTV:CAC framework the same way you would any other acquisition channel.

The UAE and GCC Angle

Two things make this more consequential for regional businesses than for their US counterparts. The first is infrastructure: a large share of UAE and GCC commercial sites run on free or entry-tier CDN plans, which is precisely the cohort whose defaults change on 15 September. The teams least likely to have a documented crawler policy are the ones most likely to have their posture flipped for them.

The second is language coverage. Arabic-language content is comparatively thin in the training and retrieval corpora that ground AI answers, which cuts both ways. Competition for citation on Arabic and GCC-specific commercial queries is materially lower than the equivalent English query — a real opening for regional businesses publishing substantive Arabic content. But that opening only exists if the retrieval crawlers can reach the content. Blocking them forfeits an advantage that will not stay open indefinitely.

For Dubai and GCC growth teams, the practical sequence is: verify CDN posture before mid-September, allow retrieval and agent crawlers explicitly rather than relying on defaults, and treat Arabic-language citation as a channel worth building while acquisition costs in it remain low. Our Dubai digital marketing engagements now include this audit as standard scope.

Where This Goes Next

Three developments are worth tracking without over-planning for. Metering is moving from per-fetch to per-use, which changes the economics for content owners with genuinely differentiated libraries. Agent traffic is growing as assistants take on more transactional work, which raises the cost of blocking the agent category specifically. And bot verification standards are maturing, which will make identity-based policy more reliable than user-agent matching.

None of that changes the near-term action. Know which bots reach you, decide deliberately per category, and never let a default decide. The teams that get hurt in the next twelve months will not be the ones that chose wrong — they will be the ones that never chose. This is the same operating discipline we apply across the Kres Labs growth playbook, and it sits alongside the agent workflows covered in AI agents for B2B marketing and the broader capability set at our AI marketing agency practice. For foundations, start with what is growth marketing.

Frequently Asked Questions