What happened
IAB Australia published a Bots and Crawler Guidance and Decision Matrix that sorts every automated visitor into five job categories: Discovery & Search (Googlebot, bingbot, OAI-SearchBot), AI Training (GPTBot, ClaudeBot, CCBot), Live AI Agents (ChatGPT-User, Claude-User), Operational & Advertising (AdsBot-Google, verification vendors), and Unknown/Unverified. Each gets one of four verdicts: allow, allow with conditions, require licensing, or block.
The framework is useful. The deadline inside it matters more. Before September 15, 2026, new domains and free-tier zones behind Cloudflare get Training and Agent bots blocked by default on any page that displays advertising. That decision happens at the edge — the CDN layer — not in your robots.txt file.
The supporting data explains the urgency. Cloudflare figures cited in the guidance put automated requests at 57.5% of web traffic as of June 2026 — the first recorded crossover past human traffic. AI-agent requests grew 45% quarter on quarter to 17.7 billion in Q2 2026. And within AI crawler traffic, roughly 52% of requests serve model training while only about 2.6% are real-time fetches triggered by an actual human asking a question.
Why this matters
Every SEO tool you own reads robots.txt and reports back. None of them see an edge block. Your robots.txt can say "allow" in plain text while Cloudflare turns GPTBot away at the door, and your rank tracker, your audit tool, and your agency's crawler will all tell you everything is fine. That gap between what your file says and what your CDN does is the whole story here.
The category split is the part store owners should sit with. A training crawler is building a model. A live agent — ChatGPT-User, Claude-User — is a person, right now, asking an AI which product to buy. The guidance cites Adobe data showing AI referral traffic converts roughly 42% better than non-AI visits. Block that category by accident and you are turning away your best-converting visitors to solve a publisher's licensing problem.
And there is a trap the guidance is honest about: one operator often runs several bots for several jobs. Blocking a training crawler does not necessarily pull your content out of the AI answers built on the same index. So a reflexive block can cost you agent traffic without actually removing your content from anyone's model. This default was designed around ad-supported publishers protecting content economics. You sell products, not pageviews. Your math is different — which is exactly why you should be the one doing it.
What to do about it
Put September 15 on the calendar
Log into Cloudflare before the deadline, open the bot and AI-crawler settings, and record what is currently on. If you are on the free tier or launching a new domain, assume the new default applies to you and make the allow/block call explicitly rather than inheriting it.
Decide by job, not by vendor
Use the matrix's categories as your worksheet. Keep Discovery & Search open — that is your Google traffic. Treat Training as a business decision about whether your content feeds someone's model for free. Think hard before blocking Live Agents: those requests are shoppers mid-question, and they are the category tied to that 42% conversion edge.
Verify at the edge, not in the file
Stop trusting robots.txt-only reports for this. Check Cloudflare's bot analytics for blocked requests from GPTBot, ClaudeBot, ChatGPT-User, and Claude-User after September 15. If AI referral sessions in your analytics drop around that date, the CDN default is the first suspect.
Know whether you're in scope
The default triggers on pages that display advertising. Inventory which of your pages run ad code of any kind. If the answer is none, the deadline is less urgent for you — but the settings review still is, because defaults have a habit of widening.