What happened
On June 24, Microsoft added a robots.txt-violations layer to Clarity's Bot Analytics dashboard. It logs bot requests that hit URLs your robots.txt disallows, shows them as a share of total bot traffic over whatever timeframe you pick, and charts the trend. You can filter by operator, bot name, activity type, and the exact URLs and paths requested — and put compliant crawlers next to violators in a side-by-side view.
Setup takes a project admin. The feature lives in the AI Visibility section of Project Settings and has to be switched on manually. It also needs a data source at the edge: a supported CDN (Fastly, Amazon CloudFront, Cloudflare, Azure Front Door, or Akamai) or the latest Clarity WordPress plugin. Clarity itself stays free.
One honest caveat baked into Microsoft's own framing: robots.txt is advisory. Clarity records that a disallowed request happened. It doesn't block anything, and it can't tell you why the crawler made the request.
Why this matters
Store owners have been arguing about AI crawlers — ClaudeBot, GPTBot, the whole roster — with almost no data. A disallow directive was a polite request you could never verify. Now the receipts are free. The conversation shifts from "I assume bots ignore me" to "this operator keeps requesting disallowed paths, and here's the trend line."
The sequencing matters more than the tool. Edge blocking is close to a one-way door for AI visibility: block a retrieval crawler that feeds an assistant your customers actually use, and your products quietly drop out of its answers. The sane order is measure first, sort operators by behavior, then write policy per crawler — never a blanket rule adopted in a bad mood.
Stay skeptical of raw violation counts, though. A single hit on a disallowed path might be a one-off link fetch, not a systematic scrape. The trend view and the operator filter exist precisely so you don't overreact to noise.
What to do about it
Enable it this week
Have a project admin open Clarity's Project Settings, find the AI Visibility section, and switch the violations layer on. Confirm your CDN is on the supported list — Fastly, CloudFront, Cloudflare, Azure Front Door, or Akamai — or update the Clarity WordPress plugin first. No edge connection, no data.
Clean up robots.txt before you read the report
A violations report inherits every mistake in your directives. Check that your disallow rules reflect current policy — faceted URLs, cart and checkout paths, and any AI-crawler blocks you actually mean — before you start judging crawlers against them.
Collect 30 days, then decide
Let the dashboard build a real timeframe before you touch edge rules. Use the side-by-side view to sort operators into compliant, occasional, and habitual violators. Then write per-operator policy at the CDN instead of one blanket block.
Weigh what each crawler sends back
Before blocking any AI operator outright, ask what its assistant does for you. Cutting off a crawler that retrieves and cites your product pages is a revenue decision, not a hygiene chore — make it with the referral data in front of you.