What it is.
AI Crawl Control lives in the Cloudflare dashboard and reports which AI services requested your pages, how often, and which of them honored robots.txt. It is available on all plans and works with no configuration.
From the same screen you set allow or block rules per crawler. Most teams end up making that call twice: once for training crawlers, and once for the bots that fetch a page live when someone asks a question.
Pay per crawl, where a crawler is charged for access, is in private beta, so read it as a signal about where this goes.
What you get.
- A list of the AI services requesting your pages, with request volume.
- A robots.txt compliance view, so you can see which crawlers ignored your directives.
- Per-crawler allow and block rules you set from the dashboard.
- All of it on any Cloudflare plan, including the free one.
How to set it up.
Confirm your domain is proxied through Cloudflare, since the data comes from traffic passing through it.
Open AI Crawl Control and read a week of traffic before you change anything.
Separate the crawlers that cite you from the ones that only train on you, then set rules per crawler.
Check the robots.txt compliance column, because a crawler that ignores the file needs a block.
Block in stages and watch referral traffic from AI assistants, so you can undo a block that cost you citations.
Revisit quarterly, because the crawler list changes faster than your policy will.
Use cases
See who is taking your pages
Read the request volume per AI service before you decide anything.
Check robots compliance
Find out which crawlers respect the file and which ignore it.
Decide per crawler
Allow the ones that send referrals and block the ones that only take.
Best for
Deciding what AI can read
Somebody has to make the call on training crawlers, and this gives you the traffic data to make it with.
Proving AI crawlers are there
Server logs are painful. This counts AI crawler requests with no log pipeline at all.
Sites already on Cloudflare
Nothing to install, so the only cost is the time you spend reading it.