Skip to content
Raptor home
Menu

AI crawler block

directives.ai_crawlers_blocked
Priority
Note
Severity
Info

What we look for

The site's robots.txt (its file of rules for crawlers) names one or more well-known AI crawlers and blocks them from the site's root address (/).

Why it matters

Crawlers that obey these rules stay off the site, so its pages are not collected for those AI products, which can make it less visible in AI answers. Many sites choose this on purpose, and then it needs no change.

How to fix

Remove a crawler's Disallow line, or its whole User-agent group, from robots.txt to let it read the site. Leave the rules as they are if the block is what you want.

What it cannot know

Raptor reads robots.txt only, and only for a list of well-known AI crawlers. A block set in a firewall or CDN does not show here.

Seen in a real crawl

From a crawl of Millbrook Hardware (the broken fixture), a test site from Raptor’s fixture suite.

Note

The site blocks some AI crawlers in robots.txt

A site-wide finding: it describes the site, so it lists no pages. It names GPTBot.

Paths on http://127.0.0.1:8001raptor 0.3.0, 17 Oct 2026

Open this finding in that crawl’s report

Where you’ll see it

The id is stable, and every surface uses it, so it’s the thing to search for or hand to an agent.

Terminal
When it is one of the five findings under Fix first, raptor crawl prints the id under it.
Check: directives.ai_crawlers_blocked
crawl.json and the CSV
The crawl JSON names the finding by this id, and so do the CSV and raptor_query.
"check_id": "directives.ai_crawlers_blocked"
The HTML report
This link opens the finding in the report, and each finding shows its id.
crawl.html#finding=directives.ai_crawlers_blocked
MCP
raptor_findings with this check_id returns the finding in full, with a fix brief ready to paste into a task. If it didn’t fire, it says why.
raptor_findings {
  "job_id": "...",
  "check_id": "directives.ai_crawlers_blocked"
}