Live report ยท updated 2026-10-17T06:05:38Z
The State of AI Crawler Permissions
How the web's most-visited sites treat AI crawlers, measured daily by BotWitness from robots.txt / llms.txt.
Who blocks which AI crawler
| AI crawler | Blocking at root | Share |
|---|---|---|
| GPTBot | 84567 of 618241 (14%) | |
| ClaudeBot | 78027 of 618241 (13%) | |
| Amazonbot | 76514 of 618241 (12%) | |
| Google-Extended | 72825 of 618241 (12%) | |
| anthropic-ai | 71450 of 618241 (12%) | |
| Applebot-Extended | 70388 of 618241 (11%) | |
| Meta-ExternalAgent | 69693 of 618241 (11%) | |
| cohere-ai | 68651 of 618241 (11%) | |
| ChatGPT-User | 67690 of 618241 (11%) | |
| Diffbot | 67144 of 618241 (11%) | |
| ImagesiftBot | 64908 of 618241 (10%) | |
| PerplexityBot | 64713 of 618241 (10%) | |
| Omgilibot | 64396 of 618241 (10%) | |
| FacebookBot | 62834 of 618241 (10%) | |
| YouBot | 62666 of 618241 (10%) | |
| TimpiBot | 60813 of 618241 (10%) | |
| OAI-SearchBot | 58230 of 618241 (9%) | |
| Claude-SearchBot | 55998 of 618241 (9%) | |
| Perplexity-User | 55754 of 618241 (9%) | |
| CCBot | 51111 of 618241 (8%) | |
| Bytespider | 47883 of 618241 (8%) |
Momentum: the last 30 days
131787 tracked domains materially changed their robots.txt AI directives in the last 30 days.
| AI crawler | Newly blocked | Newly unblocked |
|---|---|---|
| Applebot-Extended | 4571 | 32368 |
| ClaudeBot | 4781 | 32124 |
| Google-Extended | 4408 | 32412 |
| Bytespider | 4908 | 31835 |
| CCBot | 4703 | 31819 |
| GPTBot | 4736 | 31741 |
| Amazonbot | 4360 | 31923 |
| Meta-ExternalAgent | 3849 | 32302 |
The permission stack beyond robots.txt
| File | Sites publishing it | What it is |
|---|---|---|
| llms.txt | 214907 | AI-readable site guide |
| /.well-known/ai.txt | 102981 | AI usage policy file |
| RSL license | 266 | machine-readable license terms (robots.txt License: directive) |
RSL count is sites whose declared license document currently resolves; declaration history is in the archive from the day we first saw it.
Beyond domains: the AI ecosystem's own record
Also captured daily, through the same timestamped pipeline: EU GPAI training-data disclosures (4 surfaces); Web Bot Auth key directories (12 operators); MCP registry entries (21446 servers); live MCP tool manifests (1177 probed); Hugging Face model license/gating state (22 models).
Data licensing
Researchers, journalists and quantitative teams can license the underlying dataset: the longitudinal permission record across the tracked web, with per-capture hashes and timestamps. Citation of this page is free with a link.
Email [email protected]
Recently changed AI directives
- 2026-10-17 — mcp-tools:ai.inite/inite-visibility/mcp-tools removed (HTTP 200 -> None)
- 2026-10-17 — mcp-tools:ad.getle/leads/mcp-tools appeared (HTTP 200)
- 2026-10-17 — mcp-tools:ai.cloudworldmodel/cloud-world-model/mcp-tools content changed
- 2026-10-17 — mcp-tools:ai.easyterritory/ezt-mcp/mcp-tools content changed
- 2026-10-17 — mcp-tools:ai.daishi/world/mcp-tools content changed
- 2026-10-17 — mcp-tools:ai.dotsy/mcp/mcp-tools content changed
- 2026-10-17 — mcp-tools:ai.bithuman/characters/mcp-tools content changed
- 2026-10-17 — mcp-tools:ai.arcreport/ask/mcp-tools content changed
- 2026-10-17 — mcp-tools:ai.arcreport/mcp/mcp-tools content changed
- 2026-10-17 — mcp-tools:ai.capxule/capxule/mcp-tools content changed
- 2026-10-17 — mcp-tools:academy.apexai/apex-ai-academy/mcp-tools content changed
- 2026-10-17 — mcp-tools:ai.collager/ai-collage-maker/mcp-tools content changed
- 2026-10-17 — mcp-tools:ai.signal8/mcp/mcp-tools content changed
- 2026-10-17 — mcp-tools:ai.argushq/enforcement-database/mcp-tools removed (HTTP 200 -> None)
- 2026-10-17 — mcp-tools:ai.flowcastle/flowcastle/mcp-tools removed (HTTP 200 -> 503)
- 2026-10-17 — mcp-tools:ai.certscore/mcp-light/mcp-tools content changed
- 2026-10-17 — mcp-tools:ai.promptarch/mcp/mcp-tools content changed
- 2026-10-17 — mcp-tools:ai.cabrini/market-data/mcp-tools removed (HTTP 200 -> None)
- 2026-10-17 — mcp-tools:ai.corduroy-labs/updates/mcp-tools removed (HTTP 200 -> None)
- 2026-10-17 — mcp-tools:ai.plyto/crm/mcp-tools content changed
Coverage: 1022568 domains, 5272001 snapshots, 25168158 checks; archive since 2026-07-09T19:41:24Z. Each capture is SHA-256 hashed and RFC-3161 timestamped. Point-in-time lookups and certified records via the API.