Live report ยท updated 2026-10-12T06:12:01Z
The State of AI Crawler Permissions
How the web's most-visited sites treat AI crawlers, measured daily by BotWitness from robots.txt / llms.txt.
Who blocks which AI crawler
| AI crawler | Blocking at root | Share |
|---|---|---|
| GPTBot | 83072 of 617473 (14%) | |
| ClaudeBot | 76474 of 617473 (12%) | |
| Amazonbot | 74953 of 617473 (12%) | |
| Google-Extended | 71295 of 617473 (12%) | |
| anthropic-ai | 69966 of 617473 (11%) | |
| Applebot-Extended | 68813 of 617473 (11%) | |
| Meta-ExternalAgent | 68091 of 617473 (11%) | |
| ChatGPT-User | 67467 of 617473 (11%) | |
| cohere-ai | 67165 of 617473 (11%) | |
| Diffbot | 65664 of 617473 (11%) | |
| ImagesiftBot | 64698 of 617473 (10%) | |
| PerplexityBot | 64547 of 617473 (10%) | |
| Omgilibot | 64197 of 617473 (10%) | |
| FacebookBot | 62661 of 617473 (10%) | |
| YouBot | 62468 of 617473 (10%) | |
| TimpiBot | 60594 of 617473 (10%) | |
| OAI-SearchBot | 58089 of 617473 (9%) | |
| Claude-SearchBot | 55810 of 617473 (9%) | |
| Perplexity-User | 55543 of 617473 (9%) | |
| CCBot | 49758 of 617473 (8%) | |
| Bytespider | 46506 of 617473 (8%) |
Momentum: the last 30 days
136035 tracked domains materially changed their robots.txt AI directives in the last 30 days.
| AI crawler | Newly blocked | Newly unblocked |
|---|---|---|
| Applebot-Extended | 4223 | 35638 |
| ClaudeBot | 4478 | 35332 |
| Google-Extended | 4003 | 35652 |
| Bytespider | 4604 | 35029 |
| CCBot | 4409 | 35052 |
| GPTBot | 4388 | 34898 |
| Meta-ExternalAgent | 3533 | 35628 |
| Amazonbot | 3824 | 35205 |
The permission stack beyond robots.txt
| File | Sites publishing it | What it is |
|---|---|---|
| llms.txt | 214619 | AI-readable site guide |
| /.well-known/ai.txt | 103276 | AI usage policy file |
| RSL license | 262 | machine-readable license terms (robots.txt License: directive) |
RSL count is sites whose declared license document currently resolves; declaration history is in the archive from the day we first saw it.
Beyond domains: the AI ecosystem's own record
Also captured daily, through the same timestamped pipeline: EU GPAI training-data disclosures (4 surfaces); Web Bot Auth key directories (12 operators); MCP registry entries (20313 servers); live MCP tool manifests (1129 probed); Hugging Face model license/gating state (22 models).
Data licensing
Researchers, journalists and quantitative teams can license the underlying dataset: the longitudinal permission record across the tracked web, with per-capture hashes and timestamps. Citation of this page is free with a link.
Email [email protected]
Recently changed AI directives
- 2026-10-12 — mcp-tools:ai.bizmoon/regulatory/mcp-tools removed (HTTP 200 -> None)
- 2026-10-12 — mcp-tools:ai.greenlandai/greenlandai/mcp-tools content changed
- 2026-10-12 — mcp-tools:ai.cloudworldmodel/cloud-world-model/mcp-tools content changed
- 2026-10-12 — mcp-tools:ai.deepdna/knowledge/mcp-tools content changed
- 2026-10-12 — mcp-tools:ai.centralcity/central-city/mcp-tools content changed
- 2026-10-12 — mcp-tools:ai.everymcp/every-mcp/mcp-tools content changed
- 2026-10-12 — mcp-tools:ai.agentlookups/spoon-scout/mcp-tools content changed
- 2026-10-12 — mcp-tools:ai.signal8/mcp/mcp-tools content changed
- 2026-10-12 — mcp-tools:ai.plyto/crm/mcp-tools content changed
- 2026-10-12 — mcp-tools:ai.analyticslegends/sap-analytics/mcp-tools content changed
- 2026-10-12 — mcp-tools:ai.ccapi/mcp/mcp-tools content changed
- 2026-10-12 — mcp-tools:ai.agentlookups/counterscript/mcp-tools content changed
- 2026-10-12 — mcp-tools:ai.hoghogh/chistara/mcp-tools content changed
- 2026-10-12 — mcp-tools:ai.quantifyme/quantifyme/mcp-tools content changed
- 2026-10-12 — mcp-tools:ai.namewhisper/ens-tools/mcp-tools content changed
- 2026-10-12 — mcp-tools:ai.limitguard.api/trust-intelligence/mcp-tools content changed
- 2026-10-12 — mcp-tools:ai.autonomad/travel/mcp-tools content changed
- 2026-10-12 — mcp-tools:ai.bowmark/bowmark/mcp-tools content changed
- 2026-10-12 — mcp:ai.analyticslegends/sap-analytics/mcp content changed
- 2026-10-12 — mcp:ai.ankimcp/anki-mcp-server-addon/mcp content changed
Coverage: 1021387 domains, 5085871 snapshots, 23015941 checks; archive since 2026-07-09T19:41:24Z. Each capture is SHA-256 hashed and RFC-3161 timestamped. Point-in-time lookups and certified records via the API.