Live report ยท updated 2026-08-27T13:19:55Z
The State of AI Crawler Permissions
How the web's most-visited sites treat AI crawlers, measured daily by BotWitness from robots.txt / llms.txt.
Who blocks which AI crawler
| AI crawler | Blocking at root | Share |
|---|---|---|
| GPTBot | 132855 of 640257 (21%) | |
| ClaudeBot | 126543 of 640257 (20%) | |
| Amazonbot | 125776 of 640257 (20%) | |
| Google-Extended | 122214 of 640257 (19%) | |
| Meta-ExternalAgent | 119676 of 640257 (19%) | |
| Applebot-Extended | 119400 of 640257 (19%) | |
| CCBot | 96724 of 640257 (15%) | |
| Bytespider | 93066 of 640257 (14%) | |
| anthropic-ai | 68149 of 640257 (11%) | |
| ChatGPT-User | 67762 of 640257 (11%) | |
| PerplexityBot | 64906 of 640257 (10%) | |
| cohere-ai | 64816 of 640257 (10%) | |
| ImagesiftBot | 64471 of 640257 (10%) | |
| Omgilibot | 64340 of 640257 (10%) | |
| Diffbot | 64249 of 640257 (10%) | |
| FacebookBot | 63871 of 640257 (10%) | |
| YouBot | 62628 of 640257 (10%) | |
| TimpiBot | 61528 of 640257 (10%) | |
| OAI-SearchBot | 59874 of 640257 (9%) | |
| Claude-SearchBot | 57288 of 640257 (9%) | |
| Perplexity-User | 56647 of 640257 (9%) |
Momentum: the last 30 days
58642 tracked domains materially changed their robots.txt AI directives in the last 30 days.
| AI crawler | Newly blocked | Newly unblocked |
|---|---|---|
| GPTBot | 3956 | 2534 |
| Bytespider | 4076 | 2292 |
| ClaudeBot | 3892 | 2427 |
| CCBot | 3874 | 2382 |
| Amazonbot | 3838 | 2346 |
| Google-Extended | 3684 | 2375 |
| Applebot-Extended | 3759 | 2207 |
| Meta-ExternalAgent | 3928 | 1730 |
The permission stack beyond robots.txt
| File | Sites publishing it | What it is |
|---|---|---|
| llms.txt | 236185 | AI-readable site guide |
| /.well-known/ai.txt | 115451 | AI usage policy file |
| RSL license | 230 | machine-readable license terms (robots.txt License: directive) |
RSL count is sites whose declared license document currently resolves; declaration history is in the archive from the day we first saw it.
Beyond domains: the AI ecosystem's own record
Also captured daily, through the same timestamped pipeline: EU GPAI training-data disclosures (4 surfaces); Web Bot Auth key directories (12 operators); MCP registry entries (13852 servers); live MCP tool manifests (777 probed); Hugging Face model license/gating state (22 models).
Data licensing
Researchers, journalists and quantitative teams can license the underlying dataset: the longitudinal permission record across the tracked web, with per-capture hashes and timestamps. Citation of this page is free with a link.
[email protected]Recently changed AI directives
- 2026-08-27 — rakyatjpjelata.com/robots.txt removed (HTTP 200 -> None)
- 2026-08-27 — hamrahmoshaver.com/robots.txt appeared (HTTP 200)
- 2026-08-27 — vsrf.ru/robots.txt removed (HTTP 200 -> None)
- 2026-08-27 — vsrf.ru/llms.txt removed (HTTP 200 -> None)
- 2026-08-27 — pets.ca/robots.txt appeared (HTTP 200)
- 2026-08-27 — pixelio.de/llms.txt content changed
- 2026-08-27 — miun.se/robots.txt: Amazonbot now BLOCKED (was allowed); Applebot-Extended now BLOCKED (was allowed); Bytespider now BLOCKED (was allowed); CCBot now BLOCKED (was allowed); ChatGPT-User now BLOCKED (was allowed); Claude-SearchBot now BLOCKED (was allowed); ClaudeBot now BLOCKED (was allowed); Diffbot now BLOCKED (was allowed); FacebookBot now BLOCKED (was allowed); GPTBot now BLOCKED (was allowed); Google-Extended now BLOCKED (was allowed); ImagesiftBot now BLOCKED (was allowed); Meta-ExternalAgent now BLOCKED (was allowed); OAI-SearchBot now BLOCKED (was allowed); Omgilibot now BLOCKED (was allowed); Perplexity-User now BLOCKED (was allowed); PerplexityBot now BLOCKED (was allowed); TimpiBot now BLOCKED (was allowed); YouBot now BLOCKED (was allowed); anthropic-ai now BLOCKED (was allowed); cohere-ai now BLOCKED (was allowed)
- 2026-08-27 — weltwoche.ch/.well-known/ai.txt content changed
- 2026-08-27 — rajaindowin8.xyz/robots.txt appeared (HTTP 200)
- 2026-08-27 — hrexecutive.com/.well-known/ai.txt content changed
- 2026-08-27 — weltwoche.ch/llms.txt content changed
- 2026-08-27 — psca.gop.pk/llms.txt appeared (HTTP 200)
- 2026-08-27 — psca.gop.pk/.well-known/ai.txt appeared (HTTP 200)
- 2026-08-27 — hrexecutive.com/llms.txt content changed
- 2026-08-27 — nic.porn/.well-known/ai.txt content changed
- 2026-08-27 — ultimatecarpage.com/.well-known/ai.txt content changed
- 2026-08-27 — nic.porn/llms.txt content changed
- 2026-08-27 — vreme.com/llms.txt content changed
- 2026-08-27 — vreme.com/.well-known/ai.txt content changed
- 2026-08-27 — com-hotel.com/.well-known/ai.txt content changed
Coverage: 1014574 domains, 3735087 snapshots, 11192436 checks; archive since 2026-07-09T19:41:24Z. Each capture is SHA-256 hashed and RFC-3161 timestamped. Point-in-time lookups and certified records via the API.