A BotWitness report — how the web's most-visited sites treat AI crawlers, measured daily from robots.txt / llms.txt. Updated 2026-07-09T19:42:39Z.
| AI crawler | Sites blocking at root | Share |
|---|---|---|
| Bytespider | 29 of 38 (76%) | |
| CCBot | 29 of 38 (76%) | |
| ClaudeBot | 27 of 38 (71%) | |
| Applebot-Extended | 26 of 38 (68%) | |
| Diffbot | 26 of 38 (68%) | |
| PerplexityBot | 25 of 38 (66%) | |
| anthropic-ai | 24 of 38 (63%) | |
| cohere-ai | 23 of 38 (60%) | |
| Meta-ExternalAgent | 22 of 38 (58%) | |
| Claude-SearchBot | 21 of 38 (55%) | |
| Google-Extended | 21 of 38 (55%) | |
| Amazonbot | 20 of 38 (53%) | |
| Perplexity-User | 20 of 38 (53%) | |
| TimpiBot | 20 of 38 (53%) | |
| FacebookBot | 19 of 38 (50%) | |
| Omgilibot | 18 of 38 (47%) | |
| GPTBot | 16 of 38 (42%) | |
| ChatGPT-User | 14 of 38 (37%) | |
| YouBot | 14 of 38 (37%) | |
| ImagesiftBot | 13 of 38 (34%) | |
| OAI-SearchBot | 8 of 38 (21%) |
3 of 39 tracked sites publish an llms.txt.
Coverage: 39 domains, 117 snapshots, 207 checks; archive since 2026-07-09T19:41:24Z. Each capture is SHA-256 hashed and RFC-3161 timestamped. Point-in-time and certified records at https://bot-witness.com.