The crawler
This page describes the crawler that builds the BotWitness archive. It records what 1,000,000+ domains tell AI crawlers through their public permission files, daily, with independent RFC-3161 timestamps. If you found this page from your server logs, the User-Agent you saw was:
BotWitness/1.0 (+https://bot-witness.com/bot; contact: [email protected])
Exactly three paths, always at the site root:
/robots.txt
/llms.txt
/.well-known/ai.txt
Plus, only when your robots.txt declares a License: directive, the RSL license document that directive points to. It never fetches content pages, never follows links, and never downloads media. Response bodies are capped at 5MB, which these files never approach.
At most once per day. The highest-traffic domains are checked daily; most domains are checked every 7 or 30 days. A visit is three GET requests, four if your robots.txt points at a license document. That is the entire load.
We do not yet publish an egress IP list, and our requests are not yet cryptographically signed. Both are planned. Until then, the crawler's behavior is narrow enough to act as a fingerprint:
Real BotWitness traffic only ever requests the paths above, at most once a day. Hits on any other path, or at any real volume, are someone else wearing our User-Agent. If you see that, we want to know: email a log excerpt with the IP, timestamp and path to [email protected].
The crawler reads only the permission files themselves, and it does not apply robots.txt rules to those reads. That is deliberate, not an oversight: the archive is a neutral record of what each site's rules said on each date, and it stays useful as evidence precisely because it does not go blind the moment a site says no. Sites rely on that record to prove their opt-outs existed; crawler operators rely on it to prove what they were told.
If you want your domain removed from tracking anyway, email [email protected] from an address at that domain and we will remove it.
[email protected] for anything about the crawler, including impersonation reports and removal requests. See the FAQ for what the archive can and cannot prove.
