For AI companies · Crawl Compliance Attestation
Publishers can show what their robots.txt says today. When a dispute, an audit, or a procurement review asks what it said eighteen months ago, across every domain you crawled, you need an independent record. BotWitness has been capturing robots.txt across a million domains daily, hashed and RFC-3161 timestamped. One API call turns that archive into a signed attestation: what each domain permitted your crawler, dated, for any range.
The gap
robots.txt is the permission the web gave you, and it changes silently, with no public history. When a publisher alleges your bot ignored their rules, the question is what the file said at the moment of the crawl. Your own archived copy is self-serving evidence; the live file proves nothing about the past. A dated record from a neutral archive that was watching anyway is what settles it.
How it works
01 · Pick
Name the user-agent (GPTBot, ClaudeBot, CCBot…), up to 100 domains per request, and the date range at issue.
02 · Attest
For every timestamped capture in the range (plus the state entering it) the report states whether the file permitted or disallowed your crawler at the site root, with the SHA-256 of the exact bytes and the timestamp status of each capture.
03 · File
format=json for your compliance pipeline; format=pdf for a filing-ready report. Every underlying check is additionally provable via a Merkle inclusion proof against a timestamped run root.
Example
curl -H "Authorization: Bearer $KEY" \ "https://bot-witness.com/attestation?agent=GPTBot&domains=nytimes.com,wsj.com&from=2026-01-01&to=2026-07-01&format=pdf" \ -o attestation.pdf
Who needs it
Scraping and copyright claims increasingly turn on robots.txt history. Answer them with a neutral archive's dated captures, not your own logs.
EU AI Act and enterprise procurement reviews ask how you honor crawl permissions. A recurring attestation over your crawl list is the artifact that closes the question.
Your agents browse on users' behalf. Show, per domain and date, what the rules were when they visited, before someone else characterizes it for you.
Get started
Attestations are included with Monitoring plans. Crawling at fleet scale, or need domains added to coverage? Tell us your crawl list. Shipping a model too? The Defensive Exposure Audit pairs these attestations with the probe suite, run for your side.
