Developer docs

The BotWitness API

One HTTP interface over a dated, tamper-evident record of what sites told AI crawlers. Ask what a domain said to a crawler on a date, pull the signed exhibit for it, feed your server log in for judgement, or subscribe so a change reaches your systems the day it happens.

Your first call

Current lookups cost nothing and need no account, so the quickest way to see what the archive holds is to ask it something.

curl "https://bot-witness.com/permission?domain=nytimes.com&agent=GPTBot"

The reply names the file we read, when we read it, the hash of the bytes we stored, whether an independent timestamp is attached, and what the file said about that crawler.

{
  "domain": "nytimes.com",
  "resource": "robots.txt",
  "as_of": "2026-09-13",
  "tracking_since": "2026-07-09T19:41:24Z",
  "captured_at": "2026-09-13T06:33:40Z",
  "http_status": 200,
  "sha256": "54ef6b50660ed0642fc12819d5fce90b67009f73019956ec4eeef205701a9ad9",
  "timestamped": true,
  "result": {
    "agent": "GPTBot",
    "blocked_root": true,
    "verdict": "blocked at root",
    "disallow": ["/"],
    "allow": []
  }
}

Keys, and what costs nothing

Create a key in your dashboard and send it either way:

-H "Authorization: Bearer $KEY"      # or  ?key=$KEY

Three things are free and need no key at all: a current-or-recent lookup, the coverage endpoint, and the watch streams. Everything else either spends a credit or wants a plan, and the error tells you which. Responses are JSON unless you ask for format=pdf, which returns the document itself.

The record

Lookup: GET /permission

What a domain told a crawler, now or on a past date. The current state and the last thirty days are free; going further back wants a Monitoring plan. Parameters are domain, agent, and an optional as_of date.

curl "https://bot-witness.com/permission?domain=nytimes.com&agent=GPTBot&as_of=2024-06-01" \
  -H "Authorization: Bearer $KEY"

Certificate: GET /permission/certificate key

The same record as a signed Certificate of Capture, the PDF you can hand to somebody else. It spends one certificate credit, and Monitoring plans have no limit. Read a sample before you spend one.

curl "https://bot-witness.com/permission/certificate?domain=nytimes.com&agent=GPTBot" \
  -H "Authorization: Bearer $KEY" -o certificate.pdf

History: GET /history

Every directive state we recorded for a domain, in order, so you can see when it changed its mind. Without a plan you get the last thirty days and full_history comes back false.

curl "https://bot-witness.com/history?domain=nytimes.com&agent=ClaudeBot" -H "Authorization: Bearer $KEY"

Change feed: GET /changes

Changes since a date, one event per material edit, each naming the crawlers whose treatment moved. Useful when you want to poll rather than receive a webhook.

curl "https://bot-witness.com/changes?domain=nytimes.com&since=2026-01-01"

{
  "count": 6,
  "changes": [
    {
      "event": "permission.changed",
      "id": "35c487db0f2b40af8f5f466d5be0cf1f",
      "occurred_at": "2026-09-13T06:33:40Z",
      "domain": "nytimes.com",
      "resource": "robots.txt",
      "change_type": "directives_changed",
      "material": true,
      "http_status": { "old": 200, "new": 200 },
      "agents_changed": [ ... ]
    }
  ]
}

Coverage: GET /coverage free

Whether we track a domain at all, since when, and how much we hold. Ask this first if you are about to build something on a domain we may not have.

curl "https://bot-witness.com/coverage?domain=nytimes.com"

{
  "domain": "nytimes.com",
  "tracked": true,
  "tracking_since": "2026-07-09T19:41:24Z",
  "last_checked": "2026-09-13T06:33:41Z",
  "versions_captured": 9,
  "changes_recorded": 6
}

Change alerts: POST /subscriptions Monitoring

Watch a list of domains and hear about it when one of them changes its AI directives, by email or by webhook. Verify a webhook with sha256 = HMAC(secret, X-PR-Timestamp + "." + body).

curl -X POST https://bot-witness.com/subscriptions -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"kind":"email","endpoint":"[email protected]","domains":"nytimes.com,reddit.com"}'

If you run a crawler

Compliance attestation: GET /attestation Monitoring

What robots.txt permitted your named crawler across up to a hundred domains over a date range, with the verdict for every capture, as JSON or a filing-ready PDF. This is the document to keep for the day a publisher asks what you were told. What is in it.

curl "https://bot-witness.com/attestation?agent=GPTBot&domains=nytimes.com,wsj.com&from=2026-01-01&to=2026-07-01&format=pdf" \
  -H "Authorization: Bearer $KEY" -o attestation.pdf

If you have a server log

Violation check: POST /violations Monitoring

Send hits as structured JSON or as raw Apache and nginx lines. Each one is judged against the robots.txt capture that was live at its own timestamp, with RFC-9309 path matching. The call holds nothing: your hits are never written down.

curl -X POST https://bot-witness.com/violations -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"domain":"yoursite.com","agent":"GPTBot","hits":[{"time":"2026-07-10T06:12:01Z","path":"/private/x","agent":"GPTBot"}]}'

Compliance audit: POST /audit Audit / Monitoring

The same evaluation, gathered into a report you can cite: findings per crawler, every disallowed hit as an exhibit, and the governing snapshot named for each. How it works.

curl -X POST https://bot-witness.com/audit -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"domain":"yoursite.com","agent":"GPTBot","log":"<raw access-log lines>","format":"pdf"}' \
  -o compliance-audit.pdf

Probing the models

Exposure report: POST /exposure Exposure / Monitoring

Ask the models what they retained of your text. The report separates what each method can show: verbatim reproduction is a fact about the model's behaviour, while a membership statistic is a relative signal, and we label it as one rather than dressing it up as a probability. GET /exposure/config lists the models probed and the caps. What is in it, and a specimen.

curl -X POST https://bot-witness.com/exposure -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"work_title":"My Novel","corpus":"<paste your text>","acknowledge":true}'
# "format":"pdf" (or ?format=pdf) returns the filing-ready PDF
# "mode":"contamination" runs the same probes as a Benchmark Contamination Report

Opt-out efficacy report: POST /optout/report Opt-Out / Monitoring

The report that needs both halves of the archive: your domain's dated, timestamped opt-out record joined to probes of the models released since that date. How it reads.

curl -X POST https://bot-witness.com/optout/report -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"domain":"yoursite.com","corpus":"<paste your text>","acknowledge":true,"format":"pdf"}' \
  -o optout-efficacy.pdf

Copyright traps: /exposure/traps Exposure / Monitoring

Publish a timestamped random canary and watch for it later. POST creates one (body {work_title?}), GET lists yours with their last probe results, and GET /exposure/traps/<id>/probe re-asks the models now. The daily watch re-captures the public page for free; Exposure Monitoring re-probes the models on its own.

Exposure monitoring: /exposure/watches Exposure Monitoring

Enrol a work for recurring probes, which run monthly and whenever the model roster changes, with an email when the result moves. GET lists watches, POST enrols one ({work_title, corpus, acknowledge}), POST /exposure/watches/<id>/run re-probes now and is rate limited, and DELETE /exposure/watches/<id> stops it.

The other streams

Watch streams: GET /watch, GET /watch/history free coverage

The same recorder archives what the rest of the industry publishes and then edits: gpai for EU training-data disclosures, wba for agent signing-key directories, mcp and mcp-tools for registry entries and live tool manifests, and hf for Hugging Face licence and gating state.

curl "https://bot-witness.com/watch?kind=hf&q=llama"
curl "https://bot-witness.com/watch/history?kind=wba&key=chatgpt.com"

GPAI disclosure report: POST /gpai/report GPAI / Monitoring

The EU AI Act Article 53(1)(d) record as something you can file: every dated revision of each watched disclosure surface, its cadence against the six-month yardstick, and an optional search for your terms in the latest published text. How it works.

curl -X POST https://bot-witness.com/gpai/report -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"terms":["yoursite.com"],"format":"pdf"}' -o gpai-disclosure-report.pdf

Report: GET /report.json free

The aggregate figures behind the public report, in the shape the homepage reads them.

Evidence packages: per matter

A litigation-grade ZIP holding the captures, the RFC-3161 tokens, the Merkle proofs and a standalone verifier, for a domain and a date range. We assemble these by hand rather than over the API. For counsel.

When something goes wrong

Errors come back as JSON with an error key, and the status code tells you whether to fix the request, buy something, or wait.

A 402 is worth reading rather than retrying: it names the window you have and the plan that would widen it. A 404 on a domain you expected means we are not tracking it, which /coverage will confirm in one call.

Rate limits

Unauthenticated requests are limited to 60 in any 60 seconds from one IP address. A 429 carries a Retry-After header and the same number in the body, so you can sleep exactly as long as you need to. Requests carrying a valid API key or a logged-in session skip the limiter, as do the static pages and the health check.

What a plan changes

Current lookups, coverage and the watch streams stay free whatever you do. A plan widens the window and unlocks the endpoints that produce documents.

What each plan costs, and Enterprise if you want the archive in bulk rather than a query at a time.