AI-bot audit.

Why this exists

robots.txt is a deceptively complex file once AI crawlers enter the picture. The combination of explicit allows, explicit blocks, default-allow inheritance, wildcard rules, and per-path overrides means the question "is GPTBot allowed on this URL?" is rarely answerable by visual inspection alone. The AI-bot audit endpoint resolves the question programmatically: input a URL, output a clean policy map.

Teams use this for three things primarily. Site owners use it to verify their own policy after updates, catching cases where a CDN or framework default silently overrode the intent. SEO and AEO teams use it to audit competitor sites, understanding who is and is not allowing AI assistants to read their content. Compliance teams use it to verify that customer or partner sites match the access expectations stated in their agreements.

What you get back

GET /v1/robots-policy?url=https://example.com returns a JSON map of known AI crawlers and their resolved status: allowed, blocked, unspecified, or silently_allowed (the implicit allow when no rule matches).

curl -X GET \
  "https://api.crawlcrawl.com/v1/robots-policy?url=https://example.com" \
  -H "Authorization: Bearer crk_..."

# returns
{
  "GPTBot": "allowed",
  "ChatGPT-User": "allowed",
  "ClaudeBot": "blocked",
  "anthropic-ai": "allowed",
  "PerplexityBot": "unspecified",
  "Google-Extended": "allowed",
  "Applebot-Extended": "silently_allowed",
  "Bytespider": "blocked",
  "CCBot": "unspecified",
  "Meta-ExternalAgent": "allowed"
}

Reading the policy values

Four values, each meaning something different:

• allowed — the site has an explicit rule that permits this crawler. Intentional access.
• blocked — the site has an explicit rule that disallows this crawler. Intentional block.
• unspecified — the site has no rule mentioning this crawler. Behavior depends on the default rule (usually User-agent: *).
• silently_allowed — no specific rule, but the default rule permits all bots. Most public sites resolve this way unless they have explicit AI-crawler rules.

For teams running AEO programs, the practical question is usually: is this crawler effectively allowed (either allowed or silently_allowed) or effectively blocked (blocked)? An unspecified status under a permissive default is functionally allowed.

Use cases at scale

For batch audits across many URLs, the endpoint accepts a single URL per call. Combine it with the search API or a list of customer domains to audit a portfolio. Teams running multi-tenant products use this pattern to verify that customer-managed sites match the access expectations baked into their service agreements.

Pricing for AI-bot audits

Each audit counts as one lightweight request. The free tier covers significant monthly audit volume. Pro at $8/mo covers 5,000 audits; Studio at $42 covers 100,000. The audit endpoint typically consumes a fraction of a crawl's credit cost. See full pricing →

Where it fits in the platform