SEO Audit at Scale — API-Driven Orphan Detection, Broken Links, Site Diff

Why an API beats a desktop tool for SEO at scale

Desktop SEO crawlers have served the industry well for years. For agency teams running audits across many client sites, scheduled refreshes, or platform integrations, the desktop model starts to creak. Audits cannot run in CI. Findings cannot trigger ticketing systems automatically. Multi-site portfolios require per-site manual setup. The bottleneck stops being the crawl itself and becomes the orchestration around it.

An API-driven crawler replaces the orchestration. The same audit that an analyst would run interactively becomes a curl command in a cron job, a webhook into the team's Slack, a row in a database that grows over time. The audit work shifts from "the analyst sits at the keyboard" to "the system surfaces what changed."

What the audit checks

Every crawl produces several SEO primitives ready for inspection.

Orphan pages — pages reachable in the crawl but with no incoming internal links. Often a sign of broken information architecture or accidental page leaks.
Broken links — URLs returning 4xx or 5xx status codes. Both internal and outbound broken links surface in the run.
Robots policy — how each major search engine and AI crawler is treated by the site's robots.txt and meta directives.
Structured-data audit — schema.org markup, Open Graph tags, JSON-LD blocks, and hreflang relationships extracted alongside content.
Site diff — pages added, removed, and changed between any two audit runs.
Canonical and meta consistency — canonical URLs, robots meta directives, and per-page metadata returned in a uniform shape.

The endpoints SEO teams use most

GET  /v1/crawls/{id}/orphans      # pages with no inbound internal links
GET  /v1/crawls/{id}/links        # full link graph
GET  /v1/crawls/{id}/pages        # markdown + metadata + schema per page
GET  /v1/crawls/{old}/diff/{new}  # what changed between two audits
GET  /v1/robots-policy?url=...    # per-crawler policy resolution

POST /v1/actors/audit-onpage      # 25+ on-page rules in one call
POST /v1/actors/check-links       # broken-link discovery, bulk up to 200
POST /v1/actors/structured-data   # JSON-LD, Microdata, RDFa, OpenGraph, Dublin Core, Microformats
POST /v1/actors/extract-article   # clean article text + metadata + reading time
POST /v1/actors/render-diff       # static vs JS-rendered diff, ai_bot_blind_pct
POST /v1/actors/internal-link-graph # PageRank + orphan clusters over a crawl
POST /v1/actors/sitemap-audit     # 7-bucket sitemap health (ok/redirect/4xx/5xx/noindex/...)

The seven /v1/actors/* endpoints package the most common audit workflows into a single POST per URL. Use them as drop-in checks inside your existing pipeline, or chain them after a full POST /v1/crawls to audit every page in a site. The three newest — render-diff, internal-link-graph, and sitemap-audit — cover the AEO smoking-gun report, PageRank-style internal authority analysis, and one-shot sitemap health respectively.

Example: an orphan-page report from a finished crawl.

GET /v1/crawls/{id}/orphans

# returns
{
  "orphans": [
    { "url": "https://example.com/old-promo", "title": "Old Promotion" },
    { "url": "https://example.com/legacy-doc", "title": "Legacy Documentation" }
  ],
  "total": 2
}

Schedule weekly audits on cron

Run the audit every Monday at 6 AM UTC, with results delivered via webhook to your dashboard or ticketing system.

curl -X POST https://api.crawlcrawl.com/v1/crawls \
  -H "Authorization: Bearer crk_..." \
  -d '{
    "url": "https://example.com",
    "max_pages": 5000,
    "cron": "0 6 * * 1",
    "webhook_url": "https://yourstack.com/seo-audit"
  }'

For multi-site agencies, create one scheduled crawl per client site. Each run produces a dataset that pairs with the previous week's run via the diff endpoint. Your dashboard can surface "what changed this week across all clients" as a single view across many crawl IDs.

For agencies running client portfolios

Each crawlcrawl project gets a separate API key. Agencies typically run one project per client, which keeps usage, billing, and audit logs cleanly separated. Studio at $69 a month covers 100,000 page credits across multiple projects, which is enough to audit dozens of small-to-medium client sites weekly. Agency at $279 covers 500,000 credits for larger portfolios.

The benefit for agency operations is the absence of per-seat licensing. A junior team member can run the same audit as a senior analyst because the orchestration is in the API, not in a desktop application. New hires onboard in a day rather than a week.

How this fits a modern SEO program

SEO in 2026 increasingly overlaps with AEO (answer engine optimization). The same audit that surfaces orphan pages and broken links also reports the structured-data signals AI assistants rely on for citation. A team running this audit weekly is, in effect, running an AEO audit at the same time. See our AEO vs SEO guide for the strategic framing; the audit itself produces evidence for both surfaces.

Pricing

SEO audits run on the same credit pool as any other crawl. A weekly audit of a 1,000-page site costs 4,000 credits per month, which fits inside the Pro tier with headroom. Multi-site agencies usually land on Studio at $69 or Agency at $279 depending on portfolio size. See full pricing →

SEO audit at scale.