Anti-bot Scrape — Bypass Cloudflare, Akamai, PerimeterX, Captchas

When you need it, and when you do not

Most pages on the web can be fetched cleanly with a normal request. For those, the standard /v1/scan or /v1/crawls endpoints handle the work at the lowest credit cost. The anti-bot path is the one you reach for when a target site genuinely fights back: protected by Cloudflare's bot management, fronted by Akamai's bot manager, gated behind PerimeterX, locked behind a Datadome challenge, or simply rate-limiting traffic that does not look like a residential customer.

For those targets, the anti-bot scrape endpoint routes the request through our global edge network spanning 190+ countries. Browser fingerprinting, geo-appropriate origin selection, session handling, and captcha resolution happen behind one API call. You do not configure a proxy plan, you do not rotate IPs, you do not manage session stickiness. The response comes back as if your client requested the page from inside the target's preferred audience region.

Heavyweight fetch

/v1/cloud/unblock is the path you reach for when your normal scraper hits a wall. Global routing, anti-bot handling, captcha resolution — you do not tune any of it.

curl -X POST https://api.crawlcrawl.com/v1/cloud/unblock \
  -H "Authorization: Bearer crk_..." \
  -d '{"url":"https://hard-site.example","return_format":"markdown"}'

# returns
{ "url": "...", "status": 200, "content": "...clean markdown...", "cost_usd": 0.001 }

The response includes the cleaned markdown by default. Other return formats (raw HTML, structured signals, screenshot) are available through the same endpoint. For single-URL fetches where you do not know in advance whether the target needs anti-bot routing, the /v1/scan endpoint accepts a cloud_mode parameter that handles the decision per request.

Automatic escalation on single-URL fetches

If you do not know in advance whether a target site has anti-bot protection, set cloud_mode: "auto" on a /v1/scan call. Direct fetch is attempted first. If the response indicates a bot wall (403, 429, 503, Cloudflare challenge page, etc.), the request transparently escalates to the heavier anti-bot path. You only pay the higher rate on the requests that actually need it.

curl -X POST https://api.crawlcrawl.com/v1/scan \
  -H "Authorization: Bearer crk_..." \
  -d '{
    "url": "https://protected-target.com",
    "cloud_mode": "auto"
  }'

For workloads of mostly unprotected sites with occasional anti-bot pages, this is the most cost-effective configuration. For multi-page crawls where every target is expected to fight back, use /v1/cloud/crawl directly — it routes every page through the anti-bot path. For multi-page crawls of open sites, /v1/crawls stays on owned infrastructure at 1 credit per page.

What you get back

The response from an anti-bot scrape is shaped the same way as a normal scrape. Markdown by default, with structured-data signals (schema.org, Open Graph, JSON-LD, hreflang, canonical) returned alongside content. The fact that the request took the anti-bot path is reflected in the response metadata but does not change the data shape, which means downstream pipelines do not need to handle protected and unprotected pages differently.

markdown — boilerplate-stripped, heading-preserving
metadata — title, description, OG, canonical, robots-meta, hreflang
schema — extracted structured-data blocks
status — the target's response status after anti-bot resolution
cost_usd — the per-request cost for accounting

What teams use it for

Three patterns explain most anti-bot scraping volume on crawlcrawl.

The first is competitive intelligence on e-commerce. Many large retailers (Amazon, Walmart, Target, regional grocers, fashion brands) gate traffic that does not look like residential customers. A pricing-tracking team that needs to see the same prices a real shopper sees runs through the anti-bot path. The data quality is materially higher than what direct-fetch scraping returns on those targets.

The second is regulated industries and financial data. SEC filings, regulatory portals, court records, and similar sources often sit behind bot protection meant to discourage automated scraping. Teams with legitimate access needs (compliance, research, journalism) use the anti-bot path to retrieve the same documents reliably.

The third is RAG ingestion of premium content. Some documentation sites, vendor portals, and knowledge bases protect against scrapers. For teams whose RAG corpus depends on those sources, the anti-bot path is the difference between a complete corpus and one with conspicuous gaps.

Pricing for anti-bot scrapes

Anti-bot calls cost 1.5 credits each; plain calls cost 1 credit. For single-URL workloads where most targets are open, using /v1/scan with cloud_mode: "auto" keeps the average per-request cost near 1 credit because only the pages that genuinely hit a bot wall escalate to the 1.5 path.

Anti-bot capability is included at every paid tier. There is no separate "anti-bot mode" surcharge or contract; the credits in your monthly tier cover both direct and anti-bot requests. See full pricing →

Anti-bot Scrape.