Per-site scrapers.

How it works

POST /v1/cloud/fetch/{domain}/{path} runs a pre-configured scraper for the target domain.

curl -X POST \
  https://api.crawlcrawl.com/v1/cloud/fetch/quotes.toscrape.com/v1/quote \
  -H 'Authorization: Bearer crk_...' \
  -H 'Content-Type: application/json'

Browse available scrapers

GET /v1/cloud/scrapers returns the current catalog:

{
  "scrapers": [
    {
      "domain": "quotes.toscrape.com",
      "path_pattern": "/v1/quote",
      "description": "Scrape a random quote",
      "confidence_score": 0.95
    },
    {
      "domain": "books.toscrape.com",
      "path_pattern": "/v1/book",
      "description": "Scrape book listings",
      "confidence_score": 0.8
    }
  ],
  "pagination": {
    "next": "/v1/cloud/scrapers?page=2"
  }
}

When to use this vs /v1/crawls

Use /v1/cloud/fetch when a pre-built scraper exists for your target. Use /v1/crawls for arbitrary sites or full link-following.

Request a new scraper

If you need a scraper for a site not in the catalog, email [email protected].