Webhooks.

Set a webhook URL

Include webhook_url in your POST /v1/crawls request. We POST to it when the crawl completes. For monitors with return_only_changed:true, deliveries occur only when content changes.

Signature header

We send the signature in the X-Crawler-Signature header: sha256=<hex>. This is HMAC-SHA256 of the raw request body, keyed by your project's webhook secret.

Retrieve your secret

Call GET https://api.crawlcrawl.com/v1/webhook/secret with your API key. The response:

{
  "secret_hex": "a1b2c3d4e5f6...",
  "alg": "sha256",
  "header": "X-Crawler-Signature",
  "signature_format": "sha256=..."
}

Fetch this endpoint once and cache the response server-side.

Verify in Python

import hmac
import hashlib

def verify(body: bytes, header_sig: str, secret_hex: str) -> bool:
    expected = hmac.new(bytes.fromhex(secret_hex), body, hashlib.sha256).hexdigest()
    got = header_sig.removeprefix("sha256=")
    return hmac.compare_digest(expected, got)

Verify in Node.js

const crypto = require('crypto');

function verify(body, headerSig, secretHex) {
  const hmac = crypto.createHmac('sha256', Buffer.from(secretHex, 'hex'));
  hmac.update(body);
  const expected = hmac.digest('hex');
  const got = headerSig.replace('sha256=', '');
  return crypto.timingSafeEqual(Buffer.from(expected), Buffer.from(got));
}

Retry policy

On 5xx responses, we retry with exponential backoff for up to 24 hours. Failed deliveries are logged and can be retrieved via GET /v1/logs.