Competitor monitoring.

Why this is hard to DIY

Cron a scraper, diff the output, suppress noise from cookie banners and tracking pixels, sign the webhook, retry on receiver failures. Multi-tenant if you're building a product. That's a week of work; we do it as one POST.

Set up a monitor

curl -X POST \
  https://api.crawlcrawl.com/v1/crawls \
  -H 'Authorization: Bearer crk_...' \
  -H 'Content-Type: application/json' \
  -d '{
        "url": "https://example.com/pricing",
        "cron": "0 * * * *",
        "webhook_url": "https://your-site.com/hook",
        "return_only_changed": true
      }'

What the webhook looks like

{
  "event": "crawl.done",
  "id": "crawl-123",
  "status": "success",
  "page_count": 1,
  "started_at": "2023-12-01T12:00:00Z",
  "finished_at": "2023-12-01T12:00:01Z",
  "url": "https://example.com/pricing"
}

Verify the signature

import hmac
import hashlib

secret = 'your_webhook_secret'
signature = req.headers['X-Crawlcrawl-Signature']
body = req.body
expected_signature = hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
if not hmac.compare_digest(signature, expected_signature):
    return 'Unauthorized', 401

Scale

One monitor per competitor page. Agency tier: 25 active monitors. Each fires on its own cron schedule.