Competitor Monitoring — Webhook Alerts When Pricing or Copy Changes

Why this is hard to build yourself

The first version is easy. Set a cron, fetch the page, compare against the last fetch, fire a webhook on difference. The second version is hard. Cookie banners and tracking pixels create noise that triggers false alerts. Anti-bot challenges block the fetch on the third request. Receiver downtime drops alerts you needed. Multi-tenant access (if you are building a product) adds a layer of credential management. A signature verification layer adds another. Two weeks of work compounds into a quarter of "we keep fixing the scraper rather than building product."

crawlcrawl absorbs the whole pattern into one POST. The monitor primitive handles cron, fetch, hash, diff, signature, retry, dead-letter, and re-anti-bot escalation as part of the managed service. Your application stays on the receiving end of cleanly signed delta-only webhooks.

Set up a monitor in one POST

curl -X POST https://api.crawlcrawl.com/v1/crawls \
  -H "Authorization: Bearer crk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://competitor.com/pricing",
    "max_pages": 1,
    "cron": "0 * * * *",
    "webhook_url": "https://yourapp.com/hooks/competitor",
    "return_only_changed": true
  }'

The cron expression follows the standard five-field format. Hourly checks are 0 * * * *; every six hours is 0 */6 * * *; nightly at UTC midnight is 0 0 * * *. Pick a cadence that matches how fast your decisions need to move. For pricing-page monitoring of fast-moving categories, hourly is usually right. For policy-page monitoring, daily is plenty.

What the webhook looks like

{
  "event": "crawl.done",
  "id": "crawl-123",
  "status": "success",
  "page_count": 1,
  "started_at": "2026-05-16T12:00:00Z",
  "finished_at": "2026-05-16T12:00:01Z",
  "url": "https://competitor.com/pricing",
  "content_hash": "...",
  "previous_content_hash": "..."
}

The webhook fires only when the content hash differs from the previous run. Your handler can fetch the full page via GET /v1/crawls/{id}/pages to get the actual diff content; many teams cache the previous markdown and run their own diff to highlight specific changes (a price moved up, a tier got renamed, a CTA shifted).

Verify the signature

Every webhook delivery includes X-CrawlCrawl-Signature: sha256=<hex>. Verify it before processing the payload. The signature is HMAC-SHA256 over the raw request body, keyed by your project's webhook secret.

import hmac, hashlib

def verify(req, secret):
    sig = req.headers.get("X-CrawlCrawl-Signature", "").removeprefix("sha256=")
    expected = hmac.new(secret.encode(), req.body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(sig, expected)

The webhook secret comes from GET /v1/webhook/secret. Rotate it through the dashboard if it is ever compromised; old deliveries with the previous secret continue to verify cleanly during a brief grace window.

What teams monitor in practice

The most common patterns we see are pricing pages (the page that changes most often when a competitor moves), terms of service and acceptable-use policies (legal-and-security teams care), and product release pages (changes here often precede a market move). For competitive-intelligence teams in SaaS, regulated industries, or financial services, the same pattern extends to investor relations pages, careers pages (hiring signals are competitive signals), and changelog or release-note pages.

"We cut our security asset-discovery pipeline from eight services to one. The dataset diff endpoint is what closed the deal."
— Rajesh Meta, Co-founder & CTO, Quick ZTNA

Scale across many competitors

One monitor per page. There is no upper limit on monitor count; the practical limit is your monthly credit pool. Each scheduled run consumes credits at the same rate as a normal crawl, so a 24-page-per-day monitor across hourly cadence costs 24 credits per day or roughly 720 per month. Pro at $8/mo (5K pages) covers a couple of light monitors; Studio at $42 (100K) covers a small monitoring portfolio.

Pricing

Monitors are included at every paid tier. Each scheduled run counts as a normal crawl against your monthly credit pool. The cron orchestration, the diff hashing, the webhook signing and retry are all included. See full pricing →

Competitor monitoring.