Monitor — Scheduled Crawls with Change-Detection Webhooks

Why a monitor is different from a crawl

A one-off crawl is a snapshot. A monitor is the continuous version of the same shape: the crawl runs on a schedule, the result is hashed and compared against the previous run, and your webhook fires only when the content actually changed. The difference matters because production content workloads rarely care about a single snapshot. They care about deltas: which page changed since the last run, when it changed, and by how much.

For teams running content monitoring, competitive intelligence, regulatory watch, or RAG corpus refresh, the monitor primitive removes an entire layer of orchestration. The cron lives inside crawlcrawl rather than your scheduler. The deduplication happens before the webhook fires rather than after. The retry policy is HMAC-signed with exponential backoff and dead-letter terminal state. Your service either receives the delivery or knows it did not.

Create a monitor

A crawl with a cron field becomes a recurring monitor. Add return_only_changed: true and webhooks fire only on real changes.

curl -X POST https://api.crawlcrawl.com/v1/crawls \
  -H "Authorization: Bearer crk_..." \
  -d '{
    "url": "https://competitor.com/pricing",
    "max_pages": 1,
    "cron": "0 */6 * * *",
    "webhook_url": "https://you.example.com/hooks/crawl",
    "return_only_changed": true
  }'

# returns
{ "monitor_id": 42, "schedule": "0 */6 * * *", "next_fires_at": "2026-05-16T14:00:00Z" }

The cron expression follows the standard five-field format (minute, hour, day-of-month, month, day-of-week). Common patterns include hourly (0 * * * *), every six hours (0 */6 * * *), nightly at UTC midnight (0 0 * * *), and weekly on Monday mornings (0 8 * * 1). For higher-frequency monitoring, tiers above Pro support sub-hourly cadence.

Edit, pause, or delete

Active monitors are visible through the management endpoints and the dashboard. You can list them, change their schedule, pause them temporarily, or remove them entirely without recreating the original crawl configuration.

GET    /v1/crons              → list your monitors
PATCH  /v1/crons/{id}         → { "schedule": "0 9 * * *" }  or  { "enabled": false }
DELETE /v1/crons/{id}         → remove the monitor

Webhook signature and reliability

Every webhook delivery includes a signature header: X-CrawlCrawl-Signature: sha256=<hex>. The signature is HMAC-SHA256 over the raw request body, keyed by your project's webhook secret (which you fetch from GET /v1/webhook/secret). Verify the signature in your handler before processing the payload to confirm the delivery originated from crawlcrawl and was not tampered with in transit.

Delivery is retried with exponential backoff. The first attempt fires immediately on crawl completion; subsequent attempts back off across roughly five intervals before the delivery is marked dead-letter. The current state of each delivery (queued, delivered, dead) is queryable through the run record, so your operations dashboard can surface failed deliveries without polling.

What teams use monitors for

Three patterns explain most monitor usage.

The first is competitive monitoring. A team in e-commerce, SaaS, or financial services pins the competitor pricing pages, marketing landing pages, or product catalog. The webhook fires the moment a price changes, a new tier launches, or a positioning shift appears. The alerting closes the gap between competitive intelligence and action from days to minutes.

The second is content freshness for RAG. A team running a retrieval-augmented-generation product pins the documentation sources their AI tutor depends on. Daily or hourly monitors detect doc updates, fire webhooks into the ingestion pipeline, and the embeddings refresh against only the pages that changed. The corpus stays current without re-embedding the whole site.

The third is regulatory and compliance watch. Teams in regulated industries monitor agency sites, government portals, and industry standards bodies for rule changes. The signature-verified webhook becomes the audit trail.

"We cut our security asset-discovery pipeline from eight services to one. The dataset diff endpoint is what closed the deal."
— Rajesh Meta, Co-founder & CTO, Quick ZTNA

Pricing for monitors

Monitors are included at every paid tier. Each scheduled run consumes credits at the same rate as a normal crawl; the cron orchestration itself does not cost extra. The free tier supports limited monitor count for evaluation; Pro, Studio, Agency, and Scale support increasing concurrency and frequency. Enterprise plans support sub-minute cadence and dedicated capacity.

See full pricing →

Monitor.