Data retention.

Crawl data: kept until you delete it

Every page from every crawl you run stays in your account permanently, until you call DELETE /v1/crawls/{id}. There is no automatic expiry. The "permanent dataset archive" promise on the homepage is enforced in code, not just policy: no scheduled job removes crawl pages, and re-pulling a crawl via GET /v1/crawls/{id}/export costs zero credits no matter how old it is.

What this means concretely:

• A crawl you ran today is readable tomorrow, next month, and next year — with the same JSON shape, the same markdown content, the same metadata.
• If we ever need to change this (cost pressure, infrastructure migration, legal requirement), you get at least 90 days’ written notice by email, with an export window before any data is touched.
• If your account is closed (cancelled or non-payment past grace period), crawl data is retained for 30 days so you can resume or export, then permanently deleted.

When you DELETE a crawl

Calling DELETE /v1/crawls/{id} removes the SQL row and the page HTML archive in one operation:

• The crawl record and all its pages are removed from the primary database within the request.
• The HTML archive objects for each page are deleted asynchronously within a few seconds.
• Backups taken before the deletion follow the backup retention schedule below.

Once both stores are cleared, the data cannot be recovered. There is no soft-delete or trash.

Backups

Encrypted database backups are kept for 30 days rolling. Backups exist only for disaster recovery — we do not restore individual deleted crawls from backup on customer request. After 30 days, a deleted crawl is gone from backups too.

API request logs

90 days. Each request line includes path, timestamp, API key prefix (never the full key), source IP, response status, and credit cost. Used for usage reporting, abuse investigation, and customer-support diagnostics. After 90 days the per-request log is purged; aggregated daily usage counters are kept indefinitely so historical usage views work.

Account records

• Project + API key metadata: kept as long as the account exists. Deleted with the account.
• Failed-login attempts: 30 days, used for rate limiting and account-lockout enforcement.
• Web session cookies: 30 days from last activity.
• Webhook deliveries: last 1,000 attempts per project are retained for replay/debug; older entries are pruned.

Your control

• Delete a single crawl: DELETE /v1/crawls/{id}
• Export before deleting: GET /v1/crawls/{id}/export returns the whole crawl as NDJSON, zero credits.
• Delete your account: email [email protected] with your project id. We confirm within one business day and execute within seven days.
• Audit log for your project: GET /v1/audit/log returns recent key rotations, deletions, and admin actions.

Compliance requests

For GDPR, India DPDP, and equivalent regimes: email [email protected]. We acknowledge within 72 hours and respond within 30 days. Subject access, data portability, and erasure are all supported via the endpoints above; we can also run them manually if you prefer.

Changes to this page

Material changes to retention windows trigger an email to all active accounts and a banner here. Last updated 2026-05-31.