Transform API — HTML and PDF to Clean Markdown

When you already have the bytes

The crawl and scrape endpoints fetch pages for you. Sometimes you already have the HTML or PDF, captured elsewhere in your pipeline, exported from another system, or uploaded by a customer. The transform endpoint is for that case: you provide the raw document; we return clean markdown ready for vector-database ingestion.

This is a small endpoint that punches above its weight. Most teams running RAG pipelines have at least one stage where documents arrive as raw bytes (PDF uploads, archived HTML, exported docs) and need to be converted to clean text before embedding. Building that converter in-house typically takes a few weeks the first time and ongoing maintenance forever. The transform endpoint replaces that work with one API call.

HTML to markdown

curl -X POST https://api.crawlcrawl.com/v1/cloud/transform \
  -H "Authorization: Bearer crk_..." \
  -d '{
    "data": "<html>...</html>",
    "input_kind": "html",
    "return_format": "markdown",
    "readability": true
  }'

# returns
{ "content": "# Page title\n\nMain article body...", "cost_usd": 0.0001 }

PDF to markdown

Same endpoint shape; the input_kind field switches the parser. PDFs are parsed text-layer-first with structure preservation, then converted to markdown using the same readability pass as HTML. The output is suitable for direct chunking and embedding.

curl -X POST https://api.crawlcrawl.com/v1/cloud/transform \
  -H "Authorization: Bearer crk_..." \
  -d '{
    "data": "<base64-encoded-pdf>",
    "input_kind": "pdf",
    "return_format": "markdown"
  }'

Readability mode

Default true. Strips repeated layout (nav, footer, ads, comment threads, sidebars) so what you get back is just the main content. Pass readability: false if you need the full document, including peripheral content. Most RAG pipelines benefit from leaving it on; document-preservation use cases (legal review, compliance archives, full-text indexing) benefit from turning it off.

What teams use it for

Three patterns explain most transform usage.

The first is hybrid pipelines that mix crawl-fetched HTML with user-uploaded documents. The HTML side runs through the crawler endpoint; the upload side runs through transform. Both feed the same vector database with the same output shape.

The second is migration. Teams converting an existing corpus (an archive of HTML pages, a library of PDFs) to a new vector-database-backed format use transform to handle the bulk conversion without writing a custom parser.

The third is internal tools. Engineering teams expose the transform endpoint behind an internal API so non-technical colleagues can drop in any document and receive clean markdown for downstream use.

Pricing for transform

Transform requests cost a fraction of a credit per document on most workloads because the work is parsing rather than fetching. The free tier covers experimentation; production teams typically land on Pro at $8 (5K pages) or Studio at $42 (100K) depending on document volume. See full pricing →

Transform.