llms.txt Builder.

What llms.txt is, and why it matters

llms.txt is an emerging standard file (similar in spirit to sitemap.xml) that summarizes a site for AI ingestion. It lives at the root of your domain, contains a structured plain-text description of what your site offers, and gives AI assistants a clean, low-cost way to understand your business when generating answers for their users.

In 2026, AI assistants are becoming a meaningful source of high-intent referral traffic. When ChatGPT, Claude, Perplexity, or Gemini answer a buyer-intent question, the brands they cite inside the answer get the click. A well-formed llms.txt makes it dramatically easier for those assistants to understand what you do, summarize you accurately, and cite you confidently. See our full AEO vs SEO guide for the strategic context.

Generate for any site

POST /v1/llms-txt-build with a URL and the inventory cap. The crawler maps the site, builds a structured summary, and returns both the short policy file and the detailed inventory variant.

curl -X POST https://api.crawlcrawl.com/v1/llms-txt-build \
  -H "Authorization: Bearer crk_..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "max_pages": 1000}'

# returns
{
  "llms_txt": "# Your Brand\n\n> One-line description...\n\n## Products\n- ...",
  "llms_full_txt": "...detailed inventory with summaries per page...",
  "pages_used": 42
}

What you get back

Two files. The short llms.txt is a structured summary AI assistants ingest cheaply: brand description, product list, pricing summary, key links, and primary use cases. The longer llms-full.txt includes per-page summaries that let AI assistants cite specific pages with confidence. You choose which to publish; most sites publish both, with the short file at /llms.txt and the detailed file at /llms-full.txt.

The output is human-editable. The generated file is a good starting point; treat it as a draft you refine over time, adjusting wording and emphasis to match how you want AI assistants to describe your business.

Common use cases

Three patterns explain most llms.txt builder usage.

The first is launch. Teams generate the file once when they ship a new marketing site, publish both versions, and watch AI-referral traffic grow over the following weeks as assistants re-crawl and pick up the new representation.

The second is refresh on a schedule. Pair the builder with the monitor primitive to regenerate the file weekly or monthly. As your product surface evolves, the AI-readable representation evolves with it. This is the closest thing to an "AEO SEO" cron job we recommend in 2026.

The third is competitive audit. Teams generate llms.txt for competitor sites to understand how those competitors are presenting themselves to AI assistants. The structured output makes the comparison straightforward.

Pricing for llms.txt builder

An llms.txt build counts as a normal crawl against your credit pool. A 100-page inventory costs 100 credits. The free tier covers experimentation; Pro at $8/mo (5K pages) handles small refreshes; Studio at $42 (100K) handles weekly refreshes of large sites or frequent refresh cadence. See full pricing →

Where it fits in the platform