By output · llms.txt generator

llms.txt generator: build your llms.txt file and llms-full.txt from a live crawl

The short answer

An llms.txt generator crawls your site and produces the two files AI agents read: llms.txt, a curated markdown index of your best pages, and llms-full.txt, the full markdown body of those pages concatenated into one document. Worth knowing before you spend a sprint on it: Google states plainly that Search ignores llms.txt and that creating one will neither help nor harm your rankings, so this is not an SEO tactic. The payoff sits elsewhere, with the agent frameworks and developer tools that do fetch these files, and with your own retrieval stack. ClawEngine generates both from a live crawl of public, permitted pages, and regenerates them on a schedule so they never drift from the docs they describe. Plans start at $39 a month.

Clean markdown & JSON · JavaScript rendered · robots.txt respected

Last updated September 2026

Live Extraction

Endpoint · POST /v1/extract

POST

try:

Hit Extract to turn this page into clean, LLM-ready data.

robots.txt respected · public data only

Markdown · JSON · structured fields, from one API call. Crawling, rendering and extracting ...

The index file is not the hard part. An llms.txt is a heading, a one-line summary and a list of links, and if your site is twenty pages you should write it by hand this afternoon and skip the tooling entirely. We would rather tell you that than sell you something.

The work starts at llms-full.txt and the per-page markdown companions. Those need every page you list, rendered, stripped of navigation and ads, converted to clean markdown, and concatenated in a stable order. On a documentation site of five hundred pages that is a crawl, not a copy and paste, and it is stale the week after you ship it unless something regenerates it.

That is the job ClawEngine does. Point a crawl at your docs tree, get one clean markdown document per page back, and assemble the two files from the result in about twenty lines. Re-run it on a schedule and the files track the site. ClawEngine crawls public, permitted pages, reads robots.txt and honors crawl-delay, which for your own site means it behaves like a well-mannered guest on your own infrastructure.

CRAWL RENDER JS EXTRACT MARKDOWN JSON

Any URL in LLM-ready data out

robots.txt respected public data only

Why it works

What you get with llms.txt generator

Rendered, then converted

Docs sites built with React or Vue return an empty shell to a plain fetch. ClawEngine renders each page before converting, so the markdown in your llms-full.txt is the content readers actually see.

Whole tree in one job

Give the crawl a seed URL, a path prefix and a page limit, and it walks the docs tree and returns one clean markdown document per page in the same shape, ready to concatenate.

Regenerates on a schedule

Re-run the same crawl from your build or a weekly job and both files track the site, which is the difference between a live corpus and a snapshot that quietly goes stale.

What it handles

Any URL in, clean structured data out

Point ClawEngine at a public page and it crawls, renders the JavaScript and extracts clean markdown or typed JSON in one call. Define a schema for structured fields, and respect robots.txt and Terms of Service by default.

Crawls a docs tree to clean markdown
Builds llms.txt and llms-full.txt from one job
Produces per-page .md companion files
Renders JavaScript before converting
Strips navigation, ads and scripts
Re-runs on a schedule so files stay current

POST /v1/extract extraction result

200 · JSON

{
  "url": "https://example.com/products/atlas",
  "title": "Atlas Field Notebook",
  "markdown": "# Atlas Field Notebook\n\nDurable...",
  "data": {
    "name": "Atlas Field Notebook",
    "price": 24.00,
    "currency": "USD",
    "rating": 4.7
  },
  "links": [ "/products", "/cart" ],
  "metadata": { "rendered": true }
}

JS rendered · boilerplate stripped ✓ robots.txt respected

Why ClawEngine

One API that crawls, renders and extracts

Not a raw HTML dump, not a headless browser fleet to run, and not a brittle parser to maintain. One call crawls a public page, renders its JavaScript and returns clean markdown or typed JSON, built for RAG pipelines and AI agents.

LLM-ready output

Clean markdown or typed JSON with the boilerplate stripped, so the data drops straight into a vector store, a prompt or an agent without a cleanup step.

JavaScript rendered

Each page loads in a real browser environment before extraction, so single-page apps and client-rendered content come back complete, not as an empty shell.

Compliance-first

ClawEngine works on public, permitted data only. It respects robots.txt and site Terms of Service and honors crawl-delay, so responsible scraping is the default.

Code examples

Generate llms.txt and llms-full.txt from one crawl

Crawl the docs tree once, then write both files from the same result. The third sample saves the per-page .md companions the spec recommends, so every link in your index has a clean markdown version beside it.

bash Crawl the docs tree to markdown

curl -X POST https://api.clawengine.ai/v1/crawl \
  -H "Authorization: Bearer $CLAWENGINE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/docs",
    "path_prefix": "/docs",
    "limit": 500,
    "format": "markdown",
    "render": true
  }'

python Write llms.txt and llms-full.txt

import os, requests

headers = {"Authorization": f"Bearer {os.environ['CLAWENGINE_API_KEY']}"}

pages = requests.post("https://api.clawengine.ai/v1/crawl", headers=headers, json={
    "url": "https://example.com/docs",
    "path_prefix": "/docs",
    "limit": 500,
    "format": "markdown",
    "render": True,
}).json()["pages"]

# The index: H1, blockquote summary, then H2 sections of markdown links.
index = ["# Example", "", "> Example is an API for doing the thing you came here for.", "", "## Docs", ""]
for p in pages:
    title = p.get("title") or p["url"]
    summary = (p.get("description") or "").strip()[:300]
    index.append(f"- [{title}]({p['url']}.md): {summary}" if summary else f"- [{title}]({p['url']}.md)")

open("public/llms.txt", "w").write("\n".join(index) + "\n")

# The corpus: every page body inline, separated by a divider.
full = []
for p in pages:
    full.append(f"# {p.get('title') or p['url']}\nSource: {p['url']}\n\n{p['markdown']}")

open("public/llms-full.txt", "w").write("\n\n---\n\n".join(full) + "\n")

print(f"indexed {len(pages)} pages")

javascript Save the per-page .md companions

import { mkdir, writeFile } from "node:fs/promises";

const res = await fetch("https://api.clawengine.ai/v1/crawl", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.CLAWENGINE_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://example.com/docs",
    path_prefix: "/docs",
    limit: 500,
    format: "markdown",
    render: true,
  }),
});

const { pages } = await res.json();

// The spec suggests a clean markdown version of each page at page.md,
// so every link in llms.txt has a companion an agent can fetch directly.
for (const page of pages) {
  const path = new URL(page.url).pathname.replace(/\/$/, "") || "/index";
  await mkdir(`public${path.split("/").slice(0, -1).join("/")}`, { recursive: true });
  await writeFile(`public${path}.md`, page.markdown);
}

console.log(`wrote ${pages.length} markdown companions`);

llms.txt generator: the questions buyers ask

What is an llms.txt file?

It is a markdown file at the root of a site that tells an AI agent which pages matter and where to read them. The spec is short: an H1 with the project name, a blockquote summary, then H2 sections listing links in standard markdown link format with optional notes. Jeremy Howard published it on 3 September 2024, and version 2 of the spec landed on 10 August 2026.

Does Google use llms.txt?

No. Google Search Central says Google Search does not require new machine readable files to appear in Search including its generative AI features, that Search ignores llms.txt, and that creating one will neither harm nor help your visibility or rankings. Google adds that it is completely fine to maintain the file for other services that do use it. Treat llms.txt as agent plumbing, not as an SEO lever.

What is the difference between llms.txt and llms-full.txt?

llms.txt is a compact index: links plus one-line descriptions, meant for an agent that will fetch the pages it wants afterwards. llms-full.txt inlines the actual content, concatenating the full markdown body of every listed page into one document so an agent can load your whole corpus in a single request and skip the round trips. Index versus corpus, and most docs teams ship both.

How do I generate an llms.txt file for my website?

Decide which pages belong in it, get clean markdown for each one, then write the index. For a small site, curate by hand. For a documentation site, crawl the tree, convert each page to markdown, and generate both files from that output in a build step so they regenerate whenever the docs change. The curation is a judgment call you should make; the markdown conversion is the part worth automating.

Is llms.txt worth it for SEO?

Not for Google rankings, and anyone telling you otherwise is contradicting Google. It is worth doing when your product is documentation that developers and coding agents read, which is why Anthropic, Stripe, Cursor, Cloudflare, Vercel, Supabase and Mintlify publish one. If your site is a marketing site with no reference material, the honest answer is that your effort is better spent on the content itself.

How often should I regenerate llms.txt?

On the same trigger that publishes your docs. A file that describes last quarter of your product is worse than no file, because an agent that fetches it gets confidently wrong answers with your name attached. Wire generation into the build, or run a scheduled crawl weekly for sites that publish continuously. The failure mode here is silent drift, not a broken file.

Do I need llms.txt if I already have robots.txt and a sitemap?

They solve different problems. robots.txt says what a crawler may fetch, a sitemap lists every URL you want indexed, and llms.txt says which handful of pages actually explain your product and where the clean markdown lives. A sitemap of four thousand URLs is a discovery aid; an llms.txt of thirty curated links is an editorial statement about what matters.

How do I validate an llms.txt file?

Check three things mechanically before you check taste. It parses as markdown with exactly one H1 and a blockquote directly under it; every link resolves to a live page and, where you offer them, the .md companion returns markdown rather than an HTML error page; and no listed URL redirects or 404s. Broken companion URLs are the most common defect, because they are generated separately from the index and nothing tests them.

Good questions

Questions about llms.txt generator

If your site is small, no, and we would rather say so plainly. An llms.txt for a twenty page site is thirty lines of markdown you can write faster than you can evaluate a vendor. A tool earns its place when you need llms-full.txt across hundreds of pages, when those pages render client-side, and when the files have to regenerate on every docs release without anyone remembering to do it.

No. ClawEngine crawls and returns clean markdown; writing the two files and serving them from your root is a step you own, and it is a few lines in your build. We keep that boundary on purpose, because the curation of which pages belong in your llms.txt is an editorial decision about your own product that no crawler should be making for you.

You can crawl any public, permitted page, so building a markdown corpus of a vendor documentation site for your own retrieval system is a normal use. Publishing an llms.txt that claims to represent somebody else at their root is not something you can do anyway, since you cannot write to their server. ClawEngine reads robots.txt and honors crawl-delay either way.

We do not do that. ClawEngine works on public and permitted pages and does not authenticate into accounts or work around paywalls. For private docs, generate the files from your source repository at build time, where you already hold the markdown, rather than crawling the rendered site. That is the better pipeline for internal content regardless of tooling.

If your docs already live on a platform that generates both files, use it and stop reading. Mintlify hosts llms.txt and llms-full.txt automatically for projects on its platform, and a built-in feature beats an integration every time. ClawEngine is for the case where your docs are custom, spread across several properties, or where you also want a markdown corpus of sources you do not control.

Read every web scraping question

Explore more

More ways to turn the web into data with ClawEngine

See every use case See pricing Back to the web scraping API

Stop wrangling raw HTML. Get LLM-ready data.

Point ClawEngine at a public page and one call crawls, renders the JavaScript and extracts clean markdown or typed JSON, ready for your RAG pipeline or AI agent. Public, permitted data only.

See pricing

Crawl · render JS · extract markdown & JSON · robots.txt respected, public data only