By output · Website to markdown

Website to markdown API: crawl a whole site to markdown, one document per page

The short answer

A website to markdown API crawls an entire site from a single seed URL and returns one clean markdown document per page, rather than making you convert pages one at a time. You give it a starting URL, a path prefix to stay inside and a page limit; it discovers the links, renders each page, strips the shared navigation and footer, and hands back a set of documents in a consistent shape. This is how teams turn a documentation site into a knowledge base for retrieval in a single job. For converting one page you already have in hand, use the single URL endpoint instead. Plans start at $39 a month.

Clean markdown & JSON · JavaScript rendered · robots.txt respected

Last updated August 2026

Live Extraction

Endpoint · POST /v1/extract

POST

try:

Hit Extract to turn this page into clean, LLM-ready data.

robots.txt respected · public data only

Markdown · JSON · structured fields, from one API call. Crawling, rendering and extracting ...

Converting one page to markdown is easy. Converting a whole site is where it gets awkward, because you have to discover the URLs, stay inside the section you actually want, avoid re-crawling the same content under three different paths, and strip the navigation that repeats on every page. A website to markdown API handles that whole loop for you.

Give ClawEngine a seed URL and a path prefix and it walks the site, renders each page, removes the shared chrome, and returns one tidy markdown document per page with headings, lists, tables and links intact. Every document comes back in the same shape, so the output drops straight into a chunking and embedding step with no per-page special casing.

The most common reason people run this is retrieval. A product documentation site, a help center or a policy library becomes a clean corpus in one job instead of thousands of individual requests. If you only need a single page converted, the HTML to markdown API is the simpler call and it is linked below. ClawEngine crawls public, permitted pages only, respects robots.txt and Terms of Service, and honors crawl-delay.

CRAWL RENDER JS EXTRACT MARKDOWN JSON

Any URL in LLM-ready data out

robots.txt respected public data only

Why it works

What you get with Website to markdown

One seed URL, the whole site

Point the crawler at a starting URL with a path prefix and it discovers the rest, so you never assemble the page list by hand.

One document per page

Every page comes back in the same markdown shape with the shared navigation and footer stripped, so chunking needs no per-page special casing.

Scoped and capped

A path prefix keeps the crawl inside the section you want and a page limit caps the cost, so a faceted filter cannot run away with your budget.

What it handles

Any URL in, clean structured data out

Point ClawEngine at a public page and it crawls, renders the JavaScript and extracts clean markdown or typed JSON in one call. Define a schema for structured fields, and respect robots.txt and Terms of Service by default.

Crawls a whole site from one seed URL
Returns one clean markdown document per page
Scopes the crawl by path prefix and page limit
Strips repeated navigation, ads and footers
Renders JavaScript before converting
Respects robots.txt, ToS and crawl-delay

POST /v1/extract extraction result

200 · JSON

{
  "url": "https://example.com/products/atlas",
  "title": "Atlas Field Notebook",
  "markdown": "# Atlas Field Notebook\n\nDurable...",
  "data": {
    "name": "Atlas Field Notebook",
    "price": 24.00,
    "currency": "USD",
    "rating": 4.7
  },
  "links": [ "/products", "/cart" ],
  "metadata": { "rendered": true }
}

JS rendered · boilerplate stripped ✓ robots.txt respected

Why ClawEngine

One API that crawls, renders and extracts

Not a raw HTML dump, not a headless browser fleet to run, and not a brittle parser to maintain. One call crawls a public page, renders its JavaScript and returns clean markdown or typed JSON, built for RAG pipelines and AI agents.

LLM-ready output

Clean markdown or typed JSON with the boilerplate stripped, so the data drops straight into a vector store, a prompt or an agent without a cleanup step.

JavaScript rendered

Each page loads in a real browser environment before extraction, so single-page apps and client-rendered content come back complete, not as an empty shell.

Compliance-first

ClawEngine works on public, permitted data only. It respects robots.txt and site Terms of Service and honors crawl-delay, so responsible scraping is the default.

Code examples

Crawl a site to markdown

Give the crawl endpoint a seed URL, a path prefix and a page limit. It returns one markdown document per page. The second sample re-crawls on a schedule and only re-embeds the pages whose content actually changed.

bash Crawl a docs tree

curl -X POST https://api.clawengine.ai/v1/crawl \
  -H "Authorization: Bearer $CLAWENGINE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/docs",
    "path_prefix": "/docs",
    "limit": 500,
    "format": "markdown",
    "render": true
  }'

python Refresh only the pages that changed

import os, json, hashlib, pathlib, requests

headers = {"Authorization": f"Bearer {os.environ['CLAWENGINE_API_KEY']}"}
state = pathlib.Path("crawl_state.json")
known = json.loads(state.read_text()) if state.exists() else {}

res = requests.post("https://api.clawengine.ai/v1/crawl", headers=headers, json={
    "url": "https://example.com/docs",
    "path_prefix": "/docs",
    "limit": 500,
    "format": "markdown",
    "render": True,
})
res.raise_for_status()

changed = []
for page in res.json()["pages"]:
    digest = hashlib.sha256(page["markdown"].encode()).hexdigest()
    if known.get(page["url"]) != digest:
        known[page["url"]] = digest
        changed.append(page)

state.write_text(json.dumps(known))
print(f"{len(changed)} pages changed, re-embed only these")

Website to markdown API: the questions buyers ask

How do I convert an entire website to markdown?

Crawl it from a seed URL rather than converting page by page. Pass the starting URL, a path prefix so the crawler stays in the section you want, and a page limit, and the API discovers the links, renders each page and returns a markdown document per page. One job replaces thousands of individual conversions, and every document arrives in the same shape.

How do I turn a documentation site into a knowledge base?

Crawl the docs tree to markdown, then chunk on headings and embed the chunks. Markdown is the right intermediate format because the heading structure survives the conversion, which gives your chunker natural boundaries. Keep each page URL alongside its text so answers can cite the source page, which is what makes the finished assistant trustworthy rather than merely fluent.

How do I stop the crawler wandering off the section I want?

Set a path prefix and a page limit, and both matter. A prefix like /docs keeps the crawl inside the documentation tree instead of following links into the marketing site or a blog archive. The page limit is your cost ceiling and your protection against a calendar or a faceted filter that generates effectively infinite URLs.

How does a crawl handle duplicate pages?

The same content often sits at several URLs: with and without a trailing slash, under a print view, or behind tracking parameters. ClawEngine normalizes URLs and skips pages it has already fetched, but you should still deduplicate the text before embedding, because near-identical pages that survive the crawl inflate your index and distort retrieval.

How long does it take to crawl a site to markdown?

It depends on the page count and how politely you crawl, and politeness is not optional. Honoring crawl-delay is what keeps you from behaving like a denial of service against the site you are collecting from. A few hundred documentation pages is typically minutes; a large site with tens of thousands of pages is a background job you queue rather than wait on.

Should I crawl a site or use its sitemap?

Use the sitemap when there is a good one. It gives you the canonical URL list without discovery, so you skip the crawl entirely and convert a known set of pages. Fall back to crawling when the sitemap is missing, stale or incomplete, which is common on help centers. In practice many teams do both: seed from the sitemap, then crawl to catch what it missed.

Good questions

Questions about Website to markdown

Markdown is compact, readable and the format most LLM and docs tools expect. ClawEngine strips the boilerplate and keeps the structure, so you get content that embeds cleanly for retrieval without the noise of raw HTML.

Yes. The page is rendered in a real browser environment before conversion, so JavaScript-loaded content appears in the markdown. It only processes public, permitted pages and respects robots.txt and Terms of Service.

Yes, and for a docs corpus you should. Documentation changes constantly, and a knowledge base built once quietly drifts out of date until it starts giving people answers that were true last quarter. Re-crawl on a schedule, compare each page against the version you stored, and re-embed only what actually changed.

Scope, and nothing else. The HTML to markdown API converts one URL you already know and returns one document. This page is the crawl: you give a seed URL, it finds the pages itself and returns the set. Use the single-page call inside your own loop if you already hold the URL list, and use the crawl when you do not.

Read every web scraping question

Explore more

More ways to turn the web into data with ClawEngine

See every use case See pricing Back to the web scraping API

Stop wrangling raw HTML. Get LLM-ready data.

Point ClawEngine at a public page and one call crawls, renders the JavaScript and extracts clean markdown or typed JSON, ready for your RAG pipeline or AI agent. Public, permitted data only.

See pricing

Crawl · render JS · extract markdown & JSON · robots.txt respected, public data only