By use case · Documentation

Documentation scraper API to crawl API docs and knowledge bases into clean markdown

The short answer

A documentation scraper API crawls a docs site from one seed URL and returns every page as clean markdown with the navigation, version switchers, sidebars and cookie banners removed and the code blocks intact. ClawEngine follows the docs tree, renders pages that build client-side, and hands back markdown or typed JSON you can chunk straight into a vector index, so a support bot or a coding assistant answers from your real documentation instead of a stale copy. It reads robots.txt and stays on public, permitted pages. Plans start at $39 a month.

Clean markdown & JSON · JavaScript rendered · robots.txt respected

Last updated July 2026

Live Extraction

Endpoint · POST /v1/extract

GET

try:

Hit Extract to turn this page into clean, LLM-ready data.

robots.txt respected · public data only

Markdown · JSON · structured fields, from one API call. Crawling, rendering and extracting ...

Documentation is the highest-value text most companies own and the worst-shaped text to feed a model. A single page carries a left nav, a right table of contents, a version dropdown, a feedback widget and a footer, and the part you actually want, the prose plus the code samples, is maybe fifteen percent of the HTML. Dump the raw page into a chunker and the retriever spends its budget on menu items. A documentation scraper API fixes the shape at the source: crawl the docs tree once, return each page as markdown with the boilerplate gone and the fenced code blocks preserved exactly as written.

That matters more for docs than for almost any other content type, because a mangled code block is a wrong answer. Indentation, language hints and inline backticks all have to survive the trip, or your assistant hands a developer a snippet that will not run. ClawEngine renders each page in a real browser environment first, which is what most modern docs frameworks require, then strips the chrome and returns markdown, so Docusaurus, MkDocs, GitBook, Mintlify, Readme and hand-rolled sites all come back in the same shape. From there it is a normal ingestion job: chunk by heading, keep the source URL on every chunk, and re-crawl on a schedule so the index tracks the docs instead of drifting behind them.

CRAWL RENDER JS EXTRACT MARKDOWN JSON

Any URL in LLM-ready data out

robots.txt respected public data only

Why it works

What you get with Documentation

Whole docs tree in one call

Give it the docs root and scope rules, and ClawEngine crawls the tree and returns every page as clean markdown keyed by canonical URL, ready to chunk.

Code blocks survive intact

Fenced blocks keep their language hint and indentation, and tables convert to markdown tables, so a retrieved snippet is one a developer can actually run.

Renders modern docs frameworks

Docusaurus, MkDocs, GitBook, Mintlify and Readme sites build client-side. Each page is rendered before extraction, so nothing comes back as an empty shell.

What it handles

Any URL in, clean structured data out

Point ClawEngine at a public page and it crawls, renders the JavaScript and extracts clean markdown or typed JSON in one call. Define a schema for structured fields, and respect robots.txt and Terms of Service by default.

Crawls a docs site from one seed URL
Strips nav, sidebars, version switchers and footers
Preserves fenced code blocks and tables
Returns markdown or typed JSON per page
Renders JavaScript documentation frameworks
Re-crawls on a schedule so the index stays current

POST /v1/extract extraction result

200 · JSON

{
  "url": "https://example.com/products/atlas",
  "title": "Atlas Field Notebook",
  "markdown": "# Atlas Field Notebook\n\nDurable...",
  "data": {
    "name": "Atlas Field Notebook",
    "price": 24.00,
    "currency": "USD",
    "rating": 4.7
  },
  "links": [ "/products", "/cart" ],
  "metadata": { "rendered": true }
}

JS rendered · boilerplate stripped ✓ robots.txt respected

Why ClawEngine

One API that crawls, renders and extracts

Not a raw HTML dump, not a headless browser fleet to run, and not a brittle parser to maintain. One call crawls a public page, renders its JavaScript and returns clean markdown or typed JSON, built for RAG pipelines and AI agents.

LLM-ready output

Clean markdown or typed JSON with the boilerplate stripped, so the data drops straight into a vector store, a prompt or an agent without a cleanup step.

JavaScript rendered

Each page loads in a real browser environment before extraction, so single-page apps and client-rendered content come back complete, not as an empty shell.

Compliance-first

ClawEngine works on public, permitted data only. It respects robots.txt and site Terms of Service and honors crawl-delay, so responsible scraping is the default.

Code examples

Crawl a docs site into a RAG index

Point the crawler at the docs root, scope it to that path, and take markdown back per page. Chunk on headings, keep the source URL on every chunk, and re-run it on a schedule.

curl Crawl a documentation site (POST /v1/crawl)

curl https://api.clawengine.ai/v1/crawl \
  -H "Authorization: Bearer $CLAWENGINE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com/",
    "scope": "path",
    "max_pages": 500,
    "format": "markdown",
    "render": true,
    "strip": ["nav", "sidebar", "footer", "toc"]
  }'

python Chunk docs on headings and embed

import os, re, requests

BASE = "https://api.clawengine.ai/v1"
headers = {"Authorization": f"Bearer {os.environ['CLAWENGINE_API_KEY']}"}

job = requests.post(f"{BASE}/crawl", headers=headers, json={
    "url": "https://docs.example.com/",
    "scope": "path",
    "format": "markdown",
    "render": True,
}).json()

def split_on_headings(md):
    # Keeps each H2 section whole, so code blocks are never cut in half.
    parts = re.split(r"\n(?=## )", md)
    return [p.strip() for p in parts if p.strip()]

chunks = []
for page in job["pages"]:
    for section in split_on_headings(page["markdown"]):
        heading = section.split("\n", 1)[0].lstrip("# ").strip()
        chunks.append({
            "text": section,
            "url": page["url"],          # cite the source in every answer
            "heading": heading,
        })

print(f"{len(chunks)} chunks from {len(job['pages'])} pages")

node Re-crawl and upsert only what changed

import crypto from "node:crypto";

const res = await fetch("https://api.clawengine.ai/v1/crawl", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.CLAWENGINE_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://docs.example.com/",
    scope: "path",
    format: "markdown",
  }),
});

const { pages } = await res.json();
const hash = (s) => crypto.createHash("sha1").update(s).digest("hex");

// Only re-embed pages whose markdown actually moved since the last crawl.
const changed = pages.filter((p) => hash(p.markdown) !== previous[p.url]);
console.log(`${changed.length} of ${pages.length} pages changed`);

Documentation scraper API: the questions buyers ask

What is a documentation scraper?

A documentation scraper is a service that crawls a docs site and returns each page as clean text instead of raw HTML. It follows the documentation tree from a seed URL, drops the sidebar, version switcher, search box and footer, and keeps the prose, headings and code blocks. The output is markdown or typed JSON, which is the shape a RAG pipeline, a support assistant or an offline docs archive actually needs.

How do I scrape an entire documentation site?

Start from the docs root, restrict the crawl to that path or subdomain so you do not wander into the marketing site or the blog, and let the crawler follow in-scope links until the tree is exhausted. Return each page as markdown keyed by its canonical URL. ClawEngine takes the seed URL and the scope rules in one call, renders every page, and returns the whole set, so you get the full site rather than a page at a time.

How do I scrape documentation for a RAG chatbot?

Crawl the docs to markdown, chunk each page on its headings so a chunk is a coherent section rather than an arbitrary window, attach the source URL and the heading path as metadata, embed, and store. Keep code blocks whole inside their chunk. Then re-crawl weekly and upsert by URL so the index follows the docs. The retrieval quality you get is mostly decided at the extraction step, not the embedding step.

Does it preserve code blocks and tables?

Yes. Fenced code blocks come back as fenced code blocks with their language hint and indentation intact, and HTML tables convert to markdown tables rather than collapsing into a run-on line. That is the whole point for technical content: a broken snippet is worse than no snippet, because an assistant will confidently hand a developer something that does not compile.

Can it handle docs sites that render with JavaScript?

It can, because it renders. Most current documentation frameworks build the page client-side, so a raw-HTML fetch returns an app shell with no content. ClawEngine loads each page in a real browser environment and waits for the content to build before extracting, which is why Docusaurus, Mintlify and GitBook sites come back complete rather than empty.

How often should I re-crawl my documentation?

Weekly is a sensible default for a stable product, and daily if you ship often or the docs cover a fast-moving API. Key every record by canonical URL so a re-crawl upserts rather than duplicates, and diff the markdown to see what actually changed so you only re-embed the pages that moved. Stale documentation in a retrieval index is a support liability, not a neutral cost.

Good questions

Questions about Documentation

Technically it is a public site, but treat it carefully. Documentation is copyrighted text, and republishing it or training a public product on it is a different act from reading it. Check the robots.txt and the Terms of Service first. ClawEngine stays on public, permitted pages and reads robots.txt, and the durable use of this is your own docs, your vendor docs where permitted, and open-source project documentation.

ClawEngine does not target logins or gated content, so an internal wiki is out of scope for the crawler. The pattern that works is exporting the internal source through its own API or export tool and using ClawEngine for the public surface, your documentation site, your help center, your changelog and your vendor docs.

Markdown for the body, JSON when you want fields. Markdown keeps the heading structure that makes heading-aware chunking possible, which is the single biggest lever on retrieval quality for docs. Use a schema when you want specific things pulled out as fields, an endpoint name, a method, a parameter table, so you can filter on them at query time rather than hoping the embedding captures it.

Read every web scraping question

Explore more

More ways to turn the web into data with ClawEngine

See every use case See pricing Back to the web scraping API

Stop wrangling raw HTML. Get LLM-ready data.

Point ClawEngine at a public page and one call crawls, renders the JavaScript and extracts clean markdown or typed JSON, ready for your RAG pipeline or AI agent. Public, permitted data only.

See pricing

Crawl · render JS · extract markdown & JSON · robots.txt respected, public data only