How to Scrape Documentation for a RAG Chatbot

How to scrape documentation for a RAG chatbot, in short

Crawl the docs site from its root, scoped to that path, and return every page as markdown with the nav, sidebar, version switcher and footer stripped and the code blocks intact. Chunk each page on its headings so a chunk is a whole section, attach the source URL and heading path as metadata, embed, and store. Re-crawl on a schedule and upsert by URL. Retrieval quality for documentation is decided at the extraction step far more than at the embedding step.

Almost every docs chatbot that answers badly fails for the same reason, and it is not the model. It is that the index is full of navigation. A typical documentation page is maybe fifteen percent content and eighty-five percent chrome: a left nav listing every other page, a right table of contents, a version dropdown, a search box, a feedback widget, a cookie banner and a footer. Feed that page to a naive chunker and a good share of your chunks are lists of menu items that match everything and mean nothing. The retriever then hands the model junk, and the model dutifully makes something up.

Fix the input and most of the problem disappears. Here is the pipeline that works.

Step 1: crawl the docs tree, not the whole site

Start from the documentation root and scope the crawl to that path or subdomain. This matters more than it sounds. Docs sites almost always link out to the marketing site, the blog, the changelog, the status page and a pricing page, and an unscoped crawler happily indexes all of it. Then a developer asks how to rotate an API key and the retriever returns a landing page headline.

Restrict the crawl, cap the page count, and let it follow in-scope links until the tree is exhausted. Key every page by its canonical URL, because that is the identity you will use for deduplication, for citations in the answer, and for incremental re-crawls later.

curl https://api.clawengine.ai/v1/crawl \
  -H "Authorization: Bearer $CLAWENGINE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com/",
    "scope": "path",
    "max_pages": 500,
    "format": "markdown",
    "render": true,
    "strip": ["nav", "sidebar", "footer", "toc"]
  }'

The render flag is not optional for modern docs. Docusaurus, Mintlify, GitBook and most current frameworks build the page client-side, so a raw HTML fetch returns an app shell with no prose in it. This is the single most common reason a docs crawl comes back with hundreds of near-empty pages. If your first crawl looks suspiciously thin, rendering is the thing to check.

Step 2: get markdown out, with the code blocks intact

Markdown is the right intermediate format for documentation, for one specific reason: it keeps the heading structure. Headings are what let you chunk on semantic boundaries in the next step, and once you flatten to plain text they are gone.

The thing to verify before you trust any extractor on technical content is what happens to code. A fenced block must come back fenced, with its language hint and its indentation exactly as written. Tables must convert to markdown tables rather than collapsing into a run-on line of cells. This is not a cosmetic concern. A mangled snippet is worse than a missing one, because your assistant will hand a developer something that looks right and does not compile, and they will not know until they run it. Our documentation scraper API is built around this: crawl in, clean markdown out, code blocks preserved.

Step 3: chunk on headings, not on character counts

Fixed-size chunking with a sliding window is the default in most tutorials and it is a poor fit for documentation. A 512-token window cuts through the middle of a code sample and splits a parameter table from the endpoint it describes. Heading-aware chunking respects the structure the technical writer already built.

import re

def split_on_headings(md):
    # Each H2 section stays whole, so code blocks are never cut in half.
    parts = re.split(r"\n(?=## )", md)
    return [p.strip() for p in parts if p.strip()]

chunks = []
for page in crawl["pages"]:
    for section in split_on_headings(page["markdown"]):
        heading = section.split("\n", 1)[0].lstrip("# ").strip()
        chunks.append({
            "text": section,
            "url": page["url"],        # so every answer can cite its source
            "heading": heading,
            "title": page["title"],
        })

Two refinements are worth the effort. If a section runs very long, split it on H3 boundaries before falling back to a size limit, and never split inside a fenced block. And prepend the page title and heading path to each chunk before embedding, so a chunk that says "Pass the expand parameter" carries the context that it belongs to the Webhooks endpoint. Retrieval improves noticeably from that one line of string concatenation.

Step 4: keep metadata that the answer can cite

Store the source URL, the page title, the heading, the crawl timestamp and the docs version on every chunk. The URL is what turns a plausible answer into a checkable one, and a docs bot that links its sources gets trusted in a way that a bot producing bare paragraphs never does. The version field matters if you publish more than one, because "how do I authenticate" has a different answer in v1 and v3 and an index that mixes them will be confidently wrong half the time.

Step 5: re-crawl and upsert instead of rebuilding

Documentation moves. An index built once and never refreshed becomes a support liability within a quarter, because it answers questions about parameters that were renamed and endpoints that were deprecated. Run the crawl weekly for a stable product and daily if you ship fast.

Do not rebuild the whole index each time. Hash the markdown per page, compare against the previous crawl, and only re-chunk and re-embed the pages whose hash changed. On a 500 page docs site a typical week touches a couple of dozen pages, so incremental refresh costs a fraction of a full rebuild and finishes fast enough to run unattended. Delete chunks for URLs that vanished, or your bot will keep citing a page that returns a 404.

What about internal docs behind a login?

A public crawler is the wrong tool there, and any vendor who offers to log in for you is selling you a problem. Export internal sources through their own APIs, Confluence, Notion, a private repo, and use a crawler for the public surface: your documentation site, your help center, your changelog and your vendor docs. In practice most support questions are answered by public documentation anyway, and the internal material is a smaller, slower-moving set that a one-off export handles well. If you want the assistant to also act on what it reads, this is the point where teams wire it into a coding agent that plans and writes the change rather than only quoting the docs back.

How is this different from just using a sitemap?

A sitemap gives you URLs, which is the easy half. It does not render the pages, does not strip the chrome, does not preserve code blocks and does not tell you what changed. Reading sitemap.xml is a reasonable way to seed a crawl and a poor substitute for one. Many docs sites also ship an incomplete sitemap, or none at all, so a crawler that follows in-scope links catches pages a sitemap-only approach misses entirely.

Do I need a vector database for this?

For a small docs site, honestly no. A few thousand chunks with an in-process index or plain Postgres with pgvector works fine and saves you a service to operate. Reach for a dedicated vector store when you have multiple corpora, need metadata filtering at scale, or want hybrid keyword and vector search. Whichever you pick, the extraction quality upstream determines your ceiling: a great vector database over navigation-polluted chunks still returns navigation.

Where to start

Crawl one docs site to markdown and read twenty random chunks before you embed anything. If those chunks read like sections a human wrote, the rest of the pipeline is standard work. If they read like menus, fix extraction first. The broader pattern, crawl to clean text then chunk with source metadata, is the same one behind any web scraping for RAG workload, and the same RAG data pipeline handles help centers, changelogs and knowledge bases once documentation is working.