Structured Data Extraction for RAG: From Web Pages to Typed JSON

Structured data extraction for RAG turns web pages into retrievable knowledge

Retrieval-augmented generation is only as good as the data you retrieve over. If your RAG pipeline ingests raw HTML, full of navigation, scripts and inconsistent formatting, your chunks are noisy, your embeddings are muddy, and your model retrieves the wrong passages. Structured data extraction fixes this at the source by pulling clean, typed fields straight from web pages, so your retrieval layer works with knowledge instead of markup. This post shows how to feed a RAG pipeline well, using public and permitted web data only.

Why raw HTML poisons retrieval

When you scrape a page and dump the HTML into your chunker, every chunk carries baggage. A product page might split mid-table; an article might mix the byline, the share widget and a cookie banner into the same window. Embeddings encode all of that, so a query about a price can match a footer instead of a product. The fix is to extract the meaningful content and structure before you ever chunk: clean markdown for prose, and typed JSON for anything with clear fields.

Define a schema, get typed JSON back

The most powerful pattern for RAG is schema-based extraction. Instead of guessing fields from messy text, you declare the shape you want and get typed data back. For a product listing you might ask for name, price, rating and availability. For an article you might ask for title, author, published date and body. The extractor maps the page onto your schema and returns clean JSON.

# extract typed fields with a schema
curl https://api.clawengine.ai/v1/extract \
  -H "Authorization: Bearer $KEY" \
  -d '{
    "url": "https://example.com/products/widget",
    "schema": {
      "name": "string",
      "price": "number",
      "rating": "number",
      "in_stock": "boolean"
    }
  }'

Typed fields are gold for RAG. You can store them as metadata, filter retrieval on them, and render them into deterministic, hallucination-resistant context for the model.

Chunk clean markdown, not soup

For long-form content, clean markdown is the right intermediate format. It preserves headings, lists and code blocks, which gives you natural chunk boundaries. Split by heading where you can, fall back to a fixed token window with light overlap where you cannot, and attach the source URL and section title to every chunk as metadata. Those breadcrumbs improve both retrieval and the citations you can show users.

# python: page to clean markdown, then chunk and embed
page = extract(url, fmt="markdown")
chunks = chunk_by_heading(page["markdown"], max_tokens=512)
for c in chunks:
    embed_and_store(
        text=c["text"],
        metadata={"url": page["url"], "section": c["heading"]},
    )

Use metadata to sharpen retrieval

Structured fields are not just answers, they are filters. If you extracted price, category and date, you can scope a vector search to the right subset before ranking, which cuts noise dramatically. Hybrid retrieval, combining keyword filters on typed metadata with semantic search over markdown chunks, consistently beats naive top-k over raw text.

Keep the pipeline fresh and compliant

Web data drifts. Prices change, docs get rewritten, listings disappear. Re-crawl on a schedule, compare against stored hashes, and re-embed only what changed so your index stays current without reprocessing everything. Throughout, crawl only public and permitted sources, respect robots.txt and Terms of Service, and keep provenance so you can honor a source's wishes later.

Better input, better answers

RAG quality is decided long before the model runs, at the moment you turn a web page into retrievable data. Extract typed JSON where the page has clear fields, clean markdown where it has prose, attach metadata, and chunk deliberately. ClawEngine does the crawl, render and schema extraction in one call, on public and permitted data only, so your retrieval layer starts from clean knowledge. Explore structured extraction or read about rendering JavaScript pages.