How to Get Website Data Into a RAG Knowledge Base

Getting website data into a RAG knowledge base, start to finish

To get website data into a RAG knowledge base, you run five steps: scrape the page with JavaScript rendered, strip the boilerplate to clean markdown, chunk it on heading boundaries, embed the chunks, and store them in a vector database with the source URL attached. The step everyone underestimates is the second one. If you feed raw HTML into LlamaIndex, LangChain or AnythingLLM, retrieval quality suffers no matter how good the embedding model is, because the index fills with navigation and script noise instead of content. This guide walks the whole pipeline and shows the framework-specific ingestion code.

The pipeline, and where it usually breaks

Every ingestion tutorial describes the same chain: scrape, clean, chunk, embed, store, retrieve. The failures cluster in two places. The first is scraping a JavaScript site with a plain HTTP loader and indexing an empty shell. The second is embedding pages that still carry their navigation, footer and cookie banner, which pollutes the vectors. Fix those two and most retrieval problems disappear before you ever touch chunk size or rerankers.

Step 1: scrape with rendering

Modern sites build content client-side, so you need a headless browser to render the page before you read it, not just an HTTP fetch. You can run Playwright or Puppeteer yourself, or call an API that renders and returns clean output directly. The goal is the same: the fully built page, not the shell the server first sends.

Step 2: clean to markdown (the load-bearing step)

Language models are trained on linear text, but HTML is a tree, so it has to be converted before it embeds well. Cleaning means removing nav, footer, sidebar, ads and cookie banners and keeping the real headings, lists and tables. Markdown is the practical target because its structure survives chunking. This is exactly what LLM-ready output means, and it is why starting from clean markdown removes the messiest part of every framework tutorial below.

Step 3: chunk on boundaries that mean something

Chunking splits a document into pieces small enough to retrieve and embed. Sources disagree on the numbers, so treat them as ranges rather than gospel.

Start with recursive splitting around 500 tokens with 10 to 20% overlap, measure retrieval on your own questions, and only reach for semantic chunking if the numbers justify the cost. Overlap helps a chunk keep its context, but too much bloats the store and raises false positives.

Step 4 and 5: embed and store, with provenance

Embed the chunks with a model such as an OpenAI text-embedding model or a local option, and store them in a vector database like Chroma, Pinecone, Weaviate, Qdrant or Milvus. Attach the source URL, title and a fetch timestamp to every chunk. Provenance is not optional: it is what lets your assistant cite an answer and what lets you refresh a source later.

How do I ingest scraped data into LangChain?

In LangChain the chain is load, split, embed, store, retrieve. If you already have clean markdown, you skip the messiest part and hand the splitter real text instead of HTML.

from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings

# `markdown` is clean text from your scraper, not raw HTML
splitter = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=200)
chunks = splitter.create_documents([markdown], metadatas=[{"source": url}])

store = Chroma.from_documents(chunks, OpenAIEmbeddings())
retriever = store.as_retriever(search_kwargs={"k": 4})

How do I ingest scraped data into LlamaIndex?

LlamaIndex wraps each page in a Document, then handles node parsing, embedding and storage for you. Its default node parser splits around 1,024 tokens with 200 overlap, which you can tune.

from llama_index.core import Document, VectorStoreIndex
from llama_index.core.node_parser import SentenceSplitter

docs = [Document(text=markdown, metadata={"source": url})]
parser = SentenceSplitter(chunk_size=1024, chunk_overlap=200)

index = VectorStoreIndex.from_documents(docs, transformations=[parser])
query_engine = index.as_query_engine(similarity_top_k=4)

How do I add web pages to AnythingLLM?

AnythingLLM has a built-in web scraper that returns the parsed text of a page and embeds it into a workspace for you, with no manual embedding step. That is convenient for a handful of simple pages. It gets thin on JavaScript-heavy sites and large document sets, because the lightweight built-in scraper is not a full rendering crawler. The reliable pattern for those is to scrape and clean the content upstream, then feed AnythingLLM already-clean markdown or text, so its workspace embeds real content instead of a half-rendered page. Point it at documents you have already made LLM-ready and the retrieval quality follows.

What is the best way to chunk web scraped data for RAG applications?

Start with recursive character splitting at roughly 500 tokens and 10 to 20% overlap, chunk on the markdown headings rather than arbitrary character counts, and measure recall on your real questions before optimizing further. Heading-aware splitting is what keeps a paragraph attached to the question it answers, which is why clean markdown with real headings matters more than the exact token number. Semantic chunking can add a couple of points of recall but costs compute on every document, so adopt it only when a measured gap justifies it.

How do I keep the knowledge base fresh and free of duplicates?

Re-crawl on a schedule, hash the extracted text rather than the raw HTML, and re-embed only the pages whose content hash changed. HTML changes constantly even when the words do not, so hashing the clean text is what lets you skip duplicate work and keep the index from filling with near-identical chunks. A production ingestion system is really a change-detection system with scraping attached, and it depends on watching your source feeds for drift in freshness and structure. Teams running this at scale usually put monitoring on the freshness and schema of those feeds so a silently broken source does not quietly rot the index.

Why not just feed raw HTML and let the framework handle it?

Because the framework does not clean it well, and the cost lands at query time forever. Raw HTML buries content in markup, wastes tokens on tags, and produces chunks that are partly navigation, so every embedding drifts toward the boilerplate that is identical across a whole site. Your retriever then struggles to tell pages apart. Cleaning the input once, upstream, is the cheapest improvement in the whole pipeline: it applies to every future question and never touches the model.

Doing the whole first stage in one call

The scrape, render and clean steps are the same every time, so they can be a single request instead of a subsystem you maintain. ClawEngine takes a public URL and returns clean markdown or typed JSON with the boilerplate stripped and JavaScript rendered, ready to hand straight to the splitter in any of the frameworks above. It crawls whole documentation sets from a seed URL, works on public and permitted data only, and respects robots.txt and site Terms of Service. See the output shape on our LLM-ready data page, or the end-to-end flow on the RAG data pipeline page.

The short version

Getting website data into a RAG knowledge base is scrape, clean, chunk, embed, store, with provenance attached. The two failures that matter are not rendering JavaScript and not stripping boilerplate, and both happen before chunk size ever becomes the bottleneck. Clean the input, chunk on real headings, hash the text to dedupe, and LlamaIndex, LangChain or AnythingLLM will retrieve well on top of it.

Try it on any public URL with the ClawEngine web scraping API, or read what LLM-ready input actually requires in what is LLM-ready content.