Deduplicate Pages During a Crawl for RAG Ingestion

What is the best method to deduplicate pages during a crawl for RAG ingestion?

There is no single method. The best approach is layered, cheapest filter first: normalize URLs and honor rel=canonical before you fetch, hash the extracted text (not the raw HTML) to catch exact repeats, then run SimHash or MinHash over shingles to catch near duplicates. Dedupe again at the chunk level during ingestion.

Each layer costs roughly an order of magnitude more than the one above it, and each catches a different failure. If you only build one, build URL normalization: it is the only layer that stops you paying to fetch the duplicate at all. Everything below it is cleanup after the money is spent.

Layer 1: URL normalization, before you fetch anything

The same page is reachable at dozens of URLs. Tracking parameters, session IDs, sort orders and case differences all produce distinct strings that resolve to identical bytes. A crawler that treats URLs as opaque strings will happily fetch /products/widget, /products/widget/, /products/widget?utm_source=twitter and /Products/Widget as four pages.

Normalization is a set of rules applied before a URL enters the frontier queue:

• Lowercase the scheme and host. Hosts are case insensitive per RFC 3986. Paths are not, so leave the path alone unless you know the server is case insensitive (IIS often is, nginx is not).
• Drop the default port (:80 for http, :443 for https).
• Strip the fragment. The server never sees it.
• Remove tracking parameters: anything matching utm_*, plus gclid, fbclid, msclkid, mc_cid and friends.
• Sort the remaining query parameters so ?a=1&b=2 and ?b=2&a=1 collapse to one key.
• Resolve relative links against the actual base (respect <base href> if present).
• Follow rel=canonical when the page declares one, and treat the canonical as the identity for storage.

# python: URL normalization for a crawl frontier
from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode

TRACKING_PREFIXES = ("utm_", "pk_", "hsa_")
TRACKING_EXACT = {"gclid", "fbclid", "msclkid", "mc_cid", "mc_eid", "_ga", "igshid"}
DEFAULT_PORTS = {"http": 80, "https": 443}

def normalize_url(url: str, strip_trailing_slash: bool = True) -> str:
    p = urlsplit(url)
    scheme = p.scheme.lower()
    host = (p.hostname or "").lower()

    # host is case insensitive, the path is not: leave the path case alone
    netloc = host
    if p.port and p.port != DEFAULT_PORTS.get(scheme):
        netloc = f"{host}:{p.port}"

    path = p.path or "/"
    if strip_trailing_slash and len(path) > 1 and path.endswith("/"):
        path = path.rstrip("/")

    keep = [
        (k, v)
        for k, v in parse_qsl(p.query, keep_blank_values=True)
        if not k.lower().startswith(TRACKING_PREFIXES) and k.lower() not in TRACKING_EXACT
    ]
    query = urlencode(sorted(keep))

    # fragment is dropped: the server never sees it
    return urlunsplit((scheme, netloc, path, query, ""))

One honest caveat on strip_trailing_slash. Per spec, /about and /about/ are different resources, and a few sites really do serve different content at each. They are the same page almost always, so collapsing them is worth it. Make it a per-host flag, not a truth you bake in.

Session IDs and facets are what actually blow up a crawl

Tracking parameters waste fetches. Session IDs and faceted navigation waste your entire budget. A site that puts jsessionid in the URL mints a fresh URL on every visit, so the frontier never converges and the crawler runs forever over one catalog.

Facets are worse, because they are combinatorial. A category with 5 facets of 6 values each, times sort order, exposes tens of thousands of URLs that render permutations of the same 200 products. The fix is a parameter allowlist per host: decide which query keys are content bearing (page, id, q) and drop the rest, rather than blocklisting an infinite tail.

Layer 2: exact content hashing (hash the text, not the HTML)

Once a page is fetched, hashing catches what URL rules missed: mirrors, aliases, printer-friendly versions, and paths that respond identically.

Hash the extracted text, never the raw HTML. Raw HTML changes on every fetch even when the content is identical to a reader. A CSRF token, a rendered timestamp, an ad slot ID, a cache-buster in a script tag, a rotating "related posts" module: any one of them moves the bytes and your hash matches nothing, ever. Teams discover this when their dedup rate on a re-crawl comes back at zero.

SHA-256 over normalized text is cheap enough that the debate is not worth having: you will hash far faster than you can fetch. Normalize first, though. NFC-normalize Unicode, collapse whitespace runs, strip non-breaking spaces, lowercase.

# python: exact-duplicate detection on extracted text
import hashlib, re, unicodedata

def content_hash(text: str) -> str:
    t = unicodedata.normalize("NFC", text)
    t = t.replace("\u00a0", " ")           # nbsp -> space
    t = re.sub(r"\s+", " ", t).strip().lower()
    return hashlib.sha256(t.encode("utf-8")).hexdigest()

seen = {}

def ingest(url: str, extracted_text: str) -> bool:
    h = content_hash(extracted_text)
    if h in seen:
        # keep the canonical URL, record the alias, skip re-embedding
        seen[h].append(url)
        return False
    seen[h] = [url]
    return True

Note the alias list. Keep it rather than throwing it away: when a user asks where an answer came from you want the canonical URL, but you also want the other three URLs that served it, which is the same reason teams care about knowing exactly which source a record came from once it has passed through three transformations. Dedup that silently deletes provenance becomes a debugging problem six months later.

What is the difference between SimHash and MinHash?

SimHash produces one compact fingerprint per document (typically 64 bits) and measures similarity as Hamming distance between fingerprints. MinHash produces a signature of many hash values and estimates Jaccard similarity between shingle sets. SimHash is cheaper per document; MinHash with LSH is better at finding all similar pairs at scale.

SimHash (Charikar, 2002) hashes each feature (usually a word shingle) to 64 bits, then sums those bits as +1 or -1 into a 64-slot vector, weighted by feature frequency. The sign of each slot becomes the corresponding fingerprint bit, so similar documents produce fingerprints differing in few bits. Google's crawl team (Manku et al., WWW 2007) reported that a Hamming distance of 3 or less over 64 bits worked as a near-duplicate threshold on web pages. A reasonable starting point, not a law: tune it on your own corpus.

# python: 64-bit SimHash over word shingles
import hashlib
from collections import Counter

def _hash64(token: str) -> int:
    return int.from_bytes(hashlib.blake2b(token.encode(), digest_size=8).digest(), "big")

def simhash(text: str, ngram: int = 3) -> int:
    words = text.lower().split()
    shingles = [" ".join(words[i:i + ngram]) for i in range(len(words) - ngram + 1)] or words
    v = [0] * 64
    for shingle, weight in Counter(shingles).items():
        h = _hash64(shingle)
        for bit in range(64):
            v[bit] += weight if (h >> bit) & 1 else -weight
    return sum(1 << bit for bit in range(64) if v[bit] > 0)

def hamming(a: int, b: int) -> int:
    return (a ^ b).bit_count()          # python 3.8: bin(a ^ b).count("1")

# near duplicate if distance <= 3 over 64 bits
is_near_dupe = hamming(simhash(doc_a), simhash(doc_b)) <= 3

MinHash (Broder, 1997) takes a different route. Shingle the document into a set, apply k independent hash functions, keep the minimum value under each. The probability that two documents share the same minimum for a given hash function equals the Jaccard similarity of their shingle sets, so comparing signatures position by position gives an unbiased Jaccard estimate. With k=128 the standard error is roughly 1/sqrt(k), around 9%. More permutations, better estimate, more memory.

MinHash alone is still quadratic if you compare every pair. LSH is what makes it scale: split the signature into b bands of r rows, hash each band, and only compare documents that collide in at least one band. The approximate threshold is (1/b) to the power of 1/r, so you tune b and r to put the S-curve where you want it. That candidate generation is the real reason to pick MinHash: SimHash gives you a distance function, MinHash plus LSH gives you an index.

Practical rule: comparing each newly crawled page against a known set, SimHash is simpler and lighter. Deduplicating millions of documents against each other, MinHash plus LSH is the one that finishes.

Should you deduplicate before or after chunking?

Both, at different layers. Do URL and exact-hash dedup before chunking, because there is no reason to chunk a page you already have. Do near-duplicate detection at the chunk level after chunking, because two pages can be 70% different overall while sharing chunks that are 100% identical, and the chunk is what actually gets retrieved.

Page-level detection has a blind spot. Take a docs site where every page shares the same 400-word installation preamble. Page-level SimHash correctly calls those pages different, because most of their content is. But the chunker cuts that preamble into three chunks per page, so a 500-page site writes 1,500 near-identical chunks. Retrieval for "how do I install this" now returns an effectively random one of them.

Strip boilerplate at extraction and most of this goes away

The chunk-level duplicate problem is mostly a boilerplate problem in disguise. Navigation, footers, cookie banners, sidebars and legal text are identical across every page of a site by construction. If they survive extraction they reach your chunker and manufacture duplicates that downstream dedup cannot cleanly remove, because the boilerplate is interleaved with real content inside the same chunk.

Removing boilerplate at extraction is strictly better than detecting the duplicates it creates later, and it is the same fix that improves everything else about retrieval, which is why we wrote a whole piece on what makes content LLM-ready. Get the extractor right and chunk-level dedup goes from load-bearing to a safety net.

Does duplicate content hurt RAG accuracy?

Yes, in three distinct ways. It wastes context: top-k retrieval returns five copies of the same passage instead of five different ones, so the model sees one fact where it could have seen five. It skews embeddings toward shared boilerplate. And it corrupts any relevance signal that counts occurrences.

The context waste is immediate. If k=5 and three hits are the same paragraph from three URLs, you threw away 60% of the context budget you paid for. Users experience this as an assistant that is confidently incomplete, and the usual reaction is to raise k, which grows the bill without fixing anything.

The embedding skew is subtler. An embedding represents the whole chunk, so a chunk that is 40% shared navigation produces a vector substantially about navigation. Every page shares that navigation, so every vector drifts toward the same region of the space. Cosine similarity between any two chunks rises, the spread between the best and worst hit narrows, and the retriever loses its ability to discriminate. The symptom is "it retrieves the wrong page." The cause is that mathematically the pages were not that different.

How do you handle duplicate content in RAG?

Filter at the earliest layer that can catch it, and keep the record rather than deleting it. URL normalization stops the fetch. Content hashing stops the embed. Near-duplicate detection stops the index write. At query time, deduplicate the retrieved set before it reaches the model, since two chunks with different IDs can still say the same thing.

The query-time pass matters more than people expect and costs almost nothing: you are comparing at most a handful of hits. A cheap Jaccard or SimHash check across the top 20, keeping the highest-scoring member of each cluster, is enough to turn "five copies of one paragraph" into five distinct passages.

Method comparison

Layer 5: embedding similarity, and why it is usually overkill

You can catch what the earlier layers miss by comparing embeddings and dropping anything above a cosine threshold (0.95 and up is the usual range for "these are the same thing"). It is the only layer that catches a genuine paraphrase, where two documents say the same thing in different words and share almost no shingles.

The honest assessment: if the first three layers are done properly, this one finds very little and costs the most. You have to embed the document before you can decide not to store it, so the expensive operation happens regardless. The threshold is corpus dependent and fragile, and modern embedding models put plenty of unrelated text above 0.85, so you are tuning in a narrow band.

Use it in one place: at query time over your retrieved top-k, where you already have the vectors and the set is tiny. Running it as an ingestion gate over a whole corpus usually means boilerplate stripping was skipped upstream.

What most teams should actually do

Concretely, in order:

1. Normalize every URL before it enters the frontier, with a per-host parameter allowlist. Honor rel=canonical. This is 80% of the win for 5% of the work.
2. Strip boilerplate at extraction. Not a dedup technique, but it removes more duplicate chunks than any dedup technique will.
3. SHA-256 the normalized extracted text. Store the hash alongside the record. It is your dedup key and your re-crawl change detector in one field.
4. SimHash at the chunk level, Hamming distance 3 over 64 bits, tuned on a sample of your own data. Move to MinHash plus LSH only when you are batch-deduping a corpus against itself and the pairwise comparison stops finishing.
5. Dedupe the retrieved set at query time. Cheap, and it directly fixes the "five copies of one paragraph" failure users notice.

Skip embedding-level ingestion dedup until you have evidence you need it. Most teams reach for it because their chunks are full of navigation, and it does not fix that. Better extraction does.

All of it assumes the obvious: crawl public and permitted data only, respect robots.txt and each site's Terms of Service, honor crawl-delay. Dedup is a politeness feature too. A crawler that fetches one catalog page 40,000 times through facet permutations is hammering someone's origin for nothing.

The short version

Deduplication during a crawl is a funnel, not a step. URL normalization kills the duplicate before you pay for it, exact hashing kills it before you embed it, SimHash or MinHash kills the near duplicate before it reaches the index, and a small query-time pass catches the rest. Boilerplate stripping at extraction quietly does more work than any of them.

If you would rather not maintain the extractor and the frontier rules yourself, that is what our web crawler API does: one call crawls, renders the JavaScript, strips boilerplate and returns markdown or typed JSON with the canonical URL attached, which is the input the layers above assume you already have. See how the pieces fit in a full RAG data pipeline, or read on about scraping the web for LLMs and turning website data into a RAG knowledge base.