How to Limit a Crawl to the Pages You Need

The first crawl anybody runs is a single URL, and it works. The second one is a whole site, and that is where the surprises start: 40,000 pages discovered on a site that has maybe 900 real ones, a bill larger than expected, and a corpus stuffed with ?sort=price_desc variants of the same twelve products.

None of that is a crawler defect. It is what happens when you point a link-following process at a site that generates URLs. Scope is the thing you configure to stop it, and it is worth understanding properly before you run anything at volume.

How do I limit a crawl to certain pages?

You constrain it from four directions at once. Each control answers a different question, and leaving any of them unset is what lets a crawl drift. Here is what each one does and what goes wrong when it is missing.

The page budget is a safety net, not a scope rule. That distinction matters more than it sounds. A budget stops a bad crawl late, after you have already paid for the pages; path rules and depth stop it from ever queuing them. If your only control is limit, you are not scoping a crawl, you are capping your losses on an unscoped one.

How do I crawl only one section of a site?

Seed the crawl inside the section and fence it with a path rule. Seeding at the homepage and hoping the rule catches everything is the common mistake, because the crawler still has to fetch and evaluate the homepage's links before discarding them.

A documentation crawl looks like this against the crawl endpoint:

curl https://api.clawengine.ai/v1/crawl \
  -H "Authorization: Bearer $CLAWENGINE_API_KEY" \
  -d '{
    "url": "https://example.com/docs",
    "include_paths": ["/docs/*"],
    "max_depth": 3,
    "limit": 500,
    "format": "markdown"
  }'

Four lines, four decisions. Start inside /docs. Refuse anything outside it. Follow at most three hops, which on a normal documentation tree reaches every page without walking into changelog archives. Stop at 500 either way.

Set the path rule to the narrowest prefix that still contains everything you need. If the docs live at /docs but the API reference lives at /reference, that is two rules, not one broader rule that also swallows the marketing site.

What is crawl depth and how deep should I go?

Depth counts link hops from the seed, not directory nesting in the URL. A page at /docs/a/b/c/d is depth 1 if the docs index links to it directly. This trips people up constantly, because they reason about the URL structure and set a depth that is far too high for the shape of the site.

Most sites are flatter than they look. Documentation and product catalogs are built so a human can reach anything in a few clicks, and a crawler follows the same links. Depth 2 or 3 covers the large majority of well-structured sites.

When you are unsure, run the crawl at a low depth with a small budget first and look at what came back. A 50 page sample tells you the real shape of the site in about a minute, and it costs almost nothing compared to discovering the shape at 40,000 pages.

Why does my crawl return pages I did not ask for?

Almost always one of four causes, in rough order of how often they bite:

• Faceted URLs. Filters and sort orders generate a fresh URL per combination. Twelve products with four filters is potentially hundreds of URLs holding the same twelve products.
• Pagination loops. Some archives happily serve ?page=900 with an empty result and a link to ?page=901. There is no natural end.
• Cross-section links. A footer links to careers, legal and the blog from every page, so with no path rule the entire site is two hops from anywhere.
• Trailing slash and case variants. The same page reachable four ways, counted four times.

Path rules and depth handle the first three. The fourth is a deduplication problem rather than a scope problem, and it needs content-level comparison rather than URL matching, which is a separate step worth doing properly before RAG ingestion.

One more cause worth naming: on a client-rendered site, the links themselves are built by JavaScript. If your crawler reads raw HTML it finds no links at all and stops at the seed, which looks like an over-restrictive scope but is really a rendering problem. That is a different fix, covered in crawling a JavaScript website.

How many pages should I set as the limit?

Set it to roughly double what you expect, not ten times. The budget exists to catch a scope mistake, and a budget ten times your estimate will not catch anything until it has already spent ten times your estimate.

If you genuinely do not know the size of a site, check its sitemap first, or run the low-depth sample described above. Both are cheaper than guessing high.

For recurring crawls, size the budget against what changed rather than the whole site. Re-crawling 40,000 pages nightly to catch the 60 that moved is the single most common source of a surprising bill on a web crawler API, and it is avoidable with a narrower path rule and a schedule matched to how often the section actually changes.

Does limiting a crawl actually save money?

Yes, and more than people expect, because the savings compound across three things at once: you fetch fewer pages, you store and embed fewer documents, and you spend less engineering time cleaning junk out of the corpus later.

The third one is the one that does not show up on an invoice. A dataset with pagination stubs and filter duplicates in it produces worse retrieval, and the debugging happens weeks later when someone notices the assistant citing an empty archive page. Scope is cheaper than cleanup.

How do I know my crawl is still covering the site?

Scope rules quietly rot. A site moves its docs from /docs to /help, and a crawl that returned 900 pages last month returns 40 this month without erroring once. Nothing fails loudly, the corpus just stops being current.

The fix is to treat crawl output as a monitored dataset rather than a script's side effect. Record the page count, the depth distribution and the field fill rates for every run, and alert on a change rather than on an exception. Teams that already watch freshness and volume on their warehouse tables can point the same checks at a crawl feed and catch the silent halving on the day it happens.

Two numbers are enough to start: pages returned per run, and the share of pages where your required fields came back populated. A meaningful drop in either almost always means the site changed shape and your path rules need revisiting.

A practical order to set this up

Run a 50 page sample at depth 1 from the seed you plan to use, and read the URLs that came back. Add path rules for anything you do not want. Raise depth one step at a time until the page count stops growing, which is the signal you have reached the edge of the section. Then set the budget to about double the settled count and schedule it.

That takes maybe fifteen minutes and it is the difference between a crawl you can put in a budget and one you check nervously. If the output feeds a language model, the same scoping work is what keeps the crawl into an LLM-ready corpus clean enough to retrieve from, because a tightly scoped crawl needs far less filtering downstream than a broad one.