How to Crawl a JavaScript Website (Without an Empty Page)

How to crawl a JavaScript website

To crawl a JavaScript website you have to render the page before you read it. Load the URL in a real browser engine, wait for the client-side code to build the DOM, and only then extract the content. A plain HTTP fetch returns the initial HTML, which on a modern single-page app is often an empty shell with a script tag. You have three practical options: run a headless browser yourself, use a rendering API that does it for you, or find an underlying data endpoint the page already calls.

That is the whole problem in one paragraph. The rest of this guide is about which of those three you should pick, and the specific things that break once you go past a single page.

Why a normal crawler returns an empty page

A server-rendered site sends you finished HTML. Your crawler reads it, and the content is right there. A client-rendered site sends you something closer to a bootstrap file: a nearly empty <div id="root"></div>, a bundle of JavaScript, and instructions for the browser to go fetch the actual data and assemble the page locally.

If your crawler is requests.get() or curl, it never executes that JavaScript. It sees the bootstrap file and nothing else. This is why people are surprised when a page that looks full in Chrome comes back with 200 words of boilerplate and no product listings. Nothing failed. The content was never in the response.

You can check this in about ten seconds. Open the page, view source (not the inspector, which shows the built DOM), and search for a sentence you can see on screen. If it is not in the source, the page is client-rendered and you need rendering.

The three ways to crawl JavaScript sites

Look for the data endpoint first

Before you reach for a browser, open the Network tab, filter to Fetch/XHR, and reload. Often the page is calling a JSON endpoint that returns exactly the data you want, already structured. Hitting that directly is faster and cheaper than rendering, and the output needs no parsing.

Two warnings. These endpoints are undocumented, so they change whenever the site ships a release, and your pipeline breaks silently. And they are still subject to the site's Terms of Service and robots.txt, so "I found the API" is not permission. Check both before you build on it.

Run a headless browser when you need control

Playwright is the sensible default in 2026. It handles Chromium, Firefox and WebKit, has a good waiting model, and is what most open-source crawlers use underneath. If your crawl is a few thousand pages and you have somewhere to run it, this works.

The reason teams move off it is not that the code is hard. It is that browsers are heavy. Each instance wants roughly a gigabyte of RAM, they leak, they hang on pages with long-polling connections, and concurrency becomes a scheduling problem rather than a loop. At about the point you start writing a worker pool with health checks and restarts, you have accidentally become the maintainer of a browser farm.

Use a rendering API when the crawler is not your product

A JavaScript rendering API moves that fleet to someone else. You send a URL, the service renders it in a real browser and returns the built content. ClawEngine does this in the same call that crawls the site and extracts typed fields, so a client-rendered catalog comes back as clean markdown or structured JSON rather than a DOM you still have to parse.

The tradeoff is honest: you pay per request, and rendering costs more than a plain fetch on every service that offers both. If you are crawling a handful of pages a month, running Playwright yourself is cheaper. If you are crawling continuously across sites you do not control, the managed path usually wins once you price the engineering time.

How do I know if a website uses JavaScript rendering?

View the page source rather than the inspector, and search for text you can see on screen. If the text is missing from the source but present in the inspector, the content was built by JavaScript and a plain fetch will not see it. You can also disable JavaScript in your browser and reload: whatever disappears is what your crawler will be missing.

Can Google crawl JavaScript websites?

Yes, Googlebot renders JavaScript, but it does so in a second pass after the initial crawl, and that pass can lag. This is why client-rendered sites often index more slowly and less completely than server-rendered ones, and why heavy client-side rendering is a recurring reason pages underperform in organic search despite looking fine to a human visitor. If you control the site, server-side rendering or static generation removes the problem entirely.

What are the hard parts once you go past one page?

Rendering a single URL is a solved problem. Crawling a site is where the work actually lives.

Knowing when the page is done. There is no universal "ready" signal. Waiting for the load event fires too early on most SPAs. Waiting a fixed three seconds is both too slow for fast pages and too fast for slow ones. The reliable pattern is waiting for a specific selector that only exists once the real content has rendered, which means knowing something about each site.

Infinite scroll and lazy loading. Content that appears on scroll is not in the DOM until you scroll. You end up scripting scroll loops and watching for the point where nothing new loads.

Client-side routing. On many SPAs, clicking a link changes the URL without a page load. A crawler that waits for navigation events sees nothing happen.

Cost per page. Rendering is roughly an order of magnitude more expensive than fetching, in both compute and money. Crawling 100,000 pages with rendering is a real budget line, not a rounding error.

Politeness. Rendering multiplies the load you put on a site: every page pulls its scripts, images and API calls, not just the HTML. Respecting robots.txt and crawl-delay matters more here, not less.

Do I need to render every page?

No, and assuming you do is the most common way to overspend. Most sites are mixed: the blog and documentation are server-rendered while the app and search results are client-rendered. Fetch first, check whether the content you need is present, and only fall back to rendering when it is missing. On a typical mixed crawl that decision alone can cut the rendered share to a fraction of total pages, and rendering is where the cost is.

What output should a JavaScript crawler give you?

Rendered HTML is progress, not the finish line. If the data is heading for a language model, raw DOM is close to the worst possible input: it is mostly navigation, scripts and styling, so you pay tokens to embed markup and you dilute retrieval quality with boilerplate.

What you actually want out the other side is the content, cleaned: markdown for prose, or typed JSON when you know the fields you need. Getting there means rendering, then stripping boilerplate, then either converting to markdown or extracting against a schema. Each of those is a stage you can build, and each is a stage that can be part of the same call. ClawEngine collapses the four into one request and returns LLM-ready data directly, on public and permitted pages only, respecting robots.txt and site Terms of Service.

A practical decision rule

Check the network tab for a JSON endpoint first, because free and structured beats rendered and parsed. If there is none, and you are crawling a small number of pages on infrastructure you already run, use Playwright. If you are crawling across sites you do not control, at volume, continuously, or the output is feeding a RAG pipeline, use a managed API and spend your engineering time on the pipeline instead of the browser fleet.

The mistake worth avoiding is the middle path taken by accident: starting with Playwright for one site, then bolting on proxies, a worker pool, retry logic and a memory watchdog over six months until you are maintaining infrastructure that was never the point. Decide which side of the line you are on deliberately.

You can try the rendering path on any public URL with the ClawEngine web scraping API, or see how it compares against the alternatives in our roundup of the best web scraping APIs.