How to Render JavaScript Pages When Scraping (2026 Guide)

How do I render JavaScript pages when scraping?

If you have ever scraped a modern website and gotten back a nearly empty page, you have met the JavaScript rendering problem. The server sends a thin HTML shell, and the real content, products, prices, articles, comments, gets built in the browser after the page loads. A plain HTTP fetch never runs that JavaScript, so it never sees the content. This guide explains why that happens and how to capture fully rendered pages cleanly, on public and permitted data only.

Why the raw HTML is empty

Single-page applications built with React, Vue, Angular and similar frameworks ship a minimal document and a bundle of JavaScript. When a browser loads the page, that script fetches data from an API, builds the DOM, and paints the content you actually see. A library like requests or a simple curl does none of this, it just downloads the initial response. So you get the skeleton: a root div, some script tags, and none of the data.

# raw fetch of a JS-rendered page: almost nothing useful
curl https://app.example.com/products
# <div id="root"></div> ... content loads later via JS

How headless browsers fill in the content

To scrape a JavaScript page you need to render it the way a real browser would. A headless browser, Chromium driven by Playwright or Puppeteer, loads the page, executes the scripts, waits for the data to arrive, and then exposes the finished DOM. From there you can read the rendered HTML and extract content that simply did not exist in the raw response. The catch is that running headless browsers at scale is real work: memory pressure, crashes, concurrency limits and timeouts all become your problem.

The hard parts of doing it yourself

• Knowing when to read. Read too early and the content is still loading; too late and you waste time. You need smart waits on network idle or specific elements.
• Lazy loading and infinite scroll. Some content only appears after scrolling. Capturing it means scripting realistic interaction.
• Resource cost. Each rendered page spins up a browser context. Hundreds of concurrent renders demand serious infrastructure.
• Stability. Browsers leak memory and crash. A production fleet needs supervision and restarts.

Render with one API call instead

A managed JavaScript rendering scraper runs the headless browser for you and returns fully rendered content as clean markdown or JSON. You ask it to render, it waits for the page to settle, strips the boilerplate, and hands back the data, no fleet to operate. This keeps your pipeline simple: one request in, clean content out.

# render the page, wait for content, return clean markdown
curl https://api.clawengine.ai/v1/extract \
  -H "Authorization: Bearer $KEY" \
  -d '{"url":"https://app.example.com/products","render":true,"format":"markdown"}'

Render only when you need to

Rendering is more expensive than a plain fetch, so use it deliberately. Many pages are server-rendered and need no browser at all; a quick check of the raw HTML tells you whether the content is already there. Reserve full rendering for genuine single-page apps and content that depends on client-side data. A good API lets you toggle rendering per request so you pay for it only where it earns its keep. How much that toggle is worth depends on your vendor: a rendered page costs 10 credits against 1 on ScraperAPI and 5 against 1 on ScrapingBee, while others charge no render premium at all. We put every vendor's numbers side by side in what JavaScript rendering actually costs across the major APIs. For whole sites rather than single URLs, the same decision shows up at crawl level, which we cover in how to crawl a JavaScript website.

What is a JavaScript rendering API?

A JavaScript rendering API is a hosted service that loads a URL in a real browser engine, executes the page scripts, waits for the content to settle, and returns the finished page over HTTP. You send a URL and get back rendered HTML, markdown or structured JSON. It replaces the headless browser fleet you would otherwise run, patch and monitor yourself.

The practical difference from a plain scraping endpoint is the wait logic. Anyone can call page.goto(). Knowing that a product grid finishes populating 900ms after the XHR resolves, and that a different template needs a selector wait instead of a network-idle wait, is the part that takes months to get right. That is what you are actually buying. ClawEngine exposes it as a single JavaScript rendering scraper API call with a render flag.

How do I make JavaScript pages crawlable across a whole site?

Rendering one URL is a solved problem. Rendering a whole site is a queue problem: you have to discover links inside content that does not exist until after the render, then schedule those URLs without stampeding the origin. A crawler that reads only the raw HTML will find almost no links on a single-page app and will quietly stop after the entry point.

Two things fix this. First, render before link discovery, so the crawler extracts hrefs from the settled DOM rather than the shell. Second, seed from sitemap.xml where the site publishes one, because a sitemap lists URLs that client-side routing would otherwise hide behind interactions. A managed web crawler API does both and keeps one polite queue per host, so concurrency stays under the site's crawl-delay instead of fanning out.

Does rendering JavaScript slow down a scrape?

Yes, substantially. A plain fetch returns in tens of milliseconds; a rendered page typically takes one to five seconds because a browser has to boot, parse, execute and wait for network calls. That is a 50x to 100x difference in latency and a real difference in cost per page.

Which is exactly why rendering should be a per-request flag rather than a global setting. Check the raw HTML for the content you need first. If it is already there, the site is server-rendered and a browser adds cost for nothing. On a mixed site, running the cheap fetch first and falling back to a render only when the extraction comes back empty typically cuts rendered page count by more than half.

Stay polite and in bounds while rendering

Rendering does not change the rules. Crawl public and permitted pages only, respect robots.txt and Terms of Service, and honor crawl-delay so you do not overload a host. Rendering is a technique for reading content that a browser would legitimately display to any visitor, not a means to reach anything gated behind authentication or access controls. Keep it lawful and considerate.

Get complete pages without the browser fleet

JavaScript rendering is the single biggest reason naive scrapers return empty results, and running headless browsers yourself is a project in its own right. ClawEngine renders pages on demand and returns clean, complete markdown or JSON, on public and permitted data only, so you get the whole page without operating any infrastructure. Read how it works or learn about structured extraction for RAG.