Scrapy JavaScript Rendering: Playwright, Splash or an API

The symptom is always the same. The spider runs, the logs look healthy, every request returns a clean 200, and every item comes back with empty fields. You open the page in your browser and the data is obviously there. You view-source and it is obviously not.

That gap is the whole problem. Scrapy is an HTTP client with an excellent crawling framework wrapped around it. It sends a request, it gets bytes back, and it hands those bytes to your selectors. If the site ships a near-empty <div id="root"> and builds the real page in the browser afterwards, Scrapy never sees the result, because Scrapy never runs the JavaScript that produces it.

Does Scrapy render JavaScript?

No, not on its own, and this is by design rather than an oversight. Scrapy's speed comes from being asynchronous and browserless: it can hold hundreds of concurrent HTTP requests in flight on modest hardware because none of them are running a browser engine. Adding rendering means adding a browser, and a browser costs orders of magnitude more memory and CPU per page than a socket does.

So rendering in Scrapy is always an add-on. You choose which one, and you accept the operational cost that comes with it.

How do I check whether I actually need rendering?

Do this before you install anything, because it saves a surprising number of projects from a browser fleet they never needed. Fetch the page the way Scrapy does and search the raw bytes for a value you can see on screen:

scrapy shell "https://example.com/products/widget-pro"
>>> "129.99" in response.text

If that returns True, the data is in the source and your selector is simply wrong. Fix the selector. If it returns False, look for the data in a different shape before you conclude it is missing. Modern frameworks routinely embed their state as JSON inside the HTML:

• Next.js sites ship a <script id="__NEXT_DATA__" type="application/json"> block containing the full page state.
• Nuxt sites expose window.__NUXT__.
• Many ecommerce templates include a JSON-LD Product block with price and availability already structured.
• The page's own XHR endpoint is often public, returns clean JSON, and is far more stable than any CSS selector you could write against the rendered DOM.

Any of those is a better outcome than rendering. Parsing an embedded JSON payload is faster, cheaper and considerably more durable than driving a browser, because a redesign changes the markup far more often than it changes the data contract underneath.

The three ways to render JavaScript in Scrapy

scrapy-playwright, the current default

If you are starting fresh in 2026, this is the one to reach for. It plugs into Scrapy's async event loop properly, drives modern browser engines, and lets you keep everything you already like about Scrapy: the scheduler, middleware, item pipelines, throttling and feed exports all carry on working. The project ships regularly, with 0.0.47 published in June 2026.

pip install scrapy-playwright
playwright install chromium

Then enable the download handlers and mark the requests that need a browser. Marking them individually matters: rendering every request is the most common way teams turn a fast spider into a slow, expensive one.

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

# in the spider
yield scrapy.Request(
    url,
    meta={
        "playwright": True,
        "playwright_page_methods": [
            PageMethod("wait_for_selector", "div.product-price"),
        ],
    },
)

That wait_for_selector line is the part people skip, and it is the part that decides whether the spider works. Waiting for a network-idle event is a guess. Waiting for the specific element you are about to extract is a fact.

scrapy-splash, and why not to start here

Splash was the standard answer for years and there is still working code in production that depends on it. It runs as a separate service you deploy in Docker, and any interaction beyond a plain page load has to be scripted in Lua. Against a Playwright integration that speaks Python and drives Chromium, that is a lot of moving parts to take on for a new project. If you already run Splash and it works, there is no urgency to migrate. If you are choosing today, choose Playwright.

An API that renders for you

The third route removes rendering from your infrastructure rather than optimizing it. You make an ordinary HTTP request, the rendering happens server side, and finished content comes back. Inside a Scrapy spider it looks like any other request, which means you can migrate one stubborn source at a time instead of rewriting the project:

import os, json, scrapy

class ProductSpider(scrapy.Spider):
    name = "products"

    def start_requests(self):
        for url in self.targets:
            yield scrapy.Request(
                "https://api.clawengine.ai/v1/extract",
                method="POST",
                headers={
                    "Authorization": f"Bearer {os.environ['CLAWENGINE_API_KEY']}",
                    "Content-Type": "application/json",
                },
                body=json.dumps({
                    "url": url,
                    "format": "json",
                    "render": True,
                    "schema": {"name": "string", "price": "number", "in_stock": "boolean"},
                }),
                callback=self.parse_item,
            )

    def parse_item(self, response):
        yield json.loads(response.text)["data"]

Note what disappeared along with the browser: the selectors. Describing the fields you want by name rather than by position is what stops a source-site redesign from becoming an emergency deploy on your side. Our JavaScript rendering API handles the wait logic per template, and the same call can crawl a whole site rather than a single URL.

What does rendering actually cost?

More than people budget for, on either route. A headless Chromium page is commonly in the hundreds of megabytes of RSS while it is open, so the concurrency that made Scrapy attractive collapses hard once every request holds a browser tab. A box that comfortably ran 200 concurrent HTTP requests will run a small fraction of that in rendered pages, and you will feel it as timeouts and memory pressure before you see it on an invoice.

Managed rendering moves that cost into a line item instead, and most vendors price a rendered page above a plain fetch, sometimes by five or ten times. Neither option is free. The honest comparison is between a per-page fee and the servers plus the engineer hours that keep a browser fleet healthy, and the answer genuinely differs by team. If crawling is close to your product and you have the ops capacity, running it yourself is often cheaper. If crawling is a dependency, it usually is not.

One cost that hits both routes equally: rendered spiders fail quietly. A selector that stops matching after a redesign yields empty fields rather than an exception, and Scrapy will happily report a successful crawl of nothing. Assert on the shape of your items in a pipeline and drop the run loudly when a field goes null across the board, the same instinct that leads teams to put a monitor on the endpoints they depend on rather than waiting for someone downstream to notice.

Is there a Scrapy alternative for JavaScript-heavy sites?

Yes, and the honest framing is that this is a question about operational load, not about Scrapy's quality. Scrapy is free, actively developed, and after fifteen years it is still the most widely used extraction framework in Python. Nobody should leave it because it is stale.

The reason teams leave is that the thing they wanted was data, and what they built was a distributed browser platform with a crawler attached. If that describes your last quarter, it is worth comparing the managed options side by side. We rate the category honestly, including where each tool beats us, on our Scrapy alternatives comparison, and the Python-side integration details live on the Python web scraping API page.

How do I make my Scrapy spider handle dynamic content reliably?

Four habits separate spiders that survive a year from spiders that need weekly attention.

• Render selectively. Flag rendering per request, never globally. Most sites mix server-rendered and client-rendered pages, and paying browser cost on the static half is pure waste.
• Wait for elements, not time. A sleep(3) is a bet that the network is fast today. Waiting on the selector you are about to read is deterministic.
• Prefer the data contract to the DOM. If an embedded JSON payload or a public XHR endpoint exists, use it. It survives redesigns that break every CSS selector on the page.
• Validate items, not status codes. A 200 that yields empty fields is the failure mode that actually happens. Check field completeness in a pipeline and fail the run when it drops.

If you want the longer version of the rendering decision itself, independent of Scrapy, we covered it in when you actually need to render JavaScript while scraping. For the crawl-level version of the same problem, see how to crawl a JavaScript website. And if you are weighing the framework choice more broadly, requests, Scrapy or an API works through the tradeoffs.

A note on what to scrape

Rendering a page is a technical capability, not a permission. Read robots.txt and honor it, respect crawl-delay, keep concurrency polite, and stay on public and permitted data. If a site is actively refusing you after all of that, treat it as an answer rather than an obstacle. ClawEngine works on public and permitted data only and respects robots.txt and site Terms of Service by default.