Migrate a Scrapy Spider to a Crawl API

Nobody migrates a Scrapy project because Scrapy is bad. Scrapy is fifteen years old, actively maintained, and still the fastest way to crawl a server-rendered site if you want to own the loop. Teams migrate because the crawl stopped being the interesting part of their product and started being a rota: someone is on the hook when a selector breaks, someone renews the proxy contract, someone upgrades the Playwright images.

This is the mechanical version of that move. What maps to what, what has no equivalent, what it costs, and which spiders you should leave exactly where they are.

Can I move a Scrapy spider to a scraping API?

Yes, for most spiders, and the reason is that a typical spider is doing four separable jobs at once. It finds URLs, it fetches them politely, it pulls fields out of the markup, and it pushes those fields somewhere. A crawl and extract API absorbs the first three and leaves you the fourth, which is usually the part you actually wanted to write.

The spiders that do not port cleanly are the ones doing something a request-response API cannot express: logging in, filling forms, clicking through a multi-step flow, or reacting to page state between requests. Those stay in Scrapy, or move to browser automation. Be honest with yourself about how many of your spiders are really in that category. In most codebases it is one or two out of a dozen.

What actually has to change when you migrate a Scrapy spider?

Less than the size of the project suggests. A Scrapy project is mostly scaffolding, and the scaffolding is what disappears. Here is the mapping, concept by concept.

Seven of those ten rows end in "gone" and two more end in "one line". That ratio is the entire argument for the move, and it is also why the migration feels anticlimactic when you actually do it.

How do I replace Scrapy selectors with a schema?

Stop describing where the value sits in the markup and start describing what the value is. A selector says "the text inside the second span under the div with class price-box". A schema says "there is a field called price and it is a number". The first breaks when a designer ships a redesign. The second does not.

Here is a small product spider, in the shape almost everybody writes it:

class ProductSpider(CrawlSpider):
    name = "products"
    start_urls = ["https://example.com/catalog"]
    rules = (Rule(LinkExtractor(allow=r"/products/"), callback="parse_item"),)

    def parse_item(self, response):
        yield {
            "name":  response.css("h1.product-title::text").get(),
            "price": response.css("span.price-box .amount::text").get(),
            "sku":   response.xpath("//dt[text()='SKU']/following-sibling::dd/text()").get(),
        }

And the same job as one request:

curl https://api.clawengine.ai/v1/crawl \
  -H "Authorization: Bearer $CLAWENGINE_API_KEY" \
  -d '{
    "url": "https://example.com/catalog",
    "include_paths": ["/products/*"],
    "max_depth": 2,
    "limit": 5000,
    "schema": {
      "name":  {"type": "string"},
      "price": {"type": "number"},
      "sku":   {"type": "string"}
    }
  }'

Three things are worth noticing. The XPath that walked from a dt to its sibling dd is gone, and that particular pattern is the single most fragile line in most spiders. The price comes back as a number rather than the string "$24.00", so the cleanup step in your pipeline goes too. And nothing in the request mentions a browser, because rendering is not a separate decision any more.

Write the schema against what your downstream code needs, not against what the page happens to show. If a column in your warehouse is NOT NULL, that field belongs in the schema as required, so a failure surfaces at extraction time instead of at insert time. This is the same discipline that makes structured data extraction from a website safe to build on, and it is worth an hour of argument before you write any code.

How do I replace LinkExtractor and Rule?

With a seed URL, a path pattern and a depth limit. A Rule(LinkExtractor(allow=r"/products/")) is doing exactly what include_paths: ["/products/*"] does, minus the regex escaping. The one control Scrapy gives you that a path rule does not is arbitrary Python in a process_links callback, and if you were using that, keep the spider.

Depth deserves a second look during a migration, because Scrapy projects often run with DEPTH_LIMIT unset. Depth counts link hops from the seed, not directory nesting, and most catalogs and documentation trees are fully reachable at depth 2 or 3. Setting it explicitly is usually the moment someone discovers the old spider was crawling four times more pages than anyone thought.

What happens to pipelines, middlewares and settings?

Middlewares and most settings have no equivalent, because the behavior they configure now happens inside the API: proxy selection, retry policy, throttling, user agent handling, cookie jars, browser rendering. That is the point of the trade. You give up the ability to tune those knobs and you stop being responsible for them.

Pipelines are different. Validation, deduplication against your own history, enrichment and the database write are your business logic, and they should survive the migration untouched. The practical change is where they run: instead of being invoked by Scrapy per item, they run in your app over the array the API returned, or in a handler behind a webhook if the crawl is long enough to be asynchronous.

One setting genuinely worth carrying over is politeness. If your spider ran with a deliberate DOWNLOAD_DELAY against a small site, keep that intent by scoping the crawl tightly and scheduling it sensibly rather than assuming the API will be as gentle as you were.

How long does migrating one spider take?

A spider you understand, against a site that has not changed shape recently, is a half day including verification. A spider written by somebody who left, with 300 lines of pipeline logic and undocumented settings, takes longer to read than to replace. That is the honest distribution, and it is why the sensible order is easiest first.

Two of those rows are worth stating plainly, because a vendor page usually will not. If your crawl depends on getting past active blocking, the right purchase is a proxy and unblocking platform, and the Zyte API is built for exactly that, with the added advantage that it plugs into the spiders you already have through scrapy-zyte-api. And if crawling is the product rather than a dependency of it, staying put is the correct engineering decision, which the Scrapy alternatives comparison goes through in more detail.

Does the migration actually save money?

It depends on what you count. Compare only the invoice and a managed API often looks more expensive than a couple of small servers. Compare the total and the picture usually flips, because a Scrapy stack that renders JavaScript is not a couple of small servers.

The line items that disappear are the browser fleet and its memory footprint, the proxy contract, the scheduler host, and the recurring engineering hours spent on selector repair and blocked-crawl triage. That last one is the largest and the least tracked. Teams that have never measured it are usually surprised: a single engineer spending two hours a week on spider maintenance is roughly a full working month a year.

The honest counter-case is small volume on easy targets. If you crawl 20,000 server-rendered pages a month from a box you already own, self-hosted Scrapy is cheaper and will stay cheaper. Migration pays off when rendering is involved, when the page count is high, or when the maintenance is landing on people you would rather have building something else.

What breaks after you migrate?

Three things, all predictable, all worth checking on day one rather than in week three.

Field coverage moves. A schema and a selector do not always disagree, but when they do it is usually on optional fields: a subtitle that was empty in the markup and is now absent from the JSON. Run both stacks side by side for one cycle and compare fill rates per field, not just row counts.

Page counts move. Setting an explicit depth limit for the first time frequently reveals that the old crawl was reaching pages nobody intended to collect, which is a fix rather than a regression, but it does mean the numbers will not match.

Ordering and timing move. Scrapy yields items as it goes, and a crawl run returns a set. If anything downstream assumed a stream, it needs a small change. This is normally a five-line difference in the consumer.

One thing that does not change is your obligations. Moving from a spider you wrote to an API you call does not alter what you may lawfully collect. Public and permitted pages, robots.txt respected, crawl-delay honored, no logged-in content. If any part of what you collect is personal data about identifiable people, the deletion and access duties follow that data wherever it lands, and teams handling it at scale usually end up with software that can trace where a person's records live across their systems rather than a spreadsheet and good intentions.

A sensible migration order

Pick the spider with the most maintenance history and the least clever code, and port that one first. Write its schema, run one crawl, and diff the output against the last spider run field by field. Fix the schema until the diff is boring. Then repeat, and only after two or three have landed should you decide what to do with the awkward ones.

Leave the Scrapy project in the repository until the replacements have survived a full cycle of whatever cadence you run on. Deleting it early is how you find out that one pipeline was quietly doing something important. Teams working in Python usually wire the replacement in through the same requests or httpx client they already use, which the Python web scraping API guide covers end to end.

The measure of a good migration here is not that the code got shorter, although it will. It is that the next time a source site redesigns, nobody gets paged.