Migrate an HTML Scraper API to a Schema API

An HTML scraping API solves exactly one problem, and it solves it well: it gets you the page. Proxies rotate, a browser runs the JavaScript, and a document comes back that looks like what a person would see. That is genuinely hard infrastructure and it is worth paying for.

The problem is what happens next. The document arrives and your code has to figure out which part of it is the product name, which nine paragraphs are navigation and cookie banners, and what to do when the site ships a redesign on a Tuesday. That layer is not infrastructure anybody sells you. It is a file in your repository with your team name on it.

This is the mechanical guide to handing that layer over. What maps to what, what the credit math actually looks like, and which scrapers you should leave exactly where they are.

What is the difference between an HTML scraping API and a schema API?

An HTML scraping API returns a document and stops. A schema API takes a description of the fields you want and returns those fields, already typed. The fetch, the render and the extraction happen inside one request instead of being split between a vendor and your codebase.

The practical consequence is where the breakage lands. With an HTML API, a site redesign produces a successful request, a 200 response, a full page of markup, and a row of nulls in your database, because your selectors matched nothing. Nothing alerted, because from the vendor point of view the call worked perfectly. With a schema API the extraction is the vendor problem, so a layout change is something they absorb rather than something that pages you at 2am.

What actually has to change when you migrate?

Less than the size of the codebase suggests, because most of what you own is translation rather than logic. Here is the mapping, concept by concept.

That last row matters. The migration does not touch your database, your queue or your scheduler. It replaces the middle of the pipeline and leaves both ends alone, which is why it can be done one scraper at a time without a big-bang cutover.

How much does an HTML scraping API actually cost per page?

This is where most migration decisions get made, and most teams have never worked it out, because the number on the pricing page is not the number you pay. Both major HTML APIs bill in credits and apply a multiplier depending on what the request needed.

Verified from both vendors in August 2026: ScrapingBee charges 1 credit for a plain fetch and 5 for a JavaScript-rendered page, and rendering is on by default, so an integration nobody configured pays 5 credits for every page including the static ones. Premium proxy is 10 credits alone or 25 with rendering, and stealth proxy is 75. ScraperAPI charges 1 credit flat, plus 10 for render=true, plus 10 for premium=true, with 25 for the two combined and 75 at the ultra premium tier, and it publishes separate rates for Amazon at 5, search engines at 25 and LinkedIn at 30.

Run that against a real workload. A team crawling 40,000 rendered pages a month pays 200,000 credits on ScrapingBee and 440,000 on ScraperAPI for identical work. That is the arithmetic behind our ScrapingBee vs ScraperAPI pricing comparison, which lays out both credit tables side by side with the plan ladders. The point for a migration is simpler: before you compare vendors, work out your blended cost per page today, because that is the only number that makes the next comparison honest.

What does the code look like before and after?

Here is a typical HTML API integration, compressed but not strawmanned. The parsing is deliberately short; in a real repository it is a module.

import requests
from bs4 import BeautifulSoup

def fetch(url):
    r = requests.get("https://vendor.example/v1", params={
        "api_key": KEY, "url": url, "render": "true", "premium": "true",
    })
    r.raise_for_status()
    return r.text

def parse(html):
    soup = BeautifulSoup(html, "lxml")
    name = soup.select_one("h1.product-title, h1[itemprop=name]")
    price = soup.select_one("span.price, meta[itemprop=price]")
    sku = soup.select_one("[data-sku], span.sku")
    return {
        "name": name.get_text(strip=True) if name else None,
        "price": to_decimal(price) if price else None,
        "sku": sku.get_text(strip=True) if sku else None,
    }

# plus: pagination, retries, a boilerplate stripper,
# and the fallback selectors added every time the site changed

And the same job expressed as a schema. The selectors are gone because the fields are declared rather than located.

import requests

resp = requests.post(
    "https://api.clawengine.ai/v1/crawl",
    headers={"Authorization": f"Bearer {KEY}"},
    json={
        "url": "https://example.com/products",
        "path_prefix": "/products/",
        "max_pages": 500,
        "render": True,
        "schema": {
            "name": "string",
            "price": "number",
            "sku": "string",
            "in_stock": "boolean",
        },
    },
).json()

for page in resp["pages"]:
    upsert(page["data"])   # already typed, already clean

The diff is not really about line count. It is that the second version has no branch that says if name else None, because there is no selector that can silently stop matching. If a field cannot be found the response says so explicitly, which is something you can alert on.

Which scrapers should you not migrate?

Three categories, and being honest about them saves a wasted week.

Anything behind a login or a form. If the scraper authenticates, fills fields, or clicks through a multi-step flow, a request-response API cannot express it. That work stays in browser automation. We do not do it and neither does a schema API in general.

Anything on a defended target. If the site runs Cloudflare, DataDome or PerimeterX and your current vendor is paying the premium or ultra premium multiplier to get through, that multiplier is buying you something real. Do not migrate to a tool that does not defeat anti-bot systems and then act surprised. ClawEngine does not, which is exactly why the honest recommendation for those targets is to keep a proxy-first scraping API like ScraperAPI in the stack for that subset.

Anything with a per-site endpoint you rely on. If you are pulling Amazon listings or search results through a vendor endpoint built for that platform, a general crawler is a downgrade. Those endpoints encode pagination rules and rate limits somebody else maintains, and rebuilding that yourself is not a migration, it is a project.

In most codebases these categories account for one or two scrapers out of a dozen. The rest are ordinary public pages that need a browser and a parser, and those are the ones that port in an afternoon.

A sensible migration order

Start with the scraper that has the longest history of selector fixes and the least clever code. Write its schema, run one crawl, and diff the output against your last known-good run field by field. Fix the schema until the diff is boring. Only then move to the second one.

Keep both paths live for a full cycle of whatever cadence you run on, whether that is nightly or weekly. Running the old scraper and the new crawl side by side for two weeks costs a rounding error in credits and it is the only way to find the field that was quietly wrong in both. Delete the selector file after that, not before.

Watch what happens to the data downstream too. Typed output tends to reveal that fields you assumed were populated are actually sparse, which is useful and slightly demoralizing. If the scraped records are customer reviews or support content rather than catalog data, the natural next step is to unify them with the rest of your product feedback instead of leaving them in a table nobody queries.

Your obligations do not change with the vendor. Public and permitted pages, robots.txt respected, crawl-delay honored, no logged-in content, and the same duties over any personal data you collect. Handing extraction to an API moves the engineering, not the responsibility. If you want the longer version of what a declared schema buys you in practice, the structured data extraction guide covers schema design, and teams working in Python usually wire the replacement in through the same client they already use, which the Python web scraping API guide walks through end to end.

The measure of a good migration is not that the code got shorter, although it will. It is that the next time a source site redesigns, the data keeps arriving and nobody finds out from a dashboard full of nulls.