Python Web Scraping: requests, Scrapy, or a Scraping API?

The short answer

Use requests plus BeautifulSoup when you are scraping a handful of static pages and the script is allowed to break occasionally. Use Scrapy when you are crawling thousands of pages, want control of the crawl loop, and are willing to own the infrastructure it runs on. Use a scraping API when the pages render client-side, when the job has to keep running unattended in production, or when your team's time is better spent on what happens after the data arrives. Most real projects end up mixing them, and that is fine.

Why this question keeps coming up

Python scraping has an unusually gentle on-ramp and an unusually steep maintenance curve. Twelve lines of requests and BeautifulSoup will pull a table off a page in under five minutes, which makes the build-it-yourself choice look obviously correct. The costs show up later, and they show up somewhere other than where you looked: a Selenium container that needs upgrading, a proxy module nobody owns, selectors that silently start matching nothing after a redesign, and a downstream dashboard that has quietly been showing last month's numbers.

So the real comparison is not "library versus API" on day one. It is total cost across the year, where the variables are how many distinct sites you touch, how often they change, whether they render in the browser, and how much it costs your business when the data is wrong for a week.

Option 1: requests plus BeautifulSoup

The standard starting point. requests fetches the HTML, BeautifulSoup parses it, and you write selectors against the structure you see in devtools.

import requests
from bs4 import BeautifulSoup

html = requests.get("https://example.com/products/widget",
                    headers={"User-Agent": "AcmeResearch/1.0 (+contact@acme.com)"}).text
soup = BeautifulSoup(html, "html.parser")

name  = soup.select_one("h1.product-title").get_text(strip=True)
price = soup.select_one("span[data-price]")["data-price"]

What it is genuinely good at: it is free, it has no infrastructure, it is trivially debuggable, and for server-rendered pages it is fast. If you are pulling one government data table monthly, this is the correct answer and anything more is overengineering.

Where it falls down: that select_one returns None the moment a class name changes, and unless you wrapped it defensively you get an AttributeError in a cron job at 3am. It also sees only the HTML the server sent, so on any single page app the price you want simply is not in the response. And each new site means a new set of selectors, so effort scales linearly with sources.

Option 2: Scrapy

Scrapy is a full crawling framework rather than a parser. You get a scheduler, deduplication, concurrency control, middleware hooks, item pipelines and built-in throttling. If you are crawling a large site properly, those are things you would otherwise write yourself and write worse.

import scrapy

class CatalogSpider(scrapy.Spider):
    name = "catalog"
    start_urls = ["https://example.com/catalog"]
    custom_settings = {
        "ROBOTSTXT_OBEY": True,
        "DOWNLOAD_DELAY": 1.0,
        "AUTOTHROTTLE_ENABLED": True,
    }

    def parse(self, response):
        for href in response.css("a.product::attr(href)").getall():
            yield response.follow(href, self.parse_product)
        yield from response.follow_all(css="a.next", callback=self.parse)

    def parse_product(self, response):
        yield {
            "sku":   response.css("[itemprop=sku]::text").get(),
            "title": response.css("h1::text").get(),
            "price": response.css("[itemprop=price]::attr(content)").get(),
        }

What it is good at: scale and control. AutoThrottle adapts your request rate to server latency, ROBOTSTXT_OBEY is a one-line setting, and pipelines give you a clean place to validate and store items. For a crawl of 100,000 pages on sites you understand, Scrapy is hard to beat.

Where it falls down: the learning curve is real, and it does not render JavaScript. Bolting on Playwright via scrapy-playwright works but now you are running browsers, which changes your deployment from a Python process into something with memory limits, container images and a whole class of new failure modes. You still write a selector set per site, and you still own every proxy, retry and captcha decision.

Option 3: a scraping API

Here the fetching, rendering, retrying and parsing happen on someone else's infrastructure and you make one HTTP call. With a Python web scraping API you POST a URL plus a schema and get typed fields back, so the code on your side stops containing selectors entirely.

import os, requests

res = requests.post("https://api.clawengine.ai/v1/extract",
    headers={"Authorization": f"Bearer {os.environ['CLAWENGINE_API_KEY']}"},
    json={
        "url": "https://example.com/products/widget",
        "format": "json",
        "render": True,
        "schema": {"product_name": "string", "price": "number", "in_stock": "boolean"},
    })

data = res.json()["data"]

What it is good at: the failure mode changes. A source redesign is a problem for the extraction layer rather than a code change on your side, because you asked for a field by name instead of by CSS path. Rendering is a boolean. Adding the tenth site costs the same as adding the second. And because every record has identical keys, the output drops straight into pandas.DataFrame or a vector store without a cleanup pass.

Where it falls down, honestly: it costs money, per request, forever. Libraries are free. You are also dependent on a vendor's uptime and pricing, and you get less control than a Scrapy middleware gives you. If your scraping is small, static and stable, paying for an API is a worse deal than writing thirty lines yourself, and any vendor telling you otherwise is optimizing for their revenue rather than your stack.

Side by side

How to actually decide

Four questions get you to an answer faster than any feature comparison.

Does the data appear in view-source? Open the page, view source, and search for the value you want. If it is there, requests can reach it and you may not need anything heavier. If you only see an empty <div id="root">, the page renders client-side and your choice narrows to running a browser or paying someone to run one.

How many distinct sites? One or two, libraries. Ten or more, the per-site selector cost dominates everything else and an API's flat call shape wins on effort alone.

Who gets paged when it breaks? If the answer is "nobody, it's a research script", breakage is cheap and free tools are correct. If a pricing model or a customer-facing feature consumes the output, the cost of silent staleness usually exceeds the subscription within a month.

What happens to the data next? If it lands in a warehouse or gets pushed into internal systems, you will want consistent typed records rather than per-site dictionaries, because the normalization work you skip at extraction time reappears at load time. Teams that wire the output into their warehouse and internal apps feel this immediately: a uniform schema across sources is worth more than the scraping code itself.

The hybrid most teams land on

You do not have to pick one. A common and sensible setup keeps Scrapy for the crawl orchestration a team already has, and calls an extraction endpoint from inside the spider for the specific sources that render client-side or change often. That gets rendering without adding a browser fleet to the Scrapy deployment, and it lets you migrate one troublesome source at a time instead of rewriting a working project.

Equally common: requests plus BeautifulSoup for the five stable, well-behaved sources that have not changed in two years, and an API for the twenty that do. Spending money only where breakage actually costs you is a better policy than picking a side.

Whichever you pick, crawl politely

The rules do not change with the tool. Read robots.txt and respect it. Set a User-Agent that identifies you and gives someone a way to make contact. Limit concurrency, honor crawl-delay, back off on 429 and 5xx responses, and cache so you are not refetching pages that have not changed. Stay on public and permitted pages, and if a site is clearly refusing automated access after you have behaved well, treat that as an answer rather than a puzzle. There is more on where the lines sit in our guide to whether web scraping is legal.

If the output is headed for a model

One last consideration that changes the calculus. If the scraped pages are going into a RAG pipeline or an agent rather than a spreadsheet, markdown beats parsed HTML, because you want the heading structure preserved for chunking and the navigation chrome gone. Building that cleanup yourself on top of BeautifulSoup is a genuine project. Requesting markdown directly is a parameter. There is a fuller treatment in LLM web scraping in Python and in the web scraping for RAG walkthrough.