Scrapy alternatives: 11 web scraping tools compared for Python teams

Where it wins. Crawl, JavaScript rendering and typed schema extraction happen in a single API call, and robots.txt plus site Terms of Service are respected by default.

What to watch. There is no free plan, so it is priced for teams running real pipelines rather than one-off experiments.

Where it wins. Excellent developer experience, a well-loved open-source project, and markdown output tuned for token efficiency.

What to watch. The headline 1 credit a page is the plain scrape only. Asking for structured JSON adds 4 credits, so a page you want back as typed fields is 5 credits, which turns Standard from $0.83 into $4.15 per 1,000 pages. PDF parsing adds 1 credit a PDF page, a prompt injection check adds 4, zero data retention adds 1, Map is 1 credit a call, Search is 2 per 10 results, and Interact is 2 to 7 credits a browser minute. A page that comes back as a 403 or 404 still costs 1 credit; only a request that returns no document at all is free. Plan credits do not roll over except on annual Scale, and pay-as-you-go top-ups run $5.00 per 1,000 extra credits on Hobby, $2.50 on Standard, $2.00 on Growth and $1.00 on Scale, up to three times the in-plan rate.

Where it wins. The largest proxy network in the category (150M+ residential IPs across 195 countries) and hundreds of prebuilt domain scrapers and ready-made datasets.

What to watch. It is a broad platform rather than a single LLM-ready endpoint, so output usually needs cleaning before you can embed it, and the pricing surface is complex: each product has its own meter (records, requests or gigabytes), plans carry a minimum monthly commitment billed from the 1st, and unused plan volume does not roll over.

Where it wins. A marketplace of thousands of prebuilt Actors, so common targets are already solved, plus a full automation and scheduling platform.

What to watch. Costs stack in layers: the plan buys a dollar allowance, Actor compute burns it at $0.13 to $0.20 per compute unit, and residential proxies add $7 to $8 per GB on top. Unused allowance expires monthly, and output is generic JSON rather than LLM-ready markdown.

Where it wins. Very easy to adopt, dependable rendering, and a Google Search API bundled into every tier.

What to watch. JavaScript rendering is on by default and costs 5 credits, so a Freelance plan is 50,000 rendered pages rather than the 250,000 the credit count suggests. Premium proxy is 10 credits alone or 25 with rendering, and stealth proxy is 75. Responses with a 200, 404 or 410 status are billed. You also mostly get HTML back, so the cleaning, chunking and structuring work for an LLM is still yours to do.

Where it wins. Strong price per request at volume and a very simple drop-in proxy API.

What to watch. The multipliers decide the bill: a plain request is 1 credit, render 10, premium 10, screenshot 10, premium with render 25, ultra premium 30 and ultra premium with render 75, while Amazon, Walmart and eBay cost 5, Google and Bing 25 and LinkedIn 30, and clearing Cloudflare, DataDome or PerimeterX adds 10. Two policies matter more than the rates: credits do not roll over, and pay-as-you-go overage is available only on Scaling and above, so hitting 100 percent on Hobby, Startup or Business stops the pipeline until you upgrade. Only 200 and 404 responses are billed. It is proxy infrastructure first, so an LLM pipeline still needs its own parsing, boilerplate stripping and schema layer.

Where it wins. Focused on getting through Cloudflare, DataDome and PerimeterX where simpler fetchers fail.

What to watch. The multipliers set the real price, not the headline credit count: a plain fetch is 1 credit, a JavaScript-rendered page is 5, premium proxies are 10, and premium with rendering is 25, which is the ceiling. Residential bandwidth is billed at a flat 25,000 credits per GB, and Browser Sessions add 5 credits a minute on top of bandwidth. Launch at $57 therefore buys about 50,000 rendered pages. Failed requests are not charged, but 404 and 410 responses count as successful. You also get HTML back, so the cleaning and structuring work for an LLM is still yours.

Where it wins. Enterprise-grade unblocking, a large global proxy network, and dedicated parsers for major targets, plus a free Custom Parser for your own CSS or XPath rules.

What to watch. Two rules move the real bill. First, rates are set by target category, roughly $0.25 to $0.50 per 1,000 for Amazon, $0.50 to $1.00 for Google, $0.70 to $1.15 for other sources and $0.95 to $1.35 with JavaScript rendering, and the headline result count is quoted against the cheapest one. Oxylabs own maximum-results table shows the same free trial buying 2,000 Amazon results but only 769 JavaScript-rendered results from an ordinary site, a 2.6 times spread that carries up the whole ladder. Second, the billing documentation counts any 2xx or 4xx response as a successful result, so a 404 on a dead link or a 403 from a site that blocked you is billed at full rate. Whole-site crawling means buying a second product.

Where it wins. No vendor bill at all, full control, deep crawling with BFS, DFS and best-first strategies, and output already shaped for RAG ingestion. It is the most popular open-source crawler in the category, with roughly 82,000 GitHub stars.

What to watch. You own the ops: proxy rotation, browser fleet, retries, blocks and upgrades. Free software is not free infrastructure.

Where it wins. A pre-built Knowledge Graph of more than 10 billion entities you can query instead of crawling, and computer-vision extraction that classifies and structures pages with no per-site rules to write.

What to watch. The first paid tier is $299 a month and credits are consumed fast (a Knowledge Graph record costs 25 credits, a data-center proxy request doubles the cost), so for plain RAG ingestion it is expensive and heavier than you need.

Where it wins. The open-source library (MIT, 30.8k GitHub stars) is a genuine option rather than a demo, it plugs into OpenAI, Groq, Azure, Gemini or a local Ollama model, and the managed API starts at $20 a month, just above the cheapest entry plans in the category.

What to watch. The managed API bills per credit and the rate depends on the endpoint (extract costs 5 credits, stealth adds 4 to 9 depending on the render mode, a crawl adds 2 on top of per-page scrape cost), so cost per page is harder to predict. Failed requests are not charged. Self-hosting means you supply the LLM key and pay model tokens on every page.