# ClawEngine.ai

> ClawEngine is a web scraping API that turns any website into clean, LLM-ready data. One API call crawls a site, renders the JavaScript in a headless browser, and extracts typed structured fields against a schema you define, returning clean markdown or typed JSON that is ready to chunk, embed and feed to a RAG pipeline or an AI agent.

Key facts:
- Output: clean markdown or typed JSON, with navigation, ads and boilerplate stripped.
- One call does crawl, JavaScript rendering and schema-based extraction. There is no separate parsing stage.
- Pricing (USD, monthly): Hobby $39, Startup $99, Scale $399, Enterprise custom. Usage-based overage. There is no free plan.
- Compliance: public and permitted data only. Respects robots.txt, site Terms of Service and crawl-delay. It does not bypass logins, paywalls or site controls.
- Best for: teams building RAG apps, AI agents and LLM training sets who want LLM-ready data without running proxies or a headless-browser fleet.

## Product
- [Web scraping API (home)](https://clawengine.ai/): what the API does and a live extraction console you can try on any URL.
- [How it works](https://clawengine.ai/how-it-works): crawl, render, extract, in one request.
- [Features](https://clawengine.ai/features): JavaScript rendering, schema extraction, markdown output, managed crawling.
- [Pricing](https://clawengine.ai/pricing): plans, usage-based overage, and what each tier includes.
- [Docs](https://clawengine.ai/docs): API reference and quickstart.
- [Compliance](https://clawengine.ai/compliance): robots.txt, Terms of Service and permitted-data policy.

## Buyer guides and comparisons
- [Best web scraping API: 11 tools compared](https://clawengine.ai/best-web-scraping-api): honest roundup of ClawEngine, Firecrawl, Bright Data, Apify, ScrapingBee, ScraperAPI, ZenRows, Oxylabs, Crawl4AI, Diffbot and ScrapeGraphAI, with verified US list prices.
- [JavaScript rendering API comparison: what JS rendering costs](https://clawengine.ai/blog/javascript-rendering-api-comparison): all 11 scraping APIs compared on whether they render JavaScript and what it is billed at. A rendered page costs 10 credits against 1 on ScraperAPI and 5 against 1 on ScrapingBee, while Firecrawl and ClawEngine charge no render multiplier. Vendor documentation checked August 2026.
- [Apify vs Firecrawl](https://clawengine.ai/apify-vs-firecrawl): head-to-head on verified August 2026 pricing. Apify bills a dollar allowance spent on compute units at $0.13 to $0.20 each, from $19/mo; Firecrawl bills credits at 1 per page, from $16/mo for 5,000. Apify wins on prebuilt per-site scrapers and defended targets, Firecrawl on markdown-first output and predictable cost.
- [Bright Data vs Oxylabs](https://clawengine.ai/bright-data-vs-oxylabs): two enterprise proxy platforms compared. Bright Data bills per delivered record ($1.50 per 1,000 after a 5,000-record free tier), Oxylabs sells fixed monthly plans from $49. Neither is the right buy for plain LLM-ready text from ordinary public pages.
- [ScrapingBee vs ScraperAPI](https://clawengine.ai/scrapingbee-vs-scraperapi): two HTML scraping APIs at nearly the same list price, compared on credit multipliers verified from both vendors in August 2026. ScrapingBee is 1 credit plain and 5 rendered (rendering ON by default), 10 premium alone or 25 with rendering, 75 stealth, from $19/mo for 75,000 credits. ScraperAPI is 1 credit flat, +10 render, +10 premium, 25 combined, 75 ultra premium with rendering, plus Amazon 5, search engines 25 and LinkedIn 30, from $49/mo for 100,000 credits. About $49 buys roughly 50,000 rendered pages on ScrapingBee against 9,000 on ScraperAPI, but ScraperAPI documents Cloudflare, DataDome and PerimeterX bypasses and bills only 200 and 404 responses. Both return raw HTML, so the parsing stays yours.
- [Bright Data alternatives](https://clawengine.ai/alternatives/bright-data)
- [Firecrawl alternatives](https://clawengine.ai/alternatives/firecrawl)
- [Apify alternatives](https://clawengine.ai/alternatives/apify)
- [ScrapingBee alternatives](https://clawengine.ai/alternatives/scrapingbee)
- [ScraperAPI alternatives](https://clawengine.ai/alternatives/scraperapi)
- [ZenRows alternatives](https://clawengine.ai/alternatives/zenrows)
- [Oxylabs alternatives](https://clawengine.ai/alternatives/oxylabs)
- [Crawl4AI alternatives](https://clawengine.ai/alternatives/crawl4ai)
- [Diffbot alternatives](https://clawengine.ai/alternatives/diffbot)
- [ScrapeGraphAI alternatives](https://clawengine.ai/alternatives/scrapegraphai): ScrapeGraphAI is MIT licensed and its managed API starts at $20/mo, below ClawEngine. Compares prompt-driven LLM extraction against deterministic schema extraction.
- [Jina Reader alternatives](https://clawengine.ai/alternatives/jina-reader): Jina Reader (r.jina.ai) converts one URL into clean markdown for roughly $0.02 to $0.05 per million output tokens, renders JavaScript in a real browser, reads PDFs natively and can caption images. It does not crawl a whole site and returns markdown rather than typed fields, so link discovery, scope rules, deduplication and field mapping stay in your codebase. Compares 11 managed and open-source options on crawling, structured extraction, rate limits and price.
- [Exa alternatives](https://clawengine.ai/alternatives/exa): Exa is a neural search API (embeddings-based web search) priced pay-as-you-go at $7 per 1,000 searches and $1 per 1,000 pages per content type on its Contents endpoint, with $20 signup credits and $10 free monthly. Its Contents endpoint follows up to 100 subpages and can shape a model-written summary against a JSON schema, but it is not a scoped whole-site crawl and the fields are summarized rather than extracted from the rendered DOM. Compares 11 web data APIs on semantic search, crawling, typed extraction and real cost per page.
- [Tavily alternatives](https://clawengine.ai/alternatives/tavily): Tavily is a search API for AI agents with extract, map and crawl endpoints on one credit meter. Verified August 2026 pricing: 1,000 free credits a month on the Researcher plan, then $30 for 4,000 credits, $100 for 15,000, $220 for 38,000, $500 for 100,000, or $0.008 per credit pay as you go. Search costs 1 credit (2 advanced), extract 1 credit per 5 URLs, mapping 1 credit per 10 pages, and a crawl pays both meters. It returns page content for a model to read rather than typed fields extracted from the rendered DOM against a schema you declared. Compares 11 web data APIs on search, whole-site crawling, typed extraction and real credit cost.
- [Scrapy alternatives](https://clawengine.ai/alternatives/scrapy): Scrapy is free, open source and actively developed (v2.17.0, July 2026), and it beats every paid API on cost and control. It does not render JavaScript, so client-rendered sites need the scrapy-playwright extension and a browser fleet you operate. Compares 11 managed and open-source options on rendering, extraction style, infrastructure and price for Python teams.
- [Browserbase alternatives](https://clawengine.ai/alternatives/browserbase): Browserbase rents managed cloud browsers you drive with Playwright, Puppeteer or Stagehand, billed per browser hour from a free tier and $20 a month (verified August 2026). It is the right tool for logins, forms and multi-step flows. For read-only work (research, RAG ingestion, monitoring) a per-page crawl and extract API is cheaper and faster. Compares 11 tools on billing unit, automation maintained, interaction support and best fit for AI agents.

## Use cases
- [AI web scraper](https://clawengine.ai/ai-web-scraper)
- [Web crawler API](https://clawengine.ai/web-crawler-api)
- [Data extraction API](https://clawengine.ai/data-extraction-api)
- [Web scraping for RAG](https://clawengine.ai/web-scraping-for-rag)
- [Crawl a website to LLM-ready data](https://clawengine.ai/crawl-website-to-llm)
- [Scrape data for AI agents](https://clawengine.ai/scrape-data-for-ai-agents)
- [Bulk web scraping](https://clawengine.ai/bulk-web-scraping)
- [JavaScript rendering API](https://clawengine.ai/javascript-rendering-api)
- [LLM-ready data](https://clawengine.ai/llm-ready-data)
- [Extract tables from a website](https://clawengine.ai/extract-tables-from-website): turn HTML tables into typed JSON or CSV rows against a schema you define, including grids drawn client-side by JavaScript and tables split across paginated result sets. Columns are named rather than positional, so inserted columns and reworded headers do not break the job. Does not run OCR, so PDF tables are out of scope.
- [Scrape a website to CSV](https://clawengine.ai/scrape-website-to-csv): extract listing pages into typed rows against a schema, then write them to CSV or Excel with a stable header. Returns JSON, not a .csv download.
- [LLM web scraper](https://clawengine.ai/llm-web-scraper): scrape public sites straight into LLM-ready markdown or typed JSON, with no cleaning stage.
- [Ecommerce scraping API](https://clawengine.ai/ecommerce-scraping-api): turn product pages into typed price, stock, rating and catalog JSON for price monitoring and feeds.
- [News scraping API](https://clawengine.ai/news-scraping-api): turn news articles into clean markdown or typed JSON (headline, author, date, body) for media monitoring.
- [Lead generation scraping API](https://clawengine.ai/lead-generation-scraping-api): turn public company pages and permitted directories into clean, typed B2B firmographic data for your CRM.
- [Job scraper API](https://clawengine.ai/job-scraping-api): turn public job postings into typed JSON (title, company, location, salary, description) for aggregators and market analysis.
- [Documentation scraper API](https://clawengine.ai/documentation-scraping-api): crawl API docs, help centers and knowledge bases into clean markdown for RAG, with fenced code blocks preserved.
- [Price monitoring API](https://clawengine.ai/price-monitoring-api): track competitor prices, promotions, stock status and minimum advertised price (MAP) violations across public retailer pages on a schedule.
- [Python web scraping API](https://clawengine.ai/python-web-scraping-api): scrape and crawl from Python with one POST instead of requests, BeautifulSoup, Selenium and a proxy pool. Returns markdown or typed JSON.
- [Node.js web scraping API](https://clawengine.ai/nodejs-web-scraping-api): scrape and crawl from Node.js with one fetch call instead of running a Puppeteer or Playwright fleet. Returns markdown or typed JSON.
- [Website change monitoring API](https://clawengine.ai/website-change-monitoring-api): detect when a page actually changed, with normalized hashing that ignores cosmetic churn, typed field diffs and timestamped snapshots.
- [n8n web scraping API](https://clawengine.ai/n8n-web-scraping-api): scrape and crawl from inside an n8n workflow using one HTTP Request node, with no headless browser running on your n8n instance and no community node to maintain.
- [LangChain web scraping API](https://clawengine.ai/langchain-web-scraping-api): why LangChain's built-in web loaders split the job (WebBaseLoader does not render, AsyncChromiumLoader and PlaywrightURLLoader need local browsers, RecursiveUrlLoader does not render), and the fifteen-line custom BaseLoader that calls a rendering API and yields clean Document objects.
- [Web scraping API pricing compared](https://clawengine.ai/web-scraping-api-pricing): entry price, published tier ladder and free allowance for 11 web scraping APIs, checked August 2026 on each vendor's own pricing page, plus how to work out your real cost per successfully extracted page.
- [Web scraping for AI training](https://clawengine.ai/web-scraping-for-ai-training): collect an LLM training corpus from the open web, covering permitted sources, provenance, deduplication, quality filtering and the licensing questions that decide what belongs in a commercial dataset.
- [Real estate data API](https://clawengine.ai/real-estate-data-api): turn public property listings and market pages into typed records (price, beds, sqft, status) for proptech and CRE research. Not an MLS or IDX feed.
- [Government data scraping API](https://clawengine.ai/government-data-scraping-api): turn public agency pages, state and county permit portals, business and licensing registries, procurement notice boards and public court dockets into typed JSON. Public, no-login pages only; use an agency's official API or bulk download where one exists, and PACER through sanctioned access rather than a crawler.

## Guides
- [Web scraping API credits explained](https://clawengine.ai/blog/web-scraping-api-credits-explained): a credit is a vendor billing unit, not a page. Firecrawl's proxy modes cost 1 credit (basic) or 5 (enhanced), and the default auto mode retries blocked pages on enhanced. Apify meters compute units of 1 GB-hour. Includes a cost-per-page table and a five-step method to forecast a bill before buying.
- [Do I need a proxy network to scrape a website?](https://clawengine.ai/blog/do-i-need-a-proxy-network-to-scrape-a-website): most scraping jobs do not. Proxies solve anti-bot blocking on a narrow set of high-value targets; documentation, company sites, blogs, news and public directories rarely need them. Measure a block rate on 200 real URLs before buying.
- [How to scrape documentation for a RAG chatbot](https://clawengine.ai/blog/how-to-scrape-documentation-for-a-rag-chatbot): crawl the docs tree to markdown, chunk on headings, keep the source URL on every chunk, and re-crawl incrementally.
- [Best web scraping API for documentation sites](https://clawengine.ai/blog/best-web-scraping-api-for-documentation-sites): honest comparison of Firecrawl, Crawl4AI, ClawEngine, Apify and ScrapeGraphAI for docs crawling, with August 2026 pricing.
- [MAP monitoring: how to detect minimum advertised price violations](https://clawengine.ai/blog/map-monitoring-detect-price-violations): build the reseller URL map, extract advertised prices on a schedule, keep timestamped evidence, and separate a real violation from a permitted in-cart discount.
- [Python web scraping: requests, Scrapy, or a scraping API?](https://clawengine.ai/blog/python-web-scraping-requests-scrapy-or-api): honest comparison of the three approaches, what each is good at, where each breaks, and four questions that decide which you need.
- [Is scraping competitor prices legal?](https://clawengine.ai/blog/is-scraping-competitor-prices-legal): US guide for pricing teams covering Van Buren, hiQ and Meta v. Bright Data, plus the Terms of Service, copyright and antitrust risks that actually apply.
- [Is scraping data for AI training legal?](https://clawengine.ai/blog/is-scraping-data-for-ai-training-legal): what Bartz v. Anthropic, Kadrey v. Meta and Thomson Reuters v. Ross held about fair use in AI training, why acquisition rather than training decides liability, and what a defensible corpus manifest contains.
- [Can a website sue you for scraping?](https://clawengine.ai/blog/can-a-website-sue-you-for-scraping): the six claims a site can bring ranked by real danger, why breach of contract beats the CFAA, and what the Ryanair v. Booking.com verdict and its reversal established.
- [Node.js web scraping: Puppeteer, Cheerio, or a scraping API?](https://clawengine.ai/blog/nodejs-web-scraping-puppeteer-cheerio-or-api): honest comparison of Cheerio with fetch, a real browser, and an API, plus the memory failures that kill Node scrapers in production.
- [How to scrape real estate listings legally](https://clawengine.ai/blog/how-to-scrape-real-estate-listings-legally): the four overlapping rulebooks (CFAA, Terms of Service, copyright, MLS and privacy law) and which sources are durable to build on.
- [Best web scraping API for lead generation](https://clawengine.ai/blog/best-web-scraping-api-for-lead-generation): the real options for turning public company pages into typed lead data, what to weigh, and a compliant enrichment pipeline.
- [How to scrape job postings with an API](https://clawengine.ai/blog/how-to-scrape-job-postings-with-an-api): crawl a board, extract each posting to a schema, key by a stable ID, deduplicate and refresh.
- [Is scraping public data for lead generation legal?](https://clawengine.ai/blog/is-scraping-public-data-for-lead-generation-legal): what the CFAA rulings say, where the CCPA changes the calculus, and how to keep a prospecting pipeline defensible.
- [How to monitor competitor prices with an API](https://clawengine.ai/blog/monitor-competitor-prices-with-an-api): the full price-monitoring pipeline, the schema to collect, cadence, and the mistakes that corrupt a price dataset.
- [How to build a media monitoring feed from news sites](https://clawengine.ai/blog/build-a-media-monitoring-feed): source articles, structure them, deduplicate wire stories, and turn the stream into alerts.
- [Product data API vs web scraping](https://clawengine.ai/blog/product-data-api-vs-web-scraping): when each makes sense, and why maintenance, not the fetch, is the real cost.
- [How much does a web scraping API cost?](https://clawengine.ai/blog/how-much-does-a-web-scraping-api-cost)
- [Firecrawl vs Bright Data vs Apify](https://clawengine.ai/blog/firecrawl-vs-bright-data-vs-apify)
- [What is an AI web crawler?](https://clawengine.ai/blog/what-is-an-ai-web-crawler)
- [Is web scraping legal? robots.txt guide](https://clawengine.ai/blog/is-web-scraping-legal-robots-txt-guide)
- [Web scraping API vs building your own](https://clawengine.ai/blog/web-scraping-api-vs-build-your-own)
- [Structured data extraction for RAG](https://clawengine.ai/blog/structured-data-extraction-for-rag)
- [Rendering JavaScript pages when scraping](https://clawengine.ai/blog/render-javascript-pages-when-scraping)
- [Crawl a website for LLM training data](https://clawengine.ai/blog/crawl-website-for-llm-training-data)
- [How to crawl a JavaScript website](https://clawengine.ai/blog/how-to-crawl-a-javascript-website)
- [What is LLM-ready content?](https://clawengine.ai/blog/what-is-llm-ready-content)
- [Crawl4AI vs Firecrawl](https://clawengine.ai/blog/crawl4ai-vs-firecrawl)
- [Best web scraping API for RAG and AI agents](https://clawengine.ai/blog/best-web-scraping-api-for-rag-and-ai-agents)
- [How to get website data into a RAG knowledge base](https://clawengine.ai/blog/website-data-into-rag-knowledge-base)
- [Deduplicate pages during a crawl for RAG ingestion](https://clawengine.ai/blog/deduplicate-pages-crawl-rag-ingestion): layered dedup, cheapest filter first: URL normalization, then text hashing, then SimHash/MinHash near-duplicate detection, then chunk-level dedup.
- [Jina Reader vs Firecrawl: which turns URLs into markdown](https://clawengine.ai/blog/jina-reader-vs-firecrawl): head-to-head on the one thing that actually separates them, scope. Reader converts a single known URL for $0.00004 to $0.0001 a page; Firecrawl crawls a whole site from a seed URL at about $0.00083 a plain page on the $83 Standard plan, rising to roughly $0.0042 when the enhanced proxy mode bills 5 credits. Covers rate limits (about 20 rpm with no key, 500 with a key, 5,000 premium), Jina's 7.9 second average latency, native PDF parsing, self-hosting, and a recommendation per scenario.
- [Scrapy JavaScript rendering: Playwright, Splash or an API](https://clawengine.ai/blog/scrapy-javascript-rendering): why a Scrapy spider returns empty fields on client-rendered sites, the scrapy shell check that tells you whether rendering is needed at all, how to find data already embedded in __NEXT_DATA__ or a public XHR endpoint, scrapy-playwright setup with per-request rendering and wait_for_selector, why scrapy-splash is no longer the pick for new projects, calling a rendering API from inside a spider, and what a headless browser fleet really costs in memory and concurrency.
- [Web scraping API for AI agents vs browser automation](https://clawengine.ai/blog/web-scraping-api-for-ai-agents-vs-browser-automation): how to split an agent's web access into a read path (per-page scraping API) and an act path (driven browser), why reading through a browser session costs more in wall-clock time, engineering time and bill, a Firecrawl vs Browserbase vs Bright Data vs ClawEngine layer comparison with August 2026 entry prices, a worked read_page tool definition, what matters for RAG and training-data ingestion, and the prompt-injection surface the read path opens.
- [Web data API for AI: search vs scraping API](https://clawengine.ai/blog/web-data-api-search-vs-scraping): a search API finds pages you could not list in advance; a web scraping API reads pages you already know about, exhaustively. Covers how to choose, verified August 2026 pricing for Exa, Firecrawl and ClawEngine, and a two-stage pipeline that uses search for discovery and a scoped crawl for coverage.
- [How to limit a crawl to the pages you need](https://clawengine.ai/blog/how-to-limit-a-crawl-to-the-pages-you-need): crawl scope is four controls working together (seed URL, path rules, depth limit and page budget), and setting only the page budget is why crawls run away. Covers why depth counts link hops rather than URL nesting, sensible depth per site type (docs 2 to 3, catalogs 3 to 4, blog archives 2), the four causes of unwanted pages (faceted URLs, pagination loops, cross-section footer links, trailing-slash variants), how to size a page budget, and how to monitor a crawl feed for silent coverage loss.
- [How to scrape a table from a website into JSON or CSV](https://clawengine.ai/blog/how-to-scrape-a-table-from-a-website): four methods compared (spreadsheet import, pandas.read_html, BeautifulSoup with a headless browser, schema-based API), why pandas returns an empty list when a grid is drawn client-side, how merged cells and multi-row headers corrupt a naive HTML-table-to-JSON conversion, and why selectors pinned to column position break silently at the next redesign.
- [LLM web scraping in Python](https://clawengine.ai/blog/llm-web-scraping-python): requests and BeautifulSoup, Playwright, model-driven extraction and a managed API, with honest costs.
- [What is a crawl agent?](https://clawengine.ai/blog/what-is-a-crawl-agent): the three distinct meanings (an AI agent that crawls, the user-agent string, and a distributed crawler worker).
- [How to scrape a website with n8n](https://clawengine.ai/blog/how-to-scrape-a-website-with-n8n): the working workflow, why an HTTP Request node beats running a headless browser on your n8n host, how to render JavaScript, loop over many URLs, and feed results to an AI Agent node.
- [How to build an LLM training dataset by scraping the web](https://clawengine.ai/blog/how-to-build-an-llm-training-dataset): the pipeline is mostly cleaning and deduplication rather than fetching, covering source selection, provenance, near-duplicate removal, quality filters and licensing.
- [GDPR and web scraping: what US companies need to know](https://clawengine.ai/blog/gdpr-and-web-scraping-for-us-companies): when the GDPR reaches a US company with no EU office, why "publicly available" is not an exemption, the legal-basis problem for scraped personal data, and how narrow extraction shrinks exposure.
- [CCPA and web scraping: a 2026 guide for US companies](https://clawengine.ai/blog/ccpa-and-web-scraping-for-us-companies): California's "publicly available" carve-out is narrower than it looks, the 2026 CCPA thresholds ($26,625,000 revenue / 100,000 consumers / 50% of revenue from data), data broker registration, and the DELETE Act DROP deletion requests brokers must process from August 1, 2026.
- [Is scraping government websites legal?](https://clawengine.ai/blog/is-scraping-government-websites-legal): public records start from a stronger position than private sites because they are public by statute, why the CFAA is rarely the live issue after hiQ and Van Buren while breach of contract is, the PACER rules that put federal court records off limits to crawlers ($0.10 per page, automated collection that avoids billing prohibited, harvesting the free endpoints specifically named as misuse), why robots.txt still applies on .gov, and the OPEN Government Data Act route to check first.
- [Is scraping job postings legal?](https://clawengine.ai/blog/is-scraping-job-postings-legal): why the CFAA rarely applies to public postings, how hiQ won on the statute and still paid a $500,000 judgment on contract and tort claims, why employer careers pages and ATS boards beat the big aggregators, and how to keep recruiter personal data out of the pipeline.
- [Do I need permission to scrape a website?](https://clawengine.ai/blog/do-i-need-permission-to-scrape-a-website): the four separate permissions (robots.txt, Terms of Service, copyright and login walls), which ones actually bind you, how to ask a site properly, and a seven-question checklist before your next crawl.
- [Website to API: turn any website into a structured API](https://clawengine.ai/website-to-api): how to give a public site the JSON endpoint it never shipped. Covers checking for a bulk export or an undocumented XHR endpoint first, describing fields as a schema instead of writing selectors, why client-side rendering returns an empty shell to plain HTTP clients, crawling paginated listings inside one call, and the honest scope limit (public no-login pages only, no form filling, no anti-bot defeat, no OCR).
- [Best way to integrate a portal with no API](https://clawengine.ai/blog/best-way-to-integrate-a-portal-with-no-api): the build-versus-buy decision for vertical SaaS teams whose customers run systems that publish nothing callable. Compares in-house scrapers, browser automation and a managed extraction API on time-to-first-integration, ongoing cost and what breaks, explains why a pagination change is more dangerous than an outage because nothing errors, and states when browser automation or an official partner API is the correct choice instead.
- [Zyte alternatives: 11 web scraping APIs compared](https://clawengine.ai/alternatives/zyte): where the Zyte API genuinely wins (proxy unblocking on defended sites, native Scrapy integration through scrapy-zyte-api, Scrapy Cloud hosting) and where flat plans win instead. Covers Zyte's verified August 2026 pricing model: billing per successful request across five difficulty tiers each for HTTP and browser requests, volume discounts tied to a monthly commitment (25% at $100, 40% at $200, 48% at $350, 52% at $500), $5 starting credit on standard accounts and $200 on enterprise, 3,000 requests per minute on standard plans, and the fixed list of automatic extraction page types (article, forumThread, jobPosting, pageContent, product, serp and their list and navigation variants).
- [Migrate a Scrapy spider to a crawl API](https://clawengine.ai/blog/migrate-a-scrapy-spider-to-a-crawl-api): the concept-by-concept mapping from a Scrapy project to one HTTP call. start_urls becomes a seed URL, LinkExtractor and Rule become path rules plus a depth limit, CSS and XPath selectors become a declared schema, and middlewares, AUTOTHROTTLE, the scheduler, the dedup filter and the scrapy-playwright browser fleet disappear entirely while item pipelines stay yours. Includes a before-and-after code pair, which spiders not to migrate (logged-in flows, actively blocked targets, and cases where crawling is the product), the real cost comparison including untracked selector-repair hours, and the three things that measurably change after cutover.
- [Migrate an HTML scraper API to a schema API](https://clawengine.ai/blog/migrate-html-scraper-api-to-schema-api): the mechanical guide to moving off a raw-HTML scraping API. Maps selectors to declared fields, boilerplate stripping and type coercion to schema types, and the per-request credit multiplier to a plain page count, with a before-and-after Python pair. Includes the verified August 2026 credit math (a 40,000 rendered-page month is 200,000 credits on ScrapingBee against 440,000 on ScraperAPI) and the three categories not to migrate: logged-in or form-driven flows, targets defended by Cloudflare, DataDome or PerimeterX, and per-site vendor endpoints such as Amazon or search results.
- [Firecrawl vs Tavily: crawl API vs search API](https://clawengine.ai/firecrawl-vs-tavily): why these two get shortlisted together and why they meter different verbs. Firecrawl is crawl-first at 1 credit per page (Free 1,000 credits, Hobby $19/5,000, Standard $99/100,000, Growth $399/500,000, Scale $749/1M, lower billed yearly), Tavily is search-first at 1 credit per basic search with extraction billed 5 URLs per credit and credits sold pay as you go at $0.008. Converts both to cost per 1,000 crawled pages (about $0.99 on Firecrawl Standard against roughly $2.40 on Tavily), shows why the free tiers rank the opposite way (1,000 Tavily credits stretch to about 3,300 crawled pages), and states plainly that ClawEngine runs no search index. Verified August 2026.
- [Migrate Tavily Extract to a crawl API](https://clawengine.ai/blog/migrate-tavily-extract-to-a-crawl-api): the volume at which batch-priced extraction stops being the cheap option. Covers the verified credit math (a 1,000 page Tavily crawl is 100 mapping credits plus 200 extraction credits, about $2.40, against roughly $0.99 per 1,000 pages on a bundled per-page plan), a concept-by-concept mapping from URL lists to a start URL plus path prefix, a before-and-after code pair, and the three cases not to migrate: genuine discovery workloads, under a few hundred pages a month, and any pipeline that depends on the search index itself.
- [ZenRows vs ScrapingBee: pricing and credits compared](https://clawengine.ai/zenrows-vs-scrapingbee): the closest-priced pair in the proxy plus rendering category, and why the credit tables agreeing makes the choice harder rather than easier. Both charge 1 credit for a plain fetch, 5 for a JavaScript-rendered page, 10 for a premium proxy and 25 for premium with rendering, and both sell 250,000 credits at the mid tier (ZenRows Launch $57 billed yearly or $69 monthly, ScrapingBee Freelance $49, 50 concurrent requests each). ScrapingBee is cheaper per rendered page at every paid tier, from about $1.27 per 1,000 on Hobby down to $0.37 on Business+, against $1.78 to $0.46 at ZenRows annual rates. The ranking reverses on defended sites: ScrapingBee escalates hard targets to a 75-credit stealth proxy (about $14.70 per 1,000 pages) while ZenRows never exceeds 25 credits (about $5.70 per 1,000), roughly 2.6 times cheaper. Verified from both vendors in September 2026.
- [Render quality benchmarks for crawl APIs](https://clawengine.ai/blog/crawl-api-render-quality-benchmarks): how to compare web crawling APIs on the two axes no pricing page shows. Render quality is whether the returned HTML contains what a browser would display, tested against hydration timing, lazy-loaded lists, infinite scroll, tabbed content, client-side routing and blocked subresources. Extraction quality is whether the text is the article without the navigation, with tables, code blocks and heading hierarchy intact. Includes a 10 check pass-or-fail scoring rubric across both axes, a repeatable trial protocol using ten frozen URLs of your own, why price does not correlate with render quality, and what it means when every vendor fails the same check.
- [Firecrawl vs Browserbase: pricing per page](https://clawengine.ai/firecrawl-vs-browserbase): two products that get shortlisted together for AI agent work and meter completely different things. Firecrawl is an extraction API billing 1 credit per scraped or crawled page (Free 1,000 credits at 2 concurrent, Hobby $16/5,000, Standard $83/100,000, Growth $333/500,000, Scale $599/1M, billed yearly), with structured JSON output adding 4 credits per page so a typed page costs 5 credits, not 1. Browserbase is cloud Chromium billing wall-clock browser hours (Free 1 hour/3 concurrent/15 minute cap, Developer $20 for 100 hours then $0.12/hr, Startup $99 for 500 hours then $0.10/hr, plus proxy bandwidth at $12 and $10 per GB), returning a CDP endpoint you drive with Playwright, Puppeteer, Selenium or Stagehand. The two meters cross at exactly 15 seconds of browser time per page: both come to $0.83 per 1,000 pages. Below that browser hours are cheaper ($0.33 per 1,000 at 6 seconds), above it page credits are cheaper ($3.30 per 1,000 at 60 seconds), and idle time waiting on a model bills at the full rate. Verified from both vendors in August 2026.
- [Cut agent browser hours with a crawl API](https://clawengine.ai/blog/cut-agent-browser-hours-with-a-crawl-api): why agent loops built on a cloud browser overpay for the read steps that inherited a Chromium session they never needed. Covers the verified cost-per-1,000-pages table by seconds of browser time (about $0.33 at 6s, $0.83 at 15s, $1.65 at 30s, $3.30 at 60s, $6.60 at 120s on Browserbase Startup rates), a sorting table splitting agent steps into reads and actions, the concept mapping from a browser session lifecycle to one HTTP request, the prompt-injection risk that arrives with scraped content regardless of meter, and the three cases where migrating loses money: fast pages with packed concurrency, reads that only exist inside an authenticated session, and volumes under a few thousand pages a month.
- [Parallel AI alternatives: 11 web scraping APIs compared](https://clawengine.ai/alternatives/parallel-ai): where Parallel Web Systems genuinely wins (a web-scale pre-built index, 250ms to 3s Search latency, asynchronous deep research up to two hours, FindAll list building, Entity Search, and connectors for MCP, LangChain, n8n, Zapier, Google Sheets, Snowflake and BigQuery) and the three gaps that send teams looking. Parallel Extract lists at $1.00 per 1,000 URLs, but its documented default returns focused excerpts from the cached index rather than a live fetch, full page content is disabled unless requested, the minimum indexed-content age you can request is 600 seconds, and live fetch adds significant latency under a separate rate limit. Parallel also has no crawl endpoint, so covering a domain means discovering its URLs first. ClawEngine fetches live on every call, returns the whole rendered document typed to a schema you declare, and crawls from a seed URL, at $0.78 per 1,000 pages on Hobby and $0.27 on Scale. Verified from Parallel documentation in August 2026.
- [Parallel AI API pricing: cost per 1,000 requests](https://clawengine.ai/blog/parallel-ai-api-pricing): the full published rate schedule for all eight Parallel APIs and what each line actually buys. Extract $1.00 per 1,000 URLs, Search $1.00 per 1,000 turbo or fast requests and $5.00 basic or advanced (10 results each, extra results $1 per 1,000), Task $5 to $2,400 per 1,000 runs across the lite to ultra8x processor ladder, Responses $10, $50 or $250 per 1,000 by reasoning effort, Monitor $3 or $10, Entity Search $5, Chat $5 to $25, FindAll $0.10 to $10.00 fixed plus $0 to $1.00 per match. Includes rate limits (Search and Extract 600/min, Tasks 2,000/min, Chat and Monitor 300/min, FindAll 300/hour), the fact that Task and Responses bill only successful runs and price per run rather than per output field, and three worked cost scenarios covering an agent doing lookups, enriching 20,000 companies, and keeping a 100,000 page documentation corpus current. Verified August 2026.
- [Scrapfly alternatives: 11 web scraping APIs compared](https://clawengine.ai/alternatives/scrapfly): where Scrapfly genuinely wins (its Anti Scraping Protection layer, a residential proxy pool, no billing on failed scrapes, a free tier, a screenshot product and a cloud browser) and where a flat per-page crawl API costs less. Scrapfly meters API credits: 1 for plain HTTP on a datacenter proxy, 6 once JavaScript rendering is on, 25 on residential, 30 for both, with binary downloads at 3 credits per 100KB on datacenter and 10 on residential after the first free megabyte. Tiers are Free 1,000 credits at 5 concurrency, Discovery $30 for 200,000 at 5, Pro $100 for 1,000,000 at 20, Startup $250 for 2,500,000 at 50 and Enterprise $500 for 5,500,000 at 100, with 16 percent off annually. Pro and Startup are the same $0.10 per 1,000 credits, so moving up buys concurrency and a better overage rate rather than cheaper pages. Rendered pages cost about $0.90 per 1,000 on Discovery, $0.60 on Pro and Startup and $0.55 on Enterprise, against ClawEngine at $0.78 on Hobby, $0.40 on Startup and $0.27 on Scale with rendering included, so the break-even is roughly 60 percent JavaScript pages at the mid tier. The Scrapfly Crawler API returns gzipped WARC or HAR archives rather than markdown or JSON. Verified from Scrapfly documentation in August 2026.
- [Scrapfly pricing: cost per 1,000 pages](https://clawengine.ai/blog/scrapfly-pricing): the five Scrapfly credit tiers, the per-request credit multipliers, and the arithmetic that turns them into a cost per 1,000 pages. Plain HTML runs $0.15 per 1,000 on Discovery down to $0.09 on Enterprise; rendered pages run $0.90 down to $0.55; residential plus rendered runs $4.50 down to $2.73. Overage is billed at $5.00, $3.50, $2.00 and $1.20 per 10,000 credits, which is 3.3, 3.5, 2.0 and 1.3 times the committed plan rate, and the Free and Discovery tiers hard cap instead of spilling over. Failed scrapes are not billed unless more than 30 percent of traffic fails within one hour, and Anti Scraping Protection is free on pages that are not blocked but can silently upgrade a 1 credit request to a residential pool. Includes the break-even share of JavaScript pages against flat per-page pricing (84 percent at entry tier, 60 percent at mid, 39 percent at high volume) and a 200 URL test protocol using the X-Scrapfly-Api-Cost header. Verified August 2026.
- [llms.txt generator: build llms.txt and llms-full.txt from a live crawl](https://clawengine.ai/llms-txt-generator): how the two files differ (llms.txt is a curated markdown index of links plus one-line notes, llms-full.txt inlines the full markdown body of every listed page so an agent loads the whole corpus in one fetch) and how to generate both from a crawl of a live docs tree, including the per-page .md companion files the spec recommends. States plainly that Google Search Central says Google Search ignores llms.txt and that creating one will neither help nor harm rankings, so the payoff is with agent frameworks and your own retrieval stack rather than SEO. The spec was published by Jeremy Howard on 3 September 2024, with version 2 dated 10 August 2026. Also says when you do not need a tool at all: a small site should hand-write the index, and a repository that already holds clean markdown should generate both files from source at build time.
- [Mintlify pricing: docs hosting and llms.txt](https://clawengine.ai/blog/mintlify-pricing): Mintlify runs Starter at $0 with 5 editor seats (custom domain, web editor, authentication, MCP server, API playground), Pro at $450 a month with unlimited editors, the docs assistant and agent, automations, preview deployments, admin APIs and 10,000 AI credits with overage at $0.01 per credit, and Enterprise on quote with SSO, SCIM and RBAC. The finding buyers miss is that llms.txt and llms-full.txt are generated automatically on every project including the free tier, mirrored at the /.well-known/ paths, with page links carrying a .md extension and descriptions truncated to 300 characters, so the $450 buys seats and AI features rather than the files. Large sites get a split index under /_llms/ once the file would exceed 100,000 characters. Compares four routes to the same outcome: Mintlify Starter at $0, Pro at $450, generating from your own markdown source at build time for $0, and crawling a live site from $39. Verified from Mintlify pricing and documentation in September 2026.
- [Web scraping MCP server: wire a crawl tool into Claude, Cursor and Codex](https://clawengine.ai/web-scraping-mcp-server): what a web scraping MCP server actually is, which four real ones exist in September 2026, and what each charges per page. Firecrawl MCP is hosted at mcp.firecrawl.dev/v2/mcp with about fifteen tools, 1,000 free credits a month and a plain page at roughly $0.83 per 1,000 on the $83 Standard plan, rising to $4.15 per 1,000 once the JSON format is on. Bright Data Web MCP gives 5,000 free requests a month in Rapid mode with two tools, 69 tools in Pro mode, $1.50 per 1,000 requests pay as you go and $499 a month for 383,000, with stealth browsers at $8 per GB. Apify MCP runs at mcp.apify.com over Streamable HTTP and OAuth or locally through npx actors-mcp-server, priced by whichever Actor the agent calls. Playwright MCP is free and Apache 2.0 but has no crawl concept, and its own README recommends the Playwright CLI with skills for high-throughput agents because verbose accessibility trees burn tokens. ClawEngine publishes no official MCP server and says so; it gives a forty-line Python server over its crawl API instead, using MCPServer from mcp.server with the page ceiling clamped in the tool signature so the model cannot raise it. The MCP specification revision is 2026-07-28, which standardizes on stdio and Streamable HTTP and drops standalone SSE. The structural point: once an agent picks the tool calls, the tool menu becomes a spending control.
- [Firecrawl self hosted vs cloud: what the open source Docker stack costs](https://clawengine.ai/firecrawl-self-hosted): what self hosting Firecrawl actually asks of you, costed honestly. The Docker Compose stack is five services, not one: the API on port 3002, PostgreSQL as the queue backend, Redis, RabbitMQ and a Playwright rendering service. Firecrawl own documentation states that the quickstart disables API authentication (USE_DB_AUTHENTICATION=false, trusted network only), starts without durable storage, TLS or high availability, and that no single environment switch makes the stack production ready, and that you own upgrades, secrets, storage, monitoring, recovery and incident response. Screenshots and page actions are not available in the default stack; Fire Engine anti-bot is a separate service; agent and browser modes, interact and feedback, and audio and video formats all point to Firecrawl Cloud. The license point commercial teams miss: the core is AGPL-3.0 (SDKs and some UI components are MIT), and AGPL-3.0 reaches across a network boundary, so a modified copy offered as a service is expected to publish those modifications. Cloud comparison at 100,000 pages a month: Firecrawl Standard $83 for 100,000 credits at about $0.83 per 1,000 plain pages, rising to about $4.15 per 1,000 when the JSON format adds 4 credits a page, against ClawEngine Startup $99 for about 250,000 pages at about $0.40 per 1,000 with typed JSON at no surcharge. Says plainly that ClawEngine has no open source version and no Docker image, and that on-prem is an Enterprise agreement rather than a download. Verified from the Firecrawl self-host documentation and GitHub README in September 2026.
- [Apify pricing: real cost per 1,000 pages](https://clawengine.ai/blog/apify-pricing): Apify plans are Free $0 with $5 of platform credit, Starter $19, Scale $199 and Business $999, and on every tier the included usage credit equals the plan price, so the subscription is a prepaid wallet rather than a page allowance. Compute unit rates are $0.20, $0.20, $0.16 and $0.13, max RAM 16, 64, 256 and 512 GB, concurrent runs 5, 32, 128 and 256, with 10 percent off annual billing and no rollover of unused credit. The structural finding: one compute unit is one gigabyte of RAM running for one hour, so the meter is Actor runtime rather than pages fetched, and Apify cannot publish a cost per page because the number is a property of somebody else Actor. Derived at the $0.20 rate, a lean 1 GB HTTP Actor at 14,400 pages an hour works out at about $0.014 per 1,000 pages while a 8 GB browser Actor at 10 seconds a page works out at about $4.44, a spread of roughly 317 times on one plan. Actor charges sit on top through pay per usage, pay per event and monthly rental models, all deducted from prepaid credit. Proxies are separate: residential $8 to $7 per GB, datacenter overage $1 to $0.60 per IP, SERP $2.50 to $1.70 per 1,000 queries, Unblocker $1.50 to $1 per 1,000 requests, extra concurrent runs $5 each, extra RAM $1 per GB. Recommends benchmarking a candidate Actor on 500 URLs on the free tier and deriving your own figure. Verified from apify.com/pricing and the Apify platform documentation in September 2026.
- [Firecrawl MCP pricing: cost per tool call](https://clawengine.ai/blog/firecrawl-mcp-pricing): the full per-tool credit table for the Firecrawl MCP server, mapped from tool name to endpoint cost. firecrawl_scrape and firecrawl_crawl are 1 credit a page, the JSON format adds 4 so a typed page is 5, firecrawl_map is 1 credit a call, firecrawl_search is 2 credits per 10 results, firecrawl_parse adds 1 per PDF page, firecrawl_interact bills 2 credits a browser minute for code-only sessions and 7 if any call uses a prompt, and a post pulled from X through Grok costs 1 plus 29 for 30 credits, or 34 with JSON. The prompt injection check adds 4 credits a page and Zero Data Retention adds 1. Tiers on annual billing are Free $0 for 1,000 credits, Hobby $16 for 5,000, Standard $83 for 100,000, Growth $333 for 500,000 and Scale $599 for 1,000,000, so the credit rate improves about 39 percent from Hobby to Scale while typed output stays a flat 5x multiplier at every tier. Includes the per-minute rate limits by plan, the fact that batch and crawl jobs bill asynchronously as pages complete so the dashboard lags the work, a worked seven-call agent loop that costs 12 credits for one question, and four concrete controls. Verified from Firecrawl billing, rate limit and pricing documentation in September 2026.
- [CrewAI web scraping tool: hosted scraper API for agent crews](https://clawengine.ai/crewai-web-scraping-api): CrewAI documents fourteen tools in its web scraping category and eleven of them require an API key from another vendor (Firecrawl scrape, crawl and search, ScrapFly, ScrapeGraph, Spider, Browserbase, Hyperbrowser, Stagehand, Oxylabs and Bright Data). Only three run without one: ScrapeWebsiteTool (requests plus BeautifulSoup with a 15 second timeout, one URL, no JavaScript, returns the whole page text with a fixed prefix), ScrapeElementFromWebsiteTool (the same fetch narrowed to a CSS selector) and SeleniumScrapingTool (headless Chrome you install yourself, one CSS selector, a default three second wait, documented as still in development). Full table of which tool renders JavaScript, which crawls a whole site, what the agent receives and which needs a paid account. The cost mechanism buyers miss: tool output stays in the message history and is re-sent as input tokens on every remaining model call in the task, so a raw page dump is billed repeatedly, which is why returning clean markdown usually saves more than switching to a smaller model. Includes two complete CrewAI BaseTool subclasses, one for rendering a single page to markdown and one for crawling a path prefix and returning every page in a single result. Says plainly that ClawEngine does not defeat anti-bot systems, log in or fill forms, and points to Browserbase, Stagehand and Bright Data for that. Verified from the CrewAI tools documentation and the crewAI-tools source in September 2026.
- [CrewAI pricing: what it actually costs](https://clawengine.ai/blog/crewai-pricing): read firsthand from crewai.com/pricing on 4 September 2026, CrewAI publishes exactly two plans. Basic is free with 50 workflow executions a month, the visual editor, the AI copilot and GitHub integration. Enterprise is custom priced and requires a trial request; it covers SSO, RBAC, workload identity, PII redaction and policies, deployment on CrewAI cloud or your own VPC or your own infrastructure, a 45 day onboarding, and forward deployed engineering a la carte. The $25 a month Professional tier with 100 executions and $0.50 overage that most of the first page of Google still describes is not on the vendor page, and CrewAI publishes no Enterprise figure at all, so any specific annual number in a directory is an unverified estimate. A workflow execution is one run of a deployed crew, not one agent action, so 50 a month is an evaluation allowance rather than a production tier. Sets out the three separate bills a deployment generates (CrewAI platform, LLM tokens, and whichever scraping vendor fills the tool slot) and explains why the token bill usually dominates: tool output stays in the message history and is re-sent on every subsequent model call. Ends with a measurement protocol for costing your own crew in an afternoon rather than modelling it from published rates.
- [Olostep alternatives: pricing and 11 crawl APIs compared](https://clawengine.ai/alternatives/olostep): Olostep is a web data API for AI agents selling scrape, crawl, batch and search endpoints, with JavaScript rendering and residential IP addresses included on every plan including the free one. Published tiers verified from olostep.com in September 2026: free trial 500 credits, Starter $9 for 5,000, Standard $99 for 200,000, Scale $399 for 1,000,000 and a custom Enterprise tier, which is $1.80, $0.495 and $0.399 per 1,000 credits on the three paid plans. Top-up packs of 10,000 for $20, 250,000 for $200 and 2,000,000 for $1,000 are valid six months, so $2.00, $0.80 and $0.50 per 1,000, roughly a quarter more than the equivalent subscription rate in exchange for credits that do not reset monthly. The structural finding: every cost-per-1,000 figure for Olostep assumes one credit per request, and Olostep publishes no per-request-type credit table on its pricing page or in the scrape endpoint reference, which documents the credits consumed and the estimated cost in USD as fields populated after execution completes. So the headline rate is a floor rather than a forecast, and the fix is to run a hundred of your own URLs and read the credits field. Says plainly that Olostep wins on entry price, on the free tier, on bundled residential IPs, on six-month credit packs and on having a search endpoint, all of which ClawEngine lacks. Compared against ClawEngine at $0.78, $0.40 and $0.27 per 1,000 pages on published plans not yet open for purchase.
- [ScraperAPI pricing and cost per 1,000 pages](https://clawengine.ai/blog/scraperapi-pricing): verified from scraperapi.com/pricing and the ScraperAPI credits documentation on 6 September 2026. ScraperAPI now publishes six self-serve tiers plus quoted Enterprise, two more than most comparisons list: Hobby $49 for 100,000 credits at 20 threads, Startup $149 for 1,000,000 at 50, Business $299 for 3,000,000 at 100, Scaling $475 for 5,000,000 at 200, Professional $975 for 10,500,000 at 300 and Advanced $1,975 for 21,500,000 at 500, all 10 percent cheaper annually, plus a standing free plan of 1,000 credits at 5 concurrent connections and a 7-day 5,000-credit trial. Full multiplier table: standard page 1 credit, render 10, premium 10, screenshot 10, premium with render 25, ultra premium 30, ultra premium with render 75, Amazon, Walmart and eBay 5, Google and Bing 25, LinkedIn 30, and clearing Cloudflare, Turnstile, DataDome or PerimeterX adds 10, while wait_for_selector, country_code, session_number, device_type, output_format, keep_headers and autoparse cost nothing extra. Only 200 and 404 responses are billed, plus requests cancelled before 70 seconds. The finding buyers miss: pay-as-you-go overage exists only on Scaling, Professional, Advanced and Enterprise, so hitting 100 percent of credits on Hobby, Startup or Business stops the pipeline until you upgrade, and unused credits never roll over, which together force sizing on your peak month. Derived cost per 1,000 pages on every tier for plain, rendered, premium rendered and ultra rendered pages, from $0.09 to $36.75.
- [Oxylabs pricing: cost per 1,000 results](https://clawengine.ai/oxylabs-pricing): every published Oxylabs rate read firsthand from oxylabs.io pricing pages and developers.oxylabs.io on 9 September 2026. Web Scraper API is a monthly result allowance: free trial up to 2,000 results, Micro $49 up to 98,000, Starter $99 up to 220,000, Advanced $249 up to 622,500, Business $999 up to 3,330,000 and Custom by quote, with rate limits of 50 requests a second on Micro and Starter and 100 above. Rates are set by target category, roughly $0.25 to $0.50 per 1,000 for Amazon, $0.50 to $1.00 for Google, $0.70 to $1.15 for other sources and $0.95 to $1.35 when JavaScript rendering is required, which Oxylabs documentation calls feature-based billing. Residential proxies are a separate purchase at $30 for 5GB, $100 for 20GB, $500 for 125GB and $2,500 for 1TB, so $6.00, $5.00, $4.00 and $2.50 per GB, with top-ups to 100GB on the two smaller plans and 2TB on the two larger. Datacenter proxies start at $0.70 per IP, Web Unblocker is $3 per GB and the headless browser starts at $4.70 per GB. Two findings buyers miss: the headline result count is quoted against the cheapest target category, and Oxylabs own maximum-results table shows the same free trial buying 2,000 Amazon results but only 769 JavaScript-rendered results from an ordinary site, a 2.6 times spread that carries up the whole ladder; and the billing documentation states that results with 2xx or 4xx status codes are counted as successful, so a 404 on a dead link or a 403 from a site that blocked you is billed at the normal rate even though the product is marketed as success-based pricing. Says plainly that Oxylabs wins on defended targets, on residential proxy scale and on search engine results, none of which ClawEngine offers.
- [Residential proxy cost vs crawl API pricing](https://clawengine.ai/blog/residential-proxy-cost-vs-crawl-api-pricing): proxies bill gigabytes and crawl APIs bill pages, so the two prices are not comparable until you measure average page weight. Conversion table for 1,000 pages at $6.00 and $2.50 per GB across five page weights: 100KB plain HTML is $0.57 and $0.24, 500KB is $2.86 and $1.19, 1MB is $5.86 and $2.44, 2MB full browser load is $11.72 and $4.88, 5MB image-heavy pages are $29.30 and $12.21. The structural point: moving from $6.00 to $2.50 per GB by committing to a much larger plan saves 58 percent, while blocking image and font requests on a headless browser saves closer to 80 percent on the plan you already have, so fetch configuration matters more than vendor choice. Three things push real bandwidth past the estimate: retries are billed traffic, blocked responses still consume bytes and some result-metered vendors count any 2xx or 4xx as a billable success, and an uninstrumented headless browser buys the hero image, icon font, chat widget and tag manager on every page. Says plainly that residential proxies are the correct purchase for marketplaces, ticketing, travel inventory, search engine results and anything behind aggressive bot management, and for geographic targeting that needs real consumer connections. Gives a four-step audit: sort targets by whether they fight back using a plain HTTP client, measure transferred bytes rather than HTML source size, separate rendering problems from unblocking problems, and check rollover, overage and whether failed responses bill. Recommends routing by target rather than picking one vendor.
- [Firecrawl pricing plans, credits and cost per page](https://clawengine.ai/firecrawl-pricing): every published Firecrawl rate read firsthand from firecrawl.dev/pricing and docs.firecrawl.dev/billing on 10 September 2026. Free 1,000 credits a month; Hobby $16, Standard $83, Growth $333, Scale $599 billed yearly ($19, $99, $399, $749 monthly) for 5,000, 100,000, 500,000 and 1,000,000 credits. 1 credit per plain page ($0.83 per 1,000 on Standard), but the JSON format adds 4 credits so a typed page is 5 ($4.15 per 1,000). A 403 or 404 page still costs 1 credit; only a request returning no document is free. Pay-as-you-go top-ups are $5 per 1,000 credits on Hobby, $2.50 Standard, $2.00 Growth, $1.00 Scale, up to 3x the in-plan rate. Credits reset monthly except Scale (one month rollover).
- [Bright Data pricing: cost per 1,000 pages](https://clawengine.ai/blog/bright-data-pricing): every Bright Data scraping product priced firsthand on 10 September 2026. Web Scraper API, Web Unlocker API and SERP API cost $1.50 per 1,000 successful records or requests pay-as-you-go, or $499 a month for about 380,000 with extras at $1.30, each with 5,000 free a month. Crawl API falls to $1.10 on $999 (900,000) and $1.00 on $1,999 (2M). Browser API and residential proxies bill $8 per GB pay-as-you-go, so a 2MB page costs about $15.62 per 1,000 in the Browser API against $1.50 through Web Unlocker, roughly 10x. Plans carry a minimum monthly commitment billed from the 1st and unused volume does not roll over.