The ClawEngine blog
Web scraping, made practical
Practical writing on turning websites into clean, LLM-ready data: how to crawl for training data, extract structured fields for RAG, render JavaScript pages, decide between an API and your own scraper, and crawl compliantly with robots.txt. Public and permitted data only.
Migrate a Scrapy Spider to a Crawl API
A Scrapy project is mostly scaffolding, and the scaffolding is what disappears. Here is the concept-by-concept mapping from start_urls, Rule and selectors to a seed URL, path rules and a declared schema, plus the spiders you should not move at all.
Best Way to Integrate a Portal With No API
Vertical SaaS teams keep hitting the same wall: the system their customers run publishes nothing you can call. Here is how in-house scrapers, browser automation and a managed extraction API actually compare once you price the maintenance.
How to Limit a Crawl to the Pages You Need
Runaway crawls are a scope problem, not a crawler defect. Here is how seed URL, path rules, depth and page budget work together, what depth to set for each kind of site, and why a page limit alone never saves you.
Web Data API for AI: Search vs Scraping API
Neural search APIs and crawl-and-extract APIs both return clean text, and they fail in opposite ways. Here is how to tell which one your AI app needs, what each really costs, and how to run both without paying twice.
Web Scraping API for AI Agents vs Browser Automation
Most agent web access is reading, not clicking, and the two need different tools. Here is how to split the read path from the act path, what each vendor is actually for, and what the routing saves you.
Jina Reader vs Firecrawl: Which Turns URLs Into Markdown
Reader converts one URL to markdown for a fraction of a cent. Firecrawl crawls a whole site and finds the URLs for you. Here is the real cost math, where each one wins, and why comparing them on price per page gives you the wrong answer.
Scrapy JavaScript Rendering: Playwright, Splash or an API
Scrapy fetches raw HTTP, so a client-rendered page arrives as an empty shell. Here are the three ways to fix that, the check worth running before you install any of them, and what rendering really costs on each route.
How to Scrape a Table From a Website Into JSON or CSV
Four routes, from a one-line pandas call to a schema-based API, and how to pick between them. The deciding factors are whether the table is drawn by JavaScript, how many pages it spans, and whether the job has to keep working in six months.
Web Scraping API Credits Explained: Why Your Bill Is Higher
A credit is a billing unit the vendor defines, not a page. Blocked pages retrying on a premium proxy, extraction surcharges and expiring allowances are what turn a tidy plan price into a bill nobody forecast. Here is how to price your own target list first.
Do I Need a Proxy Network to Scrape a Website?
Most scraping jobs do not need proxies. Proxy networks solve anti-bot blocking on a narrow set of high-value targets, and buying one before you have measured a block rate is the fastest way to overspend on a scraping project.
JavaScript Rendering API Comparison: What JS Rendering Costs
Every major web scraping API renders JavaScript in 2026, so rendering is no longer the thing that separates them. Price is. A rendered page costs 10 credits against 1 on ScraperAPI and 5 against 1 on ScrapingBee, while Firecrawl and ClawEngine charge no render multiplier at all. All eleven tools compared on what rendering actually costs.
Is Scraping Job Postings Legal?
Scraping public job postings is broadly lawful in the US, but the risk is not where most teams look. Why the CFAA rarely applies, why hiQ won on the statute and still paid $500,000, why employer careers pages beat the big boards, and how to keep recruiter personal data out of your pipeline.
Do I Need Permission to Scrape a Website?
You do not need permission to read pages a site publishes publicly, but there are four separate permissions and most people only check one. What robots.txt, Terms of Service, copyright and login walls each decide, how to ask a site properly, and a seven-question checklist before your next crawl.
How to Scrape a Website With n8n
To scrape a website with n8n, call a scraping API from a single HTTP Request node instead of running a headless browser on your instance. The full workflow, how to render JavaScript, how to loop over many URLs, and how to feed the results to a database or an AI Agent node.
How to Build an LLM Training Dataset by Scraping the Web
Building an LLM training dataset from the web is mostly cleaning and deduplication, not fetching. The full pipeline: choosing permitted sources, crawling to clean markdown, keeping provenance, removing near-duplicates, filtering quality, and the licensing questions that decide what belongs in a commercial corpus.
CCPA and Web Scraping: A 2026 Guide for US Companies
The CCPA applies to scraped personal data about California residents once your company crosses one of three thresholds. Public does not mean exempt.
Is Scraping Government Websites Legal? A 2026 US Guide
Public records on public government pages are broadly lawful to collect in the US. The limits that actually bite are Terms of Use, robots.txt, metered systems like PACER, and privacy law once records describe people. What each one means in practice.
GDPR and Web Scraping: What US Companies Need to Know
A US company can be bound by the GDPR when it scrapes personal data from EU residents, even with no office in Europe. When it applies, why "publicly available" is not an exemption, the legal-basis problem for scraped data, and how to shrink your exposure by extracting narrowly.
How to Scrape Documentation for a RAG Chatbot
To scrape documentation for a RAG chatbot, crawl the docs tree to clean markdown with the nav and sidebars stripped and code blocks intact, chunk on headings, and keep the source URL on every chunk. The full pipeline, why fixed-size chunking fails on docs, and how to re-crawl incrementally.
Best Web Scraping API for Documentation Sites in 2026
For documentation specifically, the best web scraping API renders JavaScript, crawls a scoped docs tree, and returns markdown with fenced code blocks intact. An honest look at Firecrawl, Crawl4AI, ClawEngine, Apify and ScrapeGraphAI, with August 2026 pricing and a real recommendation.
How to Scrape Real Estate Listings Legally
Scraping public property listings is broadly lawful in the United States, but Terms of Service, robots.txt, copyright in photos and descriptions, MLS display rules and state privacy law all apply. Which sources are durable to build on, and what to collect versus what to leave behind.
Best Web Scraping API for Lead Generation in 2026
The best web scraping API for lead generation is the one that turns public company pages into clean, typed firmographic data your CRM can import, on permitted sources only. An honest look at the main options, what to weigh, and how to build an enrichment pipeline that stays compliant.
How to Scrape Job Postings With an API
To scrape job postings with an API, crawl a public board or career page, extract each posting into a fixed schema, and key every record by a stable ID so you can deduplicate and refresh. The full pipeline, the fields to collect, and how to keep a job feed compliant.
Is Scraping Public Data for Lead Generation Legal?
Scraping publicly available business data for lead generation is broadly lawful in the United States, but the constraints are Terms of Service, robots.txt and privacy law once personal data is involved. What the CFAA rulings actually say, where the CCPA changes the calculus, and how to keep a prospecting pipeline defensible.
How to Monitor Competitor Prices With an API
To monitor competitor prices with an API, extract each product page on a schedule, pull price and stock as typed fields, store every reading with a timestamp, and diff against the last. The full pipeline, the schema to collect, how often to check, and the mistakes that corrupt a price dataset.
How to Build a Media Monitoring Feed From News Sites
Build a media monitoring feed by crawling section fronts, extracting each article into clean text with headline, author and date, then filtering for the brands and topics you track. Sourcing, structuring, deduplicating wire stories, and turning the stream into alerts.
Product Data API vs Web Scraping: Which Do You Need?
A product data API returns clean, typed fields in one call; raw web scraping returns HTML you render, parse and maintain yourself. When each makes sense, the honest trade-offs, and why maintenance, not the fetch, is where the real cost lives.
Deduplicate Pages During a Crawl for RAG Ingestion
The best way to deduplicate pages during a crawl is layered, cheapest filter first: normalize URLs before you fetch, hash the extracted text for exact repeats, then SimHash or MinHash for near duplicates, and dedupe again at the chunk level. Real code for each layer.
LLM Web Scraping in Python: A Practical Guide
How to scrape websites for an LLM in Python, from requests and BeautifulSoup to Playwright, model-driven extraction and a managed API. Real code, honest costs, and when an LLM is the wrong tool for the job.
What Is a Crawl Agent? Three Meanings, Explained
A crawl agent means three different things: an AI agent that crawls the web for itself, the user-agent string a crawler declares, and the worker process in a distributed crawler. Here is what each one is, and which you actually need.
Best Web Scraping API for RAG and AI Agents (2026)
The best web scraping API for RAG and AI agents returns rendered, boilerplate-free, structured content your model can use with no cleaning stage. An honest comparison of the tools teams actually shortlist, and how to pick for your job.
How to Get Website Data Into a RAG Knowledge Base
A step-by-step guide to feeding web pages into a RAG knowledge base: scrape with rendering, clean to markdown, chunk on real headings, embed and store, with the exact LangChain, LlamaIndex and AnythingLLM ingestion code.
Crawl4AI vs Firecrawl: Which Should You Use in 2026?
Crawl4AI vs Firecrawl compared on cost, rendering, extraction and who runs the infrastructure. Both are good, and the decision is really about whether you want to operate a crawler or ship a pipeline.
What Is LLM-Ready Content? Why Raw HTML Wrecks RAG
LLM-ready content is rendered, stripped, structured and traceable, so a model can use it with no cleaning stage. Here is what that means in practice, and why fixing the input beats tuning the model.
How to Crawl a JavaScript Website (Without an Empty Page)
How to crawl JavaScript websites: render the page before you read it. The three practical approaches compared, what breaks past the first page, and why you should not render everything.
How Much Does a Web Scraping API Cost? 2026 Pricing Compared
Web scraping API pricing compared for 2026: entry plans run $16 to $49 a month, but protected pages, expiring credits and the hidden cleaning stage decide what you actually pay. Real numbers from eight tools.
Firecrawl vs Bright Data vs Apify: Which Web Scraping API for LLM Data?
Firecrawl vs Bright Data vs Apify, compared honestly on output quality, anti-bot strength, prebuilt scrapers and price. They are built for three different jobs, and buying the wrong one is the usual mistake.
What Is an AI Web Crawler? How It Differs From a Traditional Crawler
An AI web crawler renders JavaScript, strips boilerplate and returns clean markdown or typed JSON a model can actually use. Here is how it differs from a search-engine crawler, and why raw HTML wrecks RAG retrieval.
How to Crawl a Website for LLM Training Data
How to crawl a website for LLM training data the clean way: discover URLs, render pages, strip boilerplate, and export tidy markdown that is ready to chunk, embed and train on. Public and permitted data only.
Web Scraping API vs Building Your Own: An Honest Cost Breakdown
Web scraping API vs building your own scraper: a clear-eyed comparison of engineering time, proxy and headless-browser ops, maintenance, and total cost, so you can decide what to own and what to buy.
Structured Data Extraction for RAG: From Web Pages to Typed JSON
Structured data extraction for RAG: define a schema, pull typed JSON straight from web pages, and feed your retrieval pipeline clean fields instead of messy HTML. Better chunks, better retrieval, fewer hallucinations.
How to Render JavaScript Pages When Scraping (2026 Guide)
To render JavaScript pages when scraping, load each URL in a headless browser or send it to a rendering API that runs the page scripts and returns the finished content as clean markdown or JSON.
Is Web Scraping Legal? robots.txt and US Court Rulings 2026
Short answer: scraping public data is generally legal in the US. The real limits are robots.txt, Terms of Service, copyright and privacy law. What US courts have actually held, plus crawl-delay, compliance basics and public versus private data, in plain English.
MAP Monitoring: How to Detect Minimum Advertised Price Violations
How to run a MAP monitoring program that actually catches violations: build the reseller URL map, extract advertised prices on a schedule, keep timestamped evidence, and tell a real violation apart from an in-cart discount.
Python Web Scraping: requests, Scrapy, or a Scraping API?
An honest comparison of the three ways to scrape with Python: requests plus BeautifulSoup, Scrapy, and a scraping API. What each is genuinely good at, where each breaks, and four questions that decide which one you need.
Is Scraping Competitor Prices Legal? A US Guide for Pricing Teams
Collecting public competitor prices is broadly lawful in the US, but the risk sits in Terms of Service, copyright and antitrust rather than the CFAA. What Van Buren, hiQ and Meta v. Bright Data actually mean for a pricing program.
Is Scraping Data for AI Training Legal? A 2026 US Guide
Two federal courts have now held that training an AI model can be transformative fair use, and one held it was not. What Bartz v. Anthropic, Kadrey v. Meta and Thomson Reuters v. Ross mean in practice, why acquisition rather than training is where the liability sits, and what a defensible corpus manifest looks like.
Can a Website Sue You for Scraping? What US Courts Do
Websites can and do sue scrapers, but the claims that actually win are breach of contract, copyright and trespass to chattels rather than the CFAA. The six claims ranked by how dangerous they really are, what the Ryanair v. Booking.com verdict and its reversal taught us, and how to answer a cease and desist.
Node.js Web Scraping: Puppeteer, Cheerio, or a Scraping API?
An honest comparison of the three ways to scrape with Node.js: Cheerio with fetch, a real browser through Puppeteer or Playwright, and a scraping API. What each is genuinely good at, the memory failures that kill Node scrapers in production, and a decision rule that usually says start with Cheerio.
Ready to put it to work? See how it works, explore the features, or compare plans.
Reading is good. Clean, LLM-ready data is better.
Point ClawEngine at any public or permitted site and get back clean markdown, JSON, or typed structured fields in one call. Crawl at scale, render JavaScript, and feed your RAG pipelines and AI agents, robots.txt and Terms of Service respected.
Clean markdown in one call · JavaScript rendered · robots.txt respected