The ClawEngine blog
Web scraping, made practical
Practical writing on turning websites into clean, LLM-ready data: how to crawl for training data, extract structured fields for RAG, render JavaScript pages, decide between an API and your own scraper, and crawl compliantly with robots.txt. Public and permitted data only.
CCPA and Web Scraping: A 2026 Guide for US Companies
The CCPA applies to scraped personal data about California residents once your company crosses one of three thresholds. Public does not mean exempt.
Is Scraping Government Websites Legal? A 2026 US Guide
Public records on public government pages are broadly lawful to collect in the US. The limits that actually bite are Terms of Use, robots.txt, metered systems like PACER, and privacy law once records describe people. What each one means in practice.
Is Web Scraping Legal? robots.txt and US Court Rulings 2026
Short answer: scraping public data is generally legal in the US. The real limits are robots.txt, Terms of Service, copyright and privacy law. What US courts have actually held, plus crawl-delay, compliance basics and public versus private data, in plain English.
Is Scraping Competitor Prices Legal? A US Guide for Pricing Teams
Collecting public competitor prices is broadly lawful in the US, but the risk sits in Terms of Service, copyright and antitrust rather than the CFAA. What Van Buren, hiQ and Meta v. Bright Data actually mean for a pricing program.
Is Scraping Data for AI Training Legal? A 2026 US Guide
Two federal courts have now held that training an AI model can be transformative fair use, and one held it was not. What Bartz v. Anthropic, Kadrey v. Meta and Thomson Reuters v. Ross mean in practice, why acquisition rather than training is where the liability sits, and what a defensible corpus manifest looks like.
Can a Website Sue You for Scraping? What US Courts Do
Websites can and do sue scrapers, but the claims that actually win are breach of contract, copyright and trespass to chattels rather than the CFAA. The six claims ranked by how dangerous they really are, what the Ryanair v. Booking.com verdict and its reversal taught us, and how to answer a cease and desist.
Ready to put it to work? See how it works, explore the features, or compare plans.
Reading is good. Clean, LLM-ready data is better.
Point ClawEngine at any public or permitted site and get back clean markdown, JSON, or typed structured fields in one call. Crawl at scale, render JavaScript, and feed your RAG pipelines and AI agents, robots.txt and Terms of Service respected.
Clean markdown in one call · JavaScript rendered · robots.txt respected