Is Web Scraping Legal? robots.txt and US Court Rulings 2026

Is web scraping legal? It depends on what you crawl and how you behave

Is web scraping legal is one of the most common questions developers ask, and the honest answer is that it depends. Scraping is not inherently illegal, and accessing public information at reasonable rates is widely accepted. But the moment you touch private data, ignore a site's stated wishes, or overload a server, you move into territory that can be both legally and ethically wrong. This guide explains the compliance-first way to crawl: public and permitted data only, robots.txt and Terms of Service respected, crawl-delay honored, and you accountable for what you collect. It is general guidance, not legal advice; consult a lawyer for your specific situation.

Public and permitted data is the line that matters

The single most important distinction in responsible scraping is public versus private. Public data is content a site openly displays to any visitor without a login: published articles, product catalogs, documentation, public listings. Permitted data is content you are explicitly allowed to use, including your own sites and sites whose Terms of Service or an agreement grant you access. Everything else, content behind authentication, paywalls, or access controls, is off limits. ClawEngine is built for public and permitted data only, and that is the right policy for any crawl.

What US courts have actually held

Most advice on this subject is written from nerves rather than from the record. The record is narrower and more useful than the folklore, and three decisions do most of the work.

Van Buren v. United States (Supreme Court, 2021) cut back the Computer Fraud and Abuse Act's "exceeds authorized access" clause. It applies when someone obtains information from areas of a system they were never entitled to reach, not when a person with legitimate access uses data for a purpose the owner dislikes. That closed the door on the reading where breaking a website's terms could turn ordinary browsing into a federal crime.

hiQ Labs v. LinkedIn (Ninth Circuit) concluded that scraping public profile data, visible to anyone with a browser and no account, was not access "without authorization" under the CFAA. The ending is the instructive part: hiQ still lost on LinkedIn's breach of contract claim, because it had agreed to LinkedIn's user agreement. The public data was never the problem. The contract was.

Meta v. Bright Data (2024) followed the same logic. The court found Bright Data could only have breached Meta's terms if the collection happened while logged into an account, and Meta had not shown that. Logged-out collection of public pages survived.

The through-line is simple enough to keep in your head: public and logged-out is defensible; authenticated, contractual and personal is where the risk lives.

What robots.txt is and why you honor it

robots.txt is a file at the root of a site, for example at example.com/robots.txt, where the owner tells automated agents which paths they may and may not crawl. It is a clear, machine-readable statement of the site's wishes.

# a typical robots.txt
User-agent: *
Disallow: /admin/
Disallow: /checkout/
Crawl-delay: 5
Sitemap: https://example.com/sitemap.xml

The right thing to do is simple: read robots.txt before you crawl, stay out of disallowed paths, and treat the file as binding. Respecting it keeps you a good citizen of the web and reduces the chance of disputes. A compliance-first web crawler API checks robots.txt automatically and skips anything disallowed.

Honor crawl-delay and crawl gently

The Crawl-delay directive asks you to wait a set number of seconds between requests so you do not overload the server. Even when no delay is specified, crawl politely: limit concurrency, spread requests over time, and back off when a site returns errors or slows down. Hammering a host can degrade service for real users and is exactly the kind of behavior that turns a routine crawl into a problem. Gentle, rate-limited crawling is both kinder and safer.

Read the Terms of Service

A site's Terms of Service may permit, restrict or forbid automated access, and they carry weight. Before crawling at scale, check the ToS and respect what they say. When terms forbid scraping, the responsible choice is to look for an official API, request permission, or find another source. Aligning your crawl with both robots.txt and the ToS is how you stay on the right side of the rules.

Personal and private data deserves extra care

Personal data, anything that identifies an individual, sits under privacy laws such as the GDPR and the CCPA, regardless of whether it appears on a public page. Collecting and processing it carries real obligations. The safe default is to avoid scraping personal data, and to seek proper legal guidance before going anywhere near it. Just because information is visible does not mean it is free to harvest.

Practices that keep you compliant

• Crawl public and permitted data only. If it needs a login or sits behind a paywall, do not crawl it.
• Respect robots.txt and Terms of Service. Treat both as binding statements of the owner's wishes.
• Honor crawl-delay and rate limits. Never overload a server.
• Keep provenance. Record source URLs and dates so you can honor removal requests.
• Avoid personal data. Get legal advice before processing anything that identifies individuals.
• You are responsible for what you crawl. The tool follows the rules; the decision of what to collect is yours.

Is it illegal to scrape data from a website?

No, not as a general matter in the United States. Scraping publicly available data is not a crime, and federal courts have declined to read the Computer Fraud and Abuse Act as covering access to pages anyone can view. Scraping becomes unlawful when it crosses into authenticated or paywalled systems, breaches a contract you accepted, infringes copyright, or mishandles personal data.

Can a website sue you for scraping?

Yes, and this is the realistic risk rather than criminal prosecution. The common claims are breach of contract when you accepted the site's terms, copyright infringement when you copy protected content rather than facts, and trespass to chattels when your crawl degrades the site's service. All three are civil, all three are avoidable by staying logged out, storing facts rather than republishing prose, and crawling gently.

Does violating robots.txt make scraping illegal?

Not by itself. robots.txt is a convention rather than a statute, and ignoring it is not automatically a crime. It still matters a great deal in practice: disregarding a clear, machine-readable instruction from the site owner strengthens a plaintiff's story on every other claim and undermines any argument that you acted in good faith. Treat it as binding, because the cost of compliance is nearly zero and the cost of ignoring it compounds.

Is scraping data for AI training legal?

It is contested and moving. Collecting public pages sits on the same footing as any other scraping, but what you do next raises separate copyright questions that courts in the US are still working through, particularly around reproducing protected expression in outputs. The lower-risk pattern is to train or retrieve on content you own, content you have licensed, and content whose terms permit it, and to keep provenance for every document so you can remove a source cleanly if the position changes.

What data is safe to scrape?

Facts published openly on pages that permit crawling. Prices, specifications, availability, public documentation, published articles, public government records and your own sites are the durable categories, and they are exactly the fields a schema-based data extraction API is built to collect. The risky categories are consistent too: anything behind a login, anything identifying an individual, creative content you intend to republish, and any site whose terms forbid automated access or whose anti-bot systems are actively refusing you.

Is web scraping legal for commercial use?

Yes. Nothing in US law makes commercial purpose the deciding factor, and the cases that shaped this area were all commercial disputes between businesses. Collecting public, logged-out data to run a product is treated the same as collecting it for research. What changes with commercial use is not legality but exposure: you are a bigger target, you are worth suing, and you will be held to what you agreed to.

Three things do shift once money is involved. Copyright matters more, because reselling or republishing scraped descriptions, photographs or articles is infringement in a way that internal analysis usually is not. Contract matters more, because a commercial scraper is far more likely to have clicked through terms, opened an account or used an API key that came with conditions attached. And privacy law matters more, because the CCPA's data broker rules turn on selling or sharing personal information, so the same dataset can be unremarkable in-house and regulated the moment you license it out. Facts themselves are not copyrightable, which is why price, specification and availability data remains the most commercially durable category.

Is scraping government websites legal?

Generally yes, and public records start from a stronger position than private sites because a statute made them public rather than an agency choosing to share them. The usual limits still apply: read robots.txt, read the Terms of Use, and check whether an official API or bulk download exists first, because a sanctioned route beats a crawler on cost and stability.

The exception worth knowing is that some of the best-known government sources are authenticated and metered rather than open. Federal court records through PACER are the clearest case: access requires a paid account, and the Judiciary's policy prohibits any attempt to collect PACER data in a manner that avoids billing. Our guide to whether scraping government websites is legal works through the permit, licensing, procurement and docket sources in detail.

Related compliance guides

The general rules above apply everywhere, but the specifics change by industry. We have written separate walkthroughs for the situations that come up most:

• Is scraping competitor prices legal? Terms of Service, copyright in product content, and the antitrust rules that apply specifically to pricing data.
• Is scraping public data for lead generation legal? Where CCPA, CAN-SPAM and business contact data intersect.
• How to scrape real estate listings legally. MLS display rules and what falls outside a licensed feed.
• Is scraping data for AI training legal? What Bartz v. Anthropic, Kadrey v. Meta and Thomson Reuters v. Ross established about fair use, and why acquisition decides the risk.
• Can a website sue you for scraping? The six claims a site can bring, ranked by how dangerous each one actually is in US courts.
• Do I need permission to scrape a website? The four separate permissions most people collapse into one, and how to ask a site properly when it matters.
• Is scraping job postings legal? Why hiQ won on the CFAA and still paid $500,000, and why employer careers pages beat the big job boards on both legal risk and data quality.
• Is scraping government websites legal? Why public records start from a stronger position than private sites, the PACER rules that make federal court records off limits to crawlers, and how to check for an official open-data route first.
• GDPR and web scraping for US companies. When an EU regulation reaches a US-only crawler, and what to keep out of your database.
• CCPA and web scraping for US companies. Why California's "publicly available" carve-out is narrower than it looks, the 2026 thresholds, and the DELETE Act deletion requests brokers must process from August 1, 2026.

Compliance-first by design

The way to scrape with confidence is to make compliance the default, not an afterthought. ClawEngine is built for public and permitted data only: it respects robots.txt and Terms of Service, honors crawl-delay, and never aims at authentication, paywalls or private data. The responsibility for what you choose to crawl stays with you, and good defaults make doing the right thing the easy path. Read our compliance policy, or start with the website crawler API that applies these defaults on every request.