Is Scraping Data for AI Training Legal? A 2026 US Guide

The short answer

Scraping publicly available web pages to train an AI model is broadly lawful in the United States, and by mid 2026 two federal courts have held that training itself can be transformative fair use. The liability has moved somewhere more specific: how you obtained the corpus. Courts have treated training on lawfully acquired material very differently from building a permanent library of pirated works, and one court found no fair use at all where the model competed directly with the source it was trained on. So the question that decides your risk is not "is scraping legal" but "can you prove, record by record, where every document came from and that you were allowed to have it." This is general information, not legal advice.

What US courts actually decided

Three decisions did most of the work, and they point in different directions for good reasons. Reading them together is more useful than reading any one of them.

Thomson Reuters v. Ross Intelligence (February 2025) came first and went against the AI company. Ross had used Westlaw headnotes to train a legal research tool, and the court found that was not fair use. The reasoning turned on market harm: Ross built a product that competed directly with the very service whose content it consumed. That is the fact pattern to avoid. The case is on interlocutory appeal to the Third Circuit, so it is not the last word.

Bartz v. Anthropic (June 2025) split the question in a way that has become the standard framework. Judge Alsup found that training a language model on books was "quintessentially transformative" and therefore fair use. He treated the separate act of downloading and retaining a permanent library of pirated books as a different question with a different answer, since acquiring copies without paying for them is not made transformative by what you later do with them. Anthropic settled the piracy side for $1.5 billion. The training was fine. The sourcing was not.

Kadrey v. Meta (June 2025) landed within weeks and reached a similar conclusion on the training claim, granting Meta summary judgment on the basis that training was highly transformative, while flagging that a better developed record on market dilution might have produced a different outcome.

Is it legal to scrape websites to train an AI model?

In the United States, generally yes, when three things are true: the pages are public and you reach them without logging in, you are not breaching a contract you accepted, and you are not building something that directly substitutes for the source you took it from. Every published US decision that has gone badly for a defendant so far failed at least one of those three, and usually the first or the third.

What has changed since 2024 is where the argument happens. Nobody serious now leads with the Computer Fraud and Abuse Act, because Van Buren narrowed it to gated areas rather than improper purposes. The fight moved to copyright, and inside copyright it moved to acquisition. That is a much more tractable problem for an engineering team, because acquisition is something you control and can document.

Does robots.txt legally stop you from using data for AI training?

robots.txt is not a contract and ignoring it is not by itself illegal in the United States. It is still the most important file in your compliance posture, for two practical reasons. It is the clearest available evidence of whether a publisher consented, and a plaintiff who can show you crawled paths you were explicitly told to skip has a much better story about willfulness, whatever the underlying claim is.

Since 2023 publishers have had AI specific controls, and they use them. GPTBot, Google-Extended, ClaudeBot, CCBot and PerplexityBot can each be disallowed independently of Googlebot, which means a site can welcome search indexing and refuse AI training in the same file. Treat a disallow for those agents as a refusal of consent for training use, even if you technically fetched the page under a different user agent. Our full walkthrough of the file lives in the guide to whether web scraping is legal.

Is scraping for RAG different from scraping for training?

Yes, and the difference is usually favorable. Retrieval augmented generation does not bake the source text into model weights. It indexes documents, retrieves the relevant passages at query time, and cites them back. That looks far more like ordinary information retrieval, and the fair use analysis is friendlier because the use is more clearly transformative in purpose and typically points users back to the source rather than replacing it.

Two cautions that people miss. First, the copies you make to build the index are still copies, so acquisition rules apply exactly as they do for training. Second, a RAG system that reproduces long verbatim passages instead of short quoted snippets starts to look like republication, and the fact that a model produced the output is not a defense. Keep retrieved excerpts tight, attribute them, and link back. If you are building this, the mechanics are in web scraping for RAG.

Can I use scraped data to fine-tune a model for my own company?

Internal fine-tuning on public data is the lowest risk posture in this whole area, and it is where most US companies actually sit. The model stays inside your walls, it does not compete with the sources in their market, and no third party ever sees the outputs. That combination weakens both the market harm factor and the practical incentive for anyone to sue.

Risk climbs at two specific moments: when the fine-tuned model becomes a product you sell, and when the training set includes a competitor's proprietary content. A support bot fine-tuned on your own docs plus public standards material is a different animal from one fine-tuned on a rival's knowledge base. If the corpus contains material from a company whose business is selling access to that material, get counsel involved before you train, not after you ship.

What about personal data in the training set?

Copyright is only half the picture. If your crawl sweeps up names, emails, biographical detail or anything else tied to an identifiable person, US state privacy law applies regardless of whether the page was public. Under the CCPA as amended, publicly available information is largely carved out, but the carve out is narrower than people assume and it does not extend to inferences you draw or to combining sources into a profile. Several states now grant deletion rights that are painful to honor if you cannot locate a specific person's records inside a training corpus.

The engineering answer is narrow extraction. Pull a typed schema with the fields you actually need instead of hoovering full page text, and personal data mostly never enters the pipeline. It is cheaper to not collect something than to build the machinery to delete it later.

There is a security dimension too, and it is newer than the legal one. Text scraped from the open web is untrusted input, and instructions embedded in a page can be carried straight into a model's context by a RAG retriever or an agent. Teams shipping agents on web data increasingly treat prompt injection as a threat model rather than a curiosity, and the mitigation belongs at ingestion time, alongside the compliance checks.

What compliant collection looks like

Provenance is the whole game now

If you take one operational lesson from the 2025 and 2026 rulings, make it this: the corpus manifest matters more than the crawler. Anthropic's training was held to be fair use and it still wrote a very large check, because of where the files came from. A team that can produce a table showing, for every document, the URL it came from, the UTC timestamp it was fetched at, the robots.txt decision that applied, and the license or terms in force at that moment, is in a fundamentally different position from a team with a folder of text files and a good memory.

Build that table at collection time. Retrofitting provenance onto an existing corpus ranges from expensive to impossible, and "we think most of it came from public pages" is not a defense anyone wants to make. It also pays for itself operationally, because the same manifest is what lets you re-crawl only what changed, drop a source cleanly when a publisher's terms change, and answer a diligence questionnaire in an afternoon instead of a quarter.

Where the line sits for us

We take a deliberately conservative position, and it costs us some addressable market. ClawEngine crawls public, permitted pages. It reads robots.txt and obeys it, including the AI specific agents. It does not log in, does not defeat anti bot systems, and does not touch paywalled content. That rules out marketplaces and social platforms entirely, which is a real limitation and we would rather say so than sell coverage that quietly erodes. The reasoning is set out in our compliance policy.

For teams assembling a corpus on that basis, the crawl website to LLM endpoint returns clean markdown or typed JSON with the source URL and fetch time attached to every record, which is the manifest described above, produced as a side effect of the crawl. If you are still choosing an approach, the guide to crawling a website for LLM training data covers the pipeline, and whether scraping competitor prices is legal covers the adjacent commercial case.