Is Scraping Government Websites Legal? A 2026 US Guide

Is scraping government websites legal?

For public, no-login pages, yes. Government records are public because a statute made them public, not because an agency chose to be generous, and that changes the starting position compared with scraping a private company's site. A county assessor publishes property records because state law requires it. A procurement office posts solicitations because open bidding rules require it. You are not extracting something the publisher would rather keep private; you are reading the thing they were obligated to publish.

That does not make government sites a free-for-all, and the distinction that matters is between the records and the delivery system. The records are public. The web application serving them is still a system with terms, capacity limits and sometimes a billing meter attached. Almost every real dispute in this area is about the delivery system, not about whether the underlying data was public.

Why the CFAA is usually not the problem

The Computer Fraud and Abuse Act is the statute people worry about, and for public government pages it is rarely the live issue. In hiQ Labs v. LinkedIn, the Ninth Circuit held in its April 2022 opinion that scraping publicly available data likely does not constitute access "without authorization" under the CFAA, and it read the Supreme Court's decision in Van Buren v. United States as reinforcing that narrow interpretation. The logic is that authorization only becomes a question where authorization was required in the first place. On a page any member of the public can load without signing in, it was not.

The important footnote is what that case did not decide. The Ninth Circuit was explicit that its analysis covered the CFAA alone and said nothing about other theories. That mattered a great deal afterward: in November 2022 the district court granted summary judgment that hiQ had breached LinkedIn's User Agreement through its automated scraping, and the record included hiQ using fake accounts to reach data that was not public at all. So the same fact pattern produced a defendant who was probably fine on the federal computer-crime statute and clearly not fine on contract.

The practical read for 2026 is that the risk has moved. Plaintiffs increasingly lead with breach of contract and copyright rather than the CFAA, because those theories survived where the CFAA argument did not. On government sites that shift matters less than elsewhere, since agencies rarely sue and public records carry thin copyright, but it tells you where to look: read the Terms of Use, not the criminal statute.

Can I scrape PACER for court records?

Not with a general-purpose crawler, and this is the clearest line in the whole topic. PACER is the federal Judiciary's court records system, it requires an account, and it charges for access. Its published policy prohibits automated collection designed to sidestep that billing, and it names the specific pattern people try.

The right move is to go through sanctioned routes. PACER publishes developer resources including an authentication API for automated systems, and the Free Law Project's RECAP archive and CourtListener APIs redistribute documents that have already been purchased, which is often all a research project needs. If what you actually want is the reasoning inside opinions rather than docket metadata, note that a crawler is the wrong tool entirely and AI-powered case law research answers that question directly.

State courts are a different picture. Many state and county court portals publish dockets on public pages with no account and no meter, and those are ordinary public records you can extract like any other. The federal system is the metered exception, not the rule.

Does robots.txt apply to government websites?

Yes, and you should follow it. Plenty of .gov and .us sites ship a robots.txt with real disallow rules, and they are usually protecting the expensive parts of the application: search endpoints, document generators, calendar queries and anything that runs a database scan per request. Those rules exist because a careless crawler once took the site down.

Following robots.txt is also the cheapest legal insurance available. Ignoring an explicit disallow strengthens any argument that your access was unauthorized, and it is the single easiest thing for an agency to point at after the fact. Honoring crawl-delay matters just as much on this kind of infrastructure. A county permit portal may be one server under a desk, and the difference between a polite crawl and an outage is your request rate. Slow down, cache aggressively, and never re-fetch a record you already have.

Should I use an open data API instead of scraping?

Whenever one exists, yes, and you should check properly before concluding it does not. The OPEN Government Data Act of 2018 requires federal agencies to publish their information as machine-readable open data and to register the metadata in the data.gov catalog, and many states and larger counties now run their own open-data portals. A documented API or a bulk download beats scraping on cost per record, on speed, on stability across redesigns, and on the permission question, which it removes entirely.

One catch worth knowing: the data.gov catalog holds metadata about datasets, including descriptions and URLs, not the datasets themselves. So the search that tells you whether an official route exists is a two-step process, and it is worth doing before you write a parser. Scraping earns its place in the very common case where an agency publishes records to a web page and offers no machine-readable route at all, which is most county and municipal government in the United States.

What government data can you actually scrape?

The public, no-login surface, which is much larger than most teams expect. The useful test is simple: if a member of the public can load the page in a browser without signing in or paying, it is in scope.

Two technical realities show up constantly on this surface. The first is that government portals run the full span of web history, from 2003 ASP.NET postback forms to client-side React, so a crawler that cannot render JavaScript before extracting will return empty results tables and report success. The second is that these sites get redesigned on procurement cycles, which means a selector-based parser breaks on a schedule you do not control. Extracting against a declared schema rather than a set of CSS paths survives that much better.

Do public records containing personal data create privacy obligations?

Often, yes, and this is where teams that got the access question right still go wrong. Public availability is not a blanket exemption. The CCPA does carve out information lawfully made available from federal, state or local government records, which covers a genuine slice of this data: property records and court dockets sit squarely inside it. But the carve-out is narrower than it sounds, and it does not simply travel with the data once you combine, enrich or resell it, because linkage is what makes a record identifying in the first place.

Court filings, licensing registries and property records routinely contain names, home addresses and signatures. The defensible pattern is to extract only the fields you actually need, keep personal fields in separate columns so you can apply different retention rules to them, and avoid building a person-centric index when a property-centric or case-centric one answers your question. For the full picture on the storage half of the problem, see CCPA and web scraping for US companies, and for the general access rules our guide to whether web scraping is legal covers robots.txt and Terms of Service in depth.

A practical checklist before you crawl an agency site

Most of the risk in this work is removed by five minutes of checking, not by legal review.

• Look for the official route first. Check data.gov, the agency developer page, and any state or county open-data portal. A bulk file beats a crawler every time.
• Read robots.txt and the Terms of Use. Disallow rules on search and document endpoints are common and worth honoring exactly.
• Check for a login or a meter. If either exists, stop and use sanctioned access. This is the PACER line.
• Crawl slowly and off-peak. Set a conservative rate, honor crawl-delay, and schedule large pulls outside business hours.
• Declare a schema. Extract named fields rather than dumping pages, so personal data cannot arrive by accident and a redesign does not silently break you.
• Keep provenance. Store the source URL and fetch timestamp on every record so you can re-verify and audit.

Building the pipeline

Once the source is cleared, the work is ordinary extraction: drive the public search, collect detail URLs, and extract each one against a fixed schema so every record lands in the same shape. Hash the fields you care about and you get change detection for free, which is what turns a permit portal or a bid board into an alerting feed rather than a nightly full re-read.

ClawEngine does the rendering and the schema-based extraction in one call, reads robots.txt and honors crawl-delay on every request, and stays on public, no-login pages by design. If you are working through the sources above, our government data scraping API covers the permit, docket and procurement patterns in detail, and how we approach compliance sets out the boundaries we hold to. For scheduled monitoring specifically, website change monitoring handles the diffing side.

The summary a lawyer would give you is short. Public records on public pages are fair game, the delivery system still has rules, authenticated and metered systems are off limits to crawlers, and personal data brings obligations that survive the fact that you found it in a government file. Get those four right and the rest is engineering.