By use case · Government data

Government data scraping API for public records, court dockets and procurement data

The short answer

A government data scraping API extracts structured records from public agency websites, court docket portals, permit and licensing databases and procurement notice boards, and returns them as typed JSON instead of HTML you have to parse. ClawEngine renders each page, strips the boilerplate and fills a schema you define in one call, so a county permit portal or a state contract board becomes a clean table you can diff on a schedule. It works on public, no-login pages only, reads robots.txt and honors crawl-delay. Where an agency publishes an official API or bulk download, use that first: it is faster, cheaper and explicitly sanctioned. Plans start at $39 a month.

Clean markdown & JSON · JavaScript rendered · robots.txt respected

Last updated August 2026

Live Extraction

Endpoint · POST /v1/extract

GET

try:

Hit Extract to turn this page into clean, LLM-ready data.

robots.txt respected · public data only

Markdown · JSON · structured fields, from one API call. Crawling, rendering and extracting ...

Government data is public by law and awful to work with in practice. The record you need exists, but it lives behind a paginated ASP.NET search form on a county site, or in a procurement portal that renders its results table with JavaScript, or in a permit database that changes layout when the vendor contract turns over. A government data scraping API removes the parsing layer: point it at the public page, define the fields you want, and get back a typed record you can load, diff and alert on.

Start with the official route every time. The OPEN Government Data Act of 2018 requires federal agencies to publish their information as machine-readable open data and register the metadata in the data.gov catalog, and many states and large counties now run their own open-data portals or APIs. When a documented API or a bulk download exists, it beats scraping on cost, speed and stability, and it removes the question of permission entirely. ClawEngine is for the very common case where it does not: the agency publishes the data on a public web page and nowhere else.

The boundary matters here more than in most verticals. ClawEngine works on public pages that need no login, reads robots.txt and honors crawl-delay. It is not built to sign into authenticated systems, and some of the best-known government sources are authenticated and metered. PACER is the clearest example: federal court records sit behind a paid account, and the Judiciary's own policy prohibits any attempt to collect PACER data in a manner that avoids billing, along with using an automated process to repeatedly hit the free portions of the application to harvest case information. That is a source to reach through official access, not a crawler.

CRAWL RENDER JS EXTRACT MARKDOWN JSON

Any URL in LLM-ready data out

robots.txt respected public data only

Why it works

What you get with Government data

Portals in, typed records out

Point ClawEngine at a permit search, a docket list or a bid board, define your schema, and get one clean typed record per result instead of a parser you rebuild after every vendor redesign.

Public, no-login pages only

It reads robots.txt, honors crawl-delay and never signs into authenticated or metered systems like PACER, so your pipeline stays defensible and your access stays inside the rules.

Renders legacy and modern portals

Government sites run everything from 2003 ASP.NET to client-side React. Each page loads in a real browser environment before extraction, so results tables built by JavaScript come back complete.

What it handles

Any URL in, clean structured data out

Point ClawEngine at a public page and it crawls, renders the JavaScript and extracts clean markdown or typed JSON in one call. Define a schema for structured fields, and respect robots.txt and Terms of Service by default.

Extracts permits, licenses and inspection records
Tracks procurement and bid notices on a schedule
Turns public docket listings into typed rows
Renders JavaScript results tables on legacy portals
Keeps the source URL and record ID on every row
Reads robots.txt and honors crawl-delay

POST /v1/extract extraction result

200 · JSON

{
  "url": "https://example.com/products/atlas",
  "title": "Atlas Field Notebook",
  "markdown": "# Atlas Field Notebook\n\nDurable...",
  "data": {
    "name": "Atlas Field Notebook",
    "price": 24.00,
    "currency": "USD",
    "rating": 4.7
  },
  "links": [ "/products", "/cart" ],
  "metadata": { "rendered": true }
}

JS rendered · boilerplate stripped ✓ robots.txt respected

Why ClawEngine

One API that crawls, renders and extracts

Not a raw HTML dump, not a headless browser fleet to run, and not a brittle parser to maintain. One call crawls a public page, renders its JavaScript and returns clean markdown or typed JSON, built for RAG pipelines and AI agents.

LLM-ready output

Clean markdown or typed JSON with the boilerplate stripped, so the data drops straight into a vector store, a prompt or an agent without a cleanup step.

JavaScript rendered

Each page loads in a real browser environment before extraction, so single-page apps and client-rendered content come back complete, not as an empty shell.

Compliance-first

ClawEngine works on public, permitted data only. It respects robots.txt and site Terms of Service and honors crawl-delay, so responsible scraping is the default.

Code examples

Extract a public permit record in a few lines

Send the public record URL and a schema of the fields you want. ClawEngine renders the page, strips the navigation and returns a typed record. Loop it over a result list to turn a portal into a table you can diff on a schedule.

curl Extract a permit detail page (POST /v1/extract)

curl https://api.clawengine.ai/v1/extract \
  -H "Authorization: Bearer $CLAWENGINE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://permits.example-county.gov/record/BLD-2026-04817",
    "format": "json",
    "render": true,
    "schema": {
      "permit_number": "string",
      "permit_type": "string",
      "status": "string",
      "issued_date": "string",
      "site_address": "string",
      "valuation": "number"
    }
  }'

python Watch a procurement board and alert on new solicitations

import os, hashlib, json, requests

BASE = "https://api.clawengine.ai/v1"
headers = {"Authorization": f"Bearer {os.environ['CLAWENGINE_API_KEY']}"}

schema = {
    "solicitation_id": "string",
    "title": "string",
    "agency": "string",
    "due_date": "string",
    "category": "string",
}

def fingerprint(record):
    # Hash the fields you care about so an amendment reads as a change,
    # and a cosmetic edit to the page does not.
    payload = json.dumps({k: record.get(k) for k in schema}, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()

seen = {}  # load from your store; keyed by solicitation_id

for url in notice_urls:  # collected from the public bid board listing
    r = requests.post(f"{BASE}/extract", headers=headers,
                      json={"url": url, "format": "json", "render": True, "schema": schema})
    record = r.json()["data"]
    record["source_url"] = url          # keep provenance on every row
    fp = fingerprint(record)
    if seen.get(record["solicitation_id"]) != fp:
        notify(record)                  # new solicitation, or an amendment
        seen[record["solicitation_id"]] = fp

node Crawl a public docket listing into typed rows

// Crawl the public listing, then extract each detail page to your schema.
// Keep the page budget modest: municipal and court servers are small.
const res = await fetch("https://api.clawengine.ai/v1/crawl", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.CLAWENGINE_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://courts.example-state.gov/dockets/civil",
    include_paths: ["/dockets/case/"],
    limit: 200,
    render: true,
    format: "json",
    schema: {
      case_number: "string",
      filed_date: "string",
      case_type: "string",
      court: "string",
      status: "string",
    },
  }),
});

const { id } = await res.json();
// Poll /v1/crawl/{id} or register a webhook; each page returns one typed docket row.
console.log("docket crawl started", id);

Government data scraping API: the questions buyers ask

Is scraping government websites legal?

Collecting public records from government websites is broadly lawful in the United States, and the data itself is public by statute rather than by permission. US courts have repeatedly declined to treat access to publicly available web data as unauthorized access under the Computer Fraud and Abuse Act. What still binds you is the site's Terms of Use, its robots.txt, any authentication or metering in front of the records, and privacy law once the records contain personal data. This is general information, not legal advice.

Can I scrape PACER for court records?

Not with a general crawler. PACER requires an account, charges $0.10 per page, and the Judiciary's policy states that any attempt to collect data from PACER in a manner that avoids billing is strictly prohibited. It specifically names using an automated process to repeatedly access the non-fee portions of the application to collect case information as misuse, and misuse ends in account termination. Use official PACER access and its published developer resources instead.

What government data can you actually scrape?

The public, no-login surface, which is larger than most people expect: state and county permit and inspection portals, business entity and professional licensing registries, procurement and bid notice boards, city council agendas and minutes, property assessor and tax roll pages, code enforcement records, and the many state court portals that publish dockets without a paywall. If a member of the public can load the page in a browser without signing in, it is in scope.

Should I use an open data API instead of scraping?

Yes, whenever one exists. A documented API or bulk download is cheaper per record, faster, more stable across redesigns and explicitly sanctioned, which removes the permission question entirely. Check data.gov, the agency's own developer page and any state or county open-data portal first. Scraping is the fallback for the very common case where an agency publishes records on a web page and offers no machine-readable route to them.

How do I scrape a county or municipal website?

Treat it as a search-then-detail crawl. Most municipal portals expose a results list behind a query form and a detail page per record, so you drive the list, collect detail URLs, then extract each one against a fixed schema. Render JavaScript, because a lot of these portals build the results table client-side. Keep the record identifier and the source URL on every row so you can re-fetch and audit, and crawl slowly: these servers are small.

Does robots.txt apply to government websites?

Yes, and you should follow it. Plenty of .gov and .us sites ship a robots.txt with real disallow rules, often protecting search endpoints and document generators that are expensive to serve. Ignoring those rules strengthens any argument that your access was unauthorized, and it is the single easiest thing for an agency to point at. ClawEngine reads robots.txt and honors crawl-delay on every request.

Do public records containing personal data create privacy obligations?

Often, yes. Public availability is not a blanket exemption. The CCPA carves out information lawfully made available from government records, but that carve-out is narrower than it sounds and does not travel with the data once you combine, enrich or resell it. Court filings, licensing registries and property records routinely contain names and addresses. Extract only the fields you actually need and keep personal fields separate so you can apply different retention rules.

How often does government data change?

It varies enormously and that should drive your schedule, not a default. Procurement boards and court dockets move daily. Permit and inspection records typically post in batches on business days. Assessor rolls and licensing registries may update quarterly or annually. Poll on the cadence the source actually publishes, store a content hash per record, and alert on the diff rather than reprocessing the whole set.

Good questions

Questions about Government data

Yes, and that is one of the cleaner uses. Crawl the notice list on a schedule, extract each solicitation against a schema (title, agency, NAICS or category, due date, contact, document links), and hash the record. New hashes are new opportunities and changed hashes are amendments, so your alerting is a diff rather than a re-read of the whole board.

Use the bulk file. It is cheaper, faster and sanctioned, and no crawler beats a published dataset on stability. A realistic pipeline usually mixes the two: bulk or API for the sources that offer it, ClawEngine for the counties and agencies that still publish only to a web page. Keeping both behind the same schema means downstream consumers cannot tell which route a record arrived by.

Read every web scraping question

Explore more

More ways to turn the web into data with ClawEngine

See every use case See pricing Back to the web scraping API

Stop wrangling raw HTML. Get LLM-ready data.

Point ClawEngine at a public page and one call crawls, renders the JavaScript and extracts clean markdown or typed JSON, ready for your RAG pipeline or AI agent. Public, permitted data only.

See pricing

Crawl · render JS · extract markdown & JSON · robots.txt respected, public data only