How to Scrape Real Estate Listings Legally

How to scrape real estate listings legally, in short

Accessing publicly available listing pages is broadly lawful in the United States, and courts have repeatedly declined to treat scraping public data as unauthorized access under the Computer Fraud and Abuse Act. The constraints that actually bind you are the site's Terms of Service, its robots.txt, copyright in photos and descriptions, MLS rules that travel with syndicated listings, and privacy law once you collect personal information about owners or agents. Build on sources that permit crawling and the pipeline holds up. This is general information, not legal advice.

Property data is one of the most valuable public datasets in the country and one of the most legally tangled, because four different rulebooks overlap on the same web page. Understanding which rule applies to which piece of data is most of the work. The technical part is easy by comparison.

The four rulebooks, and which one bites

1. Computer access law (rarely the problem)

The Computer Fraud and Abuse Act criminalizes accessing a computer "without authorization". For years plaintiffs argued that a Terms of Service violation made scraping a federal crime. That argument has fared badly. In hiQ Labs v. LinkedIn the Ninth Circuit held that scraping publicly available data likely does not violate the CFAA, and the Supreme Court's decision in Van Buren v. United States narrowed the statute further by rejecting a broad reading of "exceeds authorized access". The practical takeaway: if the page is public and you are not defeating a login or a technical access control, the CFAA is usually not your exposure.

The corollary matters just as much. Once you create an account, accept terms and log in to reach the data, you are in an entirely different legal posture. Do not do it.

2. Terms of Service and robots.txt (the real constraint)

Several of the largest US property portals prohibit scraping outright in their terms and back it up with aggressive bot defenses. A contract claim for breach of terms is far more likely to reach you than a criminal one, and it does not require the site to prove much. Read the terms per source. If a portal says no, treat that as no, whatever the CFAA case law says about access.

robots.txt is not itself a law, but it is the clearest published signal of what a site permits, and ignoring it undercuts any argument that you acted in good faith. It is also the easiest thing to get right, because a compliant crawler reads it for you.

3. Copyright (the one people forget)

Listing photographs are copyrighted, usually by the photographer or the brokerage, and listing descriptions are frequently protectable too. Facts are not copyrightable, so a price, a bed count, a square footage and an address are safe to record. The photo of the kitchen and the four paragraphs of prose about the "chef's dream layout" are not. Copying facts into your database is a different act from republishing images and descriptions on your own site, and most of the real litigation risk in this space sits on the republishing side.

4. MLS rules and privacy law

Listings syndicated from an MLS carry display rules that travel with them, covering attribution, how long data may be retained and what may be shown. Those rules bind participants through contract, and a portal displaying MLS-sourced listings is passing that obligation along in its own terms. Separately, once your records include personal information about identifiable people, agents, owners, tenants, the California Consumer Privacy Act and its state equivalents apply, including deletion and opt-out obligations. Teams that collect contact details at scale often end up needing a process to handle those personal data removal requests rather than treating them as an edge case.

Which sources are durable to build on

The question worth asking is not "can I scrape this" but "will this source still work in eighteen months". Sources that permit crawling stay stable. Sources that prohibit it turn into an escalating fight you eventually lose, usually right when your product depends on the data.

• County and municipal portals. Assessor rolls, deed records, permit filings. Public by statute, rarely restricted, and the authoritative source for ownership and tax data. Layouts vary wildly per county, which is precisely the problem an extraction API solves.
• Brokerage and agent websites. Usually happy to be indexed, since visibility is the point. Excellent for active inventory in a specific market.
• New construction and builder sites. Structured, public, and absent from many aggregated datasets.
• Commercial listing pages. There is no national commercial MLS, so most public CRE data genuinely lives on broker sites and market reports.
• Rental portals that permit it. Check terms individually, as policies differ sharply between them.

And if you need authoritative on-market residential listings, license them. An IDX or RESO Web API feed through a broker relationship is the correct path, and no amount of scraping substitutes for it. Scraping covers the public web the licensed feed does not reach, which is a large and useful surface. Our real estate data API is built for that half of the problem and does not resell MLS data.

Practical rules for a defensible crawl

Compliance here is mostly operational discipline, and it is cheap to implement up front.

Read robots.txt and honor crawl-delay. Automatically, on every request, not as a one-time review. A crawler that ignores the published rules is hard to defend regardless of what the underlying law says.

Identify yourself. Use a descriptive user agent with a contact URL. Sites that would otherwise block you will often just email instead, and that conversation frequently ends in permission.

Rate limit conservatively. A crawl that degrades a county server is the fastest way to turn a tolerable activity into an intolerable one. Slow crawls almost never generate complaints.

Never defeat access controls. No logins, no paywalls, no CAPTCHA solving, no rotating identities to evade a block. A block is an answer.

Store facts, reference the rest. Keep prices, sizes and addresses in your database. Store photo URLs rather than copies, and think hard before republishing descriptions.

Log provenance. Record the source URL, the retrieval timestamp and the robots.txt state at the time of the crawl for every record. If anyone ever asks how you obtained a field, you want an answer with a date on it.

What does the extraction actually look like?

Once sources are settled, the technical work is small. Send the listing URL and a schema of the fields you want, and get a typed record back. Rendering matters, because most listing pages build the price and detail panel client-side and a raw HTML fetch returns nothing useful.

curl https://api.clawengine.ai/v1/extract \
  -H "Authorization: Bearer $CLAWENGINE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://listings.example.com/homes/1421-oak-street",
    "format": "json",
    "render": true,
    "schema": {
      "street": "string",
      "city": "string",
      "state": "string",
      "zip": "string",
      "list_price": "number",
      "beds": "number",
      "baths": "number",
      "sqft": "number",
      "status": "string"
    }
  }'

Notice the schema collects facts and leaves the description and photos out. That is a deliberate choice, and it is the kind of choice that keeps a dataset boring in the best sense.

Is scraping Zillow or Redfin legal?

Those portals prohibit scraping in their Terms of Service and defend against it technically, so even where public access is not a CFAA problem, you are looking at a contract claim, an arms race with their bot defenses, and MLS display rules layered on top. The realistic answer for a business is to build elsewhere: county records, brokerage sites, builder sites and CRE listings give you a durable dataset that nobody is trying to shut off. Chasing a hostile source is an engineering treadmill with a legal bill attached.

Can I use scraped listing data to train a model?

Training on facts you lawfully collected is a much stronger position than training on copyrighted descriptions and photographs. Keep the two separable from the start, so a fact-only training set is one query away rather than an archaeology project. If you plan to expose the model output publicly, get counsel to look at it, because the law on generative output over copyrighted inputs is still moving.

Building the pipeline

Settle the sources, define a schema of facts, crawl politely, log provenance, and re-extract on a schedule so you catch price cuts and status changes rather than one snapshot. The same crawl-plus-schema pattern that handles listings handles product and price monitoring too, and our compliance approach spells out exactly which sources ClawEngine will and will not touch.