Do I Need Permission to Scrape a Website?

The short answer

You do not need a website's explicit permission to read pages it publishes publicly, and US courts have repeatedly held that scraping public data is not by itself unlawful computer access. What you do need is to stay inside the boundaries that site has set: honor robots.txt, respect the Terms of Service you agreed to if you signed up for an account, never log in or defeat a technical block to get at data, and do not reuse copyrighted content in a way the copyright owner can object to. This is general information, not legal advice.

The confusion comes from treating "permission" as a single yes or no. In practice there are four separate permissions, they are granted by different mechanisms, and you can have one without the others. Sorting them out is what turns a vague worry into a checklist you can actually work through.

The four permissions, and who grants each one

Most disputes trace back to someone satisfying one of these and assuming they had all four. A page being publicly readable settles the first row and says nothing about the other three.

Is it illegal to scrape a website without permission?

Not in itself, under US federal law, when the data is genuinely public. The line of cases running through hiQ Labs v. LinkedIn established that accessing a public website without authorization is not the kind of unauthorized access the Computer Fraud and Abuse Act was written to punish, and the Supreme Court's Van Buren decision narrowed the statute further. The CFAA is about breaking into places you were not allowed to be. A page served to anyone who requests it is not that place.

The practical consequence is that the risk in ordinary public scraping is rarely criminal. It is contractual, it is copyright, it is privacy, and it is the site simply blocking you. Those are real risks with real costs, but they are a different conversation from "am I committing a crime," and conflating them is why so many teams get bad answers.

Do I need permission to scrape publicly available data?

No, not as a general rule, and this is the most common version of the question. Publishing a page to the open web is an invitation to read it, and automating that reading does not change the character of the access. What automation changes is scale, and scale is where obligations attach: you should crawl at a rate that does not degrade the site, identify yourself with an honest user agent, and honor the crawl-delay and disallow rules the site publishes. Being public is permission to read, not permission to hammer.

Public also does not mean unowned. An article you can read for free is still copyrighted, and a name and email on a public directory page is still personal data with legal duties attached. That distinction is covered in more depth in our guide to whether scraping public data for lead generation is legal, where the personal-data question is the whole ballgame.

What does robots.txt actually permit?

robots.txt is a published statement of what a site is willing to have crawled, by which agents, and how fast. It is a convention rather than a law, and ignoring it is not a crime, but it is the clearest expression of the site owner's wishes that exists. Courts and regulators look at it when weighing whether conduct was reasonable, and a plaintiff who can show you were told not to crawl a path and did it anyway is in a much stronger position than one who cannot.

Read three things in it. Disallow lines tell you which paths are off-limits, and they can differ per user agent. Crawl-delay tells you the minimum gap the site wants between requests. And the Sitemap line usually tells you exactly which pages the site wants discovered, which is often a better crawl plan than anything you would build by following links. Treating robots.txt as an input rather than an obstacle tends to produce both a safer and a more efficient crawl.

Do Terms of Service count as permission?

They count against you when you actually agreed to them. US courts distinguish browsewrap terms, a link in a footer nobody clicked, from clickwrap terms you accepted by creating an account or checking a box. Clickwrap terms are ordinarily enforceable contracts, so if you signed up for an account and the terms forbid automated access, scraping while logged in is a breach of contract regardless of whether the pages are public to everyone else.

This is why the account question matters so much. Scraping public pages anonymously and scraping the same pages while logged in are legally different acts, because the second one involves a contract you entered. If you ever find yourself reasoning that logging in is fine because the data is public anyway, stop. That is precisely the fact pattern that turns a low-risk crawl into a lawsuit, as we cover in what a website can actually sue you for.

How do I ask a website for permission to scrape it?

Send a short, specific email to the site's contact or partnerships address. Say who you are, exactly which pages or paths you want, the rate you intend to crawl at, how often you would re-crawl, and what you will do with the data. Offer to identify your crawler with a named user agent so they can see and rate-limit you. Vague requests for "your data" get ignored; a request for one path at two requests per second, refreshed weekly, for internal analysis, gets answered.

Ask when the stakes justify it: you need a high crawl rate, the data is central to your product, the site is a potential partner, or robots.txt is ambiguous about your path. Written permission overrides robots.txt for you specifically and removes the contract question entirely, which is worth having on record before you build a business on the feed. Many sites will also point you at an official API you did not know existed, which is a better outcome than permission anyway.

When is there a permitted alternative to scraping?

Check for an official API, a bulk data download, or a licensed feed before you write a crawler. Documentation sites, government agencies, marketplaces and financial data providers frequently publish one, and using it removes the whole permission question, gives you a stable schema, and does not break the next time the site redesigns. For blockchain data specifically, block explorers usually forbid automated access in their terms while the same information is available if you query an indexed on-chain dataset directly, which is faster and squarely permitted.

Where no API exists, the next best posture is to narrow what you take. Extracting three typed fields from a page is easier to defend than storing the whole document, it sidesteps most copyright exposure because you are taking facts rather than expression, and it keeps personal data out of your database by design. That is the reasoning behind extracting structured data to a schema rather than archiving raw HTML.

What happens if you scrape a website without permission?

In the overwhelming majority of cases, nothing, or you get blocked. The realistic escalation ladder is rate limiting, then an IP block, then a cease-and-desist letter, and only then litigation, and litigation almost always involves an aggravating fact: you were logged in, you defeated a technical block, you republished the content, or you built a competing product from the data. Sites reserve their legal budget for those cases.

Two behaviors move you up that ladder faster than anything else. Crawling hard enough to affect the site's performance turns an invisible visitor into an incident with a ticket attached. And rotating identities to evade a block you have already received converts a debatable situation into one where the site can point to a clear refusal you worked around. Neither is worth the data.

The rules we hold ourselves to

ClawEngine only fetches public and permitted pages. It reads and honors robots.txt, respects crawl-delay, and does not log into accounts, defeat anti-bot systems or reach past paywalls, and we will not build those features. That rules out some targets our competitors advertise, and we consider that the correct trade: a data pipeline whose legal footing depends on nobody noticing is not a pipeline you can build a company on. The compliance policy spells out where the lines sit, and the broader legal picture including the CFAA, copyright and Terms of Service is in our guide to whether web scraping is legal.

A checklist before your next crawl

• Is every target page reachable without logging in? If not, you need permission, not a workaround.
• Does robots.txt allow your paths and your user agent? Read it per path, not once for the domain.
• Did you accept Terms of Service on this site? If you have an account, the contract binds you.
• Are you taking facts or expression? Prices, specs and availability are facts. Full articles are not.
• Does the extract contain personal data? If it does not need to, schema it out at collection time.
• Is your crawl rate polite? Honor crawl-delay and back off on errors instead of retrying harder.
• Would you be comfortable if the site owner read your logs? That is the honest version of the whole test.

Answer those seven and you have a defensible position for almost any commercial crawl. If a target fails the first or third question and the data matters to your business, that is the moment to write the permission email or call counsel, not the moment to look for a clever technical route around it.