How to Scrape a Website With n8n

The short answer

The reliable way to scrape a website with n8n is to call a scraping API from a single HTTP Request node, rather than running a headless browser inside a Code node. You POST the target URL to the API, it renders the page and returns clean markdown or typed JSON, and the next node in your workflow reads that result directly. This keeps the heavy work of rendering, proxying and parsing off your n8n instance, so your automations stay light and do not break every time a site adds a bot check or Chromium ships an update.

Why not just run a browser inside n8n?

It is the first thing most people try, and it works on the first site. You add a Code node, install Puppeteer, and scrape a page. Then you point it at a second site that renders client-side, memory climbs, a Chromium upgrade breaks the container, and now your automation platform has a browser fleet bolted to it that nobody wants to own. n8n is excellent at moving data between HTTP services and terrible as a place to host a browser. The moment scraping becomes an infrastructure problem, it stops being automation.

Calling an API from the HTTP Request node inverts that. The render, the proxy rotation and the parsing all happen on the API side. Your workflow only ever handles the clean JSON that comes back, which is exactly the kind of payload n8n was designed to route. Your instance stays small, and a site adding JavaScript rendering or a bot check becomes the API's problem instead of yours.

How do I scrape a website in n8n, step by step?

To scrape a website in n8n, add an HTTP Request node, set it to POST, point it at the scraping API endpoint, and put the target URL in the JSON body. The node returns the extracted content, which you wire into whatever comes next. Here is the whole flow.

1. Add a trigger. A Schedule trigger for a recurring scrape, a Webhook trigger to scrape on demand, or a manual trigger while you build.
2. Add an HTTP Request node. Method POST, URL https://api.clawengine.ai/v1/extract, and an Authorization header holding your Bearer token. Store the token as an n8n credential, not in the node.
3. Set the body. Send JSON with the target url, the format you want (markdown or json), and render: true when the page builds client-side.
4. Read the output. The node returns the markdown or your typed fields. Feed it into a Set, IF, database or spreadsheet node with no parsing step in between.

The body for a typed extraction looks like this, pasted straight into the HTTP Request node's JSON body field:

{
  "url": "https://example.com/products/widget-pro",
  "format": "json",
  "render": true,
  "schema": {
    "product_name": "string",
    "price": "number",
    "in_stock": "boolean"
  }
}

Ask for format: "markdown" instead and you get clean, boilerplate-stripped text, which is the right choice when the destination is a language model rather than a database. The full setup, including the crawl endpoint for whole sites, lives on the n8n web scraping API page.

How do I scrape JavaScript-rendered pages in n8n?

An HTTP Request node on its own only sees the HTML the server sent, and on a modern single page app that is an empty shell with the real content loaded later by JavaScript. To capture the rendered page you either run a headless browser inside a Code node, which is the heavy path described above, or set render: true on an API that renders server side and returns the finished content. The second option keeps your workflow to one node and hands you the same markdown a user would see. If you want the mechanics of why this happens, we cover it in rendering JavaScript pages when scraping.

How do I loop over many URLs in a workflow?

For a list of URLs, put a Split In Batches node ahead of the HTTP Request node, set a modest batch size so you stay polite to the target, and let n8n iterate. Batching also protects you from firing hundreds of simultaneous requests, which is both rude to the site and a good way to get rate limited. For a whole site rather than a known list, call the crawl endpoint once with a seed URL and scope rules, and the API walks the site for you and returns the pages in a single job, so you are not rebuilding a crawl frontier inside n8n.

{
  "url": "https://example.com/blog",
  "include": ["/blog/*"],
  "max_pages": 200,
  "format": "markdown"
}

Two ways to scrape in n8n, compared

Building an AI agent in n8n that scrapes the web

n8n's AI Agent and LangChain nodes can call a scraping API as a tool. The agent decides which URL it needs, the HTTP Request node fetches clean markdown, and that text goes into the model context. Returning markdown instead of raw HTML matters more here than anywhere, because raw HTML burns tokens on markup the model does not need and drags down answer quality. A scrape-as-a-tool step that hands back boilerplate-free text is the difference between an agent that reads a page and one that chokes on it. If you are collecting pages for retrieval rather than a live agent, the scrape data for AI agents page walks through the schema side.

What do I do with the results?

Because the API returns typed JSON when you pass a schema, every record shares the same keys, so it drops into a Postgres, Airtable or Google Sheets node with no cleanup transform. A common shape is a workflow that scrapes public company pages on a schedule, extracts firmographic fields, writes them to a database, and then routes the enriched records onward, for example into a personalized cold email sequence so the freshly collected contacts get worked while they are still current. n8n is the glue; the scrape is just one clean node in the middle of it.

Keep it compliant

The same boundaries apply inside n8n as anywhere else. Scrape public and permitted pages, respect robots.txt and Terms of Service, set a real batch size so you are not hammering a host, and never point the workflow at data behind a login or a paywall. An API does not make prohibited access permissible, and no scraping tool should pretend otherwise. For the full picture on what is and is not fair game, start with our guide to whether web scraping is legal.