By use case · CrewAI

CrewAI web scraping tool: a hosted web scraper and scrape website tool for agent crews

The short answer

CrewAI does not scrape the web itself past a plain HTTP fetch. Its own web scraping catalog lists fourteen tools, and eleven of them are wrappers around a third-party vendor you have to sign up for and pay separately. The three that need no key are ScrapeWebsiteTool (requests plus BeautifulSoup, one URL, no JavaScript), ScrapeElementFromWebsiteTool (the same fetch narrowed to a CSS selector) and SeleniumScrapingTool (headless Chrome you install and maintain, still one URL, and the docs label it as in development). So the real decision is not whether to use a tool, it is which vendor fills the slot. ClawEngine fills it with one HTTP call that renders the page, strips the boilerplate, follows a whole site when you ask it to, and hands the agent markdown or typed JSON instead of a wall of page text. Plans start at $39 a month with no free tier.

Clean markdown & JSON · JavaScript rendered · robots.txt respected

Last updated September 2026

Live Extraction

Endpoint · POST /v1/extract

POST

try:

Hit Extract to turn this page into clean, LLM-ready data.

robots.txt respected · public data only

Markdown · JSON · structured fields, from one API call. Crawling, rendering and extracting ...

The first crew you build with the stock ScrapeWebsiteTool works. The second one does not, and the reason is always the same. Someone points the crew at a docs site or a pricing page that builds itself in the browser, requests fetches the empty shell, BeautifulSoup dutifully turns that shell into about forty words of nothing, and the agent writes a confident report based on a navigation menu. Nothing errors. There is no stack trace to chase. The crew simply produces worse output and you find out three tasks downstream.

The second failure is quieter and more expensive. ScrapeWebsiteTool returns the full text of the page with a fixed prefix, which means the cookie banner, the nav, the sidebar, the footer, the newsletter box and the cookie banner again all land in your agent context and get billed on every model call that touches that message. A mid-size marketing page is a few hundred useful words wrapped in a few thousand useless ones. Multiply that by five pages per task, several tasks per crew run, and a crew that runs on a schedule, and the scraping tool you picked because it was free turns into the largest single line on your model bill.

CrewAI is honest about the shape of this. Look at the web scraping section of its own documentation and you find fourteen entries, of which eleven ask you for an API key from Firecrawl, ScrapFly, ScrapeGraph, Spider, Browserbase, Hyperbrowser, Stagehand, Oxylabs or Bright Data. The framework orchestrates agents extremely well and treats web data as something you buy. That is a reasonable division of labor, and it means the useful question for a US team shipping a crew this quarter is which of those slots gives an agent the cleanest input for the least money and the least maintenance.

What an agent actually wants from a page is not HTML and not a screenshot. It is the article, in markdown, with the headings intact and the chrome gone, or a small typed JSON record when you already know which fields matter. Markdown keeps the structure a model reasons over and costs a fraction of the tokens. Typed JSON is better still when the task is deterministic, because you stop asking a language model to do a job a schema can do for free. ClawEngine returns either, from one POST, with render enabled when the page needs a browser.

Crawling is the other thing the built-in tools cannot do. ScrapeWebsiteTool takes one URL. If a research task needs a vendor's entire documentation tree or every post under a blog path, the naive fix is to have the agent discover links and loop, which burns model calls on navigation and produces a different set of pages every run. A crawl endpoint with a path prefix and a page limit does that server side, deterministically, and returns the whole set in one result the agent can reason over. Same task, one tool call, repeatable output.

Where this is the wrong purchase, plainly. If your crew reads five server-rendered pages a week, the stock tool is free and fine, and you should keep it. If your agents need to log in, fill forms, click through a checkout or defeat an anti-bot challenge, we are not the answer and Browserbase, Stagehand or a Bright Data unlocker are the tools CrewAI already documents for that. We crawl public, permitted pages, we read robots.txt, and we do not bypass logins, paywalls or CAPTCHAs. The fit is agents that read a lot of public web, need the content clean, and would rather not own a Chrome fleet.

If your crew sits alongside other frameworks, the same tool pattern works from a LangChain web scraping API loader, and the best web scraping API roundup has the per-1,000-page math for every vendor CrewAI documents. If you have already picked the Firecrawl tools, the Firecrawl alternative comparison covers the credit multipliers that decide what a rendered page really costs.

CRAWL RENDER JS EXTRACT MARKDOWN JSON

Any URL in LLM-ready data out

robots.txt respected public data only

Why it works

What you get with CrewAI

One tool call, not a scrape loop

Set a path prefix and a page limit and the whole documentation tree comes back in one result. The agent stops spending model calls on link discovery, and the same run returns the same pages.

Markdown the model can afford

Nav, cookie banners, sidebars and footers never reach the context window. The agent reads the article with its headings intact, which is a fraction of the tokens of a raw page dump on every call.

Renders before it reads

Set render to true and the page runs in a real browser first, so client-built pricing tables and docs are present. No Chrome on your machine, no driver upgrades, no fixed three second wait to tune.

What it handles

Any URL in, clean structured data out

Point ClawEngine at a public page and it crawls, renders the JavaScript and extracts clean markdown or typed JSON in one call. Define a schema for structured fields, and respect robots.txt and Terms of Service by default.

Drops into a crew as a BaseTool subclass
Renders JavaScript server side
Crawls a path prefix with a page cap
Returns clean markdown or typed JSON
Cuts the tokens a scraped page costs
Reads robots.txt and stays on permitted pages

POST /v1/extract extraction result

200 · JSON

{
  "url": "https://example.com/products/atlas",
  "title": "Atlas Field Notebook",
  "markdown": "# Atlas Field Notebook\n\nDurable...",
  "data": {
    "name": "Atlas Field Notebook",
    "price": 24.00,
    "currency": "USD",
    "rating": 4.7
  },
  "links": [ "/products", "/cart" ],
  "metadata": { "rendered": true }
}

JS rendered · boilerplate stripped ✓ robots.txt respected

Head to head

Every web scraping tool CrewAI documents, and what each one actually does

Compiled from the CrewAI tools documentation and the crewAI-tools source on 4 September 2026. The last row is a custom BaseTool wrapping ClawEngine, built from the code sample below.

Swipe to compare all columns →

Tool	Renders JavaScript	Crawls a whole site	What the agent receives	Third-party key
ScrapeWebsiteTool	No	No, one URL	Full page text, fixed prefix	None
ScrapeElementFromWebsiteTool	No	No, one URL	Text under a CSS selector	None
SeleniumScrapingTool	Yes, your Chrome	No, one URL	Text or HTML of a selector	None, but you run Chrome
FirecrawlScrapeWebsiteTool	Yes	No	Markdown or HTML	Firecrawl
FirecrawlCrawlWebsiteTool	Yes	Yes	Markdown per page	Firecrawl
FirecrawlSearchTool	Yes	No	Search results plus content	Firecrawl
ScrapflyScrapeWebsiteTool	Yes	No	Markdown, text or raw HTML	ScrapFly
ScrapegraphScrapeTool	Yes	No	LLM-extracted fields	ScrapeGraph
SpiderTool	Yes	Yes	Markdown or raw content	Spider
BrowserbaseLoadTool	Yes	No	Page content from a cloud browser	Browserbase
HyperbrowserLoadTool	Yes	Yes	Markdown or HTML	Hyperbrowser
StagehandTool	Yes	No	Result of a natural language action	Browserbase and an LLM key
OxylabsScraperTool	Yes	No	Parsed result or raw HTML	Oxylabs
BrightDataTools	Yes	Partly	SERP, unlocked page or dataset	Bright Data
Custom BaseTool on ClawEngine	Yes	Yes, path scoped	Clean markdown or typed JSON	ClawEngine, from $39 a month

Eleven of the fourteen tools CrewAI lists require a paid account somewhere else, which is the honest reading of this table: the framework expects you to buy web data, and the choice is which vendor sits in the slot. Verify current vendor pricing before you commit, since every provider in this table changes plans.

Why ClawEngine

One API that crawls, renders and extracts

Not a raw HTML dump, not a headless browser fleet to run, and not a brittle parser to maintain. One call crawls a public page, renders its JavaScript and returns clean markdown or typed JSON, built for RAG pipelines and AI agents.

LLM-ready output

Clean markdown or typed JSON with the boilerplate stripped, so the data drops straight into a vector store, a prompt or an agent without a cleanup step.

JavaScript rendered

Each page loads in a real browser environment before extraction, so single-page apps and client-rendered content come back complete, not as an empty shell.

Compliance-first

ClawEngine works on public, permitted data only. It respects robots.txt and site Terms of Service and honors crawl-delay, so responsible scraping is the default.

Code examples

A CrewAI tool that renders, crawls and returns markdown

Subclass BaseTool, declare an args_schema so the agent knows what to pass, and put the HTTP call in _run. The description is not a comment: it is the text the agent reads when it chooses a tool, so write it for the model.

python Custom BaseTool for a CrewAI agent

import os, requests
from typing import Type
from pydantic import BaseModel, Field
from crewai.tools import BaseTool

BASE = "https://api.clawengine.ai/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['CLAWENGINE_API_KEY']}"}


class ScrapeInput(BaseModel):
    url: str = Field(..., description="Absolute URL of the page to read")


class CleanPageTool(BaseTool):
    name: str = "Read a web page as clean markdown"
    description: str = (
        "Fetch one public web page and return it as clean markdown with the "
        "navigation, cookie notices, sidebars and footer removed. Runs the page "
        "in a real browser first, so use this for anything that builds itself "
        "in JavaScript or that the plain fetch returned empty."
    )
    args_schema: Type[BaseModel] = ScrapeInput

    def _run(self, url: str) -> str:
        r = requests.post(
            f"{BASE}/extract",
            headers=HEADERS,
            timeout=90,
            json={"url": url, "format": "markdown", "render": True},
        )
        r.raise_for_status()
        return r.json()["markdown"]

python Crawl a whole section, then wire both tools to an agent

class CrawlInput(BaseModel):
    start_url: str = Field(..., description="URL to start from")
    path_prefix: str = Field("/", description="Only follow links under this path")
    limit: int = Field(40, description="Maximum pages to return")


class CrawlSectionTool(BaseTool):
    name: str = "Crawl a section of a website"
    description: str = (
        "Follow links under one path prefix and return every page as markdown "
        "in a single result. Use this instead of scraping URLs one at a time "
        "when the task covers a documentation tree, a changelog or a blog path."
    )
    args_schema: Type[BaseModel] = CrawlInput

    def _run(self, start_url: str, path_prefix: str, limit: int) -> str:
        r = requests.post(
            f"{BASE}/crawl",
            headers=HEADERS,
            timeout=600,
            json={
                "url": start_url,
                "path_prefix": path_prefix,
                "limit": limit,
                "format": "markdown",
                "render": True,
            },
        )
        r.raise_for_status()
        pages = r.json()["pages"]
        return "\n\n---\n\n".join(
            f"# {p['url']}\n\n{p['markdown']}" for p in pages
        )


from crewai import Agent

researcher = Agent(
    role="Competitive researcher",
    goal="Summarize how a vendor prices its API",
    backstory="Reads vendor documentation and reports only what it can cite.",
    tools=[CleanPageTool(), CrawlSectionTool()],
)

CrewAI web scraping tool: the questions buyers ask

Can CrewAI scrape websites?

Yes, with a tool. CrewAI agents cannot fetch pages on their own, so you attach one of the tools from crewai-tools or write your own. The built-in ScrapeWebsiteTool makes a plain HTTP request with a 15 second timeout and parses the HTML with BeautifulSoup, which covers server-rendered pages and nothing else.

What is ScrapeWebsiteTool in CrewAI?

It is the default scraper in the crewai-tools package. You give it a website_url, it fetches the page with requests behind a browser-like user agent, parses it with BeautifulSoup, collapses the whitespace and returns the full page text prefixed with a fixed sentence. Fetches run through a helper that blocks private and cloud metadata ranges.

Does CrewAI ScrapeWebsiteTool render JavaScript?

No. It fetches raw HTML and never executes client-side scripts, so a single page application returns the shell instead of the content. The failure is silent: the agent gets a short string of navigation text rather than an error, and the task completes with a confidently wrong answer. Rendering needs SeleniumScrapingTool or a hosted API.

How do I add a custom web scraping tool to CrewAI?

Subclass BaseTool from crewai.tools, declare a name, a description the agent reads when deciding to call it, and an args_schema built with Pydantic. Put your HTTP call in _run and return a string. Then pass an instance in the tools list of the agent. The code sample below is a complete working example.

Can CrewAI crawl an entire website?

Not with the free tools. ScrapeWebsiteTool and ScrapeElementFromWebsiteTool each take a single URL and do not follow links. Crawling a whole documentation tree means either the FirecrawlCrawlWebsiteTool with a Firecrawl key, a Spider or Bright Data tool, or a custom tool that calls a crawl endpoint and returns the whole set in one result.

Why does my CrewAI agent run out of context when it scrapes?

Because the default tool returns the entire page as text and every word of it stays in the message history. Nav bars, cookie notices, sidebars and footers usually outweigh the article several times over, and you pay for them again on each subsequent model call in that task. Returning clean markdown, or typed JSON, is the fix.

CrewAI SeleniumScrapingTool vs a hosted scraping API: which should I use?

Selenium if the volume is tiny and you already run Chrome somewhere. It needs Chrome installed on the machine, targets one CSS selector, waits a fixed three seconds by default, and the documentation flags it as still in development. A hosted API removes the browser fleet, the driver upgrades and the wait tuning, which is most of the maintenance cost.

How much does web scraping cost inside a CrewAI crew?

Two bills, and the model one is usually larger. Page fetches are cents: ClawEngine starts at $39 a month on Hobby, $99 on Startup and $399 on Scale. The bigger number is tokens, because a page of raw text can be five to ten times the size of the same page in clean markdown, and it is re-sent on every model call in the task.

Which CrewAI scraping tools need a third-party API key?

Eleven of the fourteen listed in the web scraping section: both Firecrawl tools plus Firecrawl search, ScrapFly, ScrapeGraph, Spider, Browserbase, Hyperbrowser, Stagehand, Oxylabs and the Bright Data set. Only ScrapeWebsiteTool, ScrapeElementFromWebsiteTool and SeleniumScrapingTool run without one, and all three are single-URL fetchers.

Good questions

Questions about CrewAI

Run both, and describe them differently. The description string on a CrewAI tool is what the agent reads when it decides which one to call, so make the cheap one sound cheap and specific ("fetch one server-rendered page as text") and the hosted one sound like the heavier instrument ("render a JavaScript page or crawl a whole section, returns clean markdown"). Agents route on that text, and a crew that reaches for the free fetch on simple pages and the API on hard ones costs less than either tool alone. The pattern that goes wrong is giving two tools near-identical descriptions, because the agent then picks arbitrarily and your bill becomes unpredictable.

Markdown when the agent has to read and reason, JSON when you already know the fields. A research or summarization task benefits from headings and paragraph order, so markdown wins. A task that pulls a price, an effective date, a version number or a list of open roles should ask for a schema, because a typed record is smaller, deterministic and does not need the model to parse anything. The mistake worth avoiding is passing raw HTML into a prompt and asking the model to extract from it. That is the most expensive possible way to run a parser, and it fails differently each time.

Cache outside the agent. Agents have no memory of what a sibling agent fetched, so a crew with a researcher, an analyst and a writer will happily pull the same URL three times in one run. Put a small keyed store in front of your tool: normalize the URL, hash it with the format and schema you asked for, and return the cached body when it is fresh enough for the task. A short time to live measured in hours is usually right for research work and cuts both the API calls and the wall clock time noticeably. Log a cache hit rate for a week and you will find the number of duplicate fetches surprising.

Read every web scraping question

Explore more

More ways to turn the web into data with ClawEngine

See every use case See pricing Back to the web scraping API

Stop wrangling raw HTML. Get LLM-ready data.

Point ClawEngine at a public page and one call crawls, renders the JavaScript and extracts clean markdown or typed JSON, ready for your RAG pipeline or AI agent. Public, permitted data only.

See pricing

Crawl · render JS · extract markdown & JSON · robots.txt respected, public data only