LLM Web Scraping in Python: A Practical Guide

How do you scrape websites for an LLM in Python?

Fetch the page, render its JavaScript, strip the boilerplate, then hand the model clean markdown or typed JSON. In Python that is requests plus BeautifulSoup for static pages, Playwright when content is client-rendered, and an LLM extraction pass only where layouts are messy. A scraping API collapses all three into one call.

That is the whole job. The reason it takes teams months anyway is that each stage fails in a different way, and the failures only show up in production. What follows is the honest progression, with code you can run, and a clear statement of where each step stops being worth it.

Step 1: requests and BeautifulSoup, and where it breaks

For a server-rendered page this is still the right answer. It is fast, it has no moving parts, and a fetch costs you bandwidth and nothing else.

import requests
from bs4 import BeautifulSoup

r = requests.get(
    "https://example.com/blog/post",
    headers={"User-Agent": "my-rag-bot/1.0 (+https://mysite.com/bot)"},
    timeout=20,
)
r.raise_for_status()

soup = BeautifulSoup(r.text, "lxml")

# strip the parts that are never content
for tag in soup(["script", "style", "nav", "header", "footer", "aside", "form"]):
    tag.decompose()

article = soup.find("article") or soup.body
text = article.get_text("\n", strip=True)

print(text[:500])

Two things go wrong. The first is JavaScript: on a React or Vue site, r.text is a root div and a script bundle, and your text is 40 characters of nothing. The second is subtler. get_text() flattens the document, so headings, lists and tables all collapse into an undifferentiated wall. That wall chunks badly, which is a retrieval problem you will misdiagnose as a model problem later. If you are staying on this path, convert to markdown instead of plain text so the heading hierarchy survives.

The find("article") or soup.body fallback is also doing more work than it looks. Telling content from boilerplate is a heuristic problem, not a tag problem, because a sidebar is a div and so is the article. That single line is where per-site special cases start breeding.

Step 2: Playwright renders the page, and now you run browsers

Playwright drives a real Chromium, so the scripts execute and the DOM you read is the DOM a user sees.

# pip install playwright && playwright install chromium
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page(user_agent="my-rag-bot/1.0")
    page.goto("https://app.example.com/products", wait_until="networkidle")

    # wait for the thing you actually want, not an arbitrary sleep
    page.wait_for_selector("[data-testid='product-card']", timeout=15_000)

    html = page.content()
    browser.close()

This works, and it is where the operational cost arrives. Each page is a browser context holding real memory. networkidle is unreliable on pages that poll or stream, so you end up waiting on specific selectors, which means a per-site selector again. Chromium leaks and crashes under sustained load, so you need supervision, restarts, timeouts and a concurrency cap that you will tune by watching things fall over. Two hundred pages on a laptop is fine. Fifty thousand pages a day is a fleet, and the fleet is a team member's job.

Step 3: the LLM extraction pass

Here is the part people mean by "LLM scraper." You stop writing selectors and ask a model to read the cleaned page and fill in a schema. Always validate the output, because a model returning JSON is not the same as a model returning your JSON.

import json
from openai import OpenAI
from pydantic import BaseModel, ValidationError

class Product(BaseModel):
    name: str
    price: float
    currency: str
    in_stock: bool

client = OpenAI()

def extract(markdown: str) -> Product | None:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        response_format={"type": "json_object"},
        temperature=0,
        messages=[
            {"role": "system", "content":
             "Extract product fields from the page. Return JSON matching this schema: "
             + json.dumps(Product.model_json_schema())
             + ". Use null for anything not stated on the page. Never guess."},
            {"role": "user", "content": markdown[:40_000]},
        ],
    )
    try:
        return Product.model_validate_json(resp.choices[0].message.content)
    except ValidationError as e:
        print("rejected:", e)   # log it, retry once, then quarantine
        return None

Three notes that matter in production. Set temperature=0, since you want the same page to give the same record. Truncating at 40,000 characters is a real constraint, not politeness: a long page has to be split, and a field can land in the half you dropped. And "use null, never guess" is load-bearing, because the default behavior of a helpful model handed a page with no price is to find something price-shaped.

The economics are the honest problem. Feeding a model a cleaned 10,000 token page and asking for six fields costs input tokens on every single page, every time you refresh. Across 50,000 pages refreshed weekly, that is 500 million input tokens a week to read documents whose structure never changed. A CSS selector costs zero tokens and returns the same answer.

Step 4: one API call instead of the stack

The three stages above are always the same, which is why they can be one request. You send a URL and a schema, and get typed JSON back with the rendering, boilerplate stripping and extraction already done.

import os, requests

r = requests.post(
    "https://api.clawengine.ai/v1/extract",
    headers={"Authorization": f"Bearer {os.environ['CLAWENGINE_API_KEY']}"},
    json={
        "url": "https://example.com/products/atlas",
        "format": "json",
        "schema": {
            "name": "string",
            "price": "number",
            "currency": "string",
            "rating": "number",
        },
    },
)

record = r.json()["data"]
# {'name': 'Atlas Field Notebook', 'price': 24.0, 'currency': 'USD', 'rating': 4.7}
print(record["name"], record["price"])

For a whole site rather than one page, POST /v1/crawl takes a seed URL, follows links within the scope you set, and runs asynchronously so a 500 page site is one call rather than a queue you operate.

import os, time, requests

BASE = "https://api.clawengine.ai/v1"
headers = {"Authorization": f"Bearer {os.environ['CLAWENGINE_API_KEY']}"}

job = requests.post(f"{BASE}/crawl", headers=headers, json={
    "url": "https://example.com/docs",
    "include_paths": ["/docs"],
    "limit": 500,
    "max_depth": 4,
    "format": "markdown",
}).json()

while True:
    res = requests.get(f"{BASE}/crawl/{job['id']}", headers=headers).json()
    if res["status"] == "completed":
        break
    time.sleep(3)

for page in res["pages"]:
    print(page["url"], len(page["markdown"]))

The four approaches compared

Can an LLM scrape a website?

Not on its own. A language model cannot fetch a URL or execute JavaScript; it only reads text you give it. What people call an LLM scraper is a normal scraper (an HTTP client or headless browser) with a model bolted on at the extraction step, replacing CSS selectors with a schema and a prompt.

The distinction matters because it tells you where the failure modes live. Empty pages, timeouts, rate limits and crawl scope are all scraper problems, and no model fixes them. Hallucinated fields, drifting formats and cost per page are model problems. Teams that blur the two spend a long time prompt-tuning an extraction step that is being handed an unrendered page.

Is Python good for web scraping with LLMs?

Yes, and it is the default for this work. Python has the mature scraping stack (requests, httpx, BeautifulSoup, lxml, Scrapy, Playwright) and it is where the LLM tooling lives: the official SDKs, pydantic for validating extracted records, and LangChain or LlamaIndex if you are chunking and embedding downstream.

The one place Python is not the obvious choice is when your app is already TypeScript. The llm-scraper library is a solid TS option built on Playwright, and shipping one language usually beats a marginally better library in another. Otherwise, the fact that pydantic gives you a schema that is simultaneously your validation layer and your prompt contract is a genuine advantage worth staying in Python for.

Should you use a local LLM for web scraping?

Sometimes, and the deciding factor is volume and privacy, not quality. Extraction is a narrow, structured task, so a small instruct model running on Ollama or vLLM can do it acceptably. Once you are extracting millions of pages, the per-token API cost stops being noise and fixed local hardware starts winning.

Be realistic about the tradeoffs. A local 7B or 8B model needs a tighter prompt, a smaller context window per call, and a stricter validation and retry loop than a frontier model does, because it fails more often on long or unusual pages. Test it on a hundred pages you have hand-labeled before assuming it works. Local also solves a real compliance question: if the pages contain anything you cannot send to a third party, keeping inference in your own network is a straightforward answer.

How much does LLM web scraping cost?

Three separate costs that people tend to merge: fetching and rendering (compute, or an API plan), LLM tokens for the extraction step (per page, per refresh, forever), and engineering time, which is usually the largest and the one that never shows up on a bill.

Do the token math before you commit to LLM extraction. A cleaned page of roughly 10,000 input tokens, times 50,000 pages, is 500 million tokens per full refresh. Multiply by your model's current input rate and by how often you re-crawl. That number is often fine for 500 messy sources and absurd for 50,000 stable ones. ClawEngine's own plans are flat by volume: Hobby at $39, Startup at $99, Scale at $399, and Enterprise custom, with rendering, schema extraction and crawling included rather than sold as add-ons. There is no free tier, because the buyer here is a team running a pipeline.

The honest recommendation

Do not use an LLM to extract from a stable layout. If a site puts the price in the same element on every product page, a selector or a fixed schema is cheaper, faster, deterministic and testable, and the model buys you nothing but a bill and a hallucination surface. Reach for LLM extraction when the sources are messy or numerous or changing: 300 supplier sites that each describe a product differently, or a set of pages that get redesigned faster than you can maintain selectors. That is where paying per page beats paying an engineer to chase markup.

Open source is genuinely the right pick in a few cases. Crawl4AI is a capable Python crawler with markdown output and LLM extraction strategies built in, and if you have the ops capacity and want to run it yourself, it will do the job. ScrapeGraphAI is worth a look if you like the graph-based pipeline model. Use them when your volume is spiky, your budget is engineer hours rather than dollars, or your data cannot leave your network. Use a managed LLM web scraper when crawling is not your product and you would rather define a schema than operate Chromium.

One last thing that is easy to skip. Whatever you build, keep the source URL and fetch date attached to every record from the extraction step onward. It is what lets your assistant cite an answer and lets you refresh a stale page, and once it is in your warehouse you can ask questions of it in plain English instead of writing a query every time someone wants to know which sources went quiet last month. Reconstructing provenance later is close to impossible.

Scrape only what you are allowed to scrape

None of this is a technique for getting past a login, a paywall or a bot check. Public and permitted data only, respect robots.txt, read the site's Terms of Service, honor crawl-delay, and pace your requests so you are never the reason a small site's server has a bad afternoon. Identify your crawler in the User-Agent with a URL a site owner can use to contact you. ClawEngine works on public and permitted pages under exactly those rules. This is general information, not legal advice.

The short version

Scraping for an LLM in Python is four decisions: fetch or render, selector or model, local or hosted, build or buy. requests and BeautifulSoup are right for static pages. Playwright is right when the content is client-rendered and you can carry the ops. LLM extraction earns its per-page cost only on messy or changing sources. And an API earns its keep when you would rather ship the pipeline than maintain the scraper underneath it.

If that last one is you, the LLM web scraper API returns rendered, cleaned, typed data from one call, and the web crawler API does it across a whole site. For what happens downstream, see how the same input problem plays out in a RAG data pipeline and what actually makes content LLM-ready, or compare the tools in this category in our best web scraping API roundup.