By use case · AI training data

Web scraping for AI training: collect clean LLM training data from the open web

The short answer

Web scraping for AI training is the process of collecting text from public web pages and turning it into a clean, deduplicated corpus a model can learn from. The hard part is not fetching pages, it is getting boilerplate-free text, removing near-duplicate documents, keeping provenance for every document, and staying on sources you are permitted to use. ClawEngine crawls public sites, renders JavaScript, strips navigation and ads, and returns clean markdown with the source URL attached, which is the format a training pipeline actually wants. It works on public and permitted data only. Plans start at $39 a month.

Clean markdown & JSON · JavaScript rendered · robots.txt respected

Last updated July 2026

Live Extraction

Endpoint · POST /v1/extract

GET

try:

Hit Extract to turn this page into clean, LLM-ready data.

robots.txt respected · public data only

Markdown · JSON · structured fields, from one API call. Crawling, rendering and extracting ...

Training or fine-tuning a model is only as good as the corpus you feed it, and most of the work in building that corpus is cleanup. Raw HTML is full of navigation, cookie banners, ads and script tags that add tokens and noise without adding signal. Near-duplicate pages, the same article syndicated across ten sites, quietly bias a model toward whatever gets repeated. And a corpus with no record of where each document came from is impossible to audit, filter or partially delete later. ClawEngine is built to hand you the clean end of that pipeline.

Point it at a set of public sources and it crawls, renders any JavaScript, strips the boilerplate, and returns clean markdown for each page with the source URL and fetch timestamp attached. From there your pipeline chunks, deduplicates and filters, but it starts from readable text rather than a pile of markup. For structured corpora, define a schema and get typed JSON instead, so a dataset of, say, product descriptions or Q and A pairs comes back with consistent fields.

The part every serious team has to get right is licensing and permission, and it is where we are deliberately conservative. ClawEngine works on public and permitted pages, respects robots.txt and Terms of Service, and does not touch logins or paywalls. Whether a given source may be used to train a commercial model is a legal question about that source's terms and the copyright in its content, not something a scraper decides for you. Collecting responsibly from sources that permit it is the durable way to build, and it is the only way we support.

CRAWL RENDER JS EXTRACT MARKDOWN JSON

Any URL in LLM-ready data out

robots.txt respected public data only

Why it works

What you get with AI training data

Clean text, not raw HTML

Every page comes back as boilerplate-free markdown, so your corpus starts from readable text instead of navigation, ads and script tags that waste tokens.

Provenance on every document

Each document carries its source URL and fetch timestamp, so you can audit, filter and partially delete the corpus later without guesswork.

Permitted sources only

Crawls public pages that allow it, respects robots.txt and Terms of Service, and never touches logins or paywalls, which is the durable way to build a corpus.

What it handles

Any URL in, clean structured data out

Point ClawEngine at a public page and it crawls, renders the JavaScript and extracts clean markdown or typed JSON in one call. Define a schema for structured fields, and respect robots.txt and Terms of Service by default.

Crawls public sources at scale to clean markdown
Renders JavaScript so client-side pages are captured
Strips boilerplate that adds tokens without signal
Keeps source URL and timestamp on every document
Returns typed JSON for structured datasets
Respects robots.txt, ToS and crawl-delay

POST /v1/extract extraction result

200 · JSON

{
  "url": "https://example.com/products/atlas",
  "title": "Atlas Field Notebook",
  "markdown": "# Atlas Field Notebook\n\nDurable...",
  "data": {
    "name": "Atlas Field Notebook",
    "price": 24.00,
    "currency": "USD",
    "rating": 4.7
  },
  "links": [ "/products", "/cart" ],
  "metadata": { "rendered": true }
}

JS rendered · boilerplate stripped ✓ robots.txt respected

Why ClawEngine

One API that crawls, renders and extracts

Not a raw HTML dump, not a headless browser fleet to run, and not a brittle parser to maintain. One call crawls a public page, renders its JavaScript and returns clean markdown or typed JSON, built for RAG pipelines and AI agents.

LLM-ready output

Clean markdown or typed JSON with the boilerplate stripped, so the data drops straight into a vector store, a prompt or an agent without a cleanup step.

JavaScript rendered

Each page loads in a real browser environment before extraction, so single-page apps and client-rendered content come back complete, not as an empty shell.

Compliance-first

ClawEngine works on public, permitted data only. It respects robots.txt and site Terms of Service and honors crawl-delay, so responsible scraping is the default.

Code examples

Crawl a source into a clean training corpus

Crawl a permitted source to clean markdown, keep the source URL on every document, and write it out as JSONL your training pipeline can read. Swap markdown for a schema when you want a structured dataset.

python Crawl to markdown and write JSONL

import os, json, requests

BASE = "https://api.clawengine.ai/v1"
headers = {"Authorization": f"Bearer {os.environ['CLAWENGINE_API_KEY']}"}

job = requests.post(f"{BASE}/crawl", headers=headers, json={
    "url": "https://example.com/docs",
    "include": ["/docs/*"],        # stay on permitted, on-topic paths
    "max_pages": 1000,
    "format": "markdown",
}).json()

# one clean document per line, provenance kept for auditing and filtering
with open("corpus.jsonl", "w") as f:
    for page in job["pages"]:
        f.write(json.dumps({
            "url": page["url"],
            "fetched_at": page["fetched_at"],
            "text": page["markdown"],
        }) + "\n")

print(len(job["pages"]), "documents written")

python Deduplicate near-identical documents

import json, hashlib

seen = set()
kept, dropped = 0, 0

with open("corpus.jsonl") as src, open("corpus.dedup.jsonl", "w") as out:
    for line in src:
        doc = json.loads(line)
        # exact-duplicate guard; add MinHash/embeddings for near-duplicates
        h = hashlib.sha256(doc["text"].strip().encode()).hexdigest()
        if h in seen:
            dropped += 1
            continue
        seen.add(h)
        out.write(line)
        kept += 1

print(f"kept {kept}, dropped {dropped} exact duplicates")

python Structured dataset with a schema

import os, requests

BASE = "https://api.clawengine.ai/v1"
headers = {"Authorization": f"Bearer {os.environ['CLAWENGINE_API_KEY']}"}

# a structured corpus: identical keys per record, ready for JSONL
res = requests.post(f"{BASE}/extract", headers=headers, json={
    "url": "https://example.com/faq",
    "format": "json",
    "schema": {"question": "string", "answer": "string"},
})
res.raise_for_status()
print(res.json()["data"])

web scraping for AI training: the questions buyers ask

How is web data used to train AI models?

Web pages are crawled, stripped of boilerplate, and converted to plain text, then deduplicated and filtered into a corpus the model reads during pretraining or fine-tuning. The quality of that corpus matters more than its raw size: clean, diverse, deduplicated text produces a better model than a larger pile of noisy, repetitive pages. That is why the collection step focuses on cleaning and dedup, not just fetching as many pages as possible.

How do I collect training data for an LLM?

Start by choosing sources you are permitted to use, then crawl them to clean markdown with navigation and ads removed, keep the source URL on every document, deduplicate near-identical pages, and filter out low-quality or unsafe content. Doing the cleaning at collection time is far cheaper than trying to fix a messy corpus later. An API that returns rendered, boilerplate-free text turns the first three steps into one call.

Is it legal to scrape data to train an AI model?

Scraping public pages is broadly lawful in the United States, but training on what you collect raises separate copyright and Terms of Service questions that courts are actively working through. The conservative posture is to use public sources that permit it, respect robots.txt, avoid paywalled and login-gated content, and keep provenance so you can remove a source if needed. This is general information, not legal advice, and licensing high-value corpora is often the safer path.

Why does deduplication matter for training data?

Duplicate and near-duplicate documents make a model over-weight whatever is repeated, waste compute on redundant text, and can inflate benchmark scores through contamination. Removing near-duplicates before training measurably improves quality per token. Because deduplication compares documents across the whole corpus, it is easiest when every document arrives as clean text with a stable source URL, rather than as raw HTML you still have to normalize.

What format should training data be in?

For language model training, clean UTF-8 text or markdown with boilerplate removed is the practical target, usually chunked and stored with metadata like the source URL and a content hash. Markdown keeps useful structure such as headings and code blocks while dropping the markup noise of raw HTML. For structured or instruction-tuning datasets, typed JSON with a fixed schema is easier to validate and filter than free text.

How much data do I need to train an AI model?

It depends entirely on what you are doing. Fine-tuning an existing model for a narrow task can work with a few thousand high-quality examples, while pretraining from scratch takes billions of tokens. In both cases quality beats quantity past a point: a smaller clean, deduplicated, on-topic corpus usually outperforms a larger noisy one. Spend the effort on curation, not just on crawling more pages.

Good questions

Questions about AI training data

ClawEngine gives you clean text with a stable source URL per document, which is what makes deduplication straightforward, but the dedup pass itself runs in your pipeline where you control the threshold. A common approach is a content hash for exact duplicates plus MinHash or embedding similarity for near-duplicates. Starting from normalized markdown rather than raw HTML means the comparison is about the content, not incidental markup differences.

Yes. Pass a schema and each page returns typed JSON with your fields filled in, which suits instruction-tuning pairs, classification datasets or any corpus where you want consistent columns rather than free text. Because every record shares the same keys, it loads directly into a DataFrame or a JSONL file that a training script can read without a cleanup step.

The API only crawls public pages and honors robots.txt, so sources that disallow crawling are skipped by default. Beyond that, deciding which permitted sources belong in a commercial training set is a curation and licensing judgment you make, and keeping the source URL on every document is what lets you enforce that judgment: you can filter or remove any source cleanly because you always know where each document came from.

Read every web scraping question

Explore more

More ways to turn the web into data with ClawEngine

See every use case See pricing Back to the web scraping API

Stop wrangling raw HTML. Get LLM-ready data.

Point ClawEngine at a public page and one call crawls, renders the JavaScript and extracts clean markdown or typed JSON, ready for your RAG pipeline or AI agent. Public, permitted data only.

See pricing

Crawl · render JS · extract markdown & JSON · robots.txt respected, public data only