By output · Website to JSON

Scrape website to JSON: web scraping with JSON output from a single API call

The short answer

To scrape a website to JSON, send the page URL to an extraction API and get a clean object back, url, title, cleaned content, links and metadata, instead of raw HTML you have to parse. ClawEngine renders JavaScript first so dynamic pages come back complete, and if you pass a schema it adds the typed fields you name on top of the standard shape. Every response has the same predictable structure, ready to store or process. It runs on public, permitted pages only. Plans start at $39 a month.

Clean markdown & JSON · JavaScript rendered · robots.txt respected

Last updated August 2026

Live Extraction

Endpoint · POST /v1/extract

POST

try:

Hit Extract to turn this page into clean, LLM-ready data.

robots.txt respected · public data only

Markdown · JSON · structured fields, from one API call. Crawling, rendering and extracting ...

When you scrape a website to JSON, you want a clean object your code can consume, not a tangle of tags. ClawEngine returns exactly that: a structured JSON response with the page url, title, cleaned content, extracted links and metadata, all from one API call. JavaScript is rendered first, so dynamic pages come back complete.

Need more than the standard shape? Define a schema and ClawEngine adds typed fields for the data you care about. The result is predictable, ready to parse, and easy to store. Every request runs against public, permitted pages only, respecting robots.txt and site Terms of Service and honoring crawl-delay.

CRAWL RENDER JS EXTRACT MARKDOWN JSON

Any URL in LLM-ready data out

robots.txt respected public data only

Why it works

What you get with Website to JSON

A clean JSON object

Each page returns url, title, content, links and metadata in a predictable JSON shape, so your code parses a record instead of scraping HTML.

Add typed fields

Define a schema and ClawEngine includes the structured fields you name, so you get general content and specific data in one response.

Dynamic pages included

JavaScript is rendered before serialization, so content that loads client-side is present in the JSON just like static markup.

What it handles

Any URL in, clean structured data out

Point ClawEngine at a public page and it crawls, renders the JavaScript and extracts clean markdown or typed JSON in one call. Define a schema for structured fields, and respect robots.txt and Terms of Service by default.

Returns clean JSON for any public page
Includes url, title, content, links, metadata
Adds typed fields from your schema
Renders JavaScript before serializing
Strips boilerplate from the content
Stays on public, permitted data only

POST /v1/extract extraction result

200 · JSON

{
  "url": "https://example.com/products/atlas",
  "title": "Atlas Field Notebook",
  "markdown": "# Atlas Field Notebook\n\nDurable...",
  "data": {
    "name": "Atlas Field Notebook",
    "price": 24.00,
    "currency": "USD",
    "rating": 4.7
  },
  "links": [ "/products", "/cart" ],
  "metadata": { "rendered": true }
}

JS rendered · boilerplate stripped ✓ robots.txt respected

Why ClawEngine

One API that crawls, renders and extracts

Not a raw HTML dump, not a headless browser fleet to run, and not a brittle parser to maintain. One call crawls a public page, renders its JavaScript and returns clean markdown or typed JSON, built for RAG pipelines and AI agents.

LLM-ready output

Clean markdown or typed JSON with the boilerplate stripped, so the data drops straight into a vector store, a prompt or an agent without a cleanup step.

JavaScript rendered

Each page loads in a real browser environment before extraction, so single-page apps and client-rendered content come back complete, not as an empty shell.

Compliance-first

ClawEngine works on public, permitted data only. It respects robots.txt and site Terms of Service and honors crawl-delay, so responsible scraping is the default.

Code examples

Scrape a page to JSON in one call

Send a URL and get a clean JSON object back: url, title, content, links and metadata. Add a schema to include typed fields, or crawl a whole site to get the same shape per page.

curl Page to JSON (POST /v1/extract)

curl https://api.clawengine.ai/v1/extract \
  -H "Authorization: Bearer $CLAWENGINE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/blog/launch",
    "format": "json"
  }'

# Response
# {
#   "url": "https://example.com/blog/launch",
#   "title": "Launch day",
#   "content": "We shipped...",
#   "links": ["https://example.com/pricing", "..."],
#   "metadata": { "description": "...", "canonical": "..." }
# }

python Clean JSON straight into your code

import os, requests

r = requests.post(
    "https://api.clawengine.ai/v1/extract",
    headers={"Authorization": f"Bearer {os.environ['CLAWENGINE_API_KEY']}"},
    json={"url": "https://example.com/blog/launch", "format": "json"},
)

page = r.json()
print(page["title"])
print(len(page["content"]), "chars of clean text")
print(len(page["links"]), "links")

node Add typed fields with a schema

const res = await fetch("https://api.clawengine.ai/v1/extract", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.CLAWENGINE_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://example.com/products/atlas",
    format: "json",
    schema: { name: "string", price: "number", in_stock: "boolean" },
  }),
});

const page = await res.json();
// Standard fields plus your typed schema data in one object.
console.log(page.title, page.data.name, page.data.price);

Scrape a website to JSON: the questions buyers ask

How do I scrape a website to JSON?

Send the page URL to an extraction API and it returns a JSON object rather than HTML. The standard response includes the url, title, cleaned content, an array of links and a metadata object; pass a schema and you also get the typed fields you name. ClawEngine renders the page before serializing, so JavaScript content is included, and it works on public, permitted pages only. You parse a record instead of scraping markup.

What fields are in the JSON response?

By default the object has the page url, title, cleaned content (also available as markdown), a links array and a metadata block with things like description and canonical URL. When you define a schema, ClawEngine adds those typed, structured fields on top, so you get general page content and the specific data you asked for in one predictable shape every time.

How is scraping to JSON different from an HTML response?

An HTML response hands you the raw page, tags, scripts, navigation and all, and leaves the parsing to you. A JSON response gives you a clean, predictable object your code can read directly: title, content, links, metadata and any schema fields. The difference is where the work happens. ClawEngine does the rendering, cleaning and structuring, so you skip the brittle parser and consume a record.

Can I convert a whole website to JSON, not just one page?

Yes. Crawl the site from a seed URL and ClawEngine returns a clean JSON record per page, or send known URLs one at a time. Either way every page comes back in the same shape, so a whole-site export is one job rather than thousands of parsers. Scope the crawl with a path prefix and a page limit to keep it focused and cheaper.

Does website-to-JSON work on JavaScript-heavy pages?

It does when the tool renders. Pages that build content client-side return almost nothing to a scraper that reads raw HTML, so the JSON comes back empty. ClawEngine loads each page in a real browser environment and waits for the content to build before it serializes, so single-page apps and lazy-loaded data appear in the JSON just like static content.

How do I turn messy HTML from a web scraper into clean structured JSON for analytics?

Stop parsing the HTML and define the shape you want instead. Pass a schema naming each field and its type, and the extraction step returns records that already match it, so every row lands in your warehouse with identical keys. The alternative, hand-written selectors, breaks whenever the site changes its markup, and repairing those selectors is where most scraping maintenance time actually goes.

How do I get JSON output from web scraping instead of HTML?

Ask the API for it at request time rather than converting afterwards. Set the response format to JSON and, if you want specific fields, pass a schema describing them, and the parsing happens on the service side. The alternative, fetching HTML and running BeautifulSoup or Cheerio over it, works until the site changes its markup, at which point every selector you wrote is a silent failure that returns nulls rather than an error.

Why is my scraped JSON full of nulls and empty strings?

Almost always one of two causes. Either the page is client-rendered and you fetched it without rendering, so the values were never in the HTML you parsed, or the site changed its markup and your selectors now match nothing. Check by searching the raw response for a value you can see in your browser. If it is missing there, you need rendering. If it is present but unmapped, your extraction rules drifted.

What is the difference between JSON output and structured extraction?

JSON output is the format of the response; structured extraction is the promise about what is inside it. A tool can return JSON whose content field is still a wall of raw markup. Structured extraction means the specific values you asked for, a price as a number, a date as a date, arrive as typed fields you can query without post-processing. Ask which one a vendor means before you buy.

Good questions

Questions about Website to JSON

By default the JSON includes the page url, title, cleaned content (also available as markdown), an array of links and a metadata object. Define a schema and ClawEngine adds the typed, structured fields you specify on top of that.

Yes. ClawEngine renders JavaScript before producing the JSON, so single-page apps and dynamically loaded content come back complete. It runs only on public, permitted pages and respects robots.txt and Terms of Service.

Read every web scraping question

Explore more

More ways to turn the web into data with ClawEngine

See every use case See pricing Back to the web scraping API

Stop wrangling raw HTML. Get LLM-ready data.

Point ClawEngine at a public page and one call crawls, renders the JavaScript and extracts clean markdown or typed JSON, ready for your RAG pipeline or AI agent. Public, permitted data only.

See pricing

Crawl · render JS · extract markdown & JSON · robots.txt respected, public data only