How to Scrape Job Postings With an API

How to scrape job postings with an API, in short

To scrape job postings with an API, crawl a public board or company career page to collect posting links, extract each posting into a fixed schema (title, company, location, salary, employment type, description), and key every record by a stable ID so you can deduplicate and refresh on a schedule. The hard parts, rendering JavaScript boards and parsing a different layout per source, are exactly what a hosted extraction API removes. This guide walks the full pipeline, the fields to collect, and how to keep the feed compliant.

Job data drives a lot of products: niche job boards, recruiting-intelligence tools, ATS integrations, salary benchmarks and labor-market research. All of them need the same thing first, a clean, consistent record per posting, pulled from sources that come in a hundred different layouts. The mechanics are simple once extraction is handled, which is the whole reason to reach for an API rather than a parser per board.

What to collect from each posting

A job record you can query needs more than a title and a link. Define a schema with these fields and apply it to every source you crawl, so normalization and deduplication become tractable:

An extraction API that returns typed JSON gives you these fields directly. You describe the schema once and every posting, on every board, comes back in the same shape. Our job scraping API is built for exactly this: send a posting URL and a schema, get a clean record back.

The pipeline, step by step

There are four stages, and only the first is web-specific.

1. Discover posting URLs. Crawl the board or career page from a seed URL to collect the links to individual postings. A tool that renders JavaScript matters here, because most modern boards load their listings client-side and a raw-HTML crawl finds nothing.

2. Extract each posting to the schema. Call the extraction endpoint per posting URL. Here is the extract-and-key core in Python using ClawEngine:

import os, requests, hashlib

BASE = "https://api.clawengine.ai/v1"
headers = {"Authorization": f"Bearer {os.environ['CLAWENGINE_API_KEY']}"}
schema = {
    "title": "string", "company": "string", "location": "string",
    "salary_range": "string", "employment_type": "string", "description": "string",
}

def job_id(rec):
    key = f"{rec['company']}|{rec['title']}|{rec['location']}".lower()
    return hashlib.sha1(key.encode()).hexdigest()

def extract(url):
    r = requests.post(f"{BASE}/extract", headers=headers,
                      json={"url": url, "format": "json", "render": True, "schema": schema})
    return r.json()["data"]

feed = {}
for url in posting_urls:               # collected in stage 1
    rec = extract(url)
    feed[job_id(rec)] = rec            # same role on two boards collapses to one

3. Deduplicate and normalize. Key each record by a stable ID, a hash of company plus title plus location, or the canonical URL, so the same role posted on several boards collapses to one. Then normalize the messy fields: parse salary strings into a min and max, map employment type onto a fixed set, and standardize location.

4. Refresh on a schedule. Re-run the crawl on a cadence that matches how fast the source moves, daily is fine for most boards, and mark roles that drop off the source as expired. Because you key by a stable ID, a re-run updates existing jobs instead of duplicating them.

How do I handle boards that load listings with JavaScript?

You render them. Most modern job boards load listings and posting details client-side through JavaScript, so a scraper that reads only raw HTML captures an empty shell. An extraction API that loads each page in a real browser environment and waits for the content to build returns the title, salary and full description complete. This is the single most common reason a home-built job scraper returns blank fields, and it is why rendering is the first thing to check when a feed comes back empty.

Is it legal to scrape job postings?

Accessing publicly available job postings is broadly lawful in the United States, and US courts have repeatedly declined to treat scraping public data as unauthorized access under the Computer Fraud and Abuse Act. The constraints that still apply are the platform's Terms of Service, its robots.txt, and copyright in the posting text, so storing full descriptions and republishing them raises copyright questions even when access is lawful. Some large job boards explicitly prohibit scraping and gate content behind logins, so build on company career pages and boards that permit crawling. A compliance-first tool that reads robots.txt and stays on public pages keeps you on the right side of this. This is general information, not legal advice.

From feed to product

A clean job feed is the raw material, not the product. Once you have consistent records, the value is in what you build on top: a search experience, salary benchmarks, alerting on new roles, or matching candidates to openings. Teams building recruiting tools often pair the feed with an AI recruiter that sources and screens candidates, so the scraped roles flow straight into a matching and outreach workflow. The extraction pipeline is the foundation; the schema you defined in stage one is what makes everything downstream possible.

Build versus buy for extraction

You can write the crawler and a parser per board with Playwright, and for two or three sources that is reasonable. It stops being reasonable when a board redesigns, adds bot defenses or lazy-loads its listings, and you are debugging a broken feed instead of shipping features. A managed extraction API absorbs the rendering, the per-site parsing and the scale, and hands back the typed fields your dedup and normalization logic need. If you want to see a single posting turned into a clean object first, the scrape website to JSON endpoint is the simplest place to start.