How to Monitor Competitor Prices With an API

How to monitor competitor prices with an API, in short

To monitor competitor prices with an API, extract each competitor product page on a schedule, pull the price and stock as typed fields, store every reading with a timestamp, and compare each new record against the last to flag changes. The hard parts, rendering JavaScript storefronts and parsing a different layout per retailer, are exactly what a hosted extraction API removes. This guide walks the full pipeline: what to collect, how often, how to structure it, and the pitfalls that quietly corrupt a price dataset.

Price monitoring is one of the highest-return uses of web data for a retailer or brand. Knowing that a rival dropped a SKU 8% this morning lets you match or hold with intent instead of guessing. The data is public and sitting on product pages. The work is collecting it reliably, at the cadence the category moves, without building a scraper you have to babysit.

What data to collect from each product page

Price alone is not enough to act on. A useful price-monitoring record captures the context around the number so you can tell a real move from noise. Define a schema with these fields and apply it to every retailer you track:

An extraction API that returns typed JSON gives you these fields directly. You describe the schema once and every product, on every retailer, comes back in the same shape. That is the difference between a maintainable monitor and a pile of per-site parsers. Our ecommerce scraping API is built for exactly this: send a product URL and a schema, get a clean record back.

The pipeline, step by step

The mechanics are simple once the extraction is handled. There are four stages, and only the first is web-specific.

1. Extract. For each product URL you track, call the extraction endpoint with your price schema. A tool that renders JavaScript matters here, because most storefronts build the price client-side and a raw-HTML scraper returns an empty field.

2. Store with a timestamp. Append every reading to a table keyed by SKU and time. Never overwrite; a price monitor is a time series, and the history is where the value is.

3. Diff. Compare each new reading to the previous one for that SKU. Emit a change event when price or stock moves beyond a threshold you set, which filters out rounding and currency jitter.

4. Act or alert. Route change events to a dashboard, a repricing rule or a Slack channel. This is where the data becomes a decision.

Here is the extract-and-diff core in Python, using ClawEngine for step one:

import os, requests, datetime

BASE = "https://api.clawengine.ai/v1"
headers = {"Authorization": f"Bearer {os.environ['CLAWENGINE_API_KEY']}"}
schema = {"name": "string", "price": "number", "currency": "string", "in_stock": "boolean"}

def read_price(url):
    r = requests.post(f"{BASE}/extract", headers=headers,
                      json={"url": url, "format": "json", "schema": schema})
    d = r.json()["data"]
    return {**d, "url": url, "ts": datetime.datetime.utcnow().isoformat()}

# last_seen is your stored previous reading per URL
def check(url, last_seen):
    now = read_price(url)
    prev = last_seen.get(url)
    if prev and abs(now["price"] - prev["price"]) >= 0.01 * prev["price"]:
        print(f"PRICE MOVE {now['name']}: {prev['price']} -> {now['price']} {now['currency']}")
    last_seen[url] = now
    return now

How often should I check competitor prices?

Match the cadence to how fast the category moves. Fast-moving electronics, marketplace listings and anything with dynamic pricing can shift several times a day, so hourly or a few times daily is reasonable. Stable catalog goods are fine checked daily or weekly. Checking more often than prices actually change wastes requests and adds needless load on the source, so tune the schedule per category rather than hammering everything hourly.

Is it legal to monitor competitor prices?

Collecting prices from public product pages is broadly lawful in the United States, and US courts have repeatedly declined to treat access to publicly available data as unauthorized access under the Computer Fraud and Abuse Act. The limits that still apply are the retailer's Terms of Service, its robots.txt, and copyright in images and descriptions. Price facts themselves are not copyrightable. A compliance-first tool that reads robots.txt and stays on public pages keeps you on the right side of this. This is general information, not legal advice.

Common ways a price dataset gets corrupted

Most bad price data comes from a handful of avoidable mistakes. Regional pricing is the big one: the same URL can serve a different currency or price by location, so pin your requests to one region and always store the currency. Sale-price confusion is next; capture both the current and the struck-through price so a permanent "was $99" display does not read as a fresh cut every run. Finally, an unrendered page returns a blank or placeholder price, which a naive diff reads as a huge move, so validate that the price field is present before you record it.

Beyond price: the wider competitive picture

Price is one signal among several. Teams that watch pricing closely usually also want to know when a rival launches a promotion, changes its assortment or ramps advertising. Extending the same extraction approach to category and promo pages covers assortment, and pairing it with a view of the ads your competitors are actively running rounds out the picture with demand-side intent you cannot see on a product page alone. The data pipeline is the same shape; you are just pointing it at more of the public surface.

Build versus buy for the extraction layer

You can write the scraper yourself with Playwright and a parser per retailer, and for two or three sites that is fine. It stops being fine when a retailer redesigns, adds bot defenses or lazy-loads the price, and you are debugging a broken monitor instead of pricing your catalog. A managed extraction API absorbs the rendering, the per-site parsing and the scale, and hands back the typed fields your diff logic needs. If you want the honest trade-offs across the main vendors, see our web scraping API comparison. For a single product page turned into a clean object, the scrape website to JSON endpoint is the simplest starting point.

Start monitoring

The pipeline is four stages and the only hard one is extraction, which a schema-based API turns into a single call. Decide the products and cadence that matter, define your price schema, and let the history accumulate. Within a week you will see rivals' patterns, and within a month you will price against evidence instead of instinct.