Migrate Tavily Extract to a Crawl API

Most teams do not choose Tavily Extract. They arrive at it, because they were already using Tavily Search inside an agent, extraction was one more parameter on a client they had already installed, and it worked. That is a good reason to start somewhere and a poor reason to stay.

The moment to look again is when the workload changes shape. Search-then-read is a discovery workload: an agent asks a question, gets back candidates, and reads two or three of them. Ingestion is a different workload: you already know the site, and you want all of it, on a schedule. The tool that is excellent at the first is priced to discourage the second, and the billing tells you so if you read it carefully.

How much does Tavily Extract cost per page?

Tavily bills in API credits and sells them pay as you go at $0.008 each. Extraction is batched: every 5 successful URL extractions cost 1 credit on basic depth, or 2 credits on advanced. That works out to about $0.0016 per page basic, which is genuinely cheap for reading a short list.

Crawling is where the arithmetic changes, because a crawl is two billable operations rather than one. Tavily documents crawl as mapping plus extraction: mapping costs 1 credit per 10 pages returned, and then extraction bills those pages at the usual 5 per credit. Their own worked example is a 10 page crawl with basic extraction coming to 3 credits, 1 for the map and 2 for the extraction.

Run that out to a real ingestion job and a 1,000 page crawl is 100 mapping credits plus 200 extraction credits, so 300 credits, about $2.40. On advanced extraction it is 500 credits, about $4.00. Compare that against a crawl API that charges 1 credit per page and sells 100,000 credits for $99, which is about $0.99 per 1,000 pages, and the gap is roughly two to four times. All of these figures were verified from Tavily and Firecrawl pricing and documentation in August 2026.

Two caveats before anyone quotes that table at a vendor. Tavily also sells a Project tier priced on a slider rather than a published number, so a committed customer is not necessarily paying the pay as you go rate. And bundled crawl plans only hit their headline per-page cost if you actually use the allowance, because credits on most self-serve plans do not roll over. An underused $99 plan is worse value than pay as you go, which is exactly the trap that makes the middle row of that table a genuine coin flip.

What actually changes in the code?

Less than you would expect, because you are removing a step rather than replacing one. The Tavily pattern is search, collect URLs, extract those URLs. The crawl pattern is point at a root and describe the boundary. The list of URLs stops being something you assemble and becomes something the crawler discovers.

Here is the before and after for the common case, ingesting a documentation site into a vector store.

# Before: search, collect, extract
results = tavily.search(query="acme api documentation", max_results=10)
urls = [r["url"] for r in results["results"]]
docs = tavily.extract(urls=urls, extract_depth="basic")
# You still owe: pagination, dedupe, and the pages search never surfaced.

# After: one crawl, bounded by path
POST /v1/crawl
{
  "url": "https://docs.acme.com",
  "path_prefix": "/docs/",
  "max_pages": 2000,
  "render": true,
  "schema": { "title": "string", "section": "string", "body_markdown": "string" }
}

The line worth staring at is the one that vanished. In the first version, search decides what gets ingested, which means your knowledge base contains whatever a ranking algorithm surfaced for one query on one day. That is fine for an agent answering a question and quietly wrong for a corpus you intend to keep. A path prefix is a boundary you can reason about and reproduce next month.

When should you not migrate?

Three cases, and they are common enough that saying so matters more than the pitch.

Your agent genuinely does discovery. If the URLs are not known ahead of time, no crawler helps, because a crawler cannot find a page it was never pointed at. Keep the search API. This is the whole distinction we lay out in the Firecrawl vs Tavily comparison, and it is the first question to settle before comparing any prices.

You are under a few hundred pages a month. The free Researcher tier is 1,000 credits, which covers about 3,300 crawled pages or 5,000 basic extractions. A migration that moves you from free to a paid plan is not a saving no matter how good the per-page rate looks.

You need the search index itself. Tavily returns ranked results from an index, which is a different asset from a crawler. Nobody in the crawl category ships an equivalent, ourselves included. If ranking quality is what your product depends on, the correct move is to keep paying for it and move only the bulk ingestion elsewhere.

Should you run both?

For most agent stacks, yes, and it is cheaper rather than a compromise. Search answers the question of which sources matter, at 1 credit for a basic query. Crawling answers the question of reading them completely, at per-page rates. Paying search prices for bulk ingestion, or crawl prices for discovery, is where budgets quietly go, and the routing logic that avoids it is a conditional and a few lines.

The pattern that works: let the agent search when it needs a source it does not have, and let a scheduled crawl keep the sources you already trust up to date. Teams building this kind of tool-routing layer are increasingly shopping for ready-made agents rather than writing every integration, and the routing decision looks the same either way. One vendor for finding, one for reading.

What about the output format?

This is the reason to migrate that has nothing to do with money, and for a lot of teams it is the deciding one. Tavily Extract returns cleaned page content, which is the right answer for putting text in front of a model right now. It is not a schema. If your pipeline needs a price, a version number, an author and a published date as validated fields, you are writing that parsing layer yourself against cleaned text.

A crawl API that accepts a schema in the request removes that layer. You declare the fields, you get typed JSON, and a site redesign becomes the vendor problem rather than a row of nulls that nothing alerted on. If what you want is clean prose for embeddings, either approach delivers, and you can read more on shaping that in our guide to crawling a website into LLM-ready content.

How long does the migration take?

Half a day per pipeline, assuming you understand what it currently ingests. Most of that time is not code. It is deciding the crawl boundary, because a URL list has an implicit boundary that nobody ever wrote down, and a path prefix makes it explicit. Expect one meeting about whether the changelog and the API reference belong in the same corpus.

The sequence that avoids surprises: run the crawl against your existing corpus first and diff the page set, so you find out what search was silently excluding before you cut over. Then run both for a billing cycle and compare the real invoices rather than the estimates, including the pages you failed to fetch. Then delete the extraction path, not before.

If you want the shape of that request against your own site, the console at the top of this page runs a live crawl and returns exactly what the API would. If you are still deciding between the two categories rather than migrating within one, start with the crawl API versus search API breakdown, or compare the wider field in our roundup of the best web scraping APIs.