Best Way to Integrate a Portal With No API
Vertical SaaS teams keep hitting the same wall: the system their customers run publishes nothing you can call. Here is how in-house scrapers, browser automation and a managed extraction API actually compare once you price the maintenance.
By the ClawEngine team
August 2026 · 8 min read
Hit Extract to turn this page into clean, LLM-ready data.
robots.txt respected · public data only
Short answer: When a portal your customers rely on has no usable API, you have three real options: reverse engineer it in-house, drive it with browser automation, or buy a managed extraction API that returns typed JSON from the rendered pages. For public, no-login pages, the managed API is almost always the right call, because the work is not writing the parser, it is keeping the parser alive through every redesign. Browser automation is the correct answer only when the data genuinely sits behind a login or a multi-step form, and in-house is right when the source is strategic enough to own outright.
This problem shows up in the same shape at every vertical SaaS company. You sell into property management, legal, healthcare, logistics or fintech. Your roadmap depends on reading data out of the systems your customers already run, and a good share of those systems publish nothing you can call. The partner program takes nine months and rejects companies under a certain size. The official API is read-only, or covers three of the eleven objects you need, or does not exist.
Meanwhile each new integration unlocks a customer segment, so the pressure to ship one is real, and the engineers you would spend on it are the same engineers you want on core product. Here is how the three options actually compare once you price the maintenance rather than the build.
How do I get data from a website that has no API?
Before choosing an approach, spend twenty minutes looking for a route that already exists. In roughly a third of cases one does, and it beats everything below on cost and stability.
- Check for a bulk export. Many portals have a CSV or report download buried in a settings screen. If your customer can export it, you can often automate the export instead of the page.
- Check the Network tab. Open developer tools, filter to Fetch or XHR, reload the page. Modern portals load content from a JSON endpoint, and you will see it with its parameters. It was never meant as a public contract, so it can change without warning, but it is cleaner than parsing HTML while it lasts.
- Check for a partner or sandbox tier. Some vendors have an undocumented developer program that a customer-initiated request unlocks faster than a cold application.
If none of those exist, you are extracting from rendered pages, and the question becomes who maintains that.
Build in-house, browser automation, or a managed API?
| Approach | Time to first integration | Ongoing cost | Best when |
|---|---|---|---|
| In-house scraper | Days for the first, weeks for the fifth | Highest. Every redesign lands on your team, and nobody owns it after the author changes projects. | One or two sources that are strategic enough to own, with a named maintainer. |
| Browser automation | Days, then slow per run | High. Scripts break on selector and flow changes, and a fleet of browsers is real infrastructure. | Data behind a login, a multi-step form, or an action that must be performed rather than read. |
| Managed extraction API | Hours, and roughly the same for each new source | Lowest. You maintain a field list; rendering, crawling and retries are somebody else's problem. | Public, no-login pages, and any time you need many sources rather than one. |
The column that decides it is the third one. A scraper is cheap to write and expensive to keep alive, and the expense arrives unevenly: nothing for four months, then three broken integrations in the week a vendor ships a redesign. That is the cost people leave out of the build-versus-buy spreadsheet, and it is the one that actually determines whether integration number twelve ever ships.
If the pages you need are public and need no login, a schema-based service will turn a website into an API in an afternoon: you describe the fields you want, and every call returns the same typed record regardless of how the source lays them out. The full trade-off between renting that and running it yourself is worth reading before you commit either way, and we set it out in our note on a web scraping API versus building your own.
When browser automation is genuinely the right answer
It is worth being direct about this, because the honest boundary matters more than the pitch. If the data sits behind a customer login, or you need to submit a form, click through a wizard or perform an action inside the portal, then you need something that drives a browser session. An extraction API that reads public pages will not help you, and no amount of schema design changes that.
ClawEngine reads public, no-login pages. It does not fill in forms, hold authenticated sessions, defeat anti-bot systems or run OCR on scanned documents. When your integration needs any of those, browser automation or an official partner API is the correct tool, and choosing it early saves a month.
The same logic applies to workflow-shaped problems. A finance team that needs invoice lines out of a supplier portal usually wants accounts payable automation rather than a general crawler, because the approval routing and the payment run matter more than the extraction step. Match the tool to the job, not to the data format.
What actually breaks, in order of frequency
| Failure | What you see | How to prevent it |
|---|---|---|
| Layout change | Fields come back empty after a vendor deploy | Describe fields by name and type, never by position in the markup |
| Client-side rendering | The HTML is an empty shell and the parser returns nothing | Render each page in a browser environment before extracting |
| Pagination change | Nothing errors, the dataset is quietly short | Alert on row count per run, not just on HTTP failures |
| Rate limiting | Partial results, intermittently | Honor crawl-delay, cap concurrency, back off on errors |
The third row is the dangerous one. A pagination change does not raise an exception, it just returns fewer rows, and a silent 40 percent shortfall in a dataset your product reasons about can run for weeks. Alert on the shape of each run, not only on whether it completed.
Is it legal to integrate with a portal that has no API?
In the United States, collecting publicly available data from pages that need no login is broadly lawful, and courts have repeatedly declined to treat access to public data as unauthorized access under the Computer Fraud and Abuse Act. What still binds you is the portal's Terms of Use, its robots.txt, copyright in creative content, and privacy law once records describe individuals. If your customer holds the account and directs you to retrieve their own data, that is a materially different posture from crawling a third party at scale, and it is worth documenting which one you are in. We cover the detail in our guide to whether web scraping is legal. This is general information, not legal advice.
For regulated buyers there is a procurement angle too. Security review is easier when the answer to "what does this component do" is "it reads public pages against a schema" rather than "it holds our customers' portal credentials". Scope your integration so the smallest possible amount of it touches authentication.
How much does this cost?
Price the engineering, not the requests. A managed extraction API runs from $39 a month for roughly 50,000 pages on ClawEngine, and the published tiers are on our pricing page. Against that, one senior engineer spending a day a month keeping five scrapers alive is an order of magnitude more expensive, and it is a recurring cost that grows linearly with the number of integrations you have shipped.
The break-even is not really about volume. It is about count. One source you care deeply about can justify owning outright. Eleven sources, each unlocking a customer segment, is a portfolio problem, and portfolios of brittle scripts are what stop integration roadmaps.
A practical recommendation
Start by checking for a bulk export or an existing JSON endpoint, because the cheapest integration is the one you do not build. If the pages are public and need no login, use a managed data extraction API and keep your engineers on product; point it at the listing and let it crawl the whole site rather than wiring per-page requests yourself. Reach for browser automation only where a login or a form genuinely stands between you and the data, and build in-house only for the one or two sources important enough to staff permanently.
Whichever you choose, instrument it. Record the source URL on every row, alert on row counts and null rates per field, and re-check the vendor's terms when you scale up. Integrations rarely fail loudly; they fail quietly, and the teams that notice are the ones that measured the shape of the data rather than the exit code.
See ClawEngine turn pages into clean data
Point ClawEngine at any public or permitted site and get back clean markdown, JSON, or typed structured fields in one call. Crawl at scale, render JavaScript, and feed your RAG pipelines and AI agents, robots.txt and Terms of Service respected.