Blog · General

Scrape the site or use an API? The cost of a scraper

Web scraping looks free, but it breaks on layout changes, logins, blocks and lost data. Here is when a scraper makes sense and when an API comes out cheaper.

Published on October 18, 2026 · 4 min read

When a project needs data from another site, the first idea is usually web scraping: a program opens the page, reads what is on screen and stores it. No contract, no payment, and a tutorial gets it done in an afternoon. The problem shows up later, when the scraper becomes the part of the project that demands the most work. This article compares both paths dispassionately and shows when each one pays off.

How a scraper works

There are two main kinds:

  • HTML scraper: downloads the page and looks for the data in its code, with libraries like BeautifulSoup or Cheerio. It is lightweight, but it only works on pages that already ship the data.
  • Automated browser: opens a code-controlled Chrome, with Selenium, Playwright or Puppeteer, logs in, clicks and reads the screen. It works on modern, JavaScript-heavy sites, but it is heavy and fragile.

Some people also read the site's internal connections, such as calls to private APIs or websockets, which is faster but depends on tokens and formats the site changes whenever it likes.

Where scrapers break

ProblemWhat happens
The layout changesThe selector stops finding the data and the program returns empty or errors
The login expiresThe bot is logged out in the middle of the night and stops collecting
Anti-bot blockingChallenges, captchas and IP bans, especially for cloud servers
Request limitsThe site answers 429 or drops the connection when the bot asks too often
Browser updatesThe Chromedriver version stops matching Chrome's
MemoryOne browser per bot consumes far more than a simple script

Each of those problems has a fix, but every fix is more code to maintain, and the site on the other side does not warn you before it changes.

The cost nobody counts at the start

  • Lost data. While the scraper is broken, that period's data does not come back. With real-time data, that opens gaps in the history, and the people using it notice.
  • Servers. An automated browser needs a much bigger server than a program that only makes HTTP requests.
  • Maintenance hours. Add up the time for each fix over a year and compare it with the price of a subscription.
  • Legal and account risk. Many sites forbid automated access in their terms of use, and the account the bot uses can be blocked. In some cases, copying data en masse also runs into database rights.
  • Reliability. Even big statistics sites lose data. In a comparison we ran, the public list of a well-known site was leaving out about 20% of the rounds of a live game, as we showed in Bac Bo history round by round.

What an API solves

An API is a contract: the provider commits to a format, handles collection and answers when something goes wrong.

  • A stable format: if the source changes, the provider adapts and your integration stays the same.
  • Complete data: someone collecting full time, with alerts and redundancy, loses far less than a homemade bot.
  • History: the provider keeps the past, and you query it whenever you want, instead of depending on what your bot happened to see.
  • Less infrastructure: your program only makes HTTP requests, and runs on the smallest server there is.
  • Support: when something does not work, there is someone on the other side to answer.

When scraping still makes sense

It would be dishonest to say never. A scraper pays off when:

  • there is no API for that data, paid or otherwise;
  • you need a one-off collection, not a continuous one;
  • the volume is small and a gap in the data makes no difference;
  • the site allows automated access in its terms of use.

If the data is continuous, real time, and your product depends on it, the maths almost always favours the API.

A quick calculation

Suppose the scraper breaks twice a month and each fix takes two hours. That is 48 hours of maintenance a year, besides the bigger server and the data lost in every outage. At LynxMetrics, a live data API starts at R$ 21.90 a month, or about R$ 263 a year. If your hour is worth more than R$ 6, the API is already cheaper, before counting any other advantage.

How to test the switch risk-free

Every LynxMetrics API has a free 2-hour trial, no card, after you confirm your email. In practice, you swap only the function that fetches the data in your program, run it with the trial key and compare. If you work with Bac Bo, the migration walkthrough is in Bac Bo with Python: why your scraper keeps dying.

Frequently asked questions

Is web scraping illegal?

It depends on the site, the data and the use. Many sites forbid automated access in their terms of use, which can get the account blocked, and copying a database en masse can run into rights over it. With an API, the provider does the collecting, and your account and server are not exposed to those blocks.

Why does my scraper keep breaking?

The most common reasons are a change in the site's layout, an expired login session, anti-bot or IP blocking, request limits, and a browser update that leaves Chromedriver incompatible.

Is an API more expensive than a scraper?

On the subscription price, yes, because a scraper looks free. Adding up servers, maintenance hours and the data lost in every outage, an API for continuous data usually comes out cheaper.

Read also