Scrape the site or use an API? The cost of a scraper
Web scraping looks free, but it breaks on layout changes, logins, blocks and lost data. Here is when a scraper makes sense and when an API comes out cheaper.
Published on October 18, 2026 · 4 min read
When a project needs data from another site, the first idea is usually web scraping: a program opens the page, reads what is on screen and stores it. No contract, no payment, and a tutorial gets it done in an afternoon. The problem shows up later, when the scraper becomes the part of the project that demands the most work. This article compares both paths dispassionately and shows when each one pays off.
How a scraper works
There are two main kinds:
- HTML scraper: downloads the page and looks for the data in its code, with libraries like BeautifulSoup or Cheerio. It is lightweight, but it only works on pages that already ship the data.
- Automated browser: opens a code-controlled Chrome, with Selenium, Playwright or Puppeteer, logs in, clicks and reads the screen. It works on modern, JavaScript-heavy sites, but it is heavy and fragile.
Some people also read the site's internal connections, such as calls to private APIs or websockets, which is faster but depends on tokens and formats the site changes whenever it likes.
Where scrapers break
| Problem | What happens |
|---|---|
| The layout changes | The selector stops finding the data and the program returns empty or errors |
| The login expires | The bot is logged out in the middle of the night and stops collecting |
| Anti-bot blocking | Challenges, captchas and IP bans, especially for cloud servers |
| Request limits | The site answers 429 or drops the connection when the bot asks too often |
| Browser updates | The Chromedriver version stops matching Chrome's |
| Memory | One browser per bot consumes far more than a simple script |
Each of those problems has a fix, but every fix is more code to maintain, and the site on the other side does not warn you before it changes.
The cost nobody counts at the start
- Lost data. While the scraper is broken, that period's data does not come back. With real-time data, that opens gaps in the history, and the people using it notice.
- Servers. An automated browser needs a much bigger server than a program that only makes HTTP requests.
- Maintenance hours. Add up the time for each fix over a year and compare it with the price of a subscription.
- Legal and account risk. Many sites forbid automated access in their terms of use, and the account the bot uses can be blocked. In some cases, copying data en masse also runs into database rights.
- Reliability. Even big statistics sites lose data. In a comparison we ran, the public list of a well-known site was leaving out about 20% of the rounds of a live game, as we showed in Bac Bo history round by round.
What an API solves
An API is a contract: the provider commits to a format, handles collection and answers when something goes wrong.
- A stable format: if the source changes, the provider adapts and your integration stays the same.
- Complete data: someone collecting full time, with alerts and redundancy, loses far less than a homemade bot.
- History: the provider keeps the past, and you query it whenever you want, instead of depending on what your bot happened to see.
- Less infrastructure: your program only makes HTTP requests, and runs on the smallest server there is.
- Support: when something does not work, there is someone on the other side to answer.
When scraping still makes sense
It would be dishonest to say never. A scraper pays off when:
- there is no API for that data, paid or otherwise;
- you need a one-off collection, not a continuous one;
- the volume is small and a gap in the data makes no difference;
- the site allows automated access in its terms of use.
If the data is continuous, real time, and your product depends on it, the maths almost always favours the API.
A quick calculation
Suppose the scraper breaks twice a month and each fix takes two hours. That is 48 hours of maintenance a year, besides the bigger server and the data lost in every outage. At LynxMetrics, a live data API starts at R$ 21.90 a month, or about R$ 263 a year. If your hour is worth more than R$ 6, the API is already cheaper, before counting any other advantage.
How to test the switch risk-free
Every LynxMetrics API has a free 2-hour trial, no card, after you confirm your email. In practice, you swap only the function that fetches the data in your program, run it with the trial key and compare. If you work with Bac Bo, the migration walkthrough is in Bac Bo with Python: why your scraper keeps dying.
Frequently asked questions
- Is web scraping illegal?
It depends on the site, the data and the use. Many sites forbid automated access in their terms of use, which can get the account blocked, and copying a database en masse can run into rights over it. With an API, the provider does the collecting, and your account and server are not exposed to those blocks.
- Why does my scraper keep breaking?
The most common reasons are a change in the site's layout, an expired login session, anti-bot or IP blocking, request limits, and a browser update that leaves Chromedriver incompatible.
- Is an API more expensive than a scraper?
On the subscription price, yes, because a scraper looks free. Adding up servers, maintenance hours and the data lost in every outage, an API for continuous data usually comes out cheaper.
Read also
- What an API is and how to connect to one for the first time
APIs explained without the jargon: what they are, how REST, JSON and keys work, and how to make your first call in the browser, the terminal, Python and JavaScript.
- What LynxMetrics is: subscription data APIs
Who we are, what we do and who for: LynxMetrics sells real-time data APIs by subscription, with per-API plans and keys and a free trial.
- Is the LynxMetrics API secure? Key, account and payments
How LynxMetrics protects API keys, your account and your payments, what you should do on your side so the key does not leak, and what to do if it does.