What Is Web Scraping?

Almost every price-comparison site, market-research report and recommendation engine you have used is built on data that was scraped from somewhere. The technique is ordinary; what varies is how carefully it is done.

How web scraping works

1. 1The scraper sends an HTTP request to a target URL, through a proxy if it is running at any scale.
2. 2The server returns the page's HTML — or JSON, if you found the underlying endpoint.
3. 3The scraper parses the response and extracts the fields it needs.
4. 4The data is validated, cleaned and stored in a CSV, a database or behind an API.
5. 5The process repeats across many pages, usually in parallel and on a schedule.

Steps one and four are where the real engineering lives. Step three — the parsing — is the part people expect to be hard and is usually the easiest to fix when it breaks.

Static pages versus rendered pages

One decision shapes the cost of everything downstream. If the data is present in the HTML the server returns, a plain HTTP client such as Python requests, httpx or Scrapy is enough, and it is fast and cheap. If the page builds itself in the browser with JavaScript, you need a real browser engine — Playwright, Puppeteer or Selenium — which costs roughly twenty times the bandwidth and far more CPU per page.

What web scraping is used for

• Price monitoring and competitor research — the largest single category by volume.
• SEO and SERP tracking, where results must be collected per country and per city.
• Lead generation and market research from public directories and listings.
• Aggregation in real estate, travel and jobs — entire businesses are built on it.
• Training and evaluating AI models, where source diversity is part of the requirement.
• Brand protection, spotting counterfeit listings and unauthorised resellers.

Why proxies are essential

Websites block scrapers that send many requests from one IP or from known datacenter ranges. Proxies — especially rotating residential proxies — spread requests across many real IPs so traffic looks like separate visitors, keeping success rates high and avoiding bans and CAPTCHAs.

The arithmetic is what makes this non-optional rather than merely helpful. If a target tolerates 300 requests per hour per IP and you need 60,000 an hour, no amount of clever code changes the fact that you need roughly 200 addresses working at once. One machine with one IP cannot do it, however well written.

Location is the second reason, and it is easy to overlook. Prices, availability, delivery estimates and search rankings all vary by where the request came from. Data collected from the wrong country is not slightly noisy — it describes a different market. See the locations we cover.

What it actually costs

For a mid-sized job — 200,000 pages a month at roughly 250 KB each — the proxy line is around €60 on residential bandwidth at €1.20 per GB. Blocking images and fonts in the fetcher typically halves that. Compute is usually smaller unless you are running headless browsers, in which case it dominates. The proxy comparison works through the same job across all three proxy types.

Is web scraping legal?

Broadly, collecting publicly available data is permitted in many jurisdictions. The clearest reference point in US law is *hiQ Labs v. LinkedIn*, in which the Ninth Circuit held in 2022 that the Computer Fraud and Abuse Act likely does not reach websites open to the public — there is no gate to climb over when the page is served to anyone with a browser.

That is narrower than it sounds. The same case later turned on LinkedIn's user agreement rather than the CFAA, which is the practical lesson: contract law, copyright and data-protection rules apply independently of anti-hacking statutes. Personal data in particular carries obligations under the GDPR regardless of whether the page was public. The legal overview goes into more detail, though none of it is legal advice.

Public availability is a defence against one specific claim, not a general permission. What you collect and what you do with it matter more than how you fetched it.

Scraping responsibly

1. 1Read `robots.txt` and treat it as a statement of intent even where it is not binding.
2. 2Use the API if there is one. It is more stable, cheaper and uncontroversial.
3. 3Pace requests so your traffic is a rounding error in the site's load, not a spike in it.
4. 4Cache aggressively. Re-fetching unchanged pages wastes their capacity and your bandwidth.
5. 5Identify yourself where you reasonably can, and honour requests to stop.
6. 6Collect only what you need, especially where personal data is involved.

Where to go next

If you are choosing infrastructure, start with the best proxies for web scraping. If your scraper already works and is starting to fail, go to how to avoid getting blocked. For a specific platform, the target guides cover what works on each.