How to Avoid Getting Blocked When Web Scraping

Blocks feel arbitrary until you understand that detection runs as a pipeline. Each layer is cheaper than the next, so sites evaluate them in order and stop as soon as something looks wrong. Debugging works the same way: fix layer one before touching layer four, or you will spend money on proxies to solve a header problem.

1. Rotate IPs with residential proxies

The single biggest factor is the IP. Use rotating residential proxies so each request, or each small batch, comes from a different real home IP. This prevents the rate limits and bans that hit a single repeating address, and it clears systems that filter datacenter ranges outright.

One caveat that catches people: rotating harder does not always help. If the site is scoring your request *shape* rather than your address, every new IP simply delivers the same suspicious request from somewhere new. If your block rate is unchanged after switching to residential, the problem is layer two.

2. Send realistic headers

This is where most scrapers are identified, and it is free to fix.

• Set a real, current User-Agent and keep it plausible — a Chrome version from three years ago is a signal in itself.
• Send the full set. Accept, Accept-Language, Accept-Encoding, Referer, and the Sec-Fetch-* and Sec-CH-UA headers modern browsers include.
• Match header order. Browsers send headers in a consistent order; most HTTP libraries do not. Order alone can distinguish a real Chrome from a library claiming to be one.
• Keep the set coherent. A Chrome user-agent paired with headers Chrome never sends is worse than an honest one.
• Do not rotate user-agents within one session. A browser that changes identity between two requests is more suspicious than one that stays boring.

TLS fingerprinting

The layer beneath headers, and the one that quietly defeats otherwise well-built scrapers. Every TLS client presents a distinctive combination of cipher suites, extensions and ordering — Chrome's differs from Python's requests, which differs from Go's standard library. Sites match against known signatures, so a request can be flagged before a single HTTP header is read.

You cannot fix this with headers. The options are to use a client that mimics a browser's TLS handshake, or to run an actual browser via Playwright or Puppeteer and accept the bandwidth cost.

If your scraper works from a browser and fails from a script against the same IP, stop adjusting headers. You are being identified below HTTP.

3. Pace and randomise requests

Add delays between requests and randomise them. Aggressive, perfectly timed bursts are an obvious bot signal — and so is a perfect two-second interval, because no human is that consistent. Draw from a distribution rather than a constant, and slow down more on error responses than on successes.

4. Use sessions where needed

For multi-step flows — login, pagination, carts — use sticky sessions to keep the same IP for the whole sequence, then rotate between sequences. For independent page fetches, rotate per request. Getting this backwards produces symptoms that look exactly like detection but are self-inflicted; rotating vs sticky sessions covers the diagnosis.

5. Target the right location

Use country or city targeting so the IP location matches the content you are collecting. This improves accuracy and looks natural — but the stronger reason is consistency. An Accept-Language of de-DE arriving from a Brazilian IP with a New York timezone is three signals that disagree, and disagreement is what gets scored.

6. Handle errors gracefully

1. 1Classify the failure. A timeout, a 429, a 403, a challenge page and a genuine 404 need different responses — the error code reference covers which is which.
2. 2Detect soft blocks by content, not status. A 200 carrying a challenge page is the most common modern block.
3. 3Back off exponentially and rotate the exit — but back off first. Retrying immediately on a fresh IP teaches the site that its rate limit is being evaded.
4. 4Cap retries and quarantine problem URLs rather than looping.
5. 5Log the exit IP with every failure. Without it you cannot tell a bad exit from a bad request.

Reading the symptoms

Do proxies stop CAPTCHAs?

They reduce them substantially, because a clean residential IP does not trip the reputation check that triggers most challenges. They do not eliminate them. A CAPTCHA appearing partway through a run is almost always a rate signal, not an IP signal — the answer is to slow down, not to buy better proxies.