HTTP 429 Too Many Requests: How Rate Limits Work and How to Survive Them

429 behaves differently from every other error in a scraping stack, because the naive reaction makes it worse. A failed request that gets retried immediately looks, from the server's side, exactly like the abuse the rate limit exists to stop — so the window extends, the penalty escalates, and the client concludes the site is broken.

What the server is actually counting

Before touching your code, work out the scope of the limit. It is usually visible in one experiment: change one variable and see whether the limit follows.

Read the headers before writing any code

Well-behaved APIs tell you exactly what they want. The response usually carries more information than the status line.

• Retry-After — seconds, or an HTTP date. This is an instruction, not a suggestion.
• X-RateLimit-Limit — the ceiling for the current window.
• X-RateLimit-Remaining — what is left. Throttle *before* this reaches zero.
• X-RateLimit-Reset — when the window rolls over, usually a Unix timestamp.
• RateLimit and RateLimit-Policy — the newer standardised form; the same information in one line.

The best-tuned clients never see a 429 at all, because they watch Remaining and slow down as it approaches zero rather than waiting to be told off. Treat the 429 itself as a bug in your pacing rather than a normal part of the loop.

Backoff that actually works

Three properties separate a retry policy that recovers from one that entrenches the problem.

1. 1Exponential, not linear. Double the wait each attempt: 1s, 2s, 4s, 8s, with a sane ceiling.
2. 2Jittered. Add randomness. Without it, every worker that failed at the same moment retries at the same moment, and you recreate the spike that caused the limit.
3. 3Capped and surfaced. Give up after a fixed number of attempts and record it. A retry loop that never terminates hides a broken assumption for weeks.

When Retry-After is present it overrides all of this. The server has told you the exact moment it will listen again; arriving early is both rude and counterproductive.

Spreading load without looking like an attack

For IP-scoped limits the arithmetic is simple: if one address may make 60 requests a minute and you need 600, you need at least ten addresses and headroom on top. What matters is that each address behaves like a plausible client rather than a shard of one large machine.

• Keep a per-exit rate meaningfully below the observed ceiling, not at it.
• Vary the interval. Requests landing exactly every 900ms are a signature no human produces.
• Keep a session's cookies, headers and exit together. Rotating the IP mid-session while keeping the cookie is a strong bot signal — the trade-offs are in rotating vs sticky sessions.
• Separate read-only crawling from logged-in flows so a limit on one does not stall the other.

Rotating residential proxies fit the first case, where each request can come from a different consumer address. For anything that carries a session — carts, dashboards, queues — a static ISP proxy per worker keeps the identity stable while still letting you scale the number of identities.

429 that is not really 429

Not every rate limit uses the correct status code, and not every 429 is a rate limit.

That last row is worth guarding against explicitly. Validate the shape of successful responses, not just their status, or a throttled crawl will quietly write thousands of empty rows and look healthy the whole time.

A working policy in five lines

1. 1Track X-RateLimit-Remaining per identity and slow down at 20% left.
2. 2On 429, honour Retry-After; fall back to exponential backoff with jitter.
3. 3Pause the affected identity's queue, not the whole crawl.
4. 4Cap retries, then park the task and alert rather than looping.
5. 5Log status, identity, exit IP, endpoint and timestamp on every failure.

Rate limits are the one class of block where the polite client genuinely wins. Sites publish limits because they would rather you stay inside them than disappear behind a proxy pool, and staying inside them is almost always cheaper than the infrastructure needed to route around them.