Scraping frameworks

How to use proxies with Scrapy

Scrapy reads the proxy from the request's meta dictionary under the proxy key. Set it in a downloader middleware so every request is routed automatically, rather than repeating the configuration in each spider.

Recommended for Scrapy

Residential proxies — Scrapy is built for volume, and volume is exactly what gets a single IP rate limited. A rotating residential pool spreads the load automatically.

See residential proxies

Setup

Create a proxy middleware

Add a downloader middleware that attaches the proxy to every outgoing request. Replace PROXY_HOST, PROXY_PORT, USERNAME and PASSWORD with the endpoint and credentials shown in your seamless dashboard.

# myproject/middlewares.py
import base64

class SeamlessProxyMiddleware:
    PROXY = "http://PROXY_HOST:PROXY_PORT"
    USER = "USERNAME"
    PASSWORD = "PASSWORD"

    def process_request(self, request, spider):
        request.meta["proxy"] = self.PROXY
        creds = f"{self.USER}:{self.PASSWORD}".encode()
        request.headers["Proxy-Authorization"] = b"Basic " + base64.b64encode(creds)

Enable it in settings

# myproject/settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.SeamlessProxyMiddleware": 350,
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 400,
}

Tune concurrency and politeness

Proxies remove the IP bottleneck but not the target's tolerance. Keep AUTOTHROTTLE on so the crawl backs off when the site starts to struggle.

CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 16
DOWNLOAD_TIMEOUT = 30

AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 8.0

RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 408]

Give each request its own exit IP

For maximum spread, generate a fresh session identifier per request so the gateway assigns a different IP to each one.

import uuid

def process_request(self, request, spider):
    user = f"{self.USER}-session-{uuid.uuid4().hex[:8]}"
    creds = f"{user}:{self.PASSWORD}".encode()
    request.meta["proxy"] = self.PROXY
    request.headers["Proxy-Authorization"] = b"Basic " + base64.b64encode(creds)

Scrapy proxy FAQ

How do I set a proxy in Scrapy?

Set request.meta['proxy'] to the proxy URL, and send credentials in a Proxy-Authorization header. Doing this in a downloader middleware applies it to every request in the project without touching individual spiders.

Why does Scrapy ignore credentials in the proxy URL?

Scrapy's HttpProxyMiddleware strips inline credentials from the meta proxy value. Set the Proxy-Authorization header explicitly with base64-encoded credentials instead.

Does Scrapy need a rotating proxy?

For anything beyond a small crawl, yes. Scrapy's concurrency means a single IP reaches a target's rate limit very quickly, so a rotating residential pool is normally what makes a large crawl finish at all.