Scraping frameworks

How to use proxies with Python Requests

Requests routes traffic through a proxy via its proxies argument, which takes a dict mapping the http and https schemes to a proxy URL containing your credentials. Set it per request or once on a Session to reuse the same proxy and connection pool across calls.

Recommended for Python Requests

Residential proxies — Most Requests workloads are bulk collection against sites that inspect the network origin, which is exactly what rotating residential IPs are for.

See residential proxies

Setup

1
Install Requests
Install the library into your environment if it is not already present.
```
pip install requests
```
2
Build the proxy URL
Requests takes proxies as a dict keyed by scheme. Point both http and https at the same authenticated proxy URL. Replace PROXY_HOST, PROXY_PORT, USERNAME and PASSWORD with the endpoint and credentials shown in your seamless dashboard.
```
import requests

proxy = "http://USERNAME:PASSWORD@PROXY_HOST:PROXY_PORT"
proxies = {"http": proxy, "https": proxy}

response = requests.get("https://example.com", proxies=proxies, timeout=30)
print(response.status_code, response.text[:200])
```

Reuse a Session

A Session keeps the proxy configuration and the underlying TCP connections alive across requests, which is noticeably faster than configuring each call separately.

session = requests.Session()
session.proxies.update(proxies)
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
})

for url in urls:
    r = session.get(url, timeout=30)

Control rotation with sessions

Rotation happens at the gateway. Append a session identifier to the username to hold one IP across requests, or omit it to get a fresh IP on every call.

import uuid

# Sticky: same exit IP for every request that uses this identifier
sticky_user = f"USERNAME-session-{uuid.uuid4().hex[:8]}"
sticky = f"http://{sticky_user}:PASSWORD@PROXY_HOST:PROXY_PORT"

5
Handle failures and verify the exit IP
Retry on connection errors rather than aborting, and confirm the traffic is actually leaving through the proxy before you scale the job up.
```
r = requests.get("https://api.ipify.org?format=json", proxies=proxies, timeout=30)
print(r.json())  # should show the proxy's IP, not yours
```

Python Requests proxy FAQ

How do I use an authenticated proxy with Python Requests?

Put the credentials in the proxy URL itself, in the form http://username:password@host:port, and pass it in the proxies dict for both the http and https keys. Requests handles the Proxy-Authorization header for you.

Why does my HTTPS request fail with an SSL error through a proxy?

The https key in the proxies dict should normally still use the http:// scheme, because the proxy connection itself is plain HTTP and the TLS tunnel is established through it with CONNECT. Writing https://user:pass@host:port there is the most common cause of handshake errors.

How do I rotate IPs in Requests?

Rotation is handled by the proxy gateway rather than in your code. Send requests without a session identifier to get a new IP each time, or include one in the username to keep the same IP for as long as you need it.