Proxies in Python requests: Sessions, Rotation and the Mistakes That Cost Hours

This is the article for after the setup works. If you just need the working configuration, the Python requests integration page has it as a copy-paste block. What follows is the behaviour that shows up once the job runs for more than a few minutes.

The baseline, and why it looks wrong

import requests

proxy = "http://USERNAME:PASSWORD@PROXY_HOST:PROXY_PORT"
proxies = {"http": proxy, "https": proxy}

r = requests.get("https://api.ipify.org?format=json", proxies=proxies, timeout=30)
print(r.json())  # the exit IP, not yours

The dict key is the scheme of the destination, not of the proxy connection. For an HTTPS target, requests opens a plain connection to the proxy, issues CONNECT, and negotiates TLS with the target through the tunnel. The proxy hop itself is HTTP, so http:// is right. Writing https://user:pass@host:port under the https key tells requests to speak TLS to the proxy, which almost no proxy endpoint expects.

Credentials that contain punctuation

Generated proxy passwords routinely contain @, :, # and /, all of which are structural inside a URL. An unencoded @ splits the URL at the wrong point, so requests sends part of your password as a hostname and the proxy answers 407.

from urllib.parse import quote

user = quote("USERNAME", safe="")
password = quote("PASSWORD", safe="")
proxy = f"http://{user}:{password}@PROXY_HOST:PROXY_PORT"

Encode at the point the URL is built rather than trusting whatever put the value in your environment. A password that survives your shell may not survive a YAML file, a Docker env var and a CI secret store in sequence.

Sessions do more than tidy the code

A Session keeps a connection pool alive. Without one, every request re-establishes the TCP connection to the proxy and re-negotiates TLS with the target — and through a residential exit that handshake costs several hundred milliseconds before the target does any work at all.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(
    total=5,
    backoff_factor=1,               # 1s, 2s, 4s, 8s, 16s
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET", "HEAD"],
    respect_retry_after_header=True,
)

session = requests.Session()
session.proxies.update(proxies)
session.headers.update({
    "Accept-Language": "en-US,en;q=0.9",
})
session.mount("https://", HTTPAdapter(max_retries=retry, pool_maxsize=20))

status_forcelist covers the codes worth retrying automatically. Note what is absent: 403 and 407. Neither is transient — a 403 needs a different exit or a different request shape, and a 407 is a configuration fault that will fail identically five more times.

Rotation belongs at the gateway

Rotation is not something requests does. The exit is chosen by the proxy gateway based on the username you authenticate with, which means switching identity is a string change rather than a networking change.

import uuid

def sticky_session(user: str, password: str) -> requests.Session:
    """One worker, one exit IP, for as long as the identifier stays the same."""
    tag = uuid.uuid4().hex[:8]
    proxy = f"http://{user}-session-{tag}:{password}@PROXY_HOST:PROXY_PORT"
    s = requests.Session()
    s.proxies.update({"http": proxy, "https": proxy})
    return s

# Omit the session tag entirely and every request gets a fresh exit.

The choice between the two is the whole subject of rotating vs sticky sessions. The short version: anything that carries a cookie needs a sticky exit, and anything stateless should rotate.

SOCKS5 and the DNS leak nobody mentions

SOCKS support is an optional dependency: pip install "requests[socks]". Once installed, the scheme you choose decides where DNS resolution happens.

With socks5://, your resolver sees every hostname you visit even though the traffic itself is proxied. That defeats much of the point and, on geo-sensitive targets, resolves to the wrong CDN edge. The h suffix is one character and fixes both.

Verify the exit before you trust the run

def exit_ip(session: requests.Session) -> str:
    r = session.get("https://api.ipify.org?format=json", timeout=15)
    r.raise_for_status()
    return r.json()["ip"]

print(exit_ip(session))

Run this at worker startup and log the result. A misconfigured proxy that silently falls through to a direct connection produces a job that looks perfectly healthy right up until the target bans your office address — and the only evidence would have been this line.

Environment variables will override you

requests reads HTTP_PROXY, HTTPS_PROXY and NO_PROXY from the environment. On a machine where those are set — a corporate laptop, some CI images — traffic can route somewhere you did not intend. Pass trust_env=False on the session when you want the explicit configuration to be the only one in play.

session.trust_env = False  # ignore HTTP_PROXY / HTTPS_PROXY / NO_PROXY

A checklist worth keeping

1. 1Both dict keys use http://.
2. 2Credentials URL-encoded at build time.
3. 3One Session per worker, with a mounted retry adapter.
4. 4Timeouts on every call — requests has no default, and a hung socket waits forever.
5. 5socks5h:// if you use SOCKS at all.
6. 6Exit IP logged at startup and on every failure.
7. 7403 and 407 excluded from automatic retries.

None of this is exotic. It is the difference between a script that works on your machine and a crawler that runs unattended for a week, which is a much shorter distance than it looks from the first two lines.