How to use proxies with HTTPX

Setup

1. 1

   Install HTTPX

   Install the library, optionally with HTTP/2 support.

   pip install 'httpx[http2]'

2. 2

   Create a client with a proxy

   Pass the proxy URL when constructing the client so every request through it is routed. Replace PROXY_HOST, PROXY_PORT, USERNAME and PASSWORD with the endpoint and credentials shown in your seamless dashboard.

   import httpx
   
   proxy = "http://USERNAME:PASSWORD@PROXY_HOST:PROXY_PORT"
   
   with httpx.Client(proxy=proxy, timeout=30.0) as client:
       r = client.get("https://api.ipify.org?format=json")
       print(r.json())

3. 3

   Go async for throughput

   An async client with bounded concurrency is where HTTPX earns its keep. Keep the semaphore modest so you do not overwhelm the target.

   import asyncio, httpx
   
   async def fetch_all(urls):
       limits = httpx.Limits(max_connections=50)
       sem = asyncio.Semaphore(20)
   
       async with httpx.AsyncClient(proxy=proxy, limits=limits, timeout=30.0) as client:
           async def one(url):
               async with sem:
                   return await client.get(url)
           return await asyncio.gather(*(one(u) for u in urls), return_exceptions=True)

4. 4

   Rotate per worker

   Give each concurrent worker its own session identifier so they do not all share a single exit IP.

   import uuid
   
   def client_for_worker():
       session = uuid.uuid4().hex[:8]
       url = f"http://USERNAME-session-{session}:PASSWORD@PROXY_HOST:PROXY_PORT"
       return httpx.AsyncClient(proxy=url, timeout=30.0)

HTTPX proxy FAQ