Proxies for AI Agents: Sessions, Sizing and What Gets Browsing Agents Blocked

Automated traffic now outweighs human traffic, and bot defences have scaled to match. An agent built on Playwright or a browser-use framework shows up in that picture looking a lot like the automation sites are trying to stop, which is why the first run usually works and the fiftieth does not.

Why agents get blocked differently from scrapers

The last row is the one people underestimate. A scraper that hits a block loses one request. An agent that hits a block on step nine of twelve loses the whole task and the model tokens spent getting there, so the proxy's job is less about volume and more about not failing mid-flow.

Which proxy type fits which agent

Session design: one task, one identity

Anti-bot systems correlate the IP with the cookie jar, the TLS fingerprint and the browser profile. Keeping those four aligned is what makes a session look like one person. The simplest way to guarantee it is to create all of them together when a task starts and throw them away together when it ends.

import uuid
from playwright.async_api import async_playwright

async def run_task(task):
    session = uuid.uuid4().hex[:10]
    proxy = {
        "server": "http://PROXY_HOST:PROXY_PORT",
        "username": f"USERNAME-country-{task.country}-session-{session}",
        "password": "PASSWORD",
    }
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        context = await browser.new_context(proxy=proxy, locale=task.locale)
        page = await context.new_page()
        try:
            return await task.agent.run(page)   # every step reuses this exit
        finally:
            await context.close()               # identity ends with the task
            await browser.close()

• Match locale and timezone to the exit country. A German IP with an en-US browser in a Pacific timezone is an easy inconsistency to catch.
• Set the sticky TTL longer than your slowest task. Agents pause for the model; an exit that rotates during a pause breaks the flow.
• On failure, rotate the whole identity — new session tag, new context — rather than retrying the same step on a new IP with old cookies.
• Never share one identity across concurrent tasks. Two agents interleaving on one IP look like one very erratic user.

The mechanics of sticky versus rotating exits are in rotating vs sticky sessions, and the Playwright specifics in Playwright proxies.

What not to route through the proxy

The agent's calls to its model endpoint — OpenAI, Anthropic, Google or a self-hosted model — should go direct. Those APIs rate-limit by key and account, not by address, so rotating IPs does nothing for a 429 from the model, and routing that traffic through a residential proxy only spends bandwidth and adds latency to every step. Proxy the browser; leave the model client alone.

Sizing bandwidth for agents

Agents run real browsers, so they pay for everything a page loads. The median page is around 2.5 MB, and most of that is images, video and fonts the model never reads. Blocking those types is the largest cost lever available.

BLOCK = {"image", "media", "font"}

async def slim(route):
    if route.request.resource_type in BLOCK:
        await route.abort()
    else:
        await route.continue_()

await context.route("**/*", slim)

These are planning figures, not guarantees — measure a hundred real tasks and scale from that. The general method is in how many proxies do I need.

Being a good citizen, and staying unblocked

The web is moving toward identifying agents rather than guessing at them. Site owners can now block or charge AI crawlers at the edge, and proposals such as Web Bot Auth let well-behaved agents sign their requests so sites can recognise them. Whatever the mechanism, the practices that keep an agent welcome are the same ones that keep it unblocked:

• Respect robots.txt and terms of service for the sites your agent visits, and stay away from anything behind a login you are not authorised to use.
• Pace like a person. One agent per identity, human-scale delays, no parallel hammering of one domain.
• Prefer official APIs where they exist; they are faster, cheaper and stable across redesigns.
• Keep personal data out of scope unless you have a lawful basis to process it. See are proxies legal for the wider picture.

When the agent gets blocked anyway

Treat the response code as a diagnosis, not a retry signal. A 403 or Cloudflare 1020 means a rule matched something about the identity — change the whole identity, not just the IP. A 429 or 1015 means the agent is too fast for that site; slow it down. A CAPTCHA mid-flow usually means the session changed underneath the agent, which points back at session design.