Scraping frameworks

How to use proxies with BeautifulSoup

BeautifulSoup parses HTML but does not fetch it, so the proxy is configured on whatever client retrieves the page — usually Requests or HTTPX — and BeautifulSoup simply parses what comes back.

Recommended for BeautifulSoup

Residential proxies — BeautifulSoup work is almost always page collection at volume, which is where rotating residential IPs prevent blocks.

See residential proxies

Setup

1
Install both libraries
BeautifulSoup needs a fetching library alongside it; Requests is the usual pairing.
```
pip install beautifulsoup4 requests lxml
```

Fetch through the proxy, then parse

The proxy belongs on the request, not on the parser. Replace PROXY_HOST, PROXY_PORT, USERNAME and PASSWORD with the endpoint and credentials shown in your seamless dashboard.

import requests
from bs4 import BeautifulSoup

proxy = "http://USERNAME:PASSWORD@PROXY_HOST:PROXY_PORT"
proxies = {"http": proxy, "https": proxy}

html = requests.get("https://example.com", proxies=proxies, timeout=30).text
soup = BeautifulSoup(html, "lxml")

for item in soup.select(".product"):
    print(item.select_one(".title").get_text(strip=True))

Reuse a session across pages

A Session carries the proxy and connection pool across a multi-page crawl instead of reconnecting each time.

session = requests.Session()
session.proxies.update(proxies)

for page in range(1, 20):
    html = session.get(f"https://example.com/list?page={page}", timeout=30).text
    soup = BeautifulSoup(html, "lxml")

4
Check whether the page needs JavaScript
If the data is missing from the parsed HTML, the site renders it client-side and no proxy will change that. Look for the underlying JSON endpoint, or switch to Playwright.

BeautifulSoup proxy FAQ

How do I use a proxy with BeautifulSoup?

You do not configure a proxy on BeautifulSoup itself — it only parses HTML you already have. Set the proxy on the library that fetches the page, such as Requests or HTTPX, and pass the resulting HTML to BeautifulSoup.

Why is BeautifulSoup returning empty results?

Usually because the content is rendered by JavaScript after the initial HTML loads, so it is genuinely not in what you fetched. Check the raw response first; if the data is absent, use the site's underlying JSON endpoint or a browser automation tool.

Do I need rotating proxies for BeautifulSoup?

It depends entirely on volume. A handful of pages needs nothing special; thousands of pages from one IP will be rate limited, and rotation is what avoids that.