How to Scrape Google Search Results

SERP data drives rank tracking, competitive research, ad monitoring and a good deal of market analysis. Google offers no general search API for this, so collection means scraping — against one of the better-defended targets on the web.

Why the proxy choice decides everything

Google maintains excellent intelligence on IP ranges. Requests from known cloud and hosting ASNs are treated as automated more or less on arrival, which is why a scraper that works perfectly on your laptop returns nothing but consent pages and CAPTCHAs the moment it runs on a server.

Use rotating residential proxies with a fresh exit per request or per small batch. The proxy-for-scraping guide covers the wider selection logic; for Google the choice is effectively made for you. Mobile proxies will also work and tolerate more abuse, but at four times the cost per gigabyte they only pay for themselves if residential is genuinely failing.

Getting localisation right

Results differ by country, language and physical location, and a rank check against the wrong locale is worse than no data. The IP influences this, but the URL parameters control it.

Keep them consistent with the exit IP. A German IP with gl=us is contradictory, and while Google will answer, you have created data that does not correspond to what any real user sees. Match the proxy's geo-targeting to the parameters.

A working example

The mechanics are unremarkable — a session, a proxy, realistic headers, and generous jitter between requests.

In Python, that is a requests session pointed at the proxy endpoint, a browser-realistic User-Agent and Accept-Language matching hl, a randomised delay of a few seconds between calls, and a retry that backs off rather than hammers. Parse with selectorlib or BeautifulSoup, and treat every selector as optional.

1. 1Send the request through a rotating residential endpoint with headers that match the locale.
2. 2Detect the response type before parsing — results, consent page, or CAPTCHA.
3. 3Back off on anything that is not results, with increasing delay.
4. 4Parse organic results, ads, People Also Ask and related searches separately.
5. 5Validate the parse: a zero-result page for a common query means the layout changed, not that the query has no results.
6. 6Store the raw HTML alongside the parsed data so you can re-parse after a layout change.

That last point saves entire collection runs. When Google shifts its markup — and it will — re-parsing stored HTML costs nothing, while re-collecting costs bandwidth and time you may not have.

Reading CAPTCHAs correctly

A CAPTCHA is feedback. It means the pattern you are producing looks automated — usually too many requests per exit, too little variation in timing, or headers that do not match the claimed browser. The correct response is to reduce rate and increase exit diversity.

Solving CAPTCHAs treats the symptom and keeps the pattern that produced them. Fixing the rate removes both.

The anti-blocking guide covers header consistency and TLS fingerprinting in more depth. For Google specifically, the highest-yield changes are a lower per-exit rate, randomised intervals, and consistent header sets per session.

Costs and sizing

A SERP page is roughly 200–400 KB of HTML. Ten thousand queries a day at 300 KB is about 3 GB daily, or 90 GB a month — meaningful, and worth measuring before committing. Requesting num=20 halves the page count for a two-page-deep tracker at almost no extra size, which is the single easiest saving available.

The sizing guide has the full calculation, including how to convert a target's tolerated per-IP rate into the concurrency you need.

Legal and ethical notes

SERP results are publicly visible and scraping them is broadly accepted practice, though it conflicts with Google's terms of service. The legal overview sets out the distinction between contract, anti-hacking law and data protection. Be careful with results that contain personal data, and keep request rates proportionate.