google.com/goto: Google's anti-scraping update
- ID
- 23753
- Status
- summarized
- Published
- 12 Sep 2026, 11:14 AM
- Fetched
- 14 Sep 2026, 7:39 AM
- Provider
- Hacker News
- Category
- dev-community
- Original URL
- https://www.autom.dev/blog/google-search-goto-links
- Source URL
- https://hnrss.org/best
Summary
- Score
- 6.5
- Created
- 14 Sep 2026, 7:42 AM
- Tags
- Audience
- developersai_ml_learnerssaas_founders
What happened
Google Search is rewriting organic result links to google.com/goto?url=... instead of exposing destination URLs directly in HTML, as of late August 2026 for logged-out and private sessions. The url parameter is an opaque Google-specific encoding, not decodable offline; the real URL is only available in the Location header when you request the /goto URL without following the redirect. This forces each scraped result to make a round-trip back to Google, making bulk SERP scraping slower, noisier, and easier for Google to detect.
Why it matters
If you build anything that parses Google SERP HTML for URLs—search indexes, SEO tools, AI crawlers—you must now issue a HEAD request to each /goto link and read the Location header instead of parsing hrefs from the page. This multiplies your request volume by ~10x per SERP and gives Google a clear fingerprinting signal for sequential resolution. Anyone shipping SERP-dependent pipelines in Malaysia or elsewhere should audit their scrapers now or switch to an API that handles this.
Discussion angle
How does this change the economics of building search-based products on top of Google SERPs—and does it push builders toward official APIs or alternative search engines like Bing or Brave?