Creepy Crawlies
- ID
- 19629
- Status
- summarized
- Published
- 30 Aug 2026, 1:49 AM
- Fetched
- 01 Sep 2026, 1:36 AM
- Provider
- Hacker News
- Category
- dev-community
- Original URL
- https://people.kernel.org/monsieuricon/creepy-crawlies
- Source URL
- https://hnrss.org/best
Summary
- Score
- 7.5
- Created
- 01 Sep 2026, 1:39 AM
- Tags
- Audience
- developersvibe_codersdatabase_learnersai_ml_learnerssaas_founders
What happened
Konstantin Ryabitsev reports that AI training scrapers now consume more CPU on git.kernel.org than all legitimate access combined, including git clones. At any given time, 14 CPU cores across 5 geo-distributed nodes are dedicated solely to rendering git commits as HTML for scrapers, who are hitting billions of valid cgit URLs to re-fetch 922 duplicate forks of the same 1.48 million Linux commits instead of simply cloning the repos.
Why it matters
If you operate any public git infrastructure or web-facing content with on-demand rendering (cgit, Gitea, GitLab web views), expect aggressive AI crawler traffic to become a baseline cost line item. Block or rate-limit non-clone HTTP access paths and serve a static robots.txt that disallows cgit-style URL patterns, because the crawlers are ignoring efficient clone endpoints in favor of expensive per-commit HTML rendering.
Discussion angle
The AI industry's training-data hunger is now imposing real, measurable infrastructure costs on open-source maintainers who provide free services — should public code-hosting infrastructure start requiring authentication or aggressive rate-limiting by default, and what does that mean for the openness that made these resources valuable in the first place?