Rewinding the exponential: How we keep hardware usage down
- ID
- 34927
- Status
- summarized
- Published
- 13 Oct 2026, 11:51 PM
- Fetched
- 14 Oct 2026, 1:58 AM
- Provider
- Cloudflare Blog
- Category
- infrastructure
- Original URL
- https://blog.cloudflare.com/rewinding-the-exponential/
- Source URL
- https://blog.cloudflare.com/rss/
Summary
- Score
- 6.5
- Created
- 14 Oct 2026, 1:59 AM
- Tags
- Audience
- developersdatabase_learnerssaas_founders
What happened
Cloudflare's Data team describes how its pipeline — Logfwdr into centralized receivers, then the Buftee buffer, then consumers writing to ClickHouse — was handling roughly 2 billion events per second as of May this year, while hardware growth stayed linear. Across five years they made five changes (breaking up the monolithic message schema in 2022, adjusting compression in 2024, rearchitecting pipeline replication in spring 2026, decentralizing sampling in summer 2026, and adding aggregations to the pipeline in fall 2026) that together deferred about two and a half years of hardware expansion. The excerpt only details the 2022 schema split, where a single stream carrying every product's fields was replaced with product-specific schemas that duplicate some fields but stop internal consumers from ingesting everything and filtering it out.
Why it matters
If you run high-volume ingestion into ClickHouse or a similar store, the concrete lever here is schema splitting: stop shipping one firehose message with every product's fields and make consumers demux it, and accept field duplication in exchange for not paying to move and store fields nobody reads. Cloudflare attributes roughly 2.5 years of deferred hardware spend to five such changes at 2B events/sec — but the post only explains the 2022 change in the text available, so treat the 2026 replication, sampling and aggregation items as named but unsubstantiated until the full post is read.
Discussion angle
Which of the five levers — schema splitting, compression tuning, replication rearchitecture, decentralized sampling, upstream aggregation — is actually adoptable at small scale, and at what data volume does each start paying for itself versus just adding operational complexity?