How we could save petabytes of cache storage with Zstandard and Pingora
- ID
- 20219
- Status
- summarized
- Published
- 01 Sep 2026, 8:59 PM
- Fetched
- 01 Sep 2026, 9:53 PM
- Provider
- Cloudflare Blog
- Category
- infrastructure
- Original URL
- https://blog.cloudflare.com/cache-transcoding/
- Source URL
- https://blog.cloudflare.com/rss/
Summary
- Score
- 6.5
- Created
- 01 Sep 2026, 9:53 PM
- Tags
- Audience
- developersdatabase_learners
What happened
Cloudflare prototyped a system called Cache Transcoding that encodes eligible cached assets with Zstandard (zstd level 3) inside Pingora before writing to disk, shrinking eligible assets to roughly 1/3 of their original on-disk size on average. The trade-off is a small one-time CPU cost at cache fill time, in exchange for ongoing storage and cross-data-center bandwidth savings. The prototype was built during an internship and is not yet a shipped product feature.
Why it matters
If you operate any caching layer or CDN-like infrastructure, this is a concrete data point that zstd level 3 can cut on-disk cache footprint by ~66% with minimal CPU overhead, and that compressing at cache-write time (rather than only honoring origin content-encoding) is worth evaluating. Builders running their own edge proxies or origin-facing caches should benchmark zstd against their current compression to see if the storage/bandwidth savings justify the CPU cost at their scale.
Discussion angle
The design choice to compress once at cache-fill and decompress on every client serve — when does that trade-off flip, and what does it imply for smaller teams running their own reverse proxies vs. relying on a CDN?