Cutting our inference bill by 68% by being obsessive about optimization
- ID
- 37091
- Status
- summarized
- Published
- 20 Oct 2026, 3:12 AM
- Fetched
- 20 Oct 2026, 3:48 AM
- Provider
- Cloudflare Blog
- Category
- infrastructure
- Original URL
- https://blog.cloudflare.com/obsessive-inference-optimization/
- Source URL
- https://blog.cloudflare.com/rss/
Summary
- Score
- 8.0
- Created
- 20 Oct 2026, 3:48 AM
- Tags
- Audience
- developersai_ml_learnersai_agent_userssaas_founders
What happened
Cloudflare's AI code reviewer went from $1.19 per average review in April to $2.46 by late August, then dropped to $0.77 (down 68%) after a mid-September optimization pass. About 40% of the savings came from code changes to the reviewer itself and the rest from swapping models and tuning effort levels; median review time also fell from 4m 09s to 1m 32s. The post attributes the earlier rise mostly to model changes — moving all reviewers onto Anthropic models took cost per job from $1.38 to $2.03, and their internal standards reviewer on Claude Opus 4.7 went from about $0.25 to $0.88 per review.
Why it matters
Two concrete lessons for anyone shipping LLM features: cache hit rate alone is a misleading metric — theirs rose from 85.7% to 97% while tokens per review tripled from 0.91M to 2.99M, so total spend still climbed; and a 'better' model on one narrow job (the standards reviewer, $0.25 → $0.88) can quietly become pricier than your orchestrator. Instrument cost per unit of work with the tier mix held fixed, as they did, before you decide a model upgrade is worth it.
Discussion angle
How do you split an inference bill between 'code/architecture' fixes and 'model choice' fixes in your own stack — and would you accept a slower or cheaper model on a narrow reviewer job if it saved 3x, given they chose the pricier model because evals showed it performed better on that specific workload?