[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback
- ID
- 37787
- Status
- summarized
- Published
- 21 Oct 2026, 2:54 PM
- Fetched
- 23 Oct 2026, 6:04 PM
- Provider
- Latent Space
- Category
- developer-ai
- Original URL
- https://latent.space/p/ainews-deepseek-deepinfer-open-source/
- Source URL
- https://www.latent.space/feed
Summary
- Score
- 7.0
- Created
- 23 Oct 2026, 6:04 PM
- Tags
- Audience
- developersai_ml_learnersai_agent_userssaas_founders
What happened
DeepSeek open-sourced DeepInfer, its production inference framework for DeepSeek-V4.1-Flash, released on day one of PyTorch Conf. It is a heavily modified vLLM fork (crediting SGLang and TRT-LLM) that supports up to 72-hour prompt caching versus the 12 hours OpenAI announced, but the release withholds DeepSeek's ultra-compact KV cache, prefill/decode disaggregation, and load balancing. DeepSeek's own translated figures claim 155T tokens/day per cluster, ~1.5M cache-inclusive input tokens/sec per prefill GPU with a 96% cache hit rate, 14,000 output tokens/sec per decode GPU across ~2,400 GPUs, and >80% margin at $5-8 all-in cost per card-hour against $0.15/$0.60 per million input/output tokens.
Why it matters
The 72-hour prompt caching is the part you can actually plan around: long-running agent sessions and multi-day RAG workflows stop paying repeated prefill costs, which changes how you'd design context retention versus rebuilding state each run. But do not benchmark your stack against DeepSeek's published throughput — the KV cache, PD disaggregation, and load balancing that produce those 14k tok/s decode numbers are explicitly not in the repo, so a vanilla DeepInfer deploy will land well short. Treat the $22m/day and 81-day payback figures as self-reported vendor economics, not a reproducible result.
Discussion angle
Is a partial open-sourcing still useful? DeepSeek shipped the vLLM fork but kept the KV cache, PD disaggregation, and load balancer — so what can a team actually learn from DeepInfer that they couldn't get from upstream vLLM, and is 72-hour caching alone worth the migration?