DeepInfer: High-performance inference framework for DeepSeek models
- ID
- 37345
- Status
- summarized
- Published
- 20 Oct 2026, 5:08 PM
- Fetched
- 21 Oct 2026, 3:50 AM
- Provider
- Hacker News
- Category
- dev-community
- Original URL
- https://github.com/deepseek-ai/DeepInfer
- Source URL
- https://hnrss.org/best
Summary
- Score
- 7.5
- Created
- 21 Oct 2026, 3:50 AM
- Tags
- Audience
- developersai_ml_learnersai_agent_userssaas_startup_founders
What happened
DeepSeek open-sourced DeepInfer, the inference framework it says powers its large-scale production services, under Apache-2.0. The release supports exactly one model (DeepSeek-V4.1-Flash) and offers offline inference only, bundling FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels, plus DSpark/confidence-scheduled speculative decoding (DEEPINFER_ENABLE_MTP_VARLEN=1, DEEPINFER_USE_NEXTN_DECODE default 5), SWA bounded replay (DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact, DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140), zero-overhead async batch scheduling on by default, batch-invariant inference and deterministic sampling. The repo shows 5.4k stars, 405 forks and only 3 commits, and the HN thread drew 1,962 points and 760 comments.
Why it matters
If you are self-hosting DeepSeek-V4.1-Flash, this is the vendor's own kernel stack rather than a reimplementation, so it is the baseline to benchmark against your vLLM/SGLang setup before migrating. The catch is scoped: one model, offline batch inference only, 3 commits, no serving/API layer — so treat it as a tuning reference and a benchmark target, not a drop-in production server, and check whether the 140-token SWA replay suffix and 5-token default draft length fit your prompt-length distribution. There is no Malaysian or SEA-specific angle in this release; local relevance is limited to cost of self-hosted inference capacity.
Discussion angle
5.4k stars against 3 commits: is publishing the production kernel stack with no serving layer and a single supported model genuinely useful to outside builders, or is it mostly a signal of DeepSeek's internal architecture — and what would you need (multi-model support, an HTTP front end, published benchmarks) before switching off vLLM or SGLang?