AI and Tech Summaries - AI Weekly Malaysia

Summaries

Short AI and tech summaries with source links, signal scores, and why each update matters for builders, founders, and Malaysian tech workers.

Window

Today Yesterday This week Last week This month Last month All time

Showing 1-1 of 1 results

Date	Provider	Score	Summary
21 Oct 2026, 2:54 PM	Latent Space	7.0	[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback DeepSeek open-sourced DeepInfer, its production inference framework for DeepSeek-V4.1-Flash, released on day one of PyTorch Conf. It is a heavily modified vLLM fork (crediting SGLang and TRT-LLM) that supports up to 72-hour prompt caching versus the 12 hours OpenAI announced, but the release withholds DeepSeek's ultra-compact KV cache, prefill/decode disaggregation, and load balancing. DeepSeek's own translated figures claim 155T tokens/day per cluster, ~1.5M cache-inclusive input tokens/sec per prefill GPU with a 96% cache hit rate, 14,000 output tokens/sec per decode GPU across ~2,400 GPUs, and >80% margin at $5-8 all-in cost per card-hour against $0.15/$0.60 per million input/output tokens. Why: The 72-hour prompt caching is the part you can actually plan around: long-running agent sessions and multi-day RAG workflows stop paying repeated prefill costs, which changes how you'd design context retention versus rebuilding state each run. But do not benchmark your stack against DeepSeek's published throughput — the KV cache, PD disaggregation, and load balancing that produce those 14k tok/s decode numbers are explicitly not in the repo, so a vanilla DeepInfer deploy will land well short. Treat the $22m/day and 81-day payback figures as self-reported vendor economics, not a reproducible result. inference open-source vllm llm-infra prompt-caching cost-economics

Date

Provider

Score

Summary

21 Oct 2026, 2:54 PM

Latent Space

7.0

[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback

DeepSeek open-sourced DeepInfer, its production inference framework for DeepSeek-V4.1-Flash, released on day one of PyTorch Conf. It is a heavily modified vLLM fork (crediting SGLang and TRT-LLM) that supports up to 72-hour prompt caching versus the 12 hours OpenAI announced, but the release withholds DeepSeek's ultra-compact KV cache, prefill/decode disaggregation, and load balancing. DeepSeek's own translated figures claim 155T tokens/day per cluster, ~1.5M cache-inclusive input tokens/sec per prefill GPU with a 96% cache hit rate, 14,000 output tokens/sec per decode GPU across ~2,400 GPUs, and >80% margin at $5-8 all-in cost per card-hour against $0.15/$0.60 per million input/output tokens.

Why: The 72-hour prompt caching is the part you can actually plan around: long-running agent sessions and multi-day RAG workflows stop paying repeated prefill costs, which changes how you'd design context retention versus rebuilding state each run. But do not benchmark your stack against DeepSeek's published throughput — the KV cache, PD disaggregation, and load balancing that produce those 14k tok/s decode numbers are explicitly not in the repo, so a vanilla DeepInfer deploy will land well short. Treat the $22m/day and 81-day payback figures as self-reported vendor economics, not a reproducible result.