180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers
- ID
- 47917
- Status
- summarized
- Published
- 18 Nov 2026, 8:00 AM
- Fetched
- 19 Nov 2026, 1:59 AM
- Provider
- Hugging Face Blog
- Category
- developer-ai
- Original URL
- https://huggingface.co/blog/moe-expert-streaming
- Source URL
- https://huggingface.co/blog/feed.xml
Summary
- Score
- 7.8
- Created
- 19 Nov 2026, 1:59 AM
- Tags
- Audience
- developersvibe_codersai_ml_learnersai_agent_users
What happened
Tarek Ziadé's Hugging Face post experiments with running a 180B-parameter MoE model on a 32GB Mac by streaming experts from SSD, reporting that this approach gets within 6% of Strata on token generation. The post covers the author's setup—two RTX 4090s on Windows 11 plus an M5 MacBook with 32GB unified memory—and prior tests running Qwen 3.8 27B via llama.app and Qwen3.8-Flash-Next via Strata. It notes a 32GB M4 result of 13.5 tok/s, too slow for coding agents, and frames expert offload as a way to run MoE models that do not fit in unified memory.
Why it matters
If you run local coding agents, use the reported numbers as a decision threshold: a 180B MoE can fit with SSD expert streaming, but 13.5 tok/s on a 32GB M4 is not interactive for agent loops, so benchmark your own model and engine before treating extra VRAM as the only fix. The 6% gap to Strata also makes offload engines worth testing before upgrading a 32GB Mac.
Discussion angle
What token-per-second threshold makes a local 180B MoE good enough for coding-agent loops, and does SSD expert streaming change your buy-vs-upgrade decision for a 32GB Mac?