AI Weekly Malaysia

Nvidia announced NeMo Switchyard, a software proxy that routes inference requests to different models based on cost, latency, or quality, claiming a 74% cost reduction versus using Claude Opus 4.8 alone with roughly a six-point accuracy tradeoff. Alongside it, Nvidia released Nemotron 3.5-30B-A3B-Lightning, a 30B-parameter MoE open-weights model for low-latency general use, and Nemotron Parse, a 1B-parameter model specialized in extracting context from PDFs including charts and tables.

Why: If you're paying for frontier-model API calls on every request, model routing lets you send trivial subtasks (title generation, summarization, PDF parsing) to cheaper or self-hosted models and reserve expensive models for the prompts that actually need them. The key insight is optimizing for completion cost, not per-token price—a model at 1/10th the token price that needs 10x tokens isn't cheaper. Evaluate whether a routing layer fits your stack before committing to a single provider.

antirez (creator of Redis) published h3.c, a native C implementation of MiniMax-H3 multimodal inference for Apple Silicon using Metal shaders. The project already supports end-to-end prompt-to-video/audio generation, first/last-frame conditioning, and ordered image/video/audio references, with current work focused on Metal performance and memory optimization on M3 Max and M5 Max.

Why: If you build AI-powered media generation features, this demonstrates a viable path to run a multimodal model entirely on-device with a single C binary and no Python runtime—relevant for teams wanting to avoid per-request cloud GPU costs or data residency concerns. The project's vertical-slice approach (metadata, Metal parity, prompt encoding, then full generation) is a useful reference architecture for anyone considering native local inference over API-dependent workflows.

Meta released Muse Glimmer, a 30-billion parameter open weights LLM distilled from its proprietary Muse Spark model — its first open weights release in over a year after Llama 4 flopped and its AI group was restructured. Released under Apache 2.0 with early support on Llama.cpp, Ollama, and Unsloth, it targets local inference workloads like agents and code assistants. Benchmarks show it beating Google Gemma 4 31B and trading blows with Alibaba Qwen 3.6-27B, but it's too small to challenge leading Chinese models like DeepSeek V4 Flash or Kimi K3.

Why: If you run local AI inference or build on-device agents, Muse Glimmer is now an Apache 2.0 option on Ollama and Llama.cpp worth benchmarking against Qwen 3.6-27B for your workload — but don't commit to it as a flagship given Qwen 3.8-27B is imminent and Meta's own larger Muse Spark remains proprietary. For Malaysian builders who care about local deployment (data sovereignty, latency, cost), a 30B model under a permissive license is practically runnable on a single high-end GPU.

OpenAI is previewing an 'Ultrafast' API tier for GPT-5.6 Sol that delivers up to 14× the speed of Standard processing, generating up to 750 output tokens per second. The service is powered by Cerebras inference hardware, marking a notable infrastructure partnership for OpenAI. It launches first via the OpenAI API.

Why: If you build latency-sensitive AI features (real-time agents, voice assistants, interactive copilots), 750 tokens/sec is a concrete threshold that could shift your architecture from streaming-with-spinners to near-instant full responses. The Cerebras partnership signals that non-NVIDIA inference silicon is reaching frontier-model production, which matters for cost and vendor-lock-in planning. Malaysian builders shipping API-based products should benchmark whether Ultrafast pricing justifies migrating workloads currently on Standard tier.

CoreWeave's Q2 2026 revenue doubled YoY to $2.575B, but operating expenses of $2.624B produced a $49M operating loss and $626M net loss, with total debt at $35.6B. 93% of revenue growth came from existing customers, and just three customers accounted for 72% of quarterly revenue. CEO Michael Intrator pitched AI compute as a continuous recurring loop (training, inference, evaluation, redeployment) rather than a one-time training cost, with managed inference services targeting $250M ARR by end of 2026.

Why: If you rent GPU capacity from neoclouds like CoreWeave, this signals pricing and service-model shifts ahead: they are pushing up-stack into managed inference and are financially stretched enough that contract terms or availability could change. The extreme customer concentration (three clients = 72% of revenue) and $35.6B debt mean builders should avoid single-provider lock-in for critical inference workloads and evaluate whether the 'continuous compute loop' framing matches their actual usage pattern before committing to long-term contracts.

The trycua/cua project published a blog post on using GPU passthrough in macOS VMs to accelerate llama.cpp-based LLM inference on Apple Silicon. The post details the technical approach for passing Apple GPU resources through to a virtualized macOS environment.

Why: If you run local LLM inference on Apple Silicon but need VM isolation for CI, agent sandboxes, or multi-tenant setups, this approach could let you keep near-native Metal GPU performance inside a VM rather than falling back to CPU-only inference. Builders evaluating local agent infrastructure should test whether the passthrough overhead is acceptable for their workload before committing to a bare-metal-only deployment.

Together AI signed a $240M deal with IBM Cloud to run its OpenAI-compatible inference platform on a large cluster of Nvidia HGX B300 GPU systems, launching Q1 2027. The B300 is a conventional air-cooled 8-GPU-per-box platform, not Nvidia's top-tier rack systems, but IBM had the capacity Together AI needed. Together AI also runs services on SambaNova's Intel-collaboration platform, showing it is hardware-agnostic so long as price-performance holds.

Why: If you use Together AI's inference or fine-tuning APIs, your workloads may soon run on IBM Cloud-hosted B300 GPUs—expect potential changes in latency, throughput, or regional routing when these go live in Q1 2027. For builders comparing inference providers, the real differentiator here is GPU supply availability, not just model selection or API compatibility.

OpenAI announced 'Ultrafast' mode for GPT 5.6 Sol, claiming 14x standard processing speed and up to 750 output tokens per second. The mode is powered by OpenAI's partnership with chipmaker Cerebras and is currently in preview for a small group of customers, with broader access promised as capacity grows.

Why: 750 tokens/second would enable genuinely real-time agent workflows (incident response, customer support, live financial analysis) that are impractical at current speeds. But since access is limited to a small preview group, builders cannot plan around this yet — monitor when it opens to API customers and evaluate whether your latency-bound use cases justify the likely premium pricing.

Cerebras shares fell nearly 20% after missing earnings expectations, with hardware sales declining even as its AI cloud revenue surged 281%. The divergence signals that demand for Cerebras's wafer-scale inference/cloud offering is growing faster than its chip-selling business.

Why: For builders evaluating inference infrastructure, Cerebras's 281% AI cloud revenue jump suggests its hosted inference service is gaining real traction — worth benchmarking against Groq and standard GPU clouds for latency-sensitive workloads. The hardware sales decline also signals that the pure chip-sales model for AI accelerators remains hard to sustain.

Nvidia released Nemotron 3.5 Lightning, an open-source AI model it describes as 'lightweight' and capable of running on a single GPU on a laptop or desktop. It's Nvidia's first open-source model since CEO Jensen Huang publicly defended open-source AI on X in late July, aligning with other tech leaders urging the U.S. government to support open models. The model is free for companies to download.

Why: If you're prototyping AI agents or local inference workflows, a single-GPU open-source model from Nvidia could reduce cloud dependency and cost—but the article gives no parameter count, benchmark scores, or license terms beyond 'free to download,' so evaluate the actual model card and license before committing. For Malaysian builders operating where GPU cloud capacity is scarce or expensive, a locally-runnable model is worth a test run, but don't assume production-readiness from a press release.

French startup Kog, founded solo by Gaël Delalleau, claims 30x faster LLM inference on conventional datacenter GPUs (AMD MI300X, NVIDIA H200) via software optimization. Its demo hit 3,000 tokens/second but only with a 2B-parameter model (Laneformer 2B, now open-sourced), and Kog admits customers won't fine-tune small models, so it is pivoting to accelerate larger models — a claim still unproven. The startup generated 200 business leads and is targeting software engineering workflows where Claude Code users wait hours for results.

Why: The 3,000 TPS demo is real but narrow — it runs on a 2B model, not the large models production teams actually use. Builders should treat the '30x faster' headline as aspirational until Kog shows results on production-scale models. The open-sourced Laneformer 2B is worth examining if you work on inference optimization, but don't change your serving stack based on this.