AI Weekly Malaysia

A new paper from Multiverse Computing introduces two systems-level optimizations for LLM knowledge distillation: caching the teacher's top-K logits offline so the teacher model never needs to co-reside in VRAM with the student, and a fused chunked KL-divergence loss that avoids materializing the full vocabulary×sequence-length probability matrix. Together these cut VRAM usage far below default PyTorch or NVIDIA Megatron-Bridge implementations, making long-context distillation feasible on a single GPU instead of requiring hundreds.

Why: If you're distilling large open-source models (e.g., gpt-oss-120b with its 201,088-token vocabulary) into smaller deployable students, this approach lets you skip keeping the teacher loaded during training—potentially dropping your GPU footprint from a cluster to a single card. Evaluate the offline top-K logits caching and fused KL loss before your next distillation run, especially if you've been blocked by VRAM costs.

Writer launched Palmyra X6, a post-training variation of Z.ai's open source GLM-5.2, alongside upgrades to its agentic harness, claiming up to 50% cost cuts for basic tasks. Writer's own research found that harness efficiency changes reduced costs an average of 40% across multiple models, often more reliably than model choice itself.

Why: If you're shipping AI agents, the practical lever to pull may be your harness/orchestration layer, not just swapping models. Writer's finding that harness tweaks averaged 40% cost reductions across models suggests auditing your agent loop—prompt structure, tool-call patterns, token reuse—before paying for a pricier model.