AI Weekly Malaysia

Dharma-AI built a constraint-aware GPU allocator and benchmarked it against a FIFO scheduler across seven scenarios on identical hardware. GPU utilization rose by up to 33 percentage points and priority-weighted output rose by up to 105%, purely by changing the order of allocation decisions. The core problem is that batch-like workloads (training, batch inference, quantization) need contiguous uninterrupted GPU blocks while real-time inference is elastic and demand-driven, creating incompatible allocation shapes competing for the same GPUs in the same timestep.

Why: If you operate or pay for GPU clusters for mixed AI workloads, this post argues that scheduling order—not hardware—is the primary lever for utilization. The concrete takeaway: a FIFO scheduler with a fixed real-time inference reservation leaves significant GPU capacity stranded under contention, and a constraint-aware allocator that treats GPU-job-timestep as a binary grid can recover that capacity without buying more hardware.