AI Weekly Malaysia

Summaries

Short AI and tech summaries with source links, signal scores, and why each update matters for builders, founders, and Malaysian tech workers.

Reset

Showing 1-1 of 1 results

DateProviderScoreSummary
19 Nov 2026, 2:53 AMHugging Face Blog7.5 Why your fine-tuning job takes longer than the FLOPs say

A Dharma-AI team analysed 272 real supervised fine-tuning jobs from their production pool of NVIDIA H200 GPUs (~7,382 GPU-hours) to test how well the standard FLOP-based time estimate holds up. The closed-form 6·P·D / (N_GPU · FLOPS_peak · MFU) calculation explained little of the variance (R² = 0.31, MAE 12.41 h), while a CatBoost regressor using only pre-dispatch information reached R² = 0.98 with MAE 1.71 h and median error 0.51 h in 5-fold cross-validation. The gain comes from signals a FLOP formula cannot see: evaluation cadence, checkpoint policy, per-GPU load, and fine-tuning regime, plus the nonlinear interactions between them. This is a preview of their forthcoming paper, 'Predicting End-to-End Fine-Tuning Time for Generative Language Models'.

Why: If you plan GPU spend or queue order with a FLOPs/MFU back-of-envelope, this dataset says your estimate is off by roughly 12 hours on average — enough to leave H200s idle or push every queued experiment back. The concrete fix is to start logging four things at job submission: evaluation cadence, checkpoint policy, per-GPU load at dispatch, and fine-tuning regime — those are the inputs that got the regressor to ~1.7 h MAE, and you cannot train that model later without them.

Top