AI Weekly Malaysia

Back to items Summaries

Why your fine-tuning job takes longer than the FLOPs say

ID
47979
Status
summarized
Published
19 Nov 2026, 2:53 AM
Fetched
19 Nov 2026, 4:06 AM
Provider
Hugging Face Blog
Category
developer-ai
Original URL
https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time
Source URL
https://huggingface.co/blog/feed.xml

Summary

Score
7.5
Created
19 Nov 2026, 4:06 AM
Tags
Audience
developersai_ml_learners

What happened

A Dharma-AI team analysed 272 real supervised fine-tuning jobs from their production pool of NVIDIA H200 GPUs (~7,382 GPU-hours) to test how well the standard FLOP-based time estimate holds up. The closed-form 6·P·D / (N_GPU · FLOPS_peak · MFU) calculation explained little of the variance (R² = 0.31, MAE 12.41 h), while a CatBoost regressor using only pre-dispatch information reached R² = 0.98 with MAE 1.71 h and median error 0.51 h in 5-fold cross-validation. The gain comes from signals a FLOP formula cannot see: evaluation cadence, checkpoint policy, per-GPU load, and fine-tuning regime, plus the nonlinear interactions between them. This is a preview of their forthcoming paper, 'Predicting End-to-End Fine-Tuning Time for Generative Language Models'.

Why it matters

If you plan GPU spend or queue order with a FLOPs/MFU back-of-envelope, this dataset says your estimate is off by roughly 12 hours on average — enough to leave H200s idle or push every queued experiment back. The concrete fix is to start logging four things at job submission: evaluation cadence, checkpoint policy, per-GPU load at dispatch, and fine-tuning regime — those are the inputs that got the regressor to ~1.7 h MAE, and you cannot train that model later without them.

Discussion angle

Which of the four pre-dispatch signals (eval cadence, checkpoint policy, per-GPU load, fine-tuning regime) is easiest to capture in your current job-submission flow — and is a learned estimator worth building before you have a few hundred runs of your own?

Top