Why your fine-tuning job takes longer than the FLOPs say
- ID
- 47979
- Status
- summarized
- Published
- 19 Nov 2026, 2:53 AM
- Fetched
- 19 Nov 2026, 4:06 AM
- Provider
- Hugging Face Blog
- Category
- developer-ai
- Original URL
- https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time
- Source URL
- https://huggingface.co/blog/feed.xml
Summary
- Score
- 7.5
- Created
- 19 Nov 2026, 4:06 AM
- Tags
- Audience
- developersai_ml_learners
What happened
A Dharma-AI team analysed 272 real supervised fine-tuning jobs from their production pool of NVIDIA H200 GPUs (~7,382 GPU-hours) to test how well the standard FLOP-based time estimate holds up. The closed-form 6·P·D / (N_GPU · FLOPS_peak · MFU) calculation explained little of the variance (R² = 0.31, MAE 12.41 h), while a CatBoost regressor using only pre-dispatch information reached R² = 0.98 with MAE 1.71 h and median error 0.51 h in 5-fold cross-validation. The gain comes from signals a FLOP formula cannot see: evaluation cadence, checkpoint policy, per-GPU load, and fine-tuning regime, plus the nonlinear interactions between them. This is a preview of their forthcoming paper, 'Predicting End-to-End Fine-Tuning Time for Generative Language Models'.
Why it matters
If you plan GPU spend or queue order with a FLOPs/MFU back-of-envelope, this dataset says your estimate is off by roughly 12 hours on average — enough to leave H200s idle or push every queued experiment back. The concrete fix is to start logging four things at job submission: evaluation cadence, checkpoint policy, per-GPU load at dispatch, and fine-tuning regime — those are the inputs that got the regressor to ~1.7 h MAE, and you cannot train that model later without them.
Discussion angle
Which of the four pre-dispatch signals (eval cadence, checkpoint policy, per-GPU load, fine-tuning regime) is easiest to capture in your current job-submission flow — and is a learned estimator worth building before you have a few hundred runs of your own?