Hugging Face says a model predicts fine-tuning time more accurately
🎧 Listen to this story
In a preview of a paper not yet published on arXiv, Hugging Face analyzed 272 supervised fine-tuning (SFT) jobs run on its pool of NVIDIA H200 GPUs, totaling about 7,382 GPU-hours. The company says the estimate based only on FLOPs explained little of the observed durations (R² = 0.31; mean absolute error of 12.41 hours), while a CatBoost regressor reached R² = 0.98, a mean error of 1.71 hours and a median error of 0.51 hours in five-fold cross-validation.
The model uses information available before dispatch, including configuration, data size, model, GPU allocation, evaluation cadence and checkpoint policy. Hugging Face says FLOP calculations do not adequately capture tasks such as environment setup, loading, tokenization, evaluations, uploads and competition for network and storage in shared clusters.
Why it matters · editorial interpretation
If the results hold after the paper is published and validated, duration estimates that account for a job’s full configuration could help teams plan fine-tuning runs better than metrics based only on FLOPs. The study also highlights the operational impact of evaluations, checkpoints and shared resources on total runtime.
Sources
Research