Skip to content
RepoReady
Sign up
Menu
Docs menu

Statistical soundness: seeds, variance and checkpoints

One run or five, early stopping on validation or test data, macro- or micro-averaged F1: each choice changes reported numbers. Make them explicit in code and paper.

Last updated

In your report

Area
Manuscript
Check
Statistical Soundness

Why it matters

A single run can land at the lucky end of the distribution, a model stopped at its best test-set epoch has seen the test set, and micro- and macro-averaged F1 differ on imbalanced classes.

Kapoor and Narayanan list 17 fields where data leakage errors have been found, collectively affecting 329 papers (Leakage and the Reproducibility Crisis in ML-based Science). The NeurIPS paper checklist asks authors to state the factors of variability their error bars capture.

What good looks like

  • The test set used once, at the end: early stopping, tuning and checkpoint choice rely on validation data.
  • Scaling, imputation and feature selection fitted on training data only.
  • Results that depend on randomness repeated over several seeds or folds, with mean and spread.
  • Error bars that say what they show (SD, SE or confidence interval) and over what: seeds, folds or bootstrap samples.
  • Averaging of each metric (micro, macro or weighted) set in code and named in the paper.

How to fix it

Python. Select on validation data, evaluate the test set once per seed, and report mean and SD with explicit averaging:

python
import numpy as np
from sklearn.metrics import f1_score

SEEDS = [0, 1, 2, 3, 4]
scores = []
for seed in SEEDS:
    model = train(train_set, val_set, seed=seed,   # your training loop:
                  early_stopping_on="val_loss")    # validation split only
    y_pred = model.predict(test_set.X)
    scores.append(f1_score(test_set.y, y_pred, average="macro"))

print(f"macro-F1 = {np.mean(scores):.3f} ± {np.std(scores, ddof=1):.3f} "
      f"(SD over {len(SEEDS)} seeds)")

R. Repeat over seeds and report the spread next to the mean:

R
seeds <- 1:5
auc <- vapply(seeds, function(s) {
  set.seed(s)
  fit_and_score(train, test)   # your model: fit on train, score on test
}, numeric(1))
sprintf("AUC %.3f (SD %.3f, %d seeds)", mean(auc), sd(auc), length(seeds))

In the manuscript, state the number of runs, what the error bars show, the checkpoint rule and the averaging of each metric.

See this check on your repository

Add your manuscript to an analysis to get findings for this check, with suggested fixes.

Start free