Random seeds and determinism in research code

Why it matters

Train/test splits, weight initialization, bootstrap resampling, embeddings and MCMC all draw random numbers. Without a recorded seed, a reader who gets a slightly different result cannot tell noise from a mistake. Noting random seeds is rule 6 of the Ten Simple Rules for Reproducible Computational Research.

Seeds are not always enough: the PyTorch reproducibility notes warn that results may differ across releases and platforms, and between CPU and GPU, even with identical seeds.

What good looks like

• One seed, set near the entry point and passed to everything that draws random numbers: Python’s random, NumPy, PyTorch, scikit-learn and data-loader workers.
• The seed saved with each result and changeable from the command line or a config file.
• Notebooks and R Markdown or Quarto reports that set the seed in their first cell or chunk.
• Parallel jobs that use their own reproducible random streams, such as future.seed = TRUE in R.
• Variation that remains between runs, for example from GPU kernels, described in the README with its typical size.

How to fix it

Python and PyTorch. Take the seed from the command line and seed every generator in one place, data-loader workers included:

import argparse, random
import numpy as np
import torch

parser = argparse.ArgumentParser()
parser.add_argument("--seed", type=int, default=0)
args = parser.parse_args()

random.seed(args.seed)
np.random.seed(args.seed)
torch.manual_seed(args.seed)
torch.use_deterministic_algorithms(True)  # error on nondeterministic ops

def seed_worker(worker_id):
    worker_seed = torch.initial_seed() % 2**32
    np.random.seed(worker_seed)
    random.seed(worker_seed)

g = torch.Generator()
g.manual_seed(args.seed)
loader = torch.utils.data.DataLoader(
    dataset, num_workers=4,
    worker_init_fn=seed_worker, generator=g,
)

On GPUs, deterministic mode needs further settings, listed in the PyTorch notes. With scikit-learn, pass random_state=args.seed to every estimator and splitter.

R. Set the seed once at the top. For parallel code, future.seed = TRUE gives the same random numbers regardless of the number of workers:

args <- commandArgs(trailingOnly = TRUE)
seed <- if (length(args) > 0) as.integer(args[1]) else 42L
set.seed(seed)

library(future.apply)
future::plan(future::multisession, workers = 4)
boot <- future_lapply(1:1000, function(i) mean(sample(x, replace = TRUE)),
                      future.seed = TRUE)

Related

• Statistical soundness⁠
• Experiment logging and traceability⁠
• Pinned dependencies and containers⁠