Skip to content
RepoReady
Sign up
Menu
Docs menu

Paper–code alignment: is the method implemented?

A paper can describe a method, baselines and ablations while the repository holds only the model. Make every part of the method findable and every comparison fair.

Last updated

In your report

Area
Manuscript
Check
Paper–Code Alignment

Why it matters

Released research code is often a subset of what was run: the preprocessing lived in a private notebook, or the baselines are missing. Readers then cannot tell whether a gap between your numbers and theirs comes from the method or from a missing step.

Comparisons need particular care. Henderson et al. illustrate how reported metrics and results vary when methods are compared against common baselines (Deep Reinforcement Learning that Matters, AAAI 2018).

What good looks like

  • Every step in the Methods section has code, including preprocessing and post-processing that began in notebooks.
  • Baselines that go through the same splits, preprocessing, tuning budget and seeds as your method.
  • Ablations derived from the full configuration, each changing a single setting.
  • Hyperparameter search spaces and the values you chose saved in the repository.
  • Anything that cannot be shared, such as licensed data or models, named in the README with the reason.

How to fix it

Share splits. Create the folds once, save them, and have every model read the same file. In R:

R
set.seed(2026)
folds <- rsample::vfold_cv(cohort, v = 5, strata = outcome)
saveRDS(folds, "data/processed/cv_folds.rds")

# every model script, your method and baselines alike:
folds <- readRDS("data/processed/cv_folds.rds")

Derive ablations from the full config, so only one setting changes. In Python:

python
import copy
import yaml

with open("configs/full_model.yaml") as fh:
    base = yaml.safe_load(fh)

ablations = {
    "no_attention":    ("model", "use_attention", False),
    "no_augmentation": ("data", "augment", False),
}
for name, (section, key, value) in ablations.items():
    cfg = copy.deepcopy(base)
    cfg[section][key] = value  # exactly one change per variant
    with open(f"configs/ablation_{name}.yaml", "w") as fh:
        yaml.safe_dump(cfg, fh)

Document the mapping. A “Relationship to the paper” section in the README lists each section and figure with its script, and names anything not released.

See this check on your repository

Add your manuscript to an analysis to get findings for this check, with suggested fixes.

Start free