Manuscript–code consistency: versions, seeds, statistics

Why it matters

A reviewer asks for a one-sided test and the text changes, but the script does not. The data are downloaded again and a newer release of a reference database slips in. The paper says “averaged over five seeds” while the final script loops over three.

Each change is small, and each makes the published numbers harder to match.

What good looks like

• One command, such as make paper, that rebuilds every figure and table, supplementary ones included.
• Text and code revised together: a change to the Methods section comes with the commit that implements it.
• The manuscript names the exact code version (a tag or commit, ideally with an archive DOI) and data version (accession, release or retrieval date).
• Parameters, splits, exclusion rules, seed counts and statistical tests in the text match the scripts.
• Known differences, such as a bug fixed after submission, documented in the README.

How to fix it

Add a figure map at the repository root:

figure_2:
  command: snakemake --cores 4 results/figures/figure2.pdf
  script: workflow/scripts/plot_figure2.R
  inputs: [data/processed/counts.tsv, data/processed/metadata.tsv]
table_1:
  command: python -m analysis.table1 --config configs/table1.yaml
  output: results/tables/table1.csv
supplementary_figure_4:
  command: quarto render reports/supplement.qmd

Test reported values. In Python with pytest; in R, testthat::expect_equal() with a tolerance does the same:

import json

def test_table2_auroc_matches_paper():
    with open("results/claims/table2_auroc.json") as fh:
        result = json.load(fh)
    assert 0.90 <= result["mean"] <= 0.92  # the paper reports 0.91

State versions in the manuscript, for example:

All analyses used code release v1.2.0 (commit <hash>,
archived at <DOI>). Expression data were retrieved from
<repository> under accession <accession> on <date> and
processed with workflow/ at the same release.

Related

• Tracing numbers and figures to code⁠
• License, citation and paper version⁠
• Code availability statement template⁠