Tracing a paper’s numbers and figures to code

Why it matters

Readers rarely doubt a whole paper; they doubt a number. The useful answer is a path: this command, run on this input, wrote this file, which contains 0.91.

“Connect textual statements to underlying results” and “always store raw data behind plots” are among the Ten Simple Rules for Reproducible Computational Research. Figures that exist only as images, and metrics copied by hand from a terminal, have no such path.

What good looks like

• Reported numbers written to files by scripts, never copied by hand from a terminal or notebook output.
• Result files named after what the paper reports, such as table2_auroc.json or fig3_fold_change.csv.
• The data behind each figure saved next to the image, so values can be read without re-plotting.
• Rounding applied in the manuscript, while saved results keep full precision.
• Each result file notes the seeds and number of runs behind its value.

How to fix it

Python. Write each reported number to a file named after the claim, with its seeds and runs:

import json
from pathlib import Path
import numpy as np

seeds = [0, 1, 2, 3, 4]
aurocs = [evaluate(train_model(s), test_set) for s in seeds]  # your code

out = Path("results/claims/table2_auroc.json")
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(json.dumps({
    "claim": "Table 2, held-out cohort AUROC",
    "mean": round(float(np.mean(aurocs)), 3),
    "sd": round(float(np.std(aurocs, ddof=1)), 3),
    "n_runs": len(seeds),
    "seeds": seeds,
    "script": "analysis/evaluate_cohort.py",
}, indent=2))

R. Save the table behind each figure, not only the image:

fold_change <- summarise_fold_change(counts, design)   # your analysis
dir.create("results/figure3", recursive = TRUE, showWarnings = FALSE)
readr::write_csv(fold_change, "results/figure3/fold_change.csv")
ggplot2::ggsave("results/figure3/figure3.pdf", plot_fold_change(fold_change))

Point to the files. A “Results” table in the README with claim, command and output path gives readers a direct route.

Related

• Manuscript–code consistency⁠
• Experiment logging and traceability⁠
• Statistical soundness⁠