Reproducibility levels

The five levels

Today a check can establish L0, L1 and L2. L3 and L4 need evidence from running the code and are locked until RepoReady can produce it.

1. L4

   Robustness confirmed within tested scope

   Coming soon

   L3 plus a defined, accepted sensitivity-testing protocol. The conclusion is limited to the variations and targets actually tested.

2. L3

   Results reproduced

   Coming soon

   L2 plus independent execution and agreement with the declared results. A script finishing successfully is not enough.

3. L2

   Workflow documented

   Available now

   L1 plus complete instructions, versions, preparation and result mapping. Confirmed manuscript contradictions block this level.

4. L1

   Materials available

   Available now

   The necessary code, inputs and software are accounted for within the assessed scope, with evidence of availability or a credible access route.

5. L0

   Materials incomplete

   Confirmed material gap

   An essential material is confirmed missing, incomplete or unavailable within the assessed scope. Insufficient evidence alone does not establish L0.

When no level is set

How a level is awarded

Materials and documentation are assessed independently, and the report counts their requirements: confirmed, gaps, needs evidence, not assessed. A level is awarded only when its own requirements and every level below it are confirmed. There is no combined score.

Supporting results, such as supplementary figures, count one level down: a missing script for a supplementary figure blocks L2, not L1, and gaps in their documentation never block L2.

What a finding blocks

Each finding says what it blocks. A finding marked Needs evidence is not settled yet; it never lowers a level.

• Blocks L1A key result lacks its code, an input or the software it needs.
• Blocks L2The workflow behind a key result, or its agreement with the paper, falls short.
• Blocks automationRunning every step takes more than one locked, seeded command.
• WarningWorth fixing, but blocks no level: a placeholder link in the docs, commented-out steps, scripts or files nothing uses, an external service, an undefined environment for the results, parallel workers without reproducible random streams. Journal policy and good practice findings are always Warnings.

Review scopes

Each check runs at a review scope. A Materials check awards at most L1 and marks the L2 requirements “Not assessed in this scope”. A Materials and workflow check also assesses documentation and automation, and compares paper and code: the methods, parameters, statistics and reported numbers of every key result. It can award L2.

Levels and areas

Two areas decide the level: Materials for L1 and Documentation for L2. A confirmed contradiction with the manuscript also blocks L2. The other areas are reported beside the level.

What a check covers

A check reads the code, paper, data and documentation; the code is not executed. Analysis that starts from processed data does not establish how the raw data were prepared. A CSV can hold the scientific result: missing cosmetic plotting code alone is not a gap, but statistics inside a plotting script are part of the analysis.

Related frameworks

ACM and NISO recognise separate artifact and result properties; their badges are not a numbered ladder. These notes explain the overlap, not an equivalence or an external endorsement.

L4 is our proposed robustness extension, bound to an approved testing protocol. None of these frameworks offers a universal “robust results” tier that could simply be adopted.