Write a comment

Open Review

Manuscript title: The Reproducibility Promotion Plan for Funders - A co created set of recommendations to foster reproducible research practices

Invitation to Review date: 14.05.2026

Review submission date: 01.06.2026

Review type: original submission / revision / other

Link to this review: https://osf.io/pbnw7/files/3vyek

I have reviewed the manuscript “The Reproducibility Promotion Plan for Funders” after being invited by the journal Royal Society Open Science. I have not looked at the manuscript submitted to RSOS but I am instead reviewing the version that the authors posted publicly at https://doi.org/10.31222/osf.io/7b6mp_v1, which RSOS may use. I do so because I think that scientific discourse should happen in the open and not behind closed doors. I have chosen to review the manuscript because I agree with the baseline (“reproducibility remains insufficiently prioritized within the scientific agenda”) and I strive to support a cultural shift towards reproducibility myself.

The authors describe how they systematically conducted workshops with representatives from research and funders to develop a set of recommendations to incentivize practices that positively affect reproducibility. I took approximately two hours to read the manuscript once and write the review. I skimmed some of the additional materials.

Evaluation: I was very happy to learn about the Reproducible Research Practices recommendations and think that they are extremely useful. In my opinion, incorporating the topic of reproducibility and finding a balance between innovation and verification should be the top priority for publishers at the moment and the recommendations brought forward are a great start.

Recommendation: I recommend that you clarify the terminology and the motivation. For example, you do not mention Data Management Plans, which I think are a very important step into the direction of reproducible research and you do not mention any of the initiatives who have spent thousands of hours conducting reproduction studies. At the same time, you mention a lack of infrastructure. I would be happy if a revised version of the manuscript more clearly identified the role that funders, researchers, institutions, and publishers play and should play to support reproducibility.

Remarks in no specific order

1. I recommend that you more clearly define reproducibility since there is a fair amount of confusion around the terms replicability and reproducibility in particular (e.g., in economics, researchers generally talk of replication, for example as replication packages or replication games but mean reproduction). Relevant literature may be https://doi.org/10.5281/zenodo.15213042 or https://doi.org/10.31222/osf.io/mqfp4_v1 or https://doi.org/10.1007/s42803-023-00073-y or a book chapter that I co-authored on the distinction at https://forrt.org/replication-handbook/understanding.html. You write later (p.18) that it is difficult to define reproducibility in a meaningful way but having discussed the topics with researchers from many different fields, I think it is easily feasible and much easier than for example replicability (which is sometimes not a goal in fields that have a social-constructivist understanding of truth).

2. “Only then can they promote and facilitate a change in research culture which values transparency, openness, and reproducibility.” I suggest that you clarify these values a bit. For example, there can be a trade-off between openness and reproducibility when openness to other disciplines’ ideals and requirement of reproducibility contradict each other (e.g., when a high-performance computer or special commercial software is required to check for reproducibility). Funders should anticipate this conflict and provide researchers with guidance on how to navigate it.

3. Preregistration on the OSF: I recommend that link the actual preregistration (10.17605/OSF.IO/69XD5) instead of linking to the file on the OSF (https://osf.io/3fpbj/files/tuz62). From the link, it is not apparent that you updated the preregistration. Files can also be changed but registrations are frozen. I also suggest that you paste links as text instead of linking via words (e.g., as per the APA recommendations for citing websites) so that they will also be visible to people reading the printed-out version.

4. I very much appreciate you sharing links to the materials (e.g., https://osf.io/rdq3g/files/5ynhj). I recommend that, before publication, you register the project and replace all links with permanent links to the frozen version. This will ensure long-term availability of the materials.

5. Given that your article’s scope is beyond psychology, I recommend that you define terms such as open data and open methods (p. 5) or reproducibility vs. replicability (p. 17). I think Open Methods are particularly blurry since data can be part of the methods and methods are copyrighted and cannot be shared.

6. Whenever a study is preregistered, all deviations should be discussed (description, reason, how they affect the results). Also, please discuss the update of the preregistration (timing, changes, reasons).

7. Regarding infrastructure: There is very useful infrastructure that allows for community-sourced reproducibility checking at CODECHECK (codecheck.org.uk), which some universities, conferences, and journals are already implementing. I recommend that you mention it in your discussion about burdens on p. 18. While the Replication Wiki has been offline for more than a year, other organizations such as Institute for Replication, The Replication Network (https://replicationnetwork.com/replication-studies/) and (although in a much smaller capacity as of know) the FORRT Replication Hub with things such as the Replication Journal Federation (https://forrt.org/rjf/) are developing and providing infrastructure [COI: I am a member of the RJF].

8. Regarding collaboration between funders and publishers (p. 19): While I appreciate you discussing this, I think you should put in context, that many funders have been lavishing money upon commercial publishers with open access fees (https://arxiv.org/abs/2511.04820). I think that funders should instead ask researchers to spend money reasonably (e.g. APCs max US$400, https://f1000research.com/articles/10-20). Also, prestigious journals’ publishing strategies and both publishers’ and funders’ urge for innovation is one of the causes of low rates of replicability – something that I recommend you consider discussing. Various diamond Open Access journals are already doing far better jobs than any commercially published journal, most notably Meta-Psychology and Journal of Comments and Replications in Economics, who both do regular reproducibility checks for all accepted articles (both diamond OA). I would very much appreciate contextualizing such calls and treating them more carefully, e.g., clearly distinguishing between for-profit publishers and scholarly-led publishers and explaining why to coordinate with whom.

9. While I agree that a one-size-fits-all approach is neither feasible nor useful, I think there are some things that are widely applicable and should be part of the RPP, such as Data and Code Sharing (as much as possible within legal and ethical restrictions) and a definition of data, which should not be problematic given that data management plans are already popular. I suggest that you discuss the relevance of shared data for reproducibility (only if data is shared can anybody check reproducibility).

10. With organizations such as the Institute for Replication having conducted hundreds of reproduction studies, I think you should mention in the introduction a few estimates of reproducibility rates. If these rates were very high, no further resources should be spent on incentivizing reproducibility.

11. Regarding hosting the RPP on GitHub (p. 24): I strongly recommend using a non-commercial host that also has a long-term archiving policy such as Zenodo or OSF.

12. I recommend that you add an AI use statement (i.e., explain whether you used AI for the research and if so, specify the model and the tasks that were assisted).

Disclaimers

13. I did not check for deviations from the preregistrations.

14. Publication of this review: I have published this review at on my OSF reviews project and Pubpeer.

15. I have potential conflicts of interest since I am the managing director of an institutional open science center and an editor of a reproduction- and replication-focused diamond open access journal. I very much care for responsible spending of tax money, which creates a conflict of interest with writing reviews for commercial journals.

Recommendations for the journal

16. I recommend that another reviewer who is experienced with qualitative research also reviews the manuscript. I am not an expert on qualitative research.

17. I recommend that you ask the authors to create a frozen version of the OSF project to prevent data from being altered or deleted or archive all materials via the journal.

Competing interests

[copied from the review:] I have potential conflicts of interest since I am the managing director of an institutional open science center and an editor of a reproduction- and replication-focused diamond open access journal. I very much care for responsible spending of tax money, which creates a conflict of interest with writing reviews for commercial journals.

Use of Artificial Intelligence (AI)

The author declares that they did not use generative AI to come up with new ideas for their review.