Write a comment

This review reflects comments and contributions from Melissa Chim, Stephen Gabrielson, Dibyendu Roy Chowdhury, Pen-Yuan Hsing, and Kimberly Powell. Review synthesized by Stephen Gabrielson.

The study presents a cross-sectional survey investigating biomedical researchers' perspectives on the reproducibility of research, building upon a 2016 Nature survey on the same topic. The study aimed to explore researchers' perceptions and experiences surrounding reproducibility in the biomedical field.

Major comments:

• Given the 7% response rate, there is a potential risk that the individuals who responded differ significantly from those who didn’t. This might introduce non-response bias, which should be addressed. For instance, researchers concerned about reproducibility might be more likely to respond to a survey on the topic, skewing the results. Conversely, researchers not concerned or who have faced reproducibility challenges in their own work might avoid the survey, leading to an underrepresentation of these views.

• The authors aim to describe perceptions of reproducibility specifically in the biomedical community. However, the introduction offers examples of irreproducibility only in areas of education, psychology, and social sciences. Would recommend expanding the background/introduction/discussion to include other studies focused on reproducibility issues in biomedicine. The Reproducibility Project Cancer Biology may be a good place to start: https://elifesciences.org/collections/9b1e83d1/reproducibility-project-cancer-biology 

• Third paragraph of the discussion mentions “other stakeholders have taken actions to attempt to foster reproducibility” and note respondents perceived institutions as absent from these efforts. I’m concerned that respondents were not asked about perceptions/awareness of other stakeholder efforts. What other stakeholders are being considered? Does any evidence exist that respondents would have been more aware of those efforts? How does this claim interpret results from S4 that the majority of training resources respondents were aware of where based at the institutional-level?

Minor comments:

• The authors gave a broad definition of reproducibility near the beginning of this paper, and used it interchangeably with replicability in the rest of the text. There have been attempts to formally define reproducibility vs replicability. I suggest that the authors clarify their usage of these terms a bit more, or at least state that they use both interchangeably to mean the same thing for the purposes of this paper.

• When foundational studies are mentioned in the introduction, a quick definition and explanation of how they’re important for reproducibility would be useful.

• Can the 400 journal titles described in the sampling framework also be shared in Supplemental materials? Is the claim for exclusive focus on biomedicine based solely on MEDLINE-inclusion or where the sample titles are reviewed or mapped to specific subject domains?

• When mentioning Excel, software - even if well-known - should be formally cited. Same applies to the other software tools described.

• Could the approach for extracting author emails be published or shared in a protocol repository for greater visibility? I can see how others could benefit from working with the NLM catalog in a similar way for other kinds of meta-research.

• Were there any potential limitations of using Survey Monkey as the survey tool, which might have impacted the study?

• A detailed justification for the protocol amendments would be great. Manual extraction challenges are mentioned; however, the leap to semi-automating an entire year's extraction is a significant methodological pivot that requires clearer reasoning.

• How were the list of countries derived from respondents?

• Throughout the narrative references are made to the Nature survey, however, in Table 3 it is referred to as the Baker study- would recommend a consistency in referencing.

• Nature survey/Baker study presented results in very engaging donut and bar charts. I would recommend mirroring some of these efforts as they are easier for readers to interpret than table presentations. 

• As the emails were extracted and used for this study, it'd be great to address any data protection concerns, ensuring the study adheres to GDPR or other data protection regulations. Additionally, the email extraction method is labeled as "semi-automated." It would be beneficial to provide more information on this process to ensure the study is reproducible for other researchers. 

• Of the 24 training resources referred to in the section "Support for initiatives to enhance reproducibility", it appears that 17 (71%) are institutional resources. How might this be reflected on in paragraph 3 of the Discussion section which states "researchers perceive that institutions are absent from efforts to support... reproducibility"?

• In the Discussion section, it was very interesting regarding the lack of communication between researchers attempting to reproduce studies and the original study authors. Could the authors describe how researchers can communicate with each other, through private (email) and public channels (social media, preprints servers, etc)? A wider point is if and how this might relate to the highly competitive nature of biomedical research, where research teams are often hesitant to communicate to - let alone collaborate with or help - each other.

• The authors do a great job of showing the differences between this study and the survey from Nature.

Suggestions for future studies

• It would be of both intellectual and practical interest to consider how the study described here could be adapted to survey other research disciplines, including those that don’t follow such a strict experimental approach. What would reproducibility mean in those contexts, if relevant at all?

• Utilizing the RAND() function in Excel might not be the most robust method for random sampling given its pseudo-random nature. Future research could benefit from using more sophisticated software or programming language dedicated to scientific sampling.

• In the abstract, the authors state that "leading perceived cause of irreproducibility was a ‘pressure to publish’ with 62% of participants indicating it ‘always’ or ‘very often’ contributes." I didn't see this elaborated more in the article but would be interested in a discussion on this. Is this due to not enough time for people to share or publish research outputs other than articles? 

• Perhaps future studies could focus on outreach to early career researchers, who might have a very different perspective on reproducibility and open science than PIs who might be more mid-to-late career.

Competing interests

The author declares that they have no competing interests.