Write a comment

This review is the result of a virtual, collaborative live review discussion on April 16th, 2025, organized and hosted by PREreview as part of the PREreview Champions Program 2025. The discussion was joined by 9 people: 2 facilitators and 7 cohort members of the PREreview 2025 Champions Program. We thank all participants who contributed to the discussion and made it possible for us to provide feedback on this preprint. The authors of this review have dedicated additional asynchronous time over the course of two weeks to help compose this final report using the notes from the Live Review. 

Summary

This study mainly addresses how reproducibility and replication are conceptualized and discussed within qualitative research and what factors and practices allow or hinder their application in qualitative research. These questions, clearly articulated on page 5 of the paper, reflect a critical and timely inquiry into how traditionally quantitative-oriented open science concepts intersect with qualitative research methodologies. The authors employ an integrative review method ideal for synthesizing empirical and theoretical literature to explore these themes. Drawing from 248 peer-reviewed articles and grey literature —including work from multiple disciplines such as health sciences and social sciences, which broadens the scope and relevance of the findings—and utilizing tools like Systematic Review Facility (SyRF) for data extraction and analysis, the sources were identified through searches in databases such as Scopus, Web of Science, Dimensions, PubMed, APA PsychInfo, and JSTOR, followed by snowball sampling from included references.  The main findings show that reproducibility and replication don't work the same in qualitative research as in quantitative research. Trying to copy a qualitative study exactly is often not possible. This is because qualitative research is more subjective, often depends on the context, and involves the researcher's role and perspective. The study also points out significant challenges, like ethical issues with sharing data (e.g., protecting participant privacy). However, the authors also highlight things that can help, like keeping good records of how research is done, being open about the research process, and adapting open science tools to fit qualitative research rather than rejecting reproducibility outright, the study argues for rethinking it in qualitative terms—emphasizing transparency, traceability, and positionality. In the end, the paper argues for a more flexible and fair version of open science that respects and supports the unique ways qualitative research works while also acknowledging, as a limitation, the exclusive focus on English-language sources, which may overlook relevant perspectives from non-English-speaking contexts.

List of major concerns and feedback

1. The review only includes English-language sources, which may exclude contributions from non-English-speaking communities.. It would strengthen the manuscript to explicitly address how this limitation may affect the global applicability of the findings, particularly in regions with different qualitative traditions. 

2. The review presents a well-justified and appropriate integrative methodology, offering valuable insight into open science practices in qualitative research. However, the decision to conduct full-text screening, data extraction, and coding with a single reviewer raises concerns about potential bias and consistency. To strengthen the study’s reliability, the authors are encouraged to clarify whether any strategies—such as double coding, consensus checks, or reflexive journaling—were used during the process. If no such measures were taken, a brief justification would help provide transparency and context for this methodological choice.

3. While the authors aim to adapt reproducibility and replicability concepts to qualitative research, the justification and framework for this adaptation remain underdeveloped. Readers would benefit from clearer guidance on how these concepts should be applied in various qualitative fields. For instance, you can include specific examples, such as how reproducibility might be approached in ethnography or thematic analysis, which would make the recommendations more practical and actionable.

4. The review may be biased toward the Western world, particularly Europe and North America, as it does not include articles representing different ethical or epistemological traditions, and while this is not necessarily inadequate, it may be important to acknowledge that the scope of the conclusions is not global.  

List of minor concerns and feedback

1. The authors describe the coding process at a high level. However, one could improve by briefly explaining how disagreements in coding were addressed, whether through discussion, consensus meetings, or cross-checking. Consider specifying whether any disagreement resolution methods were used and, if so, providing a brief description to enhance transparency and reproducibility.

2. The contrast between qualitative and quantitative research seems oversimplified. Revising this framing could better reflect the complexity of both approaches. A more nuanced discussion acknowledging the diversity and overlapping elements within both paradigms would improve the theoretical framing.

3. There is sample bias towards medical/health sciences (16.9%) and psychology (13.3%), fields where Open Science norms are more entrenched, while humanities and critical social sciences are underrepresented. It may be helpful to acknowledge this imbalance in the discussion or limitations sections and briefly reflect on how it could influence the findings or their generalizability.

Concluding remarks

This paper is valuable in contributing to the conversation about open science in qualitative research. It uses strong methods and gives a fair look at both the challenges and opportunities in making qualitative research more open and transparent. 

We thank the authors of the preprint for posting their work openly for feedback. We also thank all participants of the Live Review call for their time and for engaging in the lively discussion that generated this review.

Competing interests

The authors declare that they have no competing interests.