PREreview of “Does it feel like a scientific paper?”: A qualitative analysis of preprint servers’ moderation and quality assurance processes

Summary

This study investigates the moderation processes of 13 preprint servers based on semi-structured interviews with 14 diverse representatives, uncovering a spectrum of practices shaped by local, disciplinary, and organizational contexts. It highlights key tensions in preprint moderation, such as balancing openness with quality assurance, and examines how these processes compare to peer review.

Positive feedback

Overall

“This is a very interesting and important study at an apt time when preprint servers are popular and funders are adopting preprint policies.”

Methodology

“The study includes good qualitative methods and in-depth interviews with key personnel across diverse preprint servers.”

Discussion

“I appreciate the balanced and thoughtful discussion and conclusion.”

Major issues

Data Presentation

Comment: “It is so hard to work out which servers are in which category referred to in the text, and some key information is buried in the supplemental file.”

Suggestion: Move key information, such as server classifications and moderation workflows, from supplemental materials into the main text. Summarize these in a table or visual format.

Clarification of Peer Review and Moderation Boundaries

Comment: “Some more discussion on where the boundary between [moderation and peer review] might lie would be useful.”

Suggestion: Discuss of how moderation criteria (e.g., scholarly content, novelty) align with or diverge from peer review.

Minor issues

Terminology Precision

Comment: “The authors use ‘scholarly content’ and ‘scientific research’ interchangeably.”

Suggestion: Define key terms explicitly and apply them consistently

Visualization of Findings

Comment: “The paper would benefit from a clear visualization of the moderation workflows across different servers.”

Suggestion: Add a flowchart or infographic to illustrate the moderation steps and criteria employed by various servers.

Scope Column for Tables

Comment: “Consider adding a scope column to Table 1 to clarify the disciplinary and/or geographical requirements of the preprint servers.”

Suggestion: Add a “scope” column to highlight these aspects in tables summarizing server characteristics.

Acronym

Comment: “What does STS mean?”

Suggestion: Spell out STS.

References

Comment: “Not in reference list.”

Suggestion: Add these parenthetical citations to References list:

(Ball 2021; Gibbons 1999) (Pereira and de Oliveira 2024; Swire-Thompson and  Lazer 2020).

Numbers don’t add up

Text: “Ten participants chose option 1, three chose option 2, and  zero chose option 3.”

Comment: “Did one participant decline to choose between the levels?”

Suggestion": Double-check the dataset for accuracy/report the missing participant’s choice.

Table 1 Clarity

Column 3: Country

Comment: “Is this the country where the preprint service is held or the owning organization?”

Suggestion: Clarify the meaning of “Country” in Table 1 caption.

Table from OSF Materials

5.3.1: Moderation Criteria with OSF link  https://osf.io/drtj6/. 

Comment: “The Moderation Workflows table is really helpful in understanding the discovery of the interviews. I would recommend embedding it into the article or highlighting the link more prominently.”

Suggestion: Make this edit as four others agreed with it. I, as the synthesizer, recommend using numbered lists and bullet points for columns 4-5 in that table to ease skimming.

Table Citations

Text: 5.3.2 … “Tables 3 and 4 (Appendix) ”

Comments: Should this be "Tables 2 and 3"? Is Table 4 (Appendix) referring to the Moderation Workflows supplement?”

Suggestion: Check this table citation and correct if needed.

Fairness

Text: 5.3.3. “It is therefore neither scalable…”

Comments: “Some consideration should be given here to the other modes of author verification not being applicable in this context. It's a unique solution for a reason and works well in a context where the other approaches listed may not. I think this sentence comes across as too dismissive of the approach of RINarxiv.”

Suggestion: Discuss the implications between co-authors to temper them if possible.

Align citation

Text: Appendix → “Table 2”

Comments: “No call out in main text”

Suggestions: add the Table 2 citation in the main body

Future research

Effectiveness of Moderation Practices

“A deeper study evaluating what moderation practices are the most effective…would be recommended to preprint servers.”

Author and Submission Outcomes

“I would love to know where the rejected preprints go (i.e., do they end up being published as-is somewhere?).”

Comparative Analysis with Peer Review

“A comparison between preprint server screening and initial journal submission screening would be really valuable.”

Inclusivity of Perspectives

“The study could have benefited from including perspectives of authors and users of preprint servers.”

Competing interests

Ashley Farley is a Gates Foundation employee leading the preprint policy and main manager of VeriXiv in partnership with F1000 . Theodora Bloom is Executive Editor at BMJ Publishing Group and co-Founder of the medRxiv preprint server. Martyn Rittman is the former Director of preprints.org, a preprint server. Jay Patel has completed a graduate-level course under the second author, is collaborating on Ph.D. research, and is co-mentored as a Ph.D. student by the second author.