Write a comment

This review reflects comments and contributions from Melissa Chim, Stephen Gabrielson, Allie Tatarian, Jessica Polka, and Dibyendu Roy Chowdhury. Review synthesized by Stephen Gabrielson.

This study investigates the landscape of medical education preprints on medRxiv. It aims to provide an overview of these preprints, focusing on their characteristic, impact, dissemination, and transition within the field of medical education.

Major comments:

• The study highlights the growing importance of preprints in medical education. While the conclusion in the paper provides a good summary of directions going forward for preprinting in this field, a more critical examination of the implications of these trends for educational policy and practice, especially given the lack of peer review in preprints, would be beneficial.

Minor comments:

• In the third paragraph of the Methods section, the authors note the limitations of using Genderize.io. A significant limitation of the program is missing, however: this program has trouble analyzing non-Western first names, as described in this article: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8485937. This Western bias should be pointed out. Also in the same paragraph, if the authors are acknowledging limitations to their approaches in this section, it may be helpful to note that Crossref matches are incomplete, eg: https://link.springer.com/article/10.1007/s11192-021-03900-7.

• I appreciated the section on Creative Commons licenses, as they're an important part of any discussion on open sources. Would the authors be able to comment on why more than half of preprints had no CC license? If preprinting becomes a larger part of scholarly communication training in the medical education field, an understanding of CC licenses will be an important aspect for researchers to learn.

• In the Authorship section, I would be interested in knowing the career stage of authors, although I recognize that this would be difficult to infer from preprint metadata alone. Is preprinting helping postdocs and junior faculty to get their work out there?

• In the last paragraph of the Discussion section, the authors discuss how preprints can be published in peer reviewed journals, and state that "Reports of this practice from outside of medical education suggest that between 14%⁴³ and 42%⁴⁴ of those papers deposited as preprints eventually get published in academic journals." These values seem low to me, see https://link.springer.com/article/10.1007/s11192-021-03900-7 again and also https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0281659.

• When explaining how preprints are not peer reviewed, it would be worth mentioning how medRxiv screens preprints before posting them, even if it's not as rigorous as peer review. https://connect.medrxiv.org/news/2022/06/13/screening_procedures.

Comments on reporting:

• The statistical analysis in this paper (as explained in the Results > Preprint Use section) does not acknowledge the existence of extreme outliers in their data - it appears that there are 2 or 3 medical education papers on medRxiv that were extremely popular, and their popularity has skewed the averages reported in this paper. I would like to see what the averages look like with those outliers removed, including the Altmetric and Twitter data. The upper portions of these ranges are all 12-14 standard deviations from the mean.

• In regards to the above, in the second paragraph in the Discussion section the authors describe page views from a 2018 study and compare it to the findings of this preprint, which report higher numbers. This would be a more convincing argument without the outliers that are skewing the averages. The 2018 study that is cited here does not include a range, but their standard deviation is much smaller than the standard deviation in this paper (580.7 vs 15,863.44), which makes me feel like their data is less likely to be strongly affected by extreme outliers. Specifically for PDF downloads, without the top two extreme outliers, the average number of PDF downloads goes down to 124.84--well under the average in the paper the authors have cited here.

Suggestions for future studies:

• A qualitative analysis of social media interactions around these preprints could provide interesting and deeper insights into public perception. 

• A future study might focus on medical education authors who reside outside the United States, United Kingdom, and Canada.

Competing interests

The author declares that they have no competing interests.