Skip to PREreview

PREreview of Knowledge and motivations of training in peer review: an international cross-sectional survey

by Gary S McDowell
Published
DOI
10.5281/zenodo.7293736
License
CC BY 4.0

Overall Summary:

This preprint describes quantitative and qualitative results of a survey on perceptions and motivations in peer review administered to a selected sample of recent corresponding authors of medical journal articles. 8.6% of those invited responded and completed >80% of the survey. Most respondents:

  • are male
  • are faculty
  • have published more than 20 articles
  • indicate they had never received formal training in peer review
  • agree that reviewers should receive formal training prior to being a reviewer
  • state that it is difficult finding and/or accessing training in peer review

This is a useful and well-carried out piece of work which is extremely helpful for continuing research into attitudes surrounding peer review, particularly training, motivations and incentives. It includes important recommendations about how current peer review recruitment should consider motivation and training, e.g. “Currently, journal editors may spend time emailing numerous potentially untrained peer reviewers when they could instead contact a smaller number of highly trained and motivated peer reviewers.”

Major comments:

The aim of the survey is stated as being “to provide an up-to-date perspective of international biomedical researchers' views on peer review training.” The sample of respondents is a limiting factor in how far such an analysis is possible, and so my main critiques are about the makeup of the sample, and particularly whether the authors have put these limitations into sufficient context, which overall I think could use further work:

  • It may be worth discussing the solicitation of responses in more detail from the specific subset of journals that were classified as “Medical”. I couldn’t find this list of journals in the preprint or associated documents - it would be helpful to have this to hand if it isn’t currently available, in order to more clearly define which “biomedical” researchers were included, or excluded. It may be that this is a particular subset of biomedical disciplines if this dataset has a particularly medical bias, or focuses on journals that whole disciplines of biomedicine would never consider submissions to. This could then provide a disciplinary bias to attitudes, and indeed “medical” vs “biomedical” populations can have differing opinions on publication processes (e.g. reflected in having different preprint servers, where many biomedical researchers may exclusively use bioRxiv).
  • The journal question is also important for the “international” part - my assumption going into reading this preprint was that this would tend to be dominated by responses from North America and Western Europe, and the data reflect this somewhat (but not entirely), so clarification on the extent of this geographical bias would be helpful. Particularly as there seems to be more diversity than I would have guessed, a breakdown into Global North vs Global South would be helpful. In addition, there is a bias introduced by the authors - the exclusion of journals based on language (namely English) is worth highlighting here as an introduced potential bias that could affect geographic diversity.
  • Another bias is “In instances where the journal was not open access and we did not have access via our academic institution, we replaced the journal with another randomly selected journal.” - this is therefore a sample of journals that is accessible by a specific institution, and it may be worth commenting on how often this occurred, as it has the possibility of selecting for a certain kind of journal, and therefore, a certain kind of author. This is a fair limitation based on legality of access to data. However, could the authors perhaps explain further whether the data they needed (i.e. names and corresponding author email) was paywalled? Why was this a particularly difficult hurdle to overcome? Explaining the barrier to the reader would be helpful.
  • The emails used for invitations to the survey were those for the corresponding authors. This introduces bias as I would assume it is more likely to select for faculty - or, it is likely to select for whoever is culturally permitted to be corresponding author, which can vary by discipline. This enrichment for faculty seems to be reflected in the data. While this is very useful information - this is certainly an interesting population to study - it also affects the claim that this is a survey of "biomedical researchers" per se, and so clarification on this would be appreciated. This would be helpful as it also has the potential to affect other demographics based on known biases in publishing, such as gender. But the enrichment for faculty is a strength of this dataset in my opinion, as feedback on peer review from those who are senior figures in scholarly publishing is important, and indeed is more likely to provide data about journal practices, as senior figures are perhaps more likely to have involvement with journals. This did not seem to be articulated as an intended strength here, though, but I believe it should be seen and communicated as such.

I was a little unclear on what the responses to the questions on participation in peer review may be telling us. For instance, “How many articles have your peer reviewed in the last 12 months?” does not specify if the respondent was an invited reviewer, or a co-reviewer. The possible confusion arising from this question may have affected some of the data: for example the number of respondents to the question “For how many years have you been active as a manuscript peer reviewer?” does not appear to correspond with how many respondents say they have never been involved in review in the previous question. I found the high level of participation of biomedical researchers in peer review surprising, until I considered some of the biases above e.g. that the sample is enriched for faculty. But I still found it surprising (if the question is interpreted as having been an invited reviewer, or involved in some serious way in peer review) that the number of masters, graduate students, postdocs etc reported in the dataset appear to be participants. That is, again, until I considered that if someone in these roles is allowed to be a corresponding author (because there can be controversies and arguments about who gets this position), they are likely to be in a particularly privileged position in their lab compared to many of their peers. All in all, this still makes the data very interesting - but I would urge providing greater context about who this population is, and who they are likely to reflect. I would claim that they reflect a particularly privileged group of researchers, and are therefore not reflective of the typical biomedical researcher. Selecting for corresponding authors is a good strategy for engaging with those who are allowed access to/invited to peer review - but this is a biased and privileged group generally, which may affect broader generalizations about the experiences of all biomedical researchers. This is also before considering that these are the authors who were ultimately successful in publishing in these journals which, depending on the perceived competitiveness of the journals, may affect the population further.

Another bias that is not discussed in the preprint is that the responses will be affected by who was motivated to respond, which may be reflected in/affect the opinion-based questions. For example, the people who responded may have been motivated to do so precisely because they think peer review is so important, rather than there being a general consensus that this is the case. These points may require more discussion. But this makes the correlating opinions interesting too, especially in light of the population that has been invited to respond, and so the data are very insightful, but require more context.

The methods are particularly clear and the full recruitment email and survey texts have been included in the preprint, which is very helpful for those wishing to take prompts forward in future surveys for potential comparisons. The authors registered the study protocol prior to carrying out the work which is a particular strength. In addition I appreciate the transparency in data sharing and that anonymized data, and the codebook, are included at the Open Science Framework Project page for this work: https://osf.io/wgxc2/

Minor comments:

One comment regarding the requirement for training - I have observed there are some journals (e.g. biological society journals) who do not require all reviewers to be trained, but rather are forcing early career researchers specifically to go through training before being added to the reviewer database, whereas faculty reviewers are not made to go through training. Was such a nuance, or something like it ever reported in this dataset (it seems that there are opportunities for respondents to volunteer that information, but they aren’t explicitly prompted to do so)? Any discrimination in who is "required" to be trained, would be interesting. It would also be interesting to know if there are journals that do require training of everyone who wants to be a reviewer at that journal.

Based on some of my comments above, is there anything notable about the population (n=80) who are involved with journals themselves, with respect to their responses elsewhere in the survey? This insight into the role of editors may be helpful, especially when compared with work such as Hamilton’s (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7717900/). The extent to which this subset and the rest of the cohort agree could be interesting to look at.

On the author's recommendations: while peer review does indeed take place at journals, and so there should well be an expectation that they provide training, it may be important to consider also that generic peer review training happen as part of undergraduate or graduate training at institutions. Journals are particularly focused on curation; but curation goes beyond evaluation of the validity of research, or scholarly discourse about it. I would therefore be interested to see future discussions focused on which stakeholders may be best placed to carry out training about peer review, and the wider benefits that a more devolved training landscape may have. For example, most of the undergraduates we are carrying out our peer review education with will not go to graduate school or do peer review for journals, but they will encounter science in their everyday lives in a vast array of roles. An ability to engage with and critique research seems a very important skill to learn at undergraduate levels in STEM, and pre-medicine, and is part of the case we are making for the need for peer review education separate from the specific need for labor at journals. Similar arguments could be made for the importance of journal-agnostic peer review training, which indeed could perhaps address some critiques of the way current reviewers are perceived to be approaching their reviewing tasks.

Finally - I’ve found this work useful in rethinking some ongoing projects of my own. I’ve provided some examples here, to illustrate some of the contexts where I see this work will be important/some thoughts the authors brought up that I share in the hope that they are of interest or use:

  • I’ve also been involved in work specifically assessing the role of early career researchers in peer review, particularly the hidden role in ghostwriting. We did a literature review of the evidence for training in peer review which is summarized in an appendix in our preprint (https://www.biorxiv.org/content/10.1101/617373v1.full.pdf+html) and it had not occurred to me until reading this work that there was a lack of data specifically about online or virtual training. So this work (and the work in the authors’ related preprint on online training, https://www.medrxiv.org/content/10.1101/2022.09.02.22279345v3) are important for bringing out the need for more data about this.
  • We've looked at the role that training plays as a motivation for co-reviewing (albeit with unintended consequences for hiding the labor of ECRs through ghostwriting (https://elifesciences.org/articles/48425)). Our work finds, like yours, that most ECRs do not receive formal training while believing it to be something people should acquire - it’s therefore helpful to see in this work that this perspective is also shared by authors in senior/faculty roles, and also potentially editorial roles.
  • Also, given that the authors are mapping out future work in these spaces, I will add here that we have published somewhat related recommendations specific to ghostwriting (https://www.molbiolcell.org/doi/full/10.1091/mbc.E20-10-0642) that may be supportive - or indeed, based on the author’s data, contradictory - to recommendations. In particular I will add that we have somewhat modified our positions, and indeed are more urgent of recognition of anyone as a “peer” (and are using this as a basis for undergraduate training, see https://onlinelibrary.wiley.com/doi/10.1002/leap.1472 for framing, and preliminary outcomes are published in https://www.biorxiv.org/content/10.1101/2022.10.06.511170v1.full). In particular, I note in the discussion that the authors address issues relating to peer review guidance that are specific to medical research; it may be worth considering (for us all) whether there are ways to parse out recommendations that are generic for “peer review”, and then also separately highlight discipline-specific components. For example, our ghostwriting recommendations are quite specific to “biomedicine” (likely more basic science rather than translational) - and we know that ghostwriting occurs at different frequencies across disciplines, being virtually non-existent in some while pervasive in biomedicine.

Conflicts:

I have no conflicts to report; I do not know the authors personally and have not been involved in any way with the work. I do not stand to gain or suffer financially or otherwise from this publication.

License:

This review is published under a CC-BY license.