PREreview of Tracking the development of COVID-19 related PsyArXiv preprints

A review of the manuscript ‘Tracking the development of COVID-19-related PsyArXiv preprints

Reviewer: Olmo van den Akker

In this paper, the authors investigate what happened to COVID-19 preprints posted on PsyArXiv and compared them on several metrics to published articles about COVID-19 and to PsyArXiv preprints not about COVID-19. The authors find that COVID-19 preprints were more likely to be published and received more citations than non-COVID-19 preprint, although they reported fewer open science practices. Aside from the fact that I could not access the preregistration, I could not find any large issues with the manuscript. I do think some sections could be clearer, and the discussion needs to be more solution oriented. Also, the confound of Google Scholar vs. Web of Science citations should get a bigger role in a revised manuscript. Please find my comments below.

Major comments

-          I was not able to access the preregistration because it was not public. I requested access and have now received it. However, the private status of the preregistration remains an issue given that all readers should be able to compare the preregistration and paper to assess the validity of the study, not only reviewers. At a minimum, it is important to provide a list of things that changed between preregistration and paper in the manuscript itself.

-          The discussion needs to be more solution-oriented. I give examples of this below.

-          p.22 -> “While these metrics produce a comparable number of citations, Google Scholar tends to find more citations (Kulkarni, 2009).” -> I think this limitation is glossed over too quickly. It might even be that this factor explains the full effect of articles vs. preprints. I think it is essential to discuss this more here, and also present the result with more nuance in the discussion section and the abstract.

Minor comments

p.1 -> “Primary affiliations for all article types predominantly originated from Western countries, but this was comparatively more for preprints (both related to and not related to COVID-19), even though preprints had more international authorship teams than journal articles.” -> The terminology ‘article types’ confused me a bit here so I would stick to using the terms journal articles and preprints, or perhaps use ‘both’ article types instead of ‘all’.

p.1 -> “Overall, the results demonstrate that some of the structural problems in research are still in play despite the global effort to mobilise research efforts during the pandemic.” -> Please be more concrete about which structural problems, and perhaps use a sentence to describe what the implications are.

p.2 -> “While cognitive and behavioural sciences can increase our understanding of the world around us, some scholars are sceptical that certain social and behavioural sciences are advanced enough to deal with policy problems during crises that concern life-or-death questions.” -> I’d like to see a reference to the scholars/papers that are skeptical.

p.3 -> “Researchers submit their findings in a manuscript, which is then reviewed by experts in the field (“peer review”) and undergoes an often-lengthy revision process before it is finally published.” -> often-lengthy should be changed to often lengthy.

p.3 -> “Thus, there is a need for timely knowledge production during times of crisis” -> More specifically, the COVID-19 crisis as that is what the section is about.

p.5 -> “This trend is likely to be found in psychology as well, as COVID-19-related research has been cited on average eight times more than research on other topics between 2020 and 2021 (Ioannidis et al., 2022).” -> I don’t understand how the reference supports the presumption that the trend is likely to be found in psychology as well.

p.5 -> “There appears, prima facie, to be a tension between the need for knowledge production in a crisis to be both rapid and rigorous.” -> You could consider citing https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10427813, in which this claim is also made and assessed with regard to errors in the literature.

p.5 -> “However, one should not conflate “rigour” with the journal peer-review process” -> I think it would be good to reword this a little. Perhaps say something in line of : “However, one should not assume that the journal peer review process automatically leads to more rigorous studies.”

p.5 -> “and retractions of articles reporting rapid COVID-19 research occurred even in a high-profile journal such as The Lancet (The Editors if the Lancet Group, 2020)” -> Typo: The Editors of the Lancet Group, 2020

p.6 -> “We approached this question by examining the traction and impact of PsyArXiV COVID-related preprints compared to other types of papers” -> It is not entirely clear to me what the difference is between traction and impact. Maybe it would be good to define these terms.

p.6 -> “An alternate measure of quality is the implementation of scientific practices to promote the replicability of research.” -> Although I am a big proponent of open science I do not think is is necessarily an indicator of quality. There are ample examples of studies that are transparent but poor. The main benefit of transparency is that it makes it possible to evaluate the rigor of studies, not that it improves the rigor of studies per se. That being said, I think the authors could circumvent going in-depth about this topic and instead simply use the term transparency instead of quality.

p.6 -> “Open science approaches, including transparency, reproducibility, and cooperation, can assure the validity and reliability of research findings, advance scientific progress, and inform decision-making in the epidemic context.” -> What are these “approaches”? Reproducibility is not really an approach in my opinion, and it is unclear what transparency and cooperation mean exacylly. I think it would be best to discuss several concrete approaches and not assume a homogeneous effect on science.

p.7 -> “Non-compliance with public health measures, e.g., social restrictions, are therefore not necessarily a sign of bad faith, but a response to the disproportionate impact on these groups, which needs to be considered as well (Lewandowsky et al., 2022).” -> I am not sure that they are “not necessarily a sign of bad faith”. It seems to me that it is exactly what that is: a lack of faith in the government.

p.7 -> “In the context of psychology, the top six psychological journals feature participant samples overwhelmingly from western countries, comprising 96% of all participants (Tindle, 2021).” -> I would steer away from claims about what the top journals are as there is no consensus about this. Instead, I would only specify the operationalization Tindle used.

p.8 -> “These findings underscore the need for greater scrutiny and concern about publication bias, research output from non-WEIRD countries, and its impact on psychological research. Specifically, it is important to understand if this overrepresentation was exacerbated compared to non-crisis levels.” -> Please elaborate why these findings necessitate better insight about these issues. Perhaps a concrete example would help.

p.8 -> “We created two comparison groups of (i) non-COVID-related PsyArXiv preprints published between January 2020 and 19 May 2020 (n = 167); (ii) COVID-related journal articles published up to 19 May 2020, as indexed by Web of Science and belonging to the category “psychology” (n = 70).” -> I think one or two sentences explaining why you chose these comparison groups would be valuable. It provides some more background on what exactly you want to know from this study.

p.9 -> “We removed 7 duplicates and preprints that were later withdrawn during the analysis period (initial sample n = 218; final sample n = 211).” -> What does “preprints that were later withdrawn during the analysis period” mean exactly?

p.10 -> “For the COVID-related preprints, we searched for the preprint on Google Scholar on 2 March 2021 to record whether the preprint had been published in a journal by this time, and if so, the publication date and the total number of citations for preprint and published versions.” -> How did you do this exactly? How do you make sure that you did not miss any given that it is possible that the titles of papers change between preprint and paper.

p.10 -> “International authorship teams. For each paper, we established the location of all the authors’ primary institutional affiliation and recorded an international authorship team when there were at least two authors based at institutions in different countries.” -> It seems to me that bigger countries are more likely to have mono-country authorship (because there is less need to go abroad to get the necessary expertise). Could this have skewed the results? If so, how?

p.11 -> “We used two indicators of open science practices: whether authors had pre-registered the methods and analysis and/or shared their data and materials in an open repository (at least one of the two indicators was required).” -> Why a binary measure for open science practices? It seems like you are discarding some useful information by doing that. I would be in favor of either separately including prereg and sharing as variables in the model or creating a variable denoting the number of open science practices that were presented in the paper.

p.11 -> “Analytical approach” -> The statistical model is not explained in this section while I did expect that. It is probably better to describe it here than in the results section like you do now.

p.12 -> The preprint/article was coded with a single binary variable (0 = no open science practices reported, 1 = at least one open science practice reported)” -> In this case, but also in case of some other variables, I do not think a repetition of how they are operationalized is necessary. This is still fresh in memory from the previous section.

p.16 -> “We assessed whether COVID-19 preprints were characteristically different from the comparison groups in terms of three aspects of their research …” -> This felt like too much repetition to me. I think you can just focus on the results without reiterating what measures you used exactly.

p.17 -> The last paragraph is grey as opposed to black like the rest of the text.

p.20 -> “Some authors estimate that adopting and implementing preregistration and registered reports require twice the regular duration of a research project (Allen & Mehler, 2019).”            -> More evidence can be found in a recent survey by Sarafoglou et al: https://royalsocietypublishing.org/doi/full/10.1098/rsos.211997

p.21 -> “Although one would assume that past instances in which the validity of psychological research has been questioned has put a spotlight on the adoption of open science measures (Landy et al., 2020; Open Science Collaboration, 2015; Shrout & Rodgers, 2018), this apparently does not seem to be the case” -> I don’t think you can make this general statement. There has definitely been a spotlight on open science measures but it true that this not has translated to a giant uptake (see https://journals.sagepub.com/doi/full/10.1177/1745691620979806). It could be that this was even less so for COVID-19 papers though, which could be interesting to discuss further.

p.22 -> “Subsequent research is needed to identify this potential issue and investigate how preprints differ between these repositories” -> I think this requires a more concrete discussion. How should this be studied ideally, and why exactly is that so important?

p.22 -> “Future research should document the adherence to open science standards in greater detail, to allow a more fine-grained comparison.” -> It seems to me that you can do that already by not using the binary variable. In addition, “future research” is rather vague and any concrete guidance would be welcome.

p.23 -> “On top of this, the geographic concentration of authors, already a limitation in normal times, seems regrettable in the context of a truly global pandemic” -> It might be good to note that it may make more sense in Covid-19 times, as getting together a group of international collaborations takes more time and effort, which are scarce resources in a global pandemic. Other than that, some concrete suggestions on how to develop an infrastructure that facilitates more geographical diversity would be welcome.

Competing interests

The author declares that they have no competing interests.