Avalilação PREreview de Discovery Stack Pilot: Feasibility and Outcomes of a Scientist-Designed Peer Review Model Separating Quality and Impact

This paper presents the outcomes of a pilot study of a new approach to peer review of research articles. In this new approach, the quality and impact of an article are evaluated separately. I found the paper highly interesting to read. As pointed out by the authors, established approaches to peer review are facing major challenges. There is a strong need for studying alternative approaches to peer review.

My comments on the paper are fairly minor.

My most important comment is that the idea of separating the evaluation of quality and impact is not new. The authors should acknowledge existing approaches based on this idea. Let me highlight two approaches.

First, an increasing number of journals are using a so-called soundness-only approach to peer review. These journals ask reviewers to evaluate the soundness (i.e., quality) of an article and to refrain from evaluating the impact of an article. Soundness-only peer review was pioneered by PLOS One. Nowadays it is also used by various other journals, such as Scientific Reports. There is also literature studying soundness only peer review. See https://doi.org/10.1108/JD-06-2017-0092.

Second, the journal eLife uses a publish-review-curate model in which an explicit distinction is made between the strength of evidence (i.e., quality) and the significance of the findings (i.e., impact). Strength of evidence and significance of the findings are each assessed on a five- or six-point scale. There seem to be close connections between the eLife model and the Discovery Stack model proposed by the authors. The authors should discuss these connections.

A minor comment relates to the way in which the authors use the word ‘published’. The authors for instance state: “At the time of analysis, only six of 18 manuscripts had been published. The mean from submission to publication for the published manuscripts was 231 days.” This is confusing. Articles can be published (i.e., be made publicly available) not only in journals but also on other platforms, such as preprint servers. In this case, all 18 manuscripts have already been published on a preprint server. Therefore, instead of “only six of 18 manuscripts had been published”, the authors should say “only six of 18 manuscripts had been published in a journal”. Likewise, “from submission to publication” should be “from submission to publication in a journal”.

Finally, I was unable to find the data underlying the analyses presented in the paper. In the interest of transparency and reproducibility, publishing the data is crucial.

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they did not use generative AI to come up with new ideas for their review.