PREreview of Implicit reporting standards in bibliometric research: what can reviewers' comments tell us about reporting completeness?

This paper presents a study of openly available peer review reports of bibliometric studies. The aim of the study is to identify the implicit reporting standards applied by peer reviewers and to compare these implicit standards with the explicit standards adopted in three reporting guidelines for bibliometric studies.

This is a strong paper. The paper addresses a relevant problem, it uses a sound and innovative methodology, it presents valuable results, and it is reported in a clear and well-structured way. I very much like the way in which the paper demonstrates the value of openness in peer review.

I have two small points of feedback, which are listed below.

The number of studies included in the analysis is relatively small (85). Unfortunately, many journals have not yet adopted open peer review, which means that studies published in these journals cannot be included in the analysis. However, there is also another problem, which the authors do not discuss. To the best of my knowledge, the Web of Science database provides open peer review reports only for journals that use the ScholarOne platform (see for instance https://clarivate.com/academia-government/blog/the-current-state-of-open-peer-review/). Journals that do not use the ScholarOne platform may also publish open peer review reports, but these open peer review reports cannot be found in the Web of Science database. This for instance applies to journals belonging to the Nature family, to MDPI journals, and to platforms operated by F1000 (which are not included in the Web of Science database at all). Open peer review reports for preprints also cannot be identified using the Web of Science database. I recommend acknowledging these important limitations of the Web of Science database in the paper.

“Arguably, however, items such as reporting sample sizes should always be recommended considering that bibliometric studies very often involve immense datasets, which can influence the reliability and validity of statistical analyses.”: I do not understand this sentence. It is not clear to me how the importance of reporting sample sizes relates to the fact that bibliometric studies often involve large data sets. It seems to me that reporting sample sizes is even more important for studies that involve small data sets, since such studies are more likely to suffer from reliability problems.

Competing interests

I am involved in the development of the GLOBAL reporting guideline, and in this context I am collaborating with the first author of the paper under review.

Use of Artificial Intelligence (AI)

The author declares that they did not use generative AI to come up with new ideas for their review.