Escrever um comentário

Please note, this review has been submitted to eLife as part of their review process on the manusript. eLife will integrate this review with other reviews and sign off the review in a single blind fashion. For more information on eLife’s review model see: https://elifesciences.org/inside-elife/54d63486/elife-s-new-model-changing-the-way-you-share-your-research

Summary, significance, and strength of evidence

This study addresses the gap between calls to incentivize Open Science practices through research assessment reform and its implementation by Research Performing Organizations (RPOs). To help RPOs get started and prioritize their assessment reforms, it makes a series of recommendations for action. It does so via an expert Delphi process, where consensus around six standards was arrived at. The purpose is to provide a resource for would-be reformers in Research Performing Organizations around the world, in the hope of advancing the Responsible Research Assessment and Open Science reform movements across global academia. The standards are not meant to be one-size-fits-all, but customizable to local needs and implemented selectively at the discretion of a given RPO.

Overall I was impressed with the contribution. Generally speaking it clearly sets out its methods and arguments. I recognize there is indeed a growing need to support RPOs at varying levels of maturity, to get started and orient themselves around research assessment reform, so this seems to me like it will be a useful offering.

The Delphi approach is well explained and further information is provided in supplementary files.

Some further reflection on the disciplinary make-up of the Delphi participants would be helpful, as discussed further in the Specific Feedback to the Authors section, below. Somewhat more hedging on the limitations of the manuscript would help improve the contribution, as would further details on how this contribution sits in relation to existing resources and studies on implementing open science-aware research assessment reforms.

Specific Feedback to the Authors

As the eLife model is less about accept-reject, but more about formative feedback, I will try to offer some advice for improvement, which the authors can take or leave (and so too the readers of eLife).

Please note, in performing this review, I have tried to assess the manuscript not on the basis of whether I normatively agree or not with the content of the recommendations. Parking these considerations, I have instead tried to put on my analyst’s hat and probe the quality of the design, argumentation, assumptions, literature review, and so on.

This is a Tools and Resources type contribution to eLife. The criteria on which to judge the quality of such a contribution are set out on the eLife website, but please note, when evaluating this submission, it became clear quickly this particular contribution did not seamlessly map onto the criteria/examples given on eLife’s website:

“Specifically, submissions will be assessed in terms of their potential to facilitate experiments that address problems that to date have been challenging or even intractable. Some Tools and Resources papers will be the first report of an entirely novel technology. In other cases, authors will report substantial improvements and extensions of existing technologies. In those cases, the new method must be thoroughly compared and benchmarked against existing methods used in the field. Minor improvements on existing methodologies are unlikely to be sent for peer review.” (elife-rp.msubmit.net/html/elife-rp_author_instructions.html#types)

Obviously, from reading the above text, this type of output in eLife generally relates to a different kind of work than the Delphi based recommended standards provided here.

Elsewhere, there is some guidance to be taken in evaluating this particular contribution, e.g.:

“Tools and Resources articles should fully describe the tool or resource so that prospective users have all the information needed to deploy it within their own work.”

Following the spirit rather than the letter of the evaluative criteria provided by eLife, I asked myself while reviewing: a) do prospective users have all the info needed to work with the manuscript’s findings and recommendations b) is the improvements to the existing state-of-the-art clearly set out? and c) are there other omissions or points of improvement I can flag?

I set out my responses below and provide some modest suggestions for improvement, where necessary.

a) The text and Appendix 1 provide a good solid basis for users to work with the recommended standards. I appreciate the modular approach suggested by the authors on pp10-11 and the efforts to avoid over-standardization. I would appreciate a more explicit acknowledgement here of disciplinary-specific considerations e.g. some fields that are more inductive and less hypothesis-driven not requiring pre-registration. There is a risk with any proposed standards not only of administrative burdens that the authors mention, but worse than that – red tape.

b) How do these findings build on, differ from, and generally provide value-added to what is out there already? Clearly we have a lot of statements and principles in this space, but there are also some practical tools e.g. DORA’s practical guide to implementing reforms in RPOs, DORA’s Reformscape, CoARA reforming academic careers working group’s outputs, and on the open science side the Grasp-OS project’s outputs on open science-aware research assessment reform, and some recent studies on CoARA action plan analysis. Clarification on what is new here and why we need this contribution, in light of what is already out there, would be a valuable improvement to the text as it stands.

Whilst I appreciate the sentiment of the authors planning to provide additional support to reforming institutions in deploying this resource, I do wonder if the authors would not be better off piggy-backing this resource onto existing learning networks like DORA, CoARA, GRC, etc. instead of starting up a new network.

c) Other remarks

“Recent reforms have targeted publishers and funders, but progress on RPO policies lags (19,20)” - these references are not empirical studies.

There are a lot of biomedical and health researchers in the author list – this is not a criticism per se, but it has likely shaped the kinds of priorities and recommendations brought to the fore throughout the Delphi process. At the moment, reflexivity on who comprised the panels is a relative weakness - there is an appeal to authority when stating who the contributors are – e.g. they represent certain important organisations; and there is demographic info, stakeholder info, and career stage info. But what is the intellectual backgrounds? Some of the major recommendations (prospective study registration, transparency through reporting guidelines, verification efforts) read to me as mostly relevant to more experimental, EBM, and statistically-oriented disciplines. Confronting this composition question and if necessary owning it as a limitation, would be my suggestion.

I am not an expert on Delphi methodology, but I do note that prospective study registration, transparency through reporting guidelines, and verification efforts only arrived at consensus in Round 3 (when there were fewer people participating). For the quasi-ignorant reader (like me), is this a limitation? If not, why not? Appendix X suggests the sample of participants satisfied considerations like equity and career stage, but as per above, I am wondering about diversity of disciplinary composition in the final stage – do differences at this stage and previous stages contextualize why consensus was not arrived at earlier?

One limitation is that this study is likely to become quite dated very soon, as some developments (e.g. around Generative AI declarations) are just so fast moving. This is an ‘occupational hazard’ and not a reason to abandon this kind of contribution – but some kind of acknowledgement of the dynamic nature of certain developments on which the manuscript focused and any subsequent limitation of the contribution would be helpful – otherwise it risks hubristically presenting a ‘finalized’ set of standards that are practically out of date, perhaps even by the time the final version goes up on eLife.

Did the authors discuss during the design of the Delphi whether to include a discussion item on about open research information – i.e. that the research information upon which evaluations by RPOs is based itself conforms to open science principles (e.g. Barcelona Declaration, 2024)? Was this a deliberate omission?

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they did not use generative AI to come up with new ideas for their review.