Escribir un comentario

Summary:

The manuscript describes the multi-step development of a brief self-report questionnaire (BORS) for measuring awareness and uptake of Open Research practices, and reports descriptive results from 1,274 respondents across 35 UK Reproducibility Network institutions. The topic is timely and useful, the instrument is openly available, and the developmental process is sensibly structured and transparently documented on OSF.

Major comments:

1. The claim of “validation” is not fully supported by the methods used. The abstract and conclusion state that the BORS has been “developed and validated”. What is presented, however, is primarily content development: a scoping review (or elsewhere called “exercise”), informal peer commentary, and a pilot plus main study. There is no evidence of, for instance:

– test-retest reliability, or

– convergent or discriminant validity against the existing measures identified in the scoping review.

We recommend either (a) providing evidence more consistent with a validation claim (e.g., a test-retest study), or (b) reframing the paper’s claims throughout as “development and piloting”, with content validity established via expert/peer review.

2. The peer-review step (Step 3) appears underpowered relative to its stated purpose. Six reviewers were invited by email with no structured reply template, and “no changes were suggested or made”. This is presented as though it confirms the instrument’s adequacy, but an unstructured, low-response round of peer commentary that produced no changes could equally indicate reviewer disengagement or a low bar for feedback (e.g., no prompts on specific dimensions such as clarity, completeness, or bias) rather than confirmation of quality. We suggest either providing the actual reviewer comments (even briefly, e.g., “reviewers confirmed clarity and comprehensiveness; no substantive concerns were raised regarding X, Y, Z”) or clearly acknowledging this as a limitation.

3. Support needs are likely to differ across practices. We understand the authors’ motivation for keeping the questionnaire very brief, but the item “What would help you to use more open research practices?” arguably needs to be answered separately for each Open Research practice, as the COM-B results could differ substantially between practices. Only with item-specific insights on support needs, institutes could make useful changes.

4. The recency of the data is not flagged clearly enough. The UKRN data were collected September 2021–January 2022, and the pilot in April–May 2021. Since then, the uptake of Open Sciences Practices has (hopefully) increased. The Discussion should address this directly, and the framing should make the vintage of the data transparent up front rather than leaving readers to infer it.

Minor comments:

1. The limitations of existing instruments are not made explicit. The introduction mentions other instruments measuring Open Science uptake, but it does not specify why those measures are insufficient or what their limitations are. We suggest the authors expand the “Measures of Open Research” section accordingly. Relatedly, did any of the previous measures serve as inspiration? Were items adopted from them?

2. The claimed cross-country scope is not matched by the data. The introduction reads as if the authors set out to develop “openly available, brief and, importantly, validated questionnaires to assess awareness, uptake, and support of Open Research practices that can be employed within and across institutions and countries”, but they apply their measure only within the UK and thus not across countries.

3. The basis for the content analysis is unclear. The authors should specify more clearly which free-text responses the content analysis was conducted on. Looking at the questionnaire, there do not seem to be many free-text fields.

4. Composition differences between the pilot sample and the main UKRN sample limit. The Brunel pilot sample was heavily weighted toward Computing, Engineering, Business, and Social Sciences, with Psychology comprising only 4.7%. The main UKRN sample is quite differently composed, with Psychology as the largest single discipline (17%) and Medicine & Dentistry, Biological/Sport Sciences, and allied-medicine fields also prominent. Because the Step 5 revisions were partly based on missing-data patterns observed in the pilot, it is worth commenting on whether these patterns and the resulting item-wording changes are likely to generalise to the very different disciplinary mix that later completed the full survey. A brief acknowledgment of this discontinuity, or evidence that the missing-data patterns were similar in the main sample, would strengthen the justification for the Step 5 changes.

5. Please mark the five additional identified themes in Table 2.

6. In the discussion, the authors state that the BORS is currently being used with minor changes — what are these changes, and why were they made?

7. The link to the post-peer-review version of the BORS does not work.

Future studies:

The questionnaire could be extended to include Open Educational Resources.

Acknowledgements:

– We used an AI language model to help edit this review for clarity and conciseness. All ideas included in the review are human-generated.

– We did not review the open data and analysis scripts.

Competing interests

The authors declare that they have no competing interests.

Use of Artificial Intelligence (AI)

The authors declare that they did not use generative AI to come up with new ideas for their review.