PREreview of The Prevalence of Direct Replication Articles in Top-Ranking Psychology Journals

This is a reproduction of a review that I did for a journal in March, 2024.

The current paper provides an estimate of the number of replication articles published in top psychology journals between 2010 and 2021, building on the earlier work of Makel and colleagues published in 2012. First, the good (of which there is a lot). The article is very clearly written, addressing five important questions about the prevalence of replication articles. The article is a model of transparent research practices as well. I particularly appreciated the care that was taken to explain when decisions differed from the preregistered plan. I reviewed the contents of the preregistration document, and I did not notice any undisclosed discrepancies between the plan and the final report. I also appreciated the well organized and clear data, code, and materials that were shared on OSF. I was able to download the data and interact with it easily to examine my own questions. It is not always easy to do this, and thus, I really appreciate that the authors took the time to make this information useful. I have a number of critiques to raise, as well. I will separate them into major and minor issues.

Major critiques:

1. I question the decision to restrict the focus of the analysis to only direct, between paper (i.e., in a new paper) replications that are the “main point” of an article (or one among a few main points) – this is Type A in the authors’ coding scheme. It seems that it would also be useful to examine the same questions that were raised here with the “Type B” articles from the coding scheme. These are direct, between paper replications that are not the “main point” of their respective papers. The authors here might argue that they would consider those studies to be replications, just not of the type that they were interested to understand. However, I worry that this nuance might be lost on the casual reader who would be forgiven for concluding that the 0.20% estimate would include these other replication studies as well. That is, the 0.20% estimate is likely very conservative (excluding even direct replications that other researchers might have opted to include). The issue of within article replications is also of interest (i.e., Type E - have these also increased over time?), but I can see a stronger argument for not including those replications in the topline estimate. Given that the papers have already been coded according to the authors’ type scheme, I hope it would be simple to include some more exploratory/robustness analyses presenting results with one or more different definitions of replication. (I didn’t explore enough in the data to see whether I could look at this question myself already.) This would not be a dealbreaker issue for me, if the authors had a compelling argument for why this should not be done, but it’s my current preference for these extra analyses to be included (and marked as nonregistered).

2. The paper does a nice job of providing descriptive statistics throughout the text to support the results reporting. I wouldn’t consider this a mandatory change, but I think a table highlighting key descriptive statistics for focal metrics would be a nice addition (e.g., min, max, median, mean, SD at the journal level)? This might already be in the supplemental materials submitted to the journal (I didn’t look at those).

3. Transparency and openness statement: Thanks to the authors for including this statement as required by APA journals. It has some necessary elements, but it is missing some others. First, for data, materials, and code: all three are provided on OSF, but only 2 of 3 are mentioned in the statement. Please add mention of open materials (e.g., coding scheme) to the statement. Second, authors should cite any secondary data, materials, and code in the TOP statement (cite in the reference list, not just mention). That means that the TOP factor data that you mention should be cited in some way, as should any R packages or other analysis tools that were used. Third, no reporting guideline is cited. In APA journals, we generally recommend JARS, but PRISMA might be the more fitting guideline for this study, which is essentially a version of a systematic review. In addition to citing adherence to the reporting guideline in the TOP statement, I would ask that you review PRISMA itself and make sure that you have in fact disclosed all of the necessary details. To me, it looks like your reporting is very complete, but I find that there’s always a few things that are mentioned in these reporting checklists that people routinely forget to include. You could add your completed checklist as a supplemental material on OSF, if desired.

4. Preregistration: I believe that the preregistration document is not in the registry (i.e., it is just shared as a document instead of entered in the registry). I think this was probably an unintentional oversight by the authors rather than anything more than that. I don’t think we (open science advocates) have done a good enough job explaining the value of entering studies in a registry (e.g., OSF registries) beyond simply publicly posting time-stamped documents. The main benefit is that registry entries are indexed and searchable, both via the web interface, but especially via the API. This means that in the future, researchers will be able to search the registry to learn things like “how many registered studies go unpublished?” or “how vague or specific are preregistered hypotheses?” and so on. As a remedy, I would like the authors to register the study now, indicating in their description of their registration that they are doing so retrospectively to correct an earlier omission. Ideally, they would use a structured form, like the OSF preregistration template, but they could also attach the document that they already have to an open-ended registration. The link to the registry entry is the one that would then be included in the paper.

5. I would recommend that the authors consider adding a caveat to their discussion section about what the effects of replication policies changing over time might have on their results. The TOP Factor database entries are sporadically updated, but the updates might not coincide with a change in editorial policy. Journal author guidelines are changing all the time – an editor might add a new policy, or retract an old one, either when the editorial team changes or at another time. Different publishers have different governance structures dictating when such changes are or are not allowed to be made. I’m not sure there’s a clear implication here, but I thought the issue of policies changing over time deserved some consideration.

Minor points:

6. A quibble: You write, “our sample of journals was effectively our population of interest” (p. 9). I suspect that this is not precisely accurate, however. Your population of interest is (likely) all articles published in these journals, including those that might be published there in the future (or perhaps others published in similar journals not in the sampling frame). That is, you hope that your results would extend to future hypothetical papers in these or similar journals. I would consider revising this statement on p. 9 and other statements throughout the paper suggesting that you have conducted a census (e.g., claims on p. 18).

7. In the JournalData_cleaned file, I created a variable to look at the percentage of total articles that were of the “article” type. One journal (AMPPS) had a value that exceeded 100%, suggesting an error.

8. The color scheme in Figure 3 needs to be adjusted for color blind friendliness (minimally). However, the color in Figure 3b is not particularly legible. Maybe you could facet Figure 3b by policy instead of using color so that it is more clear.

I always sign my reviews,

Katherine S. Corker

Competing interests

I produced this review as part of masked peer review at a journal. It later became clear that the authors are people with whom I have worked on open science projects, in particular through SIPS, over the years.

Use of Artificial Intelligence (AI)

The author declares that they did not use generative AI to come up with new ideas for their review.