Write a PREreview

Replication studies play an important role in scientific knowledge generation, but researchers lack a coherent account of how to conclude whether a finding has been replicated. This paper provides such an account by connecting methodological falsificationism and the Neyman-Pearson approach to statistical inferences in the context of the analysis of replication studies. From this perspective, the goal of a replication study is to examine whether the same basic statement that was made in an original study can also be made when new data are collected in a faithful empirical realization of the original test. If the original study made an ordinal claim based on a statistical decision rule, the replication study should evaluate whether the same claim can be made using the same inferential criterion. We argue that the original effect size estimate should also be treated as part of the original claim, and therefore that replication studies should additionally examine whether the effect size differs meaningfully from the original estimate. We illustrate how such comparisons can be performed for correlations by using equivalence tests and point out the importance of reflecting on which difference between replication studies is large enough to matter. More broadly, we argue that researchers should choose an analysis strategy for replication studies where their philosophy of science and statistical framework are coherently aligned.