Write a PREreview

Multimodel inference (such as multiverse analysis and many-analyst projects) is a family of sensitivity analyses that examines variation in results across alternative auxiliary assumptions of a research question. The observed variation in multimodel inferences has given rise to the idea that there is no single correct analysis, and that researchers should examine variability across a large range of plausible models instead. In this contribution, we critically reflect on the epistemological assumptions and consequences of this idea. Specifically, we argue that variation across alternative auxiliary assumptions can reveal uncertainty about how to validly test a hypothesis, but it cannot substitute for the epistemic warrant required to justify those assumptions. This is because, once a research question is translated into a specific empirical test by specifying the relevant auxiliary assumptions, there is a fact of the matter about which statistical analysis correctly implements that test and deviations from that analysis are not made epistemically acceptable merely by being included in a distribution. Thus, robustness across alternative sets of auxiliary assumptions with differing levels of justification does not, by itself, show that a theoretical claim is epistemically warranted. Researchers should invest in the empirical groundwork needed to identify valid tests of their hypotheses, rather than embracing variability across analyses and incorporating epistemic uncertainty into statistical inferences.