Écrire un commentaire

Does the introduction explain the objective of the research presented in the preprint?

Yes

Are the methods well-suited for this research?

Somewhat appropriate

The argumentation is conceptual and critiques poor methodology and interpretation in prior multiverse studies. The author reason too quickly from those flawed studies to the conclusion (that "there is only one correct analysis"). The authors need to implement their two-step (multimodel analysis and integrative experiment) approach to actually show that there is only one correct analysis for a given research question. Also, can the authors revisit the multiverse analysis on soccer players and list the various analyses and identify the percentage of analyses with unjustified model specifications? This is a lot to ask for, I know. But it would give us readers a sense of how frequently nonsensical model specifications occurs in multiverse analyses. The "only" in "only 85 out of 90 models" is unclear and seems unjustified at first glance: "when Simonsohn and colleagues (2020) ran multimodel inferences on 90 models to examine if people in the United States with black names have lower callback rates in job applications – an observation with a large effect size that is generally considered an established fact – only 85 out of 90 models" Regarding Auspurg (2025), report what was used to justify the "justified analyses". A detailed remark here would help readers grapple with who to trust (original multiverse analysts vs. Auspurg, 2025).

Are the conclusions supported by the data?

Somewhat unsupported

The conclusions overgeneralize given the critique of past flawed analyses. There may be a few "correct" analyses, but support for "only one correct analysis" is missing.

Are the data presentations, including visualizations, well-suited to represent the data?

Somewhat inappropriate or unclear

Fig. 1 is a good start to visualizing the links between theories, auxiliary assumptions, models, and data. However, it is quite busy with many arrows pointing in opposing directions. It is also quite boxy. It would be better to remove the boxes (solid and dotted) and combine the top arrows into one long arrow. The bottom two arrows can be combined into one long arrow as well. An example in Fig. 1 to concretize the constructs would helps as well (the psychology examples from the Introduction could be a good idea). Small icons, used judiciously, can be useful to create visual engagement (theory, model, data). The Noun Project, Freepik, and default icons in MS Office applications would work.

How clearly do the authors discuss, explain, and interpret their findings and potential next steps for the research?

Somewhat unclearly

The authors could briefly sketch out examples of testing auxiliary assumptions in a topic area (e.g. multiverse analysis on soccer players) and provide readers with case studies of how to proceed. We suggest outlining a procedure in which one might show how the two-step multimodel analysis then integrative experiments would work for 1-2 research questions. We also wonder whether Delphi methods might be a useful way to collapse the many branches in the multiverse down to a few. The authors did not report this and it seems worth entertaining.

Is the preprint likely to advance academic knowledge?

Somewhat likely

The paper adds value to the metascientific literature on multimodel analyses by contradicting the dominant view that such analyses are required to understand the robustness of a phenomenon/intervention. The metascientific and philosophical connections are interesting and worth expanding further.

Would it benefit from language editing?

Yes

This preprint is very readable, though a few small grammatical issues and repetitiveness require correction. Author affiliation for the third author is missing. e.g. should be i.e. in several areas (https://www.grammarly.com/blog/acronyms-abbreviations/i-e-vs-e-g/) In many areas, explicit definitions from cited literature and tables contrasting such definitions would help readers who are unfamiliar with this literature. New terminology like set-theoretical predicate is hard to keep straight, especially without illustration with examples. Remove or rewrite "Robustness is a consequence of valid tests, and not their substitute". You might consider moving the 12 Angry Men analogy to the top of the Introduction or somewhere around there instead of at the very bottom of the preprint. It's a good storytelling and pedagogical device, so don't bury it too deep in the preprint. Define "alethic relativists" in "At the same time, the observed variability plays right into the hands of alethic relativists who believe that what is true always depends on a specific viewpoint" Unclear writing in need of rewriting or explanation: "The lack of attention to validity (Schimmack, 2021; Vazire et al., 2022) is problematic enough without embracing epistemic uncertainty about how to test a hypothesis."

Would you recommend this preprint to others?

Yes, but it needs to be improved

The strong title and subheading at the end do not match the actual advice buried deep in the preprint. The advice to combine multimodel analyses with integrative experiments is nuanced, but the title and repeated statements in the preprint "There is only one correct analysis" is not nuanced. The provocative title should be replaced with a more accurate and nuanced one. Some options to get started brainstorming: 1. Not all analyses are equally valid: A [name of your philosophical position] critique of multimodel, multiverse, and other multimodel analyses 2. Achieving robustness with multimodel analyses and integrative experiments As this is a philosophical article, it may be useful to summarize the core argument as a logical syllogism (helps skimming and communication of the key ideas) with premises and conclusion(s). This syllogism could be embedded in the abstract and main body of the text. The abstract should include the advice to use integrative experiments (not all readers will finish the full preprint). Throughout the paper, consider using the physics and psychology examples from the Introduction to make the philosophical content easier for novices to understand (e.g. researcher degrees of freedom, analytic flexibility, etc.). Several areas of the paper cite only recent papers, sometimes by the study authors. Expanding the citation network to the pioneers in a given area AND other authors would credit the broader scholarly community better. Sometimes, citations are given to just the pioneer and sometimes just to the co-authors. Balancing citations across time and disciplines/venues is desirable. Key Example: "Combined with the pressure to publish (Case, 1927), and a publication culture that values positive results, the uncertainty in how a prediction should be tested has led to concerns about how the opportunistic misuse of analytic flexibility and the selective reporting of findings undermines the reliability of scientific claims (Lakens, 2025b, 2025a). "

Is it ready for attention from an editor, publisher or broader audience?

Yes, after minor changes

Competing interests

The authors declare that they have no competing interests.

Use of Artificial Intelligence (AI)

The authors declare that they used generative AI to come up with new ideas for their review.