Write a comment

Manuscript title: Anchoring-Based Causal Design (ABCD): Estimating the Effects of Beliefs

Invitation to Review date: 20.06.2026

Review submission date: 09.07.2026

Review type: original submission / revision / other

I have reviewed the manuscript “Anchoring-Based Causal Design” after being invited by a journal. I have not looked at the manuscript submitted to the journal but I am instead reviewing the version that the authors posted publicly at https://arxiv.org/pdf/2508.01677. I have done so before (https://prereview.org/reviews/20496508) because I think that scientific discourse should happen in the open and not behind closed doors. Nor should my work (i.e., this review) be only available to a specific party (in this case, a commercial journal) but the entire research community. I have decided to review the manuscript because of my experience in anchoring research and my interest in omitted variable bias in belief measurement.

The authors present a method that uses anchoring effects to study the causal effect of beliefs on decisions. Manipulating beliefs to determine causal effects is particularly difficult, which is why the use of anchors is a promising idea.

I like the promising approach, how the authors have been very transparent (e.g., open data, reported a study that “did not work”, very helpful Table 1), and that they ran several studies to assess the generalizability (e.g., spill-over effects, anchor distance, decay). I think that this research is a good starting point for future research (e.g., incorporating actual decisions that may be affected by anchored beliefs). I several suggestions for change that I hope the authors address in future versions of their article.

Specific suggestions for change in no specific order:

1. Typo: “we propose and demonstrates” should be “demonstrate” (p. 3)

2. Revision of the theory section: Insufficient adjustment and selective accessibility model have both been shown to be wrong (SAM: e.g., https://doi.org/10.1037/xge0000644; IA [my own research]: https://doi.org/10.15626/MP.2024.4137). However, I do not think that the lack of theory makes your approach less useful.

3. “persists long enough (several days)” -> I recommend that you cite a previous work for this claim

4. I suggest that you discuss all deviations from the preregistration. Specifically, in Study 1, p. 16, footnote 10: Responses were excluded above 100,000 NS but this had not been preregistered.

5. P. 17 and following: I am not sure about economics statistics reporting but from a psychological perspective, the F tests are missing degrees of freedom, I recommend reporting Cohen’s d for the standardized mean difference (see also Journal Article Reporting Standards, JARS). Moreover, I recommend that you report descriptives instead of percentage points (e.g., M, SD) and group sizes.

6. P. 12 footnote about a third related experiment not included in the report: I appreciate you mentioning this and I suggest to explain in slightly more detail why it was not included in this article (e.g., an article targeted at the specific hypothesis is planned).

7. Preregistration from footnote 17 does not exist (https://aspredicted.org/see_one.php). It asks me to enter my email first and then I get the error message “The page https://aspredicted.org/send_link.php does not exist.”

8. Figure 3: I recommend that you use a plot that shows the specific differences from t1 to t2 (“spaghetti plot”, for an example, see https://jasp-stats.org/2022/07/29/bayesian-repeated-measures-anova-an-updated-methodology-implemented-in-jasp/).

9. P. 27 “see Figure 4B”: I think this is meant to refer to Figure 4B

10. P. 31, table 4 note: I think it should say *p<.05 not p<0.5

11. References section: Some but not all articles have DOIs, I suggest to add them when possible

12. The Jacowitz and Kahneman (1995) study that you cited, was successfully replicated in the Many Labs study – maybe consider co-citing it: https://doi.org/10.1027/1864-9335/a000178; the individual report is available at https://osf.io/y36m8

13. I would like to make you aware that visual sliding scales (p. 43) can revert anchoring effects like we showed in one of our previous studies: https://doi.org/10.15626/MP.2024.4137 We have had other instances in unpublished studies and cannot say for sure when these occur. Maybe you think this is worth noting for future research though in no way is this meant to encourage you citing my work.

14. The delay distribution (Figure A4, p. 48) clearly shows a bimodal distribution. I recommend that, instead of analyzing delay vs. no delay, you treat the variable as metric and include it in a model to see how strongly estimates are biased over time (as an exploratory analysis). I would be really interested in seeing the shape of that decline. Relatedly, in the report, you wrote that anchoring persists over about 9 days but only a small fraction of the sample actually had a delay of 9 days (most had 7 or 10 days).

15. If I understood correctly, the claim that lagged effects are significantly smaller than instantaneous ones is not tested. I suggest that you run a statistical comparison, ideally with delay as metric or a wave*anchor interaction.

16. I suggest that you more thoroughly discuss whether a decision was actually changed based on an anchor. While the persistence over time and absence of spill-over effects seems to rule out the scale distortion account, there were no actual decisions, so I think it is difficult to talk about a causal effect of belief on decision. I think that this would be a good thing for future studies to investigate.

17. Please discuss for all preregistrations how you deviated, why you did so, and how it affected the results. Given that preregistrations only reduce flexibility in analyses when the analysis plan or analysis script are preregistered (http://dx.doi.org/10.2139/ssrn.4180594), I recommend that you discuss the limitation of your preregistration. In the future, I recommend that you preregister analysis scripts (for further arguments, see my own simulations at https://doi.org/10.31222/osf.io/v259t_v2).

18. I recommend that you cite all software and versions that you used.

Methodological Checks that I include in most of my reviews

(see also https://www.opennessinitiative.org/the-initiative/ - this is redundant with some of my comments above)

19. Reproducibility check with Claude Sonnet 5 and ReproAI

a. I strongly recommend a code review or reproducibility check via a journal or CODECHECK. An LLM-assisted reproducibility check claimed that the .do files contain an undefined-variable reference and cannot run in Stata. I cannot verify this statement since I have no Stata license. Also, no code seems to be available for the supplementary information’s polynomial anchor-value analysis. Apart from that, analyses recreated in R could reproduce the results.

20. Preregistration

a. Existence: one available (https://aspredicted.org/NSP_4M7), one has a broken link (https://aspredicted.org/see_one.php)

b. Specificity: too unspecific (no analysis plan, missing exclusion criteria)

c. Discussion of all deviations: missing

21. Open Data

a. Existence: on the OSF repository

b. Code sheet: Missing – lots of Q… variables; I suggest creating a codebook and a readme file with instructions for reproduction

c. Raw AND Processed data: on the OSF repository

22. Open Code

a. Existence: .do files available

b. Open Source Software: No (Stata?)

c. Version control or disclosure of versions: Missing

d. Well-commented: Cannot assess

23. Methods

a. Incentivization of participants: Disclosed

24. Results

a. Statcheck results: not applicable due to F-values missing degrees of freedom

b. Indicators of QRPs: None spotted

25. Open Materials

a. Stimuli or questionnaires: described in supplements

Disclaimers

26. I did not conduct a manual reproducibility check

27. I did not check for deviations from the preregistrations in detail

28. Publication of this review: I will publish this review on my OSF reviews project and PREreview.org.

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they used generative AI to come up with new ideas for their review.