PREreview of Is ChatGPT detrimental to innovation? A field experiment among university students

Summary and Strengths 

ChatGPT, among other automated or artificial intelligence, is of concern – especially to educators – because of its impact on human behavior. This study aimed to test the effect of continuous usage of ChatGPT in a classroom setting on three behaviors related to economic growth: innovation, effort, and risk. Students submitted essays using ChatGPT (treatment group) or not (control group); the authors subsequently measured the focal behaviors through an innovation and risk game and a real-effort task. The authors highlighted three key findings: students who used ChatGPT 1) were significantly less innovative, 2) were significantly less risk averse, and 3) exerted less effort compared to their non-AI using counterparts. 

An obvious strength of this manuscript is the quality of the argument: supporting literature was provided, the objectives were clear, the experimental designs were appropriate, the conclusions were supported with evidence, and several limitations and rebuttals were addressed. Additionally, the study is both novel and interesting. We commend the authors for using this unique opportunity to examine the impact of AI prior to its spread and extensive use. We believe the results of this study could open opportunities to look further into the negative effects of AI applications in educational contexts. 

We would recommend this article for publication following minor revisions (as outlined below) - primarily to the experimental design. 

Major Issues 

The reviewers have not identified any major issues with the manuscript.  

Minor Issues 

Methods 

• The reviewers would find it helpful if the authors provided more detail on the topic and guidelines for the assigned essays in each course.  

• Did the essays require any innovation on the part of the students? 

• Were students in the treatment group required to submit an exact output of their ChatGPT query or were they allowed to edit it at all? 

• Supplemental materials and an appendix were mentioned in the Methods and Results but were not attached to the preprint. We recommend attaching these so that all materials can be reviewed.  

• The reviewers would find it helpful if the authors provided justification or more content for why the specific methods (i.e., lemonade and bomb game) were used. 

Results 

• The reviewers would find it helpful if the authors included indicators of statistical significance on the relevant plots. 

Discussion 

• The reviewers recommend including comments on how their results can be used to inform use of AI in the classroom – how can educators work with the existence of ChatGPT to still promote innovation in students? 

Competing interests

The authors declare that they have no competing interests.