Avalilação PREreview de Benefits and Challenges of Integrating a Generative AI Assisted Reading Guide in an Undergraduate Journal Club Assignment

Summary and Strengths

Reading primary scientific literature is an important skill, yet it remains a persistent challenge for many students due to dense jargon, difficulty identifying key claims and supporting evidence, and low overall comprehension. In this study, the authors introduce a ChatGPT-assisted reading guide in an undergraduate chemistry course as an intervention to support students’ scientific reading, with a particular focus on fostering productive engagement with assigned papers. Students read and discussed two articles: the first without the AI-supported guide and the second using the guide. The authors analyzed students’ ChatGPT transcripts to characterize the types of interactions students initiated and found that most students used ChatGPT to provide background information and to request definitions or explanations of unfamiliar terms and concepts. The authors also surveyed students about perceived benefits and challenges of using AI while reading. Overall, student responses suggested strong interest in continued use of ChatGPT to support future reading of primary literature.

As generative AI becomes ubiquitous in higher education, instructors need clear, evidence-based ways to address its use without weakening students’ ability to read and reason through primary literature. This study helps operationalize “productive AI use” as a teachable skill: students can be guided to use AI in targeted, transparent ways that support comprehension while still requiring them to identify claims, evaluate evidence, and participate meaningfully in discussion.

Major issues

• I am not fully convinced by the claim that “overall, students reported that using the ChatGPT-assisted reading guide was helpful in understanding the article,” because it is unclear how “understanding” was evaluated. The primary evidence presented appears to be students’ self-reported perceptions and the types of ChatGPT prompts they generated. While these behaviors may support comprehension of terminology or concepts, they do not, on their own, demonstrate understanding of the article as a whole (e.g., identifying the authors’ central claims, interpreting figures, explaining how evidence supports conclusions). I recommend either 1) revising the language to make the claim more precise or 2) adding an explicit measure of article-level understanding (e.g., questions on figure interpretation and claims/evidence, or coded evidence from the discussion).

Minor issues

• Title

  • The title emphasizes the benefits and challenges of the reading guide, but in the current draft 1) the benefits/challenges discussion represents only a small portion of the Results (only two paragraphs), and 2) the benefits and challenges that are reported focus primarily on students’ perceptions of using ChatGPT rather than on the guide as a whole. I recommend revising the title to better fit the scope/conclusions of the study.

• Abstract

  • “Developing scientific reading skills is critical for undergraduate STEM students due to scientific literature’s unique formatting and use of specialized jargon.” This explains why reading is challenging but not why the skill is critical. I recommend revising this sentence to clarify the broader rationale (e.g., foundational to doing science, required for academic/professional success).

  • I don’t think referencing your previous study aids your abstract - you actually state similar findings (helpful in understanding parts of a paper versus the whole paper). I would use this space to expand on other parts of your project, or even the topic of productive uses of AI.

• Introduction

  • A few interventions to support students reading literature were mentioned. I would also recommend including literature from/based on the CREATE approach by Hoskins.

  • “While the studies described above make significant gains in helping students engage with scientific literature, the issue of unfamiliar language or jargon remains.” It would be helpful to include any literature here on the efforts to support students in undergrad STEM courses learning jargon in papers, such as annotations: Annotated primary scientific literature: A pedagogical tool for undergraduate courses

  • “Generative AI’s natural language…” What does “natural” mean here?

  • All of the text related to the Haraldsrud and Odden study should be its own paragraph or section within theoretical framework.

• Context of the Class

  • It would be helpful to have more context on the papers selected. If this is a course aimed at learning to read primary literature, was there any significance in the particular papers chosen? Did the instructor think it would be a good fit for the particular reading guide?

  • What were the discussion questions? Did they revolve around the prompts in the reading guide?

• Coding ChatGPT Transcripts

  • How did the Haraldsrud and Odden study on productive behaviors inform the coding? The theoretical framework would be more salient if the coding themes were organized as productive versus not productive.

  • Figure 1 (also Figure 4 and 5): I would recommend removing the duplicate title at the top, providing more description in the figure title about the numbers/percentages (what does 181% mean?), and changing the x-axis to have whole numbers since there is no such thing as half a student.

  • I like how summarizing text is acknowledged as a potentially concerning behavior - this is a tie in to the Haraldsrud and Odden study.

  • It would be interesting to characterize the transcripts as blooms taxonomy levels to see how often students are using ChatGPT to outsource higher order thinking skills.

• Conclusions/Recommendations

  • I recommend including limitations to the current study and references to similar literature.

  • “…discussing the benefits and shortfalls of generative AI, as well as ethical use of generative AI.” Were these things discussed by the instructor of the course?

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they did not use generative AI to come up with new ideas for their review.