PREreview del Search for Medical Information and Treatment Options for Musculoskeletal Disorders through an Artificial Intelligence Chatbot: Focusing on Shoulder Impingement Syndrome

Short Summary of Main Findings In this December 2022 medRxiv preprint (v2), the authors evaluated the performance of an early AI chatbot (primarily ChatGPT) in providing medical information and treatment options for shoulder impingement syndrome. They submitted a series of structured queries on symptoms, diagnosis, conservative treatments, exercises, and surgical options. The chatbot generated generally coherent, readable responses that covered common knowledge on the topic, but often lacked depth, cited no sources, occasionally included inaccuracies or hallucinations, and provided overly generic or incomplete advice (e.g., on exercise prescription or when to seek specialist care). The study highlighted both the potential accessibility of AI for patient education and its current limitations in reliability for musculoskeletal disorders.

How This Work Has Moved the Field Forward It represents one of the earliest documented evaluations of ChatGPT in a specific orthopedic/physiotherapy context (shoulder impingement), shortly after the tool’s public release. This helped spark the subsequent wave of research on large language models (LLMs) in patient education, clinical decision support, and medical information dissemination, contributing to growing awareness of both opportunities and risks of AI chatbots in healthcare.

Major Issues

• Remains an unreviewed preprint with no identified peer-reviewed journal publication.

• Very early evaluation of a rapidly evolving tool (ChatGPT-3.5 era); findings are now largely outdated given major model improvements.

• Subjective and non-standardized evaluation methods (no clear scoring rubric, inter-rater reliability, or comparison with gold-standard sources like clinical guidelines).

• Small scope (single condition, limited queries) and lack of clinical validation or real-patient outcomes.

Minor Issues

• Title is long and somewhat wordy.

• Limited discussion of ethical, liability, or misinformation risks.

• No quantitative metrics (e.g., accuracy percentages, readability scores) in some sections; results rely heavily on qualitative description.

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they did not use generative AI to come up with new ideas for their review.