AI versus Faculty: A Comparative Study of Narrative Feedback on Medical Students' Written Mental Status Exams

Abstract

Objective: The authors evaluated narrative feedback generated by ChatGPT and faculty members for medical students' Mental Status Exam (MSE) write-ups. The study compared feedback quality and usefulness and assessed whether students and academic psychiatrists could identify the feedback source.

Methods: Medical students (N=164) wrote MSEs and received blinded feedback from either a faculty member (for low-scoring write-ups, n=43) or ChatGPT (for high-scoring write-ups, n=121). Students rated the feedback's quality and usefulness and guessed its origin. Three academic psychiatrists also conducted a blinded evaluation, rating both feedback types for the low-scoring MSEs, choosing the superior version, and guessing the source.

Results: Students rated AI feedback quality significantly higher than faculty feedback (mean=4.22 vs. 3.5). Academic psychiatrists preferred the AI-generated feedback in 93% of cases. Only 29% of students receiving AI feedback correctly identified its source. Psychiatrists correctly identified AI feedback only 23% of the time and misattributed faculty feedback as AI-generated 71% of the time.

Conclusions: AI-generated feedback was perceived as high-quality by students and preferred by expert raters. The difficulty in distinguishing AI from faculty feedback suggests generative AI can produce feedback comparable or superior to human experts, offering a scalable tool to support medical education and reduce faculty workload.