A Language Model–Powered Simulated Patient With Automated Feedback for History Taking: Prospective Study
Although history taking is fundamental for diagnosing medical conditions, teaching and providing feedback on the skill can be challenging due to resource constraints. Virtual simulated patients and web-based chatbots have thus emerged as educational tools, with recent advancements in artificial intelligence (AI) such as large language models (LLMs) enhancing their realism and potential to provide feedback.
Objective
In our study, we aimed to evaluate the effectiveness of a Generative Pretrained Transformer (GPT) 4 model to provide structured feedback on medical students’ performance in history taking with a simulated patient.
Methods
We conducted a prospective study involving medical students performing history taking with a GPT-powered chatbot. To that end, we designed a chatbot to simulate patients’ responses and provide immediate feedback on the comprehensiveness of the students’ history taking. Students’ interactions with the chatbot were analyzed, and feedback from the chatbot was compared with feedback from a human rater. We measured interrater reliability and performed a descriptive analysis to assess the quality of feedback.
Results
Most of the study’s participants were in their third year of medical school. A total of 1894 question-answer pairs from 106 conversations were included in our analysis. GPT-4’s role-play and responses were medically plausible in more than 99% of cases. Interrater reliability between GPT-4 and the human rater showed “almost perfect” agreement (Cohen κ=0.832). Less agreement (κ<0.6) detected for 8 out of 45 feedback categories highlighted topics about which the model’s assessments were overly specific or diverged from human judgement.
Conclusions
The GPT model was effective in providing structured feedback on history-taking dialogs provided by medical students. Although we unraveled some limitations regarding the specificity of feedback for certain feedback categories, the overall high agreement with human raters suggests that LLMs can be a valuable tool for medical education. Our findings, thus, advocate the careful integration of AI-driven feedback mechanisms in medical training and highlight important aspects when LLMs are used in that context.
No keywords indexed for this article. Browse by subject →
Síle Kelly, Erica Smyth, Paul Murphy et al.
Ruth Plackett, Angelos P. Kassianos, Sophie Mylan et al.
Tjorven Stamer, Jost Steinhäuser, Kristina Flägel
Jeremy J Webb
Lixiang Yan, Lele Sha, Linxuan Zhao et al.
Makenna Carlson, Austin Pack, Juan Escalante
Aidan Gilson, Conrad W Safranek, Thomas Huang et al.
Ziwei Ji, Nayeon Lee, Rita Frieske et al.
J. Richard Landis, Gary G. Koch
Ted Byrt, Janet Bishop, John B. Carlin
Gunther Eysenbach
Young-Woo Jo, Myungeun Lee · 2025
Timo Breuer, Sameh Frihat · 2025
Emilia Brügge, Sarah Ricchizzi · 2024
- Published
- Aug 16, 2024
- Vol/Issue
- 10
- Pages
- e59213
You May Also Like
Aidan Gilson, Conrad W Safranek · 2023
1,763 citations
Gunther Eysenbach · 2023
680 citations
Kai Siang Chan, Nabil Zary · 2019
577 citations
Alaa Abd-alrazaq, Rawan AlSaad · 2023
528 citations
Robyn-Jenia Wilcha · 2020
354 citations