BishtRef
Articles Journals Publishers Authors Subjects Most Cited New Papers Year Stats Open Access
journal article Open Access Jul 10, 2025

Extracting epilepsy‐related information from unstructured clinic letters using large language models

Shichao Fang ORCID Ben Holgate Anthony Shek Joel S. Winston ORCID Matthew McWilliam Pedro F. Viana ORCID James T. Teo ORCID Mark P. Richardson ORCID
Epilepsia Vol. 66 No. 9 pp. 3369-3384 · Wiley
View at Publisher Save 10.1111/epi.18475
Abstract
AbstractObjectiveThe emergence of large language models (LLMs) and the increasing prevalence of electronic health records (EHRs) present significant opportunities for advancing health care research and practice. However, research that compares and applies LLMs to extract key epilepsy‐related information from unstructured medical free text is under‐explored. This study fills this gap by comparing and applying different open‐source LLMs and methods to extract epilepsy information from unstructured clinic letters, thereby optimizing EHRs as a resource for the benefit of epilepsy research. We also highlight some limitations of LLMs.MethodsEmploying a dataset of 280 annotated clinic letters from King's College Hospital, we explored the efficacy of open‐source LLMs (Llama and Mistral series) for extracting key epilepsy‐related information, including epilepsy type, seizure type, current anti‐seizure medications (ASMs), and associated symptoms. The study used various extraction methods, including direct extraction, summarized extraction, and contextualized extraction, complemented by role‐prompting and few‐shot prompting techniques. Performance was evaluated against a gold standard dataset, and was also compared to advanced fine‐tuned models and human annotations.ResultsLlama 2 13b (a 13‐billion‐parameter LLM developed by Meta) demonstrated superior extraction capabilities across tasks by consistently outperforming other LLMs (F1 = .80 in epilepsy‐type extraction, F1 = .76 in seizure‐type extraction, and F1 = .90 in current ASMs extraction). Here, F1 score is a balanced metric indicating the model's accuracy in correctly identifying relevant information without excessive false positives. The study highlights the direct extraction showing consistent high performance. Comparative analysis showed that LLMs outperformed current approaches like MedCAT (Medical Concept Annotation Tool) in extracting epilepsy‐related information (.2 higher in F1).SignificanceThe results affirm the potential of LLMs in medical information extraction relating to epilepsy, offering insights into leveraging these models for detailed and accurate data extraction from unstructured texts. The study underscores the importance of method selection in optimizing extraction performance and suggests a promising avenue for enhancing medical research and patient care through advanced natural language processing technologies.
Topics

No keywords indexed for this article. Browse by subject →

References
66
[5]
MatejkaJ.Refractory epilepsy – continuing the hunt for answers.2023Epilepsy Research Institute. Available from:https://epilepsy‐institute.org.uk/research/refractory‐epilepsy‐continuing‐the‐hunt‐for‐answers/
[10]
Zaied ANH "Electronic health records: applications, techniques and challenges" Int J Comput Appl (2015)
[31]
Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models

Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla et al.

PLOS Digital Health 10.1371/journal.pdig.0000198
[32]
NoriH KingN McKinneySM CarignanD HorvitzE.Capabilities of gpt‐4 on medical challenge problems.arXiv preprint arXiv:230313375.2023.
[33]
SinghalK TuT GottweisJ SayresR WulczynE HouL et al.Towards expert‐level medical question answering with large language models. arXiv preprint arXiv:230509617.2023.
[35]
KraljevicZ BeanD ShekA BendayanR HemingwayH AuJ.Foresight‐Generative Pretrained Transformer (GPT) for Modelling of Patient Timelines using EHRs.
[40]
MetaAI.Introducing Meta Llama 3: the most capable openly available LLM to date.2023. Available from:https://ai.meta.com/blog/meta‐llama‐3/.
[41]
TouvronH MartinL StoneK AlbertP AlmahairiA BabaeiY et al.Llama 2: open foundation and fine‐tuned chat models. arXiv preprint arXiv:230709288.2023.
[42]
JiangAQ SablayrollesA MenschA BamfordC ChaplotDS delas CasasD et al.Mistral 7B. arXiv preprint arXiv:231006825.2023.
[43]
Machine learning–XGBoost analysis of language networks to classify patients with epilepsy

L. Torlay, M. Perrone-Bertolotti, E. Thomas et al.

Brain Informatics 10.1007/s40708-017-0065-7
[45]
ShekA.AI and the sacred disease – The opportunities of electronic patient records and natural language processing to advance epilepsy care and beyond. Doctoral thesis. King's College London London.2022.
[47]
WhiteJ FuQ HaysS SandbornM OleaC GilbertH et al.A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:230211382.2023.
[48]
TopsakalO AkinciTC.Creating large language model applications utilizing langchain: a primer on developing LLM apps fast.Proceedings of the International Conference on Applied Engineering and Natural Sciences Konya Turkey 10–12.2023. 10.59287/icaens.1127
[50]
Marvin G (2023)

Showing 50 of 66 references

Metrics
4
Citations
66
References
Details
Published
Jul 10, 2025
Vol/Issue
66(9)
Pages
3369-3384
License
View
Authors
S
Shichao Fang

Department of Basic & Clinical Neuroscience King's College London London UK; King's College Hospital NHS Foundation Trust London UK; Guy's and St Thomas' NHS Foundation Trust London UK

B
Ben Holgate

Department of Basic & Clinical Neuroscience King's College London London UK; King's College Hospital NHS Foundation Trust London UK; Guy's and St Thomas' NHS Foundation Trust London UK

A
Anthony Shek

King's College Hospital NHS Foundation Trust London UK; Guy's and St Thomas' NHS Foundation Trust London UK

J
Joel S. Winston

Department of Basic & Clinical Neuroscience King's College London London UK; King's College Hospital NHS Foundation Trust London UK

M
Matthew McWilliam

Department of Basic & Clinical Neuroscience King's College London London UK; King's College Hospital NHS Foundation Trust London UK

P
Pedro F. Viana

Department of Basic & Clinical Neuroscience King's College London London UK; King's College Hospital NHS Foundation Trust London UK

J
James T. Teo

Department of Basic & Clinical Neuroscience King's College London London UK; King's College Hospital NHS Foundation Trust London UK; Guy's and St Thomas' NHS Foundation Trust London UK

M
Mark P. Richardson

Department of Basic & Clinical Neuroscience King's College London London UK; King's College Hospital NHS Foundation Trust London UK

Funding
Angelini Pharma
Cite This Article
Shichao Fang, Ben Holgate, Anthony Shek, et al. (2025). Extracting epilepsy‐related information from unstructured clinic letters using large language models. Epilepsia, 66(9), 3369-3384. https://doi.org/10.1111/epi.18475
Related

You May Also Like

ILAE Official Report: A practical clinical definition of epilepsy

Robert S. Fisher, Carlos Acevedo · 2014

4,870 citations

ILAE classification of the epilepsies: Position paper of the ILAE Commission for Classification and Terminology

Ingrid E. Scheffer, Samuel Berkovic · 2017

4,486 citations

Definition of drug resistant epilepsy: Consensus proposal by the ad hoc Task Force of the ILAE Commission on Therapeutic Strategies

Patrick Kwan, Alexis Arzimanoglou · 2010

3,977 citations

Operational classification of seizure types by the International League Against Epilepsy: Position Paper of the ILAE Commission for Classification and Terminology

Robert S. Fisher, J. Helen Cross · 2017

2,934 citations

Epileptic Seizures and Epilepsy: Definitions Proposed by the International League Against Epilepsy (ILAE) and the International Bureau for Epilepsy (IBE)

Robert S. Fisher, Walter van Emde Boas · 2005

2,913 citations