What studies show about errors in AI medical summaries
Answer in brief: across the studies we found, AI summaries of clinical text leave facts out more often than they invent them, negations are a large share of invented facts, and medication details score worst. None of the studies measured summaries of medical-legal record files, so use them to decide what to check, not to predict a tool's error rate on your files.
Limits: We can't vouch for any tool's error rate on your files: the studies we cite measured other tasks and other models, and vendor features are as each vendor's site stated on the capture date.
The studies, with what each measured
We read each study's abstract or full text on 2026-11-12. The table gives the numbers as the authors report them. Where a study measured a different task, the last column says so.
| Study | What was summarized, by whom | Error finding | Doesn't transfer because | Source, read |
|---|---|---|---|---|
| Asgari et al., npj Digital Medicine, 2025 | 450 clinical notes drafted by LLMs from consultation transcripts; 50 doctors annotated sentences | 191 of 12,999 note sentences hallucinated (1.47%), 44% of them major; 30% of hallucinations were negations; 1,712 of 49,590 transcript sentences omitted (3.45%) | Source was a conversation, not a document file | nature.com, 2026-11-12 |
| Grolleau et al., JAMA Network Open, 2026 | 1,274 hospital course summaries in live use on 1 medicine unit; physician feedback on 100 | Omissions in 25, inaccuracies in 20, hallucinations in 2; 88 rated no harm potential, 1 likely moderate harm | Inpatient notes from 1 EHR; physicians knew the patients | jamanetwork.com, 2026-11-12 |
| Deschepper et al., JMIR Medical Informatics, 2026 | 30 synthetic Dutch discharge letters summarized by GPT-4o; 31 clinicians rated | Missing information was the most common error; medication section scored lowest and had the most hallucinations; median rater agreement kappa 0.36 | Short letters in Dutch | medinform.jmir.org, 2026-11-12 |
| Shemtob et al., JAMIA Open, 2025 | 70 simulated primary care records of 10 contacts each; GPT-4 vs clinician summaries, rated blind | Similar accuracy; AI less likely to omit important information; 11 inaccuracy incidents in AI summaries, 12 in clinician summaries | Simulated, short records | PMC, 2026-11-12 |
| Van Veen et al., Nature Medicine, 2024 | 8 adapted LLMs on radiology reports, patient questions, progress notes and dialogue; 10 physician readers | Best-adapted LLM summaries rated equivalent (45%) or superior (36%) to expert summaries in most cases | Short inputs; models adapted per task | nature.com, 2026-11-12 |
| Tang et al., npj Digital Medicine, 2023 | GPT-3.5 and ChatGPT summarizing medical evidence in 6 clinical domains | Factually inconsistent summaries and overly convincing or uncertain statements; more errors over longer texts; automatic metrics tracked quality poorly | Research abstracts, older models | nature.com, 2026-11-12 |
| Mendel and UMass Amherst pilot, 2024 | 500-word summaries of 50 medical notes by GPT-4o and Llama-3 | GPT-4o: 21 summaries with incorrect and 50 with over-general information; Llama-3: 19 and 47; 92 minutes for a clinician to label 1 summary | Pilot, reported by trade press | Clinical Trials Arena via Yahoo, 2026-11-12 |
| Oukelmoun et al., EMNLP Industry Track, 2025 | French medical summaries, expert-validated omission dataset | Omission is a separate failure from hallucination and needs its own detection | Method paper; no error rate for your tool | abstract, 2026-11-12 |
| Liu et al., Transactions of the ACL, 2024 | General language models on question answering over long inputs | Performance highest when the relevant text is at the start or end of the input, lower in the middle, even for long-context models | Not medical; newer models may differ | arXiv, 2026-11-12 |
| Steinkamp et al., JAMA Network Open, 2022 | 104 million EHR notes for 1.96 million patients | 50.1% of note text duplicated from earlier text about the same patient; 54.2% in 2020 notes | Measures the records, not AI | PMC, 2026-11-12 |
4 patterns that hold across the studies
- Omissions outnumber inventions. In the 2 studies that counted both against a source, omissions came first: 3.45% of source sentences vs 1.47% of note sentences in Asgari, and 25 vs 2 summaries in Grolleau. Reading a summary for errors misses the larger category.
- Negation is a weak point. 30% of invented sentences in Asgari were negations. That is why the negation cue search is its own step.
- Medications score worst. In Deschepper, the medication section had the lowest content scores and the most hallucinations. Check doses against the page image.
- Raters disagree. Deschepper's median kappa of 0.36 means 2 clinicians often judged the same summary line differently. Decide what counts as an error before you score a tool, or your trial will measure your raters.
Why the numbers don't transfer to your files
Every study above summarized short, single-source text: a consultation, a hospital stay, a discharge letter. Medical-legal files differ in 4 ways that each add risk: they are long, so more falls in the middle of the input; they mix sources, so dates and identifiers conflict; they include scans and handwriting, so OCR errors enter before the model; and they carry duplicates and copied-forward text, so a faithful summary can still be wrong. Expect a tool's error profile on your files to differ from any published figure, and measure it with a known-answer test.
The studies also differ from each other in what they call an error. Some count sentences, some count summaries with at least 1 error, and some rate quality on a scale. That is why we don't combine them into a single rate.
What we left out, and why
- Vendor accuracy figures. Several vendors publish a single accuracy or precision number. None we read published the test set, the error definition or who scored it, so we can't compare them and don't repeat them.
- Studies of ambient scribes alone. Drafting a note from a conversation shares failure types with summarizing records, which is why Asgari is in the table, but most scribe studies measure time saved, not errors.
- Preprints we couldn't read in full. We list a study only when we could read its methods or abstract ourselves.
Questions about the evidence
Is there a study of AI summaries of medical-legal record files?
We didn't find 1 with published error counts as of 2026-11-12. The closest are studies of hospital course summaries and clinical notes, listed above. If you know of 1, email aj@medrecords.ai and we will add it.
Do newer models make these studies obsolete?
The rates change with each model; the failure types don't. Omissions, negations and medication errors appear across studies from 2023 to 2026 and across different models, so the checks stay the same.
Are AI summaries worse than human summaries?
Not clearly. Van Veen found adapted LLM summaries equivalent or better than experts' in most cases, and Shemtob found similar accuracy. Human summaries also omit and err. The difference is that you usually know how a colleague works; with a tool, you have to test it.