Medical records summary AI: how it fails and how to check it
Answer in brief: an AI summary of a medical record fails in 10 predictable ways, and the 2 most common, leaving facts out and losing context such as a negative or a "rule out", look correct on a read-through. Check every line you will rely on against its cited page, sample the rest, search the source for what the summary skipped, and test the tool on a file where you already know the answers.
Limits: We can't vouch for any tool's error rate on your files: the studies we cite measured other tasks and other models, and vendor features are as each vendor's site stated on the capture date.
Can you trust an AI summary of a medical record?
Trust it the way you trust a resident's first draft: as a fast, mostly right starting point that you check before you sign. Published studies put invented facts in a small share of sentences, but find omissions more often, and a citation only proves that a page exists. You can rely on it, for the lines you checked and within the bounds your sample sets, after 4 checks: every relied-on line opened at its source, a random sample of the rest, a search for what was left out, and a sweep for negatives, hedges, dates and sides. This site shows each check, with the math for how many lines to open.
10 ways AI record summaries go wrong
"Hallucination" gets the headlines, but it is 1 failure type of 10. Each type has a cue you can search for in the source or the summary, which is faster than rereading. The failure modes page covers each one with a synthetic example.
| Failure type | What it looks like | Search cue |
|---|---|---|
| Omission | A prior injury, an abnormal result or a consult never makes it into the summary | Source pages the summary never cites |
| Invented fact | A finding, dose or event the file doesn't contain | Lines with no citation, or a citation that doesn't say it |
| Lost negative | "Denies numbness" becomes "numbness" | denies, no, negative, without, absent |
| Hedge dropped | "Rule out rotator cuff tear" becomes "rotator cuff tear" | r/o, possible, probable, cannot exclude, suspicious for |
| Wrong date | A fax, print or signature date used as the visit date | Dates that match a fax header or e-signature stamp |
| Wrong side or level | Right becomes left, L4-5 becomes L5-S1 | left, right, bilateral, spinal levels |
| Dose or unit error | "5/325" read as 325 mg of an opioid | mg, mcg, units, combination products |
| Attribution error | What the patient reported becomes what the physician found | pt states, reports, per patient, told by |
| Copy-forward echo | A stale plan or work status repeated as if current | "unchanged", identical paragraphs across visits |
| Wrong patient | Another patient's page folded into the history | Name, date of birth and record number on every new source |
Types and cues are this site's synthesis from the studies on the evidence page and common record defects. The examples are synthetic, no real patient.
Where errors enter an AI summary
Errors don't all come from the language model. Some start at the scan, some at the split into documents, and some in the file itself before any software touches it. Knowing the stage tells you what to check: an OCR misread shows on the page image, a co-mingled page shows in the header, and an omission shows only when you go back to the source.
What the published studies say
No published study measures AI summaries of 1,000-page medical-legal files. The closest evidence comes from clinical notes and hospital summaries, and it points the same way each time:
- In 450 AI-drafted clinical notes, 1.47% of sentences contained an invented or wrong fact and 30% of those were negations; 3.45% of source sentences were left out (Asgari, npj Digital Medicine, 2025).
- Physicians who reviewed 100 AI hospital course summaries reported omissions in 25, inaccuracies in 20 and hallucinations in 2 (Grolleau, JAMA Network Open, 2026).
- Across 30 synthetic discharge letters, missing information was the most common error, and the medication section did worst (Deschepper, JMIR Medical Informatics, 2026).
- Language models use the start and end of a long input better than the middle (Liu, Transactions of the ACL, 2024). That study tested general question answering, not record summaries, but it predicts that a page buried mid-file, such as an old primary care note, is the kind a tool skips.
The lesson for a reviewer: spend at least as much time looking for what is missing as for what is wrong. The evidence page lists 10 studies with what each measured and why the numbers don't transfer directly to your files.
How to verify an AI summary in 4 passes
Reading the summary start to finish feels like checking but catches the least. These 4 passes each target different failure types, and together they cover all 10.
- Relied-on lines, 100%. List every line your opinion or plan rests on: mechanism, key imaging, procedures, work status changes, prior history. Open the cited page for each and read the sentence around it, not just the highlighted phrase. The citation checks page lists the 6 ways a citation can point to a real page and still be wrong.
- Random sample of the rest. Pick lines at random and check them the same way. The number you check sets what you can claim: 10 clean lines only show the error rate is probably under 26%, while 59 clean lines get it under 5%.
- Reverse spot check for omissions. Pick source pages at random, read each, and confirm its key facts appear in the summary. This is the only pass that finds what was left out, because a sample of summary lines can't contain a missing fact.
- Cue sweep. Search the source for negation, hedge, laterality, dose and date cues, and confirm the summary kept each one on the facts that matter. The verification checklist has the search strings ready to copy.
How many lines should you check?
"Spot-check 10 lines" is common advice, including from vendors. It is a floor, not a standard. If you check n lines picked at random and all are correct, the table shows the highest error rate you can rule out with 95% confidence (exact binomial bound).
| Random lines checked, 0 errors | Error rate you can rule out above |
|---|---|
| 10 | 25.9% |
| 20 | 13.9% |
| 30 | 9.5% |
| 59 | 5.0% |
| 100 | 3.0% |
| 300 | 1.0% |
Computed 2026-11-12 as 1 minus 0.05 to the power 1/n. Assumes lines are picked at random and errors are independent; errors cluster on hard-to-read pages, so treat the bound as optimistic. Try your own numbers in the sampling calculator.
2 practical rules follow. First, sampling is for the lines you don't rely on; the lines you do rely on get checked 100%. Second, 1 error in a small sample changes the picture: 1 error in 10 lines means the true rate could be as high as 39%, so expand the check to the whole section before you go on.
Test a tool on a file where you know the answers
A vendor demo uses a clean file. Your files have fax headers, copied-forward notes and pages from the wrong patient. The fastest honest test is a file with known traps. The synthetic test file is 12 pages on a fictional patient with 15 planted traps: a "rule out" that should stay a rule-out, a lab collected on 1 date and resulted on another, a copied-forward work status, a combination opioid dose, and an MRI report from a different patient with a similar name. Download it as plain text, run it through any tool, and score the output against the answer key.
It is synthetic, no real patient, so you can paste it into a tool you haven't signed a BAA with yet. Use your own closed files only after a BAA is in place.
What a citation proves, and what it doesn't
A page citation proves that the tool can point to a page. It doesn't prove that the page says what the line says, that the page belongs to this patient, or that nothing contradicting it sits 40 pages later. Tools also differ in how precise the pointer is. Some link a line to a whole source document, some to a page, and some highlight the sentence. A document-level link on a 300-page hospital chart saves little time; a highlighted sentence turns a check into a glance. The tool table shows which level 9 tools state on their own sites, captured 2026-11-12.
When an AI summary is the wrong tool
AI summaries save the most time on long, messy files where the risk is missing something in volume. They help least, and can hurt, in 3 situations:
- The opinion turns on 1 page. If the case rests on a single operative note or a single imaging read, read it yourself. A summary adds a layer between you and the words that matter.
- The file is mostly handwriting or poor scans. Every tool reads these worst. Expect to check far more lines, and budget for it.
- You can't check the output. A summary without page links, or one you won't have time to verify, is a liability with your signature on it.
Questions about medical records summary AI
How accurate are AI summaries of medical records?
No study we found has measured it on medical-legal record files, and none of the vendor accuracy figures we read came with a published test set. In clinical-note studies, invented facts appear in a small share of sentences while omissions are more common: physicians flagged omissions in 25 of 100 AI hospital summaries and hallucinations in 2. Measure it on your own files with a known-answer test.
What is the most common error in an AI medical summary?
Omission. In both the 2025 npj Digital Medicine study and the 2026 JAMA Network Open study, leaving facts out was more common than inventing them. Omissions are also the hardest to catch, because every line that is there can be correct.
How many lines of an AI summary should I check?
Check 100% of the lines you rely on, then a random sample of the rest. 10 clean random lines only rule out error rates above about 26%; 59 clean lines rule out rates above 5% with 95% confidence. Use the sampling calculator for your numbers.
Does a page citation mean the line is correct?
No. It means the tool linked a page. The page can support only half the sentence, describe an order instead of a result, belong to another patient, or be a duplicate of a page with a different date. Open the page and read the sentence around the highlight.
Can I test an AI summary tool without using real patient records?
Yes. The synthetic test file on this site has no real patient, so you can run it through any tool before a BAA is signed, then score the output against the 15-trap answer key.