Questions about checking AI summaries of medical records

What goes wrong most often in an AI summary of medical records?

Facts left out. In the studies that counted both, omissions outnumbered invented facts. They are also the hardest to see, because nothing in the summary looks wrong. See the evidence page.

Why would an AI summary invent a finding?

Language models write the most likely next words. When a note orders an MRI for a possible tear, "MRI showed a tear" is a likely sentence, so a tool can write it with no report in the file. Always trace a result to a result report.

Can the source records themselves cause summary errors?

Yes. Copied-forward text, duplicate pages with different dates and pages from another patient all pass through a faithful summary unchanged. These need checks on the file, not on the model. See failure modes 9 and 10.

Which part of a summary deserves the most suspicion?

Medications, negatives and dates. Medication sections scored worst in a 2026 study of AI discharge summaries, negations were 30% of invented sentences in a 2025 study, and dates fail because 1 page can carry many of them.

How long does it take to verify an AI summary?

We know of no published benchmark for medical-legal files. Labeling the errors in 1 AI summary of a short medical note took a trained clinician 92 minutes on average in a 2024 pilot, so the practical approach is to check every relied-on line and sample the rest. The checklist is built that way.

How do I pick lines at random?

Number the summary lines and use any random number generator, or take every nth line from a random start. Don't pick the lines that look suspicious; that measures your instincts, not the tool.

What is a reverse spot check?

Picking random pages of the source, reading them, and confirming their key facts made it into the summary. It is the only sampling method that finds omissions. See citation checks.

Which date should a summary use for a visit?

The date of service, not the fax, print, dictation or signature date. For labs, the collection date, plus the result date when timing matters. The date-type field guide lists 12.

Should I record how I checked the summary?

Yes, if it feeds a report you sign. A short note of what you checked, how many lines and what failed answers the question of how you know the summary was right. The verification note template takes 2 minutes.

Do all AI summary tools cite their sources?

Most medical record tools we read state some kind of source link, but the precision differs: a whole document, a page or a highlighted sentence. The tool table records what 9 vendors stated on 2026-11-12.

Can I test a tool before signing a BAA?

Yes, with synthetic data only. The test file has no real patient and 15 planted traps with an answer key. Never upload real records to a tool before a BAA is signed.

Can I use ChatGPT or another general chatbot to summarize records?

Not with real records unless the vendor has signed a BAA that covers your account; ask before you paste anything. Then check whether it cites page numbers on a long file, since every check on this site depends on them. The synthetic test file shows both in a few minutes.

Is a vendor's published accuracy figure useful?

Only if the vendor publishes the test set, the error definition and who scored it. None we read did. Your own known-answer test is more useful.

Who is responsible for errors in an AI summary used in my report?

You are, if you sign the report. MediScan, for 1, says on its own site that the tool doesn't replace the reviewer's responsibility. The tool is a drafting aid; the opinion and its factual basis are yours.

Do I have to disclose that I used AI?

It depends on your jurisdiction, the forum and the client, and the rules are changing. Check the rules that apply to your work; this site doesn't give legal advice.