Citation checks: how to verify the pages an AI summary cites
Answer in brief: a page citation proves only that a page exists. Open it, read the whole sentence around the highlight, and confirm the page is this patient's, is a result rather than an order, and isn't a duplicate with a different date. Check every line you rely on, then sample the rest: 59 clean random lines rule out error rates above 5%, while 10 clean lines only rule out rates above 26%.
Limits: We can't vouch for any tool's error rate on your files: the studies we cite measured other tasks and other models, and vendor features are as each vendor's site stated on the capture date.
6 ways a correct-looking citation misleads
Most AI summary tools now cite pages, and most vendor pages say "click to verify". The click is the easy part. These are the ways a real page can sit behind a wrong line.
| Citation problem | Synthetic example | What to check on the page |
|---|---|---|
| Supports half the line | "X-ray negative; MRI showed partial tear (p. 3)." Page 3 is the X-ray only. | Each clause of a compound line has its own source |
| Order, not result | "MRI showed a full-thickness tear (p. 5)." Page 5 orders the MRI. | The page is a report, with a result date |
| Secondhand mention | "MRI: partial-thickness tear (p. 12)." Page 12 is an addendum quoting a report that isn't in the file. | The cited page is the primary record; if not, the primary is listed as missing |
| Wrong patient | "Full-thickness tear with retraction (p. 11)." Page 11 is another patient's MRI. | Name, date of birth and record number in the page header |
| Duplicate with another date | 2 visits cited a week apart, same text, different fax headers. | Date of service, not fax date; identical text means 1 visit |
| Page number shift | Citation says p. 214; the finding is on p. 215. | Whether the tool cites PDF page index or Bates number, and whether pages were added or removed after processing |
The last row trips up exports. A summary cites PDF page 214, someone adds a cover sheet or re-Bates the file, and every citation after it is off by 1. Ask whether exported citations are live links or plain page numbers, and re-check a few after any change to the file.
Citation granularity: document, page or sentence
How precisely a tool points to its source decides how long each check takes. A link to a whole document on a 300-page hospital chart sends you hunting. A page link narrows it to 1 page. A highlighted sentence lets you check a line in seconds. This table records what each vendor's own site stated on the capture date. It is not a test of the tools.
| Tool | Source link the vendor describes | Human review stated | Built for | Source, captured |
|---|---|---|---|---|
| Abstractive Health | Sentence level: each summary sentence linked to its original note | Not stated | Clinicians at the point of care | vendor page, 2026-11-12 |
| DigitalOwl | "Click-to-evidence" to the original source document | Not stated | Life, disability and P&C insurers; legal | vendor page, 2026-11-12 |
| Dodonai | Page level: page citation for every extracted event; click opens the page | Not stated | Attorneys | vendor page, 2026-11-12 |
| InPractice | Citations to the original records; source-backed Case Chat answers | Not stated | Law firms, QMEs and IMEs, claims | vendor page, 2026-11-12 |
| Lexitas Record Insights | Hyperlinks to each record, as described in a customer quote on the page | Not stated | Legal | vendor page, 2026-11-12 |
| MediScan AI | Page level: editable summaries linked to the original record pages | Says the reviewer stays responsible for the final report | Physician evaluators, med-legal | vendor page, 2026-11-12 |
| MOS ReviewGenX | Hyperlinks to original records | Medical professionals review AI summaries | Legal and claims | vendor page, 2026-11-12 |
| Wisedocs | Document level: each summary hyperlinked to its original document | Human clinical QA | Claims | vendor page, 2026-11-12 |
| Medrecords AI (ours) | Sentence level: a chronology event opens its page with the quoted sentence highlighted | No; you verify and sign | Physicians, hospital risk teams, IME and QME | help center page, 2026-11-12 |
"Not stated" means we didn't find it on the page we read, not that the tool lacks it. Some vendors publish accuracy figures; we don't repeat them, because none we read published the test set behind the figure. Corrections: aj@medrecords.ai.
How many lines to check: the sampling calculator
Check every line you rely on. For the rest, a random sample tells you how far to trust the summary, and the size of the sample sets how much it can tell you. Enter what you checked; the calculator gives the highest error rate the sample rules out, using an exact binomial bound. It runs in your browser and sends nothing.
| Lines checked | Failures | Error rate ruled out above (95%) |
|---|---|---|
| 10 | 0 | 25.9% |
| 10 | 1 | 39.4% |
| 20 | 1 | 21.6% |
| 29 | 0 | 9.8% |
| 59 | 0 | 5.0% |
| 59 | 1 | 7.8% |
| 149 | 0 | 2.0% |
Exact binomial (Clopper-Pearson) 1-sided bounds, computed 2026-11-12. These assume random picks and independent errors. Errors cluster on poor scans and handwriting, so a clean sample from typed pages says little about the handwritten ones; sample each kind separately.
The reverse spot check: sampling for omissions
A sample of summary lines can't find an omission, because the missing fact isn't a line. Turn the sample around:
- Number the source pages after duplicates are removed.
- Pick 10 to 20 page numbers at random. Skip blank pages, cover sheets and billing pages, and note how many you skipped.
- Read each page and write down any fact that matters to your question: a finding, a result, a work status, a history item.
- Find each fact in the summary. Count the ones that are missing or wrong.
- If a page with a relevant fact is missing from the summary entirely, search for others from the same source and date range. Omissions cluster by source.
The same table and calculator apply, with "facts found on sampled pages" in place of "lines checked". Add 1 targeted step random sampling can't replace: read every page dated before the injury or event yourself. Prior history is the omission that changes opinions most often in IME and causation work.
Questions about checking citations
Is clicking 10 random citations enough?
It is a floor. 10 clean lines rule out error rates above about 26% at 95% confidence, which is a weak claim for a report you sign. Check every relied-on line, then sample 30 to 60 others if the summary is long.
What should I do when a sampled citation is wrong?
Treat it as a sign of a cluster. Check every line from the same source and section, find out why it failed (poor scan, wrong patient, duplicate), and then take a fresh random sample.
Does a highlighted sentence mean I don't need to read the page?
No. Read the sentences before and after the highlight. Negations, hedges and the speaker often sit just outside the highlighted words.