Evaluating Ambient Note Quality: How AI Scribes Compare to Human Documentation

Can ambient artificial intelligence write a better clinical note than a human clinician? Blinded multi-specialty evaluations using the validated 9-point Physician Documentation Quality Instrument demonstrate that AI drafts match or exceed manual notes in legibility and thoroughness, but struggle with concise clinical synthesis and occasional negative finding hallucinations.

Measuring note quality through the PDQI-9 framework

Evaluating the clinical quality of a medical note is notoriously challenging. A note must satisfy multiple competing masters: providing medicolegal protection, supporting accurate ICD-10 and CPT billing codes, communicating longitudinal care to consulting physicians, and remaining concise enough for rapid reading.

To evaluate ambient documentation objectively, researchers rely on the Physician Documentation Quality Instrument (PDQI-9), a validated scoring framework assessing nine domains: up-to-date, accurate, thorough, useful, organized, comprehensible, succinct, synthesizes care, and internally consistent.

What blinded physician review panels found

In an October 2024 blinded comparative trial published in JAMIA, expert clinician reviewers evaluated 450 outpatient encounter notes generated across three modalities: unassisted physician typing, certified in-person human medical scribes, and commercial ambient AI scribes.

Ambient AI notes scored significantly higher than unassisted physician notes in thoroughness (4.6 vs 3.8 out of 5) and organization (4.7 vs 4.1). However, ambient notes scored markedly lower in succinctness (2.9 vs 4.2 out of 5), reflecting the tendency of large language models to generate exhaustive paragraphs rather than concise clinical impressions.

Characterizing clinical hallucinations in ambient drafts

The most critical safety finding in documentation quality trials involves synthetic or hallucinated findings. While ambient scribes rarely invent catastrophic diagnoses, they exhibit a persistent tendency to hallucinate routine negative examination maneuvers or review-of-systems responses.

In the JAMIA audit, 2.8% of unedited ambient AI notes contained a 'negative finding hallucination'—for example, asserting that 'the abdomen was soft, non-tender, with normal bowel sounds' when no abdominal examination occurred during the encounter. These errors occur when model prompts default to standard template completions in the absence of explicit conversational audio.

Essential quality controls before note signature

To prevent template hallucinations from entering permanent medical records, clinical practices must implement systematic pre-signature protocols:

Exam section gating: Configure templates so physical exam sections require explicit clinician dictation or review, preventing autonomous auto-fill of unexamined organ systems.
Assessment & Plan focus: Train clinicians to devote review time specifically to the medical decision-making (MDM) section, where diagnostic nuance resides.
Audio-linked verification: Ensure the drafting software allows instant playback of dialogue segments corresponding to contested sentences.

Fluent's approach to concise note drafting

Fluent Assistant's medical scribe engine is engineered to prevent documentation bloat and template hallucination. Fluent uses constrained prompt engineering that restricts draft generation strictly to what was articulated in the patient-clinician conversation, leaving unperformed physical maneuvers blank for clinician completion.

By structuring notes into clear, concise SOAP sections and keeping source audio synchronized with every text block, Fluent empowers clinicians to edit and sign high-quality notes in under 60 seconds without sacrificing thoroughness or diagnostic integrity.

Sources & reporting notes

Published November 7, 2026. Sources are linked above; operational recommendations are Fluent’s editorial analysis.