State of Healthcare AI Benchmarks

Five things true at once

• Exams are saturated. Frontier models score 84–90% on USMLE-style tests — at or above physician level. One vendor markets a perfect score.
• The gap is measured. Performance falls 39–45 points from exams to practice tasks (39 benchmarks); one system fell 95% → 34% from evaluation to deployment.
• Usage ran ahead. ~⅔ of US physicians use an AI tool; one platform logs ~27M encounters a month. Adoption follows peer signal, not leaderboards.
• New benchmarks use real data, same cadence. Real EHR tasks, real clinician chats — still shipped as frozen test sets.
• The evaluation that matters is private. Decision streams, utilization, outcome reconciliation exist only as vendor telemetry — invisible to the literature.

The flagship evaluations, mid-2026

Benchmark	Steward	What it grades	Data	Cadence
USMLE / MedQA era	—	Medical knowledge recall, multiple choicethe 2023–24 generation; now table stakes	Exam items	Static, saturated
HealthBench May 2025	OpenAI	Multi-turn health conversations against physician-written rubrics5,000 conversations · 262 physicians · 48,562 criteria	Simulated	Static + "Hard" subset
MedHELM 2025	Stanford + partners	121 clinician-validated tasks across 5 categories, 35 benchmarksdecision support · documentation · communication · research · admin	Real EHR data (incl. private/gated)	Quarterly leaderboard
ARISE · MAST 2026	Stanford Medicine network	Composite "living benchmark" of clinical benchmarksNOHARM, MedAgentBench, CPC-Bench, PhysicianBench …	Mixed clinical	Rolling refresh
HealthBench Professional Apr 2026	OpenAI	The tasks clinicians actually bring to a chatbot at workcurated from 15,079 real clinician conversations	Real usage, frozen	Static
RCTs rare	Academia	Whole-workflow effect on real decisions and outcomese.g. Abaluck et al. 2026: assistance changed deliberation, not test appropriateness	Real deployments	Years per verdict

Five eras, each asking a more real question

• Each era's question absorbs the last. Each era's data gets harder to publish.
• The outcome era has no public benchmark. Its data lives in product telemetry.

Exams stopped discriminating

• When every model clears the bar, the bar carries no information. Remaining headroom lives in curated "hard" subsets.
• Every flagship is redesigning around realism: real EHR tasks, real conversations, agents.

Knowledge ≠ practice, now quantified

• The JMIR authors' verdict on exam scores: "insufficient and misleading proxies for clinical readiness."
• CMU's framing: evaluation silently assumes perfect exposure, cooperative users, frictionless action. Deployment provides none.

Usage didn't wait for the evaluations

• 69% of users say AI improved care; 71% name accuracy as their top concern. Self-report is what you get without a reconciliation layer.
• No clinician ever chose a tool because of MedQA.

Benchmarks are drifting up, not right

• Up = more scope. Real EHR tasks (MedHELM), real clinician chats (HealthBench Professional), composite clinical suites (MAST), the occasional RCT. The response to saturation is realism.
• Not right = same cadence. Curate, freeze, grade, publish. HealthBench Pro took 15,079 real conversations and froze them into a static test set.
• The benchmark community says it itself. ARISE's 2026 report: traditional QA scores are saturated; the field needs "multi-turn unstructured real-world data" and "prospective and post-deployment real-world scenarios."
• Benchmark design is converging on watching real usage. What still differs is cadence and data access — the chasm.

Evaluation is split by a public/private wall

• It's a property line. Data ownership decides which side you work on. Traffic runs one way: usage data, exported left and frozen into a benchmark. Nothing flows right.
• The shared question — did it change what happened to the patient? — needs the right side's data at the left side's rigor.

Put every effort on one timeline and the coverage gap is visible

Same artifacts, three audiences

SCOPE — exam item → the whole outpatient visit & its outcome

the chasm · a public / private wall

Public — published & citable

Private — proprietary telemetry

Field studies & surveys

Exam hall

Ops dashboards

Evaluation as product analytics

USMLE / MedQA-style exams84–90% — saturated, at/above physician level

"100% on USMLE" marketingcapability score used as a product claim

HealthBench (2025)5k simulated conversations · 262 physicians' rubrics

ARISE · MAST"living benchmark" — faster refresh, still task-scoped

MedHELM121 clinician-validated tasks · real EHR data

HealthBench Professional (2026)curated from 15k real clinician chats — frozen into a benchmark

Doximity annual survey69% *say* AI improved care — self-report outcome

JMIR systematic reviewthe 39–45 pt knowledge–practice gap, quantified

Abaluck et al.gold-standard outcome · years from question to answer

ARISE State of Clinical AI '26annual synthesis — itself calls for post-deployment eval

CI eval regressionsLLM-judge suites on every model / prompt change

Per-turn feedback streamthumbs + free-text rejection reasons

Latency & cost telemetryper turn, per org

Delivery → exposure → action funnelviewed-denominator accept rates, per clinic

Workload-adjusted utilizationminutes per appointment-provider-day, not DAU

Decision streams (app DB)who accepted, deduplicated, internal actors excluded

Post-visit outcome reconciliationEHR evidence attributed back to what the AI said

OpenEvidence query logs~27M encounters/mo — exists, unpublished as evaluation

VERDICT · publication cadence⟵ the chasm ⟶PROCESS · adaptive cadence

Views

🗺 The visit map 🩺 Useful to clinicians 🫀 Useful to patients 🎓 Useful to academia ▶ Cycle the views

What no benchmark measures

• Standard product analytics — funnels, denominators, identity hygiene — applied to the visit as the unit of work.

Most model evaluation should be product analytics

• Full argument: the long-form essay Model evaluation should look like product analytics (companion to this deck).

What should happen next

Sources