# References Canonical bibliography for the **ai-lens** visual essays. The first three sections are generated from the card registry in `site/one-chart-five-questions.html` — every benchmark, study, and product named on the timeline, with its primary sources (paper, code, announcement). The appendix is a mechanical sweep of every external link in each artifact, so nothing cited anywhere is lost. _Last regenerated: 2026-08-03._ ## Eras (the four bands) - **Are you booksmart?** — the exam era · 2019–24. Multiple-choice recall — integrated retrieval, not practice. Saturated at 84–90%, at or above physician level; falls to 45–69% on clinical tasks. Ends as marketing. [MedQA paper](https://arxiv.org/abs/2009.13081) · [MedMCQA paper](https://arxiv.org/abs/2203.14371) · [GPT-4 on USMLE](https://arxiv.org/abs/2303.13375) · [JMIR review Dec '25](https://pmc.ncbi.nlm.nih.gov/articles/PMC12706444/) - **Can it converse safely?** — the rubric era · 2023–. Physician-written rubrics over multi-turn conversations; a curated vignette stands in for the patient. [Med-PaLM (Nature)](https://arxiv.org/abs/2212.13138) · [HealthBench](https://openai.com/index/healthbench/) - **Can it do clinical work?** — the task era · 2024–. Real EHR data and agentic workflows — graded on output quality, never on adoption. [MedHELM (arXiv)](https://arxiv.org/abs/2505.23802) · [MedHELM leaderboard](https://medhelm.org/) · [ARISE report '26](https://arise-ai.org/report) - **What does deployment show — and did it change the patient?** — the deployment era · 2025–. Real use and real outcomes, one era: benchmarks curated from live telemetry (usage ran ahead — ~⅔ of US physicians before any benchmark measured their questions), while the outcome question has no public instrument; its raw material runs daily as private telemetry. [HB Pro (arXiv)](https://arxiv.org/abs/2604.27470) · [OpenEvidence 1M/day](https://www.prnewswire.com/news-releases/openevidence-achieves-historic-milestone-1-million-clinical-consultations-between-verified-doctors-and-an-artificial-intelligence-system-in-a-single-day-302712459.html) · [Doximity 2026 report](https://www.doximity.com/reports/state-of-ai-medicine-report/2026) · [Abaluck et al. — NBER w34660](https://www.nber.org/papers/w34660) · [ARISE report '26](https://arise-ai.org/report) · [the essay](model-eval-as-product-analytics.md) ## Benchmarks, studies & milestones - **MedQA** — benchmark · 2020. The first large USMLE-style QA benchmark (~12.7k exam questions) — the exam era's starting gun. [MedQA paper](https://arxiv.org/abs/2009.13081) - **MedMCQA** — benchmark · 2022. 194k exam items across 2.4k health topics — scale for the exam era. [MedMCQA paper](https://arxiv.org/abs/2203.14371) - **Med-PaLM passes** — milestone · Dec 2022. The first model past the USMLE-style pass mark, published in Nature — the exam era's proof of concept. [Med-PaLM (Nature)](https://arxiv.org/abs/2212.13138) - **GPT-4 aces USMLE** — milestone · 2023. Clears USMLE-style exams by >20 points with no medical fine-tuning. The era saturates at 84–90%, at or above physician level. [GPT-4 on USMLE](https://arxiv.org/abs/2303.13375) · [JMIR review Dec '25](https://pmc.ncbi.nlm.nih.gov/articles/PMC12706444/) - **Med-PaLM long-form** — milestone · 2023. Physician-graded long-form answers alongside exam scores — the bridge from item recall into rubric grading. [Med-PaLM (Nature)](https://arxiv.org/abs/2212.13138) - **HealthBench** — benchmark · May 2025. 5,000 simulated multi-turn conversations, 262 physicians, 48,562 rubric criteria. [HealthBench](https://openai.com/index/healthbench/) · [GitHub (simple-evals)](https://github.com/openai/simple-evals) - **Length-adjusted scoring** — patch · 2026. HealthBench revises scoring after verbose answers were found to game rubric coverage — a benchmark patching its own incentive bug. [HealthBench](https://openai.com/index/healthbench/) - **MedHELM** — benchmark · 2025. 121 clinician-validated tasks across 35 benchmarks, on real (including gated) EHR data; quarterly leaderboard. [MedHELM (arXiv)](https://arxiv.org/abs/2505.23802) · [Leaderboard](https://medhelm.org/) · [GitHub](https://github.com/PacificAI/medhelm) - **MedAgentBench** — benchmark · 2025. 300 clinician-derived agentic tasks in a virtual EHR — 70→92% in six months. Sandboxed actions, not deployments. [MedAgentBench (arXiv)](https://arxiv.org/abs/2501.14654) · [NEJM AI](https://ai.nejm.org/doi/full/10.1056/AIdbp2500144) · [GitHub](https://github.com/stanfordmlgroup/MedAgentBench) - **ARISE · MAST** — living benchmark · 2026. A composite “living benchmark” of clinical benchmarks with rolling refresh — and a public call for post-deployment evaluation. [ARISE report '26](https://arise-ai.org/report) · [MAST](https://arise-ai.org/blog/mast) · [MAST leaderboard](https://arise-ai.org/mast/technical) - **HealthBench Professional** — benchmark · Apr 2026. Curated from 15,079 real clinician conversations — the usage era's raw material, frozen into a static test set. [HB Pro (arXiv)](https://arxiv.org/abs/2604.27470) - **Abaluck et al.** — randomized evaluation · Jan 2026 (NBER w34660: Abaluck, Pless, Ravi, Sautmann & Schwartz, “Does LLM Assistance Improve Healthcare Delivery?”). Two clinics in Nigeria: health workers changed prescribing for half of patients and retrospective reviewers scored assisted plans higher — but on-site physicians re-examining the same patients saw little to no improvement, and lab-verified testing effects were mixed. The only public instrument that reaches the patient. [NBER w34660](https://www.nber.org/papers/w34660) · [ARISE report '26](https://arise-ai.org/report) - **AMIE** — milestone · Jan 2024 → Nature 2025. Google's conversational diagnostic AI — outperformed PCPs on diagnostic dialogue in a blinded OSCE-style study; the rubric era's research apex. [AMIE (arXiv)](https://arxiv.org/abs/2401.05654) · [Nature](https://www.nature.com/articles/s41586-025-08866-7) · [Google Research blog](https://research.google/blog/amie-a-research-ai-system-for-diagnostic-medical-reasoning-and-conversations/) - **Med-Gemini** — milestone · Apr 2024. Gemini fine-tuned for medicine — state-of-the-art across 10 of 14 medical benchmarks on release. [Med-Gemini (arXiv)](https://arxiv.org/abs/2404.18416) - **AgentClinic** — benchmark · May 2024. Simulated clinic with patient, doctor, and grader agents — reasoning, information-gathering, tests, diagnosis (npj Digital Medicine). [AgentClinic (arXiv)](https://arxiv.org/abs/2405.07960) · [npj Digital Medicine](https://www.nature.com/articles/s41746-026-02674-7) - **NOHARM** — benchmark + randomised study · arXiv Dec 2025 (v4 Jul 2026); v2 ranking Jul 2026. 1,100 primary-care-to-specialist consultation tasks, 12,747 expert annotations, 10 specialties; grades 20 LLMs **and 4 deployed RAG assistants**, finding potential severe harm in up to 24.6% of cases. Its randomised study of 101 US physicians measures the clinician-plus-AI pair — performance improved with assistance, but physicians under-used the recommendations. Public leaderboard. One of the very few instruments on this chart that grades shipped products and the teaming behaviour rather than the model alone — though still not downstream patient outcomes. [NOHARM (arXiv 2512.01241)](https://arxiv.org/abs/2512.01241) [Doximity write-up](https://blog.doximity.com/articles/which-ai-gives-the-safest-medical-advice) · [Ranking announcement](https://blog.doximity.com/articles/doximity-outranks-openevidence-frontier-models-in-independent-stanford-harvard-study-of-clinical-ai-safety) - **PubMedQA** — benchmark · 2019. 211.3K artificially generated Q&A pairs from PubMed abstracts — the hybrid knowledge/reasoning ancestor. [PubMedQA paper](https://arxiv.org/abs/1909.06146) - **MMLU — medical slices** — benchmark · 2021. 1,785 medical questions inside the 57-subject academic exam suite; academic validation, no clinical review. [MMLU paper](https://arxiv.org/abs/2009.03300) - **BiasMedQA** — benchmark · 2024. 1,273 MedQA items rewritten with cognitive-bias prompts — does framing sway the answer? (npj Digital Medicine) [BiasMedQA (arXiv)](https://arxiv.org/abs/2402.08113) · [npj Digital Medicine](https://www.nature.com/articles/s41746-024-01283-6) - **MedQA-CS** — benchmark · Oct 2024. OSCE-inspired clinical-skills scenarios — 1,667 instruction/input/output triplets; LLM-as-student and LLM-as-examiner tasks (EACL 2026). [MedQA-CS (arXiv)](https://arxiv.org/abs/2410.01553) · [EACL paper](https://aclanthology.org/2026.eacl-long.292/) - **ClinicBench** — benchmark · 2024 (EMNLP). 17 datasets, 11 tasks, medical-expert evaluated — “LLMs are poor clinical decision-makers.” [ClinicBench (GitHub)](https://github.com/AI-in-Health/ClinicBench) - **MEDEC** — benchmark · Dec 2024. 3,848 clinical texts from US hospital systems — find and fix the medical error. [MEDEC (arXiv)](https://arxiv.org/abs/2412.19260) - **MedXpertQA** — benchmark · Jan 2025. 4,460 expert-level items, including multimodal, from USMLE/COMLEX and 17 specialty-board sources. [MedXpertQA (arXiv)](https://arxiv.org/abs/2501.18362) - **LLMEval-Med** — benchmark · 2025 (EMNLP). 2,996 questions from real EHRs and expert-designed clinical scenarios, physician-validated LLM-as-judge. [LLMEval-Med (arXiv)](https://arxiv.org/abs/2506.04078) · [GitHub](https://github.com/llmeval/LLMEval-Med) - **CSEDB** — benchmark · 2025. 2,069 open-ended questions by 32 specialist physicians — dual-track safety and effectiveness; scores drop 13.3% in high-risk scenarios (npj Digital Medicine). [CSEDB (arXiv)](https://arxiv.org/abs/2507.23486) · [GitHub](https://github.com/Medlinker-MG/CSEDB) ## Products, launches & partnerships - **OpenEvidence at 1M/day** — usage · 2026. One million clinical consultations in a single day — usage running years ahead of public measurement, and marketing running on a perfect USMLE score. [OpenEvidence 1M/day](https://www.prnewswire.com/news-releases/openevidence-achieves-historic-milestone-1-million-clinical-consultations-between-verified-doctors-and-an-artificial-intelligence-system-in-a-single-day-302712459.html) · [the 100% USMLE claim](https://www.openevidence.com/announcements/openevidence-creates-the-first-ai-in-history-to-score-a-perfect-100percent-on-the-united-states-medical-licensing-examination-usmle) - **ChatGPT launches** — product · Nov 30, 2022. The usage detonator: generic chat, no medical instrument attached. Within three years, 230 million people a week ask it health questions. [Introducing ChatGPT — OpenAI](https://openai.com/blog/chatgpt) - **ChatGPT Health** — product · Jan 7 → Jul 2026. A dedicated health space grounding answers in linked patient portals, Apple Health, and wellness apps — built with 260+ physicians scoring outputs 600k+ times. GA to all US adults in July '26; ~300M health questions weekly. [Introducing ChatGPT Health — OpenAI](https://openai.com/index/introducing-chatgpt-health/) · [b.well connectivity (PRNewswire)](https://www.prnewswire.com/news-releases/openai-selects-bwell-to-power-secure-health-data-connectivity-for-ai-driven-health-experiences-in-chatgpt-302655598.html) · [TechCrunch, Jan 7 '26](https://techcrunch.com/2026/01/07/openai-unveils-chatgpt-health-says-230-million-users-ask-about-health-each-week/) · [US GA — TechCrunch, Jul '26](https://techcrunch.com/2026/07/23/openai-makes-chatgpt-health-available-to-all-u-s-users/) - **Claude for Healthcare** — product · Jan 12, 2026 (JPM). Announced within a week of ChatGPT Health: HIPAA-ready tools for providers, payers, and patients — native CMS Coverage Database, ICD-10, and PubMed integrations; prior auth and care coordination first. [TechCrunch, Jan 12 '26](https://techcrunch.com/2026/01/12/anthropic-announces-claude-for-healthcare-following-openais-chatgpt-health-reveal/) · [Fierce Healthcare — JPM26](https://www.fiercehealthcare.com/ai-and-machine-learning/jpm26-anthropic-launches-claude-healthcare-targeting-health-systems-payers) · [Becker's — 7 notes](https://www.beckershospitalreview.com/healthcare-information-technology/ai/anthropic-rolls-out-claude-for-healthcare/) - **Nuance DAX Copilot** — product · Sep 2023 (GA). GPT-4-powered ambient documentation goes GA — the scribe wave begins at enterprise scale (Epic-embedded by Jan 2024). [GA announcement](https://www.prnewswire.com/news-releases/nuance-announces-the-general-availability-of-dragon-ambient-experience-copilot-to-further-improve-healthcare-experiences-outcomes-and-efficiency-301939899.html) - **Abridge** — product · 2023 (Epic Pal). First ambient AI integrated into Epic's Pal program — ambient notes inside the clinician's existing workflow; $300M Series E by 2025. [Contrary Research](https://research.contrary.com/company/abridge) · [Fierce Healthcare](https://www.fiercehealthcare.com/ai-and-machine-learning/ambient-ai-startup-abridge-scores-300m-series-e-backed-a16z-and-khosla) - **Doximity Scribe** — product · Jul 2025. Ambient documentation free to verified U.S. physicians — the scribe wave reaches the physician network. [Doximity blog](https://blog.doximity.com/articles/meet-doximity-scribe) - **Doximity Clinical AI Suite** — product · May 2026. Ask, GPT, and Scribe unified into one clinical platform — 150 health systems aboard by June 2026, including 8 of the top 20 hospitals. [Announcement](https://blog.doximity.com/articles/introducing-the-doximity-clinical-ai-suite) · [150 health systems](https://blog.doximity.com/articles/150-health-systems-are-now-on-the-doximity-clinical-ai-suite) - **OpenEvidence × NEJM** — partnership · Feb 2025. Content agreement puts NEJM full text inside the answer engine — trusted sources become a product feature (JAMA followed). [Announcement](https://www.openevidence.com/announcements/openevidence-and-nejm) - **OpenEvidence Visits** — product · Aug 2025. Real-time medical intelligence in the exam room — the answer engine goes ambient, listening to the visit itself. [Announcement](https://www.openevidence.com/announcements/visits-real-time-medical-intelligence) - **EvidenceGrade™** — product · Jul 2026. Grades the strength of the evidence beneath each AI answer — evidence appraisal productized, shipped without a public benchmark. [Announcement](https://www.openevidence.com/announcements/openevidence-launches-evidencegrade-empowering-physicians-to-see-the-strength-of-cited-evidence-beneath-each-ai-answer) - **Claude ships** — product · Mar 2023. Anthropic's assistant opens for business — two years before it specializes in healthcare. [Introducing Claude](https://www.anthropic.com/news/introducing-claude) - **Doximity GPT** — product · Feb 2023. DocsGPT beta puts an LLM inside the physician network — pre-auth and appeal letters over free fax, first. [Doximity blog](https://blog.doximity.com/articles/doximity-gpt-faster-and-peer-reviewed) · [Fierce Healthcare](https://www.fiercehealthcare.com/health-tech/doximity-rolls-out-beta-version-chatgpt-tool-docs-aiming-streamline-administrative) - **GPT-4 ships** — product · Mar 2023. The model that aced the USMLE reaches consumers — clinicians included. [GPT-4 announcement](https://openai.com/research/gpt-4) - **MedGemma** — product · Jul 2025. Google's open medical models on Gemma 3 (4B multimodal, 27B text) — the focused-model bet the field later cooled on. [Google Research announcement](https://research.google/blog/medgemma-our-most-capable-open-models-for-health-ai-development/) - **Epic Art & Emmie** — product · Aug 2025 (UGM). The EHR itself splits by audience: Art for providers, Emmie for patients (plus Penny) — intent segmentation as product strategy. [CNBC — UGM 2025](https://www.cnbc.com/2025/08/20/epic-ugm-2025-epic-touts-new-ai-tools.html) · [Healthcare IT News](https://www.healthcareitnews.com/news/epic-unveils-ai-agents-showcases-new-foundational-models) ## Corpus-wide sources - JMIR systematic review (Dec 2025) — the knowledge–practice gap frame: 84–90% exam → 45–69% clinical tasks → 40–50% safety. - ARISE / MAST — turn-level agent safety gates (Stanford Medicine network); graded in `one-recommendation.html`. - Abaluck et al., NBER w34660 — the randomized-trial overpass in `one-recommendation.html`. - Doximity *State of AI in Medicine* 2026 report — physician adoption numbers. - OpenEvidence announcements index (scoured 2026-08) — partnerships (NEJM, JAMA, NCCN, ACOG, AAP, Cedars-Sinai agentic pilot), press milestones. - Doximity blog (scoured 2026-08) — Scribe, Clinical AI Suite, NOHARM v2 ranking posts. ## Prior art — evaluation frameworks & method The chart tracks instruments and products. Two adjacent literatures set the terms this essay argues inside of: the governance layer that now mandates outcome monitoring, and the epidemiological method that defines how an outcome may be attributed at all. Verified August 2026; citation figures are OpenAlex `cited_by_count` and FWCI (field- and age-normalised — 1.0 is world average for the field and year). ### The evidence gap — how little reaches real patients - **Bedi et al. — "Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review"** · JAMA 2025;333(4):319–328. 519 studies (Jan 2022 – Feb 2024): **5% used real patient-care data**, 44.5% assessed medical knowledge via licensing exams, deployment considerations appeared in 4.6%, financial impact in one. 517 citations · FWCI 44.0. [JAMA](https://jamanetwork.com/journals/jama/fullarticle/2825147) · doi:10.1001/jama.2024.21700 - **Chen, Alyakin, Seas et al. — "LLM-assisted systematic review of large language models in clinical medicine"** · Nat Med 32, 1152–1159 (2026). 4,609 studies (Jan 2022 – Sep 2025, ~3.2/day); **1,048 used real-world patient data and only 19 were prospective randomised trials**; 1,857 simulated, 1,704 exam-style. 34 citations · FWCI 69.3. [Nature Medicine](https://www.nature.com/articles/s41591-026-04229-5) ### Governance — the mandate without the instrument - **Joint Commission + CHAI — "Guidance on the Responsible Use of AI in Healthcare" (RUAIH)** · 17 Sep 2025, first release of a partnership begun June 2025. Requires AI governance structures and regular quality monitoring for "changes in outcomes" — without specifying unit, window or denominator. Governance playbooks follow, then a **voluntary AI certification** open to 22,000+ accredited organisations. [Joint Commission](https://www.jointcommission.org/en-us/knowledge-library/news/2025-09-jc-and-chai-release-initial-guidance-to-support-responsible-ai-adoption) · [CHAI](https://www.chai.org/news/joint-commission-and-coalition-for-health-ai-chai-release-initial-guidance) · [guidance PDF](https://digitalassets.jointcommission.org/api/public/content/dcfcf4f1a0cc45cdb526b3cb034c68c2) - **FAIR-AI — "A practical framework for appropriate implementation and review of artificial intelligence in healthcare"** · npj Digital Medicine, Aug 2025. Health-system templates for pre-implementation review and post-implementation monitoring; risk tiering and a "Safe AI Plan". 58 citations · FWCI 18.2. [npj Digital Medicine](https://www.nature.com/articles/s41746-025-01900-y) · doi:10.1038/s41746-025-01900-y ### Method — what "closing the loop" inherits - **Donabedian — "Evaluating the Quality of Medical Care"** · Milbank Mem Fund Q 44(3), 1966; reprinted Milbank Q 83(4):691–729, 2005. Structure → process → outcome. 6,155 citations · FWCI 225.6. - **Hernán & Robins — "Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available"** · Am J Epidemiol 183(8), 2016. Time zero, eligibility, follow-up window, outcome — the formal version of open → watch → close. 2,882 citations · FWCI 56.5. doi:10.1093/aje/kwv254 - **Glasgow, Vogt & Boles — RE-AIM** · Am J Public Health 89(9):1322–7, 1999. Reach, effectiveness, adoption, implementation, maintenance. 7,413 citations · FWCI 45.7. doi:10.2105/ajph.89.9.1322 **Where this work differs.** Target trial emulation requires eligibility criteria and a comparator; event→outcome reconciliation has neither, so it yields association, not attribution. Silent-trial / shadow-mode evaluation sits *before* go-live, whereas this reconciles *live* recommendations after the fact. Clinical decision support has measured acceptance and override rates for decades, but for discrete alerts with a click to count — the open problem is generalising that to free-text output with the label harvested from the chart. And the unit measured throughout is clinician-plus-AI, never the model alone. ### Citation weight of the instruments on the chart FWCI is the better seminality signal than raw counts (it normalises for field and year); percentile is within field-year. Ranked by FWCI: | Work | Year | Citations | FWCI | |---|---|---|---| | Med-PaLM (Nature) | 2023 | 3,479 | 453.6 | | Donabedian | 1966/2005 | 6,155 | 225.6 | | MedHELM | 2026 | 25 | 92.7 | | AMIE (Nature) | 2025 | 273 | 84.1 | | Chen — Nat Med LLM review | 2026 | 34 | 69.3 | | Hernán — target trial | 2016 | 2,882 | 56.5 | | RE-AIM | 1999 | 7,413 | 45.7 | | Bedi — JAMA review | 2024 | 517 | 44.0 | | MMLU | 2021 | 331* | 39.6 | | MEDEC | 2025 | 15 | 32.2 | | BiasMedQA | 2024 | 57 | 20.7 | | FAIR-AI | 2025 | 58 | 18.2 | | PubMedQA | 2019 | 628 | 11.8 | No reliable signal yet (too new, or split across preprint/published records): MedAgentBench (4), MedXpertQA (3), AgentClinic (22), Med-Gemini (94), MedMCQA (74), GPT-4-on-USMLE (517, no FWCI). Not indexed as works at all: **HealthBench** (an OpenAI release, not a paper) and **Abaluck et al.** (NBER working paper) — absence here is a publishing-venue artefact, not a measure of importance. *MMLU's OpenAlex record undercounts badly; the arXiv/ICLR original is cited in the five figures. ## Appendix — every external link, by artifact ### `site/eval-quadrant-gpu.html` — 19 links - - - - - - - - - - - - - - - - - - - ### `site/one-chart-five-questions.html` — 64 links - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ### `site/one-recommendation.html` — 1 links - ### `site/physician-prompting.html` — 7 links - - - - - - - ### `site/state-of-healthcare-benchmarks.html` — 19 links - - - - - - - - - - - - - - - - - - -