OpenAI o1 Outperforms Doctors in Diagnosing from Emergency Department Data
According to a publication in Science, the AI model o1-preview achieved 78.3% accuracy on the most challenging cases from NEJM, surpassing human doctors and previous versions of GPT-4. The largest gap in favor of AI was observed during the triage stage in the emergency department, when the least information about the patient is available.
While headlines scream "AI surpasses doctors in the ER," the real story is more nuanced and concerning. On April 29, 2026, the journal Science published a study that is not really about an algorithm's victory, but about a shift in the methodology of AI medical research. A team led by Peter Broder from Beth Israel Deaconess Medical Center and Harvard Medical School conducted the first blind comparison of the OpenAI o1 model with human doctors, not on training vignettes, but on real, unstructured data from the emergency department—and simultaneously demonstrated that the model fails precisely where the stakes are highest.
The Core: Not Replacing Doctors, but Redefining the Standard of Evidence
The study consisted of six experiments. Five used structured clinical cases, the sixth used 76 real cases from the emergency department of Beth Israel Deaconess Medical Center in Boston. At all stages, the o1 model outperformed doctors: during triage, the accuracy of differential diagnosis was 67.1% versus 55.3% and 50.0% for two attending physicians. The gap was largest precisely when information was minimal—at the first patient contact.
But the key finding of the study is hidden in the details. The authors emphasize that o1-preview did not show statistically significant superiority over doctors in identifying "cannot-miss diagnoses"—conditions whose omission is life-threatening. The model brilliantly generates a broad differential, but it is no better than a human at identifying the primary threat. This is not a minor caveat, but a central result for safety assessment.
The methodology is impressive: the diagnoses generated by the model and by doctors were evaluated by two additional specialists who did not know whether they were analyzing a human or machine response. Blinding worked: reviewers correctly identified the source in only 15% and 3% of cases, respectively. In other words, o1's texts are clinically indistinguishable from those written by a competent physician.
Timeline and Context: The Race for Evaluation Methods
The study did not emerge in a vacuum. As early as the 1950s, Ledley and Lusted proposed case-based benchmarks as the standard for evaluating medical computing systems. Since then, NEJM clinicopathological conferences have served as the gold standard: each generation of differential diagnosis generators has been tested on these cases.
Until 2024, machine learning models for diagnosis were so weak that comparison with doctors was meaningless. The situation changed with the advent of GPT-4, which began to approach human level in some tasks. o1-preview became the first reasoning model to undergo full-scale blind testing against doctors of varying training levels.
Fundamentally new in Broder's design is the use of real emergency department records rather than cleaned vignettes. This addresses the main criticism of previous work: training cases are neatly structured, whereas real clinical information is chaotic, redundant, and contains errors. The sixth experiment proved that the model can extract signal from noise as well as an experienced clinician.
Who Wins and Who Loses
Those who bet on the "augmented doctor" rather than the "replacement doctor" win. OpenAI gains validation of its reasoning architecture in the most demanding domain—medicine. This strengthens the company's position in negotiations with hospital networks and insurers. Harvard Medical School and BIDMC solidify their status as centers for clinical AI evaluation: a publication in Science with transparent methodology creates a template that others will replicate.
Developers of previous-generation clinical decision support systems—symptom checkers based on Bayesian networks and rules—are definitively outdated. Startups selling "AI doctors" as standalone entities also lose: Broder's study clearly shows that the model is unsafe without a physician precisely at critical points.
What the Media Doesn't Say
The first non-obvious point is the conflict of interest. Adam Rodman, one of the co-authors, is a visiting researcher at Google DeepMind, and Eric Horvitz is a Microsoft employee. This does not compromise the data, but explains why the study focuses on the model's capabilities rather than the risks of its deployment.
The second point is the model's textual limitation. o1 does not have access to images, physical exam data, or auditory information. In emergency medicine, the patient's appearance, smell, and breath sounds are critical signals. A model that works only with text is fundamentally limited.
The third and most important aspect is the regulatory gap. Concurrently with the Science publication, the journal Nature Medicine released two articles demanding that medical AI demonstrate improvements in clinical outcomes, not just accuracy on benchmarks. This timing is not coincidental: the AI research community acknowledges that most deployed systems have not proven real benefit to patients.
Forecast: The Next 30 and 90 Days
In the next 30 days, I expect that at least one major US hospital network—likely Mass General Brigham—will announce plans for a prospective study of o1 as a "second opinion" tool in the emergency department. This will not be an RCT, but an observational study assessing concordance between the model's diagnoses and the final discharge diagnosis.
In the 90-day outlook, the FDA will release a discussion paper on the status of LLMs as clinical decision support software. The key question: if the model does not outperform physicians in identifying cannot-miss diagnoses, can it be approved for autonomous use? The answer will almost certainly be negative, cooling the market for "AI diagnosticians" but accelerating the development of human-in-the-loop systems.
The main forecast: Broder's study will become a precedent that changes the standard for evaluating medical AI. From now on, it is not enough to show accuracy on a benchmark—blind comparison with doctors on real data and mandatory safety analysis at critical points are required. Startups not ready for this level of validation will lose access to venture capital within 12 months. This is not a crisis, but a market maturation: medicine demands not just accuracy, but demonstrable safety.
— Editorial Team
