OpenAI o1 Outperforms Physicians in Diagnosing Complex Emergency Cases
A study involving doctors from Harvard and Stanford found that the LLM model o1-preview surpassed humans in diagnosing clinical cases, especially under limited information during triage in emergency departments. The model achieved 78.3% accuracy in diagnosing from NEJM materials.
Not a diagnosis, but a demonstration of power: how OpenAI o1 exposes the weak link in triage and raises the question of power redistribution in the emergency department
[The Gist]: What's Really Happening
On April 30, 2026, the journal Science published a study that many rushed to call "AI beat doctors." The headline is flashy but inaccurate. What actually happened is more subtle and more important: OpenAI o1-preview demonstrated that it can surpass board-certified internists precisely at the moment where the human mind is most vulnerable—the first minutes of a patient's arrival in the emergency department, when information is critically scarce and the cost of error is maximal.
Researchers from Harvard, Stanford, and Beth Israel Deaconess Medical Center conducted six parallel experiments. The most telling was not academic vignettes but 76 real cases from the Beth Israel emergency department in Boston. The model received exactly the same data as two attending physicians: text from the electronic medical record, vital signs, a few lines from the triage nurse. No preprocessing, no hints.
During triage—when the patient first crosses the threshold—o1 made an accurate or very close diagnosis in 67.1% of cases. The first physician: 55.3%, the second: 50%. The gap is not just statistically significant. It means that in every sixth case, the model offers the correct answer where both specialists err.
Timeline and Context
The study by Peter G. Brodeur and colleagues, published on April 30, 2026, relies on a methodology that stands out from predecessors. First, for the first time, a direct comparison of LLM and physicians was conducted on unstructured, real clinical data—not on textbook cases, but on the chaos that is a real patient's medical record. Second, reviewers did not know whose answer they were evaluating—human or machine. The blind method succeeded: one physician evaluator guessed the source in only 15% of cases, the other in 3%.
This is important because previous AI studies in medicine were often criticized precisely for lacking a "human baseline" and for evaluating on clean cases far from real clinical practice. Brodeur and colleagues close both methodological gaps.
However, the context extends beyond a single publication. o1-preview, on which most experiments were conducted, is no longer the newest model. By the time of publication, OpenAI had released o3. This means that the results that shocked the medical community in April 2026 reflect not the peak of capabilities but an intermediate point on a trajectory that climbs higher each month.
Who Wins and Who Loses
Winners:
OpenAI and other developers of reasoning models. The study in Science is the gold standard of validation, opening the door to clinical trials and potentially FDA clearance for AI physician assistants. The AI market in emergency medicine is valued at $4.2 billion in 2026, with a projected growth to $12.8 billion by 2032; the Brodeur et al. study is a catalyst that could accelerate this growth.
Patients with rare and complex diseases. The model showed particular strength precisely where human physicians face cognitive biases: anchoring effect, premature closure of diagnostic search, influence of recent experience. For a patient with an atypical presentation of a heart attack or lupus masquerading as pulmonary embolism, an AI assistant is not a luxury but insurance against missed diagnosis.
Medical startups building LLM-based decision support systems. The publication gives them an argument for dialogue with hospital administration: "Want to reduce diagnostic error rate? Here's Science."
Losers:
Emergency department physicians—but not as a profession, rather as bearers of the old decision-making model. The study showed that the largest gap occurs during triage—precisely where the physician must decide quickly and on minimal information. This does not mean AI will replace the physician. But it does mean that a physician working without an AI assistant will soon be viewed like a physician who doesn't wash their hands.
Developers of AI solutions not using reasoning architecture. GPT-4o, tested in parallel with o1, showed significantly weaker results both at triage and at the first physician contact. The gap between model generations (o1-preview surpassed GPT-4 by 16 percentage points on NEJM CPC cases) means that companies invested in the previous generation of LLMs for clinical applications must urgently upgrade their stack.
Skeptics who claimed AI is incapable of clinical reasoning. The model scored 89% on management reasoning tasks versus 34% for physicians with access to search engines. This is not just "guessing the answer" but a demonstration of the ability to weigh conflicting information and make decisions under uncertainty.
What the Media Leaves Out
Non-obvious insight: the study's main sensation is not diagnosis, but management reasoning.
All headlines shout about "diagnosis in the ER" and "67% accuracy at triage." But the most shocking number in the paper is 89% versus 34% on management reasoning tasks. What is that? It's not "make a diagnosis." It's "what to do next": whether to prescribe an antibiotic, how to conduct an end-of-life conversation, what the goal of treatment is given the patient's context.
Thomas Buckley, a doctoral student at HMS involved in the study, explains: management reasoning is harder than diagnostic reasoning because it requires considering not only objective case characteristics but also subjective factors—context, patient preferences, prognostic uncertainty. The fact that a reasoning model surpasses physicians precisely in this—not just in listing symptoms—indicates that AI is beginning to master territory traditionally considered "human": decision-making under incomplete information incorporating value judgments.
Second insight: the model failed at "cannot-miss diagnoses"—and that's good news.
One of the least discussed but critically important results: o1-preview did not statistically significantly outperform physicians in identifying "cannot-miss" diagnoses—those conditions that will kill the patient within hours if missed. In the experiment with NEJM Healer cases, the model scored full marks on 78 out of 80 assessments, but on safety—identifying these cannot-miss threats—it did not differ from physicians.
This is a crucial safeguard against techno-optimism. The model broadens the differential diagnosis, finds rare diseases, but does not surpass humans in screening out the most dangerous and urgent conditions. And that is precisely the main task of a physician in the ER, as Kristen Panthagani, an emergency physician, rightly notes: "My primary goal is not to guess the final diagnosis, but to determine if you have a condition that could kill you."
Forecast: Next 30 Days and 90 Days
30 days (by mid-June 2026):
OpenAI is expected to announce results of testing o3 on the same battery of tests used in the published study. Given that o3 is already available and researchers mention it in limitations, this is a matter of weeks. If o3 shows a significant improvement in cannot-miss diagnoses, it will shift the tone of the discussion from "AI helps but does not replace" to "when will clinical trials start?"
In parallel, several large hospital systems—likely Mass General Brigham and Mount Sinai—will announce the launch of prospective trials of AI assistants in emergency departments. David Reich, chief clinical officer at Mount Sinai, has already called the study "an ideal call to action."
90 days (by mid-August 2026):
The key catalyst will be the publication of parallel studies by the same group on multimodal models. Arjun Manrai reported that the team is already testing AI on images and seeing "rapid progress." If a multimodal model capable of analyzing not only text but also X-rays shows comparable or superior performance on cannot-miss diagnoses, the conversation will shift from "second opinion" to "standard of care."
Simultaneously, professional societies (American College of Emergency Physicians, Society of Hospital Medicine) will begin issuing preliminary guidelines for integrating AI into clinical workflows. And the key question will not be "does AI replace the physician," but "who is liable if a physician overrides the model's recommendation and the patient suffers?" As Adam Rodman noted, "there is currently no formal system of accountability for AI diagnoses." By the end of summer, this legal vacuum will become the main topic at medical conferences.
— Editorial Team
