Large Language Models Outperform Physicians in Emergency Triage Diagnosis
A new study involving Harvard and Stanford found that the preview version of OpenAI's o1 model significantly outperformed physicians in making diagnoses based on limited emergency department data. The largest accuracy gap was observed precisely when the least information was available.
For several days now, I've watched the medical community and IT industry buzz over a study published in Science. Harvard and Stanford have thrown down the gauntlet again: their joint research shows that OpenAI's o1-preview large language model diagnoses more accurately than doctors. Especially during triage in the emergency department, where information is minimal and the cost of error is life.
But news summaries, as usual, skim the surface. They report the fact of "hardware victory" but miss the tectonic shifts this victory triggers in medical law, hospital finances, and physician hierarchy. As someone who consults for health tech startups, I see behind the accuracy graphs not a triumph of technology, but the beginning of the end of "clinical intuition" as a legal argument.
[The Core]: What's Really Happening
Formally, we're talking about numbers: 67.1% accuracy for AI versus 55.3% and 50% for live physicians at the triage stage. But the point isn't the numbers. The point is that AI has beaten humans not on "sterile" textbook knowledge tasks, but on "messy" real-world data—unstructured notes from electronic health records (EHR).
This means the model has learned to extract signal from the noise created by doctors themselves (confusing notes, typos, contradictory data). We used to think AI needed perfectly labeled datasets. It turns out the raw chaos of a real hospital is sufficient nutrient medium. o1-preview works like an ideal resident who reads the entire medical history in a second and doesn't suffer from cognitive biases after a tough night shift.
Timeline and Context
This isn't a bolt from the blue. It's a methodical siege that began in 1959, when computers were first tested on clinical diagnostic tasks.
- 2021–2024: Models like GPT-4 were already breathing down doctors' necks on clinical reasoning tasks (Managing Reasoning), but back then they were compared to reference tools like Google, and AI won by a landslide (89% vs 34%).
- 2025: OpenAI releases the o1 line with "chain of thought." This is a turning point: the model stopped guessing the next token and began mimicking a diagnostician's thought process—doubting, considering hypotheses, double-checking itself.
- April–May 2026: Publication in Science and large-scale tests at Beth Israel Deaconess. The gap became statistically undeniable. And what's most troubling for the profession: reviewers evaluating the diagnosis texts could not tell whether a conclusion was written by a doctor or a machine in 83.6% and 94.4% of cases.
Who Wins and Who Loses
Loses: the old-school clinician. This is not a figure of speech. The study found that the quality gap is especially large in "management reasoning"—choosing treatment tactics, antibiotics, treatment goals. This was once considered the pinnacle of skill, combining experience and subjective judgment. Now it's an algorithm.
Wins: mid-level medical staff. The most undervalued beneficiary. ER nurses often intuitively sense that a patient is "critical" but lack the authority to challenge a doctor. Now they have a tool that formally diagnoses more accurately than a doctor in 67% of cases. This is a weapon in the power struggle within the department. The model reduces anxiety: if staff once feared AI would replace them, after real implementation that fear decreases—because it becomes clear that AI requires human oversight and interpretation.
Loses: the insurance industry in its current form. If AI can predict the need for hospitalization with high accuracy at triage, it undermines the business model of many US insurance companies that profit from denying payment for "unjustified" hospitalizations. Denying coverage when an algorithm superior to a physician recommended hospitalization would be legally suicidal.
What the Media Leaves Out
All glossy articles have sidestepped the problem of "cannot-miss diagnoses." This is a secret known only to careful readers of scientific journals.
In the very same experiments where AI shone in accuracy, it did not show statistically significant superiority over doctors in identifying life-threatening conditions. The model is great at generating a broad differential, it spots rare diseases, but when it comes to the classic "mask" of a heart attack or sepsis, doctors perform just as well. AI doesn't so much prevent catastrophes as it prevents "diagnostic wandering" in complex and rare cases. But for PR purposes, it's more advantageous to show the overall average accuracy percentage rather than admit that in critical safety there's almost no difference. This is a crucial nuance that will surface in lawsuits against hospitals that blindly trust the algorithm.
Forecast: Next 30 Days and 90 Days
Next 30 days. We'll see a wave of statements from C-level executives of major US hospital chains about "partnership," not replacement. No one wants conflict with medical associations. But hospital legal departments have already started rewriting protocols. Implementing AI as a "second opinion" will become mandatory not for convenience, but to reduce malpractice risk. If AI is available and a doctor makes a wrong diagnosis at triage, it will be considered negligence. The argument "I didn't know" disappears.
90 days. Here begins the knockout game. The study was conducted on the preview version of o1-preview, which is already outdated. OpenAI has released o3, which is head and shoulders above. The accuracy gap will grow from the current ~12% to 20–25%. This will trigger the first serious crisis.
Additionally, we'll face the "Leiden effect." In the Netherlands, an attempt to implement an ML tool in real ER work failed not because of accuracy, but due to lack of EHR integration and clear protocols for acting on AI signals. Administrators will realize that buying an OpenAI subscription is only 1% of the problem. The main $50 million (or more) will have to be spent on overhauling hospital IT infrastructure, which is currently a patchwork of 90s software. Without that, any smartest AI will remain just a pretty demo at a TED conference, not a life-saving tool.
— Editorial Team
