OpenAI o1 Outperforms Doctors in Diagnosing Complex Emergency Department Cases
A Harvard and Stanford study published in Science found that OpenAI's latest LLM model, o1, accurately diagnoses 78.3% of complex clinical cases, outperforming physicians. The AI shows the greatest advantage during initial triage with minimal patient information.
As someone who has watched the race of medical algorithms from inside the industry for the past seven years, let me say this: Harvard and Stanford have just published more than just a study. They published a manifesto about the end of the 'golden era' of physician intuition in emergency medicine. The article in Science is an elegantly crafted capitulation of humans to machines in the most chaotic part of the hospital, but beneath this capitulation lies a deal that will reshape the Medicare budget.
The Core: What's Really Happening
The layperson sees the number 78.3% and thinks, 'The robot diagnoses better than the doctor.' That's a superficial view. The real revolution is that OpenAI o1 wins not through knowledge, but through resilience to 'information noise.'
When a patient arrives in the ER with abdominal pain, they don't have a neat textbook discharge summary. There's a jumble of disjointed complaints, panic attacks, and blurry lab results. At that moment, the human doctor activates 'availability heuristics': they see an obese patient with pain—think gallbladder; see a migrant—subconsciously scan for tuberculosis. These are cognitive biases burned into neurons by thousands of hours of practice.
o1-preview, unlike us, is not subject to 'affective heuristics.' The study showed that at the triage point, when information is scarcest, the AI achieved 67.1% accuracy versus 50–55% for live physicians. That's not just a gap of 12–17 percentage points. It's proof that the LLM processes the 'silence' in data better than humans. It doesn't try to fill in the picture with stereotypes when data is lacking. It calculates probabilities where the doctor starts guessing.
Timeline and Context
This breakthrough didn't happen in a vacuum. Back in 2023, everyone laughed at GPT-4 for failing basic clinical scenarios, confusing the sequence of actions in CPR. At the time, it seemed that 'reasoning' was humanity's holy grail.
Key milestones leading to this moment:
- September 2024: OpenAI releases o1-preview. Insiders know it was initially tested on math problems and coding, but Adam Rodman's team at Beth Israel Deaconess immediately saw the potential of Chain-of-Thought for medicine.
- Fall 2024 – Winter 2025: A quiet race between Google DeepMind (with their Med-PaLM) and OpenAI to be the first to publish in a journal like Science. Harvard gains access to raw EHR data without preprocessing—a huge risk.
- April 2026: Publication in Science. The main shock is not that the model won at diagnosis selection, but that in the 'clinical management' task (Grey Matters), o1 scored a median of 89%, while doctors with access to Google and GPT-4 scored a measly 34–41%. This means AI manages hospital resources better than humans.
Who Wins and Who Loses
Winners:
- Insurance companies (UnitedHealth Group, Elevance Health): The main beneficiaries. o1 showed it can work excellently under information scarcity. This means the algorithm can be placed at the system's 'entry point' to cut unnecessary hospitalizations. If AI says a chest pain patient doesn't need troponin and an ECG every 15 minutes, just an antacid and home, savings would be billions of USD annually.
- Triage Nurses: Their role will paradoxically increase. They will become the 'hands of AI' at the bedside, earning above-market salaries for their ability to verify machine input.
- Medical wrapper startups: Dozens of companies will start selling 'o1 for radiology,' 'o1 for pathology,' even if under the hood it's just an API call.
Losers:
- Ultrasound and functional diagnostic doctors in the ER: If o1 diagnoses more accurately from a sparse history than a doctor after an exam, why pay $300 for an extended exam when you can pay $0.03 per token?
- US residency system: Why spend 7 years training an internist if their clinical thinking upon graduation is inferior to a model trained for 3 months on a GPU cluster? Residents in the study showed dismal results on the R-IDEA scale (perfect score in only 16 out of 80 cases, versus 78 for o1). This demoralizes the entire medical education system based on apprenticeship.
What the Media Isn't Saying
Now for the most unpleasant part. Everyone writes about 'blind testing' where doctors couldn't distinguish human text from machine (guessing probability 15% and 3%). But no one says this means the collapse of diagnostic reproducibility.
The second insight concerns the 'forgetting curve.' The study showed the model handles a one-time snapshot of a situation (triage) well, but real emergency medicine is a dynamic process. The model doesn't track the patient in real time; it doesn't see skin color change or shortness of breath worsen. It works with a textual snapshot. That's why in critical 'cannot-miss' diagnoses, o1 did not show significant superiority over humans. The machine is great at playing 'What? Where? When?' but poor at keeping a finger on the pulse of a dying patient.
Forecast: Next 30 and 90 Days
First 30 days (by mid-June 2026):
What I call an 'epidemic of demo versions' will begin. Every self-respecting US medical center will issue a press release about a partnership with OpenAI or Microsoft to pilot an 'AI second opinion.' But real implementation won't happen. Any hospital's legal department will balk: who goes to jail if o1 misses an aortic dissection? The FDA is silent for now. Also expect a wave of criticism from the American Medical Association (AMA), insisting that 'data interpretation' is not the same as 'diagnosis.'
Next 90 days (by September 2026):
The key shift will be multimodal integration. Currently, the model only works with text, but OpenAI has already released o3. I expect leaks or preprints where the model is fed not only the complaint text but also portable ultrasound images and breath recordings. Once o3 proves superiority in multimodal data collection, large networks like Kaiser Permanente will start embedding it not as an 'assistant' but as the primary triage filter in emergency departments. Investments in NLP solutions for the ER will drop to zero because no one will want to compete with the 800-pound gorilla from Microsoft/OpenAI. The market for clinical reasoning scripts and tests for doctors will simply be destroyed.
— Editorial Team
