Microsoft's MAI-DxO scores 83% vs 36% for physicians on complex NEJM cases in Science study
Microsoft's MAI-DxO, a multi-agent diagnostic orchestrator, was correct on 83% of complex New England Journal of Medicine cases in simulation, versus 36% for generalist physicians, according to a study in Science. The data suggests most of the gain comes from stronger base models, with orchestration adding a smaller lift, mainly on complex cases.
Microsoft's MAI-DxO diagnostic system correctly diagnosed 83% of complex medical cases in a simulation, compared with 36% for generalist physicians, according to a study published Thursday in Science. Microsoft AI CEO Mustafa Suleyman calls the result a step toward "medical superintelligence."
What MAI-DxO is
MAI-DxO (DxO) is an orchestration layer, not a standalone trained model. It coordinates several AI agents that act as a panel of doctors: they question a patient, order tests, and debate a diagnosis. Microsoft tested it on top of models from OpenAI, Google, Anthropic, Meta, and DeepSeek. Best results came with OpenAI's GPT-5 and GPT-5.5.
The numbers
- Complex cases (NEJM-derived, simulated): DxO was correct 83% of the time. Primary care doctors and general internists, working individually with reference materials, were correct 36% of the time.
- GPT-5 alone: 78.9% on complex cases at about $8,400 per case, using a short, basic prompt. DxO cut cost by about $3,000 and raised accuracy by only about four points.
- GPT-5.5 (released April 2026): DxO raised accuracy on complex cases from 72% to nearly 87% at a slightly lower cost.
- Gemini 2.5 Flash: 46.4% alone, nearly 69% with DxO.
- Emergency department cases: Little improvement in accuracy or cost with GPT-5. With GPT-5.5, both configurations scored about 85% at similar cost.
Caveats in the data
The physician comparison has limits. Cases this complex are usually handled by specialists or physician teams, who were not part of the comparison. The setting is a simulation, not live patient care.
The study's own data also suggests that most of the improvement comes from more capable underlying models. The gap between weaker and stronger models was larger than the boost DxO provided. Microsoft's researchers say accuracy gains shrink as models approach the theoretical maximum.
Microsoft joins Google and OpenAI in claiming its AI can match or outperform doctors. Per-token pricing for the system was not disclosed. Costs were reported as estimated cost per case.
What's next
- Real-world evaluation: Through a new partnership with Beth Israel Deaconess Medical Center in Boston, Microsoft will evaluate DxO's clinical reasoning across hospital settings. Patients will receive standard care, and those who consent will also engage with DxO. Microsoft will compare its recommendations against the human doctors'. Dominic King, VP of Health at Microsoft AI, calls it potentially the biggest study of its kind.
- Foundation model: Microsoft is working with Mayo Clinic to train a healthcare foundation model, with release planned for next year. No specs, parameter count, or pricing have been disclosed.
- Competition: Google published a clinical study with Beth Israel in The Lancet this month on its AMIE conversational diagnostic AI in real-world urgent care visits, with physicians monitoring in real time.
Suleyman says he expects healthcare models to get "an order of magnitude better in capability within the next 24 to 36 months." That is a forecast, not a result.
What this means
The headline 83% vs 36% figure is the least informative number in the study. The comparison excludes specialists, and the 78.9% GPT-5 baseline shows a strong base model gets most of the way there without orchestration.
The more useful finding is the cost-accuracy tradeoff. DxO lifts weaker models and trims spend on hard cases, but shows little benefit on emergency department cases, where base models already score around 85%. Orchestration looks like a way to get more from cheaper models, not a durable edge as frontier models improve.
The Beth Israel trial matters more than the simulation. Prospective comparison against clinicians in live settings is the evidence regulators and health systems will weigh. Microsoft's planned Mayo Clinic model suggests it wants a proprietary foundation, rather than depending on OpenAI's models for its best results.
Related Articles
Taste-Bench: Best AI model scores 59.7% on long-term agent decisions, where random guessing yields 25%
Researchers from Microsoft and City University of Hong Kong released Taste-Bench, a 502-question benchmark that tests whether models can pick the better path at decision forks in long agent runs. The top model, GPT-5.6 Sol, scored 59.7% against a 25% guessing baseline. The authors also show a distilled Qwen3.6-27B student can lift SWE-bench Pro success from 14.6% to 33.7%.
Ai2 releases FairStress: 13M-case benchmark finds LLM fairness gaps hidden by final answers
Ai2 has released FairStress, a benchmark of 300 decision scenarios across 29 domains that expands to more than 13 million test cases. It measures whether LLMs change well-supported decisions more readily for one demographic group than another when challenged. Ai2 reports the gap appears as identity becomes more explicit.
IBM's ALTK-Evolve lifts GPT-4.1 to 67.5% Pass^4 on τ²-bench Retail without weight updates
IBM Research says its ALTK-Evolve method took a stock ReAct agent running GPT-4.1 to 67.5% Pass^4 on τ²-bench retail, above every retail entry in its October 12, 2026 leaderboard snapshot. The method appends learned guidelines to the system prompt and updates no model weights.
Microsoft's Probabilistic Bias Correction improves ECMWF forecasts for 98% of precipitation targets, runs on CPUs
Microsoft Research has published Probabilistic Bias Correction (PBC) in Nature and released it as open-source software. The researchers report it improved a leading ECMWF physics-based model for 98% of precipitation targets and 92% of temperature targets at 2–6 week lead times, and that it runs on CPUs.
Comments
Loading...