Microsoft claims its AI diagnoses complex cases better than generalists

Microsoft is staking a significant claim in the medical AI race, asserting that its new diagnostic system, MAI-DxO, correctly identified complex medical cases in a simulation more than twice as often as generalist physicians. The findings, published in Science on Thursday, position the technology not just as a helpful tool, but as a step toward what CEO Mustafa Suleyman calls “medical superintelligence.” While the headline numbers are striking, the details reveal a nuanced picture of where AI currently stands relative to human expertise.

The performance gap in the study was substantial. On complex cases drawn from the New England Journal of Medicine, the AI system achieved an 83% accuracy rate. In contrast, primary care doctors and general internists, working individually with reference materials, were correct only 36% of the time. However, a crucial caveat remains: these are the kinds of cases typically handled by specialists or teams, who were not included in the direct comparison. This raises questions about whether the AI is truly outperforming the best available human care, or simply outperforming the average generalist on tasks outside their typical scope.

The system works by coordinating multiple AI agents that function like a panel of doctors, debating diagnoses and ordering tests. Microsoft tested this orchestration on models from various providers, including Anthropic, Meta, and DeepSeek, but found the highest performance using OpenAI’s GPT-5 and GPT-5.5. The data suggests that much of the improvement stems from the underlying power of these newer models rather than the orchestration itself. For instance, using GPT-5.5, the system’s accuracy jumped from 72% to nearly 87%, a significant leap that outpaced the gains seen with older models.

Cost and efficiency also played a role in the evaluation. While the multi-agent approach reduced the estimated cost per case by about $3,000 compared to using GPT-5 alone, the accuracy gain was modest. On emergency department cases, the addition of the orchestration layer provided little benefit to either accuracy or cost. This indicates that the technology is most effective on ambiguous, complex cases and with less capable underlying models, such as Gemini 2.5 Flash, which saw its accuracy rise from 46.4% to nearly 69% with the assistance of DxO.

Behind the technology is Mustafa Suleyman, who joined Microsoft in 2024 to lead its AI unit after co-founding DeepMind. He is working with the Mayo Clinic to train a foundation model for healthcare, with a release planned for next year. Suleyman predicts a “coding style breakthrough” for medical records by 2027, allowing for highly accurate predictions of patient outcomes. He believes this infrastructure will enable real breakthroughs, stating, “I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”

To validate these claims in the real world, Microsoft has partnered with Beth Israel Deaconess Medical Center in Boston. The upcoming study will compare DxO’s recommendations against those of human doctors in both emergency and urgent care settings. Vice President of Health Dominic King noted that patients will receive standard treatment while consenting to parallel evaluation by the AI. This move follows a similar recent study by Google, as both companies race to prove that their AI can match or exceed human performance in clinical settings.

Microsoft is framing this effort as a move beyond simple benchmarking, aiming to integrate AI deeply into clinician workflows. Suleyman argues that this focus on medical superintelligence could be a key differentiator for Microsoft against other frontier labs. With generative AI in healthcare still in its early stages, the company expects capabilities to improve by an order of magnitude within the next two to three years, driven by real-world usage and iterative improvements.