Microsoft’s AI diagnostic system reached the correct diagnosis in 83.2% of complex medical teaching cases, against 36.1% for generalist physicians using reference tools, according to a paper published in Science on Thursday. The results come from past cases tested under benchmark conditions. Microsoft has not announced a product, regulatory approval or release date.
The paper, “Sequential diagnosis with language models”, was written by Microsoft researchers. They include Mustafa Suleyman, chief executive of Microsoft AI, and Dominic King, its health lead. The paper calls the system Diagnostic Orchestrator, or DxO. Microsoft’s announcement calls it MAI-DxO and says the next step is a prospective study in which patients will use the system directly.
Microsoft’s system scored 83% on NEJM cases and 80% on emergency cases
Microsoft built the test set itself. Its Sequential Diagnosis Benchmark (SDBench) turns 304 clinicopathological conference cases from the New England Journal of Medicine and 325 emergency-department admissions from the MIMIC-IV-ED database into step-by-step exercises. Each case starts with a short description. The diagnostician, either the AI or a physician, then asks questions, orders tests and commits to a diagnosis.
Using OpenAI’s GPT-5, the system scored 83.2% on the NEJM cases and 80.3% on the emergency cases. Generalist physicians averaged 22.3% on the NEJM cases without reference resources, 36.1% with them, and 46.8% on the emergency cases. Microsoft describes this as “about 3.7 and 2.3 times more accurate” on the NEJM cases and “more than 1.7 times higher” on the emergency cases. The Verge reports that the emergency cases were drawn from admissions at Beth Israel Deaconess Medical Center in Boston.
The physician study used 56 NEJM cases held back from development. It produced 764 diagnoses without reference resources and 662 with them. The paper says even the most accurate physician, at 51.8%, was “outperformed in both accuracy and cost”. On the emergency cases, the best physician scored 62.5%.
Microsoft says the system reached its diagnoses with fewer, cheaper tests
SDBench also counts the estimated cost of the tests ordered, using US hospital list prices from Kaiser Permanente Washington. Without a budget limit, the system averaged $5,431 per NEJM case. With budget limits, it scored 80.9% at $2,241 and 79.6% at $1,303. Physicians averaged $3,049 without references and $4,725 with them. Microsoft’s blog says one configuration “achieved 81% accuracy at less than half the cost of comparative systems”.
The system works like a simulated panel of doctors. Microsoft says each one proposes diagnoses, chooses questions or tests and challenges “the emerging view”. According to the paper, it asked a mean of 25.5 questions before ordering its first test. Physicians asked 5.6.
The paper says the results do not show superiority over real clinical practice
The authors set out the limits themselves. “These comparisons do not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice,” they write. The cases were text-only, with no imaging, waveforms or pathology slides. They exclude healthy or benign encounters. A software “Gatekeeper” supplied the case information, rather than real patient records or conversations with patients.
“SDBench measures diagnostic reasoning under idealized information conditions rather than bedside performance or patient outcomes,” the paper says. Its scoring checks whether the diagnosis is correct. It does not assess “therapeutic appropriateness, safety, or longitudinal management quality”. All authors are Microsoft employees who hold equity in the company, and the study had no outside funding.
The Verge reports that the published results do not include the newest frontier models. The paper says Microsoft repeated the full evaluation with GPT-5.5 before publication, without changing the method, and the system’s advantage over the basic prompt grew.
“It needs clinical validation,” Suleyman told The Verge. “That is a leap, but this is a major milestone along that journey.”
A prospective study with Beth Israel Deaconess is next, Microsoft says
Microsoft says it has a new partnership with Beth Israel Deaconess. “In a prospective study, patients will interact with the system directly, with performance measured against physicians’ independent assessments,” the blog says. King told The Verge that testing will likely “get underway in the next couple of months”, but neither he nor Suleyman would commit to a firm timeline. The Verge reports that King stressed the regulatory and compliance requirements in medicine. Microsoft has not said when or how the technology might become available.
The company is also bringing health teams from across Microsoft into Microsoft AI, including Microsoft Research scientists and the teams behind Dragon Copilot and PowerScribe. A companion paper on what Microsoft calls “medical superintelligence” appeared in BMJ Digital Health and AI. TSN has reported on Microsoft’s push to build its own AI stack.
Microsoft’s open benchmark and planned patient study are the tests that matter next
Several parts of the release let others check Microsoft’s work. The SDBench code, including the Gatekeeper, Judge and cost estimator, is published on GitHub under the MIT licence. Microsoft says the aim is for researchers to “reproduce the evaluation, test other models, and challenge or extend the findings”. The NEJM-derived dataset is available for non-commercial research under licence from NEJM Group. The paper says the emergency-case subset will be released on PhysioNet, subject to its credentialing rules.
The paper also points to specific benefits. Its error analysis found that when the system was wrong, it usually picked a clinically adjacent diagnosis at the wrong level of detail rather than an unrelated disease. Its advantage was concentrated in cases that depend on careful history-taking, such as rare exposures and travel-related infections. The authors suggest it could support physicians and be used to train them, through interactive diagnostic cases.
The paper does not measure patient outcomes, and the authors say so. Every figure comes from Microsoft’s own benchmark, run by Microsoft staff, against individual generalists rather than specialist teams. The planned Beth Israel Deaconess study is the first step towards testing the system with real patients. Google took a similar step when it tested its AMIE chatbot with real patients under physician supervision at the same hospital, in a feasibility study published in The Lancet. TSN has also reported OpenAI’s figures on health questions asked of ChatGPT, a sign of how many health questions already go to general-purpose chatbots.
King told The Verge the hospital testing should begin within “the next couple of months”; Microsoft has given no date for anything after that.
Sources
- Nori et al., “Sequential diagnosis with language models”, Science, published in print 22 October 2026, DOI 10.1126/science.aeh7930: https://www.science.org/doi/10.1126/science.aeh7930
- Microsoft AI, “Building Medical Superintelligence”, 22 October 2026: https://microsoft.ai/news/building-medical-superintelligence/
- Microsoft, SDBench code repository (MIT licence): https://github.com/microsoft/sdbench
- The Verge, “Microsoft says its AI tool is better than doctors at diagnosing conditions”, 22 October 2026, 19:00 BST: https://www.theverge.com/ai-artificial-intelligence/1016690/suleyman-microsoft-diagnostic-orchestrator-science-medical-superintelligence
Related stories
- The 13 Billion Bet: Why Microsoft Is Building Its Own AI Stack
- 600,000 People Ask ChatGPT About Their Health Every Week — And 70% Do It After Hours
- AI in Medicine: Google’s AMIE in The Lancet, Chai and GSK, and an FDA-Cleared Autism Assessment Aid
Follow @tsnmediahub on X for breaking AI news as it happens.
Follow on X
