Microsoft’s AI Claim, In Context: What The Study Measured

🔍 Read the full analysis: Microsoft’s AI Claim, In Context: What The Study Measured on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s AI diagnostic system, MAI-DxO, scored 83.2% on a set of 304 difficult medical cases, compared with 36.1% for generalist physicians using textbooks and the internet, according to a peer-reviewed Science paper published Oct. 22. The results come from a text-based simulation using historical case records, not from treating patients, so the study does not establish how the system performs in ordinary clinical care.

The paper, “Sequential diagnosis with language models,” evaluated Microsoft’s Sequential Diagnosis Benchmark, which gives a doctor or AI a short case summary and requires it to request further information, such as test results. A separate model holds the case file and supplies requested details. The benchmark scores both the final diagnosis and the estimated cost of tests used to reach it.

Microsoft reported that on the 304 complex New England Journal of Medicine cases, physicians averaged 22.3% accuracy without reference materials and 36.1% with textbooks and internet access. The study also tested 325 emergency-department cases: MAI-DxO scored 80.3%, while physicians scored 46.8%. These figures are results reported by the paper’s Microsoft-affiliated authors, not independent replications.

The system coordinates a language model acting as a panel of virtual physicians. Agents suggest diagnoses, request information, and challenge one another over several rounds. The source account says GPT-5 alone scored 78.9% on the complex cases, compared with 83.2% for MAI-DxO. That makes the underlying model’s performance a major part of the result; the orchestration layer’s reported accuracy gain was about four percentage points. Microsoft also reported lower estimated test costs with the system, though those costs were modelled rather than observed in hospitals.

What the Scores Can Tell Clinicians

The study addresses a meaningful question: whether an AI system can gather diagnostic information step by step, rather than simply choose an answer after receiving a complete case. In that respect, its benchmark attempts to represent part of the diagnostic process more closely than a conventional multiple-choice test. The results suggest that a language-model system can perform strongly on selected, difficult case records under controlled conditions.

But the scores do not show that MAI-DxO is safer or more effective than doctors in clinical practice. The physician comparison group consisted of primary-care doctors and hospital generalists working alone, while the AI system used multiple agents. The cases were challenging teaching examples, not a representative stream of patients, and the test did not include physical examinations or real-time imaging. Hospitals, insurers and patients should treat the findings as benchmark evidence, not proof that the system is ready to diagnose independently or replace clinical teams.

The cost estimates also need cautious interpretation. The source account says test prices were assigned using billing codes and a 2023 U.S. hospital price list, with a fixed physician-visit charge. Those estimates can compare choices within the benchmark, but they do not by themselves establish savings under actual hospital workflows, where staffing, equipment, follow-up and patient outcomes affect costs.

How the Benchmark Was Built

The Science paper is the peer-reviewed publication of work that appeared earlier as a June 2025 arXiv preprint. The preprint used an earlier model, OpenAI’s o3, and reported a larger gap between AI and physicians. The Oct. 22 paper presents updated figures and includes an open-book physician comparison. The source account notes that the full Science text is paywalled and some methodological detail it discusses comes from the preprint, so those details may not match the final paper exactly.

According to that account, the original physician group included 21 U.S. and U.K. generalists: 17 in primary care and four hospital generalists. They were assessed on 56 held-out cases, while the AI was assessed on all 304. The published Science comparison reports physician results of 22.3% without references and 36.1% with them. The doctors were not specialists, and the comparison was between individuals and a coordinated AI system, not between the AI and a medical team.

The benchmark’s scoring and cost methods add further limits. A model judged diagnoses against the published answer, and physician re-checks of a sample showed good but imperfect agreement, according to the source account. If a requested test result was absent from a record, a gatekeeper model generated a synthetic result. Test prices were approximations. The NEJM cases also lacked healthy patients and harmless explanations, meaning the benchmark could not measure false alarms in a typical patient population.

““more than twice as accurate as physicians using textbooks and the internet on complex cases””

— Microsoft, in its blog post, “Building Medical Superintelligence”

Clinical Performance Still Untested

The study does not establish how MAI-DxO performs with real patients, full clinical records or routine hospital workloads. It is also unclear whether its accuracy would hold across common conditions, less difficult cases, different patient groups or settings outside the selected benchmarks. Because the case collection excluded healthy people and benign explanations, the study could not show how often the system might incorrectly flag a serious disease.

There has been no independent reproduction of the reported results in the source material. Peer review means the paper went through journal review; it does not mean outside researchers have replicated the findings. The comparison also leaves open how the system would fare against specialists or physician teams, and whether its model-generated test results and estimated prices reflect real care. The final paper’s full methodological details should be checked directly, since some specifics cited in the source account come from the earlier preprint.

A Planned Study With Patients

The next cited milestone is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source material, consenting patients are expected to interact with MAI-DxO alongside their usual care. That design could provide evidence about how the system works during actual clinical encounters, although it would not by itself establish broad safety or effectiveness.

Key details remain unreported in the source material, including the study’s start date, number of participants, outcome measures and whether results will be compared with clinicians’ decisions or patient outcomes. Until those findings are available, the Science paper supports a narrower conclusion: MAI-DxO performed well on the reported simulated benchmarks, while its value in patient care remains to be tested.

Key Questions

What did Microsoft’s study find?

The Science paper reports that MAI-DxO achieved 83.2% accuracy on 304 difficult NEJM cases. Generalist physicians scored 36.1% with reference materials and 22.3% without them in the reported comparison.

Was MAI-DxO tested on real patients?

No. The reported benchmark used historical, text-based case records in a simulator. The source material describes a prospective study involving consenting patients at Beth Israel Deaconess Medical Center as a planned next step.

Did the AI outperform specialists or medical teams?

The comparison described in the source material involved generalist physicians working alone, not specialists or clinician teams. The study therefore does not establish how MAI-DxO compares with those groups.

How much did the MAI-DxO system add beyond GPT-5?

The source account reports that GPT-5 alone scored 78.9% on the complex cases, compared with 83.2% for MAI-DxO. That is an approximate four-percentage-point difference in the reported results; it does not establish how either system would perform in clinical practice.

Does peer review mean the results have been independently confirmed?

No. Peer review means the paper was reviewed for publication in Science. The source material says the results have not been independently reproduced, so external confirmation remains an open question.

Source: ThorstenMeyerAI.com