MSU Researchers Propose Method for Evaluating Medical AI Responses

Researchers at the Artificial Intelligence Center at Lomonosov Moscow State University (MSU) have contributed to the development of a method for evaluating AI-generated diagnostic responses that accounts for differences in physicians’ judgments. The method compares diagnostic conclusions produced by large language models (LLMs) with those made by physicians, using two relative metrics and an algorithm that identifies matching wording.

Tests involving 360 dialogues, seven physicians, and eight language models showed that the models’ level of agreement with experts was comparable to the level of agreement among physicians themselves. The researchers caution, however, that agreement does not mean a diagnosis is necessarily correct.

The proposed method could be used to develop and compare clinical decision support systems that analyze text-based dialogues and suggest possible diagnoses.