Agreement Between AI Language Models and BOOM Chondrosarcoma Consensus Statements: A Comparative Study

Abstract

Artificial intelligence (AI) is an exciting development makes life easier and solves many problems in daily life. The Birmingham Orthopaedic Oncology Meeting (BOOM) met in January 2024 with 309 participants from 53 countries to discuss the optimal consensus for chondrosarcomas on 21 questions. The aim of this study was to investigate how reliable the expert statements from the BOOM were compared to ChatGPT-4 and DeepseekR1.

21 questions and consensus statements in the section on chondrosarcomas in the BOOM were extracted. The answers were classified according to the level of evidence and consensus status, taking into account the consensus strength category determined in the meeting. Each statement were written separately for the ChatGPT-4 and DeepseekR1. Consensus questions and answers were written for the AI modules and they were asked to interpret these expressions as ‘’strongly disagree, disagree, undecided, agree or strongly agree’’.

BOOM participants reached a strong consensus on 19 questions. The number of people who accepted the proposition for 1 question was 52% and no consensus was reached. ChatGPT-4 and DeepseekR1 responded ‘’disagree’’ for a same question. The level of evidence for that question was ‘’low to moderate’’ and a strong consensus was reached. A significant relationship was found between the responses of ChatGPT-4 and DeepseekR1.

ChatGPT-4 and Deepseek expressed more positive opinions in the answers with high levels of evidence, while BOOM participants were able to make stronger consensus decisions by combining their clinical observations with literature knowledge, regardless of level of evidence.