A recent study published in the journal Science has raised concerns about the generalizability of clinical prediction models, particularly in the context of artificial intelligence (AI) in healthcare.

The study, conducted by Adam M. Chekroud, Matt Hawrilenko, Hieronimus Loho, Julia Bondar, and others, examined the performance of machine learning models in predicting patient outcomes in independent clinical trials of antipsychotic medication for schizophrenia.

One of the key findings of the study was that machine learning models demonstrated high accuracy in predicting patient outcomes within the specific trial in which the model was developed. However, when these same models were applied to independent clinical trials, their predictive performance dropped to chance levels, indicating a lack of generalizability.

Furthermore, the study revealed that even when data from multiple similar multisite trials were aggregated to build more robust models, the predictive performance remained poor when applied to new patient samples, highlighting the context-dependency of these models.

This research challenges the widespread optimism regarding the potential of statistical models to improve decision-making in medical treatments. The hope, often based on successful observations of model performance in limited datasets or clinical contexts, may not accurately reflect the real-world generalizability of these models.

The implications of these findings are particularly significant in the realm of precision medicine and AI-driven healthcare, where the ability to accurately predict and identify the best course of care for patients is a central promise. The study’s results suggest that caution is warranted in relying solely on machine learning models for clinical decision-making, especially without rigorous prospective testing on independent patient samples.

The study’s authors emphasize the need for further research and scrutiny in evaluating the generalizability of clinical prediction models, particularly in the context of AI applications in healthcare.

For more details, the full research article can be accessed in the journal Science, Vol. 383, Issue 6679, pp. 164-167, DOI: 10.1126/science.adg85382,7061.