Artificial intelligence, i.e. machine learning (ML) systems and their use in medical applications, are topics that currently receive a considerable amount of attention due to so-called Large Language Models (LLMs), which were a recent breakthrough in natural language processing capabilities. However, there have been earlier generations of automated clinical systems which until now have not had a lasting impact on the way medicine is practiced around the globe [1]. Looking at that history, and at what makes LLMs different, shows why interpretability is not a nice-to-have. It’s the condition for these systems to actually earn a place in clinical practice.
From expert systems to black boxes
One of the earliest “expert systems” was MYCIN [2], essentially an automated decision tree built for selecting antibiotics. It was capable for its time but already had some of the acceptance issues that will be discussed in the context of current efforts based on LLMs. Later systems, powered by neural networks and GPU advances, focused mainly on very domain-specific pattern recognition tasks in imaging-based fields such as radiology and pathology. LLMs changed the equation again: for the first time, machines could process the natural language written in medical records and reports created by and for interpretation by humans.
The gain in language capability comes at a cost. LLMs do not lend themselves well to interpretability, largely because the natural language text used for training the system is largely abstracted away. The input is fragmented into so-called “tokens,” small parts of a sentence, single words, or interpunctuation. The resulting model is essentially an opaque collection of computational data that describes the statistical probabilities of token relationships. The probabilistic path that data takes from input to resulting output during inference has no similarity to human language processing or reasoning, and intermediate representations of input data are usually compressed and non-accessible [3]. In short: LLMs can produce a diagnosis or a therapy recommendation, but they can’t easily show how they got there. That’s the black box problem.
What’s at stake without interpretability
LLMs promise gains of increased efficiency and improved performance in clinical applications. But that’s exactly what makes the black box problem worth taking seriously: a model that receives input data at one end and provides a diagnosis or therapy “recipe” at the other can be considered problematic and calls for improvements in interpretability.
The most important reason for breaking up the black box and providing insight into models is patient safety. A treating clinician has to be able to understand why a model arrives at a given diagnosis or therapy recommendation, in order to identify errors and intervene before they cause harm to a patient.
Interpretability also lets clinicians review what input data the model actually used in processing. Missing or ignored context, such as comorbidities, allergies, or medications that have to be considered, can be spotted and corrected only if there is feedback about its use in the system.
Accountability for medical decisions is at present still the responsibility of the treating clinician, and decision-making cannot be delegated ethically to a system that acts quasi-autonomously and without supervision. That supervision is only possible if the process that leads to the decision is comprehensible by human oversight. Adequate documentation of medical findings, decision process, and therapy is a requirement that is, for example, part of the medical profession’s regulations in Germany [4].
Medical decisions are required to be based on up-to-date knowledge and science, which is usually synthesized into guidelines that reflect best practices and summarize available evidence. In many jurisdictions, clinicians are expected to practice in accordance with said guidelines. Leaving big parts of medical decision-making to opaque systems is therefore also a legal risk in itself. If the system’s scientific basis is outdated or doesn’t meet accepted standards, that opens the door to malpractice claims, criminal penalties, or costly civil cases.
As medicine is a field that observes and interacts with biological systems, absolute certainty about observations and interventions’ outcomes can rarely be attained. To support decision-making, it is of interest to the clinician to receive a measure of uncertainty for a model’s output, to evaluate the robustness of a given diagnosis or therapy recommendation.
There’s a second, harder problem underneath: bias. Models are trained on real-world data, and biomedical research has long skewed toward study populations in western, industrialized countries, populations that don’t represent the genetic, phenotypic, behavioral, and cultural range of humanity [5]. Unfortunately, this bias in biomedical research has mostly continued to be carried over into training datasets for ML applications, and necessitates a shift in paradigms, as otherwise AI models might fail to realize their potential and will even endanger patients due to wrong baseline assumptions for the intended use [6]. Information about, e.g., training cohort demographics and performance metrics can help clinicians evaluate if the model will perform well for their patients.
Trust, consent, and the clinician’s own skillset
Interpretability isn’t just a technical or legal concern. It shapes trust between clinician and patient. Shared decision-making and informed consent both depend on a clinician being able to explain how they arrived at a recommendation. If clinicians cannot explain why they arrived at a given therapy recommendation, patients might be more reluctant to consent because trust is not sufficiently established, and in turn adherence might suffer or necessary treatment might be delayed or declined altogether.
Preservation of the clinician’s skillset and judgement is important but might be impaired if overreliance on systems with little insight takes hold. If the system only provides output without taking the clinician along on the path to said output by giving them windows of interpretability, there will be a danger of blindly applying its results without critically engaging with presented findings and reasoning about conclusions. In the longer term, this might lead to a loss of routine and make it harder to access existing knowledge on the spot. This is of special importance if AI systems are unavailable, as for example in emergency situations, when decisions need to be made fast and with confidence in one’s own abilities.
Interpretable AI systems are also paramount to facilitate training and learning. A “black box” that presents immediate solutions without detailing options and their advantages and disadvantages makes it harder to acquire knowledge in the first place, and there will be little motivation for the next generation of clinicians to develop a robust clinical “reasoning toolbox” and form mental decision trees based on a solid foundation of established facts.
In summary, clinicians and their patients will only be able to utilize, reason about, and trust new LLM-driven approaches if there is a way to “crack” the black box and give them a chance to engage with and observe the models, instead of blindly delegating diagnosis and therapy to the hidden algorithms within.
References
- Hirani R, Noruzi K, Khuram H, Hussaini AS, Aifuwa EI, Ely KE, Lewis JM, Gabr AE, Smiley A, Tiwari RK, Etienne M. Artificial Intelligence and Healthcare: A Journey through History, Present Innovations, and Future Possibilities. Life (Basel). 2024 Apr 26;14(5):557. doi: 10.3390/life14050557. PMID: 38792579; PMCID: PMC11122160.
- Yu VL, Fagan LM, Wraith SM, et al. Antimicrobial Selection by a Computer: A Blinded Evaluation by Infectious Diseases Experts. JAMA. 1979;242(12):1279–1282. doi:10.1001/jama.1979.03300120033020
- Baumann, J. PAICON. Internal communication
- Bundesärztekammer. Musterberufsordnung für die in Deutschland tätigen Ärztinnen und Ärzte. Dtsch Arztebl International. 2021;118(23):8816
- Norori N, Hu Q, Aellen FM, Faraci FD, Tzovara A. Addressing bias in big data and AI for health care: A call for open science. Patterns (N Y). 2021 Oct 8;2(10):100347. doi: 10.1016/j.patter.2021.100347. PMID: 34693373; PMCID: PMC8515002.
- Aichmüller-Ratnaparkhe, M. Remaining84: The missing foundation of personalized medicine. 1st ed. PAICON, 2026.
