When the Problem Isn’t the Model
A diagnostic AI model starts misclassifying new cases even though it worked correctly last month. The model has not been retrained. The code has not changed. Something is clearly off.
The instinct in most teams is to look at the model: audit the architecture, maybe even retrain it. The actual problem usually sits upstream, in the data pipeline, where a quiet change went unrecorded. The question that matters is rarely “is the model broken” It is “what changed” Without lineage, that question is answered by guesswork.
The most common defects hide in places that are difficult to reach retroactively: a new scanner deployed at one site, a preprocessing library upgrade that silently shifts output, a labeling vendor that revised its annotation guidelines without notice, a schema migration that dropped a field referenced by a feature transformation, or a new data source that introduced a subtle distribution shift [1,2]. Each of these can degrade performance in ways that look like model drift but are not. Debugging without lineage often produces fixes that mask the real cause, leaving the underlying defect to resurface elsewhere.
What Lineage Actually Tracks
Data lineage is the record of where every data point came from, how it was transformed, and which version was used by which model. It captures sources, processing logic, schema versions, annotation history, and the link between training data and the models that depend on it. Treated seriously, lineage is not documentation. It is the operational foundation of the application. Without it, the connection between a model’s behavior and the data that produced it is lost.
Lineage Compresses Time to Root Cause
With lineage in place, debugging shifts from forensic guesswork to a structured walk back through the pipeline. Given a failing prediction, a team can identify the exact training corpus version, the preprocessing stack that produced it, the source institutions and acquisition parameters, and the annotation provenance.
In computational pathology, where a model can sit between a clinician and a treatment decision, the gap between “the model is misbehaving” and “we know why” must be measured in hours, not weeks. Lineage also clarifies the scope of a problem. When a defect is traced to a specific batch or processing step, every model trained on it and every prediction served downstream can be enumerated immediately. Without lineage, the scope of the failure is unknowable, and the only safe response is broad rollback rather than targeted remediation.
From Debugging to Accountability
Lineage is not only a debugging tool. It is the substrate for accountability. Regulators are converging on the position that AI systems used in clinical decision-making must be auditable [3,4]. Auditability is not satisfied by knowing what the model does. It requires reconstructing what the model was shown, in what form, derived from which sources, under which version of the pipeline. That reconstruction is not possible after the fact unless lineage was captured at the time. A model whose training data cannot be traced cannot be defended in a regulatory review, cannot be safely updated, and cannot be trusted in clinical deployment.
Why It Has to Be Designed In
Data transformations, particularly in pathology and multi-omics workflows, are lossy with respect to provenance. A normalized image does not remember the staining batch it came from. A feature vector does not remember which schema version produced it. If lineage is not captured at the moment data flows through the pipeline, it has to be reconstructed from logs and timestamps, often unreliably. Retrofitting lineage onto a mature system is expensive enough that it is often cheaper to start over.
Effective lineage requires four things, all designed in from the beginning: versioned data stores with immutable references, deterministic and instrumented processing pipelines, structured links between training runs and the data slices they consumed, and governance that treats lineage gaps as defects.
Conclusion
Model failures in medical AI are rarely model problems. They are problems somewhere in the system around the model, most often in the data. The teams that ship reliable diagnostic systems are not the ones with the largest models or the most data. They are the ones who can answer, quickly and precisely, what changed and where.
Data lineage is what makes that answer possible. It is the difference between a system that can be maintained, audited, and trusted over time, and one that quietly accumulates risk until it fails in a way that nobody can explain.
References
-
Sculley D, Holt G, Golovin D, Davydov E, Phillips T, Ebner D, et al. Hidden technical debt in machine learning systems. Adv Neural Inf Process Syst. 2015;28:2503-11.
-
Finlayson SG, Subbaswamy A, Singh K, Bowers J, Kupke A, Zittrain J, et al. The clinician and dataset shift in artificial intelligence. N Engl J Med. 2021;385(3):283-6.
-
U.S. Food and Drug Administration. Artificial Intelligence and Machine Learning (AI/ML)-Enabled Medical Devices [Internet]. Silver Spring (MD): FDA; 2024 [cited 2026 Apr 28]. Available from: https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices
-
European Parliament and Council of the European Union. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Off J Eur Union. 2024;L 2024/1689.
