When debating the application boundaries of artificial intelligence (AI) predictive models in clinical medicine, it is clear that high predictive accuracy is desirable, but on its own, does not provide a sufficient condition for clinical application. Drawing on three example, AlphaFold’s prediction of protein structure, radiomics’ prediction of disease diagnosis and prognosis, and clinical risk scoring models’ prediction of morbidity, and engaging with David Hume’s empiricist skepticism towards causality, argue that interpretability is an indispensable condition in a discipline that values mechanistic explanation. In order for AI to evolve from a capable recommender to a decision-making machine that begins to develop a sense of individual self, several preconditions need to be fulfilled. Predictions must be falsifiable, minimally grounded in mechanistic knowledge, accompanied by partially explicable decision logics, designed to be fair to populations, and embedded in an error-tolerant architecture that enables correction and rollback. The utility of AI today lies in its ability to dramatically reduce the costs of human trial and error, but should not diminish the doctor’s right to make, learn from, and reflect on mistakes as the final accountable link in the chain.