Date of Award
Summer 8-15-2022
Degree Name
Doctor of Philosophy (PhD)
Degree Type
Dissertation
Abstract
Clinical Prediction Models (CPM) have long been used for Clinical Decision Support (CDS) initially based on simple clinical scoring systems, and increasingly based on complex machine learning models relying on large-scale Electronic Health Record (EHR) data. External implementation – or the application of CPMs on sites where it was not originally developed – is valuable as it reduces the need for redundant de novo CPM development, enables CPM usage by low resource organizations, facilitates external validation studies, and encourages collaborative development of CPMs. Further, adoption of externally developed CPMs has been facilitated by ongoing interoperability efforts in standards, policy, and tools. However, naïve implementations of external CPMs are prone to failure due to the incompatibilities between the environments of the development and implementation sites. Although prior research has described methods for estimating the external validity of predictive models, quantifying dataset shift, updating models, as well as numerous CPM-specific frameworks for guiding the development, evaluation, reporting, and systematic reviews of CPMs, there are no frameworks for assessing the compatibility between a CPM and the target environment. This dissertation addresses this critical gap by proposing a novel CPM transportability checklist for guiding the adoption of externally developed CPMs.To guide the development of the checklist, four extant CPM-relevant frameworks (TRIPOD, CHARMS, PROBAST, and GRASP) were reviewed and synthesized, thereby identifying the key domains of CPMs. Then, four individual studies were conducted, each identifying, assessing the impact of, and/or proposing solutions for the disparity between CPM and environment in those domains. The first two studies target disparities in features, with the first characterizing the non-generalizability impact of a particular class of commonly used, EHR-idiosyncratic features. The second study was conducted to identify and propose a solution for the semantic discrepancy in features across sites caused by the insufficient coverage of EHR data by standards. The third study focused on the prediction target of CPMs, identifying significant heterogeneity in disease understanding, phenotyping algorithms, and cohort characteristics of the same clinical condition. In the fourth study investigating CPM evaluation, the gap between typical CPM evaluation design and expected implemented behavior was identified, and a novel evaluative framework was proposed to bridge that gap. Finally, the APT checklist was developed using the synthesis of the aforementioned CPM frameworks as the foundation, enriched through the incorporation of innovations and findings from these four conducted studies. While rigorous meta-evaluation remains, the APT checklist shows promise as a tool for assessing CPM transportability thereby reducing the risk of failure of externally implemented CPMs. The key contributions to informatics include: the discovery of healthcare process (HCP) variables as a driver of CPM non-transportability, the fragility of clinical phenotyping used to identify CPM targets, a novel classification system and meta-heuristics for an aspect of EHR data previously lacking in standards, a novel CPM evaluation design termed the pseudo-prospective trial, and the APT checklist. Overall, this work contributes to the body of biomedical informatics literature guiding the success of informatics interventions.
Language
English (en)
Chair
Albert M. Lai
Committee Members
Philip R. Payne, Randi E. Foraker, Dennis L. Barbour, Thomas Kannampallil,
Included in
Artificial Intelligence and Robotics Commons, Bioinformatics Commons, Biomedical Engineering and Bioengineering Commons