Searchable Title

GRACE Checklist: A Validated Assessment Tool for High Quality Observational Studies of Comparative Effectiveness. Copyright: Academy of Managed Care Pharmacy.

Searchable Authors

N A. Dreyer
A Bryant
P Velentgas

Reference Type

Journal Article

Authors, Section

Dreyer, N. A.; Bryant, A.; Velentgas, P.

Title, Section

GRACE Checklist: A Validated Assessment Tool for High Quality Observational Studies of Comparative Effectiveness. Copyright: Academy of Managed Care Pharmacy.

Publication Year

2016

Journal Title

Journal of Managed Care & Specialty Pharmacy

Volume

22

Issue

10

Pages

1107-1113

Availability

online

PMID

PMID: 27668559

DOI

10.18553/jmcp.2016.22.10.1107

Abstract

DISCLOSURES: The original validation Study was published in the March 2014 issue of the Journal of Managed Care & Specialty Pharmacy. The Checklist Questions and scoring were included using a table that was originally published by this journal in 2014. Study Concept and design were primarily contributed by Dreyer and Velentgas, along with Bryant. Bryant took the lead in Data collection and analysis, along with Dreyer and Velentgas, and Data Interpretation was performed by Dreyer, Velentgas, and Bryant. The manuscript was written and Revised primarily by Dreyer, along with Bryant and Velentgas. BACKGROUND: Recognizing the growing need for robust evidence about treatment effectiveness in real-world populations, the Good Research for Comparative Effectiveness (GRACE) guidelines have been developed for noninterventional studies of comparative effectiveness to determine which studies are sufficiently rigorous to be reliable enough for use in health technology assessments. OBJECTIVE: To evaluate which aspects of the GRACE Checklist contribute most strongly to recognition of quality. METHODS: We assembled 28 observational comparative effectiveness articles published from 2001 to 2010 that compared treatment effectiveness and/or safety of drugs, medical devices, and medical procedures. Twenty-two volunteers from academia, pharmaceutical companies, and government agencies applied the GRACE Checklist to those articles, providing 56 assessments. Ten senior academic and industry experts provided assessments of overall article quality for the purpose of decision support. We also rated each article based on the number of annual citations and impact factor of the journal in which the article was published. To identify checklist items that were most predictive of quality, classification and regression tree (CART) analysis, a binary, recursive, partitioning methodology, was used to create 3 decision trees, which compared the 56 article assessments with 3 external quality outcomes: (1) expert assessment of overall quality, (2) citation frequency, and (3) impact factor. A fourth tree looked at the composite outcome of all 3 quality indicators. RESULTS: The best predictors of quality included the following: use of concurrent comparators, limiting the study to new initiators of the study drug, equivalent measurement of outcomes in study groups, collecting data on most if not all known confounders or effect modifiers, accounting for immortal time bias in the analysis, and use of sensitivity analyses to test how much effect estimates depended on various assumptions. Only sensitivity analyses appeared consistently as a predictor of quality in all 4 trees. When a composite outcome of the 3 quality measures was used, the GRACE Checklist showed high sensitivity and specificity (71.43% and 80.95%, respectively). CONCLUSIONS: The GRACE Checklist stands out from other consensus-driven and expert guidance documents because of its extensive validation efforts. This most recent work shows that the checklist has strong sensitivity and specificity, increasing its utility as a screening tool to identify high-quality observational comparative effectiveness research worthy of in-depth review and applicability for decision support.

Share

COinS