Author + information
- Received November 4, 2016
- Revision received December 7, 2016
- Accepted December 16, 2016
- Published online April 3, 2017.
- Jacob A. Doll, MDa,b,∗ (, )
- Dadi Dai, PhDc,
- Matthew T. Roe, MD, MHSc,d,
- John C. Messenger, MDe,
- Matthew W. Sherwood, MD, MHSd,
- Abhiram Prasad, MDf,
- Ehtisham Mahmud, MDg,
- John S. Rumsfeld, MD, PhDh,
- Tracy Y. Wang, MD, MHS, MScc,d,
- Eric D. Peterson, MD, MPHc,d and
- Sunil V. Rao, MDc,d
- aVA Puget Sound Health Care System, Seattle, Washington
- bDivision of Cardiology, University of Washington, Seattle, Washington
- cDuke Clinical Research Institute, Durham, North Carolina
- dDivision of Cardiology, Department of Medicine, Duke University School of Medicine, Durham, North Carolina
- eDivision of Cardiology, University of Colorado School of Medicine, Aurora, Colorado
- fDepartment of Cardiovascular Diseases, Mayo Clinic College of Medicine, Rochester, Minnesota
- gDivision of Cardiovascular Medicine, University of California, San Diego Sulpizio Cardiovascular Center, La Jolla, California
- hDenver VA Medical Center, Denver, Colorado
- ↵∗Address for correspondence:
Dr. Jacob A. Doll, VA Puget Sound Health Care System, 1660 S. Columbian Way, S111-CARDIO, Seattle, Washington 98108.
Objectives This study sought to determine variability and stability in risk-standardized mortality rates (RSMR) of percutaneous coronary intervention (PCI) operators meeting minimum case volume standards and identify differences in case mix and practice patterns that may account for RSMR variability.
Background RSMR has been suggested as a metric to evaluate the performance of PCI operators; however, variability of operator-level RSMR and the stability of this metric over time among the same operator are unknown.
Methods The authors calculated mean RSMRs for PCI operators with average annual volume of ≥50 cases in the National Cardiovascular Data Registry CathPCI Registry. Funnel plots were used to account for operator case volume. Demographic, clinical, and treatment variables of patients treated by operators with outlying high or low RSMRs (identified by RMSR greater than or less than 2 σ above or below the mean [analogous to 2 SD], respectively) were compared with nonoutlier operators. RMSR stability was assessed by calculating average annual operator RMSR during the study period and by determining if operators were consistently classified into RMSR categories in each year.
Results Between October 1, 2009, and September 30, 2014, a total of 2,352,174 PCIs were performed at 1,373 hospitals by 3,760 operators. Of these, 242 operators (6.5%) had RSMR >2 σ above the mean and 156 operators (4.1%) had RSMR >2 σ below the mean. Both high and low RSMR outlier operators treated patients with lower expected mortality risk, compared with nonoutlier operators. There was significant instability in annual operator RMSR during the study period.
Conclusions There is significant variability in risk-standardized PCI mortality among U.S. operators meeting minimum volume standards that is not explained by case mix or procedure characteristics. Operator RMSR was unstable from year to year, thus limiting its utility as a sole performance measure for PCI quality.
- coronary artery disease
- percutaneous coronary intervention
- quality measure
- risk standardized mortality rate
Percutaneous coronary intervention (PCI) has a central role in the management of patients with ischemic heart disease and has been the target of national efforts to improve quality of care and patient outcomes. Assessments of PCI quality have traditionally focused on hospital-level performance (1). Guidelines recommend all PCI programs also perform routine internal review of operator results, but do not provide specific performance standards to measure PCI quality at the operator level (2). Measurement of individual operator mortality rates is not widespread, although some states, such as New York and Massachusetts, calculate PCI mortality at the operator level, and these data are publicly reported in New York (3). Mortality rates for individual operators can be standardized for baseline patient risk, but concerns remain about appropriate risk adjustment, the stability of these measures because of low case volumes, and the potential adverse effect of public reporting on patient selection (4,5).
Prior studies of operator performance have reported an inverse relationship between operator volume and mortality (6–8) and the most recent practice guideline from the American Heart Association/American College of Cardiology collaborative recommends operators maintain a minimum annual volume of 50 cases (9). However, to date, studies have not examined variability in PCI mortality among operators meeting these volume targets. It is also unknown if differences in baseline patient risk or practice patterns account for variability in operator-level mortality.
To address these gaps in knowledge, we used the National Cardiovascular Data Registry CathPCI Registry (NCDR CathPCI) to perform an operator-level analysis of in-hospital mortality after PCI. Our objectives were to: 1) determine variability in risk-standardized mortality rates (RSMR) of operators meeting minimum case volume standards; 2) identify differences in case mix or practice patterns that may account for RSMR variability; and 3) determine the annual stability of RSMR classifications.
NCDR CathPCI Registry is a quality improvement registry sponsored by the American College of Cardiology Foundation and the Society for Cardiovascular Angiography and Interventions that collects clinical data and outcomes associated with diagnostic catheterization and PCI at more than 1,700 sites in the United States, capturing >90% of the PCIs performed in the country. Data collection and quality assurance has been previously described (10). In brief, participating institutions submit patient and procedural data using standardized data elements and definitions. A comprehensive data quality program includes training in data abstraction, data quality thresholds, and independent auditing. Data elements and definitions are available at https://www.ncdr.com/WebNCDR/docs/public-data-collection-documents/cathpci_v4_codersdictionary_4-4.pdf?sfvrsn=2.
Our study sample consisted of PCI procedures performed from October 1, 2009, to September 30, 2014. Overall, 3,428,733 PCIs were performed by 10,352 operators at 1,549 sites. Operators were identified using the National Provider Identifier number. Cases with missing or nonvalid National Provider Identifier number (n = 30,261; 0.9%) were excluded. To focus on operators meeting minimum volume standards and performing PCIs throughout the full study period, we also excluded cases performed by operators who had average annual PCI volume <50 (n = 328,387; 9.6%), or who performed <30 PCIs in any 12-month period of the analysis (n = 636,426; 18.6%). Overall, 6,523 operators (63%) were excluded because of low case volume. Excluded operators had higher observed mortality rates and practiced at lower-volume hospitals. A full description of these operators and their patient population is included in Online Table 1. We then excluded PCIs performed on patients transferred out to other acute care hospitals (n = 81,485; 2.4%) or those missing mortality status at discharge (n = 7; <0.001%), leaving our primary analysis dataset of 2,352,174 PCIs performed by 3,760 operators. As a sensitivity analysis we repeated our main analyses after excluding patients with emergency or salvage indication for PCI (n = 423,107; 12.3%), cardiogenic shock within 24 h of presentation (n = 15,660; 0.4%), or cardiac arrest within 24 h (n = 5,863; 0.2%). This second population included a total of 1,907,544 PCIs performed by 3,760 operators (Figure 1).
The expected mortality rate for each case was estimated using the CathPCI mortality risk model that includes demographic and baseline clinical variables (age, body mass index, cerebrovascular disease, peripheral artery disease, chronic lung disease, prior PCI, diabetes), presentation characteristics (ST-segment elevation myocardial infarction, glomerular filtration rate, ejection fraction, cardiogenic shock, heart failure class within 2 weeks, cardiac arrest within 24 h), and PCI lesions characteristics (in-stent thrombosis, highest risk lesion, number of diseased vessels, and chronic total occlusion) (11). In-hospital RSMR was calculated for each operator for the full study period by multiplying the ratio of observed to expected mortality by the population mean mortality (1.5%).
A funnel plot was constructed by plotting RSMR against total number of cases for each operator. A funnel plot is a method of graphically presenting outcomes for quality improvement that accounts for case volume and includes 3 lines: a central line (mean); a 2 σ control limit (analogous to a 95% confidence interval or 2 SD); and a 3 σ control limit (analogous to a 99.8% confidence interval or 3 SD). Funnel plots have previously been used to assess variation in PCI mortality (12,13).
We then compared baseline clinical characteristics, angiographic characteristics, procedural variables, and in-hospital outcomes for patients treated by low RSMR operators (>2 σ below mean), nonoutlier operators, and high RSMR operators (>2 σ above mean). Pearson chi-square test was used to compare categorical variables and Kruskal-Walis or Wilcoxon tests were used to compare continuous variables. Standardized differences were also calculated for baseline variables, expressed as the difference of the 2 means in units of SD. This comparison was repeated for very low RSMR operators (>3 σ below mean), nonoutlier operators, and very high RSMR operators (>3 σ above mean). The missing rate of all variables was low (<1%). Missing categorical variables were imputed to the most common category, and continuous variables to the medians.
We assessed the stability of operator-level RSMR by determining annual RSMR classifications (as calculated by funnel plots for each of the 5 years of the study). The frequency of annual high RSMR outlier classification was determined for operators that were high RSMR outliers or nonoutliers for the full 5-year study period.
In addition, we performed a sensitivity analysis on a lower-risk population by repeating the funnel plot after excluding patients with emergency or salvage PCI, cardiogenic shock, or cardiac arrest.
All statistical analyses were performed by the Duke Clinical Research Institute using SAS version 9.2 (SAS Institute, Cary, North Carolina).
Between October 1, 2009, and September 30, 2014, a total of 2,352,174 PCIs were performed by 3,760 operators meeting minimum volume standards at 1,373 hospitals in the United States. The median operator annual volume was 103 PCI procedures (interquartile range: 75 to 145). Overall in-hospital mortality was 1.5%. Figure 2 provides a funnel plot display of operator-level RSMR mortality rates versus number of cases for each operator. Overall, 242 operators (6.5% of total) had RSMR >2 σ above the mean (analogous to 2 SD) and were classified as high RSMR outliers. The 156 operators (4.1%) with RSMR at least 2 σ below the mean were classified as low RSMR outliers, and the remaining 3,362 operators (89.4%) were nonoutliers. Observed mortality for patients in these groups was 2.2% (high RSMR outliers), 1.5% (nonoutliers), and 0.4% (low RSMR outliers), and RSMRs were 2.7% (high RSMR outliers), 1.5% (nonoutliers), and 0.5% (low RSMR outliers).
We then examined operator, hospital, and patient characteristics stratified by operator RSMR. Compared with nonoutliers, high RSMR operators were more likely to work at rural or urban (as opposed to suburban), nonteaching, and lower PCI volume hospitals, although the individual operators had a higher median annual volume than nonoutliers. Low RSMR operators were more likely to be high-volume operators at high-volume, suburban or urban, nonteaching hospitals (Table 1).
Baseline demographic, clinical, and lesion characteristics of patients treated by low RSMR, nonoutlier, and high RSMR operators showed statistically significant differences, but standardized differences were generally <10% (Table 2). The largest differences were noted in race, insurance status, and clinical presentation. Compared with nonoutliers, high RSMR operators were more likely to care for nonwhite patients without private health insurance. In contrast, low RSMR operators were more likely to care for white, privately insured patients. Both outlier groups performed a greater percentage of elective PCIs and fewer cases for myocardial infarction, compared with nonoutliers. Patients of both outlier groups were at lower estimated risk for mortality, as calculated by the CathPCI mortality risk model (Table 1).
High RSMR operators were less likely to use radial access, fractional flow reserve, or drug-eluting stents compared with nonoutliers (Table 3). In-hospital complications including myocardial infarction, shock, stroke, and dissection/perforation were less common for low RSMR operators and more common for high RSMR operators compared with nonoutliers.
In addition, 72 operators (1.9%) had RSMR of >3 σ above the mean and 15 (0.4%) had RSMR >3 σ below the mean (analogous to 3 SD) (Figure 3). We assessed hospital, operator, and patient characteristics for these 3 groups. Differences described previously for 2 σ outliers versus nonoutliers were similarly present in this assessment of 3 σ outliers (Online Table 2). Notably, outlier operators with 3 σ or higher RSMR treated patients at lower estimated mortality risk compared with nonoutliers (1.0% vs. 1.5%; p < 0.001).
Next, we assessed the stability of provider classifications by examining outlier status for each year of the study period (Figure 4). The outlier operators with RSMR 2 σ or higher for the full study period were, when assessed annually, classified as high RSMR outliers on average 1.53 out of 5 years. Nonoutlier operators were classified as high RSMR outliers on average 0.29 out of 5 years.
Finally, we performed a sensitivity analysis of a lower-risk patient population by excluding all patients presenting with PCI classified as emergency or salvage, cardiogenic shock, or cardiac arrest. Among this population of 1,907,544 PCIs performed by 3,760 operators, 240 operators (6.4%) had RSMR >2 σ above the mean and 13 operators (0.3%) had RSMR >2 σ below the mean. Overall mortality was 0.4% and mortality rates among the operator groups were 0.04% (low RSMR operators), 0.4% (nonoutliers), and 1.1% (high RSRM outliers). Only 108 operators (2.9%) were classified as high RSMR operators in both patient cohorts. An additional 266 (7.1%) operators were identified as high RSMR by 1 but not both populations (Table 4).
This analysis from the largest ongoing PCI registry in the world indicates that there is significant variability in RSMR among U.S. operators, even among a cohort meeting guideline recommended, minimum case volume standards. This variability was also present in a lower-risk patient cohort. High operator RSMR does not seem to be driven by case mix, because these outlier operators cared for patients at lower estimated mortality risk, on average, than nonoutlier operators. Although this suggests that operator RSMR may be useful to identify opportunities for quality improvement, its poor year-to-year stability and concerns about the adverse effects of public reporting limit its use as a sole performance measure.
Variability in operator PCI mortality
Meaningful examination of any performance measure, including PCI mortality, requires risk adjustment and an understanding of common cause variation (“background noise”). This is particularly important when assessing operator-level mortality, as opposed to hospital-level mortality, because smaller sample sizes potentially exaggerate the impact of a small numbers of patient deaths. In this context, the funnel plot is a type of statistical control chart that accounts for case volume (14). The 2 σ and 3 σ lines are arbitrary limits that describe outcomes that are greater than 2 and 3 SD from the mean. Funnel plots have been used for quality improvement purposes in multiple clinical fields (15,16), and Kunandian et al. (12,13) previously assessed PCI operator performance in northeast England and New York State with funnel plots. Our study, using a U.S. national database, shows that a proportion of operators were outside of the 2 σ and 3 σ limits. We primarily focused on operators outside the 2 σ limit because most publicly reported quality measures use 2 SD as a cutoff for outlier performance.
Our study shows that these operator classifications are dependent on the patient population selected for evaluation. Notably, a similar number of operators were identified as high RSMR using both the full population and a low-risk patient cohort, but we found only partial overlap of operator classification when comparing these 2 populations. Our study cannot determine if excluding high-risk patients, as has been done for some hospital-level reporting through the introduction of a “compassionate use” variable (3,17), results in more accurate or fair assessment of quality. We demonstrate that limiting assessment to only low-risk patients would markedly impact classification of providers, but would not eliminate variability.
Operator classifications also varied year-to-year throughout the study. Some nonoutlier operators would be classified as high RSMR for 1 or more years of the study (0.29 out of 5 years on average), whereas overall high RSMR operators would have an outlier RSMR only a minority of years (1.53 out of 5 years on average). This lack of stability may be caused by small annual sample sizes with the consequent outsize impact of a small number of patient deaths.
Differences in case mix
Differences in operator RSMR may be determined by operator characteristics, such as decision-making or procedural skill. Alternatively, variability could be driven by differences in baseline patient characteristics not accounted for by risk standardization. However, in our study both high and low RSMR operators treated a lower-risk patient population relative to nonoutlier operators. High RSMR operators treated fewer myocardial infarction, emergency, and salvage patients. This finding complements a recent analysis showing the CathPCI mortality risk model performs well across the spectrum of risk. Hospitals treating more high-risk patients had better RSMR than low-risk hospitals (18). It is possible that a small number of extreme-risk patients, with expected in-hospital mortality that exceeds that predicted by the CathPCI model, may drive the mortality rates of high RSMR operators. We did not see evidence of this in average predicted mortality rates. Instead, it seems that treatment of a low-risk patient population exaggerates the impact of isolated patient deaths, leading to high RSMR. We identified only modest differences in practice patterns (i.e., use of radial access, drug-eluting stents, and fraction-flow reserve), and these are unlikely to fully account for RSMR variability.
Concerns regarding operator RSMR as a sole performance measure
Operator-level PCI mortality meets many criteria for a performance measure because it is clinically meaningful, demonstrates significant variability, and is reliably measured and adjusted for risk (19). However, our study suggests that use of this outcome as a sole performance measure has significant limitations. First, overall in-hospital mortality is low in this population, and thus small numbers of deaths could determine RSMR classifications. Second, our study shows that operator RMSR is unstable year to year and results in many operators being classified as a high RSMR outlier during some years and not during others. This instability is apparent despite exclusion of low-volume operators. More variability, and potential misclassification, would be expected if lower-volume operators were included. Third, mortality after PCI may be unrelated to the procedure or to operator performance (5,20,21). A prior analysis indicated that 79% of deaths after PCI cannot be attributed to the PCI procedure (22). Reporting of operator-specific mortality may inappropriately penalize a single clinician for systems-level quality gaps. Operator-level RSMR should not be considered in a vacuum, but instead put in the context of hospital RSMR and other measures of operator performance. Finally, reporting of operator RSMR could lead to decreased treatment of high-risk patients, such as those with ST-segment elevation myocardial infarction and/or cardiogenic shock, although these patients have been shown to benefit most from PCI (23). This trend has been seen in states that currently report hospital-level PCI mortality (3,24,25), and may result in elevated mortality after myocardial infarction (26).
Despite these limitations, operator RSMR is already being measured and reported in some settings. At the local level, hospitals currently participating in the CathPCI registry can access a confidential NCDR dashboard that reports operator RSMR (10). Our study suggests that operator-level RMSR should be considered as only 1 potential component of PCI quality assessment that should also include measures of process and structure at the hospital level. Assessment of high RSMR operators could reveal harmful practice patterns or gaps in evidence-based care that may be responsive to quality improvement mechanisms, such as random case review, standardized morbidity and mortality reviews, and selective use of proctored cases. Additionally, review of low RSMR operators may reveal positive practice patterns that can be disseminated to other operators.
Although our study uses a large nationally representative registry, it may not represent current practice in nonparticipating sites or other countries. We intentionally excluded operators with low case volume overall or in any year of the study, which excludes a substantial number of low-volume operators and all operators with fewer than 5 consecutive years in practice. Assessing mortality rates of low-volume or early career operators may require alternative methods. In addition, the CathPCI mortality model differs from the adjustment models currently used for public reporting of myocardial infarction and heart failure mortality, which are based on administrative data. Alternative methods for risk adjustment may not perform as well at the extremes of risk, and could alter the results of this study. However, any future reporting of PCI RSMR will likely be based on CathPCI (17,27). Although prior studies have shown disparities in cardiovascular outcomes associated with race and socioeconomic variables (28), these variables are not included in our model or other adjustment models used for public reporting. High RSMR operators were more likely to treat nonwhite and nonprivately insured patients and work in rural and urban (as opposed to suburban) hospitals. Operators in these settings may have less access to technologies, such as fractional flow reserve and drug-eluting stents. We only examined in-hospital mortality; 30-day mortality or other post-discharge outcomes may be more appropriate to assess quality, but are not captured in the registry.
In this large retrospective study of 3,760 operators performing 2,343,693 PCI procedures, we identified significant variability in operator-level in-hospital mortality after adjustment for demographic and clinical variables. This variability is not driven by differences in case mix or PCI procedure characteristics; in fact high RSMR operators treated patients with lower expected mortality risk. Operator RMSR was unstable from year to year; therefore, concerns remain regarding its use as sole performance measure for PCI quality.
WHAT IS KNOWN? There is variability in PCI mortality at the hospital level, and some states are currently measuring operator PCI mortality.
WHAT IS NEW? There is significant variability in risk-standardized PCI mortality among U.S. operators that is not explained by case mix or procedure characteristics. However, low event rates and year-to-year measurement instability may limit its utility as a sole performance measure.
WHAT IS NEXT? Alternative measures of PCI quality are needed, although operator PCI mortality could be 1 component of quality assessment if considered in the context of other outcomes and process measures.
For supplemental tables, please see the online version of this article.
This research was supported by the American College of Cardiology Foundation’s National Cardiovascular Data Registry. The views expressed in this manuscript represent those of the authors and do not necessarily represent the official views of the National Cardiovascular Data Registry (NCDR) or its associated professional societies identified at www.ncdr.com. CathPCI Registry is an initiative of the American College of Cardiology with partnering support from The Society for Cardiovascular Angiography and Interventions. Dr. Doll has received a research grant from Gilead. Dr. Roe has received research grants from Eli Lilly, AstraZeneca, Daiichi-Sankyo, Ferring Pharmaceuticals, Janssen Pharmaceuticals, KAI Pharmaceuticals, the Familial Hypercholesterolemia Foundation, and Sanofi; and consulting payments or honoraria from Actelion, Myokardia, Novartis, Daiichi-Sankyo, Quest Diagnostics, AstraZeneca, Eli Lilly, Merck, Janssen, Elsevier Publishers, Amgen, Bristol-Myers Squibb, Boehringer Ingelheim, Pri-Med, and Regeneron. Dr. Sherwood has received research support from AstraZeneca and Gilead; and consulting fees from Boehringer Ingelheim. Dr. Mahmud serves on the advisory board and speakers bureau of Medtronic; consulting and educational programs of Abbott Vascular; and Clinical Events Committee of St. Jude. Dr. Rumsfeld was the Chief Science Officer of the NCDR at the time of this analysis. Dr. Wang has received research grant support from Pfizer, Eli Lilly, Daiichi-Sankyo, AstraZeneca, Bristol-Myers Squibb, Boston Scientific, Gilead, GlaxoSmithKline, and Regeneron; and consulting services from AstraZeneca, Eli Lilly, Merck, Pfizer, and Premier. Dr. Peterson has received institutional grant support from American College of Cardiology, American Heart Association, Eli Lilly, and Janssen; and consulting fees (including CME) from Merck & Co., Boehringer Ingelheim, Genentech, Janssen, AstraZeneca, Bayer AG, and Sanofi. Dr. Rao has received research funding from Bellerophon; and consulting fees from Medtronic, AstraZeneca, and Boehringer Ingelheim. All other authors have reported that they have no relationships relevant to the contents of this paper to disclose.
- Abbreviations and Acronyms
- National Cardiovascular Data Registry
- percutaneous coronary intervention
- risk-standardized mortality rate
- Received November 4, 2016.
- Revision received December 7, 2016.
- Accepted December 16, 2016.
- Frey P.,
- Connors A.,
- Resnic F.S.
- Levine G.N.,
- Bates E.R.,
- Blankenship J.C.,
- et al.
- Peberdy M.A.,
- Donnino M.W.,
- Callaway C.W.,
- et al.
- Badheka A.O.,
- Patel N.J.,
- Grover P.,
- et al.
- Hannan E.L.,
- Wu C.,
- Walford G.,
- et al.
- Strom J.B.,
- Wimmer N.J.,
- Wasfy J.H.,
- Kennedy K.,
- Yeh R.W.
- Harold J.G.,
- Bass T.A.,
- Bashore T.M.,
- et al.
- Moussa I.,
- Hermann A.,
- Messenger J.C.,
- et al.
- Brennan J.M.,
- Curtis J.P.,
- Dai D.,
- et al.
- Kunadian B.,
- Dunning J.,
- Roberts A.P.,
- et al.
- Aelvoet W.,
- Terryn N.,
- Blommaert A.,
- et al.
- Solomon P.J.,
- Kasza J.,
- Moran J.L.
- Resnic F.S.,
- Normand S.L.,
- Piemonte T.C.,
- et al.
- Sherwood M.W.,
- Brennan J.M.,
- Ho K.K.,
- et al.
- ↵National Quality Forum. Measure evaluation criteria. Available at: http://www.qualityforum.org/Measuring_Performance/Submitting_Standards/Measure_Evaluation_Criteria.aspx. Accessed January 16, 2017.
- Aggarwal B.,
- Ellis S.G.,
- Lincoff A.M.,
- et al.
- Spoon D.B.,
- Psaltis P.J.,
- Singh M.,
- et al.
- Resnic F.S.,
- Welt F.G.
- Apolito R.A.,
- Greenberg M.A.,
- Menegus M.A.,
- et al.
- Moscucci M.,
- Eagle K.A.,
- Share D.,
- et al.
- Waldo S.W.,
- McCabe J.M.,
- O'Brien C.,
- Kennedy K.F.,
- Joynt K.E.,
- Yeh R.W.
- Dehmer G.J.,
- Drozda J.P. Jr..,
- Brindis R.G.,
- et al.
- Bucholz E.M.,
- Ma S.,
- Normand S.L.,
- Krumholz H.M.