Author + information
- Received April 12, 2011
- Revision received May 9, 2011
- Accepted May 14, 2011
- Published online September 1, 2011.
- Lisa Bergersen, MD, MPH⁎,⁎ (, )
- Kimberlee Gauvreau, ScD⁎,
- Susan R. Foerster, MD†,
- Audrey C. Marshall, MD⁎,
- Doff B. McElhinney, MD⁎,
- Robert H. Beekman III, MD‡,
- Russel Hirsch, MD‡,
- Jacqueline Kreutzer, MD§,
- David Balzer, MD†,
- Julie Vincent, MD∥,
- William E. Hellenbrand, MD∥,
- Ralf Holzer, MD¶,
- John P. Cheatham, MD¶,
- John W. Moore, MD#,
- Grant Burch, MD⁎⁎,
- Laurie Armsby, MD⁎⁎,
- James E. Lock, MD⁎ and
- Kathy J. Jenkins, MD, MPH⁎
- ↵⁎Reprint requests and correspondence:
Dr. Lisa Bergersen, Department of Cardiology, The Children's Hospital, Longwood Avenue, Boston, Massachusetts
Objectives This study sought to develop a method to adjust for case mix complexity in catheterization for congenital heart disease to allow equitable comparisons of adverse event (AE) rates.
Background The C3PO (Congenital Cardiac Catheterization Project on Outcomes) has been prospectively collecting data using a Web-based data entry tool on all catheterization cases at 8 pediatric institutions since 2007.
Methods A multivariable logistic regression model with high-severity AE outcome was built using a random sample of 75% of cases in the multicenter cohort; the models were assessed in the remaining 25%. Model discrimination was assessed by the C-statistic and calibration with Hosmer-Lemeshow test. The final models were used to calculate standardized AE ratios.
Results Between August 2007 and December 2009, 9,362 cases were recorded at 8 pediatric institutions of which high-severity events occurred in 454 cases (5%). Assessment of empirical data yielded 4 independent indicators of hemodynamic vulnerability. Final multivariable models included procedure type risk category (odds ratios [OR] for category: 2 = 2.4, 3 = 4.9, 4 = 7.6, all p < 0.001), number of hemodynamic indicators (OR for 1 indicator = 1.5, ≥2 = 1.8, p = 0.005 and p < 0.001), and age <1 year (OR: 1.3, p = 0.04), C-statistic 0.737, and Hosmer-Lemeshow test p = 0.74. Models performed well in the validation dataset, C-statistic 0.734. Institutional event rates ranged from 1.91% to 7.37% and standardized AE ratios ranged from 0.61 to 1.41.
Conclusions Using CHARM (Catheterization for Congenital Heart Disease Adjustment for Risk Method) to adjust for case mix complexity should allow comparisons of AE among institutions performing catheterization for congenital heart disease.
Until recently, the field of pediatric cardiac catheterization has lacked standardized nomenclature and definitions for adverse events (AE). This, in turn, led to different arbitrary definitions for adverse events being used by different institutions and inhibited the ability to establish benchmarks or expected outcomes in this specialty (1–7). Comparisons between and within institutions were further limited by the lack of ability to adjust for case mix differences. The C3PO (Congenital Cardiac Catheterization Project on Outcomes) was developed and executed to overcome these limitations in our ability to assess and compare outcomes in the field. Specifically, we sought to collect data in a uniform manner on all catheterization procedures performed at multiple institutions, including patient and procedural characteristics and the occurrence of AE (8). After 1 year of planning and development of a Web-based data entry tool, data collection commenced in 2007, and has resulted in the creation of a large multicenter dataset of congenital cardiac catheterization cases.
Cardiac catheterization in pediatrics and for adults with congenital heart disease encompasses a broad range of procedures, some of which occur infrequently, precluding assessment of risk for individual procedure types. Further, there is variation in the frequency of different procedures between centers and practitioners and a wide variety of adverse outcomes can occur in different interventions. To account for procedural diversity, we developed procedure type risk categories using both consensus and empirical methods (9). Although the procedure type risk categories have a strong relationship with AEs, other patient and/or procedural factors may also influence outcome. Thus, we sought to develop a method of case mix adjustment that would include the most important patient and procedural determinants of risk for clinically important AE in a multivariable model. The population analyzed included the entire cohort of patients undergoing catheterization for congenital heart disease, with a secondary cohort consisting of children <18 years of age. This was necessary to develop a risk adjustment method focused on pediatrics in parallel with the broader goal of developing a method for practitioners and institutions caring for both children and adults with congenital heart disease.
After institutional review board approval was obtained, 6 participating centers collected patient and procedural information and the occurrence of adverse events on all diagnostic and interventional catheterization procedures starting in February 2007. One of the 6 institutions did not start collecting data on biopsy cases until March 2009. Two additional sites became participants in April 2008 and June 2009, respectively. Thus, 8 sites contributed data to the registry (Online Appendix). Primary electrophysiology cases were not included. This analysis is limited to cases entered after August 1, 2007, because additional hemodynamic variables were added to the database and prospectively collected only after this date.
The methods for data collection and validation were previously published (8,9). Patient and procedural characteristics as well as the occurrence and type of adverse events were prospectively recorded. The following patient and procedural characteristics were recorded: type of case (diagnostic, interventional, or biopsy), center, admission status (elective, nonelective, emergent), age, sex, diagnosis, noncardiac problem, known or suspected genetic abnormality, airway management during catheterization (spontaneous respiration or mechanical support), types of interventions, case duration, and hemodynamic variables.
Because cardiac catheterization for congenital heart disease encompasses such a broad range of therapeutic and diagnostic procedures, we previously developed procedure type risk categories (Table 1). These categories were developed using both consensus and empirical methods. The categories represent procedure types with similar risk of adverse outcomes during cardiac catheterization procedures for congenital heart disease (9). As previously described, cases with multiple procedures were assigned to the category corresponding to the highest risk procedure.
AE were defined as any anticipated or unanticipated event from which injury could have occurred or did occur, potentially or definitely because of performing the catheterization. Events were recorded at the time of identification, either at the time of the case or later if determined to be related to the procedure. We used previously established and tested definitions for AE severity ranging from levels 1 to 5 (Table 2) (1,8). For this analysis, clinically important higher severity AE were defined as levels 3 and 4, or 5 AE. All AE were reviewed for proper application of seriousness definitions by the principal investigator and a designee. Any misapplication of definitions was reported to the participant and disagreements were resolved.
It was our supposition that a patient with abnormal hemodynamics would be at increased risk for clinically important AE. In previous work at The Children's Hospital in Boston, we defined abnormal hemodynamics using judgment and consensus and were able to show an association with AE (1). However, the thresholds were arbitrarily defined and not based on data. In this study, we sought to use empirical data to develop a single composite indicator of hemodynamic vulnerability based on the hemodynamic factors with the greatest ability to predict high-severity AE, while eliminating collinear variables with less explanatory value. The following 8 hemodynamic variables were assessed for inclusion in this composite measure: cardiac index, right ventricular (RV) systolic pressure, RV to systemic pressure ratio (RV ratio), systemic ventricle end-diastolic pressure (EDP), mixed venous saturation (MVsat), systemic arterial saturation (SAsat), main pulmonary artery systolic pressure (MPAsp), and main pulmonary artery mean pressure (MPAmp).
Frequency with percentage or median with interquartile range (25th and 75th percentiles) were calculated for patient and procedural characteristics of the entire cohort and for the subgroup of patients <18 years of age. Using a random sample of 75% of the cases collected between August 1, 2007, and December 31, 2009, the composite indicator of hemodynamic vulnerability was derived and multivariable models for predicting the outcome of high severity AE were developed. Model performance was then assessed both in this dataset as well as in the remaining 25% of cases.
Hemodynamic Vulnerability Indicator
Before any analyses were conducted, correlations among the 8 hemodynamic variables were explored using Pearson and Spearman correlation coefficients. For each individual variable, a receiver-operator characteristic curve was used to identify the threshold value that maximized discrimination for predicting a high-severity AE. A binary variable was then created based on this cutpoint. The entire cohort was used to derive single threshold values for cardiac index and EDP. For other variables, such as MVsat, SAsat, and MPAmp the cohort was divided into patients with single-ventricle (SV) and 2-ventricle (nonSV) physiologies. Threshold values for RV, RV ratio, and MPAsp were only assessed in the nonSV subset of the cohort.
Once the best threshold value had been determined for each of the 8 individual hemodynamic variables, the resulting binary variables were used to create a composite hemodynamic vulnerability indicator. Multivariable logistic regression with backward elimination was used to identify the subgroup of hemodynamic variables with the greatest ability to predict a high-severity AE. We started with a model containing all 8 hemodynamic variables, and removed them 1 at a time. If the area under the receiver-operator characteristic curve improved when a variable was removed, that variable was eliminated and the process repeated until eliminating an additional variable no longer improved the discrimination of the model. Once the most important hemodynamic variables were identified, we considered different methods of combining them to quantify hemodynamic vulnerability. This included using each hemodynamic risk factor as a simple binary variable indicating whether a case had any of the hemodynamic risk factors, taking the total number of hemodynamic risk factors present, and combining the number of risk factors into a score. Factors were weighted according to whether they were more or less predictive of the outcome. The simplest model with the highest discrimination was chosen as the final composite indicator of hemodynamic vulnerability.
Risk Adjustment Model
The random sample of 75% of the cases collected between August 1, 2007, and December 31, 2009, was used to create a multivariable logistic regression model for the outcome severity levels 3 and 4, or 5 AE in the entire cohort, as well as in the secondary cohort of children <18 years of age. Forward selection was used. The patient characteristic with the highest statistically significant improvement in the area under the receiver-operator characteristic curve (C-statistic), as assessed by the likelihood-ratio test, was retained in the model, and the remaining variables were reassessed. This process was repeated until there was no significant contribution to outcome prediction for any of the remaining variables.
The following patient and procedural characteristics were considered to develop the model: procedure risk category, diagnosis, age, noncardiac problem, known or suspected genetic abnormality, and hemodynamic vulnerability. Odds ratio (OR) with 95% confidence intervals (CIs) are presented for the final risk adjustment models. The Hosmer-Lemeshow goodness of fit test was used to assess calibration (10,11). Discrimination and calibration were also assessed in the validation datasets.
The final CHARM (Catheterization for Congenital Heart Disease Adjustment for Risk Method) model was used to determine risk-adjusted AE rates for severity levels 3 and 4, or 5 AE by institution for the entire cohort as well as for children <18 years of age. Standardized adverse event ratios (SAER) were calculated by dividing the observed AE rate within each institution by the expected AE rate accounting for the institution's case mix. To calculate the expected AE rate, the expected number of events was counted for each institution by summing the predicted probabilities of the outcome—generated from the final logistic regression model—for each case within that institution. The expected number of events was then divided by the total number of cases at that institution to calculate the expected rate. A SAER of 1.0 indicates that the observed event rate is equal to the expected rate given the institution's case mix complexity. If the SAER is >1.0, the institution has more events than would have been expected. If the SAER is <1.0, the institution has fewer events than expected. The 95% CIs were calculated for each SAER; if a CI does not contain the value 1.0, then the AE rate for that institution is significantly different from average (p < 0.05).
Between August 1, 2007, and December 31, 2009, 9,362 diagnostic and interventional cardiac catheterization cases were recorded in the database, 85% of which were performed in children <18 years of age (Table 3). One-half the population was composed of patients with SV circulation or complex 2-ventricle anatomy with outflow tract obstruction and/or intracardiac shunts due to septal defects. Twenty percent of the population had isolated lesions such as an atrial septal defect, patent ductus arteriosus, or valve abnormality. Genetic abnormalities were present in 12% of the cohort and noncardiac problems in 27%. Most of the procedures were performed electively (81%). When the population was limited to pediatric patients, there was a lower prevalence of spontaneous respirations as the method of airway management (25% vs. 31%). Patient and procedural characteristics were otherwise similar. The lowest procedure type risk category comprised the largest proportion of the procedures, with 42% in category 1, 29% in category 2, 19% in category 3, and 10% in category 4. The outcome for the risk adjustment model, high-severity AE, occurred in 5% of the entire cohort and at a similar rate in the secondary cohort of patients <18 years of age.
The following correlations were found among the hemodynamic variables: 1) higher cardiac index and higher MVsat; 2) higher MVsat, higher SAsat, and lower RV ratio; and 3) higher RV ratio, higher RV, higher MPAsp, and MPAmp. None of the other hemodynamic variables were associated with EDP (Fig. 1). We assigned threshold values empirically for each of the 8 hemodynamic variables (Table 4,Fig. 2). Multivariable modeling yielded 4 indicators of hemodynamic vulnerability independently associated with the occurrence of a high severity AE: EDP ≥18 mm Hg; SAsat <95% if nonSV or <78% if SV; MVsat <60% if nonSV or <50% if SV; and MPAsp ≥45 mm Hg if nonSV or MPAmp ≥17 mm Hg if SV. The number of hemodynamic variables present among these 4 categorized as 0, 1, 2, or more was the simplest composite indicator of hemodynamic vulnerability for predicting the outcome any severity levels 3 and 4, or 5 AE (C-statistic: 0.632) (Table 5).
Multivariable risk adjustment model: CHARM
The final multivariable model for the overall cohort included procedure type risk category, number of hemodynamic indicators, and age <1 year, with a C-statistic of 0.737 and Hosmer-Lemeshow test p value 0.74 (Table 6). For the age variable, we started with 5 categories (age <1 month, 1 month to <1 year, 1 to 10 years, 11 to 17 years, ≥18 years) but then collapsed categories with OR that were not significantly different from each other. In the secondary cohort of patients <18 years of age, the same 3 variables were identified; the model showed similar predictive performance and calibration, C-statistic: 0.734, and Hosmer-Lemeshow test p = 0.52. Models also performed well in the validation dataset, C-statistic: 0.734 for the entire cohort (Hosmer-Lemeshow p = 0.20) and 0.728 (Hosmer-Lemeshow p = 0.10) when limited to patients <18 years of age.
The observed adverse event rates among the 8 institutions ranged from 1.91% to 7.37% for the entire cohort (Table 7). Similarly, for patients <18 years of age, there was a wide range of observed AE rates among institutions, 1.71% to 7.86%. The CHARM model was used to determine an expected AE rate based on the case mix for each of the institutions, and SAERs were determined. For all institutions, SAER ranged from 0.61 to 1.41 in the entire cohort and 0.51 to 1.39 in patients <18 years of age (Fig. 3). Institutions C and G had lower AE rates than would be expected given their case mix, whereas institution E had a higher rate than would be expected.
The C3PO collaborative group has been accruing data on all congenital catheterizations at several U.S. centers over the course of more than 3 years, with data from nearly 10,000 cases available for this analysis. The investigative group previously was able to stratify catheterization procedure types based upon the risk for adverse outcomes, initially using expert consensus, then modifying the procedure type categories based on empirical methods (9). The risk categories were validated through comparison to a separate cohort of prospectively collected patients. In this analysis, we have identified 4 important hemodynamic variables associated with adverse outcomes including: systemic ventricular EDP ≥18 mm Hg, a SAsat <95% (or <78% if SV), MVsat <60% (or <50% if SV), and pulmonary artery systolic pressure ≥45 mm Hg (or mean ≥17 if SV). These easily and commonly measured factors of hemodynamic vulnerability, when combined with the previously validated procedure type risk categories and patient age, can be applied to make comparisons of the outcome of high-severity AE by adjusting for some of the case mix differences at different centers.
Risk adjustment has been successfully employed for many years in the fields of adult cardiology coronary intervention and pediatric cardiac surgery (12–14). This is a much more difficult process in pediatric cardiology, given the wide heterogeneity of diagnoses and types of interventions, as well as the relatively limited sample size of patients as a whole. However, despite this heterogeneity, risk adjustment models, including the RACHS-1 (Risk Adjustment in Congenital Heart Surgery) score and the Aristotle system, have been developed and have been included in the Society of Thoracic Surgeons and European Association for Cardio-Thoracic Surgery databases for congenital cardiac surgery for many years; the data have been useful in improving the ability to compare institutional outcomes adjusted for case mix and certain patient variables (15–20).
There is a comparative deficit in outcomes assessment in the realm of pediatric interventional cardiology, which has lagged behind in the development of a broadly collected, inclusive, and prospective database with sufficient anatomic and physiological information to allow for substantive interrogation. Currently, there are several groups gathering this type of data, including C3PO, the MAGIC (Mid-Atlantic Group of Interventional Cardiology), and the CCISC (Congenital Cardiovascular Interventional Study Consortium), but these efforts have been relatively recent and there is limited ability to cross-communicate between the systems (8,21–23). There is an effort to make a more widely used outcomes registry through the American College of Cardiology IMPACT (Improving Pediatric Adult Congenital Treatments) registry, which began in 2011 (24). Once the data are available, there will be a need for one or more methods allowing comparison between centers and interventionalists, given the wide variation in case type, referral patterns, and approaches to care between these groups. Physicians are understandably concerned that they be fairly compared when sensitive topics such as outcomes and adverse event rates are being critically evaluated and used for compensation and accreditation. In some states, these will also be available in the public record.
The hope is that interventionalists will be able to use these data to improve patient care and outcomes, not just to satisfy outside agencies. For a tool to be useful, it must be able to collect data in a standardized fashion, with easily understandable terminology that will be uniformly employed by the myriad of physicians and administrators who will ultimately use the system. The data must be stringently acquired and are likely to be improved by intermittent auditing for completeness and accuracy. The tool must be generalizable to the entire population and have demonstrated reliability in multiple settings. Although similar methods were used previously to derive a risk adjustment method at Children's Hospital in Boston, it was always considered preliminary because it was derived using only a single institutional dataset (25).
Hemodynamic vulnerability indicator
Empirical data informed the development of the indicator of hemodynamic vulnerability. Based on our experience, the 4 hemodynamic risk factors incorporated in the composite indicator variables also have significant face validity. However, there are some limitations with the indicator of hemodynamic vulnerability. Because of nonlinear relationships between hemodynamic values and risk of AE, we chose to use binary rather than continuous variables that meant threshold values were required. Although based on data, we admit that the thresholds may not be an important distinction for an individual patient, for example, an EDP of 17 mm Hg versus 18 mm Hg. Also, because they were empirically derived, different thresholds may have been chosen in a different dataset. Another limitation pertains to the measurement of these factors. Limited hemodynamic data may be obtained in a very ill or otherwise high risk patient resulting in the inability to account for risk due to lack of measurement. But by combining multiple variables in our final composite indicator of vulnerability variable (0, 1, or 2 or more), we sought to overcome this limitation in most cases.
It is important to consider the wide variation in observed AE rates by institution, ranging from 1.9% to 7.4%, and the narrower range of expected rates (3.1% to 5.7%) as supporting evidence of the need for a risk adjustment method to make equitable comparisons. Our analysis did show statistically significant variation in SAER at different institutions, even after risk adjustment, with 1 high and 2 low outliers. However, a number of factors potentially influencing the SAER must be considered before interpretation. First, one must consider the reliability of event reporting at different institutions, especially those with SAER <1.0. In the C3PO dataset, an independent record review and monitoring of a 10% sample revealed a 91% capture of high-severity events (9), but differential reporting remains a possibility. Second, and perhaps most importantly, we have not made any adjustment or consideration of efficacy as it relates to safety. Are institutions with lower than expected event rates avoiding taking on risk at the expense of completing all the objectives of the procedure? This must be considered carefully in the future as we try to understand what risk we must accept to achieve the best outcomes (26). Institutional comparisons cannot be regarded as definitive unless comparisons of both efficacy and safety are included. Alternatively, on the other extreme, does the 1 institution with lower 95% CI of 1.08 represent a performance outlier? Perhaps, but there may be unmeasured factors not accounted for, or optimized, in these models that are associated with higher event rates, but not associated with the quality of care. For example, although the models were developed using both pediatric and adult data, there may be factors unique to the adult congenital population that may be important to consider in a model for an isolated adult population, such as comorbidities not as common in the pediatric population. Despite the large size of the cohort used to derive and test our models, only 15% of the data was from adults >18 years of age. Finally, there may be differences in physician performance such as procedural duration, which was found to be independently associated with higher event rates in previous work (1). At present, taking into account these issues, we believe the CHARM model can be used to make comparisons of AE rates after pediatric cardiac catheterizations and can be regarded as a preliminary way to make similar comparisons for adult congenital heart procedures. As additional data for adult congenital procedures accrues, the performance of the CHARM model can be assessed further and refined if necessary.
The CHARM model to adjust for case mix complexity, based on procedure type, hemodynamic features, and age allows for equitable comparisons of adverse event rates among the institutions performing catheterization for congenital heart disease.
The authors would like to recognize the study coordinators and personnel that have made this project feasible: Gary Piercey, BS; Stephanie Valcourt, BA; Anniece Woods-Brown, RN, BSN; Joanne Chisolm, RN; Sharon Hill, MSN, ACNP-BC; Terri Mclees-Palinkas, MS, CCRC; Cyndi Murphy, RN, BSN; Rebecca Raic, MSN, RN, CPNP–PC; Holly Pfreim, RN, MSN, CNP; and Janene Allen, RN, CNP.
For a list of the expert panel and participating centers, please see the online version of this paper.
A Web-based application for data entry was developed in 2006 with funding support from the Children's Heart Foundation (Chicago, Illinois). The application was deployed on a Microsoft Internet Information Server obtained with funding support from the American Heart Association. The American Heart Association Physicians Roundtable Award (AHA-PRA) provides support for the project and career development plan for Dr. Bergersen (2006 to 2010). The Keane Operating Fund at The Children's Hospital in Boston provided the resources necessary to perform site visits and independent audits. Drs. McElhinney and Vincent are consultants and proctors for Medtronic. Drs. Beekman and Balzer are proctors for AGA Medical Inc. All other authors have reported that they have no relationships relevant to the contents of this paper to disclose.
- Abbreviations and Acronyms
- adverse event(s)
- confidence interval
- end-diastolic pressure
- main pulmonary artery mean pressure
- main pulmonary artery systolic pressure
- mixed venous saturation
- 2-ventricle physiology
- odds ratio
- right ventricle
- standardized adverse event ratio(s)
- systemic arterial saturation
- single-ventricle physiology
- Received April 12, 2011.
- Revision received May 9, 2011.
- Accepted May 14, 2011.
- American College of Cardiology Foundation
- Vitiello R.,
- McCrindle B.W.,
- Nykanen D.,
- Freedom R.M.,
- Benson L.N.
- Bergersen L.,
- Gauvreau K.,
- Marshall A.,
- et al.
- Lemeshow S.,
- Hosmer D.W.
- Hosmer D.,
- Lemeshow S.
- Peterson E.D.,
- Dai D.,
- DeLong E.R.,
- et al.,
- NCDR Participants
- Lacour-Gayet F.,
- Clarke D.,
- Jacobs J.,
- et al.,
- Aristotle Committee
- Jacobs J.P.,
- Jacobs M.L.,
- Lacour-Gayet F.G.,
- et al.
- Forbes T.J.,
- Garekar S.,
- Amin Z.,
- et al.,
- Congenital Cardiovascular Interventional Study Consortium (CCISC)