Author + information
- Received May 29, 2012
- Accepted June 21, 2012
- Published online November 1, 2012.
- Sergio Raposeiras-Roubín, MD⁎,
- Emad Abu-Assi, MD⁎,⁎ (, )
- Pilar Cabanas-Grandío, MD⁎,
- Rosa María Agra-Bermejo, MD⁎,
- Santiago Gestal-Romarí, MD⁎,
- Eva Pereira-López, MD⁎,
- Rubén Fandiño-Vaquero, MD⁎,
- Belén Álvarez-Álvarez, PhD⁎,
- Cristina Cambeiro, PhD⁎,
- Marta Rodríguez-Cordero, MD†,
- Pamela Lear, PhD⁎,
- Amparo Martínez-Monzonís, MD, PhD⁎,
- Carlos Peña-Gil, MD‡,
- José María García-Acuña, MD, PhD⁎ and
- José Ramón González-Juanatey, MD, PhD⁎
- ↵⁎Reprint requests and correspondence:
Dr. Emad Abu-Assi, Cardiology Department, Clinical University Hospital of Santiago de Compostela; and Travesía Choupana s/n 15706, Santiago de Compostela, A Coruña 15706, Spain
Objectives This study sought to compare the in-hospital prognostic values of the original and updated GRACE (Global Registry of Acute Coronary Events) risk score (RS) and the AR-G (ACTION [Acute Coronary Treatment and Intervention Outcomes Network] Registry and the GWTG [Get With the Guidelines] Database) RS in acute coronary syndromes (ACS). To evaluate the utility of recalculating risk after percutaneous coronary intervention (PCI) with newer RS models (NCDR [National Cardiovascular Data Registry] and EHS [EuroHeart Score] RS).
Background Defined in 2003, GRACE is among the most popular systems of risk stratification in ACS. An updated version of GRACE has since appeared and new RS have been developed, aiming to improve risk prediction.
Methods From 2004 to 2010, 4,497 consecutive patients admitted to a single center in Spain with an ACS were included (32.1% ST-segment elevation myocardial infarction, 19.2% unstable angina). Discrimination (C-statistic) and calibration (Hosmer-Lemeshow [HL]) indexes were used to assess performance of each RS. A comparative analysis of RS designed to predict post-PCI mortality NCDR and EHS RS versus the GRACE and AR-G RS was performed in a subgroup of 1,113 consecutive patients included in the study.
Results There were 265 in-hospital deaths (5.9%). Original and updated GRACE RS and the AR-G RS all demonstrated good discrimination for in-hospital death (C-statistics: 0.91, 0.90 and 0.90, respectively) with optimal calibration (HL p: 0.42, 0.50, and 0.47, respectively) in all spectra of ACS, according to different managements (PCI vs. conservative) and without significant differences between the 3 different RS. In patients undergoing PCI, EHS and NCDR RS (C-statistic = 0.80 and 0.84, respectively) were not superior to GRACE RS (C-statistic = 0.91), albeit in the subgroup of patients undergoing PCI who were categorized as high risk using the GRACE RS, both EHS and NCDR have contributed to decrease the false positive rate generated by using the GRACE RS.
Conclusions Despite having been developed over 8 years ago, the GRACE RS still maintains its excellent performance for predicting in-hospital risk of death among ACS patients.
- acute coronary syndrome
- GRACE (Global Registry of Acute Cardiac Events) risk score
- risk score
- risk stratification
Risk stratification plays a pivotal role in optimizing management of acute coronary syndrome (ACS). It is the basis on which decisions are made about level of care, need for transfer to a tertiary care center, and specific pharmacological and interventional treatments (1). Although clinicians can stratify risk in their patients with ACS at a general level using clinical parameters, electrocardiographic changes, and biochemical cardiac markers, determination of risk for specific adverse outcomes requires the integration of multiple sources of information (2). Indeed, American College of Cardiology/American Heart Association (ACC/AHA) guidelines state that “estimation of the level of risk is a multivariable problem that cannot be accurately quantified with a simple table” (3). Among several risk scores for predicting in-hospital mortality, the focused update of ACC/AHA ACS guidelines (4) and the current European Society of Cardiology Guidelines (5) recognize the GRACE (Global Registry of Acute Coronary Events) (6,7) and the TIMI (Thrombolysis In Myocardial Infarction) (8) risk scores (RS) as the most accurate scoring systems for in-hospital death risk assessment (9,10). Recently, Pieper et al. (11) updated the in-hospital death GRACE RS, with the inclusion of nonlinear and interaction terms, thus improving its predictive value. A more contemporary RS for in-hospital death, known as AR-G (ACTION-GWTG) RS, has also been derived from a combined population consisting of patients in the ACTION (Acute Coronary Treatment and Intervention Outcomes Network) registry and the GWTG (Get With the Guidelines) database (12). In addition, there are 2 current models for predicting in-hospital risk of death following percutaneous coronary intervention (PCI), not only in the context of an ACS: the NCDR (National Cardiovascular Data Registry) CathPCI RS (13) and the EHS (EuroHeart Score) (14). Both models are designed on the basis that the risk of death in this patient group can have its own predictors. Notably, none of the recently developed RS (i.e., updated GRACE, AR-G, NCDR, and EHS) has been externally validated. Moreover, there are no data available about whether NCDR and EHS RS provide any incremental prognostic value over the GRACE RS in the dynamic process of death risk prediction. Accordingly, we compared performance of the original and updated GRACE RS, and the AR-G RS, to predict in-hospital mortality. Additionally, in a subgroup of patients undergoing PCI, we assessed the ability of the NCDR and EHS RS models to predict in-hospital risk of death and compared their performance with that provided by GRACE and AR-G RS.
Data sources and samples
This was a retrospective study in which demographic, clinical, and angiographic data, as well as data on management and in-hospital complications, had been prospectively collected and recorded in an electronic database. Subjects were all patients with a diagnosis of ACS admitted consecutively to our hospital between January 2004 and June 2010. ACS diagnosis was validated if the patient had new onset symptoms consistent with cardiac ischemia and at least one of the following criteria: cardiac biomarkers above the higher normal laboratory limit; ST-segment deviation on electrocardiogram; in-hospital stress testing showing ischemia; or known history of coronary vessel disease. Patients were classified as having acute myocardial infarction with ST-segment elevation myocardial infarction (STEMI) or non–ST-segment elevation acute coronary syndrome (NSTE-ACS) (unstable angina and non–ST-segment elevation acute myocardial infarction). The initial cohort consisted of 4,645 patients. We then excluded the following patients: those in whom ACS was precipitated in the context of surgery, sepsis, trauma, or cocaine consumption (n = 41); and those with missing data for any variable making up an RS under assessment (n = 107). Thus, the final cohort was composed of 4,497 patients. We then divided this study population into 2 sets: first, all patients, which was used to compare the performance of the original and updated GRACE RS (6,11) and the AR-G RS (12); and second, patients admitted between January 2007 and July 2010 who underwent PCI during the index episode (n = 1,113). This second set was used to compare the performance of RS developed specifically for predicting in-hospital mortality in PCI patients—NCDR (13) and EuroHeart (14) PCI RS—with those for predicting in-hospital mortality across the entire ACS population—the original and updated GRACE RS, and the AR-G RS.
Primary endpoints were all-cause in-hospital mortality, originally designated to be predicted by all the scores explored in this study.
Risk scores calculation
RS were calculated for each patient from the sum of the individual scores assigned to each of their corresponding variables (Online Appendix), as was previously described (6,11,12); thus, the total sum of these points corresponded to the total RS as a continuous variable. In addition, patients were categorized to different risk groups according to cutoff points and intervals established by each of the RS (see Online Appendix). Accordingly, 3 risk categories were established from the original and updated GRACE RS, whereas patients were assigned to 5 risk strata using the AR-G score.
We used the following standard clinical criteria to stratify risk of death in ACS patients: presence of elevated cardiac troponin I on admission (any elevation ≥0.1 ng/dl); ST-segment deviation ≥2 mm in ≥2 contiguous leads; and/or heart failure at presentation.
Calibration and discrimination
Indexes of calibration and discrimination were used to assess performance of the various RS studied. We used the Hosmer-Lemeshow (HL) goodness-of-fit test to assess calibration (15), in which higher p values indicate better calibration. Each RS was entered, separately, into a logistic regression model to generate the individual risk probability of death. The HL statistic from the regression modeling was used as an indicator of goodness-of-fit of each RS as a global predictor variable. Discriminative power of each RS was assessed by the C-statistic, equivalent to the area under the receiver-operating characteristics curve (16). A model with a C-statistic >0.75 was considered to have meaningful discriminatory ability. Negative and positive predictive values for each RS and risk stratification according to clinical criteria were also computed for the high-risk group.
Discrete variables are expressed as frequencies and percentages, and quantitative data are presented as the mean ± SD. Chi-square test was used to compare discrete variables and the Student t test to compare quantitative variables. The discriminatory abilities of the original and updated GRACE RS, as well as the AR-G score for in-hospital death were computed and compared according to the nonparametric method described by DeLong et al. (17). A p value <0.05 was considered statistically significant. All analyses were performed using SPSS (version 17.0, SPSS Inc., Chicago, Illinois) software and Medcalc (Mariakerke, Belgium).
Table 1 shows patient characteristics, overall and in subgroups of survivors and nonsurvivors. A total of 338 patients (7.5%) were transferred from other centers, and 268 (5.9%) patients died in hospital. As expected, patients who died were older and more likely to have a greater comorbidities burden, such as diabetes, history of heart failure, vascular disease, chronic obstructive pulmonary disease, and prior malignancy, than were those who survived the index event. Similarly, they were more likely to be at Killip class ≥II at admission, to present with STEMI, to have more reduced left ventricular ejection fraction and more severe valvular heart disease. Nonsurvivor patients were also more likely to have lower admission hemoglobin, lower glomerular filtrate rate, as well as and more extensive coronary artery disease involving both the left main and proximal left anterior descending coronary arteries. Similarly, nonsurvivors had less frequently undergone PCI procedures.
Predictive accuracy of RS
Both the original and updated GRACE and AR-G RS showed excellent discrimination and calibration together with marked risk gradients (Fig. 1), regardless of whether the population was taken as a whole, categorized in terms of type of ACS (STEMI vs. NSTE-ACS), or divided into subgroups of patients undergoing PCI or not (Table 2,Fig. 2). Neither the updated GRACE RS nor the more contemporary AR-G RS outperformed the original GRACE RS in any of the subgroups analyzed (all p values ≥0.1 for comparison of C-statistic values of the 3 scores). Similarly, the discriminative capacity of the 3 scores did not differ significantly in the following risk subgroups: women; patients aged ≥75 years; and in those patients with diabetes mellitus, chronic renal failure (defined as glomerular filtration rate <60 ml/min/1.73 m2), and peripheral artery disease (Online Table 1).
Sensitivity and specificity, together with the positive and negative predictive values of the GRACE and other RS, and risk stratification based on clinical criteria, are shown in Table 3. The original GRACE RS had the highest negative predictive value (false negative rate = 0.6%, equal to clinical criteria) with an optimal specificity (62.1% vs. 11.6% of clinical criteria). The updated GRACE RS did not improve the predictive accuracy of the original version of GRACE. However, AR-G RS reduced the false positive rate when compared with the original GRACE and with clinical criteria (72.7% vs. 86.3% vs. 93.4% for the AR-G, original GRACE RS and clinical criteria, respectively) but with a significantly higher rate of false negatives (1.8%, a 3-fold increase over the original GRACE RS).
Role of contemporary RS for predicting in-hospital mortality following PCI
NCDR and EHS RS were validated in all consecutive patients undergoing PCI from January 2007 to July 2010. Of the 1,113 patients included, 48 died in hospital (4.3%). Both EHS and NCDR models showed excellent discrimination (C-statistic = 0.855 [95% confidence interval (CI): 0.833 to 0.875] and 0.894 [95% CI: 0.875 to 0.912] for the EHS and NCDR RS, respectively), and good calibration (HL p values 0.71 and 0.33, respectively). The C-statistic values for these models were lower than those obtained for GRACE, although the difference was only significant with respect to the EHS RS (p = 0.045).
Categorizing patients in terms of the original GRACE RS, in groups of low or moderate risk (mortality rate <1% in our study population), the use of EHS or NCDR RS did not provide any important prognostic information on risk following PCI. However, in the group defined by GRACE as high risk, both PCI score systems did provide important additional information. From the 1,113 patients who underwent PCI from January 2007 to July 2010, 423 were classified in the high-risk GRACE group, with an in-hospital mortality of 10.6%.
Figure 3 shows the rate of in-hospital death in the high-risk GRACE group, according to NCDR and EHS RS. In this selected sample of patients, both models (EHS and NCDR RS) showed good discrimination (C-statistics = 0.801 and 0.842 for EHS and NCDR RS, respectively), and calibration (HL p value = 0.579 and 0.640, respectively). Furthermore, both systems improved in-hospital mortality risk prediction by reducing the number of false positive rates. The application of EHS and NCDR RS in this high-risk subgroup of patients resulted in a substantial reduction of the false positive rates (from ∼90% to 40% for EHS RS and from ∼90% to 57.5% for NCDR RS), due to lower sensitivity (25.0% for EHS and 64.6% for NCDR RS).
We made several clinically important observations in this study. First, the GRACE RS is a useful tool in risk stratification in ACS, with a sensitivity similar to standard clinical criteria but with a higher specificity. Second, after discrimination and calibration analyses of the patient population as a whole, categorized by type of ACS, and stratified according to whether a patient underwent PCI, the more contemporary RS (AR-G and updated GRACE RS) failed to improve the predictive ability of the original in-hospital death of GRACE RS, which was developed over 8 years ago. Third, in patients undergoing PCI, neither EHS nor NCDR RS were superior to GRACE RS, despite the fact that the GRACE RS was developed in a cohort of patients with a much lower rate of PCI (<50% in STEMI and <30% in NSTE-ACS). Indeed, the discriminatory power of EHS RS was found to be significantly inferior to GRACE RS. However, in the subgroup of patients at high risk as defined by the GRACE scoring system, both EHS and NCDR RS minimized the overestimated risk of death (false positive rate) provided by the GRACE RS.
In our study, the predictive capacity of in-hospital death after an ACS was excellent for both the original and updated versions of GRACE RS and for the AR-G RS. The original GRACE RS has already been validated in numerous studies since its publication in 2003 (9,18–22). By contrast, previous to our study, neither the updated GRACE RS nor AR-G RS had been validated in an independent dataset of ACS patients.
In this study, we found that the original GRACE RS had a similar sensitivity to that obtained with clinical criteria, increasing specificity 4-fold and reducing the false positive rate by 6.8%. This might be due to variables, such as age, heart rate, systolic blood pressure, and creatinine level having been entered as continuous variables, resulting in more accurate risk stratification, and thus reducing the overestimation of risk of death based on clinical criteria. By contrast, the updated GRACE score did not improve any of these parameters in comparison with the original version. The AR-G RS, however, showed a false-negative rate 3-fold higher than either GRACE RS or clinical criteria, an unacceptable level given that the predicted event is in-hospital mortality, although it did increase specificity and reduce the rate of false positives obtained with GRACE. These results could be explained by analyzing the models used to derive the various RS. The updated GRACE RS was derived from the same registry (an international multicenter prospective registry) and using the same variables as the original GRACE but over a longer period (1999 to 2006) (11) and with changes to the scoring system. The main change has been to minimize the importance of Killip class but maximize that of kidney function. The discriminative capacity was similar in both versions (C-statistics = 0.907 and 0.896 for the original and updated GRACE, respectively). However, the fact that updated GRACE RS minimizes the presence of heart failure at admission might justify, at least partially, that the percentage of patients in non-high-risk groups who presented hospital death (false negatives) was 2.6× higher in comparison with the original GRACE version.
In contrast to GRACE, the AR-G RS is derived from a multicenter registry limited to the United States, between 2007 and 2008, and of hospitals who participated voluntarily, excluding patients transferred from other centers (12). The registry is, however, more contemporary than that of GRACE, with almost twice the rate of PCI (45% of patients having PCI, of which 80% are primary PCI). The composition of the 2 systems is very similar, the differences being the sole addition of peripheral artery disease and assessing markers of myocardial damage as a continuous variable. Thus, the discriminative power of each of the 2 RS is expected to be very similar. In fact, a recent study showed that the addition of ultrasensitive troponin measurement did not provide important additional information over that in GRACE RS (20). Moreover, several predictors specifically included in the new RS, such as peripheral artery disease, are at least indirectly accounted for by GRACE RS.
Both the GRACE and AR-G RS were designed for use in the emergency department during initial medical examination; however, due to the dynamic nature of ACS pathogenesis, the patient's risk should be subsequently re-evaluated as further information is obtained. In patients undergoing invasive management, therefore, it is desirable that the risk of death is recalculated after catheterization and/or PCI. The NCDR and EHS RS were designed explicitly to determine the probability of in-hospital death after PCI. The NCDR score (13) was derived from the National Cardiovascular Data Registry, with data from 181,775 procedures performed from 2004 to 2006 in centers in the United States, and the EHS RS (14) was based on the EuroHeart Survey of PCI and contained information on 46,064 consecutive patients who underwent PCI due to various indications in European centers from 2005 to 2008. To date, however, no PCI-related RS has been externally validated, nor its predictive ability compared with that of the GRACE RS.
In our study, both the EHS and NCDR RS performed very well for in-hospital mortality in patients with ACS undergoing PCI. When compared with the GRACE RS, the EHS was significantly lower, whereas the NCDR showed no significant difference. This could be explained by the composition of each scoring system. Thus, the EHS RS does not include several widely recognized variables, such as blood pressure, heart rate, renal function, ST-segment changes, or increased markers of myocardial necrosis. The emphasis of EHS is on angiographic parameters, such as multivessel disease, involvement of bifurcations or the coronary segments as left main or proximal left anterior descendent coronary artery, type-C lesions, or initial TIMI flow grade. The NCDR RS, by contrast, includes renal failure but does not recognize variables, such as blood pressure, heart rate, presence of ST-segment changes, or markers of myocardial necrosis; additionally, it assesses mortality at 30 days rather than in-hospital death.
It is important to remember that both EHS and NCDR RS were designed to assess risk after PCI not only in ACS but also in other scenarios (i.e., stable angina), whereas GRACE and AR-G RS were designed to evaluate initial risk in the ACS scenario to optimize therapeutic strategy. Our observation that the data obtained during catheterization and used in these 2 RS (NCDR and EHS) do not improve the initial prognostic assessment achieved with the GRACE or the AR-G RS and should therefore be interpreted with caution, taking into account the different patient settings. We have found a possible additive utility in patients defined as high risk according to the original GRACE RS (and who are therefore candidates for invasive management). In these patients, NCDR and EHS RS appear to show better discrimination of in-hospital risk of death due to their reduction of the rate of false positives and optimization of the use of hospital resources, although they do so at the expense of lowering sensitivity.
With advances in ACS management, a major objective has been the reduction of in-hospital mortality (23,24). In this context, the use of RS has proven to be very helpful to clinicians. It is especially important to optimize the use of hospital resources because aggressive strategies are not always tailored to higher-risk patients. Thus, there has been a proliferation of risk stratification systems. Studies such as ours provide a comparative assessment of the usefulness and reliability of RS for predicting in-hospital mortality. At present, no RS has proven superior to the GRACE model in the risk prediction of in-hospital death. European guidelines for ACS management (5) emphasize the importance of early risk stratification in ACS using the GRACE RS (Class I, Level of Evidence: B). However, neither the ACC/AHA guidelines nor their recent focused update (4) explicitly recommend the employment of GRACE, even though its high predictive value is recognized. We think that the application of risk stratification systems, and specifically the GRACE RS, could be useful for the proper management of ACS patients and the optimization of their therapeutic care. We recommend, therefore, that physicians should familiarize themselves with their use.
One limitation of this study is that inherent to retrospective studies. A further limitation of the study is its single institution nature. Only patients admitted to our institution, which is equipped to perform coronary angiography and PCI, were included; the applicability of the present results should therefore be viewed with caution in other centers with different therapeutic management patterns. However, findings from our registry offer the advantage of providing more consistent results compared with multicenter registries because of the large variability in each center's institutional protocols, hence, risk stratification may not translate to a particular center's experience. Another limitation of this study is the lack of sample size calculation to discriminate between RS; therefore, the study could be underpowered to detect possible differences between the RS's discriminative capacity, particularly for predicting death in the subgroup of PCI patients where only 48 events (i.e., in-hospital death) were observed. Finally, RS could not be calculated in 107 patients of our cohort, ostensibly because of an absence of on-admission heart rate. Nevertheless, the prevalence of missing data in our study was modest when compared with other data registries.
Despite its having been developed over eight years ago, the GRACE RS still maintains its excellent performance in predicting the in-hospital risk of death among ACS patients. Neither the updated version of GRACE RS nor new scoring systems, such as the AR-G RS, exceed the predictive ability of GRACE. In our study, the risk assessment of in-hospital death after PCI using the more contemporary PCI-based RS (i.e., EHS and NCDR scores) did not improve the sensitivity of the GRACE score although offered the advantage of reducing the false positive rate generated by using the GRACE scoring system.
For risk score calculations, please see the online version of this paper.
The authors have reported that they have no relationships relevant to the contents of this paper to disclose.
- Abbreviations and Acronyms
- American College of Cardiology
- acute coronary syndrome
- American Heart Association
- AR-G RS
- ACTION (Acute Coronary Treatment and Intervention Outcomes Network) Registry and the Get With the Guidelines (GWTG) database risk score
- confidence interval
- EuroHeart Score
- non–ST-segment elevation acute coronary syndrome
- percutaneous coronary intervention
- risk score
- ST-segment elevation myocardial infarction
- Thrombolysis In Myocardial Infarction
- Received May 29, 2012.
- Accepted June 21, 2012.
- American College of Cardiology Foundation
- Lee C.H.,
- Tan M.,
- Yan A.T.,
- et al.,
- Canadian ACS Registry II Investigators
- Yan A.T.,
- Yan R.T.,
- Tan M.,
- et al.
- Anderson J.L.,
- Adams C.D.,
- Antman E.M.,
- et al.
- Wright R.S.,
- Anderson J.L.,
- Adams C.D.,
- et al.
- Hamm C.W.,
- Bassand J.P.,
- Agewall S.,
- et al.
- Global Registry of Acute Coronary Events
- de Araújo Gonçalves P.,
- Ferreira J.,
- Aguiar C.,
- Seabra-Gomes R.
- Chin C.T.,
- Chen A.Y.,
- Wang T.Y.,
- et al.
- Peterson E.D.,
- Dai D.,
- DeLong E.R.,
- et al.,
- NCDR Registry Participants
- de Mulder M.,
- Gitt A.,
- van Domburg R.,
- et al.
- Lemeshow S.,
- Hosmer D.W.
- Gale C.P.,
- Manda S.O.,
- Weston C.F.,
- Birkhead J.S.,
- Batin P.D.,
- Hall A.S.
- Meune C.,
- Drexler B.,
- Haaf P.,
- et al.
- Yusufali A.,
- Zubaid M.,
- Al-Zakwani I.,
- et al.
- Krumholz H.M.,
- Merrill A.R.,
- Schone E.M.,
- et al.
- Bernheim S.M.,
- Grady J.N.,
- Lin Z.,
- et al.