# Improvement of Risk Prediction After Transcatheter Aortic Valve Replacement by Combining Frailty With Conventional Risk Scores

## Author + information

- Received September 12, 2017
- Revision received November 6, 2017
- Accepted November 9, 2017
- Published online February 19, 2018.

## Author Information

- Andreas W. Schoenenberger, MD
^{a},^{∗}(andreas.schoenenberger{at}insel.ch), - André Moser, PhD
^{a},^{b}, - Dominic Bertschi, MD
^{a}, - Peter Wenaweser, MD
^{c}, - Stephan Windecker, MD
^{c}, - Thierry Carrel, MD
^{d}, - Andreas E. Stuck, MD
^{a}and - Stefan Stortecky, MD
^{c}

^{a}Department of Geriatrics, Inselspital, Bern University Hospital, and University of Bern, Bern, Switzerland^{b}Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland^{c}Department of Cardiology, Swiss Cardiovascular Center Bern, Bern University Hospital, Bern, Switzerland^{d}Department of Cardiovascular Surgery, Swiss Cardiovascular Center Bern, Bern University Hospital, Bern, Switzerland

- ↵∗
**Address for correspondence:**

Dr. Andreas W. Schoenenberger, Department of Geriatrics, Inselspital, Bern University Hospital, Freiburgstrasse, CH-3010 Bern, Switzerland.

## Graphical abstract

## Abstract

**Objectives** This study sought to evaluate whether frailty improves mortality prediction in combination with the conventional scores.

**Background** European System for Cardiac Operative Risk Evaluation (EuroSCORE) or Society of Thoracic Surgeons (STS) score have not been evaluated in combined models with frailty for mortality prediction after transcatheter aortic valve replacement (TAVR).

**Methods** This prospective cohort comprised 330 consecutive TAVR patients ≥70 years of age. Conventional scores and a frailty index (based on assessment of cognition, mobility, nutrition, and activities of daily living) were evaluated to predict 1-year all-cause mortality using Cox proportional hazards regression (providing hazard ratios [HRs] with confidence intervals [CIs]) and measures of test performance (providing likelihood ratio [LR] chi-square test statistic and C-statistic [CS]).

**Results** All risk scores were predictive of the outcome (EuroSCORE, HR: 1.90 [95% CI: 1.45 to 2.48], LR chi-square test statistic 19.29, C-statistic 0.67; STS score, HR: 1.51 [95% CI: 1.21 to 1.88], LR chi-square test statistic 11.05, C-statistic 0.64; frailty index, HR: 3.29 [95% CI: 1.98 to 5.47], LR chi-square test statistic 22.28, C-statistic 0.66). A combination of the frailty index with either EuroSCORE (LR chi-square test statistic 38.27, C-statistic 0.72) or STS score (LR chi-square test statistic 28.71, C-statistic 0.68) improved mortality prediction. The frailty index accounted for 58.2% and 77.6% of the predictive information in the combined model with EuroSCORE and STS score, respectively. Net reclassification improvement and integrated discrimination improvement confirmed that the added frailty index improved risk prediction.

**Conclusions** This is the first study showing that the assessment of frailty significantly enhances prediction of 1-year mortality after TAVR in combined risk models with conventional risk scores and relevantly contributes to this improvement.

Risk stratification before transcatheter aortic valve implantation (TAVR) is important when selecting those patients with severe aortic stenosis who will most likely benefit from the intervention. The Society of Thoracic Surgeons (STS) score and European System for Cardiac Operative Risk Evaluation (EuroSCORE) are frequently used for risk evaluation before TAVR, but both have been developed for conventional cardiac surgery procedures and are therefore not precise and complete enough for patients being considered for TAVR (1–6). So far, TAVR-specific variables are missing in conventional risk scores. In recent years, evidence has grown that functional assessment, in particular frailty measures, may improve risk stratification in patients undergoing TAVR (7–19). So far, the previous studies provided evidence that frailty measures are predictors of worse outcomes independent of the conventional risk scores (STS score or EuroSCORE) and thus have the potential to improve risk prediction of the conventional risk scores (7–19). However, no previous study has evaluated the potential improvement, when frailty measures are added to the conventional risk scores in combined prediction models. Thus, the present work evaluates whether a frailty score based on multidimensional geriatric assessment (MGA) may improve prediction of 1-year mortality after TAVR in combined models with the conventional risk scores and determined the magnitude of the added value.

## Methods

### Study population

Between September 1, 2009, and June 30, 2013, consecutive patients ≥70 years of age with symptomatic, severe aortic stenosis, referred for TAVR evaluation to Bern University Hospital, Switzerland, were eligible for this prospective cohort study. Aortic stenosis was considered severe, if the effective orifice area was <1 cm^{2} or <0.6 cm^{2}/m^{2} body surface area. An interdisciplinary team of interventional cardiologists and cardiac surgeons formed a consensus on treatment selection (TAVR, surgical aortic valve repair, or medical treatment). The consensus was based on several parameters including underlying comorbidities, general clinical condition, and perioperative risk as calculated with the logistic EuroSCORE and the STS score. We excluded patients who received a treatment other than TAVR (i.e., surgical aortic valve repair or medical treatment), patients in whom TAVR was performed as emergency procedure, patients who lived abroad and were unable to undergo follow-up evaluation, and patients who refused geriatric baseline examination or in whom the examination was not possible due to logistic reasons. In all other patients, a geriatric baseline examination was performed in addition to the cardiologic examination. We also excluded patients who died while waiting for TAVR and patients in whom the time between geriatric baseline examination and TAVR was >3 months. The final study population consisted of all patients in whom TAVR and the geriatric baseline examination was performed during the study period. This study complies with the Declaration of Helsinki and was approved by the local ethics committee.

### Baseline data

All participating patients received an extensive cardiologic baseline examination during an in-hospital evaluation. In addition to the recording of the patient’s history, transthoracic or transesophageal echocardiography (for determination of left ventricular ejection fraction [LVEF], aortic valve orifice area, and transvalvular mean gradient) and cardiac catheterization (for determination of transvalvular gradient, cardiac output, aortic valve area, right side filling pressures, and presence of coronary artery disease) were performed. Anemia was defined based on a cutoff point of 122 g/l hemoglobin concentration for women and 132 g/l for men. Based on the gathered information, the logistic EuroSCORE and the STS score were calculated. All participating patients also received a geriatric baseline examination consisting of the following validated instruments: Mini-Mental State Exam (MMSE) for cognitive function (20), Timed Up and Go test (TUG) for gait function (21), Mini Nutritional Assessment (MNA) for nutritional status (22), basic activities of daily living (BADL) (23), and instrumental activities of daily living (IADL) (24). For the purpose of this analysis, the instruments were dichotomized at standard cutpoints that were defined a priori according to current literature: MMSE at <27 points (cognitive impairment) versus ≥27 points (normal cognitive function), TUG at ≥20 s (mobility impairment) versus <20 s (normal gait function), and MNA at <12 points (at risk of malnutrition) versus ≥12 points (not at risk of malnutrition) (20–22). BADL and IADL were considered abnormal, if the patient had a difficulty in performing 1 or more activities (23,24).

### Frailty index

We used the frailty index previously developed by Schoenenberger et al. (8). This frailty index was developed a priori (i.e., there was no derivation cohort and no specific intention to predict mortality). The index was calculated as summary score from the following baseline components: 2 points were assigned, if MMSE was <21 points; 1 point was assigned for each of the following: MMSE ≥21 and <27 points, TUG ≥20 s, MNA <12 points, BADL ≥1 limited activity, IADL ≥1 limited activity, and a preclinical mobility disability (defined as decreased frequency of walking 200 m or of climbing stairs during the preceding 6 months). Thus, the frailty index had a range from 0 to 7 points.

### TAVR procedure

TAVR was performed within a maximum of 3 months after the cardiologic and geriatric baseline examination. It was performed according to local expertise as previously described (25). Either a CoreValve (Medtronic Inc., Minneapolis, Minnesota) or a SAPIEN XT bioprosthesis (Edwards Lifesciences, Irvine, California) was implanted.

### Follow-up and outcomes

One year after the TAVR procedure, mortality was assessed by specially trained research assistants. The assistants first contacted the patients. If the patients were unavailable, the assistants contacted the relatives, the general physician, or, if both were unavailable, the town hall. Using this approach, we obtained information on mortality from all patients. No patient was lost to follow-up. In patients who died, the research assistant obtained the death certificate. Clinical endpoints were systematically adjudicated by an independent clinical event committee of the Swiss TAVR registry (26). In the present study, mortality corresponded to all-cause mortality. We reported standardized endpoint definitions of the Valve Academic Research Consortium (VARC) within 1 year, including myocardial infarction, stroke, bleeding, acute kidney injury, and vascular complications (27).

### Statistical analysis

We described the study population by using frequency and percentage and median (interquartile range [IQR]). Time at risk started at date of surgery and ended 1 year after surgery or date of death. Patients could have multiple VARC endpoints and we calculated for each patient the total number of VARC endpoints. We calculated mortality rates and VARC rates with 95% confidence intervals (CIs) across frailty index categories using Poisson regression models. Monotonic trend across categories of frailty index was assessed by a Mann-Kendall test (28). We used several statistical approaches to determine the predictive information of frailty index and conventional risk scores. First, we investigated the association of covariates on survival using Cox proportional hazards models, reporting hazard ratios (HRs) with 95% CI. The proportional hazards assumption was tested by Schoenfeld’s test. Because age and sex were a priori expected to be related with 1-year survival and the investigated risk scores, all models were adjusted for those variables. Second, to assess predictive information of the risk scores and the components of the frailty index, we reported likelihood ratio (LR) chi-square test statistics. We reported the difference of LR chi-square test statistic to investigate whether the frailty index (B) added predictive information to the conventional risk score (A) (null hypothesis: A+B>A), and vice versa (null hypothesis: A+B>B) (29,30). Specifically, we reported the differences of LR chi-square test statistic from a combined risk score model A+B minus the LR chi-square test statistic from the single frailty index model (B) or the single cardiologic risk score (A) (29). If the LR chi-square test difference from the latter is greater than from the former, then the frailty index adds more to the combined model than the cardiologic risk scores. Further, we reported the ratio of LR chi-square test from the model with the single frailty index (B) in the nominator and the LR chi-square test statistic from the combined risk score model (A+B) (29). If this ratio is equal to 1 the frailty index contains all the predictive information from the combined risk score model, and the cardiologic risk scores contain no predictive information. When the ratio is equal to 0 the frailty index contains no predictive information in the combined model. Third, overall model performance was reported by Nagelkerke’s R^{2} (29,31). Nagelkerke’s R^{2} ranges from 0 to 1, with higher values indicating better model performance. Fourth, to assess discriminative ability, we used the C-statistic (31). The C-statistic is a unitless index of predictive discrimination, with a value of 0.5 indicating random prediction, and a value of 1 indicating perfect prediction. Fifth, we reported 2 reclassification-related measures, namely the continuous conditional net reclassification improvement and the conditional integrated discrimination improvement, among alive (nonevent) and death (event) individuals (32–34). The conditional net reclassification improvement can be interpreted as the net proportion of events (or nonevents) assigned to a higher risk category, among dead individuals (or alive individuals) (33). The conditional integrated discrimination improvement can be interpreted as the difference in predicted risk probabilities between a new model and an old model, among dead individuals (or alive individuals) (34). In sensitivity analyses we added known risk factors (i.e., anemia, atrial fibrillation, and LVEF <35%) to the conventional risk scores to analyze the predictive performance in addition to the frailty index. The p values reported from LR chi-square test statistics or Wald test statistics were 2 sided. All continuous variables were a priori modeled as quadratic relationship. Because for none of the continuous variables a quadratic relationship was found, continuous variables were finally modeled as linear variables. The frailty index was modeled in survival regression models as a linear variable, despite its ordinal character, because of overfitting issues. The sample size calculation was based on the 10 events per predictor rule (35,36). We assumed that 4 degrees of freedom would be spent in Cox proportional hazards models (i.e., predictors age, sex, frailty index, and 1 cardiologic risk score). Further, we assumed that 20% of the patients would die within 1 year. This led to a sample size of 200 individuals. All statistical analyses were done in R 3.2.3 (R Project for Statistical Computing, Vienna, Austria) using packages rms, pROC, and survIDINRI (37–39).

## Results

### Baseline data

Between September 1, 2009, and June 30, 2013, 613 patients at least 70 years of age with severe aortic stenosis were referred for TAVR evaluation (Figure 1). Of these, 283 patients fulfilled an exclusion criterion; the vast majority of these patients were excluded, because a treatment other than TAVR was selected. Of the remaining patients, who were eligible for study inclusion, we had to exclude 34 patients because they did not receive the geriatric baseline assessment. In 16 patients, the geriatric baseline assessment was not performed due to logistic problems and 18 patients refused this assessment. The study population finally consisted of 330 patients. None of these patients was lost during the 1-year follow-up.

Baseline characteristics of the study population are shown in Table 1. The median age of the study population was 83.6 years (IQR: 80.9 to 86.7 years), and 56.4% of the patients were women. A majority of the study population had hypertension (88.5%), roughly one-quarter (25.8%) had diabetes. Dyspnea New York Heart Association functional class III or IV was present in two-thirds of the patients. The median logistic EuroSCORE was 19.2% (IQR: 11.8% to 28.4%), and the median STS score 6.0% (IQR: 4.3% to 8.6%). The median frailty index was 3.0 points (IQR: 1.0 to 4.0 points). The components of the frailty index indicated that one-third of the study population had cognitive impairment and almost one-third had mobility impairment. TAVR was performed transfemorally in 309 (93.6%) patients, transapically in 19 (5.8%) patients, and using the subclavian artery in 2 (0.6%) patients.

### One-year mortality and VARC by category of frailty index

We investigated 330 patients at baseline with 52 deaths within 1 year (total 295.3 person-years). Overall 1-year mortality rate was 0.18 (95% CI: 0.13 to 0.23). Table 2 shows 1-year mortality and VARC rates by category of the frailty index. One-year mortality rates were very low in nonfrail patients and steadily increased along with increasing frailty (p value for monotonic trend = 0.004). One-year VARC rates showed evidence for a trend across categories of the frailty index (p value for monotonic trend = 0.02).

### Prediction of 1-year survival

Table 3 summarizes the modeling results of the evaluated risk scores and components of the frailty index for the prediction of 1-year mortality. EuroSCORE, STS score, and frailty index showed strong evidence for an association with 1-year mortality in Cox proportional hazards regression models. However, C-statistics for all 3 single risk scores were <0.70. Among the 3 risk scores, the frailty index showed the best model performance (Nagelkerke R^{2} 0.078), whereas the logistic EuroSCORE was found best for discriminative ability among all risk scores (C-statistic 0.67).

The LR chi-square test statistic from a model with the single logistic EuroSCORE was 19.29 (model A1), from a model with the single STS score 11.05 (model A2), and from the single frailty index 22.28 (model B). The LR chi-square test statistic from a model with a combination of the frailty index and the conventional risk scores was 38.27 for the logistic EuroSCORE (model A1+B) and 28.71 for the STS score (model A2+B). The difference in LR chi-square test statistic between model A1+B and model A1 was 18.98, whereas the LR chi-square test statistic difference between model A1+B and model B was 15.99. Thus, the frailty index contributed more predictive information to the combined model than the logistic EuroSCORE. The LR chi-square test statistic difference between model A2+B and model A2 was 17.66, whereas the LR chi-square test statistic difference between model A2+B and model B was 6.43. Thus, the frailty index contributed more predictive information to the combined model than the STS score. Specifically, the frailty index accounted for 58.2% of the predictive information (LR chi-square test statistic ratio B/[A1+B]) in the model A1+B, and accounted for 77.6% of the predictive information (LR chi-square test statistic ratio B/[A2+B]) in the model A2+B. Of note, the increase in Nagelkerke R^{2} indicated a relevant increase in model performance for the combined scores, and the C-statistics of the combined models increased to 0.72 for the combination of the logistic EuroSCORE with the frailty index. Sensitivity analyses showed that the frailty index improved risk prediction for survival, even when known risk factors (i.e., anemia, atrial fibrillation, and LVEF <35%) were added to conventional risk scores. Net reclassification improvement and integrated discrimination improvement confirmed that the frailty index improved risk prediction when added to the conventional risk scores.

The 5 components of the frailty index also showed evidence for an association with 1-year mortality, except for IADL and preclinical mobility disability. Among the components of the frailty index, the TUG showed best model performance (Nagelkerke R^{2} 0.070), and high predictive ability for 1-year mortality (LR chi-square test 19.84).

## Discussion

For the first time, this prospective cohort study provides evidence that a frailty index based on MGA improves prediction of 1-year mortality in patients undergoing TAVR, when combined with the conventional risk scores (STS score or EuroSCORE). Furthermore, this study proves that 1-year mortality gradually increases with increasing frailty.

Previous research has documented that frailty enhances conventional risk models in patients undergoing surgical procedures other than TAVR (40). To the best of our knowledge, this study is the first providing evidence that the same is true for TAVR patients. Previous studies have shown that frailty and other functional measures may predict outcomes in TAVR patients (7–19). However, only few studies showed that functional measures predict outcomes independent of the conventional risk scores (STS score or EuroSCORE) and thus have the potential to improve risk prediction (7,11–13,15). None of the previous studies proved that risk prediction of the conventional risk scores (STS score or EuroSCORE) is truly improved in combined prediction models (7–19). In a preliminary analysis of an initial subsample of the present cohort, the number of participants and the respective number of endpoints were not powered to document an improvement (7). The improvement in risk prediction found in this study is of clinical relevance. By adding information from the frailty index, the only moderate predictive information from the conventional risk scores, when used alone, may be increased so that a good predictor results. We therefore see the need for multicentric trials to develop better risk prediction models combining information from conventional risk scores with the information from functional assessments. Even though further research is needed, we emphasize that frailty should already now be part of every pre-TAVR clinical evaluation, as it may improve clinical decision making.

Our study also shows that 1-year mortality increases with increasing frailty. We previously showed that frailty may also be the consequence of severe aortic stenosis and may be reversed by TAVR (8). However, this study suggests that there might be a level of frailty, which is irreversible. One-year mortality in the highest 2 frailty categories exceeded 50%, and this mortality rate might be considered unacceptably high for performing TAVR. Therefore, our frailty index might have the potential to guide triage in patients being evaluated for TAVR and to recommend palliative treatment in these highest-risk patients. As the number of patients was low in the 2 highest frailty categories, it is too early to derive clinical recommendations from our findings, but we see an urgent need for further research in this field. In addition to triage patients, using a frailty index based on MGA has further advantages. In contrast to other frailty scores, information from MGA (e.g., information on cognition, mobility, nutrition, disability) may be used to improve therapeutic decisions.

Among the components of the frailty index, the TUG was the functional measure with the best prediction of 1-year mortality. Hereby, our study confirms the findings of a previous study showing that mobility impairment is extremely important for the prediction of adverse outcomes (10). Our study documents that it is important to use performance-based tests such as the TUG, and not measures based on self-report such as preclinical mobility disability. Whereas the TUG was an excellent predictor, preclinical mobility disability was not.

### Study limitations

First, patients who underwent TAVR were a selection of elderly, high-risk patients. Older patients with low risk were frequently allocated to open heart surgery, whereas elderly patients with excessive risk were sometimes assigned to medical treatment. Therefore, the predictive value of the frailty index has to be reconfirmed in patients with lower risk. Second, the results of this study are based on data from a single center. Therefore, confirmation in an independent sample would be needed to document generalizability of our findings. Nevertheless, there is meanwhile enough evidence supporting the importance of frailty measures in the risk prediction of patients undergoing TAVR and surgical procedures (7–19,41). Therefore, this limitation should not give rise to argue against the implementation of frailty measures into clinical practice. Third, frailty is a relatively novel concept, and there is not yet a generally accepted operational definition of frailty in the literature (41,42). In the present study, frailty was defined as an index based on validated instruments covering key subdomains related to frailty and disability in old age. The advantage of this operational definition of frailty is the inclusion of cognitive function, an aspect not covered in other operational definitions of frailty (41). Fourth, though we report on one of the largest prospective cohorts assessed with geriatric baseline examination allowing to determine frailty to date, the number of patients with high frailty index remains low. Fifth, according to the recruitment period, patients were treated with first generation devices. There is a small risk that results might be slightly different, if newer devices are used.

## Conclusions

The present study provides evidence that our frailty index enhances the risk prediction of conventional risk models in patients undergoing TAVR. We therefore recommend adding frailty assessment to conventional risk scores such as STS score or EuroSCORE. In addition, our frailty index has the potential to triage patients, because interventional risk is unacceptably high in the highest frailty categories. Therefore, frailty evaluation should be part of every clinical evaluation of patients who are considered for TAVR. We recommend using a frailty evaluation based on MGA, because MGA reveals further important information that may be used to improve clinical decision making. Because frailty may be the consequence of severe aortic stenosis (and, as such, may potentially be reversible after TAVR) or the result of other underlying conditions (and, as such, may be irreversible), experienced geriatricians should be part of the heart teams, at least when higher-risk patients are discussed.

**WHAT IS KNOWN?** Frailty is an important measure in determining which of the patients undergoing TAVR benefit from the intervention and which do not. So far, conventional risk scores (EuroSCORE or STS score) have not been evaluated in combined models with frailty for mortality prediction after TAVR.

**WHAT IS NEW?** The assessment of frailty significantly enhances prediction of 1-year mortality after TAVR in combined risk models with the conventional risk scores. Frailty should be measured in all patients who undergo TAVR before the intervention.

**WHAT IS NEXT?** Frailty should be added to the currently used conventional risk scores. The best way to assess frailty and to combine the information from frailty assessment with the conventional risk scores requires further investigation.

## Acknowledgments

The authors wish to express their gratitude to all study nurses, to Stephan Born, and to Dik Heg for their support.

## Footnotes

This study was supported by research grants from Bern University Hospital, Bern, Switzerland; and a grant of the Swiss National Science Foundation, Bern, Switzerland (SNF Grant 32003B_135807). Dr. Windecker has received institutional research grant support from Bracco, Boston Scientific, and Terumo. All other authors have reported that they have no relationships relevant to the contents of this paper to disclose. Drs. Schoenenberger and Moser contributed equally to the manuscript.

- Abbreviations and Acronyms
- BADL
- basic activities of daily living
- CI
- confidence interval
- CS
- C-statistic
- EuroSCORE
- European System for Cardiac Operative Risk Evaluation
- HR
- hazard ratio
- IADL
- instrumental activities of daily living
- LVEF
- left ventricular ejection fraction
- LR
- likelihood ratio
- MGA
- multidimensional geriatric assessment
- MMSE
- Mini-Mental State Exam
- MNA
- Mini Nutritional Assessment
- STS
- Society of Thoracic Surgeons
- TAVR
- transcatheter aortic valve replacement
- TUG
- Timed Up and Go test
- VARC
- Valve Academic Research Consortium

- Received September 12, 2017.
- Revision received November 6, 2017.
- Accepted November 9, 2017.

- 2018 American College of Cardiology Foundation

## References

- ↵
- ↵
- Nishimura R.A.,
- Otto C.M.,
- Bonow R.O.,
- et al.

- ↵
- ↵
- ↵
- Martin G.P.,
- Sperrin M.,
- Ludman P.F.,
- et al.

- ↵
- Wang T.K.,
- Wang M.T.,
- Gamble G.D.,
- Webster M.,
- Ruygrok P.N.

- ↵
- Stortecky S.,
- Schoenenberger A.W.,
- Moser A.,
- et al.

- ↵
- ↵
- Arnold S.V.,
- Reynolds M.R.,
- Lei Y.,
- et al.,
- PARTNER Investigators

- ↵
- ↵
- Green P.,
- Woglom A.E.,
- Genereux P.,
- et al.

- ↵
- Rodés-Cabau J.,
- Webb J.G.,
- Cheung A.,
- et al.

- ↵
- ↵
- Kim D.H.,
- Kim C.A.,
- Placide S.,
- Lipsitz L.A.,
- Marcantonio E.R.

- ↵
- Kleczynski P.,
- Dziewierz A.,
- Bagienski M.,
- et al.

- ↵
- Bureau M.L.,
- Liuu E.,
- Christiaens L.,
- et al.,
- MPI_AGE Project Investigators

- ↵
- Assmann P.,
- Kievit P.,
- van der Wulp K.,
- et al.

- ↵
- Okoh A.K.,
- Chauhan D.,
- Kang N.,
- et al.

- ↵
- Shimura T.,
- Yamamoto M.,
- Kano S.,
- et al.

- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- Wenaweser P.,
- Pilgrim T.,
- Kadner A.,
- et al.

- ↵
- ↵
- ↵
- ↵
- Harrell F.E.

- ↵
- Califf R.M.,
- Phillips H.R. 3rd.,
- Hindman M.C.,
- et al.

- ↵
- ↵
- ↵
- ↵
- ↵
- Austin P.C.,
- Steyerberg E.W.

- ↵
- ↵Harrell FE Jr. rms: Regression Modeling Strategies. R package version 4.2–1. 2014. Available at: http://CRAN.R-project.org/package=rms. Accessed November 30, 2017.
- ↵
- ↵Hajime U, Cai T. survIDINRI: IDI and NRI for comparing competing risk prediction models with censored survival data. R package version 1.1–1. 2013. Available at: http://CRAN.R-project.org/package=survIDINRI. Accessed November 30, 2017.
- ↵
- ↵
- ↵