Author + information
- Received July 7, 2011
- Revision received August 16, 2011
- Accepted September 3, 2011
- Published online January 1, 2012.
- Frederic S. Resnic, MD, MSc⁎,⁎ (, )
- Tracy Y. Wang, MD, MHS, MSc†,
- Nipun Arora, MD⁎,
- Venkatesan Vidi, MD⁎,
- David Dai, PhD†,
- Fang-Shu Ou, MS† and
- Michael E. Matheny, MD, MPH, MSc‡
- ↵⁎Reprint requests and correspondence:
Dr. Frederic S. Resnic, Cardiovascular Division, Brigham and Women's Hospital, 75 Francis Street, Boston, Massachusetts 02115
Objectives This study sought to quantify the learning curve for the safety and effectiveness of a newly introduced vascular closure device through evaluation of the NCDR (National Cardiovascular Data Registry) CathPCI clinical outcomes registry.
Background The impact of learning on the clinical outcomes complicates the assessment of the safety and efficacy during the early experience with newly introduced medical devices.
Methods We performed a retrospective analysis of the relationship between cumulative institutional experience and clinical device success, defined as device deployment success and freedom from any vascular complications, for the StarClose vascular closure device (Abbott Vascular, Redwood City, California). Generalized estimating equation modeling was used to develop risk-adjusted clinical success predictions that were analyzed to quantify learning curve rates.
Results A total of 107,710 procedures used at least 1 StarClose deployment, between January 1, 2006, and December 31, 2007, with overall clinical success increasing from 93% to 97% during the study period. The learning curve was triphasic, with an initial rapid learning phase, followed by a period of declining rates of success, followed finally by a recovery to a steady-state rate of improved device success. The rates of learning were influenced positively by diagnostic (vs. percutaneous coronary intervention) procedure use and teaching status and were affected inversely by annual institutional volume.
Conclusions An institutional-level learning curve for the initial national experience of StarClose was triphasic, likely indicating changes in patient selection and expansion of number of operators during the initial phases of device adoption. The rate of learning was influenced by several institutional factors, including overall procedural volume, utilization for percutaneous coronary intervention procedures, and teaching status.
A learning curve can be defined as a proportional improvement in performance, such as the clinical outcomes of a medical procedure, with each doubling of cumulative experience (1). Whereas learning curve effects have been quantified in numerous industries outside of health care, within medicine, characterizing learning curve effects has been challenging. Analyses of surgical outcomes (2–4), interventional cardiology procedures (5,6–8), and improvements in single center outcomes (9–11) have all demonstrated the presence of learning effects for medical procedures. However, there has been limited investigation of the learning effect with regard to specific medical devices, often due to insufficient sample size to quantify the effect. In addition, learning effects must be distinguished from traditional steady-state “volume-outcome” relationships in which higher volume providers or centers have typically been shown to have improved outcomes relative to lower volume providers (12–14). Separating the impact, over time, of secular improvements in care, changes in clinical practice, including case selection, and the impact of institutional operator learning can be quite challenging, but these are critical to understanding the safety and effectiveness of new devices and procedures.
In this study, we sought to quantify the learning curve for the safety and effectiveness of a newly introduced vascular closure device (VCD) through the evaluation of the NCDR (National Cardiovascular Data Registry) CathPCI data registry. Within the broad population of centers and physicians reporting to this registry, VCD are used in more than 45% of cardiac catheterization procedures to achieve hemostasis at the femoral arteriotomy site (15). Given the relatively high usage rate, VCD represent an excellent opportunity for studying learning effects because these devices have been adopted across a broad spectrum of practice environments, clinical outcomes can be easily identified, and standardized clinical data definitions are widely accepted. Understanding the learning curve associated with VCD implants is important, because vascular complications following catheterization procedures are powerful independent predictors of morbidity and long-term mortality (15). In addition, VCD deployment failure has been shown to increase significantly the risk of vascular complications (16).
Therefore, we performed a retrospective analysis of the relationship between cumulative institutional experience and clinical device success, defined as device deployment success and absence of vascular complications, for the StarClose VCD (Abbott Vascular, Redwood City, California), approved for clinical use in the United States on December 21, 2006. We hypothesized that a learning curve could be quantified from cumulative outcomes data and that the learning rates would be affected by a variety of institution factors, including procedural volume, prior experience with VCD, teaching status, and type of procedures performed.
A retrospective analysis of the NCDR CathPCI registry data were performed to assess changes in successful deployment rates and clinical outcomes following the deployment of StarClose VCD between January 2005 and December 2007. A total of 1,650,953 patient visits, from 797 participating institutions, which included diagnostic catheterization procedures and/or percutaneous coronary intervention (PCI), were included. Patients were excluded who underwent nonfemoral vascular access procedures, who had procedures at centers providing incomplete CathPCI datasets, or who had an intra-aortic balloon pump inserted at the time of cardiac catheterization. Clinical success was defined as a procedure with successful device deployment, as reported by the implanting operator, and no reported vascular complication during the hospitalization. Vascular complications were defined per standard American College of Cardiology–NCDR definitions (17) and included access-site bleeding, retroperitoneal hemorrhage, access-site occlusion, peripheral embolization, arterial dissection, pseudoaneurysm, and arteriovenous fistula. The learning curve was estimated using the performing center (hospital) as the fundamental unit of analysis, exploring the impact of accumulating volume of StarClose VCD deployments on observed center-specific clinical success rates.
In addition, to further characterize the factors influencing the learning curve, predefined subgroups were analyzed based on type of procedure (diagnostic vs. interventional), annual institutional procedural volume, and teaching status of the institution (teaching vs. nonteaching institution). Of note, physician-specific identifier information was inconsistently documented in the NCDR registry, such that reasonable estimates of individual experience with the VCD studied were not available. From prior analyses using the CathPCI dataset, it was known that there is relatively little missing data, due to software requirements for complete case-level data submissions. For this analysis, missing data for any covariate was imputed to have a value of “none” or “negative” for categorical variables and was estimated as sex-specific mean values for continuous variables.
A logistic regression model was used to adjust for variables that might influence clinical success following StarClose deployment. The model was based on previously published clinical factors that predict rates of vascular complication following VCD use (15,18,19–21) and included age, sex, race, prior congestive heart failure, prior myocardial infarction, body mass index, peripheral vascular disease, chronic lung disease, cerebrovascular disease, renal failure, diabetes, hypertension, prior coronary artery bypass grafting, cardiogenic shock, and presentation with acute coronary syndrome. Generalized estimating equation (GEE) modeling, with clustering by center, was used to develop risk-adjusted clinical success predictions that were analyzed by institutional volume to quantify learning curve rates. A spline transformation was performed using the GEE results to quantify the effect of increasing institutional experience on clinical success during each phase of an experience curve that appeared to be noncontinuous. Institutions were stratified into quartiles based on volumes of StarClose VCD deployment, and 4 separate GEE models were fitted for each strata of institutional volume. The chi-square test was used for comparisons of categorical data, and the 2-tailed Student t test was used to compare continuous variables. A p value <0.05 was considered statistically significant for all comparisons. All statistical analyses were performed using SAS software (version 9.2, SAS Institute Inc., Cary, North Carolina).
Among the original study cohort, 107,710 procedures at 468 institutions between January 2005 and December 2007 included the deployment of at least 1 StarClose VCD and were included in this analysis. Baseline characteristics of the study population are summarized in Table 1. The study population included 40% women, 28% diabetics, and 8.2% patients with a history of peripheral arterial disease. In addition, of the 107,710 procedures analyzed, 31,012 (29.6%) were PCI procedures. Hospitals included in the analysis were representative of the CathPCI registry and included generally larger volume centers, with 57% having an annual PCI procedural volume of more than 500 cases and 47% having post-graduate teaching programs (Table 2). As predicted, there was little missing data in the analyzed dataset, with no covariates having more than 7.5% missing data. Only 1.96% of patients undergoing left heart catheterization and/or coronary intervention were excluded from the analysis due to use of nonfemoral access sites (brachial or radial) or due to the use of intra-aortic balloon.
Access site bleeding complications were documented in 461 patients (0.43%), whereas retroperitoneal bleeding was found to occur in 162 (0.15%) patients. Any vascular complication was documented in 1.33% of patients (Table 3). The overall StarClose device deployment failure rate was found to be approximately 5% and was the major determinant of clinical success.
The overall clinical success rate increased from 93% to 97% with increasing institutional experience over a 2-year period (Fig. 1). This learning effect translated into a 30% reduction in clinical failure rate with each doubling of cumulative institutional experience. Institutional learning curve effects were observed in multiple subgroups and were positively correlated with increasing annual catheterization laboratory volume of the institution. Higher rates of clinical success were observed in the diagnostic, compared with the PCI group, with difference in success declining from 5.6% during the initial phase of institutional experience to 3% with growing institutional experience (Fig. 2). The GEE models fitted for each of the institutional volume strata demonstrated adequate discrimination for the purpose of exploring the main effect (device-specific experience) on the clinical success rate adjusted by other possible confounders, with C-indexes ranging from 0.57 to 0.61.
The rate of learning (i.e., slope of the learning curve) was influenced by diagnostic (vs. interventional) procedures (more rapid improvement for interventional procedures), teaching status of the institution (more rapid with nonteaching status), and annual institutional volume (more rapid with smaller institutional volumes). Figure 3 illustrates the clinical success rate of StarClose deployment with increasing average StarClose experience per operator, stratified by annual institutional volume.
As illustrated in Figure 3, annual hospital volume of StarClose device deployments, normalized by the average experience per operator performing procedures at the institution, appears to influence significantly the rates of learning. Lower volume institutions required few StarClose deployments per operator to achieve high rates of clinical success as compared with larger volume institutions. At lower volume centers, 98% clinical success rates were predicted after the first 75 StarClose deployments per operator. At the highest volume centers (>2,400 total procedures annually), an average StarClose deployment volume of 130 cases per operator was required to achieve a clinical success rate of 98%.
In addition, a triphasic learning pattern emerged in institutions with larger annual volumes, as shown for the largest volume centers with >2,400 device deployments in Figure 4. This triphasic distribution is characterized by a rapid initial learning phase (phase I), followed by a period of declining procedural success (phase II), followed by recovery to a steady state of improvement with increasing experience (phase III). Whereas the absolute differences in the rates of procedural success were small (94% in phase I to 92% in phase II), these changes represent a 17% increase in the rate of procedural failure (p < 0.001). Changes in the characteristics of patients treated in each of the phases of the learning curve were explored in an effort to understand the potential impact of patient selection on the observed differences in outcomes and rates of learning between phases I, II, and III of the learning curve (Table 4). The proportion of patients with a variety of high-risk clinical conditions, including: diabetes, dialysis, urgent procedure status, PCI (compared with diagnostic procedures), and use of glycoprotein IIb/IIIa antagonists declined significantly from phase II to phase III in the large institutions studied (>1,600 procedures per year). These changes in the composition of patients selected to receive the StarClose VCD likely contributed to the overall improvement to observed outcomes in the later phases of experience with the device. There was also no significant change in the use of bivalirudin between phases II and III, indicating that the use of direct thrombin inhibitors was unlikely to be related to the observed differences in rates of clinical success with the StarClose device.
This retrospective exploration of more than 100,000 device deployments in over 460 U.S. healthcare institutions represents the largest analysis of medical implant device learning curve effects. Whereas successful deployment of the StarClose VCD was associated with previously reported clinical factors, including patient sex, body mass index, anticoagulation status, prior history of peripheral vascular disease, and deployment following a PCI procedure (vs. diagnostic-only procedure), our data suggest a strong association between clinical success and accumulating experience with the specific VCD. This learning effect was associated with increasing institutional clinical success, which improved from 93% to 97% over the study period. On average, the improvement in clinical success rate represented an approximate 1.5% increase in clinical success with each doubling of experience with the VCD. The rate of learning (slope of the learning curve) was influenced by diagnostic versus interventional procedure status, teaching status of the institution, and average institutional volume.
The significant differences in the learning curves seen in Figure 3, stratified by hospital volume, represents a paradox in that the trend does not follow the typical relationship between higher volume centers and lower adverse event rates (the “volume-outcome” relationship). In this analysis, it is the smaller centers that appear to achieve high steady-state procedural success rates more quickly, perhaps indicating a more rapid diffusion of learning among a smaller number of operators (a “shared learning” effect). Similarly, the difference in rates of learning between teaching and nonteaching centers may have been influenced by the presence of trainees in the larger teaching institutions, which could not be directly accounted for in this analysis. Because it is common practice in the United States to have the VCD deployment performed by a senior trainee (fellow), the presence of trainees and larger numbers of procedural assistants at teaching hospitals may well have contributed to “slower” observed learning as compared to smaller, nonteaching centers. Whereas the rates of improvement in clinical success were higher in interventional procedures, compared with diagnostic procedures, this finding is due to the very high rates of initial clinical success for diagnostic procedures using the StarClose VCD such that gains attributable to learning were more modest in diagnostic procedures.
For larger institutions, the learning curve for StarClose VCD appeared to be triphasic, with a period of declining success interposed between an initial and a final period of typical learning. This observed variation in learning curve may be due to changes in patient selection and expansion of number of operators during the initial adoption of the VCD at the institution, though only small differences in patient risks and characteristics were noted in this study.
Whereas this analysis demonstrates improvement in clinical success with StarClose, with accumulating experience, this finding does not appear to be driven by secular improvements in the successful deployment of VCD more generally. Within the same study period, the rates of successful other VCD deployment declined from 98.3% to 97.7% from 2005 to 2007, indicating that the improving clinical and deployment success in StarClose-treated patients was not reflective of overall improvement in VCD successful deployment rates.
The findings of the current analysis are consistent with the CLIP (Clip Closure in Percutaneous Procedures) trial, the initial randomized trial exploring the safety and efficacy of the StarClose VCD. The CLIP trial, which included 275 patients undergoing interventional coronary procedures (in addition to 208 in the diagnostic arm), demonstrated an overall device success rate of 86.8% in the interventional group and 94.1% in the diagnostic group, where device success was defined as attainment of final hemostasis within 5 min of device deployment and freedom from major vascular complication. These results appear to correlate very closely with the early outcomes of our current analysis of NCDR that demonstrated initial device success of 89.3% for the interventional patients compared with 95.1% for the diagnostic cases. These small differences may represent improvements in outcomes due to device refinement (comparing the commercially available StarClose platform to the initial platform studied during the CLIP trial) or potentially due to under-reporting of adverse events in the voluntary NCDR registry compared with the initial randomized clinical trial. In addition, the results of this analysis confirm the findings of earlier single-center studies of vascular closure that explored the learning effects associated with accumulating experience in using a specific VCD. Warren et al. (22) identified a sharp increase in procedural success after the first 50 deployments of the original Angio-Seal collagen plug VCD (St. Jude Medical, St. Paul, Minnesota), whereas Balzer et al. (5) reported significant decline in procedural failures using the Prostar suture-mediated closure device (Abbott Vascular) during peripheral interventional procedures after 100 cases.
Quantification of learning curve effects associated with medical device implant procedures is critical in developing expectations for “real world” results of such procedures outside of the context of pre-approval clinical trials. In addition, quantifying the learning curve of VCD and other implantable medical devices will allow a more informed assessment of the need for, and duration of, training and support as new technologies diffuse into routine clinical practice. It is important to note that learning and improvement of outcomes appears to continue even at high rates of average operator or institutional device-specific volume, which is consistent with learning curve theory. A better understanding of learning curve effects will also help inform objective evaluations of new medical device safety, as the influence of expected human learning might be separable from intrinsic device performance safety issues.
Several limitations, which limit the generalizability of the results to other medical devices or environments, are present in this study. The CathPCI registry did not have consistent documentation of individual operator-unique identifier information, thereby precluding learning curve analysis at the level of the operating physician. In addition, the registry does not collect information about which member of the clinical team actually deployed the device. The definition of VCD success is self-reported in the registry; however, we have previously shown that self-reported VCD success is closely associated with vascular complications rates following VCD deployment (16). Importantly, NCDR CathPCI data registry is a voluntary clinical quality and outcomes registry in which endpoints, such as VCD deployment failure, as well as vascular complication rates, are not universally audited or adjudicated.
Significant learning curve effects influence the safety and effectiveness of a recently introduced novel VCD in which between 75 and 130 deployments of the device, per attending physician, were required to achieve the highest levels of device deployment success and safety. The observed institutional learning curve is associated with institutional volume and teaching status and demonstrates a multiphasic pattern indicating potential variation in patients selected to undergo device deployment and the variable adoption of a new technology by providers within an institution.
Learning curve model
To quantify the magnitude of the learning curve effect, we modeled the learning curve as a cumulative success function represented as the simple log-linear equation (1):where k is the intercept constant representing predicted probability of failure for the first device implant, and b is the learning curve coefficient. As b increases, the learning curve flattens (becomes less steep) so that it takes longer to achieve an improvement in outcomes.
This study was funded, in part, by grants from National Library of Medicine (NIH R01-LM008142), the American College of Cardiology Foundation's National Cardiovascular Data Registry (NCDR), as well as by the Veteran's Administration Health Services Research and Development Service (CDP 09-387).
The views expressed in this manuscript represent those of the author(s), and do not necessarily represent the official views of the NCDR or its associated professional societies identified at https://www.ncdr.com/webncdr/DefaultCathPCI.aspx. CathPCI Registry is an initiative of the American College of Cardiology Foundation and The Society for Cardiovascular Angiography and Interventions. Dr. Resnic is a consultant for St. Jude Medical and Medtronic Corporation. Dr. Wang received research grants from Bristol-Myers Squibb/Sanofi Partnership, Merck (previously Schering-Plough), The Medicines Company, Eli Lilly/Daiichi Sankyo Partnership, Canyon Pharmaceuticals, Heartscape, AstraZeneca, and Medco. All other authors have reported that they have no relationships relevant to the contents of this paper to disclose.
- Abbreviations and Acronyms
- generalized estimating equation
- percutaneous coronary intervention
- vascular closure device(s)
- Received July 7, 2011.
- Revision received August 16, 2011.
- Accepted September 3, 2011.
- American College of Cardiology Foundation
- Thompson C.A.,
- Jayne J.E.,
- Robb J.F.,
- et al.
- Masonis J.,
- Thompson C.,
- Odum S.
- Hannan E.L.,
- Wu C.,
- Walford G.,
- et al.
- Bangalore S.,
- Arora N.,
- Resnic F.S.
- National Cardiovascular Data Registry
- Mehta S.K.,
- Frutkin A.D.,
- Lindsey J.B.,
- et al.,
- NCDR Registry
- Nikolsky E.,
- Mehran R.,
- Halkin A.,
- et al.