|
|
||||||||
Eur J Cardiothorac Surg 2006;29:56-62
© 2006 Elsevier Science NL
a National Centre of Epidemiology, Surveillance and Health Promotion, Istituto Superiore di Sanità, Via Giano della Bella, 34, I-00161 Rome, Italy
b Department of Epidemiology, ASL RME, Rome, Italy
c Ministry of Health, Rome, Italy
Received 4 May 2005; received in revised form 28 July 2005; accepted 28 July 2005.
* Corresponding author. Tel.: +39 06 49904234; fax: +39 06 49904230. (Email: fulvias{at}iss.it).
| Abstract |
|---|
|
|
|---|
Key Words: Coronary artery bypass graft Outcome Risk-adjustment Mortality
| 1. Introduction |
|---|
|
|
|---|
Even though limitations of risk-adjustment methods are known [1,12,13], comparative data, especially if adjusted using a risk function derived empirically from the observed population, serve many purposes and have the potential to provide insight and lead to quality improvement.
In Italy, there is no surveillance system aimed to regularly assess the outcomes of hospital care. A few isolated initiatives were made during the last decade, but they were at the regional level and involved only a limited number of hospitals [1416].
In early 2002, the Italian Ministry of Health, in conjunction with the National Institute of Health (ISS) launched the Italian CABG Projecta prospective study on short-term outcomes in patients who had CABG surgery, with the voluntary participation of the Italian Cardiac Surgery Centres. This study was the first opportunity to evaluate and publish performance indicators at the national level. It aimed at providing comparable data on observed and expected mortality 30 days after CABG intervention in each cardiac surgery centre, adjusting for pre-operative patients risk [17,18].
This study could also give the Italian scientific community the opportunity to build and estimate a national risk function for the outcome considered, compare it to functions obtained in other countries, and use it to assess the individual pre-operative risk.
| 2. Materials and methods |
|---|
|
|
|---|
Out of the 89 adult cardiac surgery centres, 82 agreed to participate in this prospective, on-going study. Patients considered were 1599 years old, and underwent an isolated CABG surgery (not associated with other cardiac or extra-cardiac procedures) after 1st January 2002 in one of the participating centres.
2.1 Data collection
Data collection is being carried out with standardized on-line data entry on a password protected website http://bpac.iss.it/.
This analysis concerns all isolated CABG interventions performed between 1st January 2002 and 30th September 2004.
2.2 List of variables and definitions
The scientific references about the choice of individual variables to be collected derive from a series of more extensive research protocols developed by the major international and national scientific societies (Society of Thoracic Surgery, American Association for Thoracic Surgery, European Association for Cardiothoracic Surgery, SICCH) and from other risk stratification protocols in cardiac surgery developed over the last 10 years (Parsonett 1° and 2°, EuroSCORE, New York et al.) [5,1922].
To define pre-operative patient risk for this study we gathered data on demographic characteristics (gender, age, residence, and place of birth) and on patient's health status prior to surgery. The latter included the presence of co-morbidities such as diabetes under treatment, malignant ventricular arrhythmia, cirrhosis, chronic obstructive pulmonary disease, renal failure (dialysis or pre-operative creatinine >2 mg/dl), neurological dysfunction, active endocarditis, pulmonary hypertension, cancer, extra-cardiac arteriopathy and stroke; patient's haemodynamic condition (unstable or shock); degree of ventricular dysfunction (ejection fraction, EF, <30%; between 30 and 49%;
50%); previous surgery that opened the pericardium (CABG or any other interventions); unstable angina and recent infarction (<90 days). Information on the type of intervention (CABG isolated intervention, associated with other cardiac or extra-cardiac procedures, elective or emergency, in on-pump or off-pump circulation) was also collected.
In case of death within 30 days of the intervention, date and specific cause were recorded.
2.3 Data quality assessment
2.3.1 Clinical monitoring
For each participating cardiac surgery centre, a sample of records was randomly selected from the electronic archive. Independent observers, following specific standardized operating procedures (SOPs), visited each centre and compared contents of the records transmitted to those reported in the original clinical charts. These procedures allowed us to assess the reliability and completeness of the database and to maintain constant quality control.
2.3.2 Other controls
The completeness of data in each centre was evaluated by comparing them to hospital discharge records supplied by regional and national health information systems.
2.4 Inclusion criteria for the analytical database
Cardiac surgery centres with fewer than 100 CABG isolated interventions per year were excluded from the analytical database. Moreover, only centres that provided data continuously for at least 6 months and who had fewer than 5% of patients lost to follow-up were considered to calculate the algorithm required for the risk-adjustment procedure.
2.5 End-point and follow-up
Each cardiac surgery centre was requested to carry out an active follow-up in order to define patient's life status 30 days after CABG intervention. Then, for patients classified as lost to follow-up by the centres, life status was confirmed by consulting the local death registries.
2.6 Data analysis
Almost all risk factors were recorded as dummy variables (having or not having the risk factor); age was used as a continuous variable; ejection fraction was subdivided into three classes: <30, 3050, and 50% or more, the latter being the reference group. For this variable a missing value category was used.
Univariate analyses were performed on all candidate predictive variables in order to determine significant associations with the outcome.
In order to account for joint confounding the best predictive model was developed using a multiple logistic regression analysis. First, all possible confounding variables were included in the model. Second, in order to identify independent associations with the outcome, a backward stepwise method was used (exclusion probability=0.20; inclusion probability=0.10). A set of interaction hypotheses defined a priori was also tested.
To avoid overfitting, a cross-validation procedure was applied. To this extent, patients were randomly split into two equal-size samples: sample I was used to build the predictive model (n=17,231); sample II was used as an independent database for model validation (n=17,079). The entire data set was finally used in estimating the definitive coefficients and calculate their P-values to provide more precise parameter estimates.
The HosmerLemeshow
2-test was applied to assess the calibration (accuracy) of the risk function obtained. To evaluate the model's discriminative ability to predict individual deaths, the area under the receiver operating curves (ROCs) was measured. As usually reported in scientific literature, ROC values higher than 0.700.75 were judged as proof of a good discriminative ability.
By applying the best predictive algorithm back to each centre's data set, we estimated the expected number of deaths of that centre. The risk-adjusted mortality rate (RAMR) is then calculated by dividing, for each centre, the observed number of deaths by the expected number of deaths, and by multiplying this ratio by the average mortality rate of the whole sample. This indicator is the best estimate of what the whole population's mortality rate would be if the population had the same observed/expected death ratio of that centre.
To test heterogeneity we used a significance threshold of P=0.05. A RAMR significantly lower than the average mortality rate indicates that the health care provider performance is better than the average of the whole sample (low-outlier); on the contrary, a RAMR significantly higher shows a worse performance (high-outlier).
In order to analyse the effect of the loss to follow-up on risk estimate in each centre, we performed a sensitivity analysis.
All statistical procedures were performed by STATA 8.1 statistical package.
| 3. Results |
|---|
|
|
|---|
|
|
3.1 Patients characteristics and association with 30 days mortality
A detailed description of the population characteristics and univariate analysis of the association between each pre-operative variable and the 30 days mortality rate is reported in Table 1 .
|
The overall crude mortality rate in the 64 centres was 2.61%, ranging from 0.33 to 7.63%. When split samples were drawn, crude mortality rates were 2.62% in sample I and 2.59% in sample II.
Most risk factors considered were significantly associated with the outcome at P<0.001. Neurological dysfunction and other cardiac interventions that opened the pericardium were significant at 0.05; only endocarditis and on/off-pump circulation were not significantly associated with 30 days mortality (P=0.184 and 0.835, respectively). Emergency surgery was performed in 3.8% of patients and showed an OR of 7.22 (P<0.0001). The excess of risk due to shock was 14.44 (P<0.0001), while dialysis had an OR of 6.66 (P<0.0001).
The multivariate predictive model, derived from logistic regression analysis, and the independent contribution of each variable to the outcome is presented in Table 2 . Coefficients, P-values for significant risk factors and odds ratios are reported.
|
To cross validate the model, the estimated coefficients from sample I were applied to both sample I and sample II. The area under the ROC curve in the validation sample was 0.80 and the HosmerLemeshow test showed a
2=8.87 (P=0.354).
Risk-adjusted mortality rates for each centre and their 95% confidence intervals are reported in Fig. 3 . Eight centres presented RAMRs significantly better than the national mean (2.61%) (low-outliers) while seven Centres exhibited significantly worse performances (high-outliers). There were no statistically significant differences between the RAMR of the other centres and the mortality rate of the overall population.
|
| 4. Discussion |
|---|
|
|
|---|
Because the risk-adjustment function was derived from around 34,000 CABG interventions carried out in 64 cardiac surgery centres uniformly distributed throughout the country, the algorithm constructed to standardise the comparison between hospital performances should be sufficiently stable.
In this Italian population, mortality within 30 days after CABG surgery is 2.61%. This result is very similar to those reported in other international settings, and confirms the satisfactory average level of the Italian cardiac surgery performances [59].
It must be stressed that such a systematic monitoring of hospital outcomes, as well as the risk-adjusted profiling of the CABG surgery providers at national level, represents a real change for the Italian NHS. The overall mortality rate observed in this study is acceptable mainly because it has not yet been influenced by the registry effect, a well-documented phenomenon in which improvements in performance [10,23] are noted after the establishment of a surveillance system.
Risk adjusted mortality estimates for most of the hospitals analysed are not different from the country average. Fifty-seven of the hospitals that participated in the study performed as or better than expected, meaning their RAMRs were not statistically higher than the national mean. Only seven (about 10%) performed worse than expected, with RAMRs significantly higher than the national mean.
When we compared centres specific crude and risk adjusted mortality rates, we found small differences in most cases, and only few isolated cases suggest a case-mix of patients markedly different from the whole study population. In fact, 10 centres were classified as low-outliers using unadjusted mortality rates and eight of them were confirmed as low-outliers after the adjustment; seven centres were classified as high-outliers using unadjusted mortality rates, two of them had their status not confirmed in the adjusted analysis, while two other centres emerged as significantly worse than the reference mean when using RAMR. As already observed, these results show a limited impact of risk-adjustment when comparing 30 days mortality after CABG, given the limited confounding effect of risk factors when comparing centers under study [24]. In other words, the proportion of heterogeneity of mortality rates attributable to heterogeneity of distribution of risk factors between centers seems to be small.
Obviously, the classification of Centres as outliers heavily depends on the chosen threshold for statistical significance. A less conservative approach, using P<0.1, would have implied three more high-outliers and three more low-outliers. Widely speaking, the choice of the most appropriate threshold is not only a statistical exercise and should be based on an explicit assessment of cost and benefits associated with true and false positive and negative results of the comparison.
The Italian CABG Project succeeded in collecting, prospectively, specific clinical data and in building and estimating an empirically derived Italian risk-adjustment function. In general, the meaning of risk-adjustment is to control confounding. We underline that any risk-adjustment function to be utilized for the purpose of comparison between centers or populations must be time and population specific. This statement implies a substantial difference between predictive models aimed to predict the occurrence of outcome, and explicative models aimed to control confounding in comparison of occurrence of outcome between centers or population. Even a very valid risk score having a strong association with the concerned outcome could be irrelevant for the purpose of risk-adjustment if the distribution of its values is homogeneous between centers.
Another innovative value of the Italian CABG Project is that it is the first attempt to produce outcome indicators at the national level, to profile hospitals with respect to their performances, and to publicize a report card where providers can see their own results and compare them to the Italian average.
The well known adjusted in-hospital mortality is a suitable indicator of the quality of assistance for many conditions, but bias can be introduced when the strategy of patients' post-operative management is different. The 30 days mortality rate allows for taking into account these differences and has been widely used as a valid instrument to evaluate quality of surgical centres [11,12].
There is plenty of evidence that performance information is vital to help clinicians and hospital managers understanding where quality of care problems may exist and to target improvement efforts. Throughout the Italian CABG Project observation, overall risk-adjusted patient mortality has dropped from 2.8% in 2002 to 2.4% in 2003/2004. This result cannot be underestimated and some reflections need to be made. RAMR observed in the eight centres classified as low-outliers, being around 0.9%, could represent the best performance obtainable in our Italian population. Considering that adjusted mortality rate in 2002 was 2.8% and the best effective reduction to be expected is around 1.9%, the observed drop of 0.4% (although not statistically significant) corresponds to 21% of the best reduction effectively obtainable. To conclude, this finding should encourage hospitals and surgeons to persevere in this approach, since a valid system of quality assessment and, consequently, the hospitals critical appraisal of their own results can contribute to the improvement of performances in cardiac surgery.
The future development of a national program of profiling health care providers must be carefully designed and conducted to maximize the potential benefits and minimize the possible negative effects, for both the population and the health system, potentially associated with the publication of comparisons among providers based on the outcomes [13].
| Appendix |
|---|
|
|
|---|
Study design, analysis and coordination
Scientific Committee
Representatives of participating Cardiac Surgery Centres
Other technical collaborations
| Acknowledgments |
|---|
The study was carried out with the collaboration of the Italian Federation of Medical-Scientific Societies (FISM), the Italian Society for Cardiac Surgery (SICCH) and Regional Authorities.
A special mention goes to Joseph S Gonnella (PhD, MD), Director of the Centre for Research in Medical Education and Health Care of the Jefferson Medical College of Philadelphia, PA, USA, consultant of this Study, for his precious advices and contributions to the preparation of the manuscript. The authors thank Margaret Becker for her editorial help.
| Footnotes |
|---|
| References |
|---|
|
|
|---|
This article has been cited by other articles:
![]() |
L. A. Menicanti The surgeon, the statistics and the data Eur. J. Cardiothorac. Surg., March 1, 2008; 33(3): 323 - 324. [Full Text] [PDF] |
||||
![]() |
P. D'Errigo, F. Seccareccia, S. Rosato, V. Manno, G. Badoni, D. Fusco, C. A. Perucci, and the Research Group of the Italian CABG Outcome Pro Comparison between an empirically derived model and the EuroSCORE system in the evaluation of hospital performance: the example of the Italian CABG Outcome Project Eur. J. Cardiothorac. Surg., March 1, 2008; 33(3): 325 - 333. [Abstract] [Full Text] [PDF] |
||||
![]() |
N. Echahidi, P. Pibarot, J.-P. Despres, J.-M. Daigle, D. Mohty, P. Voisine, R. Baillot, and P. Mathieu Metabolic Syndrome Increases Operative Mortality in Patients Undergoing Coronary Artery Bypass Grafting Surgery J. Am. Coll. Cardiol., August 28, 2007; 50(9): 843 - 851. [Abstract] [Full Text] [PDF] |
||||
![]() |
P. E. Antunes, L. Eugenio, J. Ferrao de Oliveira, and M. J. Antunes Mortality risk prediction in coronary surgery: a locally developed model outperforms external risk models Interactive CardioVascular and Thoracic Surgery, August 1, 2007; 6(4): 437 - 441. [Abstract] [Full Text] [PDF] |
||||
![]() |
G. G.L. Biondi-Zoccai, M. Fusaro, and L. Inglese Potentials and pitfalls of clinical outcome research studies in cardiac surgery. Eur. J. Cardiothorac. Surg., May 1, 2006; 29(5): 855 - 856. [Full Text] [PDF] |
||||
![]() |
F. Seccareccia, C. A. Perucci, D. Fusco, and P. D'Errigo Reply to biondi-zoccai et Al. Eur. J. Cardiothorac. Surg., May 1, 2006; 29(5): 856 - 856. [Full Text] [PDF] |
||||
![]() |
F. Seccareccia, C. A. Perucci, D. Fusco, and P. D'Errigo Reply to hekmat et Al. Eur. J. Cardiothorac. Surg., May 1, 2006; 29(5): 857 - 858. [Full Text] [PDF] |
||||
![]() |
K. Hekmat, U. Mehlhorn, and T. Wahlers Do we need the new Italian risk stratification model for CABG patients? Eur. J. Cardiothorac. Surg., May 1, 2006; 29(5): 856 - 857. [Full Text] [PDF] |
||||
![]() |
F. Seccareccia, C. A. Perucci, P. D'Errigo, and D. Fusco Concerning the Editorial comment by Dr Menicanti. Eur. J. Cardiothorac. Surg., May 1, 2006; 29(5): 858 - 859. [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| ANN THORAC SURG | ASIAN CARDIOVASC THORAC ANN | EUR J CARDIOTHORAC SURG |
| J THORAC CARDIOVASC SURG | ICVTS | ALL CTSNet JOURNALS |