Eur J Cardiothorac Surg 2006;29:693-697
© 2006 Elsevier Science NL
Quality assurance in congenital heart surgery
Nicholas Kang
a
,
Victor T. Tsang
a
,
*
,
Steve Gallivan
b
,
Chris Sherlaw-Johnson
b
,
Timothy J. Cole
c
,
Martin J. Elliott
a
,
Marc R. de Leval
a
a Department of Cardiothoracic Surgery, Great Ormond Street Hospital for Children, London WC1N 3JH, UK
b Clinical Operational Research Unit, Department of Mathematics, University College London, UK
c Centre for Biostatistics and Epidemiology, Institute of Child Health, London, UK
Received 14 October 2005;
received in revised form 13 January 2006;
accepted 16 January 2006.
* Corresponding author. Tel.: +44 20 78138159; fax: +44 50 74321281. (Email: tsangv{at}gosh.nhs.uk).
 |
Abstract
|
|---|
Objective: The aim of this study was to develop a graphical method of risk-stratified outcome analysis in paediatric cardiac surgery to provide a means of continuous, prospective performance monitoring and allow real-time detection of change in outcomes. Methods: Risk-adjusted survival following open-heart surgery was prospectively measured over a 15-month period (n
= 460). Outcomes were charted using variable life-adjusted display (VLAD) charts, which indicate the cumulative difference in observed minus expected survival against the cumulative number of cases performed. Risk stratification was based on RACHS-1 (risk adjustment in congenital heart surgery) risk category and age at surgery, using our previously published risk model. The probability of deviation in performance from the expected baseline level was determined using a mathematical model. Results: By the end of the series, observed survival (443/460 = 96.3%) exceeded that predicted by the risk model (434.5/460 = 94.5%), equivalent to a one-third reduction in expected mortality. Mathematical modelling indicated a 1–5% likelihood that this difference would have occurred by random variation alone, suggesting the outcomes represented genuine improvement. Conclusions: VLAD charts provide an effective, easily visualised display of surgical performance and can be applied to paediatric cardiac surgery. Early detection of change, whether improvement or deterioration, is important for ongoing quality assurance within a cardiac surgery programme.
Key Words: Audit • Cardiac surgery • Performance monitoring • Surgical outcomes
|
1. Introduction
|
|---|
The traditional design of most surgical audits is the retrospective analysis of outcome data, with formal hypothesis testing to look for differences in outcomes of statistical significance. This is the method currently employed in the United Kingdom by the Central Cardiac Audit Database (CCAD), a national database funded by the Department of Health, which looks for differences in mortality rates amongst institutions performing paediatric cardiac surgery [1].
Whilst this customary auditing process is an appropriate method of confirming outlying performance, it is only when the discrepancy in results has become of sufficient magnitude that firm conclusions can be drawn. Yet it is critical to the process of quality assurance in any surgical programme that methods are in place for the detection of deteriorating performance at an early stage, so that appropriate measures can be taken to rectify deficiencies before results become truly outlying.
A dilemma arises in trying to meet these two competing aims: the need to avoid failure to recognise poor surgical performance versus a desire to avoid raising false alarms. In order to address this dilemma, we sought to develop a method of prospective risk-stratified performance monitoring to allow early detection of divergent performance, while at the same time being able to quantify the degree of certainty that changes in outcomes had occurred other than by chance variation.
|
2. Methods
|
|---|
We applied and combined methodology from several earlier research efforts to develop our current performance monitoring system. The first step was the formulation of a risk-stratification model for congenital heart surgery. The risk adjustment in congenital heart surgery (RACHS-1) system [2], proposed in 2002, was the starting point for the development of our own risk model. We modified and refined the RACHS-1 system to develop a logistic regression equation, which provides estimates of postoperative mortality for any given operation involving cardiopulmonary bypass, based on patient age and RACHS-1 risk category. Details of this risk model are given in our earlier report [3], and the regression equation is provided in Appendix A.
The second step was the development of a graphical display of performance. Surgical performance charts, also known as control charts, exist in various forms including cumulative sum charts (CUSUM), cumulative risk-adjusted mortality charts (CRAM), and sequential probability ratio test charts (SPRT) [4–7]. We developed a form of control chart monitoring known as variable life-adjusted display (VLAD) [8–10]. These charts plot the cumulative difference in observed minus expected survival against the cumulative number of operations. Observed survival for any individual operation is equal to either 1 or 0, depending on whether the individual patient survives or dies, respectively. Expected survival for any individual operation is calculated as (1 – expected mortality). For example, if expected mortality is 5%, then expected survival equals 0.95. The expected mortality for any individual operation in this study was that derived from the regression equation, as detailed in Appendix A.
Using this method, the VLAD chart rises when patients survive, but falls when patients die. The degree to which these rises and falls occur is proportional to the risk of each case. For example, if a patient with an estimated risk of 25% survives, the chart moves up by 0.25. But if the same patient dies, the plot moves down by 0.75. In contrast, if a patient with estimated risk of 1% survives, the chart moves up by only 0.01, but moves down by 0.99 if the patient dies. In this way, high-risk cases carry the greatest ‘reward’ in the event of success but the least ‘penalty’ in the event of failure, whereas low-risk cases carry the least reward for success but incur the greatest penalty for failure.
If, over a series of operations, there is no difference between observed and expected survival, the VLAD chart terminates level with the horizontal axis. The horizontal axis therefore represents a ‘baseline’ (expected) level of performance. Performance, which deviates from this baseline, may simply reflect random variation (chance), or more important divergence (good or bad performance). In order to distinguish between these possibilities, we developed a mathematical model to display performance in terms of probability of divergence from baseline. Boundary lines were mapped onto the VLAD chart to give an indication of the level of certainty that any divergence from the expected performance had occurred due to factors other than chance alone. These boundaries were displayed graphically using coloured zones, which were constructed by plotting the distribution of probabilities of all observed variation in outcome being due to chance. Details of the mathematical models used to construct these boundaries are given in an earlier report from our group [10].
Secondary analysis of possible causes for the observed change in performance was achieved by comparing observed and expected mortality rates in subgroups of patients using a chi-square test. Subgroups were studied by age (neonate, infant, child) and RACHS-1 categories (‘low-risk’ 1–3 and ‘high-risk’ 4–6).
|
3. Results
|
|---|
Fig. 1
displays the VLAD performance chart using a 3-year dataset of 1085 operations. These cases comprised the original dataset used to formulate our risk model. The plot is seen to oscillate about the horizontal axis and finishes very close to zero. This indicates that the difference between observed and expected mortality over the entire series was very small, implying the risk model was well calibrated. The plot stay mostly within the green zone, which represents the middle 50% distribution of probabilities that divergence of this magnitude was due to chance alone. Some divergence into the yellow zone is observed in the first half of the series, indicating that poorer outcomes at this time were less likely to be due to chance, although there was still a 10–25% probability of this being random variation.
View larger version (32K):
[in this window]
[in a new window]
|
Fig. 1. VLAD chart for open-heart operations from April 2000 to March 2003. Note that the plot terminates dose to zero, indicating good calibration of the risk model which was formulated from these data. Coloured areas represent the distribution of probabilities that all outcomes could have occurred by chance variation.
|
|
Fig. 2
displays more recent data for the next 460 open-heart (cardiopulmonary bypass) operations over a 15-month period. No patient was excluded from the analysis. Of note, however, 72 patients (16%) had procedures not classified by the RACHS-1 system. Using our risk model, these miscellaneous cases are coded along with RACHS-1 risk category three cases, in accordance with our previously published method [2]. Initially the performance remains similar to baseline, but over the last 150 cases the outcomes improve more noticeably and by the end of the series, the plot has entered the dark blue zone. Since there is only a 1–5% probability that this degree of divergence could have occurred by chance, there is a correspondingly greater certainty that this represents a genuine improvement in results.
View larger version (28K):
[in this window]
[in a new window]
|
Fig. 2. VLAD chart for open-heart operations from March 2003 to May 2004. Risk-adjusted outcomes remain steady initially but then show an improvement towards the end of the series. Coloured areas represent the distribution of probabilities that all outcomes could have occurred by chance variation.
|
|
Further analysis revealed improved survival in neonates and high-risk category cases, although this did not reach statistical significance. Neonatal open-heart mortality fell from 16.5% in the initial 3-year period to 13.4% in the more recent 15-month study period. Mortality fell in high risk-category cases (RACHS-1 categories 4, 5 and 6) from 18.5% to 11.8%. Note that these two subgroups (neonates and risk categories 4–6) are not mutually exclusive, as many neonates undergo operations that fall into the higher RACHS-1 risk categories. No individual subgroup of patients or operations could be identified as displaying improved survival to a level of statistical significance.
|
4. Discussion
|
|---|
The term ‘quality control’ was coined in the 1930s, when manufacturing industries started examining statistical methods by which productivity and outcomes could be measured and standards established. Juran and Shewhart, working at the Western Electric Company, a manufacturer of telephony hardware, pioneered the principles by which quality could be measured and improved [11]. Shewhart developed so-called ‘statistical control charts’, which were able to indicate when a process (such as the proportion of manufacturing defects on a production line) had deviated from expected standards [12]. In the 21st century, the healthcare industry has embraced another similar catchphrase termed ‘clinical governance’. This has been defined as ‘a system through which organisations are accountable for continuously improving the quality of their services and safeguarding high standards of care’ [13].
The performance monitoring technique (VLAD) described in this paper is an attempt to address several key issues pertinent to the concept of quality assurance. Firstly, a standard of care is defined. This is the baseline mortality rate, derived from a risk-stratification model. Secondly, outcomes are tracked over time so that the trends and direction in performance can be monitored. Thirdly, every outcome is risk-adjusted so that variation in casemix and complexity are accounted for. Finally, an indication is provided about the probability that fluctuations in performance could be due to chance alone.
The traditional paradigm of surgical outcome monitoring employs a retrospective approach to compare outcomes using rigid statistical testing. VLAD charts offer the following advantages over traditional auditing methods:
- 1. Results can be followed prospectively, allowing for ‘real-time’ examination of outcomes.
- 2. Trends in performance over time are readily visualised using the graphical display of data. Evidence of deteriorating performance can be used as a ‘trigger’ to modify practice.
- 3. Changes in performance can be detected at an early stage. Formal statistical tests to look for differences in outcomes can only identify poor performance at a late stage, when the discrepancy in results has reached a sufficient magnitude to be labelled ‘statistically significant’. The aim of quality assurance should be to avert this ‘end-stage’ effect.
At the same time, use of control charts such as VLAD has a number of important limitations:
- 1. The accuracy of performance measurement in VLAD is dependent on the accuracy of the risk-stratification model used. All risk-models are imperfect and therefore assumptions and inaccuracies made in risk stratification are extrapolated to VLAD.
- 2. Although the probability distributions are calculated using exact methods rather than approximation, no correction is made to account for sequential testing. This means, for example, that the probability of crossing into the lower 5% tail (red zone) over a series will be greater than 5%.
- 3. Half of all doctors are below average [14]. If a number of providers are compared on the same VLAD chart, half can be expected to have ‘below-average’ performance. (Strictly speaking, half of all doctors are below the median.) This does not mean that performance was ‘below-acceptable’.
Identifying ‘below-acceptable’ performance mandates a higher burden of proof that outcomes are significantly worse than expected. This requires formal statistical testing to look for outlying results using arbitrarily chosen confidence limits or significance levels (p-values), and must correct for multiple comparisons and sequential testing. For example, the CCAD in the United Kingdom used 99% confidence limits to decide whether individual institutions’ mortality rates were significantly different from the overall averaged mortality across all institutions combined [1]. However, a criticism of this analysis was that the mortality was not risk-adjusted, and therefore may not have adequately accounted for variations in casemix [15].
Our own performance analysis indicated that a substantial improvement in outcomes was occurring in the most recent period (last 150 cases). Although we were unable to identify any specific subgroup of patients where the improved results reached statistical significance, we observed that mortality in high-risk cases (RACHS-1 groups 4–6) and neonates fell over this time period. One factor in this improved neonatal mortality was an institutional change in management of hypoplastic left heart syndrome. Members from our unit visited a centre in the United States with a reputation for excellent results in the Norwood procedure. As a result of this visit, many small but important changes were made in our practice involving a wide variety of disciplines including surgical, anaesthetic, nursing and intensive care. Mortality for stage 1 Norwood procedures has since fallen from 35% to 13%, although once again this does not reach statistical significance as overall numbers are relatively small. Nevertheless for the entire series, a one-third reduction in expected mortality was observed (3.7% vs 5.5%), which is reflected in the positive trend on the VLAD chart in Fig. 2.
The VLAD chart has acted as positive reinforcement that the changes made in hypoplastic left heart syndrome at our institution have been effective and are helping to steer our performance in the right direction, not only for this specific disease entity but probably in the management of neonates and other complex cases generally.
|
5. Limitations
|
|---|
This study has only examined the single short-term outcome measure of in-hospital survival. It is increasingly recognised that this is only the ‘tip of the iceberg’ in monitoring quality [16], albeit a rather sharp tip. In-hospital survival is perhaps a reflection of ‘surgical safety’, and as such, is a critically important component to quality assurance. The CCAD has made an important leap forwards by tracking 1-year results [1], which adds meaningful information about what the real outcome of surgery has been. Other outcome measures (e.g. neurological outcomes) may be equally, or indeed, more relevant. However, such outcomes are much more difficult to quantify and therefore have less inter-observer reliability as a tool for monitoring surgical performance.
|
6. Conclusions
|
|---|
VLAD charts provide an effective, easily interpreted graphical display of surgical performance. Alarm lines can be constructed mathematically to signal more significant changes in outcomes. Early detection of change, whether improvement or deterioration, is important for ongoing quality assurance within a cardiac surgery programme.
|
Appendix A
|
|---|
Regression equation for predicted risk of surgery in patients less than 18 years of age using cardiopulmonary bypass [2].
where log odds risk = –5.378 + [0.6359 x RACHS] + [3.115 x 1/
(age in days + 1)]
Note: miscellaneous cases not defined by the RACHS-1 classification (e.g. cardiac transplantation) are coded along with RACHS-1 risk category 3 in the logistic regression model (see reference [3]).
|
Acknowledgments
|
|---|
Research at the Institute of Child Health and Great Ormond Street Hospital for Children NHS Trust benefits from Research and Development funding received from the NHS Executive.
|
Footnotes
|
|---|
Presented at the joint 19th Annual Meeting of the European Association for Cardio-thoracic Surgery and the 13th Annual Meeting of the European Society of Thoracic Surgeons, Barcelona, Spain, September 25–28, 2005.
|
References
|
|---|
- Gibbs JL, Monro JL, Cunningham D, Rickards A. Survival after surgery or therapeutic catheterisation for congenital heart disease in children in the United Kingdom: analysis of the central cardiac audit database for, 2000–1. Br Med J 2004;328:611.[Abstract/Free Full Text]
- Jenkins KJ, Gauvreau K, Newburger JW, Spray TL, Moller JH, Iezzoni LI. Consensus-based method for risk adjustment for surgery for congenital heart disease. J Thorac Cardiovasc Surg 2002;123(1):110-118.[Abstract/Free Full Text]
- Kang N, Cole TJ, Tsang VT, Elliott MJ, de Leval MR. Risk stratification in paediatric open-heart surgery. Eur J Cardiothorac Surg 2004;26:3-11.[Abstract/Free Full Text]
- de Leval MR, François K, Bull C, Brawn W, Spiegelhalter D. Analysis of a cluster of surgical failures. Application to a series of neonatal arterial switch operations. J Thorac Cardiovasc Surg 1994;107(3):914-923.[Abstract/Free Full Text]
- Rogers CA, Reeves BC, Caputo M, Ganesh JS, Bonser RS, Gianni D, Angelini GD. Control chart methods for monitoring cardiac surgical performance and their interpretation. J Thorac Cardiovasc Surg 2004;128:811-819.[Free Full Text]
- Spiegelhalter D, Grigg O, Kinsman R, Treasure T. Risk-adjusted sequential probability ratio tests: applications to Bristol. Shipman and adult cardiac surgery. Int J Qual Health Care 2003;15(1):7-13.[Abstract/Free Full Text]
- Sismanidis C, Bland M, Poloniecki J. Properties of the cumulative risk-adjusted mortality (CRAM) chart, including the number of deaths before a doubling of the death rate is detected. Med Decis Making 2003;23(3):242-251.[Abstract]
- Lovegrove J, Valencia O, Treasure T, Sherlaw-Johnson C, Gallivan S. Monitoring the results of cardiac surgery by variable life-adjusted display. Lancet 1997;350(9085):1128-1130.[CrossRef][Medline]
- Gallivan S, Davis K, Stark J. Early identification of divergent performance in congenital cardiac surgery. Eur J Cardiothorac Surgery 2001;20:1214-1219.
- Sherlaw-Johnson C, Gallivan S, Treasure T, Nashef SA. Computer tools to assist the monitoring of outcomes in surgery. Eur J Cardiothorac Surg 2004;26(5):1032-1036.[Abstract/Free Full Text]
- Juran JM. Management of inspection and quality control. New York: Harper and Brothers; 1945.
- Shewhart W. Economic control of quality in manufactured product. New York: D. VanNostrand Company, Inc; 1931.
- Scally G, Donaldson LJ. Clinical governance and the drive for quality improvement in the new NHS in England. Br Med J 1998;317:61-65.[Free Full Text]
- Poloniecki J. Half of all doctors are below average. Br Med J 1998;316:1734-1736.[Free Full Text]
- Treasure T. Congenital heart disease. Br Med J 2004;328:594-595.[Free Full Text]
- Keogh BE. Cardiac surgical mortality: the tip of the quality assurance iceberg. Heart 2000;84:7-8.[Free Full Text]
This article has been cited by other articles:
|
|
|
|
 
D. M. Holzhey, S. Jacobs, T. Walther, M. Mochalski, F. W. Mohr, and V. Falk
Cumulative sum failure analysis for eight surgeons performing minimally invasive direct coronary artery bypass
J. Thorac. Cardiovasc. Surg.,
September 1, 2007;
134(3):
663 - 669.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
P. Sergeant
Editorial comment: Online performance monitoring in cardiac surgery
Eur. J. Cardiothorac. Surg.,
May 1, 2006;
29(5):
697 - 698.
[Full Text]
[PDF]
|
|
|
Top
Abstract
1. Introduction
