HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Author home page(s): Ani C. Anyanwu Tom Treasure Permission Requests Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Anyanwu, A. C. Articles by Treasure, T. Search for Related Content PubMed PubMed Citation Articles by Anyanwu, A. C. Articles by Treasure, T. Related Collections Cardiac - other Professional affairs

Eur J Cardiothorac Surg 2004;25:299-303
© 2004 Elsevier Science NL

Surgical research revisited: clinical trials in the cardiothoracic surgical literature

^a Cardiothoracic Unit, Harefield Hospital, London, UK
^b Cardiothoracic Unit, Guy's Hospital, London SE1 9RT, UK

Received 23 July 2003; received in revised form 5 November 2003; accepted 2 December 2003.

^* Corresponding author. Tel.: +44-20-8725-3288
e-mail: tom.treasure{at}ukgateway.net

	Abstract

Top Abstract 1. Introduction 2. Materials and methods 3. Results 4. Discussion Appendix A. Conference... References

 
Objective: Surgeons have been criticised for not undertaking high-quality research. This study examines the quality of randomised controlled trials (RCTs) in the cardiothoracic surgical literature with a view to revisiting the role of RCTs in surgical research. Methods: All RCTs published in 1998 and 1999 in three major international cardiothoracic journals were analysed for basic components of trial design and presentation. Results: We found 119 papers presented as RCTs. The median size of treatment arms per RCT was 20 (interquartile range 14–40). Of 20 dimensions of trial quality examined, 12 or more were deficient (not described or performed inadequately) in half of the studies. Key information was frequently missing. Additionally, most trials relied on ‘soft’ endpoints, using surrogate (72, 61%) rather than clinical, and numerical (102, 86%) rather than categorical outcomes suggesting lack of power to detect clinically relevant differences. Although most trials reported positive results (73, 63%), only in 18 (15%) did authors make recommendations for practice change. Conclusions: Many RCTs in surgery by virtue of their design, sample size, and insufficient power are incapable of answering the questions researchers seek to address. Surgical trials often may not exclude bias because of lack of blinding and variations in surgical technique and performance. It is arguable that for most study questions in clinical surgery, comparative analysis of large case series and databases will provide more robust evidence.

Key Words: Randomised controlled trial • Surgical research • Surgical journals

	1. Introduction

Top Abstract 1. Introduction 2. Materials and methods 3. Results 4. Discussion Appendix A. Conference... References

 
It is hard to believe that it is 7 years since the editor of The Lancet caused such a stir amongst surgical researchers by likening our efforts to comic opera [1]. The formulation and publication of the CONSORT guidelines followed in 1996 [2–4] and this was introduced in a combative style with the headline ‘Lies, damn lies, and reports of RCTs’ [5]. The target for improvement was now clearly wider than surgery where randomised controlled trials (RCTs) are indeed lamentably few. Magos and colleagues later suggested that the general medical journals were biased against publication of surgical papers [6]. The editor of the British Medical Journal in reply argued that this was a reflection of the poor quality of surgical research and the domination of case series over RCTs in surgical research [7]. It was ironic that this followed on a series of publications extolling the virtues of observational research [8], non-randomised studies [9] and the use of databases rather than prospective studies [10]. RCTs have not been a frequent component of surgical research; but is this because surgeons are poorer researchers or does it reflect particular difficulties in randomising operations? Should the RCT be regarded as the sine qua non when assessing surgical evidence? Over the past 25 years, cardiac surgery has been one of the most flourishing and expanding areas of clinical practice. The objective of this study is to discover the quality of RCTs in cardiothoracic surgery with a view to revisiting the role of randomised trials in surgical research.

Our study is an honest reflection from the inside rather than high-handed criticism from outsiders. Criticism has often come from those who are neither surgeons nor researchers and do not have to tackle the obstacles, ethical and logistic that make clinical surgical research so challenging. We are conscious that we ourselves, as surgical researchers in this era, have performed and published studies which might now be robustly criticised.

	2. Materials and methods

Top Abstract 1. Introduction 2. Materials and methods 3. Results 4. Discussion Appendix A. Conference... References

 
We hand-searched all issues of the three principal international specialist cardiothoracic journals, Annals of Thoracic Surgery, European Journal of Cardio-thoracic Surgery and The Journal of Thoracic and Cardiovascular Surgery for 1998 and 1999 to identify all randomised trials. The present study based on the 1998 and 1999 literature built on a previous exercise in which one of us had looked at cardiothoracic papers published in 1978 to see how they had stood the test of time over 20 years [11]. The inclusion criterion was any paper described as a randomised trial by the authors. Each report was tested against the revised CONSORT statement [12] for trial size, patient selection, reliability of randomisation and blinding, descriptions of withdrawals and choice of endpoints. We assessed the overall quality of trial reports with on a 20-point scale based on whether the investigators had adequately reported and satisfactorily undertaken the following CONSORT criteria

Study objective, inclusion/exclusion criteria, year of study, details and timing of interventions, definition of a primary clinical outcome, sample size based on power calculation, generation of randomisation, implementation of randomisation, allocation concealment, randomisation independent of implementer and assessor, blinding (patients), blinding (implementers), blinding (assessors), duration of recruitment, duration of follow-up, proportion of eligible patients recruited, description of dropouts, baseline group data, crossovers, complications and adverse effects.

We compared proportions using the ² test.

	3. Results

Top Abstract 1. Introduction 2. Materials and methods 3. Results 4. Discussion Appendix A. Conference... References

 
We identified 119 trials; 105 in adult cardiac surgery, 8 in congenital surgery, and 6 in thoracic surgery. There were 66 from Europe and 38 from North America, the remainder being from other parts of the world. The median trial size was 50 patients (range 11–886) with 75% randomising fewer than 100 patients. Fifty-four trials (45%) had fewer than 20 patients per treatment arm. Only 10 studies (8%) reported that the chosen sample size had been based on a power calculation. Inclusion and exclusion criteria were not given in 30 (25%) trials while baseline demographic and clinical data were not provided in 13 (11%). Most reports (81, 68%) did not state the year in which the study was undertaken in trials and two (2%) trials did not define the interventions clearly. The proportion of eligible patients that had been recruited was given in only eight (7%) studies; this proportion ranged from 40 to 100% in these eight studies.

Majority of trials (102, 86%) used quantitative surrogate measures, which generate numerical data for analysis, rather than categorical endpoints of clinical relevance. In only 24 (20%) trials was the primary outcome a directly measured clinical event. This was most evident in trials with sample size less than 50 where 4/65 (6%) used categorical variables as an endpoint compared to 13/54 (24%) in studies with 50 or more subjects (P=0.005). Although 74 trials reported positive results, only 18 recommended practice change.

Trials were scored from 0 (worst) to 20 (best) based on the number of CONSORT criteria that were fulfilled (Fig. 1) . The median score was 8 (range 3–19). As far as randomisation and blinding were concerned, in the majority of instances there was simply not enough information in the studies to decide if these vital components of study design were adequate (Table 1).

View larger version (32K): [in this window] [in a new window]   Fig. 1. Frequency distribution by quality score based on number of CONSORT criteria that were adequately described (and fulfilled). In 70% of trials, less than half of the 20 CONSORT criteria had been met.

	4. Discussion

Top Abstract 1. Introduction 2. Materials and methods 3. Results 4. Discussion Appendix A. Conference... References

The emphasis on randomised trials as the superior form of research places pressure on surgical researchers to undertake trials even when they may not be appropriate. This we believe has led to the proliferation of poorly reported, poorly designed small single centre trials in the cardiothoracic literature. By virtue of their design and insufficient power many of the trials in our study were incapable of answering the questions researchers sought to address. In a bid to provide positive results (which makes the trials more publishable [14]) investigators often relied on laboratory measurements and other surrogate numerical endpoints rather than clinical outcomes of relevance to patients and their doctors. The use of numerical endpoints and surrogate outcomes was more frequent in trials with less than 50 patients probably reflecting a bid to show statistical difference in trials with small sample size. While randomised trials are the ‘gold standard’, if design is poor or power inadequate, or inappropriate endpoints are used, data produced are no better (and could be worse) than that from non-randomised studies.

Are expectations placed on surgical researchers to perform more high quality randomised trials [1,7] unreasonable? To what extent do our findings reflect the special difficulties or sometimes impracticality that surgical researchers face in undertaking RCTs in surgery [15]? We would argue that the application and role of the RCT in surgery is different from the RCT in non-surgical specialities. Indeed the majority of the evidence governing contemporary cardiothoracic surgical practice arises not from clinical trials but from observational studies. The best known example is the evidence for the superiority of the internal mammary graft over vein grafts to the coronary arteries [16–18]. No RCT addressed this nor likely ever will [19,20], but we doubt if the many clinical trialists would regard it a matter sufficiently unproven that they would not mind which graft they had to their own left anterior descending coronary artery. The realisation that randomised blinded control groups are not obtainable for many surgical procedures has led surgical statisticians to focus on development of methods that balance for bias introduced by non-random application of interventions, therefore allowing for credible comparative research based on clinical experience rather than RCTs [21]. Perhaps this is time to set out guidance and criteria for high-quality observational studies, where in spite of the best of intentions and commitment RCTs just cannot be done, a ‘CONSORT’ for non-randomised studies.

Limitations of the surgical RCT include inability to apply blinding, problems in allocation concealment, limitations in study size and non-standardisation of the intervention under test. If the investigator, the patient or the assessor is aware of the treatment applied one cannot be certain that systematic bias has been excluded. Ability to exclude bias is the essence of the RCT and it is arguable that if blinding cannot be achieved then surgical RCTs are not necessarily an improvement on well-conducted observational studies [22]. There are exceptions where lack of blinding will certainly not influence outcomes but for surgical procedures these are few. Allocation concealment though desirable is sometimes not feasible so unblinding may take place well before the time of application of the intervention; for example, it would be unreasonable to keep a fully staffed theatre and fully staffed angioplasty suite awaiting unblinding of a patient in a surgery versus angioplasty trial. In practice, the investigators know which arm a patient has been allocated days or even weeks before the intervention, giving room for dropouts, crossover and introduction of bias. We know that worryingly few of the patients eligible were in fact randomised in all the large studies of coronary artery surgery, versus medical treatment and versus angioplasty, which were nevertheless RCTs performed as rigorously as possible.

Variation in surgical practice, surgical skill and surgeon preference have proven obstacles to large scale multicentre trials of surgery. Unlike in a drug trial, where drugs are manufactured in a standard fashion and quality tested, in surgical trials operations are performed by surgeons who being human are subject to variation in their belief, enthusiasm and commitment to one method over another and there remains the unanswerable question, is any difference found attributable to the operative method set out or the skill with which it was performed? Surgeons in a trial may be expected to perform a new procedure that they lack the experience, enthusiasm or skill to perform. Surgical trials of new procedures are therefore often restricted to enthusiasts and innovators and therefore often small in size and not necessarily generalisable to the non-enthusiast. Non-standardisation of surgical technique is also a problem in that no two operations are the same. Therefore, while in a drug trial one can be reasonably certain that all patients allocated to one arm received the same intervention, in surgical trials one cannot have the same certainty.

Having made this apologia for surgical research, there is an evident paucity and poor quality of these studies as judged by expected standards [12]. There must be room for improvement in our willingness to subject surgical techniques to the test. With more attention to trial design including power calculation and increased focus on clinical rather than surrogate end-points, it should be certain from the outset if a question can be answered by the proposed RCT. If reviewers and editors of specialist cardiothoracic journals were as rigorous in implementing guidelines for reporting RCTs such as CONSORT it might reverse the proliferation of underpowered and methodologically flawed trials in the cardiothoracic literature and also improve the awareness in the medical community that randomised trials cannot necessarily be applied to all surgical questions. Surgical research does have a future and while we encourage the use of RCTs when appropriate, the future probably does not rest in the RCT but in the development and refinement of methods for comparing non-randomised controls. We are by no means ‘anti-RCTs’. We promote the use of unbiased allocation [23] and have ourselves undertaken surgical RCTs, and will continue to do so when appropriate questions arise within our practice, but we do also recognise their limitations. Regardless of the research method used, the success of surgical research will rest in what Sir Bradford Hill laid down as the essence of the RCT over 50 years ago... painstaking, and sometimes very dull, attention to every detail [24].

	Footnotes

 
Presented at the joint 17th Annual Meeting of the European Association for Cardio-thoracic Surgery and the 11th Annual Meeting of the European Society of Thoracic Surgeons, Vienna, Austria, October 12–15, 2003.

	Appendix A. Conference discussion

Top Abstract 1. Introduction 2. Materials and methods 3. Results 4. Discussion Appendix A. Conference... References

 
Dr P. Sergeant (Leuven, Belgium): I could not agree more with you. Can you give to the audience a rough idea about the number of patients needed if you have a 1 or 2% event rate and you think that your new model or your new product or your new therapy will like improve it with 10%? Just give us an example of a certain event rate, a certain expected, with the regular alpha and beta type errors.

Mr Anyanwu: Taking a common example, if you wanted to reduce the mortality rate with coronary surgery from, say, 2.5% to 2.0% with a power of 90% and a statistical significance of 5%, you will require about 20,000 patients in each group.

Dr G. van der Heijden (Utrecht, The Netherlands): I am a little bit worried about your generic conclusion, not saying that either you or Richard Horton are wrong, but you don't present the absolute numbers in the two-year period in the three journals which you visited before you presented the 100 or so trials. So how many non-trials were there given in comparison to the trials? And the other question is, if you make such a generic conclusion on the randomized trials in cardiac surgery, would it be more wise to look first at other medical specialties in randomized trials to see whether the state of the art is the same there? As a clinical epidemiologist I know of certain specialty areas which give a poorer view on randomized trials and also prognostic cohort studies.

Mr Anyanwu: We did not look at the proportion of papers that were randomized trials because it is well accepted and demonstrated that randomized trials form a minority of surgical research. That was not the subject of our presentation. The same has been observed in most other surgical specialties. As regards nonsurgical specialties, if you, for example, picked up the New England Journal of Medicine or any other general medical journal, you would see that a sizable proportion of their papers are randomized trials, and the same is not in surgical specialties. So there is a clear difference between surgical and non-surgical research and publications.

Dr van der Heijden: Yes, but then you change the question to the overall picture on randomized trials or landmark clinical trials or landmark prognostic cohort studies, and then you visit another question, I think.

Mr. Anyanwu: We are not saying that randomized trials are not a good form of research. For nonsurgical specialties, randomized trials are the preferred form of research. The only way to compare two drugs is to do a randomized trial. But if, on the other hand, you want to compare whether you put a vein graft on the circumflex or a mammary artery on the circumflex, there are so many factors that will come into it that you are not going to show a difference with a randomized trial. That is what we are trying to say, that there is a difference between the surgical trial and the nonsurgical trial.

Mr V. Zamvar (Edinburgh, UK): I do not think you can make those generic conclusions because you have looked at the reporting of trials, not the conduct of trials. Partly the fault here lies with the editors and the editorial board members who accept publication of trials which have not been reported adequately. My question is, if you looked at surgical randomized control trials which have been reported mainly in the medical literature, The Lancet or the New England Journal or the BMJ, do you think your results would be substantially different?

Mr Anyanwu: We are not saying that randomized trials have no role. There is a role for randomized trials. I think if you do a randomized trial of several hundred patients, then you will get it published in a good journal. For example, the National Emphysema Trial, which was published earlier this year, is a landmark surgical trial. There is a role for the landmark multicenter surgical trial. As regards the issue of possible reporting bias, I do not think that the deficiencies we show reflect poor reporting as opposed to poor methodology. For example, most of the trials we looked at had a small sample size of 50 patients or less. I do not think that would constitute a lack of reporting. People will generally report all the patients they randomize and all the techniques they use.

Dr T. Aberg (Umea, Sweden): I think this is a very timely and important presentation and we should take heed of it. You mentioned that we should employ somehow an experienced trial-list and conduct multicenter studies in some circumstances.How do you see the role of the Association in trying to help in this endeavor?

Mr Anyanwu: Well, I think the role of an Association like this is crucial. Various subcommittees should identify areas relevant to the practice of the specialty as a whole, and through that to call for randomized trials as appropriate. I think this is crucial, and to some degree is happening on the thoracic side of the Society.

Dr G. Laufer (Innsbruck, Austria): May I ask you how important do you think is financial support from pharmaceutical companies? For example, in the field of heart transplantation we have good examples. There have been conducted serious prospective randomized trials, for example, involving everolimus, involving mycophenolate mofetil, even the Neoral formulation of cyclosporine was tested in a prospective randomized trial. So I think whenever there is an expensive new compound involved in the study, there is a good chance to get a big randomized study because you have the support from the industry, and the order of magnitude of the pharmaceutical industry able to support surgical research or research in general is much higher than from device companies.

Mr Anyanwu: I think I agree a hundred percent with that.

	References

Top Abstract 1. Introduction 2. Materials and methods 3. Results 4. Discussion Appendix A. Conference... References

 

1. Horton R. Surgical research or comic opera: questions, but few answers. Lancet 1996;347:984-985.[CrossRef][Medline]
2. Rennie D. How to report randomized controlled trials. The CONSORT statement. J Am Med Assoc 1996;276:649.[CrossRef][Medline]
3. Begg C., Cho M., Eastwood S., Horton R., Moher D., Olkin I., Pitkin R., Rennie D., Schulz K.F., Simel D., Stroup D.F. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. J Am Med Assoc 1996;276:637-639.[CrossRef][Medline]
4. Schulz K.F. Randomised trials, human nature, and reporting guidelines. Lancet 1996;348:596-598.[CrossRef][Medline]
5. McNamee D., Horton R. Lies, damn lies, and reports of RCTs. Lancet 1996;348:562.[CrossRef][Medline]
6. Magos A., Cumbis A., Katsetos C. Bias against publication of surgical papers. Lancet 2000;355:413.
7. Smith R. Publishing surgery. Lancet 2000;355:849.[Medline]
8. Black N. Why we need observational studies to evaluate the effectiveness of health care. Br Med J 1996;312:1215-1218.[Free Full Text]
9. McKee M., Britton A., Black N., McPherson K., Sanderson C., Bain C. Methods in health services research. Interpreting the evidence: choosing between randomised and non-randomised studies. Br Med J 1999;319:312-315.[Free Full Text]
10. Padkin A., Rowan K., Black N. Using high quality clinical databases to complement the results of randomised controlled trials: the case of recombinant human activated protein C. Br Med J 2001;323:923-926.[Free Full Text]
11. Treasure T. Advances and retreats. Lancet 1999;353(Suppl 1):SI33-SI34.
12. Moher D., Schulz K.F., Altman D.G. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 2001;357:1191-1194.[CrossRef][Medline]
13. Hossein-Nia M., Kallis P., Brown P.A., Chester M.R., Kaski J.C., Murday A.J., Treasure T., Holt D.W. Creatine kinase MB isoforms: sensitive markers of ischemic myocardial damage. Clin Chem 1994;40:1265-1271.[Abstract/Free Full Text]
14. Easterbrook P.J., Berlin J.A., Gopalan R., Matthews D.R. Publication bias in clinical research. Lancet 1991;337:867-872.[CrossRef][Medline]
15. Solomon M.J., McLeod R.S. Surgery and the randomised controlled trial: past, present and future. Med J Aust 1998;169:380-383.[Medline]
16. Loop F.D., Lytle B.W., Cosgrove D.M., Stewart R.W., Goormastic M., Williams G.W., Golding L.A., Gill C.C., Taylor P.C., Sheldon W.C., Proudfit W. Influence of the internal-mammary-artery graft on 10-year survival and other cardiac events. N Engl J Med 1986;314:1-6.[Abstract]
17. Coronary bypass with the internal mammary. Lancet 1984;2:1253–1254.
18. Lytle B.W., Blackstone E.H., Loop F.D., Houghtaling P.L., Arnold J.H., Akhrass R., McCarthy P.M., Cosgrove D.M. Two internal thoracic artery grafts are better than one. J Thorac Cardiovasc Surg 1999;117:855-872.[Abstract/Free Full Text]
19. Underwood M., Cooper G., Keogh B. Is one arterial graft enough?. Br Med J 1997;314:1213-1214.[Free Full Text]
20. Taggart D.P., D'Amico R., Altman D.G. Effect of arterial revascularisation on survival: a systematic review of studies comparing bilateral and single internal mammary arteries. Lancet 2001;358:870-875.[CrossRef][Medline]
21. Blackstone E.H. Comparing apples and oranges. J Thorac Cardiovasc Surg 2002;123:8-15.[Free Full Text]
22. Anyanwu A., O'Riordan D. Unblinded trails may not be more reliable than observational studies. Br Med J 1999;318:190.
23. Treasure T., MacRae K.D. Minimisation: the platinum standard for trials? Randomisation doesn't guarantee similarity of groups; minimisation does. Br Med J 1998;317:362-363.[Free Full Text]
24. Bradford Hill A. The clinical trial. Br Med Bull 1951;7:278-282.[Free Full Text]

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Author home page(s): Ani C. Anyanwu Tom Treasure Permission Requests Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Anyanwu, A. C. Articles by Treasure, T. Search for Related Content PubMed PubMed Citation Articles by Anyanwu, A. C. Articles by Treasure, T. Related Collections Cardiac - other Professional affairs