|
|
||||||||
Eur J Cardiothorac Surg 2004;25:299-303
© 2004 Elsevier Science NL
a Cardiothoracic Unit, Harefield Hospital, London, UK
b Cardiothoracic Unit, Guy's Hospital, London SE1 9RT, UK
Received 23 July 2003; received in revised form 5 November 2003; accepted 2 December 2003.
* Corresponding author. Tel.: +44-20-8725-3288
e-mail: tom.treasure{at}ukgateway.net
| Abstract |
|---|
|
|
|---|
Key Words: Randomised controlled trial Surgical research Surgical journals
| 1. Introduction |
|---|
|
|
|---|
Our study is an honest reflection from the inside rather than high-handed criticism from outsiders. Criticism has often come from those who are neither surgeons nor researchers and do not have to tackle the obstacles, ethical and logistic that make clinical surgical research so challenging. We are conscious that we ourselves, as surgical researchers in this era, have performed and published studies which might now be robustly criticised.
| 2. Materials and methods |
|---|
|
|
|---|
Study objective, inclusion/exclusion criteria, year of study, details and timing of interventions, definition of a primary clinical outcome, sample size based on power calculation, generation of randomisation, implementation of randomisation, allocation concealment, randomisation independent of implementer and assessor, blinding (patients), blinding (implementers), blinding (assessors), duration of recruitment, duration of follow-up, proportion of eligible patients recruited, description of dropouts, baseline group data, crossovers, complications and adverse effects.
We compared proportions using the
2 test.
| 3. Results |
|---|
|
|
|---|
Majority of trials (102, 86%) used quantitative surrogate measures, which generate numerical data for analysis, rather than categorical endpoints of clinical relevance. In only 24 (20%) trials was the primary outcome a directly measured clinical event. This was most evident in trials with sample size less than 50 where 4/65 (6%) used categorical variables as an endpoint compared to 13/54 (24%) in studies with 50 or more subjects (P=0.005). Although 74 trials reported positive results, only 18 recommended practice change.
Trials were scored from 0 (worst) to 20 (best) based on the number of CONSORT criteria that were fulfilled (Fig. 1) . The median score was 8 (range 319). As far as randomisation and blinding were concerned, in the majority of instances there was simply not enough information in the studies to decide if these vital components of study design were adequate (Table 1).
|
|
| 4. Discussion |
|---|
|
|
|---|
The emphasis on randomised trials as the superior form of research places pressure on surgical researchers to undertake trials even when they may not be appropriate. This we believe has led to the proliferation of poorly reported, poorly designed small single centre trials in the cardiothoracic literature. By virtue of their design and insufficient power many of the trials in our study were incapable of answering the questions researchers sought to address. In a bid to provide positive results (which makes the trials more publishable [14]) investigators often relied on laboratory measurements and other surrogate numerical endpoints rather than clinical outcomes of relevance to patients and their doctors. The use of numerical endpoints and surrogate outcomes was more frequent in trials with less than 50 patients probably reflecting a bid to show statistical difference in trials with small sample size. While randomised trials are the gold standard, if design is poor or power inadequate, or inappropriate endpoints are used, data produced are no better (and could be worse) than that from non-randomised studies.
Are expectations placed on surgical researchers to perform more high quality randomised trials [1,7] unreasonable? To what extent do our findings reflect the special difficulties or sometimes impracticality that surgical researchers face in undertaking RCTs in surgery [15]? We would argue that the application and role of the RCT in surgery is different from the RCT in non-surgical specialities. Indeed the majority of the evidence governing contemporary cardiothoracic surgical practice arises not from clinical trials but from observational studies. The best known example is the evidence for the superiority of the internal mammary graft over vein grafts to the coronary arteries [1618]. No RCT addressed this nor likely ever will [19,20], but we doubt if the many clinical trialists would regard it a matter sufficiently unproven that they would not mind which graft they had to their own left anterior descending coronary artery. The realisation that randomised blinded control groups are not obtainable for many surgical procedures has led surgical statisticians to focus on development of methods that balance for bias introduced by non-random application of interventions, therefore allowing for credible comparative research based on clinical experience rather than RCTs [21]. Perhaps this is time to set out guidance and criteria for high-quality observational studies, where in spite of the best of intentions and commitment RCTs just cannot be done, a CONSORT for non-randomised studies.
Limitations of the surgical RCT include inability to apply blinding, problems in allocation concealment, limitations in study size and non-standardisation of the intervention under test. If the investigator, the patient or the assessor is aware of the treatment applied one cannot be certain that systematic bias has been excluded. Ability to exclude bias is the essence of the RCT and it is arguable that if blinding cannot be achieved then surgical RCTs are not necessarily an improvement on well-conducted observational studies [22]. There are exceptions where lack of blinding will certainly not influence outcomes but for surgical procedures these are few. Allocation concealment though desirable is sometimes not feasible so unblinding may take place well before the time of application of the intervention; for example, it would be unreasonable to keep a fully staffed theatre and fully staffed angioplasty suite awaiting unblinding of a patient in a surgery versus angioplasty trial. In practice, the investigators know which arm a patient has been allocated days or even weeks before the intervention, giving room for dropouts, crossover and introduction of bias. We know that worryingly few of the patients eligible were in fact randomised in all the large studies of coronary artery surgery, versus medical treatment and versus angioplasty, which were nevertheless RCTs performed as rigorously as possible.
Variation in surgical practice, surgical skill and surgeon preference have proven obstacles to large scale multicentre trials of surgery. Unlike in a drug trial, where drugs are manufactured in a standard fashion and quality tested, in surgical trials operations are performed by surgeons who being human are subject to variation in their belief, enthusiasm and commitment to one method over another and there remains the unanswerable question, is any difference found attributable to the operative method set out or the skill with which it was performed? Surgeons in a trial may be expected to perform a new procedure that they lack the experience, enthusiasm or skill to perform. Surgical trials of new procedures are therefore often restricted to enthusiasts and innovators and therefore often small in size and not necessarily generalisable to the non-enthusiast. Non-standardisation of surgical technique is also a problem in that no two operations are the same. Therefore, while in a drug trial one can be reasonably certain that all patients allocated to one arm received the same intervention, in surgical trials one cannot have the same certainty.
Having made this apologia for surgical research, there is an evident paucity and poor quality of these studies as judged by expected standards [12]. There must be room for improvement in our willingness to subject surgical techniques to the test. With more attention to trial design including power calculation and increased focus on clinical rather than surrogate end-points, it should be certain from the outset if a question can be answered by the proposed RCT. If reviewers and editors of specialist cardiothoracic journals were as rigorous in implementing guidelines for reporting RCTs such as CONSORT it might reverse the proliferation of underpowered and methodologically flawed trials in the cardiothoracic literature and also improve the awareness in the medical community that randomised trials cannot necessarily be applied to all surgical questions. Surgical research does have a future and while we encourage the use of RCTs when appropriate, the future probably does not rest in the RCT but in the development and refinement of methods for comparing non-randomised controls. We are by no means anti-RCTs. We promote the use of unbiased allocation [23] and have ourselves undertaken surgical RCTs, and will continue to do so when appropriate questions arise within our practice, but we do also recognise their limitations. Regardless of the research method used, the success of surgical research will rest in what Sir Bradford Hill laid down as the essence of the RCT over 50 years ago... painstaking, and sometimes very dull, attention to every detail [24].
| Footnotes |
|---|
| Appendix A. Conference discussion |
|---|
|
|
|---|
Mr Anyanwu: Taking a common example, if you wanted to reduce the mortality rate with coronary surgery from, say, 2.5% to 2.0% with a power of 90% and a statistical significance of 5%, you will require about 20,000 patients in each group.
Dr G. van der Heijden (Utrecht, The Netherlands): I am a little bit worried about your generic conclusion, not saying that either you or Richard Horton are wrong, but you don't present the absolute numbers in the two-year period in the three journals which you visited before you presented the 100 or so trials. So how many non-trials were there given in comparison to the trials? And the other question is, if you make such a generic conclusion on the randomized trials in cardiac surgery, would it be more wise to look first at other medical specialties in randomized trials to see whether the state of the art is the same there? As a clinical epidemiologist I know of certain specialty areas which give a poorer view on randomized trials and also prognostic cohort studies.
Mr Anyanwu: We did not look at the proportion of papers that were randomized trials because it is well accepted and demonstrated that randomized trials form a minority of surgical research. That was not the subject of our presentation. The same has been observed in most other surgical specialties. As regards nonsurgical specialties, if you, for example, picked up the New England Journal of Medicine or any other general medical journal, you would see that a sizable proportion of their papers are randomized trials, and the same is not in surgical specialties. So there is a clear difference between surgical and non-surgical research and publications.
Dr van der Heijden: Yes, but then you change the question to the overall picture on randomized trials or landmark clinical trials or landmark prognostic cohort studies, and then you visit another question, I think.
Mr. Anyanwu: We are not saying that randomized trials are not a good form of research. For nonsurgical specialties, randomized trials are the preferred form of research. The only way to compare two drugs is to do a randomized trial. But if, on the other hand, you want to compare whether you put a vein graft on the circumflex or a mammary artery on the circumflex, there are so many factors that will come into it that you are not going to show a difference with a randomized trial. That is what we are trying to say, that there is a difference between the surgical trial and the nonsurgical trial.
Mr V. Zamvar (Edinburgh, UK): I do not think you can make those generic conclusions because you have looked at the reporting of trials, not the conduct of trials. Partly the fault here lies with the editors and the editorial board members who accept publication of trials which have not been reported adequately. My question is, if you looked at surgical randomized control trials which have been reported mainly in the medical literature, The Lancet or the New England Journal or the BMJ, do you think your results would be substantially different?
Mr Anyanwu: We are not saying that randomized trials have no role. There is a role for randomized trials. I think if you do a randomized trial of several hundred patients, then you will get it published in a good journal. For example, the National Emphysema Trial, which was published earlier this year, is a landmark surgical trial. There is a role for the landmark multicenter surgical trial. As regards the issue of possible reporting bias, I do not think that the deficiencies we show reflect poor reporting as opposed to poor methodology. For example, most of the trials we looked at had a small sample size of 50 patients or less. I do not think that would constitute a lack of reporting. People will generally report all the patients they randomize and all the techniques they use.
Dr T. Aberg (Umea, Sweden): I think this is a very timely and important presentation and we should take heed of it. You mentioned that we should employ somehow an experienced trial-list and conduct multicenter studies in some circumstances.How do you see the role of the Association in trying to help in this endeavor?
Mr Anyanwu: Well, I think the role of an Association like this is crucial. Various subcommittees should identify areas relevant to the practice of the specialty as a whole, and through that to call for randomized trials as appropriate. I think this is crucial, and to some degree is happening on the thoracic side of the Society.
Dr G. Laufer (Innsbruck, Austria): May I ask you how important do you think is financial support from pharmaceutical companies? For example, in the field of heart transplantation we have good examples. There have been conducted serious prospective randomized trials, for example, involving everolimus, involving mycophenolate mofetil, even the Neoral formulation of cyclosporine was tested in a prospective randomized trial. So I think whenever there is an expensive new compound involved in the study, there is a good chance to get a big randomized study because you have the support from the industry, and the order of magnitude of the pharmaceutical industry able to support surgical research or research in general is much higher than from device companies.
Mr Anyanwu: I think I agree a hundred percent with that.
| References |
|---|
|
|
|---|
This article has been cited by other articles:
![]() |
P. J. Karanicolas, M. Bhandari, B. Taromi, E. A. Akl, D. Bassler, P. Alonso-Coello, D. Rigau, D. Bryant, S. E. Smith, S. D. Walter, et al. Blinding of Outcomes in Trials of Orthopaedic Trauma: An Opportunity to Enhance the Validity of Clinical Trials J. Bone Joint Surg. Am., May 1, 2008; 90(5): 1026 - 1033. [Abstract] [Full Text] [PDF] |
||||
![]() |
I. Boutron, D. Moher, D. G. Altman, K. F. Schulz, P. Ravaud, and for the CONSORT Group Extending the CONSORT Statement to Randomized Trials of Nonpharmacologic Treatment: Explanation and Elaboration Ann Intern Med, February 19, 2008; 148(4): 295 - 309. [Abstract] [Full Text] [PDF] |
||||
![]() |
I. Boutron, D. Moher, D. G. Altman, K. F. Schulz, P. Ravaud, and for the CONSORT Group Methods and Processes of the CONSORT Group: Example of an Extension for Trials Assessing Nonpharmacologic Treatments Ann Intern Med, February 19, 2008; 148(4): W-60 - W-66. [Abstract] [Full Text] [PDF] |
||||
![]() |
I. Boutron, P. Ravaud, and R. Nizard The design and assessment of prospective randomised, controlled trials in orthopaedic surgery J Bone Joint Surg Br, July 1, 2007; 89-B(7): 858 - 863. [Abstract] [Full Text] [PDF] |
||||
![]() |
F. D. Rubens and H. Nathan Lessons learned on the path to a healthier brain: dispelling the myths and challenging the hypotheses Perfusion, May 1, 2007; 22(3): 153 - 160. [Abstract] [PDF] |
||||
![]() |
T. Treasure The evidence on which to base practice: different tools for different times Eur. J. Cardiothorac. Surg., December 1, 2006; 30(6): 819 - 824. [Full Text] [PDF] |
||||
![]() |
R. M. Sade "Surgical research or comic opera" redux. Ann. Thorac. Surg., October 1, 2006; 82(4): 1173 - 1174. [Full Text] [PDF] |
||||
![]() |
R. Tiruvoipati, S. P. Balasubramanian, G. Atturu, G. J. Peek, and D. Elbourne Improving the quality of reporting randomized controlled trials in cardiothoracic surgery: The way forward. J. Thorac. Cardiovasc. Surg., August 1, 2006; 132(2): 233 - 240. [Abstract] [Full Text] [PDF] |
||||
![]() |
R. M. Sade Reports of clinical trials: Ethical aspects. J. Thorac. Cardiovasc. Surg., August 1, 2006; 132(2): 245 - 246. [Full Text] [PDF] |
||||
![]() |
T. Caus, Y. Seree, P. Marin, M. Khairi, A. Bakkali, J. C. Guillen, J. L. Bonnet, and D. Metras Off-pump coronary surgery in selected patients: better early outcome but more recurrence of angina? Interactive CardioVascular and Thoracic Surgery, August 1, 2005; 4(4): 322 - 326. [Abstract] [Full Text] [PDF] |
||||
![]() |
N. Alphonso, C. Tan, M. Utley, R. Cameron, J. Dussek, L. Lang-Lazdunski, and T. Treasure A prospective randomized controlled trial of suction versus non-suction to the under-water seal drains following lung resection Eur. J. Cardiothorac. Surg., March 1, 2005; 27(3): 391 - 394. [Abstract] [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| ANN THORAC SURG | ASIAN CARDIOVASC THORAC ANN | EUR J CARDIOTHORAC SURG |
| J THORAC CARDIOVASC SURG | ICVTS | ALL CTSNet JOURNALS |