Evaluating the importance of sham-controlled trials in the investigation of medical devices in interventional cardiology

Abstract

Cardiovascular medicine is one of the specialties that has relied most heavily on evidence from randomised clinical trials in determining best practice for the management of common disease conditions. When comparing treatment approaches, trials incorporating random allocation are the most appropriate method for protecting against treatment allocation bias. In order to protect against performance and ascertainment bias, trial designs including placebo control are preferable where feasible. In contrast to testing of medicines, treatments based on procedures or use of medical devices are more challenging to assess, as sham procedures are necessary to facilitate blinding of participants. However, in many cases, ethical concerns exist, as individual patients allocated to sham procedure are exposed only to risk without potential for benefit. Accordingly, the potential benefits to the general patient population must be carefully weighed against the risks of the exposed individuals. For this reason, trial design and study conduct are critically important to ensure that the investigation has the best chance of answering the study question at hand. In the current manuscript, we aim to review issues relating to the conduct of sham-controlled trials and discuss a number of recent examples in the field of interventional cardiology.

Abbreviations

CI: confidence interval

PCI: percutaneous coronary intervention

PRO: patient reported outcome

RCT: randomised clinical trial

SAE: serious adverse event

Introduction

Rapid advances in cardiovascular medicine in recent years have contributed to an important decline in cardiovascular mortality in the western world1,2. Much of this progress has been driven by the development and iteration of medical and surgical devices and procedures. Central to the introduction of these improvements into daily clinical care has been objective testing in randomised clinical trials (RCTs). Indeed, cardiovascular medicine has been one of the specialties to see the most widespread adoption of evidence-based practice driven by results from clinical trials3. Objective testing of new treatments and care pathways allows doctors, providers and patients to assess the risk and benefits of interventions, facilitates regulators in deciding whether to approve new treatments, and enables payers to assess whether the benefits of treatment are affordable and economically viable.

THE ROLE OF RANDOMISED CLINICAL TRIALS IN CARDIOVASCULAR MEDICINE

A range of clinical study designs can contribute to the evidence used in the assessment of therapeutic interventions3. However, RCTs are recognised to afford the highest grade of evidence and have the strongest influence on clinical practice guidelines4. This is due to the ability to isolate the true effects of tested treatments from the medical decision that leads to the assignment of one treatment or another to the individual patient (assignment bias) (Table 1). Random treatment allocation using appropriate methods has many advantages including a high likelihood that both measured and unmeasured confounding factors and measurement errors will be equally distributed between the treatment groups under investigation. Moreover, by the nature of their design and scientific conduct, RCTs usually, though not invariably, have higher quality data sets compared with observational studies and so-called real-world data, due to the prospectively planned and performed data collection and trial oversight5. In addition, data monitoring, independent event adjudication and core laboratory assessment of quantitative metrics are frequently performed.

Central to the conduct of these trials is the concept of equipoise. Broadly speaking, it is only considered ethical to conduct trials where uncertainty regarding the risk/benefit of a therapy or intervention is thought to exist. However, the concept of equipoise is in itself difficult to define adequately, and usually depends on the opinion of experts, who may hold differing views6. For example, many professionals may not be comfortable that equipoise exists justifying a trial of intravenous diuretics in acute decompensated heart failure, although randomised data showing benefit may be scant or non-existent. The best method to resolve this dilemma is uncertain, though it is recognised that conduct of trials that violate the notion of equipoise for the development of evidence to support treatment and healthcare policy decisions may be permissible in certain situations6.

RANDOMISED TRIAL DESIGN AND MEASURES TO PROTECT AGAINST BIAS

In conducting RCTs, random treatment allocation alone is not sufficient to protect adequately against several sources of bias. A number of other issues must be addressed to ensure the internal validity of the trial (Table 1, Figure 1). These include the use of appropriate methods to generate the randomisation sequence, and effective allocation concealment to ensure that investigators cannot detect the treatment allocation sequence and introduce selection and/or allocation bias7. Central randomisation using web-based systems or interactive voice response systems are most robust; sealed, opaque and sequentially numbered envelopes may still be justifiable in 2018 in specific cases even though they are more likely to be undermined.

Figure 1. Influence of potential bias in blinded and unblinded randomised trials.

Data from RCTs remain susceptible to bias if an open-label treatment is used, although the risk of bias can be minimised by specifying primary and secondary outcome measures capable of objective evaluation8,9. Other important measures minimising the risk of bias in RCTs include blinding adjudicators of adverse events to prevent ascertainment (or detection) bias, blinding operators to prevent performance bias, and ensuring completeness of follow-up and intention-to-treat analysis to prevent attrition bias. In addition, it is important to ensure the enrolment of a sufficient number of patients to ensure appropriate power so that a true difference can be detected if it exists or the null hypothesis can be accepted with an appropriate probability of certainty.

Clinical investigations are often limited in their external validity for clinical practice, due to the inclusion of only selected patients, typically with lower risk and adverse event rates than the patients who were not included. This is known as selection bias. The use of screening logs is important to capture the degree of selection, though this provides only limited protection. Moreover, the conditions pertaining in a clinical trial including protocol-directed care, frequent contact with patients, and intensive follow-up may not be reproducible in everyday practice. Finally, patients and care providers may also change their behaviour when included in a clinical trial, because they are being observed. For example, compliance to medication or lifestyle measures (from the patient perspective) or delivery of care (from the care provider perspective) may be better than under normal circumstances. This has been described as the Hawthorne effect10.

IMPORTANCE OF PATIENT-REPORTED OUTCOMES

At present, there is increased realisation of the importance of trials assessing quality of life11, in addition to or instead of assessment of endpoints such as mortality. Indeed, although the use of patient-reported outcomes (PRO) is frequent in some specialties, these measures have been less frequently used in trials in interventional cardiology. One analysis of cardiovascular medicine trials found that 65 of 413 trials (16%) used an instrument to assess PRO. Moreover, the majority of trials (122 of 174 trials, 70%) where use of a PRO would have been important based on the study question (e.g., expected impact on symptomatic relief or quality of life) did not include this element12. In trials assessing subjective outcome measures, double-blinded trial design is critical. In pharmaceutical trials this may be accomplished by use of comparator or placebo medications of identical taste and appearance, or by the use of double dummy. In testing of medical devices and interventions, use of a sham control may be necessary to isolate the true effect of the intervention under investigation.

PLACEBO EFFECTS AND USE OF PLACEBO OR SHAM TREATMENTS IN RANDOMISED TRIALS

The impact of an interaction with the patient and the healthcare system may elicit clinical benefits independent of whether or not a biological effect was achieved by the treatment or intervention under investigation. This effect is typically termed placebo effect. The term placebo is broadly taken to mean “falsely pleasing”, believed to derive from the name applied to professional mourners at funerals in medieval times, who attended to chant prayers for the dead in return for gifts from the family of the dead and were known by some as “placebos” after one of the prayers they chanted13. Failure to consider this aspect of treatment may lead to overestimation of the benefit of a given intervention or treatment. It is particularly appropriate to take measures to counteract this effect where the focus of the assessment is symptoms (e.g., pain), or metrics of quality of life that could be affected by this (e.g., blood pressure, exercise tolerance).

The placebo response to an intervention is a complex phenomenon. Potential benefit is mediated by many factors including patient preconditioning, expectations of treatment, and the experience of the healthcare interaction13. For example, preconditioning to a treatment, due to prior benefit from the intervention, can result in rapid biochemical changes and enhance treatment benefit14. Moreover, delivery of positive messages can improve expectation of benefit and result in symptom relief and anxiety reduction15. One study showed that, in patients undergoing coronary angiography, delivery of positive messages resulted in core laboratory measurable coronary vasodilation16. In addition, frequent interactions with care givers and doctors in a reassuring environment may also confer symptom relief and anxiety reduction.

A placebo can be used for a pharmacological agent, or for a physical or psychological intervention. In these latter cases, it is usually referred to as a sham intervention. A placebo should be similar in every respect to the treatment being investigated with the exception that the active ingredient or component is lacking. However, for a variety of reasons, successful blinding of treatment can be difficult to achieve. Ideally, therefore, studies including placebo or sham treatment should assess the success of blinding. Evidence suggests that this is infrequently done17.

In the case of sham treatments, it may also be important to blind the operator in order to eliminate performance bias (Table 1), which may contribute to outcome differences between groups. In an RCT investigating intravascular sonotherapy to reduce neointimal hyperplasia, both patient and operator were blinded to treatment received18. This was possible because, after the catheter was placed in the vessel, a memory device loaded set-up and treatment data into the instrument, eliminating the need for operator control. In the trial, a central telephone randomisation service generated a code, which was entered on the device allowing either active or sham treatment to be delivered. In contrast to earlier preclinical studies19, the sham-controlled randomised trial showed no treatment effect. This type of trial represents the optimal approach but is not feasible in many settings (e.g., stenting versus no treatment).

Although the use of placebo or sham treatments is well established in research, it is not without controversy. In particular, it should not be used when a proven treatment exists for a given indication. Specific provisions regarding the restrictive use of placebo treatments are stated in the Declaration of Helsinki (Table 2)20. Evidence regarding the impact of the placebo effect on treatment response is not universally supportive of a meaningful effect. A meta-analysis by Hrobjartsson and colleagues analysed data from 130 trials in which treatment arms compared placebo treatment with no treatment21. Overall, the authors failed to detect evidence of clear benefit with placebo treatment. The results seemed consistent regardless of whether pharmaceutical, psychological or physical (sham) treatments were considered. On the other hand, variation was seen according to whether a binary or continuous outcome variable was assessed, with evidence of benefit only in case of the latter. Moreover, RCTs using pain as the outcome showed benefits in favour of placebo, although this benefit seemed to decrease with increasing sample size.

When interpreting trials incorporating placebo treatment or sham intervention, a number of specific issues need to be borne in mind. One issue not infrequently encountered in RCTs is that of regression to the mean22. This statistical phenomenon occurs when by chance a set of measurements at baseline is lower or higher than expected, for example due to the inclusion of patients with a random high or low frequency or severity of symptoms. Repeat measurement of the same parameter in a second setting will usually result in a re-equilibration of the measured value towards the real mean for the population. Accordingly, it would be incorrect to attribute the observed change to a placebo effect; in fact, it is caused by a statistical phenomenon.

SELECTED CLINICAL TRIALS IN INTERVENTIONAL CARDIOLOGY USING SHAM CONTROLS

A number of RCTs provide examples of the use of sham controls in interventional cardiology18,23-30. An overview of the principal characteristics of selected trials is provided in Table 3 and a summary of the primary endpoint analyses of four trials is shown in Figure 2.

Figure 2. Summary results of primary endpoint analysis of selected randomised trials using a sham control procedure in interventional cardiology. A) DIRECT trial. B) SYMPLICITY HTN-3 trial. C) COSIRA trial. D) ORBITA trial. CS: coronary sinus; SBP: systolic blood pressure

CATHETER-BASED LASER REVASCULARISATION

Catheter-based laser myocardial revascularisation showed favourable results at six months for patients with refractory angina, due to a significant improvement in angina and exercise duration in an early trial31. Subsequently, a phase II trial recruited patients with coronary artery disease, proven ischaemia and refractory angina despite optimal medical therapy, and incorporated a sham-control arm26. During the laser revascularisation or sham procedures, patients were sedated, and had blindfolds and earphones to minimise involuntary unblinding. If the patient was randomised to placebo, the laser (already in the room) was turned on but no further procedure was performed.

The primary endpoint, change in exercise treadmill duration from baseline to six months, was similar for both the active and sham treatment patients (Figure 2A), and this persisted at 12 months. The primary safety endpoint, a composite of 30-day major adverse cardiac events, occurred in 4.1%, 8.2% and 2.0% of the low-dose active treatment, high-dose active treatment and placebo patients (p=0.117). Six and 12 months after the index procedure, there were no statistically significant differences in cumulative death, acute myocardial infarction or repeat revascularisation among the groups.

CATHETER-BASED RENAL DENERVATION

Early experience with catheter-based radiofrequency denervation of the renal arteries for treatment of resistant hypertension showed a large treatment effect (e.g., reduction in office blood pressure) in unblinded trials. The SYMPLICITY HTN-3 (Renal Denervation in Patients With Uncontrolled Hypertension) study compared renal denervation using radiofrequency energy with a sham procedure in patients with severe resistant hypertension despite three or more antihypertensive medications27.

The primary efficacy endpoint was the mean change in office systolic blood pressure at six months, with a superiority margin of 5 mmHg. There was no significant difference between groups in the change in office systolic blood pressure at six months (−14.13±23.93 mmHg in the denervation group and −11.74±25.94 mmHg in the sham-procedure group), for a difference of −2.39 mmHg (95% confidence interval [CI]: −6.89 to 2.12; p=0.26 with a superiority margin of 5 mmHg) (Figure 2B). The study was also powered for the change in mean 24-hour ambulatory systolic blood pressure at six months; no significant difference was seen between the two groups. Consistent findings were noted in diastolic blood pressure. The primary safety endpoint, a composite of major adverse events within six months, did not differ between groups (1.4% in the renal denervation group and 0.6% in the sham-procedure group; p=0.67).

PERCUTANEOUS INTERVENTION FOR STABLE ANGINA

In unblinded trials, percutaneous coronary intervention (PCI) has also shown significant improvements in exercise time, angina relief, and quality of life32,33. In the recent ORBITA (Objective Randomised Blinded Investigation with optimal medical Therapy of Angioplasty in stable angina) trial, patients were enrolled if they had stable angina or equivalent symptoms and at least one angiographically significant non-occluded lesion (≥70%) in a single vessel (other than the left main coronary artery) that was clinically appropriate for PCI23. After a six-week escalating medical optimisation phase, where guideline-directed antianginal therapy was optimised, 200 patients with residual stable angina symptoms on a mean of three antianginal medications were randomised to PCI with drug-eluting stents or a sham procedure.

Patients in the sham group received auditory isolation and were sedated for at least 15 minutes after invasive physiological assessment. All outcome assessors were blinded. The difference between PCI and placebo groups in the change in treadmill exercise time at six weeks (primary endpoint) was 16.6 seconds (95% CI: –8.9 to 42.0) (p=0.20) (Figure 2D). The increase in exercise time was 28.4 seconds in the 104 patients with data available in the PCI group (95% CI: 11.6 to 45.1), and 11.8 seconds in the 90 patients with data available in the placebo group (95% CI: –7.8 to 31.3). There were no differences between PCI and placebo in several additional secondary endpoints, including physical limitation, angina frequency, and quality of life. The dobutamine stress echocardiography peak stress wall motion score index improved significantly with PCI (–0.07, 95% CI: –0.11 to –0.04, p<0.0001).

INTERPRETATION OF EVIDENCE

Blinded RCTs in which an investigational therapy is compared with a placebo are common for pharmacological agents but less common for medical devices or surgical interventions, in spite of the fact that in some cases the magnitude of the placebo effect with devices may be greater than with drugs34. There are a number of reasons for this. First, ethical concerns exist regarding the conduct of a trial in which patients are exposed to a sham procedure or implant. Patients are therefore exposed to risk of harm from the procedure without any prospect of benefit, in contrast to placebo-controlled trials with pharmaceutical agents. Second, the recruitment of patients is challenging as they are often reluctant to be potentially assigned to an invasive sham procedure; this may exacerbate selection bias.

However, sham-controlled trials are important and have contributed to stopping the development of many interventions, which were ineffective, preventing their widespread clinical use. In assessing the ethics of sham-controlled RCTs, investigators are faced with the challenge of weighing the foreseeable and preventable harm of a sham procedure against the risk of mistakenly believing an invasive procedure or implant to be useful when it is actually not. Sham-controlled trials present direct risks for subjects in the control group who undergo a sham procedure. For example, in the recent ORBITA trial, the proportion of serious adverse events (SAEs) in the control group was higher (8/95) compared with the PCI group (0/105). Moreover, at the time of the randomisation procedure, four of 95 patients in the control group suffered a coronary dissection requiring crossover to stenting. One approach to ameliorate this issue is by considering patients’ preference in the study design, preferentially enrolling patients who are actively demanding a device-based therapy. In such situations, ethical concerns may be less of an impediment to enrolment35.

Against the background of these ethical concerns, it is critically important that any potential risks to the individual patient are minimised by protocol design. In addition, in this type of trial more than any, it is vital that rigorous scientific methods are used in the conduct, in order to ensure that the investigation has the highest likelihood of answering the clinical question in a manner that is relevant for practice. Careful selection of primary endpoints is key because these studies are frequently small and surrogate endpoints rather than clinical outcomes are typically appraised. For example, although the ORBITA trial failed to detect differences in the primary endpoint between the treatment groups, due to a higher than expected standard deviation, the trial was not well powered to detect such differences. In addition, the time horizon of assessment at six weeks was too short to provide clinically meaningful results. Indeed, it was subsequently reported that 85% of patients in the sham-procedure group underwent PCI at the end of the six-week follow-up period (Al-Lamee R. Controversies in interventional cardiology I: interventional management in stable coronary artery disease. Presented at: SCAI Scientific Sessions 2018. April 25, 2018. San Diego, CA, USA). Trial design issues require careful weighing by institutional review boards, as inconclusive studies may unnecessarily expose patients in the control group to risks.

Conclusion

The use of sham control in trials in interventional cardiology is of considerable importance in the evaluation of medical devices and procedures. However, controversies exist in relation to the exposure of patients in the control group to the risk of the sham treatment and extreme care must be exercised to avoid misuse. Accordingly, the potential benefits to the general patient population must be carefully weighed against the risks of the exposed individuals. For this reason, trial design and study conduct are critically important to ensure that the investigation has the best chance of answering the hypothesis being tested in a reliable and relevant manner.

Guest Editor

This paper was guest edited by Alec Vahanian, MD, PhD; Cardiology Department, Hôpital Bichat, and University Paris VII, Paris, France.

Conflict of interest statement

R. Byrne reports lecture fees/honoraria from B. Braun Melsungen AG, Biotronik, Boston Scientific and Micell Technologies, and research grants to the institution from Boston Scientific and Celonova Biosciences. F. Mahfoud is supported by Deutsche Gesellschaft für Kardiologie, and has received speaker honoraria and consultancy fees from Medtronic and ReCor. J. Fajadet reports receipt of educational grants from Abbott, Boston Scientific, Medtronic, and Terumo. S. Windecker reports research grants to the institution from Abbott, Amgen, Boston Scientific, Biotronik, Medtronic and St. Jude. P. Jüni has received research grants to the institution from AstraZeneca, Biotronik, Biosensors International, Eli Lilly and The Medicines Company, and serves as unpaid member of the steering group of trials funded by AstraZeneca, Biotronik, Biosensors, St. Jude Medical and The Medicines Company. A. Baumbach reports research grants to the institution from Abbott Vascular, and advisory/speaker fees from AstraZeneca, Keystone Heart, MicroPort, Sinomed, and The Medicines Company. M. Haude reports research grants from Abbott, Biotronik, Cardiac Dimensions, Medtronic, OrbusNeich, speakers bureau from Abbott, Biotronik, Cardiac Dimensions, Lilly, Medtronic, OrbusNeich and Volcano, and being a consultant to Abbott, Biotronik, and OrbusNeich. W. Wijns reports grants from Abbott Vascular, grants from MicroPort and grants from MiCell and he is co-founder of Argonauts, an innovation facilitator. D. Capodanno has no conflicts of interest to declare. The Guest Editor is a consultant for Edwards Lifesciences.