Can contradictory results from RCTs that tested the same hypothesis be true simultaneously? Analyses of the VALIDATE-SWEDEHEART and BRIGHT-4 trials

Evidence-based medicine (EBM) is a high priority in healthcare1. The prevailing view of whether research is reliable places the most significant weight on findings from randomised controlled trials (RCTs). Some contend that these are insufficient to meet decision-makers’ needs in practice. A well-executed RCT establishes that an intervention works somewhere (i.e., in a trial). However, decision-makers must examine if it will work for us (i.e., in our society)2. In healthcare, it cannot be automatically assumed that the underlying social and physical systems in which interventions are conceived are causally analogous to the target location2. Differences in institutional, psychological, and physical elements create diverse causal and probabilistic linkages. For example, Sweden, China, and the USA have healthcare systems that differ in many respects. Citizens in these nations conceptualise, interact, and respond differently to events and behaviour in their societies that are causally related to healthcare outcomes.

The concept of external validity assumes that the same results obtained in an RCT also apply to the target population. In a controlled research setting, such as an RCT, a causal relationship between a specific factor and a particular outcome may consistently occur, provided that the experimental conditions remain unchanged. However, this relationship may not necessarily persist outside the RCT context due to the influence of other environmental factors that could negate the positive effects observed in the controlled setting. Key components necessary to achieve the therapeutic benefits reported in the RCT might be absent in the causal chain when applied in real-world situations, thus affecting the outcome. The medical community has embraced EBM with enthusiasm. However, EBM has been criticised over the years, especially by philosophers of science, who have questioned the premise that RCTs should serve as the primary basis for creating clinical practice guidelines. The claim that RCTs are the primary basis for guidelines must be backed by data demonstrating that their clinical results are superior to other clinical evidence. However, no definitive evidence has been shown so far. While EBM does advocate the cautious use of clinical evidence other than that from RCTs, its emphasis on the importance of RCTs has led to an often uncritical acceptance of their results without an appreciation of their limitations. As a result, erroneous conclusions have been reached, which has caused harm to patients in the real world.

The VALIDATE-SWEDEHEART trial (ClinicalTrials.gov: NCT02311231) – a registry-based RCT – investigated the clinical effect of bivalirudin versus heparin strategies without glycoprotein IIb/IIIa inhibitors3. It was one of the largest randomised trials comparing these anticoagulation strategies without glycoprotein IIb/IIIa inhibitors in ST-elevation myocardial infarction (STEMI) and non-ST-elevation myocardial infarction (NSTEMI) patients. The study has shown that compared to heparin alone during percutaneous coronary intervention (PCI) for acute coronary syndrome (ACS), bivalirudin did not reduce the composite of death, myocardial infarction (MI), or major bleeding in a large population receiving contemporary recommended treatments at 180 days.

The results from the VALIDATE-SWEDEHEART study agree with analyses from the HEAT-PPCI and MATRIX trials but not with the ACUITY and HORIZON trials. However, the latter 2 trials were conducted in different clinical settings when PCI was predominantly performed via the femoral artery, with much higher utilisation of glycoprotein IIb/IIIa inhibitors and no second-generation P2Y₁₂ inhibitors such as prasugrel and ticagrelor.

Bivalirudin has been used in Sweden since 2005. The utilisation of the antithrombotic agent had been steadily increasing, and it reached a plateau between 2012 and 2013 when ~80% of all PCI procedures were performed with bivalirudin. However, after the publication of VALIDATE-SWEDEHEART in 2017, the utilisation of bivalirudin decreased radically. Since 2018, it has been used in <1% of all PCI procedures in Sweden (Figure 1A).

The recently published BRIGHT-4 study has reignited interest in bivalirudin4. In BRIGHT-4, anticoagulation with bivalirudin plus a high-dose infusion post-PCI significantly reduced the 30-day composite rate of all-cause mortality or Bleeding Academic Research Consortium (BARC) types 3-5 major bleeding compared with heparin monotherapy in patients with STEMI undergoing primary PCI. These data conflict with the VALIDATE-SWEDEHEART study. The discrepancy between VALIDATE-SWEDEHEART and BRIGHT-4 could be related to significant population differences (Table 1). Patients with STEMI in VALIDATE5 were older, had fewer previous strokes, had less post-infarction heart failure, were less often treated with clopidogrel, had diabetes less often, and were less often active smokers. Indeed, the significant interaction between treatment and the CRUSADE score strongly suggests the limited external validity of BRIGHT-4, at least for Swedish patients.

Another reason for the conflicting results may be due to the more basic features of the scientific method concerning the concept of causality. It is a generally accepted fact that causality established within the boundaries of a tightly controlled RCT may not replicate in other settings. The inability to reproduce results is due to the lack of supporting factors in the chain of causation present within the specific RCT environment but absent in settings outside the specific RCT environment (Figure 2). This critical limitation of RCTs has been discussed in detail elsewhere2; however, despite the subject matter’s theoretical and practical importance for medical decision-making, it is not sufficiently taken into account.

Can the contradictory results from VALIDATE-SWEDEHEART and BRIGHT-4 both be accurate simultaneously? The answer is yes. Deaton and Cartwright suggest that causality determined by well-designed and -executed RCTs is “local”, meaning it may not be applicable under different conditions beyond those tested in the study. In other words, the conditions present in the RCT are not the sole factors required for the causal relationship, but they are essential for the observed effect. When these conditions change, the causal relationship may no longer hold true2. Many different things contribute to the cause of an event, but none of them are enough by themselves to cause it. These different things are necessary, but not sufficient, which means that they need to work together to create the condition that leads to the event. Some of these links in the chain of causation will be present in the RCT environment but may be absent from the specific environment where the intervention is implemented (e.g., hospital, region, country). This concept can be illustrated by the “Rube Goldberg machine” (Figure 2).

The argument is that VALIDATE-SWEDEHEART and BRIGHT-4’s contradictory, but valid, results support this concept. The 30-day mortality rate for patients with STEMI in Sweden did not change significantly after the discontinuation of bivalirudin use (Figure 1B). If the “BRIGHT-4 effect” (25% reduction in 30-day mortality) was present in Sweden, we should be able to detect an increase in mortality among STEMI patients after 2018.

Due to the limitations of RCTs, phase 4 postapproval studies should be used to introduce novel therapies, diagnostic procedures, health programs, and other interventions that aim to improve our societies’ health and well-being cost-effectively. Integrating RCTs into the processes of a healthcare organisation has the potential to increase our knowledge and improve the efficacy of healthcare significantly. We should only invest in the most effective treatments and promptly abandon those that do not work within our healthcare systems, which are often different from the conditions created within RCTs.

While important limitations of RCTs are regularly discussed among academics, in scientific literature, and to some extent at scientific meetings, healthcare providers, who are the primary users and implementers of RCT-generated knowledge, discuss it insufficiently.

Should we resume using bivalirudin in Sweden considering BRIGHT-4? The answer is no. In the VALIDATE-SWEDEHEART study, we tested the hypothesis of whether bivalirudin is superior to heparin and demonstrated that bivalirudin is not superior under the conditions that exist in Sweden; however, bivalirudin could work under other causal conditions or in other environments (e.g., other countries).

Figure 1. Utilisation of bivalirudin in Sweden. A) Utilisation of bivalirudin during percutaneous coronary intervention in Sweden from 2005 to 2022 in patients with ST-elevation myocardial infarction (STEMI). The bar plot shows the 30-day mortality rate as percentages for each year. The corresponding counts are also provided above the bars. The Y-axis values range up to 80% to emphasise the differences in utilisation of bivalirudin over the years. B) Mortality at 30 days in patients with STEMI in Sweden between 2005 and 2022.

Table 1. Differences in patient characteristics between BRIGHT-4 and VALIDATE-SWEDEHEART.

Figure 2. Rube Goldberg machine. A causal chain can be conceptualised as a sequence of events in which one occurrence precipitates another, ultimately culminating in an outcome. This can be likened to the intricate workings of a Rube Goldberg machine, where an initial trigger activates a cascade of interconnected mechanisms, ultimately leading to a desired result. For instance, the vibration of an alarm clock on a table might instigate a series of interactions (A-I) that concludes with the stapling of papers. Such machines aptly illustrate the interconnectedness of events and the potential consequences that may arise from them. However, like the elaborate nature of Rube Goldberg machines, real-life situations often exhibit a greater degree of complexity than that which may be initially apparent. It is essential to recognise that factors of significance within a controlled experimental setting, such as a randomised controlled trial, may not necessarily be present or have the same impact in real-world contexts. Consequently, it is imprudent to assume that findings derived from experimental studies will consistently translate to real-world applications.

Conflict of interest statement

The author has no conflicts of interest to declare.