Assessing the performance of the PRECISE-DAPT and PARIS risk scores for predicting one-year out-of-hospital bleeding in acute coronary syndrome patients

Abstract

Aims: The PRECISE-DAPT and PARIS risk scores (RSs) were recently developed for bleeding risk assessment in percutaneous coronary intervention (PCI) patients treated with dual antiplatelet therapy (DAPT). We aimed to assess the performance of these RSs for predicting out-of-hospital bleeding in patients with acute coronary syndrome (ACS).

Methods and results: Retrospectively, we studied 1,926 consecutive ACS patients treated with PCI and DAPT. The performance of RSs for predicting one-year BARC type 2, 3 or 5 bleeding and BARC type 3 or 5 bleeding was assessed and compared. Both RSs were effective for the prediction of bleeding events. For BARC type 2, 3 or 5 bleeding, the c-statistic values for PRECISE-DAPT and PARIS were 0.61 and 0.63 (p=0.29), respectively. The two scores displayed equal c-statistics of 0.73 for predicting BARC type 3 or 5 bleeding. PARIS significantly outperformed PRECISE-DAPT in terms of indices of categoryless net reclassification improvement and integrated discrimination. Decision curve analyses also favoured PARIS.

Conclusions: Within our cohort, PARIS and PRECISE-DAPT were fairly to moderately effective for the prediction of bleeding. Their predictiveness varies according to the bleeding severity. PARIS-derived bleeding risk assessment was associated with a higher net benefit compared to PRECISE-DAPT-based bleeding risk assessment.

Abbreviations

ACS: acute coronary syndrome

BARC: Bleeding Academic Research Consortium

DAPT: dual antiplatelet therapy

PARIS: patterns of non-adherence to antiplatelet regimens in stented patients

PCI: percutaneous coronary intervention

PRECISE-DAPT: predicting bleeding complications in patients undergoing stent implantation and subsequent dual antiplatelet therapy

Introduction

Dual antiplatelet therapy (DAPT) represents the cornerstone in the secondary prevention of thrombotic events in acute coronary syndrome (ACS)1. However, the benefit of such therapy can be counteracted by an increase in haemorrhagic complications which may be associated with a risk of mortality similar to or even greater than coronary thrombotic events2-4.

Although 12 months of DAPT is still the standard of care in ACS patients1, several studies have suggested that 12 months of DAPT may not be necessary after drug-eluting stent (DES) implantation in selected patients and that six months or even three months of DAPT may be sufficient, as the benefits of 12 months of DAPT may be outweighed by the risk of bleeding5-10. On the other hand, recent studies suggest that ticagrelor and prasugrel are not infrequently prescribed to patients at high risk of bleeding with low risk of ischaemic events11,12, suggesting a lack of concordance between perceived and actual risk in the ACS process of care. Integrating a quantitative risk tool into clinical practice may address the aforementioned gaps, thereby leading to more rational use of therapeutic resources and improving clinical outcomes.

The PRECISE-DAPT13 and PARIS14 risk scores (RSs) have recently been developed to help physicians in stratifying post-discharge bleeding risk in ACS patients treated with DAPT after in-hospital percutaneous coronary intervention (PCI). Both prediction scales performed moderately well in their development cohorts (c-statistic ~0.70). However, the comparative predictiveness of these scales in ACS patients is still poorly characterised.

Accordingly, in this study, we assessed the clinical utility of the PRECISE-DAPT and PARIS RSs in the prediction of post-discharge bleeding in contemporary ACS patients treated with PCI and DAPT. We tested the hypothesis that the PRECISE-DAPT score would perform at least as well as the PARIS score in predicting bleeding complications over follow-up.

Methods

PATIENT POPULATION

This was a retrospective observational study based on the CardioCHUVI (Cardiología del Complejo Hospitalario Universitario de VIgo) PCI registry, in which the study subjects were all patients with a final diagnosis of ACS admitted consecutively between January 2012 and March 2015 to our department, and who were treated with in-hospital PCI.

The initial cohort of the present study comprised 2,064 patients. Since the aim of the present study was to assess bleeding risk during follow-up in patients treated with in-hospital PCI who received DAPT at discharge, we excluded patients who died in-hospital (n=74) and those discharged without DAPT (n=43). Also excluded were patients with missing data for computing the RSs and/or with missing data on bleeding status over follow-up (n=21). Therefore, 1,926 patients constituted the final study cohort. The study complied with the Declaration of Helsinki.

ENDPOINTS AND DEFINITIONS

Our primary endpoint focused on the clinical utility of the PRECISE-DAPT and PARIS scores for classifying the risk of one-year out-of-hospital bleeding in ACS patients treated with in-hospital PCI and DAPT at discharge.

In PRECISE-DAPT, two bleeding events were considered: Thrombolysis In Myocardial Infarction (TIMI) major bleeding, and TIMI major or TIMI minor bleeding, while, in PARIS, bleeding was defined as Bleeding Academic Research Consortium (BARC) type 3 or 5 bleeding.

In this study, we compared both scores regarding their capacity to predict BARC type 2, 3 or 5 and BARC type 3 or 5 bleeding15, at one year after hospital discharge.

Our study only includes the first bleeding event that occurred over follow-up. Adjudication of events was performed by personnel unaware of the point scores of interest for this analysis.

The ascertainment of bleeding status during follow-up was carried out between February and March 2016. For each patient we reviewed the electronic medical record and all the medical attendances and hospital records.

STATISTICAL ANALYSIS

Baseline and clinical characteristics were compared between this registry population and the derivation cohorts of the PRECISE-DAPT and PARIS scores. Continuous variables are presented as mean±standard deviation (SD) and medians (interquartile ranges [IQR]), to enable comparison of our clinical data with those from the PRECISE-DAPT and PARIS cohorts.

Categorical variables are presented as proportions. Differences between continuous variables were compared using the Wilcoxon rank-sum test and Student’s t-test. The χ2 test was used to compare the categorical variables.

RISK SCORE CALCULATION AND CATEGORISATION

Bleeding RSs in each patient were calculated on the basis of the definitions used in their development cohorts (Supplementary Table 1, Supplementary Figure 1)13,14.

The PRECISE-DAPT scoring system assigns patients into four risk strata (very low: ≤10, low: 11-17, moderate: 18-24, and high: ≥25 points), whereas PARIS categorises patients into three risk groups (low: <3, moderate: 3-7, and high: ≥8 points).

For our study, patients were categorised into three bleeding risk strata for all scores by considering the very low and low risk categories in PRECISE-DAPT as a unique category (i.e., low risk). This was done to enable comparisons between the RSs regarding their bleeding risk classification systems.

PERFORMANCE OF THE RISK SCORES

The total RSs, as continuous variables, were entered into separate Cox regression models to test their association with bleeding events.

The ability to separate high bleeding risk patients from lower bleeding risk patients was checked by Kaplan-Meier curves and compared using the log-rank test. The magnitude of the association between each of the three predefined risk categories from the RSs was calculated and expressed as hazard ratios (HR) with their 95% confidence intervals (95% CI); the low risk category was considered as a reference category.

Indices of discrimination and calibration were used to assess the RSs’ performance. To assess discrimination, using the total RS as a global prognostic indicator, we calculated and compared the Harrell c-statistic for censored time-to-event data, for both scores16. We further assessed the net reclassification improvement index (NRI) and integrated discrimination index (IDI)17. For the NRI calculation, individuals were compared based on their bleeding risk using the three categories of the two RSs13,14. Since the probability of bleeding was set at different thresholds in the respective risk categories of PRECISE-DAPT and PARIS, we further analysed possible improvement in the discrimination ability of one score vs. the other by means of the “categoryless” NRI17.

We assessed calibration by using the Grønnesby and Borgan χ2 test, and plotted observed vs. predicted outcomes.

Decision curve analyses (DCA) were used to quantify the net benefit of the prediction scores; the higher the net benefit, the better the RS, in terms of clinical usefulness17. The theoretical range of net benefit is from negative infinity to the incidence of disease.

A two-sided p<0.05 was considered statistically significant. All statistical analysis was performed using Stata 13 (StataCorp, College Station, TX, USA) and the statistical package for R 3.2.1 (R Foundation for Statistical Computing, Vienna, Austria).

Results

PATIENT CHARACTERISTICS

Compared to the PRECISE-DAPT derivation cohort, CardioCHUVI patients showed significantly different clinical features (Table 1). In relation to the five variables comprising PRECISE-DAPT, CardioCHUVI patients had a similar age and prior bleeding rate, but had significantly higher haemoglobin, white blood cell counts, and CrCl values.

The rate of prasugrel prescription was similar between PRECISE-DAPT and this cohort. Only 3.9% of patients were treated with ticagrelor in PRECISE-DAPT compared to 22.9% in CardioCHUVI (p<0.001).

Compared to PARIS, patients in CardioCHUVI also showed significantly different clinical characteristics. With respect to the six variables comprising PARIS, CardioCHUVI patients were older, had lower body mass index, and higher prevalence of active smoking as well as CrCl <60 mL/min. The prevalence of anaemia was similar between the two cohorts. The rate of triple therapy at discharge in CardioCHUVI was 8.2% vs. 4.8% in PARIS (p<0.001). Clopidogrel prescription was significantly higher in PARIS compared to CardioCHUVI, but the prasugrel prescription rate was similar. No patients were treated with ticagrelor in PARIS vs. 22.9% in CardioCHUVI.

PRECISE-DAPT varies from 5 to 78 points, whereas PARIS values range from 0 to 15 points.

While both RSs classified the same proportion of patients into low risk strata, there were substantial differences between the scores regarding the definition of intermediate/high bleeding risk (Figure 1). PARIS categorised significantly more patients as having moderate risk compared to PRECISE-DAPT (42.6% vs.19.8%; p<0.001). In contrast, PRECISE-DAPT assigned more patients to the high risk category compared to PARIS (39.4% vs. 14.6%; p<0.001).

Figure 1. Distribution of patients in the risk categories of the PRECISE-DAPT and PARIS scores.

BLEEDING RATES DURING FOLLOW-UP

During 331±71 days, one hundred and thirty-six (7.1%) patients developed BARC type 2, 3 or 5 bleeding, whereas 53 (2.8%) patients suffered BARC type 3 or 5 bleeding. Mean time to BARC type 2, 3 or 5 bleeding was 178±10 days, and 168±17 days for BARC type 3 or 5 bleeding.

The rate of BARC type 2, BARC type 3, and BARC type 5 bleeding was 4.3% (n=83), 2.6% (n=51), and 0.1% (n=2), respectively.

Most bleeding events originated from gastrointestinal (40.4%; n=55/136) and genitourinary (n=24/136=17.6%) tracts. The intracranial bleeding rate was 6.6% (n=9/136).

BLEEDING RISK ASSESSMENT BASED ON THE RISK CATEGORIES OF THE RSs

One-year Kaplan-Meier curves using the three risk categories in this study are shown in Figure 2 and Figure 3. Both RSs showed highly significant predictive prognostic values (log-rank test, p<0.001). The observed bleeding rates for the two scores increased monotonically from low- to high-risk categories for both bleeding events. However, Kaplan-Meier curves diverged in a more pronounced way with PARIS (for BARC type 2, 3 or 5 bleeding, χ² values were 29 for PARIS vs. 11 for PRECISE-DAPT; for BARC type 3 or 5 bleeding, χ² values were 55 vs. 27, respectively).

Figure 2. One-year Kaplan-Meier curves for BARC type 2, 3 or 5 bleeding. A) Using PRECISE-DAPT risk strata. B) Using PARIS risk strata.

Figure 3. One-year Kaplan-Meier curves for BARC type 3 or 5 bleeding. A) Using PRECISE-DAPT risk strata. B) Using PARIS risk strata.

The risk of bleeding was more strongly captured by the PARIS classification system compared to the PRECISE-DAPT classification system, especially in the high-risk strata, as seen by the HR values in Figure 2 and Figure 3.

DISCRIMINATION

The c-statistic values indicated modest discriminative power for both scores for predicting BARC type 2, 3 or 5 bleeding (c-statistic=0.61 and 0.63 for PRECISE-DAPT and PARIS, respectively; p=0.29 for comparison). However, for predicting BARC type 3 or 5 bleeding, the c-statistic values showed moderate discriminative power (c-statistic=0.73 for both scores) (Table 2).

When using a categoryless NRI index and IDI, PARIS significantly outperformed PRECISE-DAPT for both bleeding events (Table 2).

CALIBRATION

Calibration of observed against predicted bleeding was good for both RSs, although with PARIS the predicted probability approximated more closely the observed probability than did PRECISE-DAPT (Figure 4).

Figure 4. Observed and predicted rates of BARC type 2, 3 or 5 bleeding and BARC type 3 or 5 bleeding by PRECISE-DAPT (red circles) and PARIS (blue rhombus) risk scores. A) BARC type 2, 3 or 5 bleeding. B) BARC type 3 or 5 bleeding.

DECISION CURVE ANALYSES

Figure 5 shows comparisons of the decision curves from classifying individuals using the RSs, assuming that all patients will bleed (all positive or all are at high risk of bleeding), and assuming that no patients will bleed (all negative or all are at low risk of bleeding; horizontal line at zero).

Figure 5. Decision curves for the PRECISE DAPT- and PARIS-derived risk thresholds for predicting bleeding. A) BARC type 2, 3 or 5 bleeding. B) BARC type 3 or 5 bleeding.

The DCA showed that both PRECISE-DAPT and PARIS appear to be superior to the scenario of not using the RSs, irrespective of the outcome of interest.

When compared to the strategy of “assuming all patients as low risk”, the two RSs were superior at any risk threshold probability.

When compared to the strategy of “assuming all as high risk”, PRECISE-DAPT and PARIS were only superior at a risk threshold of >4% for BARC type 2, 3 or 5 bleeding (Figure 5A, Supplementary Table 2), and at a risk threshold of ≥1% for BARC type 3 or 5 bleeding (Figure 5B, Supplementary Table 3).

When comparing the DCA of both scores, PARIS was superior to PRECISE-DAPT for the two bleeding outcomes. For BARC type 2, 3 or 5 bleeding, applying the PARIS-derived probability at a risk threshold of 4%, the use of PARIS would result in a net benefit gain of +3.3% (+3.2% for PRECISE-DAPT), compared with the scenario of “assuming all as low risk”, and of +0.1% (0.0% for PRECISE-DAPT), compared with the scenario of “assuming all as high risk” (Supplementary Table 2). In other words, the net benefit using PARIS leads to the equivalent of 79.2 (vs. 76.8 for PRECISE-DAPT) and 2.4 (vs. 0 for PRECISE-DAPT) fewer false-positive results per 100 patients compared to “assuming all as low risk” and “assuming all as high risk”, respectively.

For BARC type 3 or 5 bleeding, the DCA also showed that PARIS was better than PRECISE-DAPT. Applying a PARIS-derived probability at a bleeding risk threshold of 1%, the use of PARIS would result in a net benefit gain of +1.8% (+1.6% for PRECISE-DAPT) compared with the scenario of “assuming all as low risk”, and of +0.1% (‒0.1% for PRECISE-DAPT) compared with the scenario of “assuming all as high risk” (Supplementary Table 3). In other words, the net benefit using PARIS leads to the equivalent of 178.2 (vs. 158.5 for PRECISE-DAPT) and 9.9 (vs. ‒9.9 for PRECISE-DAPT) fewer false-positive results per 100 patients compared to “assuming all as low risk” and “assuming all as high risk”, respectively.

Discussion

In this validation study, we compared, for the first time, the PRECISE-DAPT and PARIS scores for the prediction of one-year out-of-hospital bleeding in ACS patients treated with in-hospital PCI and DAPT at discharge. We found that PRECISE-DAPT and PARIS were effective for the prediction of both of the bleeding events being studied. The discriminatory capacity exhibited by both scores was moderately good in terms of c-statistic values for predicting BARC type 3 or 5 bleeding, although their predictiveness was rather modest at predicting BARC type 2, 3 or 5 bleeding.

In our analysis, the use of both RSs was superior to the strategies of not using the RSs for bleeding risk classification, as observed in the DCA. This means that the use of PRECISE-DAPT and PARIS is of clinical value to guide clinical decisions in bleeding risk stratification. However, PARIS was superior to PRECISE-DAPT, as shown by the categoryless NRI, IDI, and DCA. This held true for BARC type 2, 3 or 5 bleeding as well as for BARC type 3 or 5 bleeding.

Importantly, in the DCA, the net benefit with PARIS compared to PRECISE-DAPT emerged at a risk threshold of 1% for BARC type 3 or 5 bleeding. This finding is particularly important since BARC type 3 or 5 bleeding is considered as a major bleeding event and taking into account that the superiority of PARIS, in relation to PRECISE-DAPT, has emerged at a clinically reasonable risk threshold.

Nevertheless, for BARC type 2, 3 or 5 bleeding, the DCA showed that the superiority of PARIS compared to PRECISE-DAPT arose at a bleeding risk threshold of 4% or above. Since this cut point indicates per se a high risk of bleeding, the incremental prognostic value of PARIS over PRECISE-DAPT in this setting is conspicuous by its absence. However, it is important to note that both scores were never worse than the strategy of “not using the RSs” and, because use of these scores is based on routinely collected data, it has no obvious downside. Therefore, the scores would be of use for clinicians who, at least some of the time, need to support their judgement by a prediction tool if a patient’s predicted probability of a bleed is low.

The relative performance of the different RSs can be explained, at least partially, by their respective composition and prevalence of risk factors18. For instance, the rate of triple therapy prescription at discharge, in our data set, was 8.4% compared to 0% in PRECISE-DAPT (5.1% in PARIS). Data from prior studies demonstrated that there is a threefold to fivefold increase in bleeding rates associated with triple therapy compared with various combinations of DAPT19,20. Indeed, we found that the strongest predictor of bleeding was triple therapy prescription at discharge (HR ~3) (Supplementary Table 4).

On the other hand, the PARIS score was derived from a “real-world” population, in contrast to the more selected derivation population of the PRECISE-DAPT study which included data from eight randomised clinical trials.

PRECISE-DAPT was externally validated in the Bern PCI registry which included “all” PCI patients13. In contrast to our results, in the Bern PCI registry, PRECISE-DAPT outperformed PARIS. However, individuals in our study were all ACS patients while in Bern PCI only 54% had ACS, and, as in the PRECISE-DAPT study, no patient in the Bern PCI external validation cohort was treated with triple therapy. Reconciling these inconsistent results is challenging as they concern different populations and different types and definitions of events.

Limitations

This was a retrospective study with the limitations inherent to this type of design. Moreover, we used treatment at discharge as a principle of intention-to-treat analysis, as we did have data on DAPT duration during follow-up. However, this principle was also applied in the PLATO and Bern PCI external validation cohorts used in the development of the PRECISE-DAPT score, and in the PARIS development cohorts13,14.

Our findings should be interpreted in the context of the use of clopidogrel in 69.8% and 21.7% of bare metal stent (BMS) patients. The number of bleeding events did not allow analysis by subgroup (BMS vs. DES and clopidogrel vs. prasugrel or ticagrelor) that would have been ideal to generalise the study findings to the overall set of ACS patients.

Of note, the PARIS score was developed to predict the risk of bleeding and thrombotic coronary events. In this study we only validated the version for bleeding risk prediction. However, clinicians should firstly be assured that PARIS bleeding estimates are not misleading. In this regard, our study is still of clinical value.

BARC criteria were used to define bleeding in our study and in PARIS, in contrast to PRECISE-DAPT where bleeding definitions were based on TIMI criteria. This point could have affected the comparability of the scores. However, BARC bleeding criteria were also used as an alternative bleeding definition in the external validation cohorts of PRECISE-DAPT. Additionally, BARC bleeding criteria are currently considered the standard bleeding definition. Finally, in this study, data on risk scores and/or bleeding status during follow-up were not available in 21 patients.

Conclusions

In conclusion, we compared for the first time the PRECISE-DAPT and PARIS scores for the prediction of out-of-hospital bleeding in real-life ACS patients treated with PCI and DAPT. Within our cohort, although both scores were effective at predicting bleeding, the PARIS score appeared to be superior to the PRECISE-DAPT score at stratifying one-year out-of-hospital BARC type 2, 3 or 5 bleeding and BARC type 3 or 5 bleeding. In the context of a comprehensive clinical assessment process, this tool may support clinical decision making for DAPT type and duration.

Conflict of interest statement

The authors have no conflicts of interest to declare.

Supplementary data

Supplementary Table 1. Variables comprising the PARIS bleeding risk score.

Supplementary Table 2. Net benefit of using the PRECISE-DAPT and PARIS scores compared to alternative strategies for identifying BARC type 2, 3 or 5 bleeding risk conditional on different risk thresholds.

Supplementary Table 3. Net benefit of using the PRECISE-DAPT and PARIS scores compared to alternative strategies for identifying BARC type 3 or 5 bleeding risk conditional on different risk thresholds.

Supplementary Table 4. Association (hazard ratio) between the predictors comprising the PRECISE-DAPT and PARIS risk scores and bleeding events as obtained in CardioCHUVI.

Supplementary Figure 1. Variables comprising the PRECISE-DAPT bleeding risk score.

To read the full content of this article, please download the PDF