TAVI risk scoring using established versus new scoring systems: role of the new STS/ACC model

Abstract

Aims: The aim of this study was to validate the recently developed STS/ACC TAVR in-hospital mortality risk score for predicting in-hospital mortality after transcatheter aotic valve implantation (TAVI) and to compare its ability to predict 30-day mortality with that of four other established risk models (EuroSCORE I, EuroSCORE II, STS-PROM, and German AV Score).

Methods and results: The study cohort included 946 consecutive patients who underwent TAVI between 2013 and 2015. Each of the five scores was fitted as a continuous linear variable into a logistic regression model estimating 30-day mortality. The STS/ACC TAVR score was additionally analysed for in-hospital mortality. C-statistics and likelihood ratio (LR) test p-values were estimated for each model to describe the model fit. The ability of the STS/ACC score to predict in-hospital mortality was similar to the reported STS/ACC TVT registry data (this study’s C-statistic 0.65 vs. STS/ACC TVT registry 0.66). The STS-PROM score (C-statistic=0.68; LR p<0.0001) and the new STS/ACC TAVR score (C-statistic=0.68; LR p<0.0001) were superior to the other scores (EuroSCORE I [C-statistic=0.55; LR p=0.02], EuroSCORE II [C-statistic=0.58; LR p=0.02], German AV Score [C-statistic=0.62; LR p<0.01]) for prediction of 30-day mortality.

Conclusions: These data show the superiority of the STS-PROM and STS/ACC TAVR scores compared with other existing risk calculation models in predicting 30-day mortality after TAVI in a German all-comers population. The STS/ACC TAVR score, however, is easier to calculate (12 vs. 28 variables), and may thus gain wider acceptance and be accompanied by improved inter-observer reliability.

Abbreviations

ES I: EuroSCORE I

ES II: EuroSCORE II

GARY: German Aortic Valve Registry

German AV Score: German Aortic Valve Score

LR: likelihood ratio

SAVR: surgical aortic valve replacement

STS/ACC TAVR: Society of Thoracic Surgeons/American College of Cardiology TAVR in-hospital mortality risk score

STS/ACC TVT: Society of Thoracic Surgeons/American College of Cardiology Transcatheter Valve Therapy

STS-PROM: Society of Thoracic Surgeons Predicted Risk of Operative Mortality

TAVI: transcatheter aortic valve implantation

Introduction

Transcatheter aortic valve implantation (TAVI) is being used increasingly worldwide, especially since outcomes comparable to those of conventional surgery have been demonstrated for high-risk patients1. In addition, the indication for using TAVI has been gradually extended to lower-risk patients2,3. Although accurate risk prediction is essential for patient-centric care and informed consent, a reliable risk score model estimating the TAVI procedural success is not yet available, despite more than ten years of clinical experience with TAVI. In order to design a TAVI-specific risk score based on a larger cohort, the Society of Thoracic Surgeons/American College of Cardiology (STS/ACC) Transcatheter Valve Therapy (TVT) Registry TAVI in-hospital mortality risk score (STS/ACC TAVR score) was recently published4. It is based on the STS/ACC TVT Registry that captures all commercial TAVI procedures performed in the USA4.

Although a validation cohort was used for internal validation, this risk score has not been externally validated. The present study was designed to evaluate the ability of this new model to predict in-hospital and 30-day mortality in an all-comers German TAVI cohort and to compare it with previously established risk scores in terms of the ability to predict 30-day mortality.

Methods

The study cohort included 946 consecutive patients who underwent TAVI between January 2013 and December 2015 at our centre. Data from our institutional database for these patients were used to calculate procedural risk scores, including EuroSCORE I (ES I)5, EuroSCORE II (ES II)6, Society of Thoracic Surgeons Predicted Risk of Operative Mortality (STS-PROM)7, German Aortic Valve Score (German AV Score)8, and the newly developed STS/ACC TAVR score (Table 1)4. In-hospital and 30-day mortality was available for all patients. Each of the five procedural scores was fitted as a continuous linear variable into a logistic regression model estimating 30-day mortality. The analysis was repeated by modelling each score using restricted cubic splines with three knots9,10. The C-statistic, Somer’s D, Brier score, and likelihood ratio (LR) test p-values were estimated for each of the ten 30-day models to describe the model discrimination and calibration. Receiver operating characteristic (ROC) curves were generated for each score, and differences between the scores’ C-statistics and 95% confidence intervals were calculated, with p-values being adjusted for the multiple testing false discovery rate. To assess calibration visually, plots of observed versus predicted mortality were generated for each model, with Hosmer-Lemeshow goodness-of-fit p-values estimated for all models. The STS/ACC in-hospital mortality score was additionally analysed for in-hospital mortality. All analyses were conducted using SAS Version 9.3 (SAS Institute Inc., Cary, NC, USA). The study was approved by the Institutional Review Board at our centre.

Results

A total of 946 patients (median age 82.1 [interquartile range 78.2-85.8] years; 51.5% female) underwent TAVI during the study period. Common comorbidities were chronic obstructive pulmonary disease (20.4%), prior stroke (14.6%), coronary artery disease (64%) and prior cardiac surgery (21.7%). Baseline characteristics are listed in Table 2. The rate of transapical TAVI was high (37.2%) and decreased over the study period. Risk scores and in-hospital and 30-day mortality data were available for all patients. As indicated by the risk scores (median ES I 21.1, ES II 5.0, STS-PROM 5.0) and comorbidities, most patients were at high or intermediate risk, which is representative of the current clinical practice in Germany11. During the hospital stay, 46 patients (4.9%) died; mortality at 30 days was 6.3% (60/946).

The C-statistic for the STS/ACC TAVR score for the prediction of in-hospital mortality in our cohort was 0.65 (Figure 1). The C-statistics for the STS/ACC TAVR (0.68) and STS-PROM (0.68) scores for prediction of 30-day mortality indicate superiority to the other scores (ES I [0.55], ES II [0.58] and German AV Score [0.62]; using either model - continuous variables or cubic splines) (Table 3, Figure 2). In comparing the C-statistics, 30-day mortality was predicted significantly better by the STS/ACC TAVR score than by ES I (p=0.023) or ES II (p=0.055). The STS-PROM score was also superior to ES I (p=0.023) and ES II (p<0.01). The calibration plots of observed versus predicted mortality (Figure 3), Somer’s D, and Brier score (Table 3, Figure 3) confirmed these results.

Figure 1. Ability of the STS/ACC score to predict in-hospital mortality. Receiver operating characteristic (ROC) plot showing the prognostic values of the STS/ACC TAVR score for the prediction of in-hospital mortality. STS/ACC TAVR score: Society of Thoracic Surgeons/American College of Cardiology TAVR in-hospital mortality risk score

Figure 2. Ability of five different risk scores to predict 30-day mortality. Receiver operating characteristic (ROC) plots showing the prognostic values of the STS/ACC TAVR score, STS-PROM, ES I, ES II and German AV Score for the prediction of 30-day mortality. ES I: EuroSCORE I; ES II: EuroSCORE II; German AV Score: German Aortic Valve Score; STS/ACC TAVR score: Society of Thoracic Surgeons/American College of Cardiology TAVR in-hospital mortality risk score; STS-PROM: Society of Thoracic Surgeons Predicted Risk of Operative Mortality

Figure 3. Calibration plots for splined risk models. Calibration plots showing deciles of observed 30-day mortality versus predicted 30-day mortality for the STS/ACC TAVR score, STS-PROM, ES I, ES II and German AV Score. ES I: EuroSCORE I; ES II: EuroSCORE II; German AV Score: German Aortic Valve Score; STS/ACC TAVR score: Society of Thoracic Surgeons/American College of Cardiology TAVR in-hospital mortality risk score; STS-PROM: Society of Thoracic Surgeons Predicted Risk of Operative Mortality

Discussion

This is the first study to evaluate the STS/ACC TAVR in-hospital mortality score in a non-US population undergoing TAVI and to investigate its ability to predict 30-day mortality compared with other, established, risk models. The similarity between C-statistics for in-hospital mortality in our cohort compared with the validation cohort in the original report (0.65 vs. 0.66) shows that the risk model is able to predict in-hospital mortality in a patient cohort outside of the population in which it was developed; thus, our study provides external validation of the new score. In our German cohort, there was similar precision to the STS/TVT Registry patients4.

As TAVI was initially developed for inoperable and high-risk patients, risk assessment has always been of particular importance and is recommended by the guidelines. The developers of the STS/ACC TAVR score opted for an in-hospital model, as complete data were easier to capture for this outcome than 30-day mortality. Data were thus available for almost all patients in the registry (only 1.2% missing data for in-hospital mortality in the registry)4. Thirty-day mortality, however, is the standard criterion that is routinely assessed for preoperative evaluation and recommended by the guidelines. We demonstrate that the STS/ACC TAVR score can be used for prediction of 30-day outcome and performs better than other scores for this purpose (ES I and II and German AV Score).

The standard models for risk stratification in cardiac patients are currently the STS-PROM in the USA and the ES I and ES II in Europe12,13. All of these models, however, were designed for conventional cardiac surgery and are based on a patient population that is different from the typical cohort of elderly TAVI patients with comorbidities. Consequently, several studies have shown the inability of these models to assess procedural risk properly14-17. Some groups demonstrated superiority of STS-PROM over ES I and II14,18, whereas others favoured ES II19,20. The present analysis confirms that STS-PROM outperforms ES I and II and is on a par with the new STS/ACC in-hospital mortality score for TAVI patients.

In an effort to achieve more accurate risk assessment, various TAVI- or aortic valve-specific risk scores have been developed over the last few years8,21-23. Different strategies were used for this purpose. Some groups developed and evaluated their model based on multicentre data sets. For example, Seiffert and colleagues developed their model based on data from two hospitals and evaluated the score on patients who underwent TAVI at a third centre23. However, due to a small event rate after 30 days and a small patient cohort, their study was not powered to develop a risk model for 30-day mortality; therefore, their one-year mortality risk score had a low C-statistic of 0.6. Similarly, the TAVI2-SCORe was developed to predict one-year mortality based on 511 patients who underwent TAVI at two centres in the Netherlands and Italy21. The authors claimed that in their analysis the TAVI2-SCORe outperformed STS, ES I, and ES II; however, the performance of the score was evaluated on the same patient cohort that was used for model development. This might invalidate the results.

As it is necessary to use larger cohorts to develop reliable models for 30-day mortality, registry data are used to design a specific risk score. The FRANCE-2 risk score is a 21-point predictive score based on the French Aortic National CoreValve and Edwards (FRANCE 2) registry24. The development cohort consisted of 2,552 patients and the validation cohort 1,281 patients. The C-statistic of the development cohort was 0.67; however, it proved to be only 0.59 in the validation cohort and thus this risk score was not superior to the established models.

The German AV Score, which was published in 2013, uses national registry data and includes both TAVI (n=573) and surgical aortic valve replacement (SAVR) (n=10,574) patients8. Thus, it uses a more disease-specific (rather than procedure-specific) approach. By including SAVR patients, the model was designed to help inform decision making regarding SAVR and TAVI. As with the STS/ACC TAVR and FRANCE-2 scores, it is based on a large registry and captures all TAVI procedures performed nationwide. In contrast to the US STS/ACC TVT Registry, the German Aortic Valve Registry (GARY) is not limited to commercial TAVI procedures25,26. Kötting and colleagues reported a C-statistic of 0.808, which is, to our knowledge, the highest value reported for a TAVI score; however, the score was evaluated on the same cohort that was used for risk development and only 5% of the patients underwent TAVI8. Thus, none of the recently developed TAVI risk score models, including TAVI2-SCORe, FRANCE-2, and the German AV Score, has proved to be superior to previously established models.

Limitations

Our study has a few limitations. Although we included almost 1,000 patients in this study, larger and more geographically diverse patient cohorts are needed to confirm our findings. This is especially important as each of the risk scores considered here was developed based on different patient populations. Another principal limitation of this analysis is the retrospective validation of the STS/ACC model. However, the data were collected prospectively. This method allows prompt validation of the risk score model after its development. Comparing the new risk model to other TAVI-specific scores, such as the OBSERVANT or FRANCE-2 score, would help to determine the clinical significance of the new model compared to existing ones. However, the currently available data in our database do not allow us to calculate these scores. Thus, we focused on the most commonly used risk models which are recommended by the current guidelines. In addition, we report the ability of the STS/ACC TAVR in-hospital mortality risk score to be reasonably accurate in predicting 30-day mortality although it was only designed for in-hospital mortality. Our assessment, however, showed the score to be superior to several commonly used 30-day risk models.

Conclusions

Here we demonstrate that, although the STS/ACC in-hospital mortality score was developed based on a US population and was designed for in-hospital mortality prediction, it is superior to the German AV Score and ES I and II in predicting 30-day mortality after TAVI. Although it requires significantly fewer variables (12 vs. 28) for risk assessment and can thus be applied more easily and quickly, we found it to be on a par with the STS-PROM score. Although all of these data support the STS-PROM and STS/ACC TVT TAVR in-hospital mortality scores for risk estimation, it is important to keep in mind that none of the models’ C-statistics exceeds 0.7, the cut-off generally considered as satisfactory for risk prediction27. The developers of the STS/ACC in-hospital mortality score are currently working on a 30-day mortality score based on the STS/ACC TVT Registry data. Until then, both the STS-PROM and STS/ACC TAVR in-hospital score seem to be the best tools for risk prediction in TAVI patients.

Acknowledgements

We thank Elizabeth Martinson, PhD, of the KHFI Editorial Office for her editorial assistance.

Conflict of interest statement

The authors have no conflicts of interest to declare.