Abstract
Aims: We aimed to test the feasibility of calculating SYNTAX score from coronary computed tomographic angiography (CCTA) compared to from invasive coronary angiography (ICA).
Methods and results: SYNTAX score was independently and blindly calculated from CCTA and from ICA in 104 patients, age 57±10, with significant (>50%) stenoses in 1.7±0.7 vessels. The level of agreement was assessed by Cohen’s kappa. Agreement between ICA and CCTA for conventional vessel-based analysis (presence of >50% stenosis per vessel) was substantial with kappa=0.66 and sensitivity, specificity and accuracy of 74%, 90% and 80%, respectively. The mean SYNTAX score was 14.2±10.0 by ICA and 10.3±6.9 by CCTA, with a significant underestimation of 3.9±8.2 by CCTA (p<0.001). Weighted kappa was 0.33, indicating only fair agreement. When only good quality CCTA were included, kappa improved to 0.56. Analysis of the cause of the bias showed ICA to identify more lesions per patient (2.2±1.3 vs. 1.7±1.0, p<0.001), while the mean score per lesion was not different (6.4 vs. 5.9, p=ns).
Conclusions: CCTA, despite having a good agreement with ICA by conventional vessel-based analysis, showed only fair agreement for the calculation of SYNTAX score, and cannot be currently used as a substitute for diagnostic ICA for this purpose.
Introduction
The SYNergy between percutaneous coronary intervention with TAXus and cardiac surgery (SYNTAX) score was developed for, and validated as, a means of risk stratifying and discriminating outcomes of patients with complex coronary disease undergoing percutaneous coronary intervention (PCI) compared with coronary artery bypass graft surgery1, 2. More recently the SYNTAX score has been shown to be closely related to outcome after PCI, in elective patients3-8 and in the presence of acute coronary syndromes9. Therefore the SYNTAX score is currently a key determinant in therapeutic decision-making processes. The latest European guidelines on revascularisation have recommended creation of a heart team which serves the purpose of a balanced multidisciplinary decision process10. According to this approach, in stable patients with complex coronary disease, PCI should no longer be automatically performed immediately following diagnostic angiography; rather the case should be discussed by a heart team in order to procure an optimal therapeutic plan. This decision would take into account multiple factors, and among them the patient’s SYNTAX score would be an important input. The disadvantage of this approach is that, in those cases where PCI is chosen, the patient would undergo a staged procedure at a later date. The advent of coronary computed tomographic angiography (CCTA) now offers the possibility of having an accurate depiction of the coronary anatomy prior to invasive coronary angiography (ICA). This offers the enticing possibility of convening the heart team prior to ICA in order to make a tentative therapeutic decision based on the CCTA diagnosis. Moreover, if it were possible to estimate SYNTAX score, based on CCTA, the ability to make a guided decision based on anatomy, coronary disease complexity and clinical predictors would be even stronger. We thus aimed to evaluate the feasibility of calculating SYNTAX score from CCTA and compare accuracy to that of the reference standard, ICA.
Methods
STUDY POPULATION
We searched our database for all patients who underwent both ICA and CCTA within two months of each other who had at least one >50% lesion according to the ICA report. Post-coronary artery bypass patients were excluded. Approval for the study was obtained from the local ethics committee.
CCTA SCAN
The CT scan was performed on one of two 64-slice scanners (Philips Brilliance 64; Philips Healthcare, Cleveland, OH, USA) or (LightSpeed® VCT; GE Healthcare, Milwaukee, Wisconsin, USA) by a retrospective helical scanning technique with tube current modulation, following IV injection of 80 ml of contrast media. Patients with heart rate >65 bpm prior to scanning were given either oral and/or IV beta blockers with the aim of lowering the heart rate to below 60-65 bpm. Prior to CCTA, a non-contrast scan was performed to calculate the calcium score and in order to plan the CCTA scan range.
ICA PROCEDURE
Diagnostic ICA was performed by standard techniques in multiple projections on a Siemens AXIOM Artis dTC cardiac angiographic system (Siemens Healthcare, Forchheim, Germany) and stored digitally.
SYNTAX SCORE TRAINING AND ANALYSIS
Two invasive cardiologists and two experts in CCTA (one cardiologist and one radiologist) simultaneously participated in the tutorial provided by the SYNTAX study website, including self-assessment. This was followed by reviewing several actual cases together to ensure that definitions were understood and agreed upon. In cases of ambiguity, a common interpretation was reached. For example, thrombus on CCTA was defined as an amorphous filling defect in a vessel, often with low attenuation, in patients admitted for acute chest pain.
After finalising the results from the first 10 cases of the study, the team examined each other’s results simultaneously to ensure that there were no significant differences in interpreting the score. Since it had already undergone extensive validation, ICA analysis was performed, in a blinded fashion, by a single reader (AK), whose results were used as the standard reference. A second reader (EA) reviewed 52 cases for the purpose of evaluating in-house inter-user variability. CCTA was scored separately and blindly by two different readers (JL and SA) who later reviewed the cases simultaneously to achieve a consensus reading which was used in all further analysis. Each user’s individual results were used only to calculate inter-rater variability. Scores were calculated in an Excel worksheet per lesion, per artery and per patient. For lesion-based analysis, only those lesions that were identified by both modalities were used for analysis. In addition to SYNTAX score evaluation, conventional vessel-based analysis was performed regarding the presence or absence of at least one stenosis >50% for each of the three main coronary arteries and the left main trunk.
The optimal phase/phases was/were selected and loaded to the Extended Brilliance Workspace (Philips Healthcare, Cleveland, OH, USA), where all image analysis was performed. Centrelines were semi-automatically identified and curved multiplanar reformatted images created for diagnostic purposes (Figure 1). These reformatted images were rotated through 360 degrees and, in cases where a lesion was suspected, cross-sections through the vessel were examined. Axial images were also available when required. SYNTAX score was documented for each segment. For each artery image, quality was evaluated on a scale from 1 to 5, where 5 is excellent, 4 is good, 3 is satisfactory with minor artefacts present, not affecting diagnosis, 2 is non-optimal image quality with artefacts which impair diagnostic accuracy but with vessel assessment still possible and 1 is unable to diagnose. Aortic root enhancement was recorded, as was heart rate and its standard deviation, the latter as a measure of heart rate variability (Table 1).
Figure 1. An example of SYNTAX scoring differences between CCTA (A,C) and ICA (B). By CCTA, the lesion appears most significant immediately after the bifurcation (arrow), in the proximal left anterior descending artery (7 points), while the left main coronary appears only mildly involved. The cross-sectional images (C, top) help confirm this impression, since the lumen just before the bifurcation (central cross-section) appears to be only mildly narrowed relative to the proximal vessel, while the cross-section on the extreme right shows a highly calcified plaque with only minimal lumen. A trifurcation lesion is scored (3 points), as well as heavy calcification (2 points) with a total score of 12 points. By ICA the lesion is identified as a distal left main stenosis (12 points) and as a trifurcation, totalling 15 points.
STATISTICAL ANALYSIS
Continuous variables are expressed as mean and standard deviation, and ordinal variables are presented as frequencies. Agreement between modalities and inter-rater agreement is evaluated by Cohen’s kappa with weighting applied for non-binary parameters. This analysis was based on SYNTAX score tertiles, as described previously3-8. We have reported the strength of agreement by kappa based on the following guidelines of Landis and Koch11: none (<0), slight (0-0.2), fair (0.21-0.4), moderate (0.41-0.6), substantial (0.61-0.8), almost perfect (0.81-1). Bland-Altman analysis was also applied as an additional measure of agreement between techniques and between raters. Since the data is not normally distributed according to the Shapiro-Wilk test, nonparametric statistical tests were applied. The Wilcoxon test was used for related data and the Mann-Whitney U test for independent data. Data analysis was performed on SPSS statistical software version 16.0 (SPSS, Chicago, IL, USA) and Analyse-it® 1.05 software (Analyse-it Software Ltd., Leeds, UK).
Results
STUDY POPULATION
Between the period from mid-2008 to March 2011, 111 patients were identified, of whom seven were later excluded due to the poor image quality on their CCTA precluding accurate diagnosis. The final patient population consisted of 104 patients, of whom 102 patients initially underwent CCTA and only two underwent CCTA after ICA. The mean time period between tests was 4.1±6.9 days. Eighty-six patients (83%) were inpatients at the time of CCTA, which was mostly performed for investigation of acute chest pain. Indications for ICA and clinical characteristics of the 104 patients who were included in the final analysis are presented in Table 2.
CCTA image quality was generally satisfactory to good and mostly performed at a heart rate of under 65 with good arterial enhancement (Table 2). The mean calcium score was 597. This demonstrates the diffuse nature of atherosclerosis in our study population.
CONVENTIONAL ARTERY-BASED ANALYSIS
Analysis of results of CCTA compared to ICA for identification of >50% stenosis per coronary artery (left main, left anterior descending, left circumflex and right coronary arteries), revealed that CCTA had a sensitivity of 71% (139/196), specificity of 94% (207/220) and accuracy of 83% (346/416). Agreement between modalities was substantial with a kappa of 0.66 (0.59-0.73).
PATIENT SYNTAX SCORES
Patient SYNTAX scores were divided into tertiles based on results of ICA, as described in previous studies3-8. Tertile ranges were 0-8, 8-17, 17-51.
The mean SYNTAX score was 14.2±10.0 by ICA and 10.3±6.9 by CCTA (Table 3), representing a significant underestimation of patient SYNTAX score by CCTA by a mean of 3.9±8.2 (p<0.001), using Bland-Altman analysis (Figure 2). Performing a separate analysis for SYNTAX scores below or above the mean value reveals that most of the bias occurs for the higher SYNTAX scores (7.3±10.0) compared to only 0.9±4.7 for the lower SYNTAX scores. Weighted kappa was 0.33, indicating only fair agreement between the two modalities. If only good quality CCTA were used (40 cases with graded image quality score 4 or 5), kappa improved to 0.56. ICA identified more lesions per patient (2.2±1.3 vs. 1.7±1.0, p<0.001), with a moderate agreement regarding number of lesions (kappa=0.49) and more diseased vessels per patient (1.8±0.9 vs. 1.4±0.7, p<0.001). The mean score per lesion was not different (6.4±5.8 vs. 6.0±4.7, p=ns). More lesions were found by ICA for each of the major coronary arteries (Table 3), though, interestingly, agreement per vessel analysed separately was better than the overall agreement. The various components that make up the SYNTAX score were also analysed for agreement between the two modalities. Trifurcation and bifurcation lesions were identified to a similar degree by CCTA and ICA, however the level of agreement was only fair (kappa=0.43 for trifurcation lesions and kappa=0.32 for bifurcation lesions.) More total occlusions were identified by ICA (22 vs. 14) with a moderate level of agreement (kappa=0.6). Tortuous lesions were identified almost exclusively by ICA and lesions were graded as being long much more frequently by ICA (33 cases) compared to 15 cases by CCTA with a slight level of agreement (kappa=0.12). Calcified lesions were identified to a similar extent (23 vs. 26 cases) however agreement was only fair (kappa=0.36). Thrombus was rarely identified by either technique (9 vs. 6 cases, kappa=0.36).
Figure 2. A Bland-Altman plot comparing patient-based SYNTAX scores based on the two techniques.
LESION SYNTAX SCORES
To assess lesion to lesion agreement we first isolated only those 148 lesions that were identified by both techniques and then divided the lesion SYNTAX scores into tertiles [0-3, 3-7, 7-34]. For comparison, ICA identified 229 lesions and CCTA 178 lesions. Lesion-based agreement was substantial with kappa=0.69 and a smaller underestimation (6.3±4.9 for CCTA vs. 7.3±6.3 for ICA, p<0.05) (Figure 3). Examining the various parameters that make up the score, results were similar to those obtained for patient-based analysis (Table 4). Of 38 bifurcation lesions identified by ICA, CCTA identified 25 as such, and three others as trifurcation lesions, while 18 lesions were called bifurcation lesions by CCTA but not by ICA. However ICA tended to grade bifurcation lesions more commonly as complex (types D-G) in 26/38 (68%) compared to CCTA, 11/43 (26%). Weighted kappa of 0.52 indicates a moderate level of agreement. Eleven of 21 occlusions were correctly identified by CCTA. The 21 occluded lesions had a mean lesion length of 26±12 mm on CCTA compared to non-occluded lesions that had a length of 8±7 mm, p<0.001. In cases of correct identification, lesion length tended to be longer than for false negatives (31±10 mm vs. 21±13 mm, p<0.05). A lesion length of ≥ 12 mm identifies 19/21 (90%) occlusions but has 22/126 (17%) false positives. A lesion length of ≥ 20 mm identifies 14/21 (67%) of occlusions and has 11/126 (9%) false positives.
Figure 3. A Bland-Altman plot comparing lesion-based SYNTAX scores based on the two techniques.
INTER-RATER VARIABILITY
By conventional artery-based analysis by CCTA there was agreement regarding 89% of vessels concerning the presence of a >50% stenosis, giving a kappa value of 0.77 (0.7-0.83). For SYNTAX score per patient, the two raters scored an average of 10.7±7.4 and 9.4±7.6, with a moderate level of agreement (kappa=0.51 [0.35-0.67]).
For ICA, mean SYNTAX scores were 13.9±10.0 and 12.9±8.1, with an excellent level of agreement (kappa=0.84 [0.75-0.94]).
Discussion
The ability to non-invasively calculate the SYNTAX score would have important implications in patient management. To the best of our knowledge, this is the first study to investigate the possibility of calculating SYNTAX score from CCTA. We have shown, in a patient population characterised by extensive coronary artery disease, that in spite of good accuracy in diagnosing the presence of significant (>50%) stenosis per coronary artery, the assessment of SYNTAX score by CCTA underestimated the total score and the degree of agreement with ICA was only fair. Sub-analysis showed that the major reason for this discrepancy was identification of more lesions by ICA, whereas for lesions identified by both modalities, a substantial agreement was found. There was better per patient agreement in the studies with good image quality.
Previous studies have shown that CCTA is highly accurate in the diagnosis of significant stenotic disease when compared with ICA and it has been shown to have an excellent negative predictive value12, 13. However since CCTA is mainly indicated in patients with a low to intermediate probability of having coronary artery disease, these studies have usually incorporated a high proportion of normal and non-significant studies, in which CCTA is especially useful. There have been few studies in populations with a very high incidence of coronary artery disease. The diffuse nature of the disease and especially the presence of large amounts of calcium may drastically affect the diagnostic accuracy of CCTA14. Nonetheless it was surprising that CCTA, which is generally regarded as being oversensitive in coronary artery disease diagnosis with a high proportion of false positive calls, in this study actually underestimated disease severity relative to ICA. This occurred with regard to both the number of diseased vessels and the number of lesions per vessel. The SYNTAX score per lesion, on the other hand, did not differ significantly between CCTA and ICA. This suggests that, in the presence of diffuse coronary artery disease, following identification of a positive finding, the readers tended to be conservative about diagnosing additional stenoses, in borderline cases.
Sub-analysis of the various components of the SYNTAX score brings out some interesting and important differences between the two techniques. Total occlusions, which may be difficult to diagnose by CCTA due to its static nature, were actually identified in the majority of cases (11 of 21 lesions), and were characterised as being much longer than non-occluded lesions, a cut-off of 20 mm being highly specific and 12 mm being very sensitive. In contrast, agreement regarding the presence of long lesions above 20 mm was poor. Here CCTA would be expected to have an advantage, since it allows the measurement of the lesion in 3-D including visualisation of the plaque itself, without foreshortening or magnification. This disagreement warrants further investigation since, potentially, CCTA may allow optimal stent sizing and placement. Both techniques identified a similar number of bifurcation lesions; however, ICA more commonly graded them as complex lesions, meaning that the ostium of the side-branch was involved. This may be related to the inability of ICA to differentiate between plaque in the main vessel at the intersection as opposed to one within the side branch, or it may be that branch vessels make diagnosis by CCTA more challenging, especially in the presence of calcified plaque. Despite being much more sensitive to the presence of calcium, CCTA did not score lesions as being heavily calcified any more than ICA, however, agreement was only fair. Similarly coronary thrombus, even though there are no agreed-upon diagnostic criteria for CCTA, was identified to a similar extent by each technique with a fair agreement. Interestingly, based on the definitions of SYNTAX score, only one lesion was scored as being tortuous by CCTA as opposed to 10 cases by ICA, despite CCTA being a 3-D technique with the ability to visualise angles clearly. This suggests that ICA may overestimate the severity of tortuosity.
Study limitations
This study is retrospective in design, with a referral bias, consisting largely of inpatients admitted for acute chest pain. There are only a small number of patients with high SYNTAX scores (>32), based on the results of the landmark SYNTAX trial2, which differentiated between patients who had better outcome with bypass surgery as opposed to PCI. Since this study has mainly looked at the lower range of SYNTAX scores, results may not reflect higher scores indicating the presence of more severe disease. Potential limitations of CCTA in this respect include evaluation of diffuse disease, especially when heavily calcified or involving smaller calibre vessels, total occlusions, bridging collaterals and presence of thrombus. These parameters are largely related to the poorer spatial resolution and lack of dynamic imaging of CCTA. Good quality CCTA studies, characterised by sharp images without blurring or motion and clear visualisation of plaque, can help compensate for these disadvantages, and indeed, our results show superior results in cases with good image quality. The strengths and limitations of each technique are summarised in Table 5.
Nonetheless the SYNTAX score has also been studied and found useful in numerous other study populations with lower SYNTAX scores3-8. Capodanno et al15 state that the SYNTAX score can be considered as a means by which the coronary angiogram is appropriately studied in the detail that it deserves to make appropriate treatment decisions in a variety of clinical scenarios. Our data shows a better agreement at lower SYNTAX scores, with a large bias being present mainly at higher scores, based on Bland-Altman analysis (Figure 2 and Figure 3).
To put things into perspective, several studies have evaluated inter-observer and intra-observer agreement for SYNTAX score evaluation by ICA, performed by invasive cardiologists. Studies of inter-observer reproducibility have reported values of kappa of 0.522 and 0.5616, while intra-observer reproducibility studies report values of 0.612 and 0.5117. Another study18 evaluated inter-observer reproducibility following a basic tutorial (kappa=0.33), which improved substantially after intensive training (kappa=0.76). Bifurcation lesions, particularly, had consistently poor reproducibility. These data suggest that better training and calibration of results between ICA and CCTA physicians may improve agreement between the two techniques. This is especially relevant in the case where radiologists interpret CCTA, since they lack familiarity with ICA and its intricacies, which are required for interpreting the SYNTAX score.
In summary, this study of patients with significant coronary artery disease shows that, despite CCTA having a substantial agreement with ICA by conventional vessel-based analysis, as well as for calculation of SYNTAX score on a per lesion basis, per patient agreement was unsatisfactory on a 64-slice scanner. Since agreement appears to improve for better quality studies and after extensive training, it is reasonable to expect improved results in future studies with new-generation scanners and improved simultaneous training of ICA and CCTA raters.
In the meantime, clinical decisions based on SYNTAX score calculated from CCTA cannot be recommended.
Conflict of interest statement
The authors have no conflicts of interest to declare.
