Systematic reviews and meta-analyses have always been popular among clinicians and scientists,in part due to the appeal of a single scientific paper summarising all the available data on a specific topic. As shown in Figure 1, systematic reviews and meta-analyses have been performed more and more often in the last years within the field of interventional cardiology and applied to the assessment of potential advantages and drawbacks of many newly developed techniques, therapies and devices.
Figure 1. PubMed search (from 1985 to 2008) with the words: “meta-analysis AND percutaneous AND coronary”.
The number of these meta-analyses that are published each year has however reached such incredible heights that, at this point, the outburst qualifies as an epidemic. Even though the authors of this editorial have contributed to some extent to this epidemic, they feel compelled to react to the developing meta-analytic rage in the cardiovascular literature.
In the past, it was not unusual that articles of this kind became seminal landmark documents which went on to influence clinical research and contribute to changes in practice. A good example is one that can be taken from the meta-analyses that demonstrated, very early on during the development of the therapy, of the advantages of primary percutaneous coronary intervention over thrombolytic therapy in acute ST segment elevation myocardial infarction1. Superiority of the invasive strategy became apparent, despite a huge difference in the number of studies that had been performed; very numerous large, industry-sponsored trials using different types of thrombolytic agents, as opposed to a small number of studies on primary percutaneous intervention, many of which were not powered to assess clinical outcome events and were often investigator-driven2.
Presently, the rapidly growing body of clinical studies and registries has stimulated the elaboration of several meta-analyses on various relevant clinical topics, often in the presence of original studies that are either few, small, inconclusive, or all of the above. This epidemic outburst of meta-analytic rage leads to the publication of numerous manuscripts, many of which only contribute to increasing the already long list of anonymous, never-quoted papers. In addition, we fear that the publication of poor quality, inappropriate, irrelevant or contradictory manuscripts will undermine the perceived, true value of meta-analyses, a type of investigation that should still be seen as providing the highest level of available evidence, when executed properly.
The holy grail of meta-analyses, similar to original work, should not be restricted to the “search for the p-value” that confers significant benefit of one type of treatment over another. Similar to original studies, meta-analyses are prone to errors and bias, dependent on the way they have been conducted and reported. We would like here to reflect on how to read and interpret meta-analyses, also keeping in mind that p-values are only one of the many results and conclusions that a meta-analysis can provide.
Definitions
Conventionally, a “meta-analysis” is performed after a systematic review of all data related to a specific clinical contest has been completed. A “systematic review” provides an overview of all articles published in the literature focusing on a clinical problem3,4. The term “systematic” implies that all the steps underlying the review process are explicitly and clearly defined, and therefore could be reproduced independently by others, as desirable. In this process, a formal set of pre-specified methods is applied to: 1) study search (extensive search of the literature for original studies), 2) study selection, 3) study appraisal, and 4) data abstraction. The term “meta-analysis“ is used to describe a statistical method that ends by pooling together results and data from several different original studies, to provide more precise and valid results (see Table 1 for further definitions).
Thus, not all systematic reviews include a meta-analysis, as not all topics are suitable for sound and robust pooling of data. Sometimes, meta-analyses can be conducted outside the realm of a systematic review (usually they are called “pooled analyses”), without extensive and thorough literature searches, but then the results of the meta-analysis are best viewed as hypothesis-generating only. This is mainly the reason why meta-analyses, without a properly performed systematic review, entail a significant risk of bias.
Strength of meta-analyses
Systematic reviews and meta-analyses have several strengths3-5. They use systematic literature searches and compile the whole body of evidence on a particular question. The standardised processes for search, appraisal and selection of original studies allow reproducibility and objectivity. Thorough evaluation for internal validity and risk of bias in the individual original studies clearly identifies the limitations of these studies. Indeed, the greatest strength of systematic reviews is often their ability to pinpoint weaknesses and fallacies in apparently sound original studies6.
Quantitative synthesis by means of meta-analysis also substantially increases statistical power and provides narrower confidence intervals for statistical inferences. Assessing the effect of an intervention in different settings and at different times gives estimates and inferences of greater external validity. In addition, meta-analyses provide not only pooled estimates of effects, but also heterogeneity and inconsistency between included studies. Specific statistical analyses, such as the pooled estimates of effects, are an essential component of meta-analyses and should be reviewed critically. These parameters allow a critical appraisal of the overall quality of the meta-analysis, and also of the original studies included. They should always be reported and discussed by researchers performing a meta-analysis. Heterogeneity among original studies included in a meta-analysis is conventionally demonstrated by p values <0.10 at Cochrane Q heterogeneity χ2 test7, while inconsistency among original studies is typically testified by I2 values >50%8. Clinical and statistical variability may be exploited by advanced statistical methods such as meta-regression, with the possibility of generating novel hypotheses9.
Weaknesses of meta-analyses
Although meta-analyses are useful instruments of study, they cannot substitute appropriately sized and conducted randomised clinical trials3,10. While some authors believe that meta-analyses from homogeneous, randomised and controlled trials represent the apex of the evidence-based medicine pyramid, others have maintained that very large and simple randomised clinical trials should always be preferred, when available11,12. Indeed, meta-analytic pooling should be performed only if statistical homogeneity and consistency are confirmed, but this is not suitable when there is significant statistical heterogeneity or inconsistency. Conversely, a systematic and adequate appraisal of the potential reasons for this statistical variability can be justified. Additionally, small study bias is a major threat to the validity of meta-analyses3,10,12. Especially when datasets are large, small original studies are more likely to be reported, published and quoted if their results are significant. Conversely, small non-significant studies often fail to reach publication, and thus may be easily missed, even after thorough literature searches. Combining results from these ‘biased’ small studies with those of larger studies –which are usually published even when negative or non-significant– may inappropriately deviate summary effect estimates away from the true value. Although several graphical and analytical tests are available, small study publication bias is always potentially present in a meta-analyses and must not be forgotten13. Another major threat to the validity of a meta-analysis, as in any other research project, depends on conflicts of interest and how meta-analytical studies are funded. It is well known that reviewers with underlying financial conflicts of interest are more likely to draw conclusions that favour an intervention, which benefits the source of any financial gain14. These facts should encourage a more critical review of work performed by scientists with declared conflicts of interest. Needless to say, there are also more or less obvious conflicts of interest that are not openly declared by authors. In any case, the overall internal validity of the studies themselves, such as the blinding of patients, physicians, adjudicators and analysts, should always be evaluated.
Finally, meta-analyses, as much as original studies, are prone to statistical errors. Alpha error is defined as the risk of incorrectly dismissing a null hypothesis, despite it being true. Also, in meta-analyses, the risk of biased estimates and alpha error may be present3. Minor differences, in few and rare events, may give nominally significant results (with borderline significant p-values, for example p=0.04) which, however, may not be reliable or clinically relevant. In any case, reliance on the combined appraisal of p-values and 95% confidence intervals is recommended. Another solution, is to use more stringent cut-off thresholds, such as 0.01 for p-values and 99% for confidence intervals. An additional, useful rule of thumb, is to trust meta-analyses reporting on at least 100 pooled events per group under comparison. The beta error is the risk of erroneously accepting a null hypothesis despite it being false. This error is also common in meta-analyses, especially when they include few studies with low event counts. This lack of sufficient statistical power (defined as 1-beta) is even more common with meta-regression analyses, which are usually underpowered because of the few studies included and regression to the mean phenomena15.
Directions for the future
Systematic reviews and meta-analyses are powerful methodologies to assess the available evidence on a specific topic. However, their value mainly depends on the application of adequate methods and the quality of the individual studies that are included.
Prospective and well planned design is pivotal and offers a potential solution to avoid duplicate meta-analyses, at times providing opposite conclusions. It would be desirable to mandate registration of ongoing and planned meta-analyses, where topics, protocols and contributions can be made public, before the work actually starts. This same concept has been introduced successfully for original studies, whilst protocols are posted on dedicated freely accessible web-sites, such as www.clinicaltrials.gov, before beginning patient enrolment. Broader, collaborative, research efforts are needed to set-up international research groups able to design, conduct and disseminate individual patient data meta-analyses, which can combine results from individual clinical trials in an unbiased and rigorous way16.
The readership should be fully aware that the summing-up of several doubtful results only yields one big uncertainty. Even well conducted meta-analyses cannot repair the flaws of included studies when these are underpowered, or their data is poorly controlled or when they lack independent evaluation by professional core-laboratories and clinical research organisations. The inclusion of registry data in meta-analyses is highly questionable, because registry data can be tarnished by unrecoverable selection bias or unknown confounders17 and meta-analyses aim at the analysis of the evidence that stems from randomised trials.
Based on these considerations, one can easily formulate a number of common sense recommendations that hopefully could be endorsed by the scientific community. The fundamentals pertain to the question of why and when a meta-analysis is needed and how it should be properly performed in accordance with well defined rules7,8. In many instances where the field is uncertain, uncertainty calls for an appropriate trial rather than for an inappropriate analysis of the existing, yet inconclusive data. Finally, it seems essential to bring an end to the wild dissemination of systematic analyses and meta-analyses that do not comply with the above-mentioned recommendations. This calls for a thorough peer review process and consensus among journal editors. If not, the risk exists that the value of meta-analyses will be degraded due to the perception that they indeed no longer express the highest level of scientific evidence, but rather “manipulation” of this evidence instead.
