Current Role and Recommendations for Systematic Review and Meta-Analysis in Dentistry

Introduction: Systematic reviews and meta-analyses (SRMAs) are at the top of the evidence hierarchy in evidence-based healthcare, but there is increasing criticism about their overuse and perceived redundancy.

Objectives: To clarify their important role, outline common misconceptions, and propose recommendations to enhance their methodological and reporting quality in dentistry.

Methods: This communication paper synthesizes current literature and expert commentary on the strengths and limitations of SRMAs, referencing Cochrane and PRISMA guidelines. It identifies key areas where SRMAs are underestimated or misjudged, explores statistical and reporting challenges, and makes recommendations from leading methodological frameworks to support future SRMAs.

Results: SRMAs provide crucial benefits: mitigating publication bias, enhancing statistical power, and enabling subgroup and network meta-analyses. They are instrumental in identifying gaps in current evidence and generating future research directions. However, criticisms from reviewers—such as misinterpretation of high I², undervaluation of non-randomized studies, and exclusion of high-risk-of-bias studies-often reflect misunderstandings. This article argues that, when conducted rigorously, SRMAs offer valid and clinically meaningful insights, even when heterogeneity is present or the evidence level is considered low.

Conclusions: Discouraging SRMAs based on flawed assumptions undermines evidence-based practice. Instead, emphasis should be placed on improving methodological rigor, involving expert statisticians, and adhering to established reporting guidelines. Quality SRMAs, when designed and interpreted properly, are vital to dental and orthodontic advancement.

Evidence-based decision-making in healthcare depends on rigorous, high-quality research. While single-center studies offer valuable information about a specific population, their generalizability is often constrained. In contrast, systematic reviews and/or meta-analyses (SRMAs) synthesize results from multiple populations. For clinical interventions, randomized controlled trials (RCTs) are widely regarded as the gold standard. However, their applicability to broader populations is limited due to the controlled nature of their study environments. Moreover, most primary studies calculate sample sizes based on overall populations rather than specific subgroups, often leading to insufficient statistical power to produce valid p-values in their stratified results.

In recent years, criticism of systematic reviews and meta-analyses (SRMAs) has increased significantly [1-3], driven by the perception that they discourage readers [2,4]. A recent letter to the editor presented a vague statistical argument, claiming that a search using the keyword “orthodonti*” with the filter for systematic reviews in period of 2000-2023 yielded 2,246 results, while applying the filter for randomized controlled trials (RCTs) in the same timeframe produced 2,306 results. However, this comparison is inherently flawed, as SRMAs do not exclusively include RCTs of interventions; they can also incorporate non-randomized trials, prospective cohort studies, and other study designs. Furthermore, SRMAs are now commonly conducted for purposes beyond intervention assessment, including prevalence estimation, diagnostic accuracy, and correlation analyses. On average, a single SR/MA synthesizes data from approximately ten primary studies.

This paper emphasizes the need to improve SRMA quality rather than discourage authors from conducting them.

Why We Need SRMAs?

We Need SRMAs for the Following Reasons

• Readers often place greater trust in journals with high impact factors, as, as these journals typically publish studies reporting statistically significant findings, or groundbreaking results that challenge existing knowledge [5]. However, this preference can contribute to publication bias, where studies with non-significant results remain unpublished. SRMAs play a crucial role in overcoming this bias by synthesizing evidence from studies published in lower-impact journals and grey literature.
• Funded studies are more likely to report statistically significant results compared to non-funded studies [6]. SRMAs help address this disparity by synthesizing evidence from both funded and non-funded studies, providing a more balanced assessment of the available literature.
• Most primary studies have limited statistical power especially on subgroup levels to detect true differences, often due to small sample sizes or variability in study design. By synthesizing similar effect sizes into a larger pooled sample, SRMAs enhance statistical power, improving the ability to detect true effects and reducing the risk of Type II errors.
• SRMAs play a crucial role in identifying gaps within the existing literature. Researchers interested in conducting innovative studies can utilize these reviews to recognize methodological limitations, unexplored areas, or inconsistencies in current evidence. By addressing these deficiencies, they can design more robust and impactful research that advances the field.
• SRMAs provide readers with a comprehensive synthesis of the available literature, including its limitations and conclusions, within a single paper. This approach eliminates the need for extensive independent searches, saving time and effort while offering a critical appraisal of existing evidence in a structured and accessible format.
• Ethnicity can be a significant factor influencing treatment effects, as genetic, environmental, and cultural differences may affect clinical outcomes. SRMAs serve as valuable tools for synthesizing data from diverse ethnic populations, enabling subgroup analyses to assess variations in treatment responses across different ethnic groups. 
• A common criticism of SRMAs is that they involve the "comparison of apples and oranges," referring to concerns about combining studies with significant methodological or clinical heterogeneity. However, the Cochrane guidelines have addressed these issues by introducing rigorous, evidence-based methodologies for conducting high-quality SRMAs.
• Network meta-analysis (NMA) allows for indirect comparisons of multiple interventions by synthesizing data from existing studies, even when direct head-to-head trials are unavailable. This approach enables researchers to rank treatments based on their relative efficacy without the need for additional resources or patient recruitment. 
• Prediction intervals in SRMAs help assess the potential impact of future studies on treatment efficacy if conducted in a similar population.
• The certainty of evidence (evidence table) in SRMAs combines risk of bias, effect size, number of studies, number of participants, and study design to rank evidence as “very low,” “low,” “moderate,” or “high,” helping readers assess the strength of the current evidence.

Common Criticisms of SRMAs from Reviewers and Readers/Researchers

Misinterpretation of High I-square: Many critics assume that a high I² value indicates bias in meta-analyses. However, I² merely quantifies the dispersion of effect sizes across studies and does not directly reflect clinical heterogeneity. Differences in effect sizes are not always due to bias but may result from valid factors such as genetic and environmental influences or variations in intervention dosage. While sources of heterogeneity (high I²) should be explored through subgroup analyses, sensitivity analyses, and meta-regression, these efforts are often limited by the lack of reported confounders in the included studies. Therefore, a key advantage of SRMAs is their role in guiding future primary studies to ensure comprehensive reporting of all relevant confounders.

Systematic review is enough: Many reviewers suggest dropping meta-analysis and concluding with a systematic review. However, systematic reviews often rely on vote counting, where conclusions are drawn based on the majority of studies favoring the experimental group. This approach can be misleading, as it does not account for study quality, effect sizes, or statistical power. A previous meta-analysis on bracket failure rate without primer demonstrated how a single high-powered study among four (in which three studies were not significant) significantly influenced the overall estimate [7].

Drop non-randomized studies of interventions (NRSI): One of the common criticisms on SRMAs is the inclusion of NRSI. NRSI may be included when randomized trials are insufficient or infeasible. They help fill gaps by providing data on long-term or rare outcomes. NRSI are also useful for studying interventions unlikely to be randomized. Additionally, they can inform future trials by identifying weaknesses in existing evidence. In some cases, NRSI are considered when intervention effects are very large or few randomized trials exist. A meta-analysis of 2,746 primary studies in 346 meta-analyses using a meta-epidemiological framework found no strong evidence of systematic overestimation or underestimation of treatment effects in NRSI versus RCTs [8].

Drop high risk of bias studies: If a sensitivity analysis reveals that excluding studies with a high risk of bias doesn't change the overall effect size, it's generally recommended to include those studies in the analysis, but to acknowledge the potential for bias in the interpretation of the results. 

Low level of evidence: Some critics argue that this review provides low-level evidence and may not influence current clinical practice. However, low-level evidence can stem from a real small effect size or ethical constraints preventing randomization. Despite its limitations, it is always better than having no evidence.

Recommendations for Quality SRMAs

• Before embarking on new SRMAs, authors should search PROSPERO and existing literature to ensure a strong justification that the new review will enhance existing knowledge and potentially influence clinical practice.
• The latest Cochrane Handbook and PRISMA guidelines should be followed.
• An expert statistician should be part of the review team.
• The abstract of SRMAs should be structured to emphasize limitations and future recommendations.
• A plain-language summary (highlights) should be included in SRMAs to present key findings in non-statistical terms.
• The journal editor should assess whether a newly submitted primary paper addresses gaps in recent reviews on the topic. Improved primary studies, adherence to standard reporting guidelines and data availability in online repositories can contribute to better SRMAs. Although authors of primary studies frequently declare that data will be made available upon reasonable request, researchers conducting systematic reviews and meta-analyses often faced challenges in accessing this data. Direct inquiries via email frequently go unanswered, or authors respond that the data is unavailable.
• Inadequate statistical methods are common in orthodontic meta-analyses, including failure to control for clustering effects and unit-of-analysis errors, incorrect estimation of tau-square, and omission of prediction intervals. The journals should have an expert statistician on the editorial board or at least one among the reviewers to assess these issues.
• Submitting authors of SRMAs should include a proposal for high-quality future primary research at the end of their review, considering ethical constraints and real-world feasibility based on the identified limitations.

Quality and well-conducted SRMAs are still needed in dentistry. They guide researchers to critically appraise and utilize SRMAs for informed decision-making.