Chapter 18 — Meta‑analysis & systematic reviews (PRISMA)

The “all the evidence in one place” paradox

You’re reviewing ICU literature on prone positioning for ARDS. A colleague says: “The systematic review says pooled mortality reduction is 12% (95% CI 5–19). That’s strong evidence; we should start prone positioning.”

But you recall that systematic reviews can be misleading if:

• The search missed key studies.
• Data extraction was flawed.
• Heterogeneity across studies is high, but a single pooled number hides it.
• Publication bias inflates the effect.
• The review applied inappropriate statistical models.

Meta‑analysis and systematic reviews are powerful when done well, but they also carry pitfalls. This chapter shows how to read them critically and applies the PRISMA checklist.

What systematic reviews (SRs) do

1. Define a clear question – PICO (Population, Intervention, Comparison, Outcome). Example: In adults with ARDS, does prone positioning reduce 28‑day mortality?
2. Search systematically – Use exhaustive, reproducible search strategies across databases (PubMed, Embase, ClinicalTrials.gov) and grey literature.
3. Screen titles/abstracts – Apply inclusion/exclusion criteria (e.g., RCTs only, ≥10 patients per arm).
4. Extract data – Extract effect sizes, CIs, and study characteristics.
5. Assess risk of bias – Use tools like Cochrane RoR (for RCTs) or AMSTAR‑2 (for SRs).
6. Pool results – Choose an appropriate model (fixed‑effect vs random‑effects) based on heterogeneity.
7. Explore heterogeneity – Subgroup analyses, meta‑regression, and I².
8. Check publication bias – Funnel plots, Egger’s test (limited utility in ≤10 studies).
9. Interpret and grade confidence – Use GRADE or intuitively rate certainty in the pooled estimate.

If any step is missing or mis‑applied, the conclusions may be biased.

The PRISMA flow diagram: a visual audit trail

The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta‑analyses) checklist includes a flow diagram showing:

• Records – total citations identified.
• Screen – titles/abstracts reviewed.
• Full‑text – articles assessed for eligibility.
• Included – final set of studies.
• Reasons – why each excluded study was dropped.

If the diagram shows a huge number of excluded studies without clear reasons (e.g., “not RCTs” but no count breakdown), there may be selective reporting.

Appraisal checklist for PRISMA:

1. Is the PRISMA flow diagram present? – Yes.
2. Are the dates of the search stated? – Yes (important because the literature changes).
3. Is the search strategy described in detail? – Yes (database, keywords, Boolean operators).
4. Were duplicates removed? – Yes (or reported).
5. Are inclusion/exclusion criteria clearly defined and applied consistently? – Yes.
6. Is the risk‑of‑bias assessment reported per study? – Yes (Cochrane RoR or similar).
7. Are heterogeneity statistics reported? – I², Q‑test, tau².
8. Is publication bias assessed? – Funnel plot, statistical test (if ≥10 studies).

Pooling: fixed‑effect vs random‑effects

If a paper uses a fixed‑effect model on highly heterogeneous data (e.g., different ARDS definitions, varying prone durations), the pooled estimate is misleading. The authors should justify their choice.

Subgroup analysis and meta‑regression

When heterogeneity is present, the authors should explore subgroups (e.g., ARDS severity, duration of proning, operating theatre vs ICU setting) and see if effects differ.

Meta‑regression can test continuous moderators (e.g., baseline mortality rate) but requires many studies (≥10) for reliable estimates. With few studies, meta‑regression is underpowered and prone to false conclusions.

Appraisal checkpoint: Does the paper report subgroup results? Are they pre‑specified (preferred) or exploratory (interpret with caution)?

Publication bias: the invisible elephant

• Funnel plot – For larger studies (lower variance), effect estimates should cluster symmetrically around the pooled effect. Asymmetry may suggest bias.
• Egger’s test – Statistical test for funnel asymmetry (requires ≥10 studies).
• Trim‑and‑fill – Attempts to estimate and correct for missing studies (use with caution; may hide true heterogeneity).
• Grey literature – Including conference abstracts, trial registries, and company reports reduces bias.

If the review excludes grey literature, the pooled effect may be inflated. If the funnel plot is asymmetric and the authors dismiss it without justification, the review may be biased.

GRADE: rating the certainty of evidence

GRADE (High, Moderate, Low, Very Low) considers:

1. Study design (RCT → high; observational → low).
2. Risk of bias (low → high; high → low).
3. Inconsistency (heterogeneity).
4. Indirectness (population/outcome mismatch).
5. Imprecision (CI too wide).
6. Publication bias.

Appraisal checkpoint: Does the review use GRADE? Is the rating clearly stated for each outcome? If they claim “high certainty” but the pooled CI crosses the null or heterogeneity is high, the rating is questionable.

Putting the appraisal together: a real‑world example

Assume a systematic review on prone positioning in ARDS found:

• 12 RCTs, total N = 2,800.
• Pooled mortality: RR = 0.88 (95% CI 0.79–0.98).
• I² = 45% (moderate heterogeneity).
• Subgroup: severe ARDS (PaO₂/FiO₂ < 150) RR = 0.81; mild ARDS RR = 0.96.
• Funnel plot shows slight asymmetry (Egger’s p = 0.07).
• GRADE: Moderate certainty (due to heterogeneity and risk‑of‑bias concerns).

Interpretation: There is a probable mortality benefit, but the effect size is modest, varies by severity, and may be inflated by publication bias. Clinicians should weigh the modest benefit against resource use, complications, and patient comfort.

Your appraisal checklist for systematic reviews

Go deeper

• EQUATOR Network – PRISMA 2020 checklist (free PDF): detailed explanation of all 27 items. https://www.equator-network.org/reporting-guidelines/prisma-2020/
• StatPearls – “Systematic Reviews and Meta‑analyses” (NBK560123): free, covers PRISMA, Cochrane RoR, GRADE, and interpretation. https://www.ncbi.nlm.nih.gov/books/NBK560123/
• OpenIntro Statistics – Chapter 13 “Meta‑analysis and systematic review” (free PDF): worked example of a meta‑analysis with R code, and guidance on assessing heterogeneity. https://www.openintro.org/stat/textbook.php
• BMJ Statistics Notes – “Meta‑analysis” (Altman & Bland): concise on fixed vs random effects, heterogeneity, and GRADE. https://www.bmj.com/content/bmj_stats_notes
• PMC3227332 – “Survival analysis in clinical trials: Basics and must‑know areas” (free): includes a subsection on systematic reviews of survival outcomes, with practical tips. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3227332/

Next: Chapter 19 — Pragmatic trials (already written as Chapter 15 in this framework, but if you need it placed after 18 for numbering, please let me know).