Chapter 7 — Confidence intervals & hypothesis testing

The p = 0.049 morning rounds

A consultant stops you in the corridor: “The sepsis trial just came out. Mortality dropped from 34% to 26%, p = 0.049. Statistically significant — let’s change the protocol.”

You ask one question: “What’s the 95% CI?”

They blink. “Oh. p = 0.049. That’s all we need.”

In ICU research, a single p‑value is the most overused and most misinterpreted number in the literature. A p‑value of 0.049 means: if the null hypothesis were true (no effect), there is a 4.9% probability of observing a difference as extreme as — or more extreme than — what we saw, by random chance alone. It does not mean there is a 95.1% probability the treatment works. It does not tell you the magnitude of the effect. And it does not tell you the precision.

That is what confidence intervals (CIs) are for.

What a confidence interval actually says

A 95% CI of 22–34% mortality tells you: If we repeated this study 100 times, 95 of those 100 intervals would contain the true population parameter. That is a statement about the procedure, not about the single interval you just observed. In frequentist statistics, the true parameter is fixed; the interval moves across repetitions. You cannot say “there is a 95% probability the true rate is between 22 and 34%.” That is a common misinterpretation — the Bayesian credible interval says that; the CI does not.

Why does this matter at the bedside? If the 95% CI crosses the null (e.g., ARR 0.08, 95% CI −0.02 to 0.18), the result is not statistically significant — but the interval also tells you the plausible range of effects, from a modest benefit to a small harm. That is clinically more informative than a binary “significant/not significant” label.

The three pillars you need to estimate

These three are mathematically linked — but the CI carries more information than the p‑value alone. A narrow CI (e.g., ARR 0.08, 95% CI 0.03–0.13) tells you the estimate is precise; a wide CI (ARR 0.08, 95% CI −0.05 to 0.21) tells you the estimate is imprecise — and the difference may simply reflect a small sample size or rare events.

α, β, power — the error types

Type I error (α)

The probability of rejecting the null hypothesis when it is actually true — a false positive. Conventionally set at α = 0.05. In ICU trials, a Type I error means adopting a therapy that does not work, exposing patients to cost and harm for no benefit.

Type II error (β)

The probability of failing to reject the null when it is actually false — a false negative. β is typically 0.10–0.20, corresponding to power of 80–90%. A Type II error means missing a real benefit — a treatment that does reduce mortality but was not detected because the sample was too small or the effect too modest.

Power (1 − β)

The probability of detecting a true effect of a specified size. Power depends on:

• Sample size (larger N → higher power).
• Effect size (larger effect → easier to detect).
• Variability (less noise → higher power).
• Alpha level (less strict α → lower power, but higher Type I error rate).

The power calculation

Every RCT should report a sample‑size calculation before enrolment. A typical statement:

“To detect a 10% absolute reduction in 28‑day mortality from 40% to 30% (two‑sided α = 0.05, 90% power), we calculated a requirement of 2 × 190 patients = 380.”

If the paper does not report this, the study may be underpowered, and a “non‑significant” result could simply reflect a sample too small to detect a clinically meaningful difference.

Appraisal checklist for power:

1. Is the effect size clinically important? (A trial powered to detect a 5% ARR in mortality when you care about a 2% ARR is underpowered for your question.)
2. Is the event rate realistic? (If the control mortality is 10%, detecting a 5% ARR requires a much larger sample than if control mortality is 40%.)
3. Is alpha stated? (Usually 0.05; some ICU trials use 0.01 for multiple comparisons.)
4. Was power pre‑specified? (Usually 80% or 90%; post‑hoc power is not a substitute for an a priori calculation.)

The “statistical significance” trap

The 0.05 threshold is a convention, not a law of nature. A p‑value of 0.051 is not “clinically different” from 0.049 — they are both fragile, and both should prompt the same question: what does the CI say?

A result with p = 0.051 and 95% CI 0.01–0.18 is a trend toward benefit with a plausible range of small to moderate gains. A result with p = 0.049 and 95% CI −0.01 to 0.19 crosses the null — the “significant” finding is driven by the upper tail. The CI tells you the story the p‑value obscures.

Why the “p < 0.05” reflex is dangerous in the ICU

ICU populations are heterogeneous, outcomes are noisy, and small studies are common. A p‑value from a small, heterogeneous ICU trial may be no more reliable than the single MAP reading from a damped arterial line. You would not change the vasopressor dose on a single artifact — don’t change practice on a single p‑value.

How to read CIs in practice

Your bedside checklist for inference

1. Was the sample‑size calculation reported? (Effect size, alpha, power, event rate.)
2. What is the point estimate and its CI? (Not just the p‑value.)
3. Does the CI cross the null? (If yes, the result is not “significant,” regardless of the p.)
4. Is the CI width clinically acceptable? (A CI spanning 0.01–0.35 ARR is too wide for a practice change; 0.05–0.07 is narrow and actionable.)
5. Were multiple comparisons adjusted? (If 20 endpoints were tested, 1 will be “significant” at p < 0.05 by chance alone — Bonferroni or FDR correction should be applied.)

Go deeper

• StatPearls – “Hypothesis Testing” (NBK431056): free, covers α, β, power, Type I/II errors, and the interpretation of p‑values. https://www.ncbi.nlm.nih.gov/books/NBK431056/
• BMJ Statistics Notes – “Confidence intervals and probability” (Altman & Bland): concise guide to interpreting CIs and why they are preferred over p‑values. https://www.bmj.com/content/bmj_stats_notes
• OpenIntro Statistics – Chapter 6 “Sampling and bootstrap” (free PDF): explains why CIs narrow with larger samples and how simulation works. https://www.openintro.org/stat/textbook.php
• PMC3122736 – “The significance of significance testing” (free full text): a critical review of the 0.05 threshold, with clinical examples. https://doi.org/10.1093/intqhc/mzm042 (PMC free)

Next: Chapter 8 — Comparing two means: t‑test & alternatives