The “same mean, different story” dilemma
You’re looking at two ICU papers back to back.
Paper A: “Mean SOFA score at 24 h: 5.2 (SD 2.1) in the intervention arm vs 6.1 (SD 2.3) in control. Two‑sample t‑test p = 0.03.”
Paper B: “Mean lactate at 6 h: 4.2 (SD 3.8) µmol/L vs 3.9 (SD 3.5). Two‑sample t‑test p = 0.41.”
Your colleague says: “Paper A shows a real effect — significant p‑value. Paper B shows nothing.”
But pause: Are you sure the t‑test is appropriate for SOFA? And for lactate? And what if the sample sizes are tiny, or the data are heavily skewed? The t‑test is the workhorse of ICU statistics, but it has assumptions — and violating them can give you a false positive (or a false negative).
In this chapter we will cover when the t‑test is legitimate, what to do when it isn’t, and the non‑parametric alternatives that ICU researchers actually use.
The one‑sample, two‑sample, and paired t‑test
Test | When it’s used | Null hypothesis |
One‑sample t | Comparing a single sample mean to a target value (e.g., mean APACHE II in your ICU vs a reference of 25). | The sample mean equals the target. |
Two‑sample t | Comparing means of two independent groups (e.g., intervention vs control). | The two population means are equal. |
Paired t | Comparing two measurements on the same subjects (e.g., SOFA before vs after a protocol, or pre‑ vs post‑treatment lactate in the same patients). | The mean difference between pairs is zero. |
The two‑sample t‑test is the most common in RCT and observational ICU literature. Its validity rests on three assumptions:
- Independence — observations within and between groups are unrelated. (Clustering by nurse or ICU violates this — see Chapter 11.)
- Normality — the data in each group (or the difference scores for paired) follow a normal distribution.
- Equal variances (for the classic Student’s t) — the two groups have the same population variance. If violated, use Welch’s t.
When the normal assumption fails — and what to do
ICU data are notoriously messy. SOFA scores are ordinal counts; lactate is often right‑skewed; length of stay has a long tail. If you run a t‑test on such data, the p‑value may be unreliable.
Simulating the problem
Imagine 30 patients in each arm. The true distribution of SOFA is Poisson‑like: most patients have SOFA 3–5, but a few have SOFA 15+. The mean might be 6 in both arms, but the t‑test could yield p = 0.04 just because one arm happened to have a slightly higher mean and a few extreme values. That is a Type I error inflated by non‑normality.
The solution: Welch’s t‑test
Welch’s t does not assume equal variances. It adjusts the degrees of freedom downward, making the test more conservative when variances are unequal. Most modern statistical software (R, Python, Prism, SPSS) defaults to Welch’s t when you ask for a “two‑sample t test.” If a paper reports a t‑test without specifying which version, assume the worst — check whether they tested equal variances (e.g., Levene’s test) and chose accordingly.
The solution: non‑parametric alternatives
When normality cannot be justified, use a test that makes no assumption about the underlying distribution.
Situation | Recommended test |
Two independent groups, non‑normal, continuous or ordinal | Mann‑Whitney U test (also called Wilcoxon rank‑sum). Reports a P and a common language effect (e.g., “there is a 78% probability that a randomly selected patient from arm A has a lower SOFA than from arm B”). |
Paired observations, non‑normal | Wilcoxon signed‑rank test. Uses the magnitude and direction of within‑pair changes. |
Small sample, very sparse data | Exact permutation test (computationally intensive but exact). |
Appraisal checklist for t‑tests:
- Was normality assessed? — Histogram, Shapiro‑Wilk, or at least a statement that n ≥ 30 invokes the CLT. If n < 30 and no normality check, be skeptical.
- Equal variances? — Was Levene’s or F‑test reported? If not, was Welch’s t used? If unsure, assume unequal variances.
- Are the data truly continuous? — SOFA is ordinal; t‑test may be inappropriate. Mann‑Whitney is safer.
- Is the sample size adequate? — For non‑normal data, small n (< 20 per group) makes even Mann‑Whitney underpowered; consider exact methods.
The magic of the P‑value… and its limits
A p‑value from a t‑test tells you: assuming the null is true, and assuming the data are normally distributed (or n is large enough for the CLT), what is the probability of seeing a mean difference this extreme?
It does not tell you:
- The size of the difference (effect size).
- Clinical importance.
- Whether the result would replicate.
That is why, alongside any t‑test, you should always report (or look for) the mean difference and its 95% CI.
Example
Group | Mean SOFA | SD | n | Mean difference | 95% CI | p (t‑test) |
Intervention | 4.8 | 2.1 | 28 | — | — | 0.04 |
Control | 6.1 | 2.3 | 30 | −1.3 | −2.5 to −0.1 |
The p‑value says “significant.” The CI says “the true mean difference is very likely between −2.5 and −0.1.” If the CI includes 0, the p‑value is misleading (this happens with small n and unequal variances). If the CI is clinically meaningful (e.g., a reduction of 1 SOFA point may not alter mortality), the p‑value alone over‑sells the result.
Your appraisal checklist for comparing two means
# | Question | Why it matters |
1 | What test was used? (Student’s t, Welch’s t, Mann‑Whitney, Wilcoxon) | Determines whether equal‑variance assumption was addressed. |
2 | Was normality checked? (histogram, Shapiro‑Wilk, n ≥ 30 CLT justification) | Small‑sample non‑normal data invalidate the t‑test. |
3 | Were variances equal? (Levene’s test reported, or Welch’s t used) | Unequal variances → use Welch’s, not Student’s. |
4 | Is the outcome actually continuous? (SOFA ordinal, lactate possibly skewed) | If not continuous, consider rank‑based test. |
5 | Is the mean difference reported with a 95% CI? | The CI tells you precision and clinical significance; p alone does not. |
Go deeper
- StatPearls – “t‑test” (NBK552515): free, covers one‑sample, two‑sample, paired t, assumptions, and Welch’s alternative. https://www.ncbi.nlm.nih.gov/books/NBK552515/
- OpenIntro Statistics – Chapter 4 “Inference for numerical data” (free PDF): worked examples of t‑tests, Welch’s, and Mann‑Whitney with R code. https://www.openintro.org/stat/textbook.php
- BMJ Statistics Notes – “The t‑test” (Altman & Bland): concise guide to when to use which version, with practical ICU‑style examples. https://www.bmj.com/content/bmj_stats_notes
- PMC3122736 – “The significance of significance testing” (free full text): critical review of p‑value over‑reliance, with t‑test examples. https://doi.org/10.1093/intqhc/mzm042 (PMC free)
Next: Chapter 9 — Comparing two proportions: χ², Fisher’s exact, RR/OR