False Discovery Rate, explained.
False discovery rate is the expected proportion of false rejections among all rejections, with that proportion defined as zero when none are rejected.
Why it happens
Thirty independent synthetic tests contain 25 null p-values uniform on 0–1 and five signal p-values uniform on 0–0.04. Compare unadjusted thresholds with the Benjamini–Hochberg step-up rule p(i) ≤ iα/30 and reveal which flagged tests were null.
False discovery rate is the expected proportion of false rejections among all rejections, with that proportion defined as zero when none are rejected.
Read the result
Compare flagged counts before revealing truth. After revealing, distinguish a batch's false discovery proportion from the expected error rate targeted by the procedure.
A worked example
Thirty tempting research results
Set the nominal level to five percent and inspect unadjusted flags without looking at the hidden null labels.
Switch to Benjamini–Hochberg, which compares ordered p-values with rank-dependent cutoffs rather than using five percent everywhere.
Reveal truth, record discoveries and false discoveries, then generate another batch to see that realised error proportions vary.
OPTIONAL DEEPER DETAILGo deeper: inside the model
Inside this model
Thirty independent synthetic tests contain 25 null p-values uniform on 0–1 and five signal p-values uniform on 0–0.04. Compare unadjusted thresholds with the Benjamini–Hochberg step-up rule p(i) ≤ iα/30 and reveal which flagged tests were null.
Where this idea is useful
A practical use
A research team comparing many product metrics should plan how it will control false discoveries across the family of tests.
A common misconception
“A five-percent FDR means every selected claim has a 95% chance of being true.”
FDR concerns an expected proportion across a selection procedure. It is not a posterior probability attached to each selected hypothesis.
What this explanation leaves out
- Signal p-values are an authored distribution, not a power calculation. FDR control relies on assumptions about valid p-values and dependence; it is not a guarantee about each batch or a posterior probability for one result.
Does no discovery mean the procedure failed?
Not necessarily. With weak evidence, selecting nothing can be appropriate. The false discovery proportion is defined as zero for that batch, not undefined.
Which tests belong to the same research question before you start searching for small p-values?
Associated thinkers
Associations marked provisional are awaiting source review.
Further reading
Explore the original research or the teaching reference behind this experiment.