P-values and multiple testing: why searching harder can create false discoveries

What a p-value does—and does not—say

A p-value evaluates how incompatible a specified test statistic is with a null model under the test's assumptions. It is not the probability that the null hypothesis is true. It also does not measure effect size, practical value or the chance of a successful replication. Before interpreting the number, identify the null, the statistic, the sampling process and whether the analysis was chosen before inspecting the data.

The same numerical threshold can mean different things in a single planned test and in a broad exploratory search. If you only report the smallest result, readers do not see the opportunities that produced it. Recording what was searched is therefore part of interpreting the evidence, rather than an administrative detail to add after a convincing result appears.

Why a family of tests changes the problem

A valid null p-value has a controlled probability of falling below a chosen threshold. Running many tests creates more chances for that to happen. The exact dependence structure matters, and counting tests is not always straightforward. Searching segments, thresholds, model variants and outcomes can introduce flexibility even when the final report contains just one tidy table.

Open the false-discovery-rate experiment. It generates thirty independent tests: twenty-five nulls and five authored signals. You know the mix only because this is a teaching model. In a real analysis, the identity of true null hypotheses is usually unknown. The reveal button helps you distinguish what the selection rule knows from what the simulator knows behind the scenes.

Worked example: compare two discovery policies

Do not aim for a prescribed number of flags. The seeded dataset makes the comparison reproducible, but another seed can change the discoveries. The task is to explain how the rule selects evidence and what its error target means, including when nothing passes the rule.

1. Choose a five-percent level and inspect unadjusted flags. Each p-value is compared with 0.05, regardless of its position among the other tests.
2. Switch to Benjamini–Hochberg. Sort the thirty p-values and find the largest rank i satisfying p(i) ≤ i × 0.05 / 30; reject through that rank.
3. Reveal synthetic truth and calculate false flags divided by all flags. Generate another round to see why that realised proportion is not a guaranteed batch-level ceiling.

False discovery rate is an expectation

The false discovery proportion is a batch-level quantity: false rejections divided by total rejections, defined as zero if there are none. False discovery rate is the expectation of that quantity over the procedure's repeated sampling. This is distinct from the chance of any false positive and from the expected share of null hypotheses rejected. Confusing those denominators makes error guarantees sound stronger or weaker than they are.

UC Berkeley's FDR notes discuss the role of dependence and formal control. The experiment uses independent valid null p-values to make the basic rule inspectable. It does not imply that any correlated collection of observational analyses inherits the same guarantee. The synthetic signal distribution is also not a study's estimated power or a recipe for deciding how many participants to recruit.

Connect discoveries with precision and recall

The precision-and-recall game gives a concrete denominator exercise. Opening a larger prefix retrieves more relevant items but can add irrelevant ones. Precision asks how much selected material is useful; recall asks how much useful material was found. These are related to the intuition behind false and missed discoveries, though a retrieval task and an inferential error guarantee are different mathematical objects.

Stanford's retrieval-evaluation chapter explains why ranked-list precision can rise and fall as recall increases. In research, demanding more evidence can reduce false claims while missing weaker real effects. The right policy depends on the scientific purpose and consequences. An exploratory lead for a later study is different from an action justified as established knowledge.

Make the search visible before declaring a winner

A clear analysis record separates a question chosen in advance from patterns discovered while exploring. Exploration is productive; its conclusions need language and follow-up that reflect how they were found. You can learn from a promising result without presenting it as if it were the only hypothesis anyone considered.

• List outcomes, segments, windows and analysis variants that were searched.
• Define the hypothesis family and error criterion before interpreting adjusted results.
• Report effect sizes and uncertainty alongside significance decisions.
• Keep exploratory claims distinct from pre-specified confirmation.
• Plan an independent follow-up for a result selected from a large search.

Small samples add another kind of surprise

The law-of-small-numbers game shows why tiny groups can produce extreme rates even without different underlying probabilities. Multiple testing concerns opportunities for selection; sample size concerns variability within those opportunities. Both can contribute to a dramatic winner that later looks ordinary. Increasing sample size and adjusting a test family solve different problems.

Avoid replacing one shortcut with another, such as dismissing every small study or trusting every adjusted p-value. Ask what was measured, how groups were formed, how much searching occurred and what evidence would change your conclusion. A result can be statistically detectable but practically unimportant, and a non-discovery can remain compatible with meaningful effects the study could not resolve.