Confidence intervals vs statistical power: what an inconclusive study really tells you

Start with the question an interval answers

A confidence interval comes from a rule applied to a sample. Under its assumptions, a 95% procedure covers the fixed population parameter in approximately 95% of repeated samples. It does not mean that 95% of individual observations lie inside the interval. It also does not assign a frequentist 95% probability to the fixed parameter after one interval has been observed.

Open the confidence intervals experiment. Forty synthetic studies estimate the same zero mean with known variance. Reveal the generating value and count the intervals that cover it. Then increase the sample size. Their widths shrink because the standard error is proportional to one divided by the square root of sample size. The coverage rule itself has not become a different confidence level. Penn State’s interval lesson explains the sampling interpretation behind this distinction.

Power asks about a test under a particular effect

Power is a rejection probability under a specified alternative. Asking whether a study has enough power without naming an effect size is incomplete: the same design can detect a large shift reliably and miss a small shift often. A meaningful effect should be chosen for the decision, rather than selected afterwards to make the design look adequate.

In the statistical power simulator, observations have known standard deviation one and the test uses a two-sided critical value of 1.96. Change the true mean shift and sample size separately. The expected rejection probability is calculated from the model; the hundred simulated study outcomes are one fluctuating realisation. Penn State’s power material provides the formal connection between alternatives, sample size and rejection probability.

Worked example: compare precision before making a claim

Use the following settings as a teaching comparison, not an estimate of any medical, commercial or social intervention. The experiment supplies the effect and variance by construction, which is exactly the information real investigators must justify rather than assume.

1. Set the generating shift to 0.3 and the sample size to twenty. Record expected power and the number of rejections among the hundred simulated tests; do not treat those two numbers as identical quantities.
2. Raise sample size to one hundred while preserving the effect and threshold. Compare the new power, then visit the interval board and observe how a larger sample narrows the uncertainty around repeated mean estimates.
3. Return the generating shift to zero. Rejections are now false positives under the null, not discoveries of the previous shift. Keep the approximately five-percent test size separate from power at a meaningful nonzero alternative.

Ask whether uncertainty crosses your decision boundary

Suppose an organisation cares about whether a change saves enough time to justify its implementation cost. A narrow interval around a negligible saving and a broad interval spanning a loss and a useful gain have very different implications, even if both studies miss a conventional significance threshold. Statistical detection and practical value are not interchangeable objectives.

Write the decision threshold before interpreting the result. Identify which values in the interval would lead you to adopt, reject or investigate further. This does not turn a confidence interval into a decision rule automatically; it makes the missing costs, benefits and uncertainty explicit. The right summary may be that the evidence rules out a large benefit but leaves a smaller one unresolved. That is more informative than declaring that nothing happened.

A short review before sharing an inconclusive result

Use this review to prevent a statistical label from replacing the question the study was meant to answer. A transparent design can still yield uncertainty; reporting that uncertainty accurately is useful progress rather than a failure to produce a headline. Keep exploratory findings visibly separate from decisions planned before the data arrived.

• State the estimand, unit of measurement and population to which the result is intended to apply.
• Report an effect estimate and its uncertainty, alongside the test and assumptions used to obtain them.
• Name the practically meaningful alternative before discussing whether the design could detect it.
• Check selection, dependence and measurement problems that increasing sample size alone cannot remove.
• Record which unresolved values would actually change the next decision, rather than relying only on a significance label.

If you keep looking, use a design that expects you to

Repeatedly checking a fixed-sample test and stopping whenever it looks favourable changes the sampling procedure. The original fixed-sample guarantee need not survive. The sequential probability ratio test instead accumulates likelihood evidence between two declared coin models and stops at a predeclared boundary. A stream that reaches the display cap without crossing remains inconclusive.

That board is deliberately narrow: it compares probabilities 0.4 and 0.6, uses nominal error targets and includes discrete boundary overshoot. It is not a general replacement for study planning. The practical lesson is to make the observation and stopping process part of the design, then describe what the resulting evidence can support. Precision, power, sampling quality and stopping rules belong in the same conversation, while each retains its own meaning.