Sampling bias vs publication bias: why more visible data can still mislead

Coverage comes before response count

The sampling frame defines who has a route into a study. If an online-only survey excludes offline households and the excluded group differs on the outcome, increasing online responses cannot close that coverage gap. This is a different issue from ordinary random variation among representative samples.

The sampling-bias activity uses an invented town with eighty online and twenty offline households. Sixty online households support a proposal; only five offline households do. The complete ledger makes the target quantity known, allowing you to inspect the selection mechanism directly. OpenStax’s sampling chapter provides university-level context. The numbers do not estimate real internet access or political opinion; their purpose is to separate a coverage problem from an impressive-looking response count.

Publication adds a selection stage after data collection

Publication bias concerns which results become available to readers. Even if each underlying study follows the stated sampling model, selecting what is shown according to its findings changes the visible evidence. The missing results matter because the visible subset was not chosen independently of the statistic being interpreted.

Open the publication-bias file drawer. One hundred synthetic studies estimate a true mean of zero, but a journal displays only two-sided z scores beyond 1.96. Rosenthal’s file-drawer paper supplies primary methodological context. Notice that selected absolute estimates are exaggerated even when positive and negative estimates balance to a signed mean near zero. A small selected mean does not by itself show that selection is harmless, just as this stipulated rule does not measure actual publication practices in a field.

Worked example: reconstruct the excluded population

Use the fictional household counts to make the target, frame and selected estimate explicit. The arithmetic is exact because the complete population is supplied; a real survey usually requires additional information to assess the missing group.

1. Choose the online-only frame and calculate sixty supporters among eighty households, giving seventy-five percent. This is an accurate summary of the selected frame even though it is not the complete-town support rate.
2. Reveal twenty offline households with five supporters. Combining groups gives sixty-five supporters among one hundred households, or sixty-five percent, which differs by ten percentage points from the online summary.
3. Imagine collecting more online responses without adding offline access. Precision within that frame could improve, but the unsupported population generalisation remains. Document the frame rather than describing the gap as ordinary sampling noise.

A correct group summary can still support the wrong inference

The ecological-fallacy exercise shows two districts with a positive relationship between library access and passing. Detailed cells then reveal that access and passing are unrelated within each district in the stipulated table. The between-district association does not establish the same individual relationship or a causal effect of access.

This does not make aggregate evidence useless. It means the level of the claim needs justification. A district-level resource question can be answered at that level; an individual causal claim needs more. Keep the group summary, the individual comparison and the causal interpretation separately labelled. Otherwise an accurate calculation can be repeated confidently while the unsupported step occurs in the sentence describing what the calculation supposedly means.

A short selection audit before sharing a result

Treat this as an evidence map rather than a checklist that mechanically certifies a study. Several selection stages can interact, and one can remain unresolved even when another is well documented. The useful output is a clear statement of what was observed, what remains missing and which conclusions depend on assumptions about that missing information.

• Name the intended population and identify who had no opportunity to enter the sampling frame.
• Separate nonresponse, coverage and random variation rather than treating them as one generic uncertainty.
• Ask whether result availability depended on significance, direction, novelty or another outcome-related filter.
• Keep aggregate descriptions distinct from claims about individuals and from causal explanations.
• Report the assumptions needed to extend beyond the observed frame or visible set of studies.

A robust summary does not repair selection

The robust-statistics board shows how a median can resist one extreme observation while an arithmetic mean moves substantially. That is useful sensitivity analysis, but it does not recover people excluded from a sample or studies hidden in a file drawer. Resistance to one departure is not a general guarantee that the evidence answers the intended question.

Likewise, range restriction changes observed correlation by selecting a narrow predictor band while preserving the generating relationship. Each board isolates a particular stage of the problem. Use the distinctions together: inspect the frame, availability rule, measurement and level of inference before selecting a numerical summary. More data can improve an answer only after it is clear which question those data are actually positioned to answer.