Correlation vs causation: how to tell whether a change actually worked

Start with the change you want to explain

Write the claim as an intervention: giving access to this course changes a specified outcome for a specified population over a specified period. That sentence is more useful than asking whether training and performance are correlated. It identifies the treatment, the result, the people and the time horizon. It also exposes ambiguities: being offered a course, enrolling in it and completing it are different interventions.

The missing comparison is the same population without the intervention. You cannot usually observe both possibilities for the same person at the same time. Research designs try to construct a defensible substitute. Before choosing a statistical tool, ask why the observed comparison group should represent that missing possibility. A complex model cannot rescue a comparison whose assignment process has no plausible causal interpretation.

Check who selected into the treatment

In the confounding experiment, treatment adds ten outcome units within each background group. Nevertheless, the combined treated group can look much stronger because it contains more people with a background advantage. The larger raw gap is partly a different mixture of people. Changing the treatment's composition changes the reported association even though its causal effect stays fixed.

Pearl and Paz's work on confounding equivalence provides a graphical causal perspective on adjustment. For a practical first pass, draw arrows from plausible background conditions to both participation and outcome. Prior experience, motivation, eligibility and location are candidates to investigate, not a magic list to adjust automatically. Some measured variables are consequences of treatment or selection and can make adjustment worse.

Worked example: an impressive course result

These numbers are from Modic's authored model, not an evaluation of an actual course. The purpose is to audit what a combined average conceals. You can reproduce every arithmetic step before considering a real dataset, where the causal effect and group counterfactuals would remain uncertain.

1. Set the advantaged treated share to seventy percent. Expected control mean is 32 and treated mean is 58, giving a raw gap of 26 units.
2. Inspect background rows: control versus treated is 20 versus 30 in one group and 60 versus 70 in the other. Both differences are ten.
3. Randomise expected assignment to balance both background shares at fifty percent. The combined gap becomes ten; the previous extra sixteen came from composition.

Compare changes when starting levels differ

The difference-in-differences experiment starts two groups at different levels. That does not prevent a useful comparison of changes. If the untreated trends would have remained parallel, subtracting the comparison group's change removes the shared trend. The important assumption is about an unobserved future change, rather than a requirement that both groups begin with identical averages.

Turn up the treated group's extra trend. The policy effect remains eight, but the estimate rises because it absorbs the trend difference. Show the no-policy line to inspect the failure directly. Real studies do not get that reveal button. They need institutional knowledge, diagnostics, robustness checks and a defensible design. A calm-looking pre-period graph is evidence to consider; it is not proof that nothing else changed after the intervention.

Use encouragement only if its shortcut is credible

The instrumental-variables experiment asks whether encouragement changes outcomes by changing treatment uptake. Divide the encouragement's outcome difference by its uptake difference. Under the displayed homogeneous model with no direct shortcut, that ratio recovers the treatment effect. Add a direct encouragement bonus and the same arithmetic produces the wrong answer.

Scott Cunningham's causal-inference textbook develops these research designs in more depth. A useful invitation example is an encouragement assigned independently of potential outcomes that affects attendance but does not otherwise change the outcome. That exclusion claim needs an argument. In richer settings the estimated response may apply to people whose behaviour the invitation changes, rather than everyone. An almost irrelevant instrument is not rescued by an impressive-sounding name.

A causal-claim checklist you can actually use

Write a short design note before looking for a favourable result. Distinguish assumptions you can partially investigate from claims the design cannot verify directly. If a claim survives, the note should explain why the comparison is informative; if it fails, you should still know which observation was useful and which inference went beyond it.

• Define treatment, outcome, population and time horizon in one sentence.
• Explain what would stand in for the untreated counterfactual.
• List plausible selection mechanisms and concurrent changes.
• Ask which design assumption would fail under each alternative explanation.
• Report uncertainty, spillovers and the population to which the result can apply.

Keep the conclusion as narrow as the design

Correlation is valuable for prediction and description even when it cannot identify an intervention effect. A customer characteristic might help forecast retention without being something that would improve retention if changed. Keep that distinction in the wording of a recommendation. Saying a result predicts an outcome is a different claim from saying an intervention causes the outcome.

The experiments make assumptions visible by giving you access to their data-generating rules. Real work is harder precisely because those rules are uncertain. Use the models to ask sharper questions, not to label an observational comparison causal merely because it resembles a familiar diagram. The most useful next action may be collecting better assignment information or running a well-designed experiment rather than adding another regression specification.