Brier score explained: how to judge a probability forecast fairly

The binary Brier score is squared probability error

For one event, encode occurrence as one and nonoccurrence as zero. Subtract that outcome from the forecast probability and square the difference. Average across events for the binary Brier score. The score ranges from zero to one under this convention, and lower is better. An eighty-percent forecast earns 0.04 if the event occurs and 0.64 if it does not.

Glenn Brier's original forecast-verification paper introduced a broader categorical formulation. The binary version used here scores one event probability and is half the sum over both binary categories. Naming the convention matters when comparing published numbers. You cannot compare scores reliably if one source doubles a binary score or averages across a substantially different set of cases.

Try committing forecasts before seeing the outcomes

The Brier-score game gives ten synthetic events with known generating probabilities. Set your probabilities or copy the generator's, then commit. Outcomes are sampled independently from those known values and revealed only after the forecast controls lock. This prevents adjusting the probabilities after seeing which events occurred.

The generator information makes this a teaching exercise rather than a forecasting competition. In a real journal you do not know the true probabilities. Here, knowing them lets you compare realised scores with expected loss. A less faithful forecast can win a small realised batch by luck even though the generating probability minimises expected squared error for each event.

Worked example: honest uncertainty can lose one round

Suppose the event truly has probability eighty percent. Compare forecasting eighty percent with forecasting certainty. The following arithmetic uses a stipulated probability to make the distinction between realised and expected scores inspectable; it is not a measured success rate for any real forecaster.

1. If the event occurs, the eighty-percent forecast scores (0.8−1)² = 0.04, while a one-hundred-percent forecast scores zero and looks better that round.
2. If the event fails, the scores are 0.64 and one. That less frequent outcome penalises the certainty forecast more heavily.
3. Average using the true probability: honest forecasting has expected loss 0.8×0.04 + 0.2×0.64 = 0.16, versus 0.2 for certainty. The expected advantage does not guarantee winning every batch.

Calibration compares stated probabilities with frequencies

Among many comparable events forecast near seventy percent, do roughly seventy percent occur? That is the intuition behind calibration. The game groups forecasts into three bins and reports observed event rates, but ten cases produce very sparse groups. An empty group provides no evidence, and a group of two or three events cannot settle whether a forecaster is calibrated.

Calibration alone also does not establish useful discrimination. A model that always predicts the overall event rate can be calibrated while failing to distinguish high- and low-risk cases. The precision-and-recall experiment offers a separate lesson about selecting cases and counting relevant outcomes. Classification thresholds, ranking quality and probability calibration should not be collapsed into a single claim that a system is accurate.

Compare against a reference on the same cases

A raw score is easier to interpret alongside a sensible reference forecast scored on the identical events. The game shows the generator's realised score because the simulation knows it. In a real forecasting exercise, possible references include a pre-specified base rate or a simple model trained without using the evaluation outcomes. Choosing the reference after inspecting results can make a performance claim misleading.

The central-limit-theorem experiment helps separate variability in individual outcomes from variability in averages. Its assumptions should not be transferred uncritically to a journal of dependent events. Ten related election questions or ten milestones in the same project may provide less independent information than their count suggests. Evaluate the case mix and dependence alongside any average score.

Set up a forecasting journal you can audit

A forecasting record needs enough specificity that a different reader can resolve the event without guessing what you meant. A probability entered after the result is known is commentary rather than a forecast. Revisions can be useful, provided their timestamps are preserved and the evaluation rule says which forecast will be scored.

• Write an event, deadline and unambiguous resolution criterion.
• Record probability and timestamp before resolution.
• Preserve revisions and choose an evaluation policy in advance.
• Score a reference forecast on the same cases.
• Inspect calibration with sample sizes and uncertainty visible.
• Review errors without rewriting the original question or probability.

Use the score to learn, not to create false certainty

The NOAA memorandum on verification of precipitation probabilities illustrates a real verification context and explicitly distinguishes an abbreviated binary Brier convention. The scoring idea is portable, but the event definition and operational setting remain important. A score summarises forecast quality under a specified loss; it does not by itself tell you which action to take.

Decisions also require consequences and constraints. A well-calibrated small probability can justify action when the cost of missing the event is large, while a likely event may not justify an expensive response. Keep forecasting, scoring and deciding connected but distinct. A good review asks what the probability meant, whether it was honestly stated, how the score compares with a reference and what information could improve the next forecast.