Brier Score

A probability forecast is awkward to grade. A 70% call on an event that never happened was not exactly wrong, since the forecast conceded a three-in-ten chance of that result, and the same call on an event that did happen was not exactly right. Grading a record takes a method that scores the stated number rather than the side it leaned toward. The Brier score is that method, and it has been the standard one since the meteorologist Glenn W. Brier introduced it in 1950 to verify rain forecasts. Forecasting tournaments rank participants with it, research on prediction market accuracy grades prices with it, and any trader who writes down a probability before entering a position accumulates a record it can score. This guide covers the metric itself: the arithmetic, the benchmarks that give a number meaning, and where its main alternative differs.

What is a Brier score?

The Brier score is the average squared difference between forecast probabilities and actual outcomes, computed across every forecast in a record. Each outcome is recorded as 1 if the event occurred and 0 if it did not, each forecast is a probability between the two, and the score is the mean of the squared gaps. Lower is better. A perfect record scores 0, and the worst possible record, full confidence in the wrong outcome every time, scores 1 under the convention used for binary questions.

The metric originated in weather verification. Glenn W. Brier proposed it in 1950 for grading probability-of-precipitation forecasts, which posed the same problem traders face now. A 70% rain forecast followed by a dry afternoon is not refuted, yet a bureau issuing such forecasts for years still needs to know whether its numbers mean anything. Brier's answer was to stop asking whether individual forecasts were right and start measuring how far stated probabilities sat from outcomes on average.

Two properties explain why the score spread from meteorology to forecasting tournaments and market research. First, it is a strictly proper scoring rule, meaning the expected score is minimized by reporting exactly the probability you believe. Shading toward certainty to look decisive, or toward 50% to look safe, worsens the expected score, so the metric can anchor leaderboards without rewarding strategic exaggeration. Second, it prices confidence rather than direction. A hit rate treats a cautious 55% and an emphatic 95% as the same call and credits whichever side the number leaned toward, while the Brier score grades the number itself, which is the part of a forecast that carries most of its information.

The score compresses an entire record into one figure. The related concept of calibration, whether events forecast at 70% actually occur about 70% of the time, unpacks that figure into a curve and is the subject of Calibration.

How is the Brier score calculated?

For a record of forecasts on binary questions, the score is:

$$\text{Brier score} = \frac{1}{N} \sum_{t=1}^{N} (f_t - o_t)^2$$

The formula subtracts each outcome, 1 if the event occurred and 0 if it did not, from the forecast probability, then squares the differences and averages them across all N forecasts. Squaring is what gives the score its character. Consider a single 70% forecast under both outcomes:

$$(0.70 - 1)^2 = 0.09 \qquad\qquad (0.70 - 0)^2 = 0.49$$

A 70% call on an event that occurs adds 0.09 to the average, and the same call on an event that does not occur adds more than five times as much. A 50% forecast adds 0.25 whatever the outcome. The penalty grows with the square of the miss, so the score is gentle with hedged errors and severe with confident ones.

Over a full record, the effect is easiest to see in a worked comparison. Two forecasters assess the same five contracts and lean the same way on every one, differing only in stated conviction. Each cell shows the forecast with its squared error in parentheses, rounded to three decimals.

Both forecasters leaned toward the correct outcome all five times, so a hit rate calls the records identical. The Brier score separates them by a factor of almost five, because Forecaster B's probabilities sat closer to what happened. Conviction is only rewarded when events cooperate, though. Suppose contract 4 had resolved No instead. Forecaster A's error rises from 0.160 to 0.360, lifting the overall score to 0.217, while Forecaster B's error jumps from 0.010 to 0.810 and the score to 0.197. One confident miss nearly erases the advantage built across four confident hits. A strong score therefore requires knowing when to commit as much as which way to lean.

What is a good Brier score?

The scale has four fixed anchors:

The line at 0.25 does the most practical work. A constant 50% forecast scores exactly 0.25 no matter what happens, so any record above that line did worse than stating no opinion at all, and on genuine toss-up questions, beating it is the minimum evidence that a forecaster knows something.

The anchor moves as soon as the questions stop being toss-ups, which is the detail that makes raw scores easy to misread. Suppose a set of contracts on low-probability events, each of which occurs 20% of the time in the long run. A forecaster with no insight into any individual event who simply writes 20% every time collects 0.04 of error on the four in five that do not occur and 0.64 on the one in five that does, which averages to 0.16. That comfortably beats 0.25 while reflecting nothing beyond knowledge of the base rate. On those questions 0.16 is the floor set by public information, and the same 0.16 on true toss-ups would be a strong record. A Brier score is therefore only meaningful relative to the difficulty of its question set.

Shared question sets are what make the published numbers interpretable. The IARPA tournament behind the modern forecasting-craft literature ranked thousands of participants by Brier score on a common stream of geopolitical questions, a design that carried into public platforms such as Good Judgment Open. The cleanest recent reference points come from ForecastBench, a benchmark that scores humans and language models on an identical set of genuinely unresolved questions. At its launch evaluation, the median superforecaster forecast scored 0.096, the median public forecast 0.121, and the best model 0.122, against the 0.25 uninformed baseline. The ordering and the gaps are the informative part; the specific values belong to that question set and would shift on an easier or harder one.

Time to resolution moves scores as well. As an event draws near, remaining uncertainty drains out of the question, and scores tend to improve mechanically with no change in forecaster skill. Comparisons therefore fix the horizon along with the questions, which is why one large analysis of market accuracy evaluated thousands of contracts at 7, 30, and 90 days before resolution.

A final piece of context is what a single number conceals. The standard decomposition of the score separates it into three parts: calibration error, called reliability; decisiveness, called resolution; and the irreducible uncertainty of the questions themselves. Two records can arrive at the same figure in different ways, one honest but timid, the other bold but slightly miscalibrated. How that decomposition grades an entire market's price history belongs to Market Accuracy.

How does the Brier score compare to the log score?

The main alternative grades each forecast by the logarithm of the probability it placed on what actually happened:

$$\text{Log score} = -\ln(p_{\text{outcome}})$$

A forecast that assigned probability 1 to the realized outcome scores 0, and the penalty grows without limit as the probability assigned to reality approaches zero. A 70% forecast on an event that occurs scores about 0.36, a 30% forecast on the same event about 1.20, and a 1% forecast about 4.61. Where the Brier penalty is capped at 1 per question, the log score can let a single forecast dominate an entire record.

Both rules are strictly proper, so under either one the best strategy is to report honest probabilities, and the choice between them comes down to how mistakes should be priced. The Brier score's squared penalty flattens near the ends of the scale, registering almost no difference between 95% and 99.9%. The log score treats that same step as large, on the reasoning that 99.9% asserts the remaining doubt has shrunk by a factor of fifty. The difference matters most in the tails. For contracts priced in single-digit cents, where the live question is whether the true probability is 3% or 1%, a Brier-scored record scarcely notices the distinction while a log-scored record does. Metaculus builds its baseline and peer scores on the log score and averages them over a forecast's lifetime, while the tournament lineage descending from the Good Judgment Project has stayed with the Brier score.

Both rules share limits that no scoring choice removes. A score separates skill from luck only across many resolved questions, and a short record is mostly noise, easily flattered by one fortunate call on a tail event. A self-selected question mix is a second blind spot, since easy questions inflate any record. The limit that matters most for traders is that both rules grade stated probabilities rather than decisions. Neither knows the stakes, the costs, or the price available at the time, so a strong score does not establish that trading on those forecasts would have been profitable. A trader's ledger runs the same comparison with money attached. Each position is an implied probability judgment taken at a price, and it pays off only by the margin between that judgment and the market's, net of costs. The distinction between scoring well and holding an edge over a price is drawn out in Superforecasting.

For an individual record, the choice between the two rules is secondary. Either one, applied consistently to forecasts written down before resolution, turns opinions into an auditable track record, and the audit is what improves the next forecast.

Related guides