Calibration
Calibration curves compare forecast probabilities with outcomes, helping you assess confidence in an analyst's predictions or your own.
Probability forecasts express how much uncertainty remains about an outcome. An analyst who expects a team to win may give it a slight advantage or consider a loss very unlikely. Counting correct picks gives both forecasts the same credit when the team wins.
Forecast calibration helps you assess that confidence using a record of predictions and results. You can check whether your most confident predictions hold up as often as you expect, or whether outcomes you consider unlikely happen more often than you allow for. In prediction markets, those estimates also affect what you are willing to pay for a contract, so a persistent calibration error can change whether a trade appears worthwhile.
What does it mean to be well calibrated?
A forecaster is well calibrated when events happen about as often as their probabilities suggest. Across a large record, events given a 70% chance should happen about 70% of the time. The same agreement should hold at the other probabilities the forecaster uses.
If someone's 90% forecasts repeatedly come true only 70% of the time, their confidence runs ahead of their results. If those forecasts come true about nine times in ten, you have evidence that their use of 90% is appropriate for those questions.
Suppose an analyst gives a tennis player a 70% chance of winning a match, and the player loses. A record of calibrated forecasts at 70% includes roughly 30% losses, so this result fits within the uncertainty the analyst expressed. To assess the analyst's calibration, you need their other forecasts and what happened afterward. One result earns a good or bad Brier score, but gives little information about a recurring calibration error.
The analyst's 70% record can include different one-time events, such as match wins, earnings beats, or rate cuts, as long as each received that probability before its outcome was known. Forecast verification also calls calibration reliability, and uses a reliability diagram to compare those probabilities with observed frequencies.
Seven of ten hypothetical events given a 70% chance happen. The tennis loss is one of the three misses, so it fits within a record whose outcomes match its forecasts.
How do you read a calibration curve?
A calibration curve, also called a calibration plot or reliability diagram, places forecast probabilities on the horizontal axis and observed event frequencies on the vertical axis. The diagonal marks exact agreement. A point at 70% across and 65% up means that events assigned 70% happened 65% of the time.
Here is a hypothetical record of 500 football forecasts. Each question asks whether the home team will win, and each forecast was recorded before kickoff.
| Forecast home-win probability | Matches | Home wins | Observed home-win rate |
|---|---|---|---|
| 10% | 100 | 20 | 20% |
| 30% | 100 | 35 | 35% |
| 50% | 100 | 50 | 50% |
| 70% | 100 | 65 | 65% |
| 90% | 100 | 80 | 80% |
Each row becomes one point on the curve, with the 50% group on the diagonal. The 10% group sits above it because home wins were more common than predicted, while the 90% group sits below it because they were less common. Both ends suggest that the forecaster expressed more certainty than the results supported.
Each point compares the forecast probability with the home-win rate for 100 hypothetical football matches. Home teams given a 10% chance win 20% of the time, while those given 90% win 80%, making the curve flatter than the diagonal and suggesting too much certainty at both ends.
The counts tell you how much evidence sits behind a gap. One result moves a group of ten by ten percentage points, so even a calibrated forecaster's chart will scatter around the diagonal. Error bars that reach the diagonal leave agreement plausible at that probability. A point beyond a chart's expected range of variation gives stronger evidence of a mismatch. The sample-size examples show how to read that uncertainty.
Actual records often contain many different values, so charts collect nearby probabilities into groups called bins. One bin might contain forecasts from 60% up to, but excluding, 70%. Its horizontal position should be the average probability of the forecasts inside it. If they average 62%, that is the value to compare with their observed frequency, even though the bin's midpoint is 65%. Bröcker and Smith show how using bin midpoints can distort a reliability diagram.
The ten hypothetical forecasts cluster near the bottom of the 60% to 70% bin, so their average is 62%. Plotting the group at the bin's 65% midpoint would shift it three percentage points to the right of its average forecast.
Binning involves a tradeoff. Wide bins combine more observations but can hide differences within the group. Narrow bins preserve those differences while leaving fewer outcomes behind each point. Equal-count bins aim for similar numbers of forecasts in each group, so their probability ranges vary. In a rainfall example, Dimitriadis and colleagues showed that changing the number of equal-width bins from nine to ten to eleven substantially changed the apparent calibration pattern.
The chart's labels also tell you which event it counts. A full-range diagram might track the probability of Yes from 0% to 100%. A confidence chart can instead track whichever answer the forecaster favored. A 20% Yes forecast then becomes 80% confidence in No, with a successful forecast counted when No occurs. For binary questions, that chart starts at 50%. The same predictions can look different under these two conventions, so comparisons need matching axes and outcome definitions.
A 20% Yes forecast appears at 80% on a scale that follows the favored answer, No. The full Yes scale runs from 0% to 100%, while the confidence scale covers only 50% to 100%.
How can you tell if forecasts are overconfident or underconfident?
Compare your confidence in the outcome you favored with how often that outcome occurred. If your 80% favorites win only 65% of the time over a substantial record, you have been overconfident in those forecasts. If they win 90% of the time, you have been underconfident.
A chart that tracks Yes probabilities across the full 0% to 100% range needs a different reading. A 20% Yes forecast expresses 80% confidence in No, so certainty increases toward either end of the scale. Koehler, Brenner, and Griffin distinguish four patterns on these charts.
| Pattern | What happens on a full-range chart | What the forecasts are getting wrong |
|---|---|---|
| Overprediction | Points tend to sit below the diagonal. | The named event happens less often than predicted. |
| Underprediction | Points tend to sit above the diagonal. | The named event happens more often than predicted. |
| Overextremity | Low probabilities sit above the diagonal; high probabilities sit below it. | The forecasts are too close to 0% and 100%. |
| Underextremity | Low probabilities sit below the diagonal; high probabilities sit above it. | The forecasts are too close to 50%. |
On these schematic calibration charts, overprediction puts outcome frequencies below the forecasts, while underprediction puts them above. Overextremity flattens the curve around 50% because the probabilities are too extreme, and underextremity steepens it because they are too close to 50%.
Below the diagonal means the named event happened less often than predicted. Whether that amounts to excessive confidence depends on which outcome was favored. If 30% Yes forecasts come true only 10% of the time, the forecaster was too cautious about No. Lowering Yes from 30% toward 10% increases confidence in No. If 90% Yes forecasts come true 80% of the time, lowering the probability reduces excessive confidence in Yes. The football record shows overextremity at both ends.
Both hypothetical groups put Yes too high, with forecasts of 30% and 90% against observed rates of 10% and 80%. The first group is underconfident in No, giving it 70% when it occurs 90% of the time, while the second is overconfident in Yes.
In their review of expert judgment, Koehler and colleagues found overprediction associated with rare events, underprediction with common events, and overextremity with difficulty telling cases apart. They also describe well-calibrated rain forecasts by professional meteorologists.
In a study by Moore and colleagues, 2,860 participants made 494,552 forecasts on 344 geopolitical questions over three years. Their average confidence in the favored answer exceeded its hit rate by about three percentage points. That average combines different confidence levels. At 100% confidence, their chosen answers were right only 84% of the time.
For your own forecasts, look for repeated gaps within a probability range and subject. The number of distinct outcomes behind each gap helps determine how much weight to give it.
How many forecasts do you need to measure calibration?
The count that matters is the number of independent outcomes near each probability you want to assess. If every event truly has a 70% chance, about 96% of groups of ten produce between four and nine occurrences, while about 96% of groups of one hundred produce between 61 and 79. Six successes out of ten therefore fit comfortably within ordinary chance variation.
When independent events each have a 70% chance, larger groups tend to produce outcome frequencies closer to 70%. About 96% of groups of ten fall between 40% and 90%, compared with 61% to 79% for groups of one hundred.
Consider groups of 70% forecasts in which exactly 70% of the events happened. The table gives 95% confidence intervals calculated using the Wilson method described by NIST. These calculations assume independent outcomes with a common underlying probability.
| Events that happened | Resolved forecasts | Observed frequency | Approximate 95% confidence interval |
|---|---|---|---|
| 7 | 10 | 70% | 39.7% to 89.2% |
| 70 | 100 | 70% | 60.4% to 78.1% |
| 700 | 1,000 | 70% | 67.1% to 72.8% |
With seven events out of ten, the interval includes long-run rates as different as 50% and 80%. With 700 out of 1,000, it includes neither. More independent outcomes make smaller differences distinguishable, but a record of thousands may still contain only a handful of forecasts near 95%.
Each group has a 70% outcome rate, with a narrower 95% Wilson confidence interval as the number of independent outcomes grows. The interval for seven events out of ten includes long-run rates of 50% and 80%, while those for 70 out of 100 and 700 out of 1,000 exclude both.
Confidence intervals surround the estimated outcome rate. If the interval includes the forecast probability, the observed gap remains compatible with calibration at that point. A consistency band instead surrounds the diagonal and shows how much a calibrated system's chart is expected to vary. Points outside that band suggest more error than ordinary sampling variation explains. The Metaculus track record provides a public example of a calibration curve with a band for perfect calibration.
Five groups of 100 hypothetical football forecasts appear with 95% confidence intervals around their observed home-win rates and a 95% consistency band around the diagonal. The 10% and 90% groups fall outside the band, and their intervals miss the diagonal, suggesting gaps larger than ordinary sampling variation.
Repeated updates need special care. If you record a 70% forecast every hour for the same rate decision, all those entries eventually receive the same outcome. One hundred updates can produce a chart point based on one decision. Counting them as one hundred independent tests would make the evidence look much more precise than it is.
One hypothetical rate cut settles all 100 hourly forecasts made at 70%. Counting those entries as independent outcomes produces a misleadingly narrow 95% interval of 96.3% to 100%, while counting the single decision gives 20.7% to 100% and leaves 70% plausible.
Separate questions may share an outcome or a common influence. Several Bitcoin price thresholds on the same closing date depend on one price, and one injury may affect several matches involving the same team. The frequencies remain countable, but uncertainty estimates need to account for that dependence.
Pooling subjects creates another issue. Imagine 50 sports forecasts and 50 crypto forecasts, all assigned 70%. If 25 sports events and 45 crypto events occur, the combined record lands exactly on 70%. Separately, the observed rates are 50% and 90%. Checking relevant subjects or time horizons can reveal errors the combined chart hides, although each smaller group will have wider uncertainty.
All 100 hypothetical forecasts assign 70%, and 70 events occur overall. The combined rate hides the difference between the two groups of 50, where sports events occur 50% of the time and crypto events 90%.
How does calibration differ from accuracy?
Calibration asks whether stated probabilities match outcome frequencies. Accuracy can mean how often someone picks the winning side, or how well their full probability estimates score. A forecaster can be well calibrated while making little distinction between easier and harder questions.
Suppose the home team wins 60% of matches in a particular competition. Someone who always predicts a 60% home-win probability would be calibrated if that rate continues. They would also pick the winner 60% of the time by always choosing the home team. Their forecasts would give you no help identifying which particular matches favor the away team.
A second forecaster might separate the matches into two equally sized groups where home teams win 40% and 80% of the time. Assigning those groups probabilities of 60% and 90% puts the more likely home wins first, yet overpredicts both groups. Lowering the estimates to 40% and 80% fixes that calibration error without changing the ranking.
In this hypothetical competition, a constant 60% home-win forecast matches the overall rate, while forecasts of 60% and 90% separate the two groups but overpredict both. Adjusting those estimates to the groups' observed rates of 40% and 80% corrects calibration and preserves their ranking.
ROC AUC measures how well forecasts rank eventual wins above losses. Changing the second forecaster's estimates preserves that ranking and its AUC while correcting the calibration error. The measures answer different questions about forecasts:
| Quality | The question it answers |
|---|---|
| Calibration | Do events assigned a given probability occur at that rate? |
| Hit rate or classification accuracy | How often does the chosen answer turn out to be right? |
| Discrimination | Do the forecasts rank events that happen above events that do not? |
| Resolution | Do the forecast groups have different observed outcome rates? This use of resolution concerns forecast quality, while market resolution means settling a contract. |
| Sharpness | How concentrated are the predicted possibilities? For a binary event, probabilities nearer 0% or 100% express more certainty. |
The Brier score decomposition combines calibration, resolution, and the overall mix of outcomes. Better separation between forecast groups can outweigh a calibration error, so a lower score can accompany poorer calibration.
Sharper binary forecasts use more probabilities near 0% and 100%, expressing greater certainty. Changing the constant 60% home-win forecast to 90% makes it sharper immediately, but it overpredicts wins if the home-win rate stays at 60%. The useful improvement would be to identify matches that really have higher or lower chances, then assign probabilities that match those groups' results. This follows the principle of maximizing sharpness subject to calibration.
What is expected calibration error?
Expected calibration error, or ECE, averages the gaps between forecasts and outcome frequencies within groups of similar probabilities, called bins. Each bin contributes in proportion to its number of forecasts, so larger groups carry more weight. Lower values mean closer agreement under that grouping.
For binary Yes/No forecasts, multiply each bin's absolute gap by its share of all forecasts, then add those contributions. Using absolute gaps keeps errors above and below the diagonal from cancelling. Maximum calibration error reports the largest bin gap instead of the average. Naeini and colleagues define both measures.
In the football example, the five gaps are 10, 5, 0, 5, and 10 percentage points. Each group contains one fifth of the forecasts, so ECE is six percentage points, or 0.06 on a zero-to-one scale. Maximum calibration error is ten percentage points, or 0.10.
Five equal groups of hypothetical football forecasts have gaps of 10, 5, 0, 5, and 10 percentage points between their predictions and home-win rates. Weighting each gap by its group's 20% share gives an expected calibration error of 6 points, or 0.06, while the largest gap gives a maximum calibration error of 10 points.
Putting all 500 football forecasts into one bin reduces ECE to zero. The average predicted probability is 50%, matching the overall home-win frequency of 50%, although the original table shows errors at both ends. The bin definitions and counts reveal which probabilities were combined, and the curve shows where their errors occurred. Those details make a reported ECE interpretable.
With five bins, 500 hypothetical football forecasts have an expected calibration error of 0.06. Merging them into one gives a 50% average forecast and a 50% home-win rate, so the measure falls to zero even though the forecasts and outcomes are unchanged.
Merging the football groups changes ECE without changing any individual forecast or outcome, so it leaves the Brier score untouched. That score averages squared errors for individual forecasts, while ECE uses absolute gaps between group averages and outcome frequencies. Even the Brier score's calibration component squares the gaps, making it more sensitive to large misses.
How can you test your own calibration?
For a quick confidence calibration test, Quantified Intuitions offers exercises with answers you can check immediately. To assess forecasts about the subjects you follow, record probabilities before events resolve, then compare each probability group's forecasts with its outcome frequency. A spreadsheet or prediction tracker is enough for that longer record.
These tools differ in the questions they use and the feedback they give:
| Tool | What you do | Feedback |
|---|---|---|
| Quantified Intuitions | Answer confidence questions, including estimates of ranges. | Calibration charts and interval scores. |
| Pastcasting | Forecast resolved questions using information available at an earlier date. | Log scores relative to the original crowd forecast, measuring broader forecasting performance. |
| Fatebook | Record and resolve your own questions. | A calibration chart and Brier scores, as described by its creators. |
| Metaculus | Forecast public questions and update as evidence arrives. | Calibration curves and scores, also shown in its public track record. |
| Good Judgment Open | Forecast public questions and compare reasoning with others. | Brier scores and comparisons with the crowd. |
Range exercises assess interval calibration by checking whether your ranges contain the answer as often as you claim. For example, ranges intended to have a 90% chance of containing a company's quarterly revenue should contain it about nine times in ten across comparable forecasts. Too many misses show that the ranges claim more coverage than they achieve. Recording which side each miss falls on helps identify whether the ranges are too narrow or centered too high or low.
Seven of ten hypothetical quarterly revenue figures fall within ranges assigned 90% confidence, giving 70% coverage in this record. All three misses fall below the ranges, suggesting the forecasts were centered too high.
For your own record, preserve the exact question, your probability, its timestamp, and the source that will settle the answer. “Will the central bank cut its policy rate at the next scheduled meeting?” has a definite event and deadline. “Will rates fall soon?” leaves both open to interpretation. For market questions, the contract's resolution criteria define the event your probability needs to describe. Keep your estimated probability separately from any trade price.
Once questions resolve, compare the forecasts with their results:
- Record each resolved outcome as Yes or No using the stated criteria. Keep unresolved and excluded questions visible, with reasons for exclusions.
- Choose a consistent forecast time, such as one day before each event. Preserve earlier estimates, but use only the chosen observation for each question in this comparison.
- Group comparable probabilities into bins. For each bin, calculate the average forecast probability and divide the number of Yes outcomes by the number of resolved questions.
- Compare the two percentages and retain the count. If 100 forecasts average 70% and 65 resolve Yes, the observed frequency is 65%, five percentage points below the forecast. Plot that point at 70% across and 65% up on a calibration curve.
In that example, a Wilson 95% confidence interval for the outcome rate is about 55% to 74%, assuming independent, comparable questions. It includes 70%, so that gap leaves calibration plausible. The sample-size section explains how counts and related outcomes affect this reading. Keeping subject and lead time in the record also lets you check whether a gap recurs in particular kinds of forecast.
How can you improve forecast calibration?
Use outcome feedback to identify which probabilities repeatedly miss their mark, and review the evidence and base rates behind them. When rare events occur less often than you predict, their estimates need to move down. When favorites win more often than you predict, their estimates need to move up. Feedback and probability training have helped forecasters make such corrections in studies of geopolitical and weather forecasts.
In Moore and colleagues' geopolitical forecasting study, untrained individuals' average confidence in their favored answers exceeded their hit rate by 4.3 percentage points. The gap was 2.0 points for trained individuals and 0.7 points for trained teams. The Superforecasting guide covers the wider Brier-score gains and the dispute over how much they demonstrate improved ability after accounting for question choice and timing.
Average confidence in the favored answer exceeded its hit rate in all three groups from Moore and colleagues' geopolitical forecasting study. The excess was smallest for trained teams, at 0.7 percentage points, compared with 2.0 for trained individuals and 4.3 for untrained individuals.
Dutch weather forecasters received detailed feedback on a year's predictions in a study reported by Koehler and colleagues. Their next year's curve moved closer to the diagonal, although they continued to overpredict rare weather events.
If high-confidence earnings forecasts fail unusually often, use your original notes to check which companies you chose for comparison. A reference group containing only companies with improving guidance would be a poor starting point for companies without it.
A historical curve also suggests a possible numerical correction. Suppose 100 of your forecasts at 90% produced 80 Yes outcomes. Replacing 90% with 80% would remove that gap in the existing record. To find out whether the adjustment helps on new questions, keep both your original probabilities and the adjusted versions. Later outcomes let you compare their calibration on the same questions.
Compare forecasts on similar subjects at similar lead times. If later estimates concern easier questions or arrive much closer to the answer, the change in circumstances may explain their closer agreement with outcomes.
How does probability calibration work in machine learning?
Model calibration checks whether a model's probabilities agree with observed outcomes. For a binary classifier, cases assigned an 80% probability of belonging to a class should belong to it about 80% of the time. For a forecast model, the same check compares its event probabilities with subsequent outcomes.
A model's design affects the errors it tends to make. In scikit-learn's worked comparison, naive Bayes produces probabilities too close to 0% and 100%, while a random forest avoids the extremes. The logistic regression model, whose assumptions match the data, is well calibrated in that example.
Recalibration learns a correction from examples with known answers. Platt scaling, also called sigmoid calibration, fits a smooth S-shaped mapping. Isotonic regression fits a more flexible mapping that never decreases as the original score increases, with greater risk of overfitting on small datasets. Both need calibration examples separate from the model's training data. A further test on unseen cases checks whether the correction holds up.
The hypothetical model predicts 5% and 95% for groups whose observed rates are 17% and 80%, expressing too much certainty at both ends. Platt scaling fits a smooth correction curve, while isotonic regression follows the groups with steps that stay level or rise.
In a 2017 study of image and document classifiers, Guo and colleagues found that modern neural networks were often overconfident. Temperature scaling, which adjusts how concentrated their probabilities are, improved calibration on most datasets they tested.
What do calibration slope and intercept mean?
A calibration slope summarizes whether a model's probabilities are too extreme or too moderate. Under the logistic recalibration convention, its target is one. A value below one suggests the model exaggerates the differences between high- and low-probability cases. A value above one suggests it understates those differences.
Calibration in the large compares the mean predicted probability with the overall event rate. A calibration intercept calculated with the slope fixed at one measures this overall mismatch. Its target is zero, with negative values suggesting overprediction and positive values suggesting underprediction. Van Calster and colleagues explain these measures and why a curve is still needed to reveal local errors. They advise against the Hosmer-Lemeshow test, whose grouped pass-or-fail result says little about an error's size or shape.
Are LLM confidence scores calibrated?
A chatbot's written “90% sure” needs its own test against verified answers, because that percentage is a different output from the probabilities assigned to answer options. In Kadavath and colleagues' 2022 study, larger models were well calibrated on multiple-choice and true-or-false questions presented in suitable formats. The 2023 GPT-4 technical report found good calibration before additional training, then substantially worse calibration afterward on the same multiple-choice questions.
Both findings concern answer-option probabilities. For GPT-4, expected calibration error on the tested subset of the MMLU benchmark rose from 0.007 to 0.074.
An AI answer is paired with a record of answers checked against evidence. The record gives the verified results needed to compare written confidence with how often answers are correct.
How are Polymarket and Kalshi calibration charts made?
A prediction market calibration chart pairs historical prices with settled outcomes, groups those observations by implied probability, and compares each group's average with its Yes frequency. The choices of price, observation time, and included markets determine what record the chart describes.
Each settled contract has many past prices but one outcome. To use its price history in the market screener for a calibration check, you need a consistent rule for which observation represents the forecast.
Polymarket's accuracy page builds its “Expected vs Actual” chart from resolved markets, using prices one month, one week, one day, twelve hours, and four hours before resolution. Its methods text leaves the bins and weighting unspecified, including how contracts from the same event are counted.
Kalshi Research's paper plots prices three months, one month, one week, and one day before close, counting individual contracts. It compares contracts with observations at each horizon against a fixed set already open a month ahead. That separates changes in prices from changes in which markets enter the sample. Further comparisons exclude near-certain prices to examine markets whose outcomes remain uncertain, or measure election forecasts from election day. The Brier Score guide works through how those choices change the reported scores.
Findings also vary by subject. Le's 2026 preprint found political favorites winning more often than prices implied on both venues, while short-horizon sports prices were closer to calibrated. Market Accuracy examines that evidence in detail.
Which prices belong in a market calibration curve?
A market calibration chart needs both a price measure and a sampling time. The latest trade and the midpoint between bid and ask quotes may differ, especially when trading is sparse. The chart's method should state which it uses and how it handles stale or missing observations.
The selected record and clock represent the price and observation time that must be chosen before a historical market forecast can be compared with its outcome.
Timing might mean a fixed period before the event, a fraction of the market's lifetime, or the final available price. A month-long market's halfway point and a two-day market's halfway point leave very different amounts of time for news to arrive.
Four hours before resolution, as in Polymarket's chart, can still be after the answer is public. Prices then reflect a known outcome, so close agreement says little about how well they anticipated the event. Kalshi's comparison using election day addresses the same timing problem by measuring how far ahead of the vote each price was recorded.
Calibration City offers observation times such as halfway through a market's life, at close, or at another fraction of its duration, alongside an average price over its lifetime. Its implementation notes explain the price conventions and data coverage behind those comparisons. A chart using prices a week before the event needs to be compared with another using the same lead time.
Why do prediction markets look underconfident?
Prices look underconfident when high-priced outcomes happen more often than implied and low-priced outcomes less often. Averaging prices over a market's life can create that shape. Studies using prices observed at specified horizons have also found underconfidence, with the cost of tying up capital proposed as one explanation.
In their study of Intrade, Page and Clemen found prices more compressed toward 50% farther from expiration. In their model, money committed to a contract earns no interest, so traders whose beliefs are close to the price stay out. Buying a favorite ties up more money than buying a longshot, which discourages participation more on the favorite's side. That imbalance leaves favorites underpriced and longshots overpriced, pulling both toward 50%. Averaging prices over a market's life can create a similar shape for a different reason, by mixing estimates made before and after news arrives.
Consider a simplified set of equally long markets that stay at 50% for the first half of their lives. Halfway through, the outcomes become known with certainty. The half that resolve Yes move to 100%, and the half that resolve No move to 0%. Assume those prices remain until settlement.
The hypothetical markets spend half their lives at 50%, then move to 100% or 0% when their outcomes become known. Averaging each full price path gives 75% for the Yes market and 25% for the No market, even though neither was a quoted price.
The eventual Yes markets have a lifetime average of 75%, and the eventual No markets average 25%. Grouping markets by those averages gives a 100% Yes frequency at 75% and a 0% Yes frequency at 25%, creating an underconfident curve. The original 50% forecasts matched the overall Yes frequency, and the later forecasts correctly reflected known outcomes.
The hypothetical markets' lifetime averages produce a curve that looks underconfident, with no Yes outcomes at 25% and all Yes outcomes at 75%. Their 50% prices before the news match the overall 50% Yes rate and put that point on the diagonal.
Grouping by lifetime average sorts these markets using information that arrived later. The 75% and 25% averages were never quoted in this example. A curve built from prices observed before the news avoids this distortion, leaving any underconfidence in those earlier prices to be explained by other mechanisms.
What does a market calibration chart count?
A market calibration chart can count contracts, whole events, or repeated price observations. Trading activity can give some observations more weight than others, changing which markets dominate the curve.
Under the per-contract counting used in Kalshi's paper, a championship with eight contenders and exactly one winner contributes eight observations, one Yes and seven No. That event receives eight times the weight of an event with a single Yes/No contract. Giving each contract equal weight also lets a lightly traded contract count as much as a heavily traded one.
Each square is a contract counted once in a calibration chart. The championship contributes one Yes and seven No results, giving it eight times the weight of the event with a single contract.
Daily sampling adds more observations from long-running markets. Volume weighting depends on whether the weight belongs to the contract or to an individual trade. A chart using each contract's total volume favors heavily traded markets even if little trading occurred at the sampled price. Using each trade's size instead emphasizes the prices at which more contracts actually changed hands.
Does better calibration help prediction market traders?
Better calibration helps a trader put a more defensible value on a contract and judge how much money to expose to the outcome. At a 70% probability, a one-dollar Yes contract has an expected value of 70¢ before costs. Paying that same price leaves no expected advantage. The Prices and Probabilities guide explains how purchase prices and costs turn an estimate into an expected gain or loss.
Suppose your record shows that events you assign 90% have happened only 80% of the time across many comparable forecasts. For a new contract costing 85¢, the original estimate suggests a 5¢ expected gain before costs. Adjusting to the historical 80% rate instead suggests a 5¢ expected loss.
The 85¢ price falls between the original 90% estimate and the adjusted 80% estimate for a hypothetical one-dollar Yes contract. Using the trader's historical 80% rate changes an expected 5¢ gain into a 5¢ loss before costs.
The Kelly criterion calculates a fraction of available capital intended to maximize long-run compound growth under the assumed probabilities. In a simplified example with no fees, a 50¢ Yes contract with a known 60% chance gives a Kelly fraction of 20%. Entering an overconfident estimate of 70% doubles that fraction to 40%, although the contract's actual chances are unchanged. On a $1,000 account, those positions risk $200 and $400 respectively if No occurs.
Across repeated independent positions at that same price and true probability, committing 20% of the current account each time gives long-run compound growth of about 2% per position. Committing 40% instead makes that growth rate slightly negative. The oversized positions erode the account over time even though each contract has positive expected value.
Across independent positions in hypothetical 50¢ Yes contracts with a true 60% chance and no fees, long-run growth peaks near 2% per position when 20% of the current account is committed. An overconfident 70% estimate calls for committing 40%, where growth falls to about −0.2%.
The forecasts you trade on also deserve a separate review. You may forecast many events but buy only where your estimate differs sharply from the market. Agreement across the full record can hide errors in that selected group. Keeping estimates for both traded and untraded questions lets you examine whether the probabilities behind your positions match their outcomes.
Brier Score
How to score probability forecasts and separate calibration from other aspects of accuracy.
Market Accuracy
Evidence on Polymarket and Kalshi forecasts across subjects and time horizons.
Superforecasting
How forecasters form, update, and review probability estimates.
Base Rates
How past frequencies inform a forecast and how to choose comparable cases.