Superforecasting
Superforecasters' methods help you turn evidence into probability estimates and review your predictions against what actually happens.
Forecasting requires you to weigh evidence that can point in different directions. A company may report progress on a new phone while a supplier warns of delays. Estimating the chance that the phone will ship on time makes that judgment explicit. Recording the probability and your reasons lets you revisit the forecast as new reports arrive, then examine which assumptions held up once the outcome is known.
Superforecasting grew out of research into people who did this unusually well. The Good Judgment Project tracked probability forecasts across many questions to identify sustained accuracy and investigate the habits behind it. For a prediction market trader, developing an estimate also provides a basis for judging a contract's price. Keeping your own forecast record lets you assess whether those judgments are improving over time.
What is a superforecaster?
A superforecaster is someone with a record of unusually accurate probability forecasts across many questions. The term comes from the Good Judgment Project, which identified its strongest participants through forecasting tournaments. Good Judgment's account of the selection process says the original designation went to the top 2% within each experimental group, among participants who answered at least 50 questions in a season.
Superforecasting means making probability judgments that can be checked and improved. You specify an outcome, assign a probability, revise it when the evidence changes, and keep a record of how accurate you were.
A probability contains more information than a prediction of who wins. Someone assigning a team a 55% chance and someone assigning it 95% both expect the team to win. They disagree substantially about how surprising a loss would be. Over many forecasts, that difference can be measured.
The two forecasts favor the same team, with win probabilities of 55% and 95%. A loss has a 45% chance under the first forecast and just 5% under the second, so it would be much more surprising to the second forecaster.
What did the Good Judgment Project find?
The Good Judgment Project was a research program led by Philip Tetlock and Barbara Mellers at the University of Pennsylvania that tested how people could forecast world events more accurately. Thousands of volunteers took part in the IARPA forecasting tournament from 2011 to 2015. The project found that some participants sustained unusually accurate forecasts, while training and teamwork improved scores in the original experiments. The tournament studies and four-year training analysis report those results.
IARPA funded the competition through its Aggregative Contingent Estimation program, which investigated ways to collect and combine forecasts. Participants assigned probabilities to questions about elections, economic developments, and other world affairs. Researchers scored the forecasts against outcomes using the Brier score, an error measure where lower is better.
A globe, source reports, and a pencil represent the research behind the Good Judgment Project's forecasts about world events.
The project tested several ways of producing a collective forecast. Its strongest system outperformed the intelligence community's prediction market, while comparisons of markets and polls showed how much results depended on who participated and how forecasts were combined. Later research challenged the explanation for the training and teamwork advantages.
What is the Superforecasting book about?
Superforecasting: The Art and Science of Prediction, by Philip Tetlock and Dan Gardner, argues that people can improve their forecasts through careful reasoning and practice with feedback. It tells the stories of Good Judgment Project volunteers and explains their habits, including comparing similar cases, assigning probabilities, and reconsidering judgments. The publisher's description highlights their success against the collective judgment of intelligence analysts, a comparison examined below.
The book's practical advice ends with the ten commandments for aspiring superforecasters. These encourage readers to make their reasoning explicit enough to revise and to investigate mistakes after outcomes become known.
Tetlock's earlier Expert Political Judgment followed 284 experts who made more than 82,000 forecasts. They barely outperformed informed non-experts and failed to beat simple statistical rules, according to Tschoegl and Armstrong's review. Within that study, foxes, who considered several explanations, did better than hedgehogs, who relied on one dominant theory. In a rate forecast, a fox would consider whether weak employment offsets the case for higher rates from inflation. A hedgehog committed to an inflation-only theory would give the employment report little weight.
For the fox, inflation favors higher interest rates while weak employment weighs against them. The hedgehog relies mainly on inflation, so the same employment report has little effect on the forecast.
Are superforecasters real?
The Good Judgment Project found that selected top performers continued to outperform other participants over two further tournament years. Their scores improved after selection while the comparison groups' scores worsened, which is evidence against the advantage being only a lucky first season. Mellers and colleagues' 2015 study followed their performance across new questions and several measures of accuracy.
After selection, the researchers placed the top performers in elite teams. That changed both who they worked with and their access to other strong forecasters' reasoning. Selection and the new working conditions changed together, making their separate contributions difficult to measure.
The designation marks a selection cutoff within a tournament. The evidence supports differences in forecasting skill across participants, without establishing a sharp boundary between two kinds of people.
Each dot is one of 100 hypothetical forecasters, ordered by accuracy, with the top 2% selected. The next few participants are nearly as accurate as the two who qualify.
What makes a superforecaster?
Measured superforecaster traits included strong reasoning ability, enjoyment of difficult problems, and a willingness to reconsider views. The selected group also spent more effort gathering information and revisiting questions. Mellers and colleagues found differences in both abilities and behavior:
- Reasoning and knowledge. The group scored at least one standard deviation above the general population on tests of reasoning and accumulated knowledge. They also knew more about the political subjects they were forecasting.
- Open-mindedness. They scored higher on measures of willingness to consider opposing arguments and change their minds, and they enjoyed solving demanding problems.
- Research effort. Across the second and third years, they clicked an average of 255 news stories in the project's news reader, compared with 55 and 58 in the two comparison groups.
- Continued attention. They attempted more questions and revised their forecasts more frequently. Updating frequency was the strongest single behavioral predictor of accuracy in the researchers' analysis.
In the Good Judgment Project's news reader, superforecasters averaged 255 clicks per person across the second and third years. The next-best forecasters on regular teams averaged 55 and the remaining participants 58, less than a quarter as many in either group.
These were observed associations. They show what distinguished the group, without measuring what would happen if someone simply read more articles or made more updates. The authors describe superforecasters as “partly discovered and partly created,” reflecting the combination of ability, practice, and working conditions.
How does superforecasting work?
A forecaster defines the outcome, estimates its probability using comparable cases, and adjusts for the evidence about the current case. Breaking the question into smaller parts helps identify which assumptions drive the estimate. The Good Judgment Project's top performers also used finer probability distinctions, as Mellers and colleagues documented.
Defining a forecast question and its resolution criteria
A forecast needs an outcome, a deadline, and a way to establish what happened. “The company will launch soon” leaves all three open. “The company will begin shipping its new phone to US retail customers by September 30” gives you a question you can research. An accompanying rule could name the company's announcement that US retail shipments have begun as the resolution source and specify that preorders do not count.
A forecast that the phone will begin US retail shipments by September 30 gives “launch soon” an outcome and a deadline. The resolution rule names the company's announcement that shipments have begun as the source and excludes preorders.
A traded contract found through Polymarket and Kalshi market search must resolve on shipments to match the phone-shipping forecast. Its resolution criteria establish whether an announcement, a preorder campaign, or actual shipments count as the outcome.
Inside view vs outside view
The outside view starts with how often an outcome occurred in comparable cases, called its base rate. The inside view considers what distinguishes the particular case you are forecasting.
For the phone-shipping question, suppose 24 of 40 comparable products began shipping to US retail customers by their announced deadlines. That hypothetical record gives a starting estimate of 60%.
The 40 hypothetical products include 24 that began US retail shipments on time and 16 that were delayed or canceled. Keeping both groups in the count gives a base rate of 60% for meeting the shipping deadline.
When choosing a reference class for this forecast, include delayed and canceled products as well as those that shipped on time. Products already through final testing face different remaining obstacles from products announced before development began, so the comparison group should also match the stage of development.
A supplier problem may lower the estimate. A completed milestone may raise it, provided the comparison group did not already share that milestone.
When building a comparison group from closed and settled markets in the screener, count outcomes using comparable resolution rules. For a phone-shipping forecast, a contract about actual shipments addresses the outcome you need, while one about a product announcement measures a different event.
Breaking a forecast into conditional steps
Decomposition means estimating the parts that determine the outcome. For the hypothetical phone, suppose final testing must finish in time for US retail shipments to begin by September 30. You estimate an 80% chance that testing finishes in time and a 75% chance that shipments begin by that date if testing finishes in time. Multiplying 0.80 by 0.75 gives a 60% probability of meeting the shipping deadline.
The second estimate already assumes testing is complete. It can account for the remaining certification and distribution work, including problems caused by the testing process. If the chance of finishing testing in time falls to 40%, while the conditional shipping estimate remains 75%, the overall probability falls to 30%.
In this hypothetical forecast, the phone ships by September 30 in 75% of cases where testing finishes in time and cannot meet the deadline otherwise. Reducing the chance of timely testing from 80% to 40% halves the overall shipping probability from 60% to 30%.
The base-rate estimate and the decomposed estimate examine the same question from different directions. Comparing them can expose a missing assumption. If the historical record suggests 60% but your breakdown gives 90%, you need to explain what makes this phone's prospects so much better than the comparable products.
Adjusting probabilities for different thresholds
For a price forecast, a higher threshold has an equal or lower probability of being exceeded when the deadline and other terms stay the same. In a Good Judgment Project test, with the euro at $1.37, superforecasters' estimates averaged 73% for it exceeding $1.38 before a specified deadline and 48% for it exceeding $1.40. The comparison group averaged 51% and 42%. Both groups assigned a lower probability to the higher threshold, but the gap was 25 percentage points for superforecasters and 9 for the controls. Only the superforecasters' difference was statistically significant.
Superforecasters' average estimates for the euro exceeding $1.38 and $1.40 by the same deadline were 73% and 48%, a gap of 25 percentage points. The comparison group distinguished the thresholds less sharply, with estimates of 51% and 42%, only 9 points apart.
In an arbitrage comparison of Bitcoin contracts, a higher threshold priced above a lower one for the same date would imply probabilities in the wrong order if the terms match. Each contract's settlement rules determine whether reaching the higher threshold necessarily satisfies both.
Choosing a precise probability
Superforecasters used an average of 57 distinct probability values across their forecasts, compared with 29 and 30 in the comparison groups. Rounding their estimates to the nearest ten percentage points reduced accuracy in the 2015 study. Their finer distinctions carried information.
A forecast of 65% can still acknowledge uncertainty in its assumptions. For example, your notes might say that reasonable supplier-delay estimates produce probabilities between 60% and 70%, with 65% as your best estimate. That records both a probability to score and the uncertainty to investigate.
How do superforecasters update probabilities?
Superforecasters revisited their estimates more often than other tournament participants. Even before selection, they averaged 2.77 forecasts per question, compared with about 1.5 in the comparison groups. The study identified updating frequency as the strongest behavioral predictor of accuracy. The size of a revision should reflect how strongly the new evidence favors an outcome.
A Bayesian update asks how much more likely the evidence is under one outcome than another. Suppose your starting probability that a phone begins shipping to US retail customers by September 30 is 60%. In a hypothetical historical record, a favorable testing report appeared in 80% of cases where shipments began by the deadline and 20% where they did not. Imagine 100 comparable cases:
| Eventual outcome | Cases before seeing the report | Cases with a favorable report |
|---|---|---|
| US retail shipments began by the deadline | 60 | 48 |
| Shipping deadline missed | 40 | 8 |
| Total | 100 | 56 |
Among the 56 cases with a favorable report, 48 met the shipping deadline. Your updated probability is 48 divided by 56, or about 86%. The report appears four times as often when the deadline is met as when it is missed.
Keeping only cases with a favorable testing report leaves 48 that met the shipping deadline and 8 that missed it. That changes the estimate from 60% across all 100 hypothetical cases to about 86% among the 56 with the report.
The difficult part is estimating those reporting rates. If the company publishes reassuring reports regardless of its progress, the report deserves less weight. Evidence equally common before success and failure leaves the probability unchanged. Writing down why you revised the estimate preserves that judgment for review.
Several articles repeating one report provide a single underlying observation. Following the links in Bluesky posts behind a discussion trend can reveal that a journalist's summary and posts quoting it all depend on the same testing report. Posts by the same author count separately toward discussion volume. Before updating again, check what each source adds to the evidence already in your estimate.
Posts about a phone's testing peak at 380 on Thursday in this hypothetical week. The three leading linked sites carry the report, a summary, and a thread quoting that summary, all based on one testing report.
Time can change a forecast without a headline. A week with no progress on testing leaves fewer days for certification and distribution before the shipping deadline. The remaining work can become harder to finish even if no new obstacle appears.
What are the 10 commandments of superforecasting?
The ten commandments for aspiring superforecasters are Tetlock and Gardner's advice for choosing questions, estimating probabilities, and learning from outcomes. Good Judgment reproduces the book's list. In paraphrase:
- Use triage to choose questions where research can improve the estimate.
- Divide difficult questions into smaller parts whose assumptions you can examine.
- Balance inside and outside views by comparing similar cases with the particulars of this one.
- Avoid under- and overreacting by adjusting probabilities to the strength of the evidence.
- Examine clashing causal forces that push toward different outcomes.
- Express degrees of doubt numerically, using distinctions your reasoning supports.
- State a probability while checking for excessive confidence or excessive caution.
- Review the reasoning behind successes and failures using what was knowable at the time.
- Help teammates explain their arguments and challenge yours constructively.
- Develop these habits through repeated practice with feedback about accuracy.
The book calls the tenth practice the “error-balancing bicycle.” Its additional eleventh commandment cautions against treating the advice as binding rules for every case. The evidence on forecasting training comes from short probability-reasoning lessons given to tournament volunteers.
A cyclist carries research notes while keeping the bicycle balanced. Tetlock and Gardner's “error-balancing bicycle” compares that continual adjustment with learning from forecasts and their outcomes.
Does forecasting training improve accuracy?
In the Good Judgment Project's randomized experiments, volunteers who received brief forecasting lessons produced more accurate forecasts. Across four tournament years, trained participants had 6% to 12% lower Brier scores than untrained controls in the reported yearly results. A Brier score measures probability error, so a reduction means more accurate forecasts. Chang and colleagues analyzed participants who answered more than 30 questions per year.
The probability-reasoning module was designed to take less than an hour and covered comparison classes, updating, scoring, and common reasoning errors. The studies measured the effect of those lessons together. For scale, lowering a hypothetical Brier score from 0.20 to 0.18 is a 10% reduction in forecasting error.
Whether the better scores show an improvement in underlying forecasting ability is disputed. Hauenstein and colleagues reanalyzed the first two years using a model that accounted for question selection and response timing. The estimated effects of training and teamwork became smaller, disappeared, or sometimes reversed. Their argument concerns how much of the score improvement came from choosing different questions or forecasting at different times.
Barbara Mellers argues that choosing questions and deciding when to submit forecasts are themselves parts of forecasting skill. Her proposed follow-up experiment would have everyone answer the same questions within the same window. That design would test whether trained participants forecast more accurately with those choices held fixed.
The two hypothetical forecasters choose different questions and submission times under the tournament rules. Barbara Mellers' proposed follow-up would give trained and untrained participants the same questions and a shared window to answer them, so the comparison would depend less on those choices.
Do forecasting teams beat individuals?
People working in teams made more accurate forecasts than independent participants in the original Good Judgment Project experiments. In the first year, teams also outperformed people who could see a summary of other participants' estimates but could not discuss them. Sharing judgments helped, and discussion added a further advantage in that experiment.
The elite teams were particularly active. They shared more news, asked more questions, and replied to teammates more often than the comparison group of strong forecasters on regular teams. Their estimates also converged more as questions developed. Mellers and colleagues observed these behaviors alongside better accuracy.
For a central-bank decision, one teammate might focus on inflation releases while another examines employment data and a third reads officials' speeches. Discussion can expose different interpretations of the same evidence. Each person can then revise a probability while retaining the reasons for any remaining disagreement.
A later reanalysis disputed whether teamwork improved underlying ability once question selection and submission timing were taken into account. Mellers treats those choices as part of forecasting skill. The training and teamwork debate concerns that distinction, while the original score advantage remains an observed result.
When teammates draw on different evidence, combining their forecasts brings together more information than any one member has. Their average probability can therefore be too cautious, which is one reason researchers use extremizing, an adjustment away from 50%. Baron and colleagues also found groups whose forecasts needed the opposite adjustment because they were already too confident.
In this hypothetical team, forecasters following different evidence put the chance of a rate cut at 58%, 71%, and 66%, for an average of 65%. Extremizing moves that average farther from 50%, to 78% in this example.
What is a good superforecaster Brier score?
In one Good Judgment Project analysis, superforecasters averaged 0.25 in the first week and 0.07 in the final week of questions open for at least a month. Trained regular teams averaged 0.40 and 0.16 in the same periods. These scores used the 0-to-2 convention, where lower is better. The tournament paper's Table 1 reports the comparison for the second year:
| When forecasts were made | Superforecasters | Trained regular teams |
|---|---|---|
| First week | 0.25 | 0.40 |
| Middle two weeks | 0.19 | 0.28 |
| Final week | 0.07 | 0.16 |
On the binary 0-to-1 scale, ForecastBench's launch evaluation reported 0.096 for the median forecasts of its 39 superforecasters on a 200-question comparison set. That is the score of a combined forecast across people. The tournament figures above summarize individual forecasts at different stages of a question.
For binary outcomes, calculating the Brier score from errors for both Yes and No gives twice the score obtained from the Yes probability alone. That is why the 0-to-2 convention gives 0.20 where the 0-to-1 convention gives 0.10.
A useful benchmark also matches the questions and the time remaining before resolution. The table shows how much the same groups' scores can improve as an answer approaches. Good Judgment Open averages active forecasts across days, so an early estimate continues to affect the result even if the final estimate is accurate. A final-only score measures a different part of that record.
Superforecasters also showed better calibration than the comparison groups in the 2015 study. Their stated confidence agreed more closely with how often outcomes occurred.
Did amateur forecasters beat CIA analysts?
A Good Judgment Project forecasting system scored 0.15, compared with 0.23 for the US intelligence community's prediction market, on an error measure where lower is better. The project's volunteers supplied the forecasts for the winning system. The intelligence-community market combined predictions from cleared employees and contractors across several agencies.
Goldstein and colleagues' report compared 139 unclassified questions posted to both systems between August 2013 and May 2014. Its mean daily scores used the 0-to-2 convention:
| Forecasting system | Average Brier score |
|---|---|
| Good Judgment Project's best aggregation method | 0.15 |
| Good Judgment Project's prediction market | 0.21 |
| Intelligence community's prediction market | 0.23 |
| Unweighted average of Good Judgment Project survey forecasts | 0.32 |
The two prediction markets were statistically comparable. The best project method weighted recent forecasts by participants' track records and adjusted the resulting probabilities. Researchers selected it after comparing roughly 20 methods, although several performed almost as well.
Across 139 Good Judgment Project questions, a simple average of survey forecasts had an average daily Brier score of 0.32 on the 0-to-2 scale. The best method used selected forecasters' latest estimates, weighted them, and moved the combined probability away from 50%, producing a lower score of 0.15.
The comparison concerned collective forecasts of public questions, on which access to classified information did not necessarily give analysts an advantage.
Do superforecasters beat prediction markets?
The Good Judgment Project's fourth-season study found that forecasters' track records mattered more for forecast accuracy than whether they traded or submitted probabilities. Across 136 questions, small groups of superforecasters produced about 21% lower Brier scores, meaning smaller forecast errors, than larger groups of other forecasters. Their advantage held in both markets and team polls, with no statistically significant accuracy difference between the two formats.
Small groups of superforecasters made smaller forecast errors than larger groups of other participants in both formats. Across 136 questions from the Good Judgment Project's fourth season, their Brier scores were 20% lower in prediction markets and 23% lower in team polls.
Polls on forecasting platforms combine submitted probabilities statistically, while markets combine trading decisions through prices. The tournament's markets used play money. In the fourth season, elite participants chose their format, so the comparison between markets and polls was not randomized.
An earlier study of 261 events excluded elite forecasters and found that markets beat simple poll averages. Team polls beat markets once researchers gave greater weight to recent forecasts and participants with stronger track records, then adjusted the combined probabilities using past results.
In Superforecasting, Tetlock and Gardner reported that superteams beat the tournament's prediction markets by 15% to 30%, as quoted in Rufus Pollock's chapter notes. Tetlock acknowledged that liquidity could be limited because participants used play money and there were too few traders.
In its own comparison with Polymarket, Good Judgment reported an average daily Brier score of 0.135 for its professional forecasters, compared with 0.159 for Polymarket. It used 1,756 paired daily forecasts of raise, hold, or cut decisions across 25 meetings of the Federal Reserve, European Central Bank, Bank of England, and Bank of Japan. Coverage varied by meeting, so the Polymarket comparison includes different amounts of forecast history for each decision.
How does superforecasting help prediction market traders?
Superforecasting gives a Polymarket or Kalshi trader a probability estimate to compare with the price available to trade. Accurate forecasts can still lose money when the purchase price is too high. For a standard event contract paying $1 if Yes occurs and $0 otherwise, the difference between probability and price determines expected profit before costs.
Suppose you estimate a 70% chance of a rate cut and can buy a Yes contract at a 66-cent ask. If Yes occurs, you gain 34 cents. If No occurs, you lose 66 cents. Weighting those outcomes gives an expected profit of 4 cents per contract: 70% of 34 cents minus 30% of 66 cents.
A hypothetical 1-cent fee reduces the trade's expected value to 3 cents per contract. If the true probability is 60%, the same 66-cent purchase has an expected loss of 6 cents before fees.
Buying a $1 Yes contract at 66 cents yields an expected 4-cent profit at a 70% chance of a rate cut, or 3 cents after a hypothetical 1-cent fee. If the true chance is 60%, the same purchase has an expected 6-cent loss before fees.
For a longer hypothetical record, suppose you forecast 100 events at 70%, and exactly 70 occur. You buy one Yes contract for each event and hold all of them to settlement.
| Price paid for each contract | Total cost | Total returned at settlement | Profit before fees |
|---|---|---|---|
| 66 cents | $66 | $70 | $4 |
| 75 cents | $75 | $70 | −$5 |
Both rows have the same calibration, meaning the frequency of outcomes matches the stated probability. The different prices turn the same forecast record into a profit or a loss.
An entry-price history also leaves your original estimate unknown. Buying at 66 cents is consistent with believing 67%, 70%, or 85%. Recording your probability and reasons separately lets you score the judgment that led to the trade. A score calculated from purchase prices measures the accuracy of those prices.
Buying Yes for 66 cents leaves the same trade record whether you estimated 67%, 70%, or 85%. The price alone does not identify your estimate, so you need a separate record to score your own forecast.
The Polymarket leaderboard ranks accounts by profit and loss. Position size and execution affect that ranking alongside judgment. In an interview about their 2026 working paper, Gómez-Cram and colleagues describe persistent trading skill in about 3% of accounts in their Polymarket sample. They identified those accounts by comparing trading profits with simulated results from chance. That tests trading performance directly, while a record of stated probabilities permits a separate test of forecasting accuracy.
Can AI beat superforecasters?
Superforecasters outperformed the strongest tested AI setup in ForecastBench's launch evaluation, reported in 2025. An earlier experiment found no statistically significant accuracy difference between the combined forecasts of a group of language models and those of a general human forecasting crowd. ForecastBench's Brier scores below use the binary 0-to-1 scale, where lower means smaller errors:
| Evaluation | AI setup | Human comparison | Result |
|---|---|---|---|
| ForecastBench launch evaluation, 200-question human comparison set | Claude 3.5 Sonnet, given crowd forecasts on market questions. | Median forecasts from 39 superforecasters who worked individually and in groups. | Binary Brier scores were 0.096 for the superforecaster median and 0.122 for the model. The expert group's advantage was statistically significant. |
| Wisdom of the Silicon Crowd, reported in 2024 | Combined forecasts from 12 language models. | A crowd of 925 human forecasters. | Across 31 binary questions over three months, the combined AI and human forecasts were statistically indistinguishable. |
ForecastBench's leaderboard compares AI and human forecasters on an ongoing basis. It adds questions about future events and scores submitted forecasts as outcomes become known, so later systems face questions whose answers were unknown when they forecast them.
How do you become a superforecaster?
The route to Good Judgment's professional designation is a sustained record of accurate forecasts on Good Judgment Open. In its explanation of how to become a superforecaster, the company asks candidates to forecast at least 100 questions and emphasizes average accuracy per closed question, comment quality, and collegiality. Selected candidates may enter a three-month probation period.
Where can you practice forecasting?
Good Judgment Open and Metaculus let you forecast real events and compare your record with other participants. Quantified Intuitions provides shorter exercises with quicker feedback. A spreadsheet or notebook can preserve your reasoning alongside any of these tools.
| Tool | What you can practice |
|---|---|
| Good Judgment Open | Forecasting real events, revising probabilities, discussing reasons, and comparing scored results with the crowd. |
| Metaculus | Making forecasts on specified questions, explaining your reasoning, and reviewing individual and community performance. |
| Quantified Intuitions | Calibration exercises and Pastcasting, which uses past events to give quicker feedback. |
| A spreadsheet or notebook | Keeping your own questions, probabilities, sources, and updates together. |
For a first practice set, choose questions you can revisit and that will resolve soon enough to provide feedback. Mix subjects you follow with some outside your strongest area. Record a starting estimate before reading a community forecast so you can later see what your own research contributed.
How to keep a forecasting journal
A forecasting journal preserves what you believed and why. Each update gets a new entry, leaving the original estimate available for review. Useful records connect the probability to the evidence available at the time:
| Record | Why it helps later |
|---|---|
| Exact question, deadline, and resolution source | Establishes what you were trying to predict. |
| Probability and time of entry | Shows your confidence with the information available then. |
| Starting base rate and main assumptions | Keeps the reasons for your starting estimate. |
| Evidence that would change your view | Lets you check whether you responded when it arrived. |
| Each update with its reason | Shows why you changed your mind. |
| Outcome, score, and a review of the reasoning | Helps separate a mistaken assumption from an unlikely outcome. |
After resolution, compare both numbers and assumptions. Suppose you gave a phone an 80% chance of beginning US retail shipments by September 30 because you trusted one supplier. Shipments begin by the deadline, but only after that supplier fails and a replacement is found. Your forecast gets a good score even though the supplier assumption was wrong.
An 80% forecast of phone shipments beginning by September 30 scores well when the deadline is met, with a squared error of 0.04. The supplier the forecaster trusted failed, so a good score still leaves a mistaken assumption to examine.
Finding repeated forecasting errors
Review batches of forecasts as well as individual cases. If repeated misses came from assuming that a supplier's announcement meant production was ready, the next forecast needs a better test of production readiness. If events you gave a 70% chance occur only half the time across a substantial record, review what led you to that confidence.
Assigning a 70% chance to each of 50 events implies about 35 occurrences if the forecasts are calibrated. Only 25 occur in this hypothetical record, a gap of 10 that suggests the forecaster was overconfident.
Base Rates
Comparable cases provide a starting point for a probability estimate.
Brier Score
Probability forecasts can be scored against what actually happened.
Calibration
A forecasting record shows how confidence compares with observed outcomes.
Wisdom of Crowds
Combining forecasts requires attention to shared information and errors.