Base Rates
Base rates help you build a forecast from comparable cases and judge how much new evidence should change its probability.
Forecasting an event involves deciding how much the details of one case should change what you would expect from similar cases. For a company promising a product launch, that means looking at how often comparable companies met their deadlines. Their record gives you an initial probability to weigh against the company's progress.
A useful comparison counts delays and cancellations alongside successful launches. It also compares companies at similar stages, because a finished product awaiting shipment faces different uncertainties from one still in development. Base rates make that comparison explicit, so you can judge whether evidence about the current case supports a higher or lower probability.
What is a base rate in statistics?
A base rate is how often an outcome occurs in a defined group. If 40 of 100 comparable cases had the outcome, its observed base rate is 40%. A probability estimated from such recorded frequencies is an empirical probability. Using base rates in forecasting means starting from that frequency, then considering what makes the particular case more or less likely to follow the pattern.
The group is called a reference class, such as basketball players listed with an injury or officeholders seeking another term. How you define that group changes what its percentage tells you. “Incumbents win 90% of the time” leaves out which office, which elections, and whether people who retired were counted.
A prior probability is the estimate you hold before considering a particular piece of evidence. It can come from a base rate, a model, or an earlier round of research. In Bayesian updating, incorporating the evidence gives a revised estimate called the posterior probability, which can become the prior for the next update. If a historical rate gives you a 40% prior and evidence about the case raises it to 70%, that 70% is your posterior.
In the grid of 100 hypothetical comparable cases, 40 had the outcome, giving a 40% base rate. That rate supplies the prior for a new case, and evidence about the case raises the estimate to a 70% posterior.
What are base rates in psychology?
In psychology, a base rate is the prevalence of a trait, condition, or behavior in a population. When a condition has a low base rate, even a test that rarely misclassifies people can produce more false positives than true positives because so many more people are in the group without the condition. Meehl and Rosen's analysis of psychological tests explains how this imbalance affects the interpretation of test results.
What is the base rate fallacy?
The base rate fallacy is giving too little weight to how common an outcome is when interpreting evidence about a particular case. It is also called base rate neglect. A positive test, a convincing description, or an apparently reliable witness can receive so much weight that the underlying frequency barely affects the answer.
How much weight people give the base rate varies with the task and how the information is presented. In a review of the research, Jonathan Koehler found that people generally used base rates to some degree, with greater use when the information was learned through experience or expressed as frequencies.
A base rate fallacy example
Suppose a disease affects 1 in 1,000 people, and a test gives false positives to 5% of people without it. In Manrai and colleagues' 2014 study, 61 physicians, residents, and medical students were asked for the chance that someone with a positive test had the disease, with no other information about symptoms.
Twenty-seven participants answered 95%, a result that follows from subtracting the false-positive rate from 100% without accounting for the disease's rarity. Fourteen gave the correct answer of about 2%, calculated by assuming that the test detected everyone with the disease.
With those assumptions, testing 1,000 people would produce about one true positive and 50 false positives. The false positives come from the much larger group of 999 people without the disease, even though the test misclassifies only 5% of them. About one of the 51 positive results would belong to someone with the disease, giving a probability of roughly 2%.
A hypothetical test detects the one person with the disease among 1,000 people and gives false positives to 5% of the other 999. That leaves about 51 positive results, of which only one, roughly 2%, belongs to someone with the disease.
The taxicab problem
In Maya Bar-Hillel's study of Kahneman and Tversky's taxicab problem, a witness says a cab involved in an accident was green. The city has 15% green cabs and 85% blue cabs, and the witness correctly identifies each color 80% of the time. Of the study's 52 participants, 36% put the chance that the cab was green at 80%, matching the witness's accuracy.
For every 100 cabs in those proportions, the witness would correctly call 12 of the 15 green cabs green. They would also mistakenly call 17 of the 85 blue cabs green. Of the 29 cabs identified as green, 12 really are green, giving 12 divided by 29, or about 41%.
In a city with 15 green cabs and 85 blue cabs, a witness who identifies each color correctly 80% of the time would call 12 green cabs and 17 blue cabs green. Of the 29 cabs identified as green, 12 would actually be green, or about 41%.
Applying the same reasoning to company news
The medical and cab examples use Bayes' theorem to combine a starting probability with the evidence's reliability. For company news, the same distinction matters when a reliable source reports acquisition talks. The source may accurately establish that talks happened while few comparable talks end in a completed deal. Estimating the chance of an acquisition means including the talks that fell through in the base rate, alongside those that succeeded.
The two company folders are open on the meeting table, and the agreement is still unsigned. The acquisition remains uncertain while talks are under way.
How does the base rate fallacy differ from the representativeness heuristic?
The base rate fallacy occurs when you give too little weight to a relevant frequency. The representativeness heuristic means judging probability by resemblance to a category or familiar pattern. It can cause base rate neglect when a description seems more persuasive than the composition of the group.
In Kahneman and Tversky's engineer-and-lawyer experiment, participants read short descriptions of people drawn from a group of engineers and lawyers. They received identical descriptions but different information about the group's composition, either 30 engineers and 70 lawyers or 70 engineers and 30 lawyers.
Participants judged a person who sounded typical of an engineer largely on that resemblance, and their estimates changed little when the group's proportions changed. Without a description, their median estimates matched the stated proportions.
The descriptions stayed the same in both versions of Kahneman and Tversky's experiment. Engineers accounted for 30 of 100 people in one group and 70 of 100 in the other, giving starting probabilities of 30% and 70% for being an engineer.
A young company might resemble a famous success because it has a charismatic founder and enthusiastic customers. Estimating how often such companies succeed means counting other companies that shared those features, including the ones that struggled or closed.
The fallacy can also occur with numerical evidence, without a stereotype or resemblance judgment. In the cab problem, people substituted a witness's 80% color-identification accuracy for the probability that a cab was green, overlooking the city's proportions of green and blue cabs.
What is reference class forecasting?
Reference class forecasting estimates an outcome from the record of comparable cases. You choose a relevant group, measure what happened, and use that record to estimate the new case. The approach grew from Kahneman and Tversky's work on the outside view. Bent Flyvbjerg developed the procedure adopted for UK transport projects in 2004 and describes that adoption as the method's first practical use.
For a Yes-or-No question, the method produces a probability. For a budget or deadline, it can produce an amount or date that a chosen share of past cases stayed within. The UK procedure used 172 completed road projects, half of which had cost overruns of no more than 15%, while 80% had overruns of no more than 32%. Applying those proportions to a hypothetical £100 million estimate gave a budget of £115 million to cover half the historical overruns, or £132 million to cover 80%. The chosen budget depended on how much risk of an overrun the planners would accept.
A hypothetical £100 million road estimate becomes a £115 million budget when adjusted to cover half the overruns in 172 completed UK road projects. Covering 80% of those overruns raises the budget to £132 million.
How do past projects reveal the planning fallacy?
The planning fallacy is underestimating the time or resources a task will take. Comparable projects show how long similar work actually took, including delays a team's own schedule may leave out.
Kahneman's curriculum-writing team estimated that a finished draft would take another 18 to 30 months. In his account with Dan Lovallo, an expert on the team then recalled that roughly 40% of comparable projects had been abandoned. The completed ones had taken seven to ten more years from a similar stage, and he saw no advantage that would help this team finish sooner. Its project eventually took about eight more years.
Kahneman's curriculum team estimated another 18 to 30 months but needed about 8 more years, within the 7 to 10 years taken by comparable projects that finished. Roughly 40% of comparable projects were abandoned.
The team's estimate used the inside view, which builds a forecast from the case's plans and progress. The outside view starts with outcomes among comparable cases, including those abandoned before completion. The Superforecasting example applies both views to a shipping deadline.
How do you choose a reference class?
Choose past cases that faced the same question, at a similar stage, under conditions relevant to the outcome. Defining those requirements before calculating the rate makes it harder to keep changing the group until it produces a number you prefer.
Suppose you want to estimate whether an injured basketball player will appear in a particular game. “Players who came back from injury” already excludes people who did not return. “Players listed as questionable two days before a game” gives you a group whose outcomes can still go either way.
The most useful restrictions depend on what would change the chance of playing:
| Choice | Why it matters |
|---|---|
| Same outcome | Appearing for any minutes differs from starting or playing a full game. |
| Same observation time | A status report two days before tipoff contains less information than the final lineup. |
| Similar reason for absence | An injury, illness, and planned rest can follow different processes. |
| Comparable reporting practices | The same status label can become less informative if teams change how they use it. |
Matching the league and injury category may improve the comparison, though it reduces the number of observations. Requiring the player's exact age, team, month, and opponent as well could leave only a few cases. Before adding those restrictions, consider whether each difference would affect the chance of playing enough to justify excluding the other observations.
The records can come from different sources. For basketball, pair dated injury reports, such as the NBA's participation-status reports, with game box scores showing who played. Brookings' congressional tables record reelection attempts and results. The Base Rate Book groups historical company growth by starting size and forecast horizon. When compiling your own comparison, record each case as it stood at the chosen time and tally its eventual outcome.
Past contracts found by searching market titles and resolution rules can also supply cases, but contracts naming the same token may ask different questions. A token becoming transferable, appearing on an exchange, and reaching a price threshold belong in separate comparison groups. Use the settlement rules to match each contract to the outcome you are counting.
The reference class problem
The reference class problem is that one case belongs to many groups, each with a different historical record. A company can be compared with other companies in its industry, businesses of its size, or firms growing at a similar rate.
When reasonable groups disagree, keep their counts visible. Suppose 18 of 30 companies in an industry met their original launch dates, including 4 of 10 companies similar in size to the one you are forecasting. Requiring a size match changes the rate from 60% to 40%. That gap might reflect a difference related to company size or variation in a small sample. If size affects access to manufacturing capacity, for example, it gives you a reason to favor the smaller group. When several classes remain plausible, Jacob Steinhardt's forecasting notes suggest using simple classes or averaging across reasonable ways of forming them.
Both rows contain the same 30 hypothetical companies, but the second emphasizes the 10 similar in size to the company being forecast. Counting that smaller group changes the on-time launch rate from 18 of 30, or 60%, to 4 of 10, or 40%.
Overlapping groups also share evidence. Three reference classes containing many of the same cases do not provide three independent confirmations.
How do rule changes affect a base rate?
Historical cases become less useful when a rule change alters the process that produced their outcomes. Separating cases from before and after the change lets you compare the rates under each set of conditions.
Suppose a new approval requirement adds a testing stage before a product can launch. For a launch forecast under that requirement, start with products that faced it, then compare their on-time rate with the older group. If the first wave faced a temporary processing backlog, keep those cases separate from later launches so that the extra delay does not carry into a forecast made after the backlog has cleared.
A prototype waits beneath a testing probe beside an unchecked form, with approval still needed before it can launch.
For a product with no close precedent, familiar manufacturing and approval steps can still have their own records of success and failure. Those records can inform estimates of the steps involved, even when there is no useful rate for the whole launch.
How does survivorship bias distort a base rate?
Survivorship bias changes a base rate by leaving out cases that failed, disappeared, or stopped being observed. An archive of products that reached the market, for example, leaves out cancellations when estimating the chance of an on-time launch.
Suppose 100 announced products had deadlines that have now passed. Sixty launched on time, 20 launched late, and 20 were canceled, giving an on-time rate of 60%. An archive retaining only the 80 products that launched would report 60 out of 80, or 75%. Removing cancellations raises the reported success rate even though the number of on-time launches stays at 60.
Both bars contain the same 60 on-time launches from a hypothetical set of 100 announced products. Dropping the 20 cancellations leaves 80 products in the archive and raises the reported on-time rate from 60% to 75%.
In investment research, Elton, Gruber, and Blake's study of mutual funds found that funds which disappear tend to have performed poorly. Keeping only the survivors changes the performance record of the group an investor originally could have chosen.
Survivorship bias is a form of selection bias, which occurs when the way cases are chosen skews the result. A launch comparison drawn from closed and settled Polymarket and Kalshi listings covers only questions somebody chose to list. It can omit less prominent companies and routine announcements that never received markets.
Other choices about which records to count can distort the rate:
- Cases that have not finished. A company still within its promised launch window has an unknown outcome, so counting it as a failure lowers the rate prematurely. Keeping only early successes raises it instead. Comparing groups whose deadlines have all passed avoids both distortions.
- Cases selected using future information. A group of companies that eventually obtained funding leaves out the possibility of failing to obtain it. That group answers a question about companies after funding, even if the spreadsheet starts their histories earlier.
- Several records of one outcome. Ten news reports about a launch provide one launch outcome, and contracts on two venues can settle from that same event. Counting each record as a separate case gives the launch extra weight in the rate.
How reliable is a base rate from a small sample?
A small sample gives an unstable estimate because each case carries a large share of the result. Three successes in five cases and 60 in 100 both produce a 60% observed rate. Changing one outcome moves the first estimate by 20 percentage points and the second by one point.
A confidence interval expresses this uncertainty as a range calculated from the observed count. For independent cases with a common probability, the Wilson method in NIST's handbook gives these approximate results:
| Observed outcomes | Observed rate | 95% Wilson confidence interval |
|---|---|---|
| 3 successes in 5 cases | 60% | 23% to 88% |
| 60 successes in 100 cases | 60% | 50% to 69% |
The wider interval shows how little five cases establish about the group's underlying rate. Even if that rate were known to be exactly 60%, the next case would still have a 40% chance of failure.
The dots mark the same 60% success rate for 3 of 5 cases and 60 of 100. The bands express the uncertainty around each estimate, with 95% Wilson intervals of about 23% to 88% for five cases and a narrower 50% to 69% for 100.
The sample size needed depends on the distinction you want to make. Separating a rate near 20% from one near 80% takes less evidence than distinguishing 1% from 3%. For a contract that settles at $1 or zero, those last two probabilities imply average settlement values of one cent and three cents. A two-cent purchase falls between them, so a difference that is hard to measure can change the expected return.
Shared causes further limit what a count tells you. Several injured players on the same team may face the same coaching decision, and several products may depend on one supplier. Their outcomes provide less independent information than equally many unrelated cases. The intervals above also cannot repair missing failures or a poorly chosen comparison group.
Estimating probability when an outcome has never occurred
With no observed occurrences, the rule of succession gives a nonzero estimate for the next case, while the rule of three gives an upper confidence bound for the group's rate. Both use the number of opportunities in which the outcome could have happened.
Before UMBC beat Virginia in March 2018, No. 16 seeds had lost all 135 matchups against No. 1 seeds in the men's NCAA tournament since 1985. UMBC's student newspaper reported the record after the upset. Treating those 135 games as independent trials with a shared upset probability gives two different calculations:
- The rule of succession estimates the next outcome's probability as 1 / (n + 2), where n is the number of opportunities with zero occurrences. With 135 games, that is 1 divided by 137, or about 0.73%. It initially gives equal weight to every possible probability between zero and one. Steinhardt's notes explain why a prior informed by the subject may fit better.
- The rule of three gives an approximate upper 95% confidence bound of 3 / n for the underlying rate. Here that is 3 divided by 135, or about 2.2%. Hanley and Lippman-Hand derive the approximation. The 2.2% is an upper bound from that procedure, not the rule's forecast for the next game.
Before UMBC's 2018 upset, No. 16 seeds had lost all 135 men's NCAA tournament games against No. 1 seeds. On the 0% to 3% scale, the dot marks the rule of succession's estimate of about 0.73% for the next game, and the band ends at the rule of three's approximate 95% upper confidence bound of 2.2% for the underlying rate.
For a particular game, evidence about the two teams can justify adjusting the estimate from their seed history. Applying these rules to a crypto protocol is harder because the number of opportunities for failure may be unknown. Fifty days without a failure can include very few serious attempts to break it.
How do you update a base rate with new evidence?
When historical records contain the new evidence, count how often the outcome occurred among the cases with that evidence. That frequency is the updated estimate. You can calculate the same change by converting the starting probability to odds, multiplying by the evidence's likelihood ratio, and converting the result back to a probability.
A Bayesian updating example
Suppose you assemble 100 past cases of basketball players listed as questionable two days before a game and define the outcome as playing any minutes. In this hypothetical group, 40 played and 60 did not, giving a prior probability of 40%.
You now learn that the player completed a full practice the day before the game. To use that report, you look at the past cases:
| Practice report | Played | Did not play | Total |
|---|---|---|---|
| Completed full practice | 30 | 10 | 40 |
| Did not complete full practice | 10 | 50 | 60 |
| Total | 40 | 60 | 100 |
Among players who completed full practice, 30 of 40 played, so including the report raises the estimate from a prior probability of 40% to a posterior probability of 75%.
Of the 100 basketball players in this hypothetical group, 40 played after being listed as questionable. Limiting the comparison to those who completed full practice leaves 30 who played and 10 who did not, raising the estimated chance from 40% to 75%.
Updating with a likelihood ratio
A likelihood ratio measures how much more common the evidence is when the outcome occurs than when it does not. In the basketball example, full practice was reported in 30 of the 40 cases where the player appeared, compared with 10 of the 60 cases where they did not. Dividing 30/40 by 10/60 gives a likelihood ratio of 4.5. In this comparison of two outcomes, the ratio is also called a Bayes factor.
The odds form of the Bayesian updating formula is:
The prior probability of 40% gives odds of 40 to 60, or 2 to 3. Multiplying those odds by 4.5 gives 3 to 1. Three chances in favor out of four total gives the posterior probability of 75%, matching the direct count in the table. A likelihood ratio below one would lower the probability. A ratio of one would leave it unchanged because the evidence is equally common under both outcomes.
For a basketball player with a hypothetical 40% chance of appearing, the odds are 2 to 3. A likelihood ratio of 4.5 multiplies the playing side from 2 to 9 while the other side stays at 3, giving a 75% posterior.
“Update your priors” means revising an earlier estimate as evidence arrives. When you lack a record of how often a report precedes each outcome, choosing its likelihood ratio involves judgment. You can see how much that judgment changes the forecast by trying several plausible ratios. If the practice report's ratio were 2, the same 40% prior would become about 57%, while a ratio of 6 would give 80%.
The three calculations start from the same 40% prior and use different likelihood ratios for the evidence. A ratio of 2 gives a posterior of about 57%, while 4.5 gives 75% and 6 gives 80%.
Counting the same news twice
Counting the same evidence twice moves a forecast farther than that evidence supports. In the basketball example, the 75% estimate already includes a report of full practice. Starting from 75% and applying the report's likelihood ratio of 4.5 again would produce about 93%, with no additional information.
In this hypothetical basketball forecast, the full-practice report raises the chance of playing from 40% to 75%. The dashed extension reaches about 93% by applying the report's likelihood ratio of 4.5 again, even though there is no new evidence.
A coach's positive comment may largely repeat what the practice report already established. Multiplying likelihood ratios measured separately across the original group assumes conditional independence. Knowing about full practice must leave the chance of a positive comment unchanged among players who played, and separately among those who missed the game.
For the second update, compare the playing rates of players who had all completed full practice, separating those with a positive comment from those without one. Practice status is the same in both groups, so this measures the comment's additional value without assuming independence.
Several posts quoting the same coach still rest on one observation, even if they appear on different sites. Conversely, two reports on one site can contain separate observations, such as full practice one day and a new injury during warm-ups the next.
The three hypothetical posts about a basketball coach's comment share one observation despite linking to three different sites. The two posts linking to one reporter's site carry two observations because they report different events, a full practice and a later injury during warm-ups.
How often do incumbents win reelection?
From 1946 through 2024, US House incumbents seeking reelection won 92.6% of their attempts and senators won 81.0%, calculated from Brookings' congressional tables. Sitting presidents who contested a general election from 1948 through 2024 won 8 of 12 attempts, or about 66.7%, counted from National Archives election results.
These rates count successful attempts divided by all attempts within each stated group:
| Office and period | Attempts won | Who is counted |
|---|---|---|
| House, 1946–2024 | 14,677 of 15,850, or 92.6% | Incumbents seeking reelection, including primary defeats, in Brookings Table 2-7. |
| Senate, 1946–2024 | 920 of 1,136, or 81.0% | Incumbents seeking reelection, including primary defeats, in Brookings Table 2-8. |
| President, 1948–2024 | 8 of 12, or 66.7% | Sitting presidents who reached the general election. |
The offices face different electorates, and the much larger House sample would dominate a combined rate. The presidential group includes officeholders who first took over during a term, but excludes those who withdrew before the general election and former presidents seeking a return to office. Since this group begins with sitting presidents already on the general-election ballot, using its rate at an earlier stage would leave out the chance that a president never reaches that ballot.
US House incumbents made 15,850 reelection attempts from 1946 through 2024, compared with 1,136 for the Senate over the same period and 12 presidential general-election attempts from 1948 through 2024. The combined win rate is 91.8%, close to the House's 92.6%, because the House contributes most of the attempts.
The House rate also changed considerably by election year, from a low of 79.3% in 1948 to a high of 98.3% in 1988 and 1998 in the Brookings record. National swings can move many House races together, so the pooled 92.6% combines elections with very different outcomes for incumbents.
For 2024 alone, 95.9% of House incumbents seeking reelection won, while those reelected were 85.7% of the full House membership, which also counts retiring members. The broad House rate mixes safe and competitive seats. For a forecast about a close race, narrow the group using what was known about competitiveness before the election. Selecting races afterward by their final margins would use information unavailable at forecast time.
What are base rates in investing?
Base rates in investing show how often businesses or investments have achieved particular results. They help put a company's growth forecast, profit margin, or expected recovery beside the record of comparable firms.
Michael Mauboussin, Dan Callahan, and Darius Majd's The Base Rate Book analyzes the largest 1,000 global companies by market capitalization from 1950 through 2015. Across three-year periods, compound annual sales growth after inflation reached at least 15% in 9,904 of 53,266 observations, or about 18.6%.
Suppose a company with $5 billion in annual sales forecasts compound annual growth of at least 15% after inflation over the next three years. In the book's $4.5 billion to $7 billion starting-sales group, about 14.7% of observations reached that target, below the 18.6% across all sizes. Signed customer commitments might support raising that probability for this company if comparable commitments have reliably turned into revenue.
In The Base Rate Book's record of the largest 1,000 global companies from 1950 through 2015, about 18.6% of three-year observations reached at least 15% compound annual sales growth after inflation. Restricting the group to companies starting with $4.5 billion to $7 billion in sales lowers that frequency to about 14.7%.
Regression to the mean occurs when unusually high or low observations tend to be followed by less extreme ones. For example, a spectacular quarter driven partly by temporary orders can be followed by growth closer to the company's usual level once those orders end.
The book includes companies that later disappeared, but each growth sample keeps only companies that survived the measured period. Because of this remaining survivorship bias, an estimate made at the start must account for the chance of disappearance before the period ends.
How often do prediction markets resolve No?
In a 2026 preprint using data through December 31, 2025, 61.9% of resolved Kalshi markets and 59.6% of the Polymarket markets in three studied categories resolved No. The overall share depends on which questions and alternatives were listed.
In its sample of 210,608 Kalshi markets, the No share among resolved contracts was 58.7% for sports, 59.3% for crypto, 59.8% for politics, and 76.0% for weather. Its Polymarket comparison covers 124,881 resolved markets across three categories, with No shares of 64.7% for sports, 55.7% for crypto, and 69.3% for politics.
Each category bar crosses 50%, meaning more markets resolved No than Yes in the 2026 preprint's data through December 31, 2025. The No share ranges from 55.7% for Polymarket crypto to 76.0% for Kalshi weather.
The weather figures show why the contract wording matters. Asking whether a temperature will exceed a high threshold produces more No outcomes than asking about a lower threshold. In a multi-outcome event with one winner, each losing alternative adds another No to the count. Listing more alternatives changes the overall frequency even when the underlying event stays the same.
When the central bank leaves rates unchanged in this hypothetical example, only the Hold contract resolves Yes. Listing cut and hike sizes separately increases the No share from 2 of 3 contracts, about 66.7%, to 4 of 5, or 80%.
Whether always buying No is profitable depends on the prices paid and costs. To test that idea against history, match outcomes to contracts available at similar prices and similar times before settlement. A resolution tally leaves out those purchase opportunities.
For an individual contract, a relevant base rate can start the comparison. In the hypothetical basketball example, 40% of players listed as questionable played in their scheduled game, rising to 75% among those who later completed full practice. If a contract on that same playing outcome were available at 70 cents and settled at $1 or zero, the 75% estimate would imply an average settlement value of 75 cents, five cents above cost before fees.
Comparing the report's publication time with the contract's price history in the Market Screener can show whether the price changed after the news arrived. If you start from a market price that already reflects the report, applying the report's multiplier again would count the same news twice.
A hypothetical Yes contract on a basketball player appearing in Friday's game rises from about 40 cents to 70 cents after Thursday's full-practice report. The later price already includes the market's response to that report.
How do superforecasters use base rates?
Superforecasters use the frequency of comparable events as a starting probability, then adjust for the particular case and new evidence. Tetlock and Gardner's guidance recommends balancing that outside view with the details of the case, including when an event initially seems unusual.
In Mellers and colleagues' tournament study, the best forecasters initially assigned about 20% to a lethal confrontation involving government forces in the South or East China Sea during the last four months of 2011. That matched the historical frequency for four-month periods. They lowered their estimates as time passed without a qualifying incident and less time remained for one to occur. A lethal incident occurred in December.
For the last four months of 2011, the starting estimate uses a 20% historical rate for a lethal incident in the South or East China Sea. Assuming the same chance each month and no incident so far, the calculated chance in the time left falls to about 15.4% on October 1, 10.6% on November 1, and 5.4% on December 1.
In Chang and colleagues' study, self-reported use of comparison classes was associated with better Brier scores. Randomly assigned forecasting training also improved accuracy, but combined several reasoning principles, so its effect cannot be attributed to base-rate instruction alone.
To test your own adjustments, preserve both the original base rate and the revised probability in a forecasting journal. Record the comparison group, the count, and the reason for the change before the outcome is known.
The Brier score measures the squared error of a probability forecast. Use matching forecast times for the original and adjusted estimates, then average their scores across the same resolved questions. The estimates with the lower average score performed better on that record. A calibration check can also show whether outcomes you assigned about 80% probability happened about that often.
Superforecasting
How forecasters define questions, weigh evidence, and revise probabilities.
Brier Score
How to compare probability forecasts with the outcomes that follow.
Calibration
How stated probabilities compare with observed frequencies across forecasts.
Prices and Probabilities
How contract prices, implied probabilities, and trading costs relate.