Judgmental forecasting is a method used to predict future events by answering specific questions. Philip Tetlock made it famous in his book Superforecasting in 2015. If you prefer text over video, scroll below and read what is judgemental forecasting about and how it differs from regular futurology.
First, forecasters answer specific questions with exact, measurable predictions. They also share how uncertain they are about their answers.
Second, various forecasts from many forecasters are combined into a single prediction, which is usually more accurate than any one forecast. This idea is called the "wisdom of the crowds." It works because each prediction has some information others might miss, and the group's combined knowledge cancels out individual biases.
Third, forecasters' success is measured by checking the accuracy of their past predictions when the future event happens. These scores help show how good a forecaster is at predicting the future. It can also be used to weigh predictions from better forecasters more heavily. This feedback helps forecasters improve over time.
Judgmental forecasting is mainly used for complicated problems involving many sectors or situations where there isn't enough data to make predictions based on exact models.
What kind of forecasting questions can we ask?
Most questions ask the forecaster to choose between two mutually exclusive outcomes of future events (so-called binary questions). Some questions ask the forecaster to estimate the date of some future event, or to estimate the quantity of something at a particular date (so called numeric range questions).
They generally take the form "Will (event) X happen by (date) Y?" or "When will (event) X occur?" or "What will the value or quantity of X be by (date) Y?"
Quantifying uncertainty
When forecasters answer a question such as “Will there be active warfare between the United States and China by the end of 2026?”, their answer usually does not take the form of a plain “Yes” or “No.” This is because an integral part of the forecasting practice is quantifying uncertainty. If the future is genuinely unknown, no one can be 100% sure that something will or will not happen. Instead, forecasting questions are usually answered by expressing your confidence in your answer in the form of a probability. E.g. “I am 90% confident the answer is Yes.”
Calibration
It can be a bit philosophically complicated to precisely define what such a probability statement formally means but you can think about it for example this way. If you answer 100 unrelated questions with 90% confidence, you should be correct on approximately 90 of them.
The skill of giving correct confidence estimates is called calibration.
Imagine the above scenario where you answer 100 questions with 90% confident answers. If you are correct on about 90 of them, you are well calibrated. If you are correct on much less (e.g. 80), you are overconfident (your predictions are less reliable than you claim), if on much more (e.g. 99), you are underconfident.
Calibration can be trained, as will be discussed later.
Other than YES/NO questions
This gets a bit more complicated in questions that can have more than two possible answers.
For example, if the question has a small finite number of possible answers (e.g. which of these states will first develop some technology) you can simply assign a probability value to each answer (and these probabilities must sum up to 100%).
Even trickier are numerical range questions, where the number of possible answers may be large, even infinite (e.g. when considering decimal numbers). In this case, you can for example predict the probabilities that the answer falls to some predefined number ranges (e.g. 1-100, 100-1000).
Another option is to give a confidence interval. E.g. when you are asked for a 90% confidence interval, you should answer a range of numbers that you are 90% confident (in the sense suggested above) the answer will lie in.
A more general option is to answer by giving a probability distribution – a mathematical construct that can more generally describe how likely you consider different answers to be.
There are many kinds of probability distributions, for example the uniform distribution (e.g. which considers all the numbers in a given range equally likely and numbers outside of it 0% likely). A more realistic one is the normal distribution, which is described by its mean (the average or expected value) and variance (a number that describes how “spread out” the distribution is, i.e., how likely you are to see numbers far from the mean). A complete discussion of probability distributions is outside the scope of this article.
Conditional questions
A special kind of question are conditional questions, e.g. questions of the form “If (event) X happens (does not happen), will (event) Y follow?” A conditional binary question can be resolved either positively (both X and Y happen), negatively (X happens but Y does not) or ambiguously (X does not happen). If the question is resolved ambiguously, forecasts for it are considered neither correct nor wrong and cannot be scored.
