Forecasting tournament step by step guide
How to run a forecasting tournament
A forecasting tournament is a competition in which participants submit predictions for future events, and the accuracy of their predictions is evaluated against the actual outcomes of those events.
1. Decide on the amount and type of questions
A typical forecasting tournament consists of 15 to 100 questions. The fewer the questions, the greater the influence of luck on the outcome. Conversely, a larger number of questions leads to more accurate insights into who the top forecasters in your group are. However, increasing the number of questions also raises the time commitment for both organizers and participants.
A well-established approach is to post four questions per week over a six-week period. This structure helps participants stay focused on the current questions without feeling overwhelmed. Based on past experience, participants spend an average of one hour answering each question.
At the start of the tournament, it’s important to decide on the type of questions you’ll be asking your participants. If you choose to include only yes-no questions, you can take advantage of Brier scoring, which is built into the Confido app. This will allow you to automatically view participants’ scores. Learn more about the automatic scoring feature.
However, if you prefer to use a different scoring method for yes-no questions or to include numeric range questions, you’ll need to export the results as a CSV file (Confido supports CSV export) and compute the scores yourself according to whatever formula you choose.
2. Decide about the level of collaboration among participants
Will your forecasting question be answered by individuals? Or by teams? Or entire departments?
You can also decide if you want participants to provide comments on their forecasts and nurture a collaborative instead of competitive environment. Best comments can be also rewarded after the tournament.
3. Think about questions you want to ask
You can not only find out who the best forecasters in your org are but you can also find out better data about the future of your industry. Therefore you can think about what kind of topics you would like to cover in your tournament.
Of course, you can also forecast just for fun – sports, fashion, celebrities etc.
4. Develop quantified questions about the future
Once you have your Confido workspace, create a room “xxx forecasting tournament.” Then add all your questions. Every forecasting question should be specific, clear and resolvable. You can also read more on what a good forecasting question looks like on our website.
For every forecasting question you can set a schedule: when it opens (will be seen by participants), when it closes (when forecasting on this question ends) and when it will be scored.
5. Inform your participants
Once you know you will organize a tournament, it is a good idea to organize a workshop or online call where you explain what a forecasting tournament is and how it works. You can also announce an award for the best forecasters. Be ready to answer lots of questions.
6. Help your participants to prepare for the tournament
It is important for participants to realize before the tournament that we can express our uncertainty as a percentage.
Sharing a calibration exercise can serve us for this. You can either prepare your own calibration training in Confido or use some of the existing ones.
7. Start the tournament and share questions with your participants
When inviting participants to your Confido room, make sure all your participants have the role of “forecasters.”
You can also share your questions with moderators, who can update and change questions, and observers, who can just see what is happening.
8. Talk with participants
After publishing the questions, tournament participants often have a lot of additional questions and need to make sure of the wording of the questions. You need to set aside time for them, especially on days when new questions are posted.
9. Resolve your questions
Once you know what the correct answer was, resolve the question. Participants usually prefer to answer questions that will resolve quite soon. Quick feedback will also help them improve as the tournament progresses.
In almost every tournament, at least one question cannot be resolved because of unclear definition and disputes.
10. Evaluate the accuracy of participants’ forecasts
The evaluation of accuracy can be done in a variety of ways, but typically involves comparing the participants’ predictions to the actual outcomes using some sort of scoring system.
The Brier score is the most common scoring system in forecasting tournaments. It measures the average squared difference between the predicted probability of an event and the actual outcome. Other systems, such as log-likelihood or precision-recall, use different metrics. The Confido app includes built-in Brier scoring, allowing you to automatically generate scores for all participants. You can set it up with a few clicks.
If you choose a different scoring method, you can still use the Confido app by exporting a CSV file of all predictions and manually scoring participants’ responses based on your preferred system.
11. Announce winners
Organize a party or meeting where people can share how they were thinking about the questions and where you recognize the best forecasters.
You can reward not only those with the best scores, but also those who had the best comments in the tournament and helped others with their insights.
Get all the benefits from running a forecasting tournament
Improved decision-making
Forecasting tournaments help participants hone their prediction skills, which can lead to better decision-making in various strategic domains within the organization.
Collective intelligence
By aggregating forecasts from diverse participants, forecasting tournaments can produce more accurate predictions than any single individual, leveraging the wisdom of the crowd.
Incentivizing learning
Tournaments often offer rewards or recognition for top performers, which can motivate participants to learn and improve their forecasting abilities.
Identifying talent
Tournaments can help organizations identify individuals with exceptional forecasting abilities who may be valuable assets in strategic planning or other decision-making processes.
Encouraging collaboration
Tournaments can foster a spirit of collaboration among participants, as they often involve sharing insights, discussing methodologies, and learning from one another.
Generating valuable data
The forecasts and accompanying data generated by tournaments can be useful for the organization itself.
“We used Confido for our internal forecasting tournament, the platform allowed us to effortlessly organize a competition with participants from different government departments, and they provided excellent technical support when necessary.”