When you need to forecast a complex issue, most people’s first inclination is to ask a domain expert. But is this really a good idea?
Evidence from forecasting tournaments in the geopolitical domain has shown that expertise is poorly correlated with forecasting accuracy.
Expertise can blind as much as it can enlighten, and what you know matters less than how you think. But does this hold true in other domains? And if forecasting is a skill independent of domain expertise, who should decision makers rely on to make high-stakes forecasts?
The experiment: a forecasting contest on infectious disease
Hypermind had the opportunity to investigate this question in the context of a large-scale pre-pandemic experiment to forecast infectious-disease outbreaks. Over the course of 15 months, from January 2019 to march 2020, we pitted 310 forecasters against one another to predict outcomes on 19 different diseases such as Ebola, cholera, influenza, dengue, and eventually Covid-19.
Some forecasters were public-health experts recruited by the Johns Hopkins Center for Health Security, while some were experienced forecasters from the Hypermind prediction market. Each participant’s individual accuracy was tracked using the Brier score, a professional forecasting accuracy metric.
Results: most experts do no better than chance
Individually, the average domain experts performed at the level of chance (the proverbial dart throwing monkey). And so did the average skilled forecaster, reflecting just how difficult these forecasting problems were.
But, amazingly, the crowd as a whole, given a few mathematical tweaks, beat all individuals.
In other words, the smartest forecaster on disease prediction is not a person, but a crowd. It is also notable that the crowd of skilled forecasters with no particular domain expertise were just as accurate as the crowd of public-health experts.
Of course, the best forecasters were both domain experts and reliable generalist forecasters, meaning that expertise still matters, but so does forecasting skill.
Who should you trust ? Three takeaways when it comes to making predictions
- Do not trust individual experts, only trust crowds of experts
- Crowds of skilled forecasters are just as accurate as experts
- Leverage whichever is most easily available and affordable
Watch Emile Servan-Schreiber’s in depth presentation of the research at the 2021 DIMACS workshop on forecasting :
We predict what you can’t. Hire our crowd of elite forecasters to predict what matters to your organization
Just curious about how Hypermind can help you predict uncertain events ? Shoot us an email at email@example.com