Crowd beats top public health experts on Covid-19 forecasting contest

When you need to forecast a complex issue, most people’s first inclination is to ask a domain expert. But is this really a good idea?

Evidence from forecasting tournaments in the geopolitical domain has shown that expertise is poorly correlated with forecasting accuracy.

Expertise can blind as much as it can enlighten

Expertise can blind as much as it can enlighten, and what you know matters less than how you think. But does this hold true in other domains? And if forecasting is a skill independent of domain expertise, who should decision makers rely on to make high-stakes forecasts?

The experiment: a forecasting contest on infectious disease

Hypermind had the opportunity to investigate this question in the context of a large-scale pre-pandemic experiment to forecast infectious-disease outbreaks. Over the course of 15 months, from January 2019 to march 2020, we pitted 310 forecasters against one another to predict outcomes on 19 different diseases such as Ebola, cholera, influenza, dengue, and eventually Covid-19.

Aucun texte alternatif pour cette image

Some forecasters were public-health experts recruited by the Johns Hopkins Center for Health Security, while some were experienced forecasters from the Hypermind prediction market. Each participant’s individual accuracy was tracked using the Brier score, a professional forecasting accuracy metric.

Results: most experts do no better than chance

Individually, the average domain experts performed at the level of chance (the proverbial dart throwing monkey). And so did the average skilled forecaster, reflecting just how difficult these forecasting problems were.

But, amazingly, the crowd as a whole, given a few mathematical tweaks, beat all individuals.

The smartest forecaster on disease prediction is not a person, but a crowd.

In other words, the smartest forecaster on disease prediction is not a person, but a crowd. It is also notable that the crowd of skilled forecasters with no particular domain expertise were just as accurate as the crowd of public-health experts.

Of course, the best forecasters were both domain experts and reliable generalist forecasters, meaning that expertise still matters, but so does forecasting skill.

Who should you trust ? Three takeaways when it comes to making predictions

  1. Do not trust individual experts, only trust crowds of experts
  2. Crowds of skilled forecasters are just as accurate as experts
  3. Leverage whichever is most easily available and affordable

Watch Emile Servan-Schreiber’s in depth presentation of the research at the 2021 DIMACS workshop on forecasting :

Aucun texte alternatif pour cette image

We predict what you can’t. Hire our crowd of elite forecasters to predict what matters to your organization

Just curious about how Hypermind can help you predict uncertain events ? Shoot us an email at contact@hypermind.com

To learn more or schedule a demo. We’ll answer any question you have.