Public health decision makers can supplement traditional disease surveillance by harnessing crowd wisdom. In a joint study, Hypermind and the Johns Hopkins Center for Health Security demonstrate how the wisdom of crowds provides accurate and timely forecasts on infectious disease outbreaks such as malaria, dengue, and Covid-19.
When you need to forecast a complex issue, most people’s first inclination is to ask a domain expert. But that may not always be a good idea. Evidence from geopolitical forecasting tournaments shows that expertise is poorly correlated with forecasting accuracy.
Expertise can blind as much as it can enlighten, and what you know matters less than how you think.
But does this hold true in other domains such as public health ? And if forecasting is a skill independent of domain expertise, who should decision makers rely on to make high-stakes forecasts, individual experts or crowds ?
Can crowd forecasting make accurate and timely predictions on a variety of diseases ?
The goal of the study was to develop an evidence base for the use of crowd-sourced forecasting as a way to confidently provide information to decision makers in order to supplement traditional surveillance efforts and improve response to infectious disease emergencies.
In a joint study, Hypermind and Johns Hopkins set up a large-scale pre-pandemic experiment to forecast infectious-disease outbreaks (read our in depth peer-reviewed publication).
The problem with traditional disease surveillance: late and incomplete data
Traditional disease surveillance is critical for a sound public health response. Yet, according to senior scholar Dr. Tara Sell, and Dr. Crystal Watson, project leads and senior scholars at Johns Hopkins, traditional methods present limitations in how well they can predict future outcomes:
"Real-time and predictive outbreak information is often limited and can make it difficult for practitioners to respond effectively before an outbreak has reached its peak.
In many cases, data collected through traditional surveillance methods often lags days or weeks behind an unfolding epidemic due to delays in collecting, reporting and analyzing data.
Moreover, surveillance data may be abundant and timely for some epidemics or regions of the world, and poor and time-lagged for others, making it difficult to respond effectively across hazards and geographies."
Over the course of 15 months, from January 2019 to march 2020, we pitted 310 forecasters against one another to predict outcomes on 19 different diseases such as Ebola, cholera, influenza, dengue, and eventually Covid-19.
Most forecasters were public-health experts recruited by the Johns Hopkins Center for Health Security, while some were experienced forecasters from the Hypermind prediction market. Each participant’s individual accuracy was tracked using the Brier score, a professional forecasting accuracy metric.
70% of forecasters were medical experts selected by Johns Hopkins, 30% were champion forecasters chosen by Hypermind.
Key finding 1: most experts can't predict infectious disease any better than chance
Forecasting a complex problem like infectious disease is hard, very hard.
Most participants cluster around the level of error of “blind chance”: it is what you would expect from the proverbial dart-throwing monkey picking answers at random.
Although most participants were medical professionals, only very few of them produced substantially better forecasts than our theoretical dart throwing chimp, and many did much worse.
A note on measuring prediction accuracy: we measured the average prediction error of each forecaster over all questions, using a standard method for computing the error of a probability forecast (Brier, 1950).
Key finding 2: a simple average of all forecasts outperforms 99% of individual experts
Simply averaging individual forecasts produced aggregate crowd forecasts that outperformed all but 6 participants, or 99% of them, visible below the pink line.
To understand the uncanny power of averaging judgments, check out our explainer of the wisdom of crowds.
The key idea: when combined in the right way, errors in judgment can cancel each other out while information can add up. This phenomenon is what’s behind Galton’s famous ox weighing competition, or the jellybeans experiment, but it also works when lawyers try to estimate damages from cases, or when predicting the likelihood of future events like our prediction market.
Key finding 3: a weighted average of all forecasts outperforms every single individual experts
When enhanced by a few intuitive statistical transformations, the crowd forecasts (below in red) beat even the best forecaster in the crowd.
The outcome probabilities forecasted by the crowd were also well “calibrated”, in the sense that they were closely correlated with the actual outcome frequencies in the real world : about 20% of all outcomes forecasted with 20% probability did occur, while 80% of all outcomes forecasted with probability 80% did occur, and so on at every level of probability.
Key finding 4: skilled forecasters performed just as well as domain experts
In other words, the smartest forecaster on disease prediction is not a person, but a crowd. It is also notable that the crowd of skilled forecasters with no particular domain expertise were just as accurate as the crowd of public-health experts.
The smartest forecaster on disease prediction is not a person, but a crowd.
The outcome probabilities forecasted by the crowd were also well “calibrated”, in the sense that they were closely correlated with the actual outcome frequencies in the real world : about 20% of all outcomes forecasted with 20% probability did occur, while 80% of all outcomes forecasted with probability 80% did occur, and so on at every level of probability.
Key finding 5: when the crowd says "it's X% likely", it's X% likely
Of course, some the best forecasters were both domain experts and reliable generalist forecasters, meaning that expertise still matters, but forecasting skill matters just as much.
How we optimized our crowd's forecasts to outperform every single expert:
- Weighting: Individual forecasts were weighted, with greater weight given to forecasters who updated their predictions frequently and who had a better past record of accuracy. Initially everyone is weighted equally, but individual differences in behaviour and performance progressively emerge as forecasting questions are settled throughout the contest.
- Culling: The pool of individual forecasts was then reduced so that only the 30% most recent forecasts – likely the most informed – were retained for aggregation, while the others were ignored.
- Averaging: The weighted forecasts were then averaged.
- Extremizing: Finally, an extremization step was used to sharpen the resulting forecast and compensate for collective under-confidence.
Crowd forecasting as a supplement to policy makers
According to Dr. Tara Sell, Senior Scholar at Johns Hopkins, crowd forecasting constitutes a quick win to improve current methods, by allowing decision makers to spot trends earlier, which means better response and more lives saved, and by quantifying uncertainty around key questions.
"Crowd forecasting efforts in public health may be a useful addition to traditional disease forecasting, modeling, and other data sources in decision making for public health events.
Such crowd-sourced forecasting can help to predict disease trends and allow public health and policymakers time to prepare and respond to an unfolding outbreak.
These efforts will never replace traditional surveillance methods, since surveillance data is the gold standard and is also needed to verify prediction platform outcomes, but they can supplement traditional methods.
By providing rapid synthesis of the knowledge and expectations of experts and informed amateurs, crowd-sourced forecasting can help inform decision-making surrounding implementation of disease mitigation strategies and predict where disease may cause problems in the near future.
The crowd accurately predicted explosive growth and spread of [Covid-19] but forecasts in some instances also provided indications of uncertainty, likely due to poor disease reporting, testing, and surveillance early in the outbreak."
Who should you trust ? Three takeaways when looking for insights about the future:
- Do not trust individual experts, only trust crowds of experts
- Crowds of skilled forecasters are just as accurate as experts
- Leverage whichever is most easily available and affordable (experts of skilled forecasters)
We help you predict what others can't. Hire our crowd of champion forecasters to tame uncertainty
Curious about how Hypermind can help you predict uncertain events ? Take a look at our Prescience platform, our panel of vetted forecasters or write to us at contact@hypermind.com
