Crowd forecasting infectious disease: a new tool for public health decision makers

Forecasting infectious disease

Public health decision makers can supplement traditional disease surveillance by harnessing crowd wisdom. In a joint study, Hypermind and the Johns Hopkins Center for Health Security demonstrate how the wisdom of crowds provides accurate and timely forecasts on infectious disease outbreaks such as malaria, dengue, and Covid-19.

When you need to forecast a complex issue, most people’s first inclination is to ask a domain expert. But that may not always be a good idea. Evidence from geopolitical forecasting tournaments shows that expertise is poorly correlated with forecasting accuracy.

Expertise can blind as much as it can enlighten, and what you know matters less than how you think.

But does this hold true in other domains such as public health ?

And if forecasting is a skill independent of domain expertise, who should decision makers rely on to make high-stakes forecasts? Individual experts or crowds ?

Can crowd forecasting make accurate and timely predictions on a variety of diseases ?

https://youtu.be/wFKPJd3SqCM

The experiment: a crowd forecasting contest on infectious disease

The goal of the study was to develop an evidence base for the use of crowd-sourced forecasting as a way to confidently provide information to decision makers in order to supplement traditional surveillance efforts and improve response to infectious disease emergencies.

In a joint study, Hypermind and Johns Hopkins set up a large-scale pre-pandemic experiment to forecast infectious-disease outbreaks (read our in depth peer-reviewed publication).

The problem with traditional disease surveillance: late and incomplete data

Traditional disease surveillance is critical for a sound public health response. Yet, according to senior scholar Dr. Tara Sell, and Dr. Crystal Watson, project leads and senior scholars at Johns Hopkins, traditional methods present limitations in how well they can predict future outcomes:

Real-time and predictive outbreak information is often limited and can make it difficult for practitioners to respond effectively before an outbreak has reached its peak.

In many cases, data collected through traditional surveillance methods often lags days or weeks behind an unfolding epidemic due to delays in collecting, reporting and analyzing data.

Moreover, surveillance data may be abundant and timely for some epidemics or regions of the world, and poor and time-lagged for others, making it difficult to respond effectively across hazards and geographies.”

Over the course of 15 months, from January 2019 to march 2020, we pitted 310 forecasters against one another to predict outcomes on 19 different diseases such as Ebola, cholera, influenza, dengue, and eventually Covid-19.

Most forecasters were public-health experts recruited by the Johns Hopkins Center for Health Security, while some were experienced forecasters from the Hypermind prediction market. Each participant’s individual accuracy was tracked using the Brier score, a professional forecasting accuracy metric.

70% of forecasters were medical experts selected by Johns Hopkins, 30% were champion forecasters chosen by Hypermind

 

70% of forecasters were medical experts selected by Johns Hopkins, 30% were champion forecasters chosen by Hypermind

Key findings

Most experts do no better than chance

Individually, the average domain experts performed at the level of chance (the proverbial dart throwing monkey).

The average skilled forecaster performed just as well, reflecting just how difficult these forecasting questions were.

most-experts-perform-no-better-than-random

The crowd’s judgment outperforms all individual experts

We measured the average prediction error of each forecaster over all questions. For this we used a standard method for computing the error of a probability forecast (Brier, 1950).

In the following graph of the results, every dot represents one forecaster. The higher a dot is, the worse the forecaster’s predictions are.

Most participants tend to cluster around the same level of error, which happens to be the same as “blind chance”: it is what you would expect from the proverbial dart-throwing monkey picking answers at random. Although most participants were medical professionals, only very few of them produced substantially better forecasts than a monkey, and many did much worse.

 

In other words, the smartest forecaster on disease prediction is not a person, but a crowd. It is also notable that the crowd of skilled forecasters with no particular domain expertise were just as accurate as the crowd of public-health experts.

Of course, the best forecasters were both domain experts and reliable generalist forecasters, meaning that expertise still matters, but on equal footing with forecasting skill.

The smartest forecaster on disease prediction is not a person, but a crowd.

How we optimized our crowd’s forecasts to outperform every single expert:

As mentioned earlier, we further enhanced the collective forecast by use of a four-step aggregation algorithm that embodied some common-sense ideas:

  1. Weighting: Individual forecasts were weighted, with greater weight given to forecasters who updated their predictions frequently and who had a better past record of accuracy. Initially everyone is weighted equally, but individual differences in behavior and performance progressively emerge as forecasting questions are settled throughout the contest.
  2. Culling: The pool of individual forecasts was then reduced so that only the 30% most recent forecasts – likely the most informed – were retained for aggregation, while the others were ignored.
  3. Averaging: The weighted forecasts were then averaged.
  4. Extremizing: Finally, an extremization step was used to sharpen the resulting forecast and compensate for collective under-confidence.

Crowd forecasting as a supplement to policy makers

According to Dr. Tara Sell, Senior Scholar at Johns Hopkins, crowd forecasting constitutes a quick win to improve current methods, by allowing decision makers to spot trends earlier, which means better response and more lives saved, and by quantifying uncertainty around key questions.

Crowd forecasting efforts in public health may be a useful addition to traditional disease forecasting, modeling, and other data sources in decision making for public health events.

Such crowd-sourced forecasting can help to predict disease trends and allow public health and policymakers time to prepare and respond to an unfolding outbreak.

These efforts will never replace traditional surveillance methods, since surveillance data is the gold standard and is also needed to verify prediction platform outcomes, but they can supplement traditional methods.

By providing rapid synthesis of the knowledge and expectations of experts and informed amateurs, crowd-sourced forecasting can help inform decision-making surrounding implementation of disease mitigation strategies and predict where disease may cause problems in the near future.

The crowd accurately predicted explosive growth and spread of [Covid-19] but forecasts in some instances also provided indications of uncertainty, likely due to poor disease reporting, testing, and surveillance early in the outbreak.

Who should you trust ? Three takeaways when looking for insights about the future:

  1. Do not trust individual experts, only trust crowds of experts
  2. Crowds of skilled forecasters are just as accurate as experts
  3. Leverage whichever is most easily available and affordable (experts of skilled forecasters)

Prescience aggregator Covid-19 daily deaths France

We predict what others can’t.

Hire our crowd of champion forecasters to tame uncertainty

Curious about how Hypermind can help you predict uncertain events ? Shoot us an email at contact@hypermind.com

Table of Contents

To learn more or schedule a demo. We’ll answer any question you have.