How crowd forecasting can help anticipate infectious disease outbreaks

Collective intelligence pushes the limits of forecasting. Crowd forecasting shines when too many variables are involved for a single expert to handle, or when there is too little data to feed an artificial intelligence.

How do we know this? Before and during the Covid-19 pandemic, Hypermind teamed up with the Johns Hopkins Center for Health Security in a large-scale research study aiming to test the epidemiological forecasting skills of several hundred public health experts and other medical professionals.

The results were astonishing:

  • Most individual experts forecast as accurately as a dart throwing monkey.
  • But the crowd’s forecasts outperform even the best individual forecasters in the crowd.
  • Crowd forecasts can accurately predict outcome probabilities.

Table of Contents

Crowd forecasting infectious disease with Johns Hopkins

In a joint study, Hypermind and Johns Hopkins set up a large-scale pre-pandemic experiment to forecast infectious-disease outbreaks (read our in depth peer-reviewed publication).

The goal of the study was to develop an evidence base for the use of crowd-sourced forecasting as a way to confidently provide information to decision makers in order to supplement traditional surveillance efforts and improve response to infectious disease emergencies.

Over the course of 15 months, from January 2019 to march 2020, we pitted 562 forecasters against one another to predict outcomes on 19 different diseases such as Ebola, cholera, influenza, dengue, and eventually Covid-19.

19

Infectious diseases

15

months

61

forecasting questions

562

forecasters

70% of forecasters had a professional medical background

No Data Found

Example forecasting question:
“How many WHO member states will report more than 1000 confirmed cases of COVID-19 before April 2, 2020?”

Less than 15

16 to 30

31 to 45

more than 45

Key finding 1: most experts can't predict infectious disease any better than chance

We measured the average prediction error for each forecaster over all the contest’s questions, and this is what it looks like. 

In this graph every dot is one forecaster. The graph plots the prediction error, so the higher a dot is, the worst that forecaster is. These higher dots are the least accurate forecasters, while these lower points are the most accurate.

Forecasting a complex problem like infectious disease is hard, very hard. 

Most participants cluster around the level of error of “blind chance”: it’s the accuracy you would expect from the proverbial dart-throwing monkey picking answers at random. 

Although most participants were medical professionals, only very few of them produced substantially better forecasts than our theoretical dart throwing chimp, and many did much worse.

The enhanced crowd forecast outperformed all experts during our infectious disease prediction contest.

Key finding 2: a simple average of all forecasts outperforms 99% of individual experts

Simply averaging individual forecasts produces an aggregate crowd forecast (in pink) that outperformed all but 6 participants (99% of forecasters).

A simple averaging of individual forecasts produced crowd forecasts that outperform 99% of individuals

Key finding 3: a weighted average of all forecasts outperforms every single individual experts

When enhanced by a few intuitive statistical transformations, the crowd forecasts (below in red) outperformed even the best forecaster in the crowd.

"The crowd is a better forecaster than the best individual forecaster in the crowd."

Key finding 4: skilled forecasters performed just as well as domain experts

In other words, the smartest forecaster on disease prediction is not a person, but a crowd. It is also notable that the crowd of skilled forecasters with no particular domain expertise were just as accurate as the crowd of public-health experts.

How we optimized our crowd's forecasts to outperform every single expert:

Weighting

Give more weight to forecasters who updated their predictions frequently and who have a better track record of accuracy. Start off by weighting everyone equally, adjust weighting as data builds up.

Culling

Reduce the pool of individual forecasts so that only the 30% most recent forecasts – likely the most informed – were retained for aggregation.

Averaging

Average these weighted forecasts into an aggregate forecast . Averaging works to reduce error because forecasters all make different mistakes that can cancel each other out.

Extremizing

Finally, extremize forecasts to sharpen the crowd forecast and compensate for collective under-confidence.

Key finding 5: when the crowd says "it's X% likely", it's X% likely

The outcome probabilities forecasted by the crowd were also well “calibrated”, in the sense that they were closely correlated with the actual outcome frequencies in the real world : about 20% of all outcomes forecasted with 20% probability did occur, while 80% of all outcomes forecasted with probability 80% did occur, and so on at every level of probability.

Of course, some the best forecasters were both domain experts and reliable generalist forecasters, meaning that expertise still matters, but forecasting skill matters just as much.

The calibration chart shows how often a predictions lign up with reality.

Why use crowd forecasting ?

traditional methods are only as good as the available (structured) data

"Real-time and predictive outbreak information is often limited and can make it difficult for practitioners to respond effectively before an outbreak has reached its peak.

In many cases, data collected through traditional surveillance methods often lags days or weeks behind an unfolding epidemic due to delays in collecting, reporting and analyzing data.

Moreover, surveillance data may be abundant and timely for some epidemics or regions of the world, and poor and time-lagged for others, making it difficult to respond effectively across hazards and geographies."

Crowd forecasting gives decision makers a synthesis of current knowledge and helps them assess uncertainty

By providing rapid synthesis of the knowledge and expectations of experts and informed amateurs, crowd-sourced forecasting can help inform decision-making surrounding implementation of disease mitigation strategies and predict where disease may cause problems in the near future.

The crowd accurately predicted explosive growth and spread of [Covid-19] but forecasts in some instances also provided indications of uncertainty, likely due to poor disease reporting, testing, and surveillance early in the outbreak."

Who should you trust ? Three takeaways when looking for insights about the future:

1

Do not trust individual experts, only trust crowds of experts

2

Crowds of skilled forecasters are just as accurate as experts

3

Leverage whichever is most easily available and affordable: experts or skilled forecasters, or both !

Prediction is the essence of intelligence

It is easy to make fun of people trying to predict the future, but in fact all of us do it all the time. Predictions are essential to our ability to navigate a world where uncertainty is everywhere. The decisions you make, in your life, for your business, or for your country, cannot be smart unless they are informed by reliable predictions. That is why human brains are wired to make predictions all the time.

Cognitive scientists, such as Yann Le Cun, the artificial-intelligence expert who co-invented deep-learning algorithms, often says that prediction is the essence of intelligence itself.

So if every brain is a forecasting machine, what happens when many brains try to make predictions together? They become a super forecasting machine. This is the promise of so-called “crowd forecasting”: using the wisdom of crowds to predict the future.

Prediction markets vs prediction polls

Crowd forecasting usually takes place on a prediction market or a prediction poll, each method having its advantages and weaknesses.

The two methods yield similar results in terms of prediction accuracy, but prediction polls are easier for most people to participate in because they don’t require you to be familiar with financial markets.

1

Prediction markets

A prediction market is an online betting platform where people buy and sell predictions from each other. 

It looks and feels like a financial market, but instead of trading company stocks, participants trade predictions that end up being right or wrong. Shares of correct predictions will eventually be paid 100 points, while shares of wrong predictions will be worth none. A prediction’s “market price” measures a its probability of coming true, according to the many diverging opinions of a crowd of forecasters.

2

Prediction polls

A “prediction poll” is a contest where participants are competing to give the most accurate probabilities for future events. 

Each person shares their probability forecasts without a central marketplace. Sometimes it’s useful to show forecasters what the crowd thinks before they make their own estimate.

Then, smart algorithms consolidate and optimizes everyone’s guesswork into a reliable collective forecast.

Community of forecasters

Our international community of thousands of minds makes numerical predictions on specific issues.

Prediction market + algorithm

Our prediction markets and proprietary algorithms combine their diverging perspectives according to the science of collective intelligence.

Reliable forecasts

Anticipate strategic issues: business environment, KPI, geopolitical events, economy.