How do we know this? Before and during the Covid-19 pandemic, Hypermind teamed up with the Johns Hopkins Center for Health Security in a large-scale research study aiming to test the epidemiological forecasting skills of several hundred public health experts and other medical professionals.
The results were astonishing!
It is easy to make fun of people trying to predict the future, but in fact all of us do it all the time. Predictions are essential to our ability to navigate a world where uncertainty is everywhere. The decisions you make, in your life, for your business, or for your country, cannot be smart unless they are informed by reliable predictions. That is why human brains are wired to make predictions all the time.
Cognitive scientists, such as Yann Le Cun, the artificial-intelligence expert who co-invented deep-learning algorithms, often says that prediction is the essence of intelligence itself.
So if every brain is a forecasting machine, what do you think happens when many brains try to make predictions together? That’s right. They become a super forecasting machine. That is the promise of so-called “crowd forecasting”: using the wisdom of crowds to predict the future.
Crowd forecasting is usually done in one of two ways.
The two methods yield similar results in terms of prediction accuracy, but prediction polls are easier for most people to participate in. So that’s the method we will use for the prediction contest to which you are invited to participate at the end of this course.
To feel the awesome power of crowd forecasting, you need to see it in action. So consider this experiment in disease prediction that was run in 2019 and 2020 by the Center for Health Security at Johns Hopkins University in collaboration with Hypermind. The goal was to predict infectious disease outbreaks around the world. So we recruited hundreds of public-health experts, other medical professionals, and also some champion forecasters from the Hypermind prediction market. In all, 70% of the participants had some professional medical background.
Over 15 months, before after the COVID-19 outbreak, we asked them to predict the severity of outbreaks for 19 infectious diseases. We asked 61 questions that had a total of 217 possible answers. So they had to give probabilities for 217 predictions, only 61 of which would come true.
For example, at the beginning of the COVID-19 pandemic, we asked “How many WHO member states will report more than 1000 confirmed cases of COVID-19 before April 2, 2020?”, and there were 4 possible answers: Less than 15, 16 to 30, 31 to 45, or more than 45.
We then measured the average prediction error of each forecaster over all the question, and this is what it looks like. In this graph every dot is one forecaster. The graph plots the prediction error, so the higher a dot is, the worst that forecaster is.
These higher dots are the least accurate forecasters, while these lower points are the most accurate.
You can see that most people tend to cluster around this level of performance, which is the level of “blind chance”. That’s the same error a monkey would achieve by throwing darts at the answers.
Except our participants were not monkeys, but for the most part, medical professionals… Yet only very few of them could make substantially better forecasts than a monkey, and many did much worse.
In fact only 6 individuals managed to beat the crowd, so the basic collective forecast beat 99% of the individuals.
Furthermore, when you enhance the crowd aggregation with a few simple statistical transformations, it outperforms all individuals: the crowd is a better forecaster than the best forecaster in the crowd.
"The crowd is a better forecaster than the best individual forecaster in the crowd."
Another way to check the accuracy of the crowd forecasts is to ask: “how many outcomes forecasted with probability p did occur?”
To test this, we gathered all the outcomes that were given, say, 30% chance of happening, and then we looked at how many of those outcomes did in fact occur. We found that about 30% of them did occur.
We looked at all the outcomes that were forecasted to have 50% chance of happening, and we found that half of them did occur.
We looked at all the outcomes given 80% chance of happening, and found that 80% did happen. You get the idea.
There was an almost perfect alignment between the predictions and reality at every level of probability. Here’s our calibration data on over 500 000 forecasts on geopolitical, economic and business prediction questions, and for the 61 questions we asked health experts in our experiment with Johns Hopkins.
It’s not just us. This same amazing calibration pattern has been observed in various domains where crowd forecasting has been applied.
For example when Google asked 1,500 employees to forecast business-relevant outcomes like new product launches or sales forecasts, or when Hypermind analyzed half a million predictions made in its prediction market on 400 geopolitical and economic questions over 5 years.
What that means is not that crowd forecasts can tell you with certainty what will or won’t happen in the future. Only divine beings could do that, perhaps.
But it reveals the uncanny ability of collective intelligence to discern the true chances that something will or won’t happen. It’s not quite divine, but it’s powerful nonetheless.
So if predictions are indeed essential to our ability to navigate a world where uncertainty is everywhere, crowd forecasting is a key resource to help make large organizations, businesses and governments as smart as they can be.