How crowd forecasting outperformed major pollsters in the 2022 French presidential elections
Hypermind’s crowd sourced forecasts for the election results outperformed pollsters on 90% of days, with about twice the accuracy of France’s three main polling agencies.
While “beating the polls“ may not be within the reach of even even the most deserving or persevering presidential candidates, the latest elections have proven it remains easier than “beating the odds,” the other favorite mantra of laggard candidates.
In other words, it is a safer bet to consult gamblers than listen to pollsters when it comes to predicting election outcomes.
We’ve now had two opportunities, both in 2017, and again in 2022, to confirm Hypermind’s crowd forecasts outperformed those of Ifop, Ipsos and OpinionWay, the main French opinion polling firms.
Rolling surveys vs. forecasting contests
Polling relies on representative samples
Our study focuses on so–called “rolling” polls. Published daily, they update one third of a representative sample of voters while eliminating the oldest data.
“While a traditional barometer can be compared to a succession of snapshots, the rolling survey allows us to recreate the film of the election race in real time” explains Ifop on its website.
During the last twelve weeks of the 2022 presidential campaign, the three combined rolling polls (Ifop, Ipsos, OpinionWay) surveyed about 100,000 voters’ preferences.
Crowd forecasting relies on aggregated bets
Instead, Hypermind relies on an entirely different approach: we consolidate the collective intelligence of our community of forecasters via an online betting platform.
This method of predicting the future originates from the crowd forecasting research programs conducted over the last decade by IARPA, the research arm of the US intelligence agencies.
Instead of polling the preferences of representative samples of voters, a contest is held to predict the ultimate
outcome of the election.
To that end, a few hundred amateur political scientists continuously update their predictions, with rewards for the most accurate among them.
The resulting collective forecast is updated daily by calculating a weighted average of the most recent 50% of
individual forecasts. In this average, more weight is given to forecasters who update their predictions frequently or with track records of accurate predictions established during previous contests.
The 2022 Macron-Le Pen duel: how did Hypermind compare with pollsters ?
2x closer
to Macron's final score
1.3 % points away versus 2.9 for pollsters (average of 3 pollsters)
better 87% of the time
Our forecasts outperformed pollsters on 128 days out of 146
Compared to the 143 rolling polls published between February 1st and April 22 on a Macron-Le Pen duel in the second round, the collective estimate of the panel of forecasters, taken a few hours before the publication of each poll, was closer to the result in 128 cases, or 90% of the times.
In detail, Hypermind’s estimate proved better in 95% of comparisons with Ifop, 91% of comparisons with OpinionWay, 77% of comparisons with Ipsos, and 87% of comparisons with the average estimate of the three firms.
The average difference between the daily estimates and Emmanuel Macron’s final score in the second round was 1.3% for Hypermind compared to 2.9% for the three rolling polls’ estimates.
Some may argue that such a comparison is flawed because the voting intentions recorded by a poll do not claim to be predictive of election outcomes.
Pollsters often feel the need to remind us that “by definition, polls are never wrong, because they are not intended to predict.“ While this is technically true, it is also disingenuous as consumers of election polls crave predictions.
This reinforces the need for a complementary approach that does not pretend to do anything other than pure forecasting, especially if it anticipates the results better.
One could also argue that Hypermind got lucky this year. But the betting crowd’s advantage has also been observed in the 2017 presidential election.
Just a fluke ? Our performance on the 2017 Four-way duel
Five years ago, the original Macron–Le Pen duel in the second round offered little suspense. Instead, it was in the first round, and particularly in the four–way duel between Emmanuel Macron, François illon, Marine Le Pen and Jean–Luc Mélenchon that the election was decided.
At the time, Ifop and OpinionWay published “victory predictions” based on the percentages of their samples that believed in the victory of each candidate. These predictions could be compared to the probabilities of victory produced by Hypermind’s prediction market, even if they were subtly different: when an pollster proclaimed “x% of the voters believe in the victory” of a candidate, the market reported rather that “the collective estimation of the bettors is that this candidate has a y% probability of winning”.
Over a period of 10 weeks preceding the 1st round, the prediction market’s daily forecast was more accurate than Ifop’s rolling in 93% of cases, and better than that of OpinionWay’s in 89% of the time.
To measure the error of such forecasts, we typically use the “Brier score”. A perfect prediction will get a score of 0 while a random prediction on 5 possible and mutually exclusive answers (Macron, Mélenchon, Fillon, Le Pen, someone else), will have an error of 0.8.
When you compare brier scores, the daily forecast of the Hypermind prediction market was on average 27% better than that of Ifop, and 19% better than OpinionWay.
Forecasting elections before polls: betting on collective intelligence
These results, on two consecutive French presidential elections, show how reliable it is to look to the collective intelligence of hundreds of amateur political scientists via online betting. But this will not surprise those who know the history of political betting.
“Scientific” opinion polls were invented by the American George Gallup only in 1936. Before then, how were the results of American elections predicted? By betting at the bookmakers on the sidewalks of Wall Street !
In the archives of the daily newspapers of the time, researchers discovered that these ancient political markets of the pre-war era were almost never wrong: in 14presidential elections between 1880 and 1932, the oddsmakers’ favorite lost only once.
Even more surprisingly, the relative odds of the candidates were already predicting margins of victory as well as polls of voting intentions have since done.
Somehow, the Wall Street presidential markets were able to quantify public opinion as accurately as polls later did. In other words, the invention of Gallup did not improve our ability to anticipate the outcome of an election. Collective intelligence already knew how.
Today, modern prediction markets, like ours, are bringing this venerable method back to life. Thanks to the Web and algorithms, performance has been further improved by eliminating bookmakers from the process.
The biggest losers of the 2022 presidential race
It is interesting to replay the film of the 2022 election from the perspective of the prediction market. It helps to establish a ranking of the biggest losers, i.e., those who have most spoiled their chances.
It is not Jean-Luc Mélenchon, who, according to the forecasters, has never had more than a 5% chance of winning.
Nor Marine Le Pen, whose chances peaked just shy of 20% just before the 1st round.
Valérie Pécresse turned out to be more disappointing, given almost a 30% chance of winning the day after her brilliant victory in the LR primary.
The medal of the biggest “loser” goes in fact to Éric Zemmour. First for having wasted the potential of his lightning breakthrough in October, when he reached almost a 40% chance of winning, then for having had his wings broken at the end of February by the Russian invasion in Ukraine after a brief comeback that placed him again ahead of Le Pen and Pécresse, his two rivals on the right.
Vladimir Putin’s interference in the presidential campaign seems to have been the determining factor.
We can see that until then, Emmanuel Macron’s status as the favorite did not guarantee him victory.
His rating had remained stable between 50%and 65% chances of victory from the beginning of 2021 until the end of February. In the end, this election will have been less influenced by French economic grievances than by Russian military violence.
