Do opinion polls have value?
Methodology under fire as election predictions have been wildly off in past
EDMONTON – With the provincial election campaign underway, public opinion polls are already surfacing — and with them, concerns about their methodology, accuracy and value to the public after they failed miserably in previous votes.
Three years ago, not one poll available to the public accurately predicted the Progressive Conservatives would surge past the front-running Wildrose. The next year, in B.C., 10 polling companies failed to forecast the Liberal win over the NDP.
So, what can be done? Are accurate polls a thing of the past, relegated to a time when everyone had a land line and answered their calls?
“There’s no excuse I think for what happened in Alberta and British Columbia. That’s not even close to accurate, so there was something we need to learn about,” said Abacus Data CEO David Coletto.
“But even in more recent provincial elections in Ontario and Quebec, where the polls generally did OK, the assumption is always that we have to be dead on accurate. Polls can’t always do that. If they do, it’s more luck than anything. They’re meant to be snapshots. There is error built into it.”
Still, there is a huge divide in methodology, and for the public, that’s not always readily apparently when political parties are waving various numbers around.
On Wednesday, for example, the Wildrose was pointing to a Think HQ poll that found their party, the PCs and the NDP in a dead heat; another poll from Mainstreet Technologies released Thursday is showing a similarly tight race. They follow on the heels of two polls last week, one by Mainstreet and the other by Insights West, that also both found the Wildrose and PCs were neck and neck.
The Insights West and Think HQ polls were from online panels; the Mainstreet Technologies ones were Interactive Voice Response polls — often dubbed “demon diallers” — which are automated phone calls.
Calgary-based pollster and pundit Janet Brown laments the proliferation of such online panel surveys and IVR polls, which she considers “cheap and fast methodologies” that skew toward people motivated by anger.
“The political parties are paying to have polling done right and the media is looking for free polling — and the media get what they pay for,” Brown says, referring to the trend for the online and IVR pollsters to provide their results to media organizations for free.
She stresses the need for properly constructed and administered live telephone polls, which are more costly to do.
Yet in the 2012 election all manner of polls conducted within a week of the April 23 vote were wrong. Theirs was a majority of error, not a margin.
Live telephone polls by Return on Insight and Leger Marketing conducted between April 13 and 16, 2012, hinted at a narrowing race but gave Wildrose the nod, as did IVR polls by Forum Research on April 22 and Abacus Data on April 18-19, and an online poll conducted April 17-18 by ThinkHQ.
So if every method of polling returned similar results in 2012, are they all faulty?
“There is no silver bullet,” said Return on Insight president Bruce Cameron. “There is no single methodology that works all the time.”
Adds Ian Large, Leger Marketing vice-president for Alberta: “What we learned in 2012 in Alberta is more about politics and less about polling. … Campaigns really matter and it’s possible to blow it at the last minute.”
Election day surprises are never good for companies whose business is to know the results beforehand, plus or minus three per cent, 19 times out of 20, of course. The Alberta and B.C. whiffs were part of a bad run as polling companies also fared poorly in the 2011 Ontario election.
But the industry is most certainly in a state of flux.
One-third of Canadians no longer have land lines and caller-ID means many others won’t answer unfamiliar calls on their cellphones or land lines. That means it takes longer to do a poll, which drives live survey costs higher and making it less likely media organizations will shell out the $10,000-$20,0000 needed to partner with a major polling firm.
“I would imagine in the next couple of years there will not be telephone surveys being done and instead we’re going to be looking for new ways to collect data online,” said Coletto, who has used IVR in the past, but will not do so again. “One of the more recent methodologies would be to randomly pull people off websites.”
Cameron believes the future lies in a hybrid methodology combining live telephone delivery and online recruitment of respondents that approaches the randomness of a phone sample. He said those hybrids are under development in Canada.