Methodology of the Abacus Data National Poll
Over the next few weeks, we are going to be releasing a number of studies from an omnibus survey we conducted last weekend (October 29 to November 1).
We interviewed 1,001 Canadians who were randomly selected from a national online panel of about 400,000 Canadians. We partnered with a very respectable, online research company called Research Now. They use multiple techniques to recruit panelists, are careful in maintaining their panel, and ensure they have an engaged group of potential research participants. Abacus Data has licensed state-of-the-art survey technology and programs all its online surveys in-house. We purchase sample from companies like Research Now as we would if we were conducting telephone surveys.
One of the criticisms of online research is that you cannot get a truly random probability sample with it. This is correct and is the reason why you cannot have a true margin of error with online surveys. However, advances in online research and the growth of online panels, like those maintained by Research Now, Vision Critical, or YouGov, means that we can mimic random sampling techniques or get results similar to that of random or stratified sampling.
For our most recent survey, we took a number of steps to ensure that the participants were representative of Canadians.
1. We set quotas to match the distribution of a number of demographic and geographic variables to the 2006 Census. Quotas were set for gender, age, region, language, education, and type of community (urban or rural).
2. Even with the quotas, we statistically weighted the data by gender, age, and region to ensure that met the appropriate proportion of these attributes.
The growth of online surveys is a response to declining participation in telephone surveys. Fewer and fewer Canadians are participating in telephone surveys. Even with random sampling techniques and the inclusion of cellphone numbers into the sample, it has become more difficult to get people to answer questions – we all know people who won’t answer the phone if they don’t recognize the number that’s calling. When people stop answering surveys, response bias increases and this can hurt the accuracy of surveys.
While there are certain challenges with online research (older and lower income Canadians are less likely to be online and to join research panels), the size and breadth of some of the available online panels and the way panels are recruited mitigate many of these challenges. Moreover, the benefits of online research make it an acceptable alternative when conducting quantitative and qualitative research.
Online surveys allow researchers to use interactive techniques and multimedia within the survey. You can present more information within the text of the survey than you can by phone. It is also easier to reach younger voters or consumers who are an important consumer segment. I don’t have a land line and am part of a growing number of Canadians who rely solely on a mobile phone.
At the end of the day, each survey methodology has its drawbacks. As a result, pollsters must be honest about the potential sources of bias in their research and work hard to mitigate them.
I’m excited about the future of online research. It is the future of survey research and Abacus Data will become a part of a group of research professionals working to improve and fine-tune the methodology to get the most accurate and effective research results for the public, the media, and most importantly, our clients.