SURVEY MYTHS DEBUNKED - Online Surveys Do Not Use Real Random Sampling
Sampling is often considered the bogeyman of online surveys. Firstly, not everyone in any given population has Internet. Secondly, not everyone who is online can be reached by online surveys – many virtual groups and most companies simply do not allow their members’ or employees’ email addresses to be used in surveys. And thirdly, even among onliners whose emails are readily available for surveys, not everyone wants to participate in a survey.
So how can online research surveyors claim that they actually do random sampling? It’s just not possible, is it?
Why Random Sampling Is a Big Deal
Random sampling is the survey industry’s standard on professional surveys. It is what the best regarded and most reliable surveys do. The logic is hard to argue with: a small group that correctly mirrors the rest of the population will best represent the views and perspectives of the entire population.
And there is hard experience to back this up with. If you have been following our series on Surveys That Changed the World (as I’m sure you have), you know the story of how Gallup changed the history of surveys by pioneering modern sampling. The company became the very giant it is today because its founder, George Gallup, predicted an American election outcome using a sample of only 50,000 respondents and got it right, whereas others used samples of multiple millions of respondents and got it wrong.
Since then, the one universally accepted article of faith among research survey practitioners is that faulty sampling is the number-one risk of any survey.
Also, if you have been following our series on Survey Myths Debunked (as I’m sure you have too), you know there are four random sampling techniques – simple random sampling, systemic random sampling, stratified sampling, and cluster sampling. For a detailed look of those techniques plus the various types of non-random sampling techniques check out our blog article The Effect of Sample Size On a Survey’s Results.
So, yes, it is a big deal if online surveys have a unique – call it a genetic - inability to get random sampling right. That would basically make them inferior to other survey methods and pretty much unreliable. But is it true that online surveys are incapable of getting random sampling right?
The Trouble With Online Random Sampling
First off, random sampling is a problem to all types of surveys, not just the online version. Not everyone has a phone or an easily accessible residence or office from where a survey can be conducted. And not everyone who does have that wants to participate in a survey.
But it is also true that online surveying has issues with random sampling that are unique to the Internet. The main one is that people tend to participate in an online survey about a topic they like, and will rarely bother with a survey about less ‘interesting’ topics. This is a particularly dangerous problem, because it tends to reinforce or ‘confirm’ conventional views or prejudices within a small group.
This problem is made worse by the huge increase in the number of online surveys in recent years. People have simply become swamped and therefore less enthusiastic about participating in online surveys. Some research firms have faced down that challenge by resorting to rewarding participants with gifts, certificates, and cash. But that has brought in another problem: it risks upsetting sampling by attracting the most opinionated people or weakens the value of the data collected as people rush through questionnaires just to earn a reward.
But, in both online and offline surveys, there are established ways of going around the challenges of random sampling. That is where the myth about online surveys being incapable of nailing random sampling falls flat.
How Online Surveys Tackle Random Sampling Challenges
Perhaps the biggest misunderstanding about online surveys is the belief that they are ‘normal’ surveys except that they are conducted through the Internet. Online surveys are in fact different and are special species of surveys. Except for very rare cases, online surveys are about gauging the views or perspectives of narrowly defined segments of a population, and not the entire population. Most online surveys are business or market oriented affairs targeted at a small population segment, such as, say, males of ages 24- 35 living and working in City X, or enthusiasts of a particular sport or professionals in a particular field.
Online surveys therefore rarely need to sample an entire population. They only sample a segment of the population, and mostly a segment that is all over the Internet. Random sampling that is restricted to such groups is a lot easier to do, and online surveyors have little trouble nailing it down.
And even when, in those rare instances, online surveys are confronted with situations that require random sampling of an entire population, there are, here too, established ways dealing of with that challenge.
The usual approach, which is also used by offline surveys confronted by random sampling headaches, focuses on two things: the fact that not everyone has Internet access, and the fact that those with a particular interest in something are likely to unfairly dominate the sample.
Both problems are sorted out using ‘score’ or ‘data’ weighting. The concept is fairly complex but, essentially, it means adjusting the survey data so that the final sample reflects the known population ratios as drawn from the population census or other credible population sample. In other words, the sample is checked against already verified samples to ensure it covers a proper cross-section of the population.
Similarly, the data is ‘weighed’ to ensure that those with an interest in the survey topic do not dominate the sample and therefore tilt the results. This is done by analyzing the response data and ensuring that the ratio of those who love the survey topic – as shown by their response to questions deliberately set to gauge their attitude towards the topic – are balanced by those who are more ambivalent or disinterested.
What it all means is that, no, online surveys do not have a random sampling genetic deficiency. They may or may not fail the test of random sampling depending on how well they are conducted – just like in any other type of survey. But never because they just can’t hack it.
Is there something about surveys that you think is confusing or often done wrong? Give us a shout in the comments and we will do our best to help clear it up!
Photo credit: Lego (thanks, Lego)
You might also like...
What To Do For a Higher Response Rate
How Knowing Your Respondents Can Help Improve Your Survey Results
Tips and Tricks: 5 Easy Ways To Increase Your Response Rate
Feb 21, 2014