Ways to Avoid Sampling Bias in Surveys

Sampling bias is something that can easily creep into surveys when the methods used unwittingly favour certain outcomes over others. While it is something that can be committed by even the most experienced professionals, any steps you can take to avoid it are crucial if you are to maximise the validity of your results and what you are able to achieve with them going forward.

What is Sampling Bias?

Sampling bias, also referred to as sample selection bias, refers to errors that occur in surveys and research studies when participants are not accurately selected.

Preferably you should select your online survey participants at random. If you fail to do this, you will run the risk of severely impacting the validity of your results and findings, as your sample will not accurately reflect your population of interest.

A famous example of Sampling Bias from yesteryear

Unfortunately sampling bias is something that can occur all too frequently. Probably one of most famous and impactful examples of this was during the Truman-Dewey United States presidential race of 1948.

During the race, a political telephone survey was conducted nationwide whose results implied a heavy landslide win for Dewey over Truman. However, the study made one large fatal error in failing to take account of the fact that telephones were still a relatively new and expensive form of technology.

Due to their cost, only a small number of wealthy families owned and kept a telephone in their home. Subsequently, this meant that only relatively wealthy families took part in the telephone survey, and these participants tended to support Dewey, in contract to lower-middle and lower-class families who were more likely to support Truman.

By failing to properly examine the population sample that owned telephones during this era, the researchers conducting the telephone survey committed sampling bias, resulting in severely skewed response data. Rather than distribute a survey to a sample that more accurately represented the population of the United States, the researchers ended up with inaccurate and unrepresentative insights, which failed to predict Truman, as the eventual winner of the presidential race.

How to avoid Sampling Bias

When you are designing your survey, there are three steps you should take to eliminate bias:

1. Correctly identify your survey goals
2. Select clearly defined requirements for your target audience
3. Give all potential respondents an equal chance of taking part in your survey

You will also need to ensure you have selected the right people, whose profiles fit your survey’s goals. However, depending on the survey you are looking to conduct and the profiles you have available among your existing contacts, this can sometimes be more challenging than at other times.

If for example you wanted to interview a specific demographic group, but didn’t possess a sufficient number of these profiles within your existing contacts, a Live Consumer Panels service could get your survey in front of your chosen audience and generate responses, as and when you required them. The great thing about this service is that you can tailor your number of responses and demographic/lifestyle segments to meet your budget requirements.

The process of how to set this up, as well as a more in-depth look at the benefits of survey panels is explored in our ‘How to get the Right Survey Audience for your Online Questionnaire’

More tips to combat Sampling Bias in your surveys

There are also some tried and tested techniques you can use to avoid sampling bias these include:

Simple Random Sampling

Using this method samples are selected strictly by chance. This ensures that there are equal odds for every member of the population to be chosen as a participant in any chosen study.

The great thing about simple random sampling is that no effort is required on the part of your potential participants. For example, a computer can be used to randomly select names from a master list, with those chosen names becoming participants in a study.

Another popular method of generating a random sample using a spreadsheet like excel is to add the formula “=RAND()” to every row of your list of contacts. By default, this will assign a random decimal value between 0 and 1 to all your contacts. If you then order the list based on this generated random value, you can then select any continuous group of people in the list (the top 100, or the bottom 100, for example) knowing that their place in the list has been generated at random. This method isn’t perfect (computers and electronic devices can struggle with true random number generation), but is useful for all but the largest studies.

Stratified Random Sampling

Another technique that can be used to avoid sampling bias is stratified random sampling.

Using this approach, those conducting the survey will be able to examine the population they will be working with in their study and comprise an accurately representative sample accordingly.

For example, stratified random sampling is effective if there are 1,000 individuals in a population and 10 people from the population are required to conduct a study. If 500 members of the targeted population are working women, and 500 members of the population are working men, then the researchers’ sample should accurately reflect this.

In this scenario, it means that the sample must be comprised of five women and five men.

Stratified random sampling helps researchers become aware of this information prior to building their sample, to help ensure they avoid sampling bias.

Checking and rechecking to remove bias from your surveys

If you are about to create a fresh survey and want the best chance of avoiding sampling bias, you may like to revisit some of the advice and ideas we have discussed in this blog post.

It’s also important to consider that no matter how long you may have been in the survey creation business, sampling bias is something that can be committed by even the most experienced professionals. So, irrespective of what methodology you have chosen to create accurately representative samples, its imperative to check and double check your work if you are to ensure you don’t make costly mistakes further down the line.