How to Reduce Bias in Surveys

When it comes to your survey, you’ll want to do everything you can to avoid bias, which can result in inaccurate or unrepresentative data.

From sampling bias, nonresponse bias and response bias to order bias and even social desirability bias. It’s important to be able to recognise the different types of bias and stop them from happening in your surveys, as they can easily skew your results, leaving you with data that you are unable to take reliable decisions with

So, to help you we take a look at these different types of bias and offer some guidance to help prevent them from creeping into your surveys.

Sampling bias

Sampling bias, also referred to as sample selection bias, is an issue that can arise in your survey, if your survey participants are not accurately selected.

For example, let’s say you were trying to get the views of a specific demographic within your customer base, maybe all customers in the age range of 45-54. In order to make sure your data was truly representative of that group, you would need to share your survey in a way that gives all people within that group an equal chance of responding to it.

If you only shared a link to your survey via social media, it’s very likely you would risk biasing your results, given that some of your customers may not use or have social media accounts.

How to reduce sampling bias

The simplest way to reduce sampling bias in this scenario would be to share your survey using a wider range of distribution channels.

From a direct link to your survey, a web popup or a survey embedded into your website to sending it out via email, there are many more ways than just social media platforms for sharing your survey.

If you have the resources, you may also consider other types of survey such as hiring telephone or in-person interviewers to reach out to respondents that still don’t have access to the internet in their home. Alternatively, you can also mail out paper surveys to any segments of your population who are not tech savvy.

Nonresponse bias

Getting your sampling right is a good start, but you still need to address the bias that can be introduced through your respondents.

Regardless of how careful you’ve been to send a survey invite to everyone, there will always be some people who are either unwilling or unable to respond. Systematically different to those who do respond, this group introduces what is referred to as nonresponse bias into your survey.

How to reduce nonresponse bias

Increasing your response rate, which will increase the chances that everyone in your population (the group you need to draw conclusions about) is represented, is the best way to counter nonresponse bias.

Some methods you can employ to increase your response rates include:

• Sending respondents, a pre-notification email containing information about the survey to come
• Sending them a personalised invite
• Sending them a reminder

Response Bias

Besides those that don’t respond, you also need to think about those that do respond and the bias they can introduce to your survey through their answers. These can be imported both consciously and subconsciously, which can display itself in a number of ways.

Firstly, many people want to appear agreeable, which can lead to many survey respondents telling you what they think you want to hear. So, if your survey included agree/disagree Likert scale type questions these people would be more likely to select agree.

Similarly, many survey respondents will be motivated to answer in such a way that associates them with behaviours and characteristics that are desirable, while denying more undesirable characteristics and traits. This is often referred to as social desirability bias.

Finally, there are also respondents who have a tendency to only provide extreme or neutral responses. So, if you presented them with a 5-point answer scale the first group would always choose 1 and 5s, while the second group would consistently select the middle number 3 as their answer.

How to reduce response bias

The good news is that online surveys already help reduce the potential for response bias, as the questions are self-administered, making it easier for respondents to be honest.

Other ways in which you can reduce response bias include:

• Asking neutrally worded questions
• Think about using anonymous surveys
• Ensuring your answer options are not leading

Order bias

One of the final areas that can introduce bias into your surveys is the order of your survey questions and answers. This is where questions positioned earlier in your survey might influence how respondents choose to answer questions that appear later in your survey.

Consider the questions in the satisfaction survey below:

1. Please rate your satisfaction with our support

2. Please rate your satisfaction with our account managers

3. Please rate your satisfaction with our survey software

4. Please rate your overall satisfaction with SmartSurvey

In the above example, the placement of the overall satisfaction question at the bottom, risks being biased by the satisfaction questions above it. This is because depending on the respondents’ previous answers, they could cast a positive or negative influence on this final question. More commonly known as the assimilation effect, this is where the response to the last question is more similar to the questions before it, than it would be if that question preceded it or was asked on its own.

Besides the bias that is introduced through how you’ve positioned your questions, the order of your answer options can also bias your results, with those completing self-administered surveys usually preferring the first few answer options in a list.

Subsequently, any tools that you get with your survey software, that can enable you to randomise the order of your questions, answers and even pages, will be hugely valuable.

How to reduce order bias

There are a number of things you can do to reduce order bias in your surveys including:

• Limiting the volume of scale questions in your survey
• Consider grouping your survey by topic
• Randomising your questions and answer options
• Leaving demographic questions until later in our survey

Take your time to check your survey before distributing it

With so many different types of bias that your survey could be vulnerable to, it’s essential to double check it before you issue it. And it doesn’t matter how long you’ve been creating surveys, as it only takes one issue to skew your results.

If you can revisit some of the ideas we’ve outlined in this blog post, you should be able to identify and remove issues that might otherwise lead to survey bias.

It’s definitely worth taking the extra time to do this, as the reliability and validity of your data will depend on it.