Common Types of Survey Bias and How to Avoid Them

44% of companies base most of their decisions on data, according to SPGlobal.

While market research can give you access to a plethora of insights, their reliability will depend on the quality of your data. The more skewed it is, the smaller its ROI.

The first step to making sure your data is credible is by eliminating survey bias.

In today's piece, we're going to cover what survey bias is, discuss its most common types, and tell you how to avoid them. Let's get started.

Table of Contents

• What is survey bias?
• Why Survey Bias Matters
• Types of Survey Bias

By eliminating or at least reducing survey bias by designing your surveys well, you improve the credibility of your data.

Why Survey Bias Matters

What happens if you make decisions based on manipulated data (that's what survey bias actually is)? It can seriously harm your business, as the insights you rely on don't reflect the reality you operate in.

Let's take a closer look at why you should avoid bias in questionnaires.

Making Misguided Business Decisions

Market research can generate tons of data, but it's also time-consuming and costly. It will only pay off if the data you use is objective.

If you fail to eliminate survey bias, then not only will you waste your marketing budget, but also put your organization at risk.

Missing Out on Important Information

If your survey is biased, respondents might not be able to provide a truthful answer. For example, they might not see a suitable option to select from your closed list, or would rather provide a text reply.

What's more, some survey recipients might decide not to mention something that's bugging them. This is especially true if the survey creator's opinion shines through the questions.

As a result, some voices will remain unheard, or you'll only hear part of the story.

Risk of Discriminating Against Certain Groups

Some surveys are biased because their creators forget to factor in context.

Let's say you wanted to ask your employees if their family members were affected by new immigration laws in your country. However, the majority of those who've received the survey have no relatives abroad.

As a result, 95% of responses were "no." If you don't look into the 5% who said "yes" and read into their stories, then this could count as discriminatory behavior.

Types of Survey Bias

Below, we discuss eight common survey bias types. In a nutshell, they circle around four main factors:

• The survey creator's own beliefs.
• Lack of responses to certain questions.
• Respondents telling you what they think you want to hear.
• Small sample sizes or non-representative respondent groups.

1. Sampling Bias

Sampling bias, also known as purposive sampling, happens when certain groups are systematically more likely to be selected in a sample than others.

For example, White respondents may make up a disproportionate amount of your research group.

Purposive sampling may work when surveying a smaller group. For example, you may want to survey people in a specific income bracket who live in one city.

However, if you want to test a larger population, you should avoid sampling bias. Otherwise, you'll get skewed results.

Example of Sampling Bias

Imagine a situation where you run a survey on the importance of using natural ingredients in cosmetics.

If you only focus on young women who lead a healthy lifestyle, are vegetarian, and only buy organic food, they'll be more likely to say it's super important to them. The general population might think differently.

Pro tip: If you want to make sure your sample is varied, try to eliminate obstacles that can stop some people from participating. Consider using various methods to distribute your survey, such as email, in an app, or placing it on your website.

2. Non-Response Bias

Whenever you run a survey among a large group, you expect that some people won't respond, and that's fine.

However, if the opinions of those who didn't respond drastically differ from those who replied, it will result in non-response bias.

Example of Non-Response Bias

A non-response survey bias example that you're probably familiar with happens during elections.

Some people don't make it to a polling station because they have other obligations, like work or childcare, or they don't feel their vote makes a difference.

These individuals might have completely different worldviews from those who do vote. However, since they don't vote, the party whose electorate does may win, even though many people disagree with them.

Pro tip: Survey delivery matters. Before you send your survey, test it. Run it on various devices to verify what the experience will look like for your respondents. Once you're sure the links work, watch the surveys you're sending and their response rates.

3. Survivorship Bias

Survivorship bias occurs when, despite targeting the right market segment, you only get to talk to a fraction of them, as the rest is no longer available. For example, their details have changed and you cannot get in touch with them.

Those whom you can contact are called the "survivors." Since they represent a certain group, you don't get the full picture, which leads to bias.

Example of Survivorship Bias

Let's say you've noticed that your employee attrition rate has skyrocketed over the past six months.

It causes concern among the management board, so you decide to investigate the issue. However, instead of running exit interviews, you run a survey.

Unfortunately, you can't get in touch with employees who have already left the organization. So, you have no choice but to send your survey to current employees.

As you can imagine, their views might differ strongly from those of workers who have already left the company.

Pro tip: The first step to avoiding survivorship bias is awareness. Verify your sample prior to sending your survey. If it turns out that a lot of those included in it are no longer available, look for other sources. It's best to use both qualitative and quantitative data.

4. Question Order Bias

If you place your survey questions in a specific order, then it will lead to question order bias.

For instance, if your respondent answers a question positively or negatively, they feel obliged to answer the following question in the same way.

This is due to the participant's desire to maintain consistency, even though in reality, they think differently.

Example of a Question Order Bias

If you asked physics students to solve a tough physics problem and then asked them how much they like physics, what do you think their answer would be?

Those who found the task difficult would probably say they don't enjoy physics. Meanwhile, those who tackled it quickly would say they do.

Pro tip: Randomize your questions and answers. There are survey platforms that allow you to resequence and reshuffle your answers and questions easily.

5. Conformity Bias

This type of bias in questionnaires happens when your respondents decide to answer with what they believe will be the "mainstream" answer. While it might reflect how they really feel, it's not always the case.

Some might choose such an approach if they fear repercussions — especially if they aren't sure the questionnaire is anonymous. Others might simply feel that their unpopular opinion won't matter.

Finally, some respondents feel an obligation to give you a positive reply. This can be true if they're being paid for filling out the survey.

Example of Conformity Bias

Let's imagine you run a B2B company. You send out a survey asking respondents to assess how happy they are working with you on a scale from 1 (lowest) to 5 (highest).

Some respondents might decide to give a "4" or "5", even if they'd rather cooperate with a different account manager or if their invoices are always overdue.

They might be afraid to point out these issues, especially if your company generates revenue.

Pro tip: A good way to tackle conformity bias is to anonymize responses. Make it clear to respondents that you won't be able to trace the survey back to them. This way, you'll encourage honesty and create room for more diversity in replies.

6. Dissent Bias

Dissent bias takes place when respondents hit "disagree" on all the statements in the survey.

A few factors contribute to dissent bias. Firstly, those who fill out the survey want to get through questions as quickly as possible. Secondly, if the survey is too long or complex, they might become tired. This is known as survey fatigue.

The main issue here is that it's hard to distinguish between biased surveys and those that actually reflect the respondent's sentiment.

Example of Dissent Bias

An ecommerce company wants to incentivize its customers to fill out a survey by offering a 10% discount coupon. Some respondents might not give the questionnaire a read and hit the lowest scores just to get their code.

Such an approach backfires for the company, and the replies have zero credibility and business use.

Pro tip: If you want thorough responses, consider selecting a smaller, but more relevant survey sample. Look at your customer segments — who are most likely to answer thoroughly?

Dive into your CRM and look at previous surveys. Try to spot long-time customers who've filled out a questionnaire in the past or interacted with your customer support. They're more likely to pay the survey the attention it requires.

7. Neutral Response Bias

Not all topics trigger strong emotions. So, in theory, there's nothing alarming if some of your respondents pick neutral options, like a "3" score on a Likert question scale.

Still, you need to make sure that you're asking the right respondents and that the question you're asking is specific. Otherwise, the neutral scores might come from those who don't understand it or don't have an opinion.

Picture asking a random group if they like the taste of Brie de Meaux when they don't know a thing about French cheese.

Example of Neutral Response Bias

On a scale from 1–5, how do you feel about the following exercises?

Cycling / Running / Swimming / Stretching

(On a scale from 1 to 5, where 1 is Hate and 5 is Love)

Those who exercise regularly will probably have a favorite sport, and their rankings will vary. But what about respondents who aren't physically active? Their answers will most probably be neutral.

They're unlikely to have an extreme attitude, as they don't do sports.

Pro tip: Choose your sample wisely. Make sure that the topic of the survey is relevant to them. As in the example above, if you're running a survey about sports, don't send it to people who don't engage in any physical activity.

8. Voluntary Response Bias

This type of bias in questionnaires takes place among respondents who volunteer to fill out a survey. These individuals tend to have strong feelings about the subject at hand.

This can lead to two negative occurrences.

First, respondents can over-report on one particular area and leave others unanswered.

Second, you might collect replies on opposite sides of the scale only without hearing from moderate respondents.

Example of Voluntary Response Bias

Let's say that you're an editor at a popular online news site and want to hear your readers' opinions on abortion rights.

If the survey has multiple questions or requires open-ended replies, it's going to require determination to complete the survey. This will likely eliminate those who don't have strong views on the subject.

Pro tip: Consider the channels you can distribute your survey in to gather more objective responses. Think of social media channels or forums where members seem to display a wider variety of viewpoints.

Eliminating Survey Bias

Before hitting send on your survey, it's important to ensure that it's as objective and user-friendly as can be.

Collecting genuine, high-quality responses comes down to a few factors — most importantly, choosing the right respondent sample and proper wording.

Be sure to show your survey to a few team members to minimize the risk of unconscious bias or tampered results.