Nonresponse Bias: What to Avoid When Creating Surveys

Nonresponse bias is used to categorize the group of people who don't answer surveys for intentional or unintentional reasons.

In this post, we'll discuss everything you need to know about nonresponse bias, including what it is, what causes it, and what you can do to avoid it when crafting surveys at your business.

Table of Contents

• What is nonresponse bias?
• Common Causes of Nonresponse Bias
• Examples of Nonresponse Bias
• How to Reduce Nonresponse Bias
• Takeaway

In simpler terms, nonresponse bias occurs when survey participants either can't or won't respond to a survey question or an entire survey. Reasons for nonresponse vary from person to person.

To be considered a form of bias, a source of error must be systematic in nature. Nonresponse bias is not an exception to this rule. If a survey method is developed in a way that makes it more likely for specific groups of potential respondents to refuse to participate or to be absent during a survey, it has created a systematic bias.

Let's say there's a survey question asking if people have cheated on a standardized test. People who have cheated may refuse to respond to this survey. This can lead to nonresponse bias.

Nonresponse bias accounts for the percentage of people who forget, don't understand, or simply don't want to take their survey. This makes nonresponse bias a common problem for researchers.

Response Bias vs. Nonresponse Bias

Nonresponse bias refers to the group of people who don't respond to your survey. Meanwhile, response bias focuses on the participants who did respond. Response bias also looks at the reasons why people would answer survey questions inaccurately or untruthfully.

For example, if a participant felt pressured by a survey maker to say their experience with a brand was great, that could create response bias. However, if the participant felt intimidated by the question and decided not to answer, that would be considered nonresponse bias.

Nonresponse bias is caused by a few different factors that your business should be aware of. You'll review each one, as well as some examples of nonresponse bias, in the sections below.

Common Causes of Nonresponse Bias

Poor Survey Construction

If your survey is too long or too difficult to understand, participants will be quick to abandon it. In fact, research shows that completion rates for surveys drop between 5%-20% when it takes longer than seven or eight minutes to complete it.

Pro tip: Be sure that your surveys are brief, user-friendly, and engaging if you want to keep your response rates high.

Incorrect Target Audience

Some people just aren't the right audience for surveys. Maybe they don't have the time to fill them out, or maybe they're just not interested in providing your brand with personal information.

Regardless of the reason, it's important to leverage your customer data when distributing surveys and target groups that would be most likely to participate.

Pro tip: You can try changing up your survey's trigger and timing if you notice that your primary audience doesn't seem engaged.

Refusal to Participate

For some customers, the very last thing that they want to do is participate in a survey. We've all fallen into this category at one point or another.

In these cases, timing is critical. If a person is adamant about not participating in your survey, then it may not be the right time for them. It doesn't mean that they'll never fill out one of your surveys. Rather, you just need to find the right time and channel to connect with these individuals to gather their feedback.

Failed Survey Delivery

Sometimes the survey doesn't make it to the recipient. For example, if you're sending a survey via email, it could get lost in a spam folder. This still gets recorded as nonresponse.

Pro tip: The best surveys have safeguards in place that account for this variable. For instance, you can add a tracking pixel to your emails to confirm that your messages are reaching your participants. Or, if you're managing a mail-in survey, you can include a return address for any surveys that don't reach a mailbox.

A simple system like this will help you keep track of surveys that were never completed because they never were seen by a survey taker.

Accidental Omission

Another common cause of nonresponse bias is forgetting to complete the survey altogether. It may not be that your survey was dull or irrelevant. Instead, it may be that the participant was distracted and pulled away from the form before they could complete it. While it's hard to prevent these incidents, ideally, they should only make up a small fraction of your nonresponses.

Let's see this concept in action with common examples of nonresponse bias.

Examples of Nonresponse Bias

1. Outdated Customer Information

Let's say you wanted to send a survey to a group of customers who attended one of your events two years ago. While you have all of their email addresses recorded in your CRM, you haven't gathered any new information on them since their last event.

When you send your survey, you notice that your delivery and open rates are extremely low. Turns out, your customers have changed their email addresses and aren't checking the inboxes that you have listed under their names.

This would be considered nonresponse bias. While these customers were technically sent the survey, they never had a real chance to interact with it.

2. Requesting Sensitive Information

In this example, you're running a cybersecurity company that's working on a new way to encrypt email addresses. To get an idea of your customers' current security setup, you distribute a survey asking them to provide various passwords to different accounts as well as their reasoning for why they chose those words or phrases.

To your surprise, no one completes your survey. In fact, a few customers even call your support team to report your survey as potential fraud.

Image source

3. Forgotten Survey

For this example, let's say you sent a mail-in survey to your customers about a month ago. Your survey's instructions are to submit the form by the end of the month so participants can receive a special promo offer.

Just as the end of the month approaches, a global crisis strikes and your company quickly pivots its marketing, sales, and customer service efforts to adapt to the new environment. You decide to put your survey on hold because customers are more focused on other needs than providing feedback.

This would also be considered nonresponse bias because participants never got a fair chance to submit the survey. If they waited until the end of the month, they wouldn't have had the same opportunity to collect the promo offer as those who submitted the survey earlier.

Now that you're familiar with how nonresponse bias can affect your surveys, let's review what you can do to avoid it.

1. Reconsider your survey's trigger and timing.

You want to present your survey at a moment when participants will be most likely to respond. Further, the survey should only take a few minutes to complete. Approaching people at the right time and in the right way will make it easier for you to encourage them to participate.

Your customer journey map is an excellent resource to use when reviewing your survey's timing. You can look for moments of delight where customers would want to share an experience, or you can look for pain points where people would want to voice their criticisms.

Targeting these memorable interactions is the key to approaching participants at the optimal point in their customer journey.

2. Optimize your survey's design.

Survey length can be one of the biggest factors that determine whether participants fill out your survey. The ideal survey length is no more than 20 questions long and should take roughly seven or eight minutes to complete. Anything longer than that will lead to lower response and completion rates.

Another important factor to consider is your survey's design. If you stack questions on top of each other, viewers may be intimidated by all of the scrolling they'll have to do just to reach the end of the page.

Pro tip: If you have access to customer feedback software, you can trigger questions one at a time, so customers can focus on each one individually and won't be overwhelmed by a 30-question list.

3. Review your survey questions.

The type of questions that you're asking will also influence participation. Long-response answers and comment boxes get exhausting over time. Double-barreled questions, like the example below, can be confusing for participants and lead to abandonment.

Image source

Your survey should primarily include close-ended questions like multiple choice or Likert scales. These give customers a fixed number of responses to choose from which makes the survey much easier to fill out.

4. Leverage customer data.

Targeting the right participants for your survey can also play a major role in its success. If you have a CRM, you can leverage customer data and send your survey to people who would be more willing to complete it.

Buyer personas would be very helpful in this situation. You can use them to identify target audiences that would be more likely to participate in a survey.

You can also look at past interactions with individual accounts to see if anyone recently interacted with your brand and may want to provide feedback.

Not only can this help you attain valuable insights from your customers, but it can also help you connect with accounts that are potentially at risk of churn.

5. Provide options for omission.

While your survey should primarily offer closed-ended questions, it's important to include options to omit individual answers as well. You can either not require all questions to be answered or you can provide a dedicated multiple-choice option that participants can choose for omitting each question.

Here's an example.

Image source

When asking for feedback, your customers shouldn't feel pressured or forced to provide information. Most would rather abandon your survey altogether than willfully answer a question that makes them feel uncomfortable.

Including this omission option provides a safety net for those who may want to skip a specific section. After all, it's not worth losing all of their answers over just one survey question.

6. Reward customers with incentives.

Offer incentives for completed surveys to boost engagement. This could be a discount or a small prize that's obtained once a participant fills out your form.

Pro tip: If you're not sure what to offer, a good place to start is with your customer loyalty program. You can borrow some of its rewards or promotions and apply them to your survey.

If your loyalty program is point-based, you can offer participants loyalty points that can be added to their accounts. This is a great way to acquire new signups for your loyalty program and increase overall customer retention.

7. Keep the survey length short.

The length of a survey is usually the deciding factor for most participants. Time is precious, and most people don't want to spend their time answering survey questions. It is up to you to get them to participate in your survey by keeping it as concise as possible.

Pro tip: A survey of 30 questions is the industry standard because it takes about 8 minutes to complete.

8. Make sure the information is confidential.

Some participants might refuse to answer a question because they fear their responses will be made public. If you can, let them know that they will be anonymous and that their information will be strictly confidential. This can help quiet their fears.

Nonresponse Bias and Your Surveys

Nonresponse bias is very common and can be detrimental to survey results.

People are increasingly refusing to participate in surveys, leading researchers to use "convenience samples." Convenience sampling is a method where survey researchers collect data from participants who are willing and available.

When that pool of available participants isn't the right fit for the survey, it can result in misleading or incorrect discoveries.

Once you've handled nonresponse bias, you can continue to master different types of surveys and questionnaires.