How to Improve Survey Response Based on Research
Designing health-research surveys online is difficult enough and dealing with non-response on top of that can make it seem impossible to get good data.
Factors that Affect Non-Response Rates
There are a multitude of factors that affect nonresponse rates for surveys; notably, the perception of burden and incentives are two big ones.
Burden
The burden that it takes a participant to complete a survey is a significant but albeit ambiguous factor. It’s often thought that survey burden means the length of time it takes to complete a survey, but studies have shown that it’s inconclusive whether this factor actually makes a difference for nonresponse rates.1
Another measure of burden is the value that a participant feels that the survey brings; one study found that those who believed the survey to be useful found it less burdensome but those that thought it was an invasion of privacy found it more burdensome.1
Incentives
Another factor that can affect nonresponse rates is incentives. Although it seems simple, it can be a highly effective technique to reduce nonresponse. One researcher found that even a small incentive of $1-2, as opposed to nothing, improves cooperation and reduces nonresponse rates.1
Another interesting finding is that incentives need not be monetary. While money and rewards are the typical incentives, a study found that the top 3 reasons to participate in a survey were: 1) altruistic reasons, 2) monetary incentives and 3) interest in the survey topic.1 Thus, providing information on the topic afterhand or appealing to the altruistic nature of the survey may also be an incentive.
Survey Design Questions
Along with survey burden and incentives, the actual survey questions are a big factor that can affect not only the response rate but also the quality of data that you are able to collect. While survey questions must ultimately be tailored to your study requirements, here are some considerations that could help improve the response rate and quality of your data.
Designing Survey Questions to Overcome Biases
Agreement Bias
Certain question formats can hinder the quality of data that you collect from a survey as it can increase the likelihood of biases. Yes/No questions are classically cited as questions that increase acquiescence bias (aka agreement bias), leading people to simply agree with a statement regardless of what they actually think.2 Yes or no questions might create an inherent bias and thus avoiding these questions wherever possible when designing surveys may improve the quality of data that’s collected.
Demand Bias
When the choice is between two statements or options, research suggests that agreement bias can be reduced when both options are presented as neutrally as possible; for instance, one option should not sound ‘bad’ and the other ‘good.’3 If the options are presented in this unipolar way,3 your survey may also be subject to demand bias and participants may try to respond to answers that they think you want to hear.4 Another thing to consider are the images and word choice that’s being used for your survey questions. Even the difference between the words ‘house’ and ‘home’ could make a big difference in how a participant perceives a question.4
Extreme/Neutral Responding Bias
Many of us as researchers and survey participants have encountered and participated in the extreme/neutral responding bias. When given sliding scale questions, participants may simply choose an extreme response or neutral response. Extreme response bias in particular was found to be correlated with people that are older or those that have a lower level ofeducation.5 Some things to consider if you are including a sliding scale question in your survey are what the scale is measuring, the length of the scale, and the visual layout of the scale.6
Often these types of questions can be difficult to analyze because participants might have a different meaning to sliding scale measures like ‘satisfied’ and ‘very satisfied.’7 Even when a scale has numerical values, are the participants treating each number on a scale equadistant?7 If using these types of questions, consider how a participant might view each item on the scale and clarify the measure with clear language so as to avoid confusion.
It may not be possible to always design your survey according to all the principles that reduce bias but being aware of them will help you through data interpretation.
You can now create secure HIPAA-compliant surveys directly in Honeybee with Honeybee Forms, our custom survey builder. The drag and drop building is super easy to use and is included free to increase the quality of the participants coming through Honeybee. If you’re looking for a flexible tool to create surveys, check out how to do it on Honeybee!
References
1. Read “Nonresponse in Social Science Surveys: A Research Agenda” at NAP.Edu. doi:10.17226/18293
2. The 5 Most Harmful Survey Response Biases (And How to Avoid Them). Paperform Blog. Published March 8, 2021. Accessed August 25, 2021. https://paperform.co/blog/survey-response-bias/
3. Hinz A, Michalski D, Schwarz R, Herzberg PY. The acquiescence effect in responding to a questionnaire. Psychosoc Med. 2007;4:Doc07.
4. Avoiding Bias in Survey Forms (a.k.a Skewing Research Data to Prove a Point) | by Christine Kane | UX Planet. Accessed August 25, 2021. https://uxplanet.org/avoiding-bias-in-survey-forms-a-k-a-skewing-research-data-to-prove-a-point-6f08494bca2
5. Meisenberg G, Williams A. Are acquiescent and extreme response styles related to low intelligence and education? Personality and Individual Differences. 2008;44(7):1539-1550. doi:10.1016/j.paid.2008.01.010
6. DeCastellarnau A. A classification of response scale characteristics that affect data quality: a literature review. Qual Quant. 2018;52(4):1523-1559. doi:10.1007/s11135-017-0533-4
7. S. westland@leeds.ac.uk. The dangers of Likert scale data. Colourchat. Published February 18, 2014. Accessed August 25, 2021. https://colourware.org/2014/02/18/the-dangers-of-likert-scale-data/