Survey Research Services

In questionnaire design text, it is both important that question-wording should express what is being measured, and that appropriate response options are provided. However, there are times the latter is overlooked, and a mismatch occurs between the question wording and response options. So, the big question is, how detrimental are these mismatches? Below, we discuss the effects of badly worded survey questions (how mismatches between survey questions and response options affect a study). We hope this information will help you weigh whether it is more critical for questionnaires to firmly match survey wording, or give more significance to more straightforward and less wordy options.

Common Sample of Mismatched Questions

Effect of Mismatched Questions

• Higher Item Nonresponse

Mismatched questions will result in higher item nonresponse because of the difficulty in responding due to additional cognitive processing. In responding to survey questions, respondents must identify the question, understand it, recover relevant information, create a judgment, and then report (Jenkins and Dillman 1997; Tourangeau, Rips, and Rasinski 2000). Consequently, there is a probability some respondents will skip items, or refuse to answer when they are troubled to exert additional intellectual effort. In an experiment by Dillman et al. (2014), they found that the mismatched version caused greater item nonresponse, and differences in response distributions, compared to the matched version. The study varied the question stems paired with forced-choice response options to be matched or mismatched with check-all question stems.

• Longer Response Time

Mismatched versions will take longer to finish compared to matched versions because of the added time-consuming comprehension and mapping complications. Additionally, the mismatch may only be identified by the respondents when they are mapping their answer to the response options and determine they are not appropriate. In this case, they have to exert additional effort in finding an adequate answer in the response option to replace their answer. Van der Zouwen and Dijkstra (2002) determined that mismatches or poorly designed response options (inadequate range of response alternatives) were greatly connected with swaying from the classic interviewer/respondent interaction in a question/answer sequence. An example of a classic interaction is an interviewer asking exactly worded questions to respondents, and receiving an adequate answer from them. Mismatch tends to result in respondents asking the interviewer to clarify their question further, resulting in longer survey completion time.  

Confirmation by a Recent Study of the University of Nebraska-Lincoln: “The Effects of Mismatches Between Survey Question Stems and Response Options on Data Quality and Responses”

The experiment divided the respondents into two groups. The first group received a survey version with all answer options and response options matched, and the 2nd group a slightly mismatched version. Their theory was that mismatched questions would result in higher item nonresponse and longer response time.

Example Matched and Mismatched Survey Questions from the Study

Results of the Study

The item nonresponse rate is higher in the mismatched than in the matched version. The probability of an item being unanswered in the mismatched version compared to the matched version is 1.6 times. 

Response time was longer in the mismatched than in the matched version. During the survey conducted via telephone interview, the researchers found that the interviewees took more time to complete the mismatched version, compared to the correctly matched one. 

The effect of the mismatch is not greater for those with lower cognitive abilities based on age or education. 

Summary

The study confirmed that mismatches weaken data quality in both mail and telephone surveys. Although, it may be less detrimental for the mail survey because it is self-administered, and respondents can see both question stem and response options. Researchers should design their questionnaires carefully, taking into consideration the proper relationship of the question stem and response option. Moreover, in analyzing survey results, researchers should review both stem questions and response options to determine if a mismatch could have made an impact on the results. 

For more information on Survey Design, survey mailing services, and data capture services, quantitative data collection services in general, contact us!

When done correctly, surveys are the golden key in social science research – helping us uncover the attitudes and behaviors of a target population like no other platform. As with all scientific research, however, they are susceptible to bias. Bias is a sneaky, subtle characteristic that can creep into any part of your research in the form of leading questions, skewed sample selection, respondent social pressures, and more. In fact, bias is so prevalent in our everyday lives, it is often difficult to spot. 

A primary goal of research is to minimize bias – if not entirely eliminate it. In so doing, results can be trusted as an honest reflection of the attitudes and behaviors of the total target population. Thankfully, survey science has been around a long time and the most common biases are well documented. 

As a leading provider of mail and multi-modal surveys, we help research professionals avoid bias every day. Following are the 8 most common data capture services survey bias errors we encounter, with helpful tips on how to avoid them. 

1. Social Desirability Bias

Respondents have a propensity to answer questions in a way that makes them look good according to social norms. Such topics as taking care of the environment, spending time with one’s children, etc., are ripe for socially acceptable responses that don’t quite match up with reality. This also applies to group norms as well, such as attending church, exercising and more. 

How to avoid it: Don’t use yes or no questions for these topics. Have respondents select from alternatives or use a ranking or rating scale. 

2. Acquiescence Bias

Respondents tend to agree and give a “yes” response to most questions, especially if they haven’t given the topic much thought before. For example, would you want your washing machine to have more preset options? Sure, why not. Next question.

How to avoid it: Don’t use yes or no questions for these topics. Have respondents select from alternatives or use a ranking or rating scale. 

3. Question Order Bias

Question order matters. Among the most common culprits of question order bias is “letting the cat out of the bag” too soon. For example, if you are doing a brand awareness survey and mention your brand name too early, you will inadvertently affect how people rate their familiarity with your brand on subsequent questions. This also holds true for response option order. A respondent might remember a choice that appeared in an earlier question and be more likely to select that response on later questions. 

How to avoid it: Use a logical question sequence that goes from general to specific. Manage response order with randomization.

4. Habituation Bias

When a series of questions are worded similarly or use a similar structure, respondents tend to answer in a less engaging way. Instead, many spot the pattern and go on autopilot to get through the survey with minimal energy. This adversely affects data quality, as respondents do not give each question the consideration it deserves. 

How to avoid it: Vary question-wording and keep it conversational. 

5. Sponsorship Bias

When respondents know who commissioned the survey, it can influence responses. Their existing feelings and opinions about the brand or organization can taint even the most general questions in the survey. This is particularly troublesome in product surveys. 

How to avoid it: Do not use any logos on the invitation, survey form or any other collateral. Declare that the survey is being moderated independently of any brand or organization.

6. Confirmation Bias

This bias occurs on the researcher side when the survey itself is conducted to confirm a hypothesis, rather than simply gauge opinion. It is particularly common in political circles. Such researchers will give extra weight to responses that confirm their belief, and dismiss evidence to the contrary. In many cases, they will pose leading questions, such as “Don’t you agree that taxes are too high?” rather than a more neutral “Which of the following describes your view on taxes?”  Confirmation bias is a natural human phenomenon, and is not always easy to spot, even within ourselves. It is simply part of the way we process and evaluate information in our daily lives.

How to avoid it: Do not use leading questions. Continually reevaluate responses and challenge them against your preconceptions.

7. Culture Bias

Our own cultural experience influences our thoughts and assumptions about other cultures, which can cause unintended bias in research. This phenomenon, known as ethnocentrism, is defined as “judging another culture solely by the values and standards of one’s own culture.” In some cases, the assumptions made can be downright offensive. 

How to avoid it:  Embrace the principle of cultural relativism – that an individual’s beliefs and activities should be understood and evaluated in terms of that individual’s own culture. Have unconditional positive regard and be mindful of their cultural assumptions, too. 

8. Halo Effect Bias

People have the tendency to hold an overall impression of something based on only one characteristic. This so-called halo effect can introduce bias on both the moderator and respondent side. For example, a moderator may make an assumption about a respondent-based on a first positive impression. A respondent may respond to a series of questions on a brand based solely on their feeling about one attribute.

How to avoid it: Choose question order carefully and stick to one topic at a time. Continually remind yourself why each question is being asked and hold off analysis until later.

While bias is an integral part of the human experience, it can be minimized in research to deliver an honest reflection of the attitudes and behaviors of your target population. By looking out for these common survey bias errors, you will be well on your way to survey success.

For more information on survey bias or any aspect of multi-modal data collection, contact us today!