In questionnaire design text, it is both important that question-wording should express what is being measured, and that appropriate response options are provided. However, there are times the latter is overlooked, and a mismatch occurs between the question wording and response options. So, the big question is, how detrimental are these mismatches? Below, we discuss the effects of badly worded survey questions (how mismatches between survey questions and response options affect a study). We hope this information will help you weigh whether it is more critical for questionnaires to firmly match survey wording, or give more significance to more straightforward and less wordy options.
Common Sample of Mismatched Questions
Effect of Mismatched Questions
- Higher Item Nonresponse
Mismatched questions will result in higher item nonresponse because of the difficulty in responding due to additional cognitive processing. In responding to survey questions, respondents must identify the question, understand it, recover relevant information, create a judgment, and then report (Jenkins and Dillman 1997; Tourangeau, Rips, and Rasinski 2000). Consequently, there is a probability some respondents will skip items, or refuse to answer when they are troubled to exert additional intellectual effort. In an experiment by Dillman et al. (2014), they found that the mismatched version caused greater item nonresponse, and differences in response distributions, compared to the matched version. The study varied the question stems paired with forced-choice response options to be matched or mismatched with check-all question stems.
- Longer Response Time
Mismatched versions will take longer to finish compared to matched versions because of the added time-consuming comprehension and mapping complications. Additionally, the mismatch may only be identified by the respondents when they are mapping their answer to the response options and determine they are not appropriate. In this case, they have to exert additional effort in finding an adequate answer in the response option to replace their answer. Van der Zouwen and Dijkstra (2002) determined that mismatches or poorly designed response options (inadequate range of response alternatives) were greatly connected with swaying from the classic interviewer/respondent interaction in a question/answer sequence. An example of a classic interaction is an interviewer asking exactly worded questions to respondents, and receiving an adequate answer from them. Mismatch tends to result in respondents asking the interviewer to clarify their question further, resulting in longer survey completion time.
Confirmation by a Recent Study of the University of Nebraska-Lincoln: “The Effects of Mismatches Between Survey Question Stems and Response Options on Data Quality and Responses”
The experiment divided the respondents into two groups. The first group received a survey version with all answer options and response options matched, and the 2nd group a slightly mismatched version. Their theory was that mismatched questions would result in higher item nonresponse and longer response time.
Example Matched and Mismatched Survey Questions from the Study
Results of the Study
The item nonresponse rate is higher in the mismatched than in the matched version. The probability of an item being unanswered in the mismatched version compared to the matched version is 1.6 times.
Response time was longer in the mismatched than in the matched version. During the survey conducted via telephone interview, the researchers found that the interviewees took more time to complete the mismatched version, compared to the correctly matched one.
The effect of the mismatch is not greater for those with lower cognitive abilities based on age or education.
The study confirmed that mismatches weaken data quality in both mail and telephone surveys. Although, it may be less detrimental for the mail survey because it is self-administered, and respondents can see both question stem and response options. Researchers should design their questionnaires carefully, taking into consideration the proper relationship of the question stem and response option. Moreover, in analyzing survey results, researchers should review both stem questions and response options to determine if a mismatch could have made an impact on the results.