One thing you must always think about when developing a survey is to encompass all possible responses; this means conducting secondary research (even some qualitative research) to ensure that you are covering everything. This is more commonly referred to as collectively exhaustive in market research terms, part of the MECE principle. The results of not taking the time to ensure your selected responses question covers the entire realm of possible answers could be devastating to your research.
Let’s say a quick service restaurant chain is conducting market research with their their customers regarding satisfaction. A portion of the survey includes a satisfaction grid series. The factors are measured on a 1-10 scale (“1” being not at all satisfied and “10” being very satisfied). The series asks to rate the following factors of customer satisfaction (we’ve attached the theoretical mean scores on the 1-10 scale):
- Restaurant Staff (5.2)
- Accuracy of the order (9.5)
- Quality of the food (9.5)
- Cleanliness of the restaurant (9.9)
- Overall Satisfaction (8.1)
When analyzing the results the restaurant’s management team is quite surprised to find that people are satisfied with all of the categories except for the restaurant’s staff. Upon conducting follow-up research to learn more about why the respondents were so dissatisfied with the staff, they come to find that the staff was actually quite friendly and helpful. It turns out the real problem was with the wait time. By leaving out the category, the respondents took out their frustration with the wait time on the closest applicable category, the restaurant staff. As you can see, this original information can be quite misleading.
While there are ways to try and get around this issue, like including open-ended questions for respondents to take out their grievances or other concerns, it’s best to make sure the satisfaction series is all-encompassing. And by this, in no way do I mean creating a 45 question satisfaction series (where your respondent falls asleep, even while in the process of straightlining their answers).
This same line of thinking also applies to multiple choice questions. If somebody doesn’t find a response that exactly matches what they were hoping for, they will select the closest option, which will result in inaccurate data. This is why the “Other, please specify:” category is so important. If you insist on limiting the respondents to a few specific answers, use the appropriate wording and go in knowing that they are selecting the closest fit (ie: Which category below best fits?).
Taking the time to develop and think through every aspect of a survey is absolutely crucial. It can’t be said enough: in order to get quality and precise data on the back-end, you have to put in the work on the front-end of market research design.
