It’s Day 4 of scale week here at the Bunker.  Today, we put together some scaling errors and ways in which scales should not be used.  These problems can lead to biased results, unintentional participant error and/or improper analysis.  Here are a few things that should be avoided if you, as a market researcher, are looking for accurate and representative results.

1) Biased/Weighted Scales

As you can see in the above scale, the level of agreement is covering more than half the scale making it weighted towards agreement.  Not only does offering more options for agreement than disagreement make the scale biased, but it creates the potential for unintentional respondent error.  People typically tend to view scales as having 2 extremes, with the middle being neutral.  The analysis that is performed could also be misrepresentative as a result.  If every respondent selected the middle of the scale, one might state 100% of the respondents are in agreement with the new ice cream being delicious.

2) Reversing Scale Order within a Survey

Reversing scale order within a survey typically won’t be as blatantly obvious as the example above, but it does happen.  As seen in the example, it’s likely that the respondent agreed with both statements but misread the scales or assumed the scales were identical.  Reversing scale order within a survey can create confusion among the respondents; one might compare it to a cereal box game in which you attempt to spot the difference between two pictures.  Some might argue that reversing scale orders can be used in an attempt to combat straight-lining of answers, but also, it can create unintentional respondent error among those who are trying to legitimately answer questions.   There are various techniques that should be used to prevent straight-lining, but this is not one of them.  Click here to read our post that goes into further detail about data quality/integrity.

3) Using Different Point Scales (5 pt, 7 pt, 10 pt)

This scaling issue may not have as much of an effect as others, but it is important to keep all scales identical within a survey.  Respondents put themselves in a state of mind where they are evaluating in terms of a certain point on a scale.  By jumping between different 5, 7, and 10 point scales, it makes it difficult for the respondent to rate statements or questions using consistent judgment.  The other problem with using different scales within a survey is that they are difficult to compare when it comes time for analysis.

In order gather quality data, it is important to use market research scales properly.  Throughout the survey it is important that all the scaling questions be uniform to reduce bias and other problematic errors.  Also, like other questions in market research, they should be mutually exclusive and collectively exhaustive by not doubling up/overlapping categories and by covering all answers.  Avoiding these problems will only benefit your analysis in the end.

• Click Here to view the Day 1 post on Constant Sum Scaling.
• Click Here to view the Day 2 post on Semantic Differential Scaling.
• Click Here to view the Day 3 post on Likert Scales.
• Click Here to view the Day 5 post on 6 Tips for Using Rating Scales.