Data weighting is a technique that is commonly used in market research. Many people reading this will already know what the concept means. If you’re not one of them, it refers to the practice of adjusting data results to either overcome sampling bias or to give more or less significance to factors based on their estimated relevance to the question at hand. Last year, the excellent Voice of Vovici blog featured this post that provided a good overview of data weighting as it applies to market research, and we recommend it to anyone who wants to learn more on the subject.

Here in the Bunker, we sometimes use weighting in survey analysis, and will continue to do so in the future. But we try to use the technique sparingly and proceed with a lot of caution when we do use it. We suspect that we probably do less weighting than a lot of our market research peers. The reason is that we feel it’s a tool that can easily be overused or misapplied.

Weighting is most effective when you have reliable, precise information about what the actual numbers should look like. A common example we run into in the Bunker is with surveys we do for school districts. There’s a built-in response bias (and to some extent there’s often a sampling bias) in favor of households with school children in such surveys. The response to a school district survey might skew as high as two-thirds parents of students and one-third non-parents, although typically, the districts we work with, the proportions are just the opposite:  about a third of the households will have children who are currently students in the district. In such cases, it’s appropriate to weight the data according to known demographics. But what if you don’t know exactly what the overall population looks like? Any weights you assign will be guesses — educated guesses perhaps — but still subject to the possibility that your estimates are off, which will in turn affect the accuracy of the results. The website of the National Council on Public Polls contains a nice, concise explanation (located under the heading “Bad Weighting”)  on why it’s often very problematic to weight data according to what you assume the overall population looks like.

As mentioned, in a previous paragraph, we think weighting is often overdone. We have seen analyses where certain subgroups are given double weight, or even more. Without getting heavily into the math involved, weighting to that extent increases the degree of error and the significance of the data, especially if the number of respondents involved is relatively small. Personally, I feel uneasy weighting anything by a factor of more than around 1.3 or 1.4.

We try to address this issue by making sure that our sampling isn’t so skewed that such drastic weights are necessary. That approach doesn’t help when the data has already been collected, but it does eliminate the mindset of thinking it’s okay to be sloppy in the project design because you can “fix” the issues later with weighting.

Another issue with weighting is that, if you do it, you need to be prepared to justify and clearly communicate your assumptions to the client and potentially to any constituency they might be sharing the results with. For example, if you have conducted a public opinion survey for a community on issues of a sensitive nature (i.e. a potential school closing, allowing certain large-scale construction, etc.) people will scrutinize your methodology very closely.  If they learn that you have weighted the data to amplify the opinions of certain groups relative to others, controversy is likely to follow. In such cases, it will often be hard to convince people that the weighting is a valid analysis technique. They will see it as unfairly stacking the deck in one side’s favor, and if you haven’t done it properly, they’ll be right. That’s not to say you should never use weighting in that kind of project, just that you’ll really need to make sure you can thoroughly justify it – which is ultimately a good rule of thumb to use any time the question of whether or not to weight data arises.

Have questions about weighting data in market research – click on the ‘Have a Question’ box in the top right corner of the blog or call Research & Marketing Strategies (RMS), a market research vendor, at 315-635-9802.