One of our clients recently came across a blog post written by Sensei Marketing titled “Is the Consumer Survey Dead?” In it, the author, Sam Fiorella, discusses how more insight can be garnered from observing social interactions of consumers with each other rather than speaking to a third-party through traditional market research. Here is a snippet from the blog post:
“Serious question: is the customer survey dead? Outside of linear data collection, I argue that there’s more insight to be gained from what customers say about you to others than what they say about you to you. Collecting survey data is like counting the number of followers a business has on Facebook or Twitter. You’ll add more numbers to your pile of data but have you really learned anything? As a basis for strategic planning, surveys are notoriously fallible because they can be easily skewed to dictate an outcome; surveys rarely include context or external situational factors that more accurately indicate the customer’s opinion and attitude. To even attempt to glean some insights from customer surveys, marketers must consider the data collection method, visual layout, respondent effort requested, question wording, question order, format, structure, behaviors to be measured, etc. Even with all that information, one must accept a margin of error based on the sample size, accuracy of data input, etc.”
Here is our staff’s take on the blog post:
George Kuhn, Director of Research Services at RMS – The blog post seems more of a plug for social analytics and sentiment analysis and less as a dismissal of consumer surveys but the arguments are good concerns. The points about traditional market research are nothing new and have been cautions for years. If anything, it further confirms the need for clients to use a third-party and survey writing expert to mitigate controllable issues such as question wording, question order, format, structure, behavior tested, etc.
A strong concern I have with using market research methodologies and applying them to understand dialogue on message boards and social media is the impact of “influence” variables. Meaning users who have no strong opinion on a product or service one way or another are predisposed and influenced by other users’ comments and ratings. For example, someone visits a page to rate their most recent hotel stay and they want to rate the hotel stay as a perfect 10, sees a comment on the hotel Facebook page that says the “bed was uncomfortable” and thinks, “you know what, it was a bit uncomfortable, so I am going to rate the stay as an 8.” Although this factor was something they never considered as impacting their rating beforehand and it wasn’t in their consideration set, they were influenced by other feedback, which I think is a major concern with social analytics that the author discusses.
If anything, the article supports my growing belief that using only one methodology may not produce truly representative results. It’s important to look at other methodologies (e.g., incorporating observational research or mystery shopping with a survey, or incorporating shop-alongs with a consumer survey) so we can cover all angles.
Chris Coville, Senior Research Associate at RMS – There seems to be a never-ending stream of articles that state “this or that is dead” and jump to a conclusion. Many blog posts and articles always talk about social analytics, but most of this analysis only applies to national or international consumer companies (Coca-Cola, McDonald’s, etc.). For smaller and mid-size market research firms who consult with much smaller clients, we aren’t going to find a whole lot of “conversations” that can be analyzed over social media. You might find an occasional review that can be taken as anecdotal evidence, but nothing that is going to be even close to representative.
Vance Marriner, Senior Research Analyst at RMS – In addition to the points George and Chris raised, I think another concern is how self-selecting online commenters are. There are a lot of people (probably a sizable majority) who would never be inclined to make either a positive or negative statement about a product on social media or a message board. Those who do will tend to be at the extremes – the very dissatisfied and the product cheerleaders. Meanwhile the silent majority goes un-analyzed because no one is actively engaging them. There has always been a concern with traditional surveys that the results tend to over-exaggerate the two extremes, but that concern would be much, much greater with social analytics.
Another concern would be those types of analytics are more open to manipulation from companies who tamper with online message boards and social media by posting positive or negative reviews online. The more it catches on, the more incentive there will be to cheat. Consumer surveys that use proper sampling techniques don’t have that vulnerability.
As Chris said, there’s a tendency in marketing to always declare the old approach dead while promoting the new approach as the end-all, be-all answer to everything. In reality, it almost never works out like that. Surveys will evolve and we may do less of them over time, but there will always be a place for them. Social analytics are useful, but they are just one more specialized tool to add to the market research toolbox. And just to stretch the toolbox analogy to its breaking point, just because we now have a shiny new social analytics ratchet set, that doesn’t mean we can or should throw away the hammer, screwdriver, pliers, et. al.
Paul Dybas, former Business Development Specialist at RMS – I agree with George’s point about confirming the need to use a third-party. Survey results are not the final product of market research. The report is where we really get into the analysis of the data – weighting the responses, accounting for market fluctuations and other contextual factors. This is in addition to the question wording, order, etc. that we already pay close attention to. Also, the author reiterates how crucial it is for a research vendor to have very strong business ethics and remain independent so that we are not trying to “dictate the outcome.”
What are your thoughts on traditional market research? Is it dead or evolving? We welcome your comments below! Research & Marketing Strategies (RMS) is a market research firm located in Syracuse, NY. If you would like more information about us please contact our Business Development Director, Sandy Baker, at SandyB@RMSresults.com or by calling (315) 635-9802.