One of our clients recently came across a blog post written by Sensei Marketing titled “Is the Consumer Survey Dead?” In it, the author, Sam Fiorella, discusses how more insight can be garnered from observing social interactions of consumers with each other rather than speaking to a third-party through traditional market research. Here is a snippet from the blog post:
“Serious question: is the customer survey dead? Outside of linear data collection, I argue that there’s more insight to be gained from what customers say about you to others than what they say about you to you. Collecting survey data is like counting the number of followers a business has on Facebook or Twitter. You’ll add more numbers to your pile of data but have you really learned anything? As a basis for strategic planning, surveys are notoriously fallible because they can be easily skewed to dictate an outcome; surveys rarely include context or external situational factors that more accurately indicate the customer’s opinion and attitude. To even attempt to glean some insights from customer surveys, marketers must consider the data collection method, visual layout, respondent effort requested, question wording, question order, format, structure, behaviors to be measured, etc. Even with all that information, one must accept a margin of error based on the sample size, accuracy of data input, etc.”
Here is our staff’s take on the blog post:
George Kuhn, Director of Research Services at RMS – The blog post seems more of a plug for social analytics and sentiment analysis and less as a dismissal of consumer surveys but the arguments are good concerns. The points about traditional market research are nothing new and have been cautions for years. If anything, it further confirms the need for clients to use a third-party and survey writing expert to mitigate controllable issues such as question wording, question order, format, structure, behavior tested, etc.
A strong concern I have with using market research methodologies and applying them to understand dialogue on message boards and social media is the impact of “influence” variables. Meaning users who have no strong opinion on a product or service one way or another are predisposed and influenced by other users’ comments and ratings. For example, someone visits a page to rate their most recent hotel stay and they want to rate the hotel stay as a perfect 10, sees a comment on the hotel Facebook page that says the “bed was uncomfortable” and thinks, “you know what, it was a bit uncomfortable, so I am going to rate the stay as an 8.” Although this factor was something they never considered as impacting their rating beforehand and it wasn’t in their consideration set, they were influenced by other feedback, which I think is a major concern with social analytics that the author discusses.
If anything, the article supports my growing belief that using only one methodology may not produce truly representative results. It’s important to look at other methodologies (e.g., incorporating observational research or mystery shopping with a survey, or incorporating shop-alongs with a consumer survey) so we can cover all angles.
Chris Coville, Senior Research Associate at RMS – There seems to be a never-ending stream of articles that state “this or that is dead” and jump to a conclusion. Many blog posts and articles always talk about social analytics, but most of this analysis only applies to national or international consumer companies (Coca-Cola, McDonald’s, etc.). For smaller and mid-size market research firms who consult with much smaller clients, we aren’t going to find a whole lot of “conversations” that can be analyzed over social media. You might find an occasional review that can be taken as anecdotal evidence, but nothing that is going to be even close to representative.
Vance Marriner, Senior Research Analyst at RMS – In addition to the points George and Chris raised, I think another concern is how self-selecting online commenters are. There are a lot of people (probably a sizable majority) who would never be inclined to make either a positive or negative statement about a product on social media or a message board. Those who do will tend to be at the extremes – the very dissatisfied and the product cheerleaders. Meanwhile the silent majority goes un-analyzed because no one is actively engaging them. There has always been a concern with traditional surveys that the results tend to over-exaggerate the two extremes, but that concern would be much, much greater with social analytics.
Another concern would be those types of analytics are more open to manipulation from companies who tamper with online message boards and social media by posting positive or negative reviews online. The more it catches on, the more incentive there will be to cheat. Consumer surveys that use proper sampling techniques don’t have that vulnerability.
As Chris said, there’s a tendency in marketing to always declare the old approach dead while promoting the new approach as the end-all, be-all answer to everything. In reality, it almost never works out like that. Surveys will evolve and we may do less of them over time, but there will always be a place for them. Social analytics are useful, but they are just one more specialized tool to add to the market research toolbox. And just to stretch the toolbox analogy to its breaking point, just because we now have a shiny new social analytics ratchet set, that doesn’t mean we can or should throw away the hammer, screwdriver, pliers, et. al.
Paul Dybas, former Business Development Specialist at RMS – I agree with George’s point about confirming the need to use a third-party. Survey results are not the final product of market research. The report is where we really get into the analysis of the data – weighting the responses, accounting for market fluctuations and other contextual factors. This is in addition to the question wording, order, etc. that we already pay close attention to. Also, the author reiterates how crucial it is for a research vendor to have very strong business ethics and remain independent so that we are not trying to “dictate the outcome.”
What are your thoughts on traditional market research? Is it dead or evolving? We welcome your comments below! Research & Marketing Strategies (RMS) is a market research firm located in Syracuse, NY. If you would like more information about us please contact our Business Development Director, Sandy Baker, at SandyB@RMSresults.com or by calling (315) 635-9802.
There is certainly value in doing surveys when you bring nonspecific questions to an audience – they over emphasize their memories in a biased but leveragable way.
You can simplify research for general sizing, such as by looking for counts of brand reference + count of positive/negative comments.
Social media is much more beneficial and honest if #1: you choose a discussion point worthy of a long term web-string conversation; #2: get people to choose sides to create friendly completion; # CARROT.
It seems like the real question is about quantitative research as qualitative research is used more today than ever. Traditional quantitative research relies on stated behaviors and we rely on the consistency of response among known archetypes to manage bias and attribute behaviors. The “exactness” of the study constructs provides very succinct answers but it also increases the margin of error by eliminating variables that were considered to be too small for measurement. The problem with that type of solution is that the number of indistinct variables is large. Today, in many cases, they account for more influence collectively than the so-called major variables.
Analysis of big data provides very discreet measurements of actual purchase regardless of stimuli. It negates the value of questions that were once the bread and butter of quantitative studies and it brings into focus the impact of all variables. What were are discovering is the variance between stated behavior and actual behavior is much larger than anticipated. That begs the question of why, and introduces the concept of behavioral measurement. Those are not the types of questions than can be easily mapped back to a tidy data set. While lemmings still exist, there are a lot more herons and they don’t exhibit collective behavior.
Because much of the data collection is done in real time, the analysis relies on continual data collection and variances within a time line rather than a shift in pattern. What does remain constant is the need for product evolution. There is still a great opportunity for research to be done in that arena and people are responsive to product innovation, development and attribute measurement. Those things are inherently quantifiable and self-selection is not an issue because it shows engagement. Most people take surveys in order to have their opinions count, not have neurological testing done so its a win/win for everybody. If the focus is reflective of “better mouse traps” people are more willing to engage. Enjoining qualitative assessments with short quantitative product surveys creates a more productive environment.
[…] articles and add a fresh perspective, sit down with a colleague and talk out loud about a topic and recap your conversation on the blog. Some of our most highly viewed blogs took less than 30 minutes to put together and post. […]