“How many completed surveys will we need?”
That’s a common question everyone has as they are about to embark upon a new market research project. Whether you are doing market research in Syracuse, market research in Central NY, or global market research – the answer depends on a variety of factors, but in many cases the magic number is 400, as in 400 completes.
There are two predominant reasons 400 is often the number of completes Research & Marketing Strategies (RMS) aims for in market research:
1) Margin of Error. The Research Bunker at RMS always recommends that a survey have a margin of error of +5% or lower at the 95% confidence level. What that means, in plain English, is that 95 out of 100 times the survey is conducted using a proper random sample, the results will yield a value within five points (plus or minus) of the current result. As an example, if 65% of survey respondents prefer Brand X, you can be reasonably sure that the actual preference of the overall population being sampled is somewhere between 60% and 70%. That’s a reasonable leeway for many types of marketing-based decisions. In large populations, 400 completes, or something very close to it, is the number that yields that +5% margin of error.
At this point, you might be thinking that you would like more precision from your survey data. A difference of 5 percentage points either way can have a large impact on decision-making in some situations. Why not collect more surveys and make your data more accurate? That brings me to the second reason that 400 is the magic number in market research.
2) Through a somewhat counter-intuitive quirk of mathematics and sampling theory, increasing the number of completes does not improve the margin of error at the same ratio. Meaning doubling the number of completes will not cut the margin of error in half. The graph below shows how this version of the Law of Diminishing Returns applies to market research.
The graph above shows the margin of error for a survey at various complete thresholds based on a population size of 100,000. As a rule of thumb, the graph above is still applicable as long as the number of completes is less than 5% of the total population, so 100,000 is no different from 1,000,000 in the examples above. Increasing your sample size from 100 to 200 completes lowers the margin of error significantly – almost 3 full percentage points (9.8% to 6.9%). The jump from 200 to 300 yields less of an improvement, but still a difference of over 1 percentage point (6.9% to 5.7%). 400 is the magic number to get us our +5% margin of error. But more importantly, notice what happens after 400 completes – the curve begins to flatten out. At that point, each incremental increase of one-hundred completes yields smaller and smaller improvements on the margin of error. In other words, 400 completes is usually the point that offers the best value, the greatest “bang for the buck” in market research.
However, there are cases where it does make sense to go beyond 400 completes and get something closer to 800 or even 1,000. These situations include the following:
- If a +5% margin of error is simply too broad for the subject at hand. This is usually the case in political polling where political races are often decided by small margins. Political polling usually involves 1,000 completes or more.
- The need for a thorough analysis of sub-populations within the total pool of survey respondents. An example of this would be if you wanted to examine how respondents in different age, gender or ethnic categories answered key questions. To achieve that, you’ll need to increase the number of total completes. For example if you want to create 2 separate marketing campaigns for males and females, you shouldn’t base your decisions on 200 completes from each but rather look to attain 400 completes from each, so each will have its own +5% margin of error.
- If you are in the (enviable) situation where both sample costs and the cost to administer each survey are relatively low. An example of this would be a client who already owns the sample list and is conducting an online survey. In this case, the cost difference between collecting 400 surveys and 4,000 would be be low or even negligible if you sent out a survey of a few questions with no open-ends. Analyzing frequencies of 400 completes is no different from analyzing frequencies of 4,000 completes for an analyst.
If you are looking to conduct a survey with a reliable margin of error – make it a goal to attain 400 completes.
Click here to view a margin of error and sampling size tool provided to you by the Custom Insights website.
[…] and a high level of reliability by county (this is why RMS did not recommend doing 400 completes – click here). The sponsoring hospital was blinded at the beginning of the hospital survey but was then revealed […]
[…] RMS started with a telephone survey to collect a statistically significant sample of patients (approximately 400). The same survey script was used in monthly surveying to a sample of patients from a list […]
[…] one with the highest likelihood to purchase. Obviously in this situation you should stress the margin of error and how the statistics could change over the next 600 completes. The last thing you want to […]
[…] Probability sample methods in most circumstances are the way to go with market research, unless you are hoping to utilize the benefits of a specific non-probability sampling method, or you are limited by things such as budget or access to a full population list. These methods allow the market research to confidently state the results are representative of the population (to a certain margin of error, of course). […]
[…] data is representative of 450 completes or 50 completes. It also gives the clients insight to the margin of error for the particular […]
[…] mean you should trash your own work – only that you should point out things like a survey’s margin of error upfront. Most people will appreciate your candor and become less defensive once they realize […]
[…] read more about why 400 survey completes is considered to be the magic number click here. If you’d like to further discuss how RMS can complement your business through market research […]
The content was instead grabbing and intriguing enough to receive all probable nuances to remember.
I really do get pleasure from studying the content and the composing
manner of the author, etc.. I recommend you write such kinds of posts daily to provide the
audience like me all the necessary information. In my opinion, it’s
far better to be ready for all the unexpected circumstances beforehand,
so thanks, it was rather cool.