“How many completed surveys will we need?”

That’s a common question everyone has as they are about to embark upon a new market research project. Whether you are doing market research in Syracuse, market research in Central NY, or global market research – the answer depends on a variety of factors, but in many cases the magic number is 400, as in 400 completes.

There are two predominant reasons 400 is often the number of completes Research & Marketing Strategies (RMS) aims for in market research:

1) Margin of Error. The Research Bunker at RMS always recommends that a survey have a margin of error of +5% or lower at the 95% confidence level. What that means, in plain English, is that 95 out of 100 times the survey is conducted using a proper random sample, the results will yield a value within five points (plus or minus) of the current result.  As an example, if 65% of survey respondents prefer Brand X, you can be reasonably sure that the actual preference of the overall population being sampled is somewhere between 60% and 70%. That’s a reasonable leeway for many types of marketing-based decisions. In large populations, 400 completes, or something very close to it, is the number that yields that +5% margin of error.

At this point, you might be thinking that you would like more precision from your survey data. A difference of 5 percentage  points either way can have a large impact on decision-making in some situations. Why not collect more surveys and make your data more accurate? That brings me to the second reason that 400 is the magic number in market research.

2) Through a somewhat counter-intuitive quirk of mathematics and sampling theory, increasing the number of completes does not improve the margin of error at the same ratio. Meaning doubling the number of completes will not cut the margin of error in half. The graph below shows how this version of the Law of Diminishing Returns applies to market research.

The graph above shows the margin of error for a survey at various complete thresholds based on a population size of 100,000.  As a rule of thumb, the graph above is still applicable as long as the number of completes is less than 5% of the total population, so 100,000 is no different from 1,000,000 in the examples above. Increasing your sample size from 100 to 200 completes lowers the margin of error significantly – almost 3 full percentage points (9.8% to 6.9%). The jump from 200 to 300 yields less of an improvement, but still a difference of over 1 percentage point (6.9% to 5.7%). 400 is the magic number to get us our +5% margin of error. But more importantly, notice what happens after 400 completes – the curve begins to flatten out. At that point, each incremental increase of one-hundred completes yields smaller and smaller improvements on the margin of error. In other words, 400 completes is usually the point that offers the best value, the greatest “bang for the buck” in market research.

However, there are cases where it does make sense to go beyond 400 completes and get something closer to 800 or even 1,000. These situations include the following:

•  If a +5% margin of error is simply too broad for the subject at hand. This is usually the case in political polling where political races are often decided by small margins.  Political polling usually involves 1,000 completes or more.
•  The need for a thorough analysis of sub-populations within the total pool of survey respondents. An example of this would be if you wanted to examine how respondents in different age, gender or ethnic categories answered key questions. To achieve that, you’ll need to increase the number of total completes.  For example if you want to create 2 separate marketing campaigns for males and females, you shouldn’t base your decisions on 200 completes from each but rather look to attain 400 completes from each, so each will have its own +5% margin of error.
• If you are in the (enviable) situation where both sample costs and the cost to administer each survey are relatively low. An example of this would be a client who already owns the sample list and is conducting an online survey. In this case, the cost difference between collecting 400 surveys and 4,000 would be be low or even negligible if you sent out a survey of a few questions with no open-ends.  Analyzing frequencies of 400 completes is no different from analyzing frequencies of 4,000 completes for an analyst.

If you are looking to conduct a survey with a reliable margin of error – make it a goal to attain 400 completes.

Click here to view a margin of error and sampling size tool provided to you by the Custom Insights website.