We have a lot of discussions with clients about statistical significance. Some clients want to place too much importance on statistical significance, others too little.
The basic motivation behind statistical testing is our desire to make statistical inferences about differences in results, based on samples (sometimes we see clients applying statistical significance testing to population data e.g. total website visits in two different months – not appropriate!), over time, between different subgroups and the like.
A basic tenet of statistical inference is that it is possible for numbers to be different in a mathematical sense but not significantly different in a statistical sense. For example, a sample of patients is asked to rate their level of satisfaction with their primary care physician. The physician receives an 8.2 average rating. The average for all physicians tested is 8.4. There is a mathematical difference between the two numbers, but is the difference significant in a “statistical sense” and is it managerially important? Three different concepts can be applied to the notion of differences.

  • Mathematical differences. By definition, if numbers are not exactly the same, they are different. This does not, however, mean that the difference is either important or statistically significant.
  • Statistical significance. If a particular difference is so large that it is unlikely to have occurred due to chance or sampling error, then the difference is statistically significant. It should also be noted that we can never say that two numbers are significantly different with absolute certainty. We always test at some level of confidence — 95% is typical. If results are significantly different at this level then we can say we are 95% confident that they are really different. But, there is a 5% chance that they are not and that the difference we observe is really due to sampling error.
  • Managerially important differences. If results are different to the extent that the difference matters from a managerial perspective, we can argue that the difference is important. For example, the difference in satisfaction scores for two health plans might be statistically significant but yet so small as to be of little practical or managerial significance.

Ultimately, from a business and practical perspective, differences that are managerially important are those that should drive decision-making. We hope that you find this overview helpful and encourage you to keep this tip handy for future reference.