Statistical Significance

Refers to whether the results of an experiment or the observations from a collected set of data are due to chance

What is Statistical Significance?

Statistical significance refers to whether the results of an experiment or the observations from a collected set of data are due to chance. Researchers conduct hypothesis testing to determine statistical significance (or the lack thereof).

 

Statistical Significance

 

Financial analysts often construct models, into which they input data, to determine what actions a business takes make a statistically significant difference. For example, a business might examine the number of sales leads generated from two different advertisements to see if a change in their advertising provides a significantly higher number of positive responses from potential customers.

 

Summary

  • Statistical significance refers to whether or not the variations in a set of collected data are due merely to random chance or a significant factor other than chance.
  • Researchers commonly conduct hypothesis testing to determine whether their theory is valid.
  • Companies test various changes in their methods of operation to make decisions about the most effective way to market their products or services.

 

Hypothesis Testing

Statistical significance is commonly determined through hypothesis testing. The hypothesis tested is a researcher’s theory or belief about something prior to testing out their theory – it is referred to as the alternative hypothesis.

The alternative hypothesis stands in contrast to what is called the null hypothesis. If testing shows that the researcher’s belief about something is true, then the alternative hypothesis is validated. If instead, testing disproves the researcher’s belief, then the null hypothesis is validated.

Integral to hypothesis testing is something known as the “p-value.” The p-value indicates what the probability is that the data observed in testing a hypothesis is in line with the results of random chance.

The statistically significant range of possible p-values is determined by the researcher. Typically, a p-value of 5% or less indicates statistical significance, validating the alternative hypothesis. In practical English, such a p-value indicates that the observed data demonstrates only a 5% (or lower) probability of being due solely to chance.

 

Statistical Significance - Hypothesis Testing

 

Practical Example

Consider a company that wants to test the idea that its stock being mentioned on a certain business television show attracts a significant number of new investors. It may then construct an experiment where they arrange to get their stock mentioned on the show every other Tuesday for three months. The company can then compare the number of new investors in their stock on the Tuesdays when their stock is mentioned on the show with the number of new investors on Tuesdays when their stock is not mentioned.

The number of new investors on the Tuesdays when their stock is not mentioned would be taken as the baseline average of new investors. If then, the average number of new investors on the Tuesdays when their stock is mentioned on the show is substantially higher than the baseline average, the company can conclude that the result of their stock being mentioned on the show is statistically significant and that it is to the company’s advantage to arrange for it to be mentioned.

 

Statistical Significance in Business

All types of businesses commonly conduct experiments to determine statistical significance. One of the most obvious instances is the actions of pharmaceutical companies in drug testing. All drugs are tested to confirm whether someone taking the drug possesses a statistically significant effect in terms of how it affects the medical condition for which the drug is to be prescribed.

Companies that conduct business online often test for statistically significant results from variations in advertising. The variations tested might be the content of ads, where online the ads are run, what time of day the ads are run, etc. Such testing enables a company to construct and deliver their advertising in a manner that is most effective and efficient in generating sales.

Price variations are another area where businesses often conduct research to see whether there is a significant impact on their sales revenues as a result of changing the price of their product. Such type of research helps the business determine the optimal price level that generates the maximum amount of revenue and/or profit.

Overall, testing for statistical significance helps the executives of a company to make wise decisions concerning how they operate their business and market their products or services.

 

Keep Learning

CFI is the official provider of the global Certified Banking & Credit Analyst (CBCA)™ certification program, designed to help anyone become a world-class financial analyst. To keep advancing your career, the additional resources below will be useful:

  • Basic Statistics Concepts in Finance
  • Empirical Evidence
  • P-Value
  • Total Probability Rule

Financial Analyst Certification

Become a certified Financial Modeling and Valuation Analyst (FMVA)® by completing CFI’s online financial modeling classes and training program!