We have to draw the line somewhere<\/h4>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2A7ZkV0yoRXok1o7McHSi-FA.jpeg?ssl=1\")
Photo by Siora Photography<\/a> on\u00a0Unsplash<\/a><\/figcaption><\/figure>\nIt\u2019s become something of a meme that statistical significance is a bad standard. Several recent blogs have made the rounds, making the case that statistical significance is a \u201ccult\u201d or \u201carbitrary.\u201d If you\u2019d like a classic polemic (and who wouldn\u2019t?), check out: https:\/\/www.deirdremccloskey.com\/docs\/jsm.pdf<\/a>.<\/p>\nThis little essay is a defense of the so-called Cult of Statistical Significance.<\/p>\n
Statistical significance is a good enough idea, and I\u2019ve yet to see anything fundamentally better or practical enough to use in industry.<\/p>\n
I won\u2019t argue that statistical significance is the perfect<\/em> way to make decisions, but it is\u00a0fine<\/em>.<\/p>\nStatistical significance does not equal business significance\u200a\u2014\u200abut so\u00a0what?<\/h3>\n
A common point made by those who would besmirch the Cult is that statistical significance is not the same as business significance. They are correct, but it\u2019s not an argument to avoid statistical significance when making decisions.<\/p>\n
Statistical significance says, for example, that if the estimated impact of some change is 1% with a standard error of 0.25%, it is statistically significant (at the 5% level), while if the estimated impact of another change is 10% with a standard error of 6%, it is statistically insignificant (at the 5%\u00a0level).<\/p>\n
The argument goes that the 10% impact is more meaningful to the business, even if it is less\u00a0precise.<\/p>\n
Well, let\u2019s look at this from the perspective of decision-making.<\/em><\/p>\nThere are two cases\u00a0here.<\/p>\n
The two initiatives are separable.<\/h4>\n
If the two initiatives are separable, we should still launch the 1% with a 0.25% standard error\u200a\u2014\u200aright? It\u2019s a positive effect, so statistical significance does not lead us astray. We should launch the stat sig positive\u00a0result.<\/p>\n
Okay, so let\u2019s turn to the larger effect size experiment.<\/p>\n
Suppose the effect size was +10% with a standard error of 20%, i.e., the 95% confidence interval was roughly [-30%, +50%]. In this case, we don\u2019t really think there\u2019s any evidence the effect is positive, right? Despite the larger effect size, the standard error is too large to draw any meaningful conclusion.<\/p>\n
The problem isn\u2019t statistical significance. The problem is that we think a standard error of 6% is small enough in this case to launch the new feature based on this evidence. This example doesn\u2019t show a problem with statistical significance as a framework. It shows we are less worried about Type 1 error than alpha =\u00a05%.<\/p>\n
That\u2019s fine! We accept other alphas in our Cult, so long as they were selected before the experiment. Just use a larger alpha. For example, this is statistically significant with alpha =\u00a010%.<\/p>\n
The point is that there is <\/em>a level of noise that we\u2019d find unacceptable. There\u2019s a level of noise where even if the estimated effect were +20%, we\u2019d say, \u201cWe don\u2019t really know what it\u00a0is.\u201d<\/p>\nSo, we have to say how much noise is too\u00a0much.<\/p>\n
Statistical inference, like art and morality, requires us to draw the line somewhere.<\/p>\n
The initiatives are alternatives.<\/h4>\n
Now, suppose the two initiatives are alternatives. If we do one, we can\u2019t do the other. Which should we\u00a0choose?<\/p>\n
In this case, the problem with the above setup is that we\u2019re testing the wrong hypothesis. We don\u2019t just want to compare these initiatives to control. We also want to compare them to each\u00a0other.<\/p>\n
But this is also not a problem with statistical significance. It\u2019s a problem with the hypothesis we\u2019re\u00a0testing.<\/p>\n
We want to test whether the 9% difference in effect sizes is statistically significant, using an alpha level that makes sense for the same reason as in the previous case. There\u2019s a level of noise at which the 9% is just spurious, and we have to set that\u00a0level.<\/p>\n
Again, we have to draw the line somewhere.<\/p>\n
Now, let\u2019s deal with some other common objections, and then I\u2019ll pass out a sign-up sheet to join the\u00a0Cult.<\/p>\n
Statistical significance is arbitrary.<\/h3>\n
This objection to statistical significance is common but misses the\u00a0point.<\/p>\n
Our attitudes towards risk and ambiguity (in the Statistical Decision Theory sense) are \u201carbitrary\u201d because we choose them. But there isn\u2019t any solution to that. Preferences are a given in any decision-making problem.<\/p>\n
Statistical significance is no more \u201carbitrary\u201d than other decision-making rules, and it has the nice intuition of trading off how much noise we\u2019ll allow versus effect size. It has a simple scalar parameter that we can adjust to prefer more or less Type 1 error relative to Type 2 error. It\u2019s\u00a0lovely.<\/p>\n
People misinterpret frequentist p-values to indicate the probability that the effect is zero. To avoid this mistake, they should use Bayesian inference.<\/h3>\n
Sometimes, people argue that we should use Bayesian inference to make decisions because it is easier to interpret.<\/p>\n
I\u2019ll start by admitting that in its ideal setting, Bayesian inference has nice properties. We can take the posterior and treat it exactly like \u201cbeliefs\u201d and make decisions based on, say, the probability the effect is positive, which is not possible with frequentist statistical significance.<\/p>\n
Bayesian inference in practice is another\u00a0animal.<\/p>\n
Bayesian inference only gets those nice \u201cbelief\u201d-like properties if the prior reflects the decision-maker\u2019s actual prior beliefs. This is extremely difficult to do in practice.<\/p>\n
If you think choosing an \u201calpha\u201d that draws the line on how much noise you\u2019ll accept is tricky, imagine having to choose a density<\/em> that correctly captures your\u200a\u2014\u200aor the decision-maker\u2019s\u200a\u2014\u200abeliefs\u2026 <\/em>before every experiment! This is a very difficult problem.<\/p>\nSo, the Bayesian priors selected in practice are usually chosen because they are \u201cconvenient,\u201d \u201cuninformative,\u201d etc. They have little to do with actual prior\u00a0beliefs.<\/p>\n
When we\u2019re not specifying our real prior beliefs, the posterior distribution is just some weighting of the likelihood function. Claiming that we can look at the quantiles of this so-called posterior distribution and say the parameter has a 10% chance of being less than 0 is nonsense statistically.<\/p>\n
So, if anything, it is easier to misinterpret what we\u2019re doing in Bayesian land than in frequentist land. It is hard for statisticians to translate their prior beliefs into a distribution. How much harder is it for whoever the actual decision-maker is on the\u00a0project?<\/p>\n
For these reasons, Bayesian inference doesn\u2019t scale well, which is why, I think, Experimentation Platforms across the industry generally don\u2019t use\u00a0it.<\/p>\n
The Church of Statistical Significance<\/h3>\n
The arguments against the \u201cCult\u201d of Statistical Significance are, of course, a response to a real problem. There is<\/em> a dangerous Cult within our\u00a0Church<\/em>.<\/p>\nThe Church of Statistical Significance is quite accepting. We allow for other alpha\u2019s besides 5%. We choose hypotheses that don\u2019t test against zero nulls,\u00a0etc.<\/p>\n
But sometimes, our good name is tarnished by a radical element within the Church that treats anything insignificant versus a null hypothesis of 0 at the 5% level as \u201cnot\u00a0real.\u201d<\/p>\n
These heretics believe in a cargo-cult version of statistical analysis where the statistical significance procedure (at the 5% level) determines what is true instead of just being a useful way to make decisions and weigh uncertainty.<\/p>\n
We disavow all association with this dangerous sect, of\u00a0course.<\/p>\n
Let me know if you\u2019d like to join the Church. I\u2019ll sign you up for the monthly\u00a0potluck.<\/p>\n
Thanks for\u00a0reading!<\/p>\n
Zach<\/p>\n
Connect at: https:\/\/linkedin.com\/in\/zlflynn<\/a><\/p>\n![\"\"](\"https:\/\/medium.com\/_\/stat?event=post.clientViewed&referrerSource=full_rss&postId=996e8c62f3e8\")
<\/p>\n
\nIn Defense of Statistical Significance<\/a> was originally published in Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\n
It\u2019s become something of a meme that statistical significance is a bad standard. Several recent blogs have made the rounds, making the case that statistical significance is a \u201ccult\u201d or \u201carbitrary.\u201d If you\u2019d like a classic polemic (and who wouldn\u2019t?), check out: https:\/\/www.deirdremccloskey.com\/docs\/jsm.pdf<\/a>.<\/p>\n This little essay is a defense of the so-called Cult of Statistical Significance.<\/p>\n Statistical significance is a good enough idea, and I\u2019ve yet to see anything fundamentally better or practical enough to use in industry.<\/p>\n I won\u2019t argue that statistical significance is the perfect<\/em> way to make decisions, but it is\u00a0fine<\/em>.<\/p>\n A common point made by those who would besmirch the Cult is that statistical significance is not the same as business significance. They are correct, but it\u2019s not an argument to avoid statistical significance when making decisions.<\/p>\n Statistical significance says, for example, that if the estimated impact of some change is 1% with a standard error of 0.25%, it is statistically significant (at the 5% level), while if the estimated impact of another change is 10% with a standard error of 6%, it is statistically insignificant (at the 5%\u00a0level).<\/p>\n The argument goes that the 10% impact is more meaningful to the business, even if it is less\u00a0precise.<\/p>\n Well, let\u2019s look at this from the perspective of decision-making.<\/em><\/p>\n There are two cases\u00a0here.<\/p>\n If the two initiatives are separable, we should still launch the 1% with a 0.25% standard error\u200a\u2014\u200aright? It\u2019s a positive effect, so statistical significance does not lead us astray. We should launch the stat sig positive\u00a0result.<\/p>\n Okay, so let\u2019s turn to the larger effect size experiment.<\/p>\n Suppose the effect size was +10% with a standard error of 20%, i.e., the 95% confidence interval was roughly [-30%, +50%]. In this case, we don\u2019t really think there\u2019s any evidence the effect is positive, right? Despite the larger effect size, the standard error is too large to draw any meaningful conclusion.<\/p>\n The problem isn\u2019t statistical significance. The problem is that we think a standard error of 6% is small enough in this case to launch the new feature based on this evidence. This example doesn\u2019t show a problem with statistical significance as a framework. It shows we are less worried about Type 1 error than alpha =\u00a05%.<\/p>\n That\u2019s fine! We accept other alphas in our Cult, so long as they were selected before the experiment. Just use a larger alpha. For example, this is statistically significant with alpha =\u00a010%.<\/p>\n The point is that there is <\/em>a level of noise that we\u2019d find unacceptable. There\u2019s a level of noise where even if the estimated effect were +20%, we\u2019d say, \u201cWe don\u2019t really know what it\u00a0is.\u201d<\/p>\n So, we have to say how much noise is too\u00a0much.<\/p>\n Statistical inference, like art and morality, requires us to draw the line somewhere.<\/p>\n Now, suppose the two initiatives are alternatives. If we do one, we can\u2019t do the other. Which should we\u00a0choose?<\/p>\n In this case, the problem with the above setup is that we\u2019re testing the wrong hypothesis. We don\u2019t just want to compare these initiatives to control. We also want to compare them to each\u00a0other.<\/p>\n But this is also not a problem with statistical significance. It\u2019s a problem with the hypothesis we\u2019re\u00a0testing.<\/p>\n We want to test whether the 9% difference in effect sizes is statistically significant, using an alpha level that makes sense for the same reason as in the previous case. There\u2019s a level of noise at which the 9% is just spurious, and we have to set that\u00a0level.<\/p>\n Again, we have to draw the line somewhere.<\/p>\n Now, let\u2019s deal with some other common objections, and then I\u2019ll pass out a sign-up sheet to join the\u00a0Cult.<\/p>\n This objection to statistical significance is common but misses the\u00a0point.<\/p>\n Our attitudes towards risk and ambiguity (in the Statistical Decision Theory sense) are \u201carbitrary\u201d because we choose them. But there isn\u2019t any solution to that. Preferences are a given in any decision-making problem.<\/p>\n Statistical significance is no more \u201carbitrary\u201d than other decision-making rules, and it has the nice intuition of trading off how much noise we\u2019ll allow versus effect size. It has a simple scalar parameter that we can adjust to prefer more or less Type 1 error relative to Type 2 error. It\u2019s\u00a0lovely.<\/p>\n Sometimes, people argue that we should use Bayesian inference to make decisions because it is easier to interpret.<\/p>\n I\u2019ll start by admitting that in its ideal setting, Bayesian inference has nice properties. We can take the posterior and treat it exactly like \u201cbeliefs\u201d and make decisions based on, say, the probability the effect is positive, which is not possible with frequentist statistical significance.<\/p>\n Bayesian inference in practice is another\u00a0animal.<\/p>\n Bayesian inference only gets those nice \u201cbelief\u201d-like properties if the prior reflects the decision-maker\u2019s actual prior beliefs. This is extremely difficult to do in practice.<\/p>\n If you think choosing an \u201calpha\u201d that draws the line on how much noise you\u2019ll accept is tricky, imagine having to choose a density<\/em> that correctly captures your\u200a\u2014\u200aor the decision-maker\u2019s\u200a\u2014\u200abeliefs\u2026 <\/em>before every experiment! This is a very difficult problem.<\/p>\n So, the Bayesian priors selected in practice are usually chosen because they are \u201cconvenient,\u201d \u201cuninformative,\u201d etc. They have little to do with actual prior\u00a0beliefs.<\/p>\n When we\u2019re not specifying our real prior beliefs, the posterior distribution is just some weighting of the likelihood function. Claiming that we can look at the quantiles of this so-called posterior distribution and say the parameter has a 10% chance of being less than 0 is nonsense statistically.<\/p>\n So, if anything, it is easier to misinterpret what we\u2019re doing in Bayesian land than in frequentist land. It is hard for statisticians to translate their prior beliefs into a distribution. How much harder is it for whoever the actual decision-maker is on the\u00a0project?<\/p>\n For these reasons, Bayesian inference doesn\u2019t scale well, which is why, I think, Experimentation Platforms across the industry generally don\u2019t use\u00a0it.<\/p>\n The arguments against the \u201cCult\u201d of Statistical Significance are, of course, a response to a real problem. There is<\/em> a dangerous Cult within our\u00a0Church<\/em>.<\/p>\n The Church of Statistical Significance is quite accepting. We allow for other alpha\u2019s besides 5%. We choose hypotheses that don\u2019t test against zero nulls,\u00a0etc.<\/p>\n But sometimes, our good name is tarnished by a radical element within the Church that treats anything insignificant versus a null hypothesis of 0 at the 5% level as \u201cnot\u00a0real.\u201d<\/p>\n These heretics believe in a cargo-cult version of statistical analysis where the statistical significance procedure (at the 5% level) determines what is true instead of just being a useful way to make decisions and weigh uncertainty.<\/p>\n We disavow all association with this dangerous sect, of\u00a0course.<\/p>\n Let me know if you\u2019d like to join the Church. I\u2019ll sign you up for the monthly\u00a0potluck.<\/p>\n Thanks for\u00a0reading!<\/p>\n Zach<\/p>\n Connect at: https:\/\/linkedin.com\/in\/zlflynn<\/a><\/p>\n In Defense of Statistical Significance<\/a> was originally published in Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\nStatistical significance does not equal business significance\u200a\u2014\u200abut so\u00a0what?<\/h3>\n
The two initiatives are separable.<\/h4>\n
The initiatives are alternatives.<\/h4>\n
Statistical significance is arbitrary.<\/h3>\n
People misinterpret frequentist p-values to indicate the probability that the effect is zero. To avoid this mistake, they should use Bayesian inference.<\/h3>\n
The Church of Statistical Significance<\/h3>\n
<\/p>\n
\n