[WAT] How to fail statistics.
A thing just happened to me in my Research & Design class.
My professor overheard me talking about significance testing to a classmate and decided to correct me in front of the whole class.
Significance testing, she said, should be done even when working with population data. "Even if the Whole Universe is your data set."
...
An explanation for those of you who don't grok the statistics:
Significance testing is used to determine if it's reasonably safe to treat a sample statistic as a population parameter. The standard for this is a low probability that a given statistic arose as a result of pure chance (IOW, you got your result because it's real, rather than just convenient noise). You get a "p value" or "alpha" which estimates the probability that your result is the product of random noise. (Note: your data and conclusions can still be wrong for any number of other reasons.) A high alpha suggests that your sample statistic doesn't necessarily say anything about the population you're studying.
But here's the thing. If you've managed to collect data on EVERY SINGLE MEMBER of the population? Your alpha is simply going to be zero. There is no possibility that you got that number by pure chance - you got it by collecting ALL OF THE DATA.
It gets better: My professor noted that you can take the numbers (presumably using N instead of n, which is just broken) and get a p value for your population data. "The maths don't care, they just want numbers."
This is true. It is as true as saying that I can put a pint of motor oil, a sneaker, and a kitten into my blender, hit liquefy and get a sludgy, largely homogeneous mixture out of the deal.
THAT DOESN'T MEAN I SHOULD TREAT THIS PRODUCT LIKE A FUCKING SMOOTHIE!
There are assumptions that go into significance testing, and one of those is that YOU NEED TO TEST FOR SIGNIFICANCE.
There are assumptions in smoothie recipes, namely that you're not a horrible monster.
When you throw that assumption out all kinds of broken things go wrong.
So I asked her, "Why on earth would you do that?" (Hoping she'd realize the error of her ways, silly me.)
"Because it's more convincing if you can say your results are statistically significant."
My professor overheard me talking about significance testing to a classmate and decided to correct me in front of the whole class.
Significance testing, she said, should be done even when working with population data. "Even if the Whole Universe is your data set."
...
An explanation for those of you who don't grok the statistics:
Significance testing is used to determine if it's reasonably safe to treat a sample statistic as a population parameter. The standard for this is a low probability that a given statistic arose as a result of pure chance (IOW, you got your result because it's real, rather than just convenient noise). You get a "p value" or "alpha" which estimates the probability that your result is the product of random noise. (Note: your data and conclusions can still be wrong for any number of other reasons.) A high alpha suggests that your sample statistic doesn't necessarily say anything about the population you're studying.
But here's the thing. If you've managed to collect data on EVERY SINGLE MEMBER of the population? Your alpha is simply going to be zero. There is no possibility that you got that number by pure chance - you got it by collecting ALL OF THE DATA.
It gets better: My professor noted that you can take the numbers (presumably using N instead of n, which is just broken) and get a p value for your population data. "The maths don't care, they just want numbers."
This is true. It is as true as saying that I can put a pint of motor oil, a sneaker, and a kitten into my blender, hit liquefy and get a sludgy, largely homogeneous mixture out of the deal.
THAT DOESN'T MEAN I SHOULD TREAT THIS PRODUCT LIKE A FUCKING SMOOTHIE!
There are assumptions that go into significance testing, and one of those is that YOU NEED TO TEST FOR SIGNIFICANCE.
There are assumptions in smoothie recipes, namely that you're not a horrible monster.
When you throw that assumption out all kinds of broken things go wrong.
So I asked her, "Why on earth would you do that?" (Hoping she'd realize the error of her ways, silly me.)
"Because it's more convincing if you can say your results are statistically significant."