Well over a year ago now…but still a hot topic!
AMERICAN STATISTICAL ASSOCIATION (ASA) STATEMENT ON STATISTICAL SIGNIFICANCE AND P-VALUES
The ASA is the world’s largest community of statisticians in the United States. Members serve in industry, government and academia in more than 90 countries, advancing research and promoting sound statistical practice to inform public policy and improve human welfare.
In March 2016, the American Statistical Association (ASA) released a “Statement on Statistical Significanceand P-Values” with six principles underlying the proper use and interpretation of the p-value.
“What we hope will follow is a broad discussion across the scientific community that leads to a more nuanced approach to interpreting, communicating, and using the results of statistical methods in research.”
This guidance on p-values was written to improve the conduct and interpretation of quantitative science as well as remind the importance of reproducibility.
Particularly over recent years, the scope of statistics and the importance of choosing the correct techniques for interpretation of large complex datasets has increased dramatically. More than ever, statistical analysis needs to be appropriately applied in order to correctly interpret data captured.
The statement by the ASA is very clear in emphasising that good statistical practice is essential, and that there should be a focus on good study design, conduct, varieties of numerical and graphical summaries of data, clear understanding of what the study is trying to achieve, complete reporting, clear interpretation of the results in context, as well as logical and quantitative understanding of data summaries. The p-value is more than an arbitrary threshold and should not replace well balanced statistical arguments.
“The contents of the ASA statement and the reasoning behind it are not new—statisticians and other scientists have been writing on the topic for decades,” Utts said. “But this is the first time that the community of statisticians, as represented by the ASA Board of Directors, has issued a statement to address these issues.”
“The issues involved in statistical inference are difficult because inference itself is challenging,” Wasserstein said.
The statement’s six principles on p-values:
1. P-values can indicate how incompatible the data are with a specified statistical model.
2. P-values do not measure the probability that the studied hypothesis is true, or the
probability that the data were produced by random chance alone.
3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
4. Proper inference requires full reporting and transparency.
5. A p-value, or statistical significance, does not measure the size of an effect or the
importance of a result.
6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.
In conclusion, we recommend that before solely relying on a p-value to tell you whether your results are important or not, consider the SIZE of the difference you have seen, whether it is consistent with other results and what you had expected, and if you can, size the experiment correctly before you start to ensure that any differences that are found to be significant statistically are also MEANINGFUL!
For the full statement and extended text, visit here
For further discussions and to hear more on this topic, why not attend one of our courses or talks?