Choosing alpha post hoc: the danger of multiple standard significance thresholds

A fundamental assumption of classical hypothesis testing is that the significance threshold α is chosen independently from the data. The validity of confidence intervals likewise relies on choosing α beforehand. We point out that the independence of α is guaranteed in practice because, in most fields, there exists one standard α that everyone uses – so that α is automatically independent of everything. However, there have been recent calls to decrease α from 0.05 to 0.005. We note that this may lead to multiple accepted standard thresholds within one scientific field. For example, different journals may require different significance thresholds. As a consequence, some researchers may be tempted to conveniently choose their α based on their p-value. We use examples to illustrate that this severely invalidates hypothesis tests, and mention some potential solutions.