Science Is Broken. Can Better Statistics Help?

This article was originally posted on RealClearScience.

Science is going through a bit of a crisis. Thanks largely to the pioneering work of Dr. John Ioannidis, scientists — particularly biomedical researchers — are grappling with the reality that much of the scientific literature cannot be replicated and is simply incorrect. The reasons are manifold, among them the tendency of top-notch journals to publish “flashy” research, publication bias, flaws in peer review and the inadequacy of statistical tests. It is this latter problem that new research in PNAS hopes to address.

Typically, biomedical scientists use a significance level of 0.05 to test new hypotheses or to announce discoveries. In plain English, this means that scientists are willing to accept a 5% probability that the data they collected occurred merely by chance. The scientific community has agreed that the arbitrary “5% chance of being wrong” standard is sufficient for most research.

Not so, says Dr. Valen Johnson of Texas A&M University. He believes the 0.05 significance level needs to be replaced. Using the Bayes factor — a ratio that measures the probability of competing hypotheses — Dr. Johnson shows that the 0.05 standard is terribly insufficient. (See figure.)

According to his analysis, a significance level (denoted with the Greek letter “alpha” on the X-axis) of 0.05 corresponds to a Bayes factor of around 3 to 5. That’s rather pathetic, as it means the newer, more fascinating hypothesis is only 3 to 5 times more likely to explain the data than the old, boring one.

Significance levels more stringent than 0.05 yield much more convincing Bayes factors. For 0.01, the Bayes factor range is 12 to 20; for 0.005, the range is 25 to 50; and for 0.001, the range is 100 to 200. A Bayes factor of 100 or greater is considered “decisive” evidence in favor of the new hypothesis.

Dr. Johnson advises the scientific community to replace its 0.05 significance level with 0.005. If his statistical analysis is correct, it’s difficult to argue with him.

Source: Johnson VE. “Revised standards for statistical evidence.” PNAS. Published online before print, 11-Nov-2013.