Via Andrew Gelman’s blog:

“Five Ways to Fix Statistics”, comment feature in Nature.

As debate rumbles on about how and how much poor statistics is to blame for poor reproducibility, Nature asked influential statisticians to recommend one change to improve science. The common theme? The problem is not our maths, but ourselves.

Summary of positions in the feature below (as far as I can discern):

Leek

  • Main problem is that people misunderstand statistical analysis very easily. The process of data analysis can’t be summarized in purely computational procedures.

  • $p$-values are fine method to make decisions if used correctly.

  • An example of method that easily misleads the analysis: pie charts. Humans have difficulty of analyzing the information .

  • The old procedures for statistical analysis that were developed in the data-poor era are a poor fit for the modern age of massive data sets.

  • Leek makes a case for his own research topic, which he calls “epidemiology on how people collect, manipulate, analyse, communicate and consume data”.

McShane and Gelman

  • NHST “encourages researchers to investigate so many paths in their analyses that whatever appears in papers is an unrepresentative selection of the data”.

  • Point of NHST was prevent seeing effects in a noise, but in practice, this does not work, not at least in the fields where the effects are small and the data is noisy and “situation-dependent”.

  • The framework centered on deciding between “no effect” or “statistically significant effect” leads to maligned procedures.

  • $p$-values should not be abandoned altogether, but they should abandoned as a sole measure of significance and “considered as just one piece of evidence among many, along with prior knowledge, plausibility of mechanism, study design and data quality, real-world costs and benefits, and other factors.”

  • Open science could help but isn’t a panacea. Instead scientists should invest in better experiments and ready to accept that sometimes one can’t escape uncertainties and data simply is not enough to make a decision between “an effect” or “no effect”.

Colquhoun

  • One should consider also FPR, false positive risk.

  • “The best solution is to specify the prior probability needed to believe in order to achieve an FPR of 5%, as well as providing the P value and confidence interval.”

  • “Imagine the healthy scepticism readers would feel if, when reporting a just-significant P value, a value close to 0.05, they also reported that the results imply a false-positive risk of at least 26%. And that to reduce this risk to 5%, you’d have to be almost (at least 87%) sure that there was a real effect before you did the experiment.”

Nuijten

  • The procedures of scientific community itself are more important than the particular statistical methods.

  • We need conventions such as preregistering analysis plans, public data and published code to help the science to stay open and honest.

Goodman

  • Goodman also argues that norms within the scientific community should be improved.

  • Culture norms are important. Because they arise from mimicry, one should do their best to let the good practices proliferate: “In clinical research, the idea that a small randomized trial could establish therapeutic efficacy was discarded decades ago. In psychology, the notion that one randomized trial can establish a bold theory had been the norm until about five years ago.”

Reference

“Five Ways to Fix Statistics”. Nature, 2017. doi: 10.1038/d41586-017-07522-z