In 2011, a team of physicists reported a startling discovery: neutrinos traveled faster than the speed of light by 60 billionths of a second in their 730 kilometer trip from Geneva to a detector in Italy. Despite six months of double checking, the bizarre discovery refused to yield. But rather than celebrating a physics revolution, the researchers published a cautious paper arguing for continued research in an effort to explain the observed anomaly. In time, the error was tracked to a single incorrectly connected fiber optic cable. This example reminds us that real science is more than static textbooks. Instead, researchers around the world are continuously publishing their latest discoveries with each paper adding to the scientific conversation.
Published studies can motivate future research, inspire new products, and inform government policy. So it’s important that we have confidence in the published results. If their conclusions are wrong, we risk time, resources, and even our health in the pursuit of false leads. When findings are significant, they are frequently double-checked by other researchers, either by reanalyzing the data or by redoing the entire experiment. For example, it took repeated investigation of the CERN data before the timing error was tracked down. Unfortunately, there are currently neither the resources nor professional incentives to double check the more than 1 million scientific papers published annually.
Even when papers are challenged, the results are not reassuring. Recent studies that examined dozens of published pharmaceutical papers managed to replicate the results of less than 25% of them. And similar results have been found in other scientific disciplines. There are a variety of sources for irreproducible results. Errors could hide in their original design, execution, or analysis of the data. Unknown factors, such as patients’ undisclosed condition in a medical study, can produce results that are not repeatable in new test subjects. And sometimes, the second research group can’t reproduce the original results simply because they don’t know exactly what the original group did.
However, some problems might stem from systematic decisions in how we do science. Researchers, the institutions that employ them, and the scientific journals that publish findings are expected to produce big results frequently. Important papers can advance careers, generate media interest, and secure essential funding, so there’s slim motivation for researchers to challenge their own exciting results. In addition, little incentive exists to publish results unsupportive of the expected hypothesis.
That results in a deluge of agreement between what was expected and what was found. In rare occasions, this can even lead to deliberate fabrication, such as in 2013, when a researcher spiked rabbit blood with human blood to give false evidence that his HIV vaccine was working. The publish or perish mindset can also compromise academic journals’ traditional peer-review processes which are safety checks where experts examine submitted papers for potential shortcomings.
The current system, which might involve only one or two reviewers, can be woefully ineffective. That was demonstrated in a 1998 study where eight weaknesses were deliberately inserted into papers, but only around 25% were caught upon review. Many scientists are working toward improving reproducibility in their fields. There’s a push to make researchers raw data, experimental procedures, and analytical techniques more openly available in order to ease replication efforts.
The peer review process can also be strengthened to more efficiently weed out weak papers prior to publication. And we could temper the pressure to find big results by publishing more papers that fail to confirm the original hypothesis, an event that happens far more than current scientific literature suggests. Science always has, and always will, encounter some false starts as part of the collective acquisition of new knowledge. Finding ways to improve the reproducibility of our results can help us weed out those false starts more effectively, keeping us moving steadily toward exciting new discoveries.