En español
NIDA

Reproducibility Issues May Be Science’s Growing Pains

November 20, 2014

I have written before about the reproducibility crisis in the health sciences. High percentages of published findings are failing to be reproduced, and this has prompted NIH Institutes and other funding agencies, as well as scientific journals and universities, to take measures to improve how science is taught, funded, conducted, and published. Among other things, it is crucial to make sure that the less exciting work of replication (and publishing of negative results) is properly incentivized and that researchers are completely clear and transparent when describing their methods.

Like our colleagues in other Institutes, we at NIDA are committed to addressing the problem, however I have also come to think we should not let ourselves become too depressed by it. The crisis of reproducibility does not merely reflect a failure of scientists or scientific organizations; it also is an inevitable effect of the rapidly increasing power and complexity of the science we are doing. The biological systems we study are unimaginably complex, they are dynamically changing, and they have many many forms of variance that only become apparent when subjected to increasingly complex tools and methods of analysis. The complexity of the analyses we are doing now, and the huge data sets being generated, are unprecedented, so we should not be surprised that we are encountering much more fine-grained variability than we ever expected or were prepared for.

This was driven home to me recently in the context of my research on dopamine signaling in obesity. I have consistently found a significant reduction in dopamine D2 receptors in the striatum of obese individuals. However, some of my collaborators at Vanderbilt found an increase in these same receptors. When I visited the lab and met with my colleagues there, a likely reason for the discrepancy soon became apparent: Whereas I had scanned my subjects during the day, they had been scanning subjects at night. The time of day was a detail neither of us thought to consider or mention when publishing our methods, yet it turned out to make a big difference in dopamine receptor expression.

Image of brain scans of control and Obese subjects showing reduced DA D2 receptor activity from control

As our ability to analyze biological systems increases in resolution, so too does the number of variables that can and need to be taken into account. Even when they do everything right and act in good faith, different laboratories may be unable to replicate findings because of nuances that might not have mattered as much in an earlier era, when research focused on coarser-grained phenomena.

If anything, the reproducibility crisis gives us a humbling reminder that science is still in its adolescence; failures of replication are growing pains—and they may not end anytime soon. As the science keeps advancing at an accelerating rate, and as we bring greater resolution to bear on our objects of study, we are likely to see more and more variability of findings. In light of this, we can maximize the quality of our data by ensuring that experiments follow proper methodology, including those involved in analysis, and by enhancing the transparency of our published findings.

This page was last updated November 2014

Comments

So true

The reproducibility crisis is well understood with the pro marijuana lobby whose efforts are currently at full speed ahead toward universal acceptance and legalization of marijuana. Here is one of the most serious and drastic course changes in American history about to take place, yet the science lags.

For instance, if a professional psychiatrist wants to air his concerns about the possibility that substantial marijuana abuse induced the mental illness that lead to the Boston Marathon bombing (or any of the other similar horrendous acts occurring that also involved marijuana use),, the scientist should not be chastised for voicing his concerns. It's one thing suggesting concerns, and another entirely making proclamations. It is a distinct possibility marijuana was the ultimate cause- a process involving indoctrination with drug abuse.
Scientists need to speak up and voice concerns, or suspicions, even without the concrete verifiable science that invariably lags. One big question is are violence and drugs so often associated with violence because drugs are illegal, or because drug abusers' minds are altered and conducive to violent acts because of the drugs? It's a monumental question whose answer in my opinion is because the drugs themselves are inducing mental illness. The users couldn't care less if the drugs are illegal. Questions like this must be addressed, and hopefully answered eventually.
The science will come. The question is how much further do we damage our country?

Undocumented confounders

It is true that the circadian component of the neuronal processes is among the undocumented confounders and as such is essentially being ignored. Moreover, the current mouse models are unusable to study the circadian component, as they are themselves damaged in this area due to mutations.

Find Help Near You

The following website can help you find substance abuse or other mental health services in your area: www.samhsa.gov/Treatment. If you are in an emergency situation, people at this toll-free, 24-hour hotline can help you get through this difficult time: 1-800-273-TALK. Or click on: www.suicidepreventionlifeline.orgExternal link, please review our disclaimer.. We also have step by step guides on what to do to help yourself, a friend or a family member on our Treatment page.

Get this Publication

    About This Blog

    Welcome to my blog, here I highlight important work being done at NIDA and other news related to the science of drug abuse and addiction.

    Receive Nora’s Blog Articles in your Email!
    You will only receive messages related to Nora’s Blog

    Dr. Nora Volkow: Video Highlights

    NIDA Director, Dr. Nora D. Volkow Videos

    • National Committee for Quality Assurance (NCQA): Quality Talks, October 2016
      Treating Addiction Within the Health Care System
    • APA TV, May 2016
      APA TV chats with Dr. Nora Volkow, the Director of NIDA to hear about her lecture on drug abuse and the opioid epidemic.
    • Kentucky Educational Television, May 2016
      One to One with Bill Goodman: Dr. Nora Volkow
    • TEDMED, January 2015
      Why do our brains get addicted?
    • The World Science Festival, April 2014
      The Moth: The Brain's Addiction - Nora Volkow
    • Rockburn Presents, November 2012
      Dr. Nora Volkow
    • Brookhaven National Laboratory WBNL Video, October 2012
      Chemistry celebration: FDG: Contribution to Our Understanding of Addiction
    • CBS 60 Minutes, April 2012
      Hooked: Why Bad Habits Are Hard to Break 
    • Science Times, June 2011
      Dr. Nora Volkow