file-drawer effect

The file-drawer effect refers to the practice of researchers filing away studies with negative outcomes. Negative outcome refers to finding nothing of statistical significance or causal consequence, not to finding that something affects us negatively. Negative outcome may also refer to finding something that is contrary to one's earlier research or to what one expects.

The practice of reporting and publishing only positive-outcome research creates a misrepresentation of the subject under investigation, especially if a meta-analysis is done.

One criticism of parapsychology has been that its researchers have ignored studies with negative outcomes. In 1975, the American Parapsychological Association established an official policy against the selective reporting of only positive results.

Little research seems to have done on the extent of the practice of scientific researchers to file away studies with negative outcomes. Brian Martinson, an investigator with the HealthPartners Research Foundation, led a study for the scientific journal Nature (published in June 2005). Martinson and his colleagues sent a survey to thousands of scientists funded by the National Institutes of Health (which commissioned the study) and received 1,768 responses with usable data from the 3,600 surveys mailed to mid-career scientists. They received 1,479 responses with usable data from 4,160 surveys sent to early-career scientists (Martinson et al. 2005). The respondents were allowed to remain anonymous. Of the scientists who responded, 6.0 percent admitted  to having "tossed out data because the information contradicted their previous research." And more than 15 percent admitted they had ignored observations because they had a "gut feeling" they were inaccurate.* Martinson et al. write:

our approach certainly leaves room for potential non-response bias; misbehaving scientists may have been less likely than others to respond to our survey, perhaps for fear of discovery and potential sanction. This, combined with the fact that there is probably some under-reporting of misbehaviours among respondents, would suggest that our estimates of misbehaviour are conservative.

See also meta-analysis and positive-outcome bias.

further reading

websites

Filed under F (for forgotten) by Dan Vergano, May 16 2001

Stenger, Victor J. (2002). "Meta-Analysis and the File-Drawer Effect." Skeptical Briefs.

Scientists who do not publish trial results are "unethical" by Gavin Yamey, BMJ

Bias against negative studies in newspaper reports of medical research by G. Koren and N. Klein

Scientists behaving badly by Brian C. Martinson, Melissa S. Anderson, and Raymond de Vries.  Nature 435, 737-738. June 9, 2005.

Publication Bias: The “File-Drawer” Problem in Scientific Inference by Jeffrey D. Scargle

news

 It's a scandal drug trial results are still being withheld by Dr. Ben Goldacre "...the chances of a completed trial being published are roughly 50%....trials with positive results are twice as likely to be published as those with negative results...."

The drugs don't work: a modern medical scandal by Dr. Ben Goldacre "Seven trials had been conducted comparing reboxetine against a placebo. Only one, conducted in 254 patients, had a neat, positive result, and that one was published in an academic journal, for doctors and researchers to read. But six more trials were conducted, in almost 10 times as many patients. All of them showed that reboxetine was no better than a dummy sugar pill. None of these trials was published."--Ben Goldacre [The Guardian article is an excerpt from Bad Pharma: How Drug Companies Mislead Doctors and Harm Patients published in the UK September 25, 2012; to bepublished in the U.S. January 8, 2013.]