Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2016 Dec 20:5:e21451.
doi: 10.7554/eLife.21451.

Publication bias and the canonization of false facts

Silas Boye Nissen  1 , Tali Magidson  2 , Kevin Gross  3 , Carl T Bergstrom  4
Affiliations

Publication bias and the canonization of false facts

Silas Boye Nissen et al. Elife. .

Abstract

Science is facing a "replication crisis" in which many experimental findings cannot be replicated and are likely to be false. Does this imply that many scientific facts are false as well? To find out, we explore the process by which a claim becomes fact. We model the community's confidence in a claim as a Markov process with successive published results shifting the degree of belief. Publication bias in favor of positive findings influences the distribution of published results. We find that unless a sufficient fraction of negative results are published, false claims frequently can become canonized as fact. Data-dredging, p-hacking, and similar behaviors exacerbate the problem. Should negative results become easier to publish as a claim approaches acceptance as a fact, however, true and false claims would be more readily distinguished. To the degree that the model reflects the real world, there may be serious concerns about the validity of purported facts in some disciplines.

Keywords: false positive; hypothesis testing; none; publication bias; replication crisis.

PubMed Disclaimer

Conflict of interest statement

CTB: Reviewing editor, eLife. The other authors declare that no competing interests exist.

Figures

Figure 1.
Figure 1.. Conducting and reporting the test of a claim.
In our model, a scientific claim is either true or false. Researchers conduct an experiment which either supports or fails to the support the claim. True claims are correctly supported with probability 1-β while false claims are incorrectly supported with probability α. Next, the researchers may attempt to publish their results. Positive results that support the claim are published with probability ρ1 whereas negative results that fail to support the claim are published with probability ρ0. This process then repeats, with additional experiments conducted until the claim is canonized as fact or rejected as false. DOI: http://dx.doi.org/10.7554/eLife.21451.002
Figure 2.
Figure 2.. A time-directed graph represents the evolution of belief over time.
In panel A, the horizontal axis indicates the number of experiments published and the vertical axis reflects the observer’s belief, quantified as the probability that the claim is true. The process begins at the single point at far left with an initial belief q0. Each subsequent experiment either supports the claim, moving to the next node up and right, or contradicts the claim, moving to the next node down and right. At yellow nodes, the status of the claim is as yet undecided. At green nodes, it is canonized as fact, and at blue nodes, it is rejected as false. The black horizontal lines show the evidentiary standards (τ0 and τ1). The red path shows one possible trajectory, in which a positive experiment is followed by a negative, then two positives, then a negative, etc., ultimately becoming canonized as fact when it reaches the upper boundary. Panel B shows the same network, but with the vertical axis representing log odds and using color to indicate the probability that the process visits each node. In log-odds space, each published positive result shifts belief by the constant distance d1>0 and each negative result by a different distance d0<0. Shown here (in both panel A and B) is a false claim with false positive rate α=0.2, false negative rate β=0.4, publication probabilities p0=0.1 and p1=1, and initial belief q0=0.1. In this case, the claim is likely to be canonized as fact, despite being false. DOI: http://dx.doi.org/10.7554/eLife.21451.003
Figure 3.
Figure 3.. ROC curves reveal that true claims are almost always canonized as fact.
In the receiver operating characteristic (ROC) curves shown here, the vertical axis represents the probability that a true claim is correctly canonized as fact, and the horizontal axis represents the probability that a false one is incorrectly canonized as fact. Panel A: lax evidentiary standards τ0=0.1 and τ1=0.9. Panel B: strict evidentiary standards τ0=0.001 and τ1=0.999. Error rates and initial belief are α=0.05, β=0.2, and q0=0.5. Each point along the ROC curve corresponds to a different value of the negative publication rate, ρ0, as indicated by color. Grey regions of the curve correspond to the unlikely situations in which ρ0>ρ1=1, i.e., negative results are more likely to be published than positive ones. The figures reveal two important points. First, when negative results are published at any rate ρ0≤1, the vast majority of true claims are canonized as fact. Second, when negative results are published at a low rate (ρ0 less than 0.3 or 0.2 depending on evidentiary standards), many false claims will also be canonized as true. DOI: http://dx.doi.org/10.7554/eLife.21451.004
Figure 4.
Figure 4.. Publishing negative outcomes is essential for rejecting false claims.
Probability that a false claim is incorrectly canonized, as a function of the negative publication rate. Throughout, initial belief is q0=0.5, and individual data series show false positive rates α=0.05 (yellow), 0.10,…,0.25 (red). Top row: weak evidentiary standards τ0=0.1 and τ1=0.9. Panel A: false negative rate β=0.2. Panel B: β=0.4. Panels C–D: similar to panels A–B, with more demanding evidentiary standards τ0=0.001 and τ1=0.999. DOI: http://dx.doi.org/10.7554/eLife.21451.005
Figure 5.
Figure 5.. False canonization rates are relatively insensitive to initial belief, unless experimental tests are inaccurate and evidentiary standards are weak.
Probability that a false claim is mistakenly canonized as a true fact vs. prior belief for various negative publication rates. Top row: weak evidentiary standards τ0=0.1 and τ1=0.9. Panel A: false positive rate α=0.05, false negative rate β=0.2, and publication rate of negative results ρ0=0.025 (light green), 0.05,0.1,0.2,0.4 (dark green). Panel B: α=0.2, β=0.4, and ρ0=0.1 (light green), 0.3,0.4,0.5,1 (dark green). Panels C–D: similar to panels A–B, with more demanding evidentiary standards τ0=0.001 and τ1=0.999. DOI: http://dx.doi.org/10.7554/eLife.21451.006
Figure 6.
Figure 6.. Strengthening evidentiary requirements does not necessarily decrease canonization of false facts.
In panel A, the false positive rate is α=0.05, the false negative rate is β=0.2, the original belief in the claim is q0=0.5, and the evidentiary standards are symmetric τ1=1-τ0. In panel B, the false positive rate is increased to α=0.25 while the other parameters remain unchanged. Particularly in this latter case, increasing evidentiary standards does not necessarily decrease the rate at which false claims are canonized as facts. DOI: http://dx.doi.org/10.7554/eLife.21451.007
Figure 7.
Figure 7.. Scientific activity will tend to increase belief in false claims if too few negative outcomes are published.
Expected change in log odds of belief vs. negative publication rate for (A) false and (B) true claims. Lines show false positive rates α=0.05 (yellow), 0.10,…,0.25 (red). Other parameter values are false negative rate β=0.2 and positive publication rate ρ1=1. DOI: http://dx.doi.org/10.7554/eLife.21451.008
Figure 8.
Figure 8.. p-hacking dramatically increases the chances of canonizing false claims.
Probability that a false claim is canonized as fact vs. fraction of negative outcomes. Throughout, all positive outcomes are published (p1=1), and the nominal false positive rate is αnom=0.05, the false negative rate is β=0.2, and evidentiary standards are strong (τ0=0.001 and τ1=0.999). Curves show actual false positive rates αact=0.05 (yellow), 0.10,…,0.25 (red). Compared with Figure 4C, in which the nominal rates are equal to the actual rates, the probability of canonizing a false claim as fact is substantially higher. DOI: http://dx.doi.org/10.7554/eLife.21451.009
Figure 9.
Figure 9.. Publishing a larger fraction of negative outcomes as belief increases lessens the chances of canonizing false claims.
Probability that a false claim is mistakenly canonized as a true fact vs. baseline probability of publishing a negative outcome. The baseline probability of publishing a negative outcome is the probability that prevails when belief in the claim is weak. The actual probability of publishing a negative outcome increases linearly from the baseline rate when belief is 0 to a value of 1 when belief is 1. All other parameters are the same as in Figure 4. DOI: http://dx.doi.org/10.7554/eLife.21451.010
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Arbesman S. The Half-Life of Facts: Why Everything We Know Has an Expiration Date. Penguin; 2012.
    1. Baker M. 1,500 scientists lift the lid on reproducibility. Nature. 2016;533:452–454. doi: 10.1038/533452a. - DOI - PubMed
    1. Begg CB, Berlin JA. Publication bias: a problem in interpreting medical data. Journal of the Royal Statistical Society. Series A. 1988;151:419–463. doi: 10.2307/2982993. - DOI
    1. Begley CG, Ellis LM. Drug development: Raise standards for preclinical cancer research. Nature. 2012;483:531–533. doi: 10.1038/483531a. - DOI - PubMed
    1. Bernstein E, Caudy AA, Hammond SM, Hannon GJ. Role for a bidentate ribonuclease in the initiation step of RNA interference. Nature. 2001;409:363–366. doi: 10.1038/35053110. - DOI - PubMed

Publication types

LinkOut - more resources