How does under-reporting of negative and inconclusive results affect the false-positive rate in meta-analysis? A simulation study

Data from individual studies

Meta-analyses of clinical trials with two arms and a binary outcome were simulated. However, the results of the simulations are applicable to other study designs as well because the distribution of the log-OR is approximately normal, similarly to the distribution of other commonly used effect size measures. Similar to another simulation study,²⁶ the sample size was modelled using the exponential of a normal distribution. This approach gives a right-skewed distribution, which is a realistic model. Based on the characteristics of the meta-analyses from the Cochrane Database of Systematic Reviews,²⁷ a mean of 4.51 and a variance of 1.47 was chosen. With these values, the median sample size equalled 91 and the IQR was 166. Following other simulation studies,^{19 20 26 28 29} equal sizes were used for the treatment group and control group.

As in other simulation studies,^{19 20 26} the probability of the event in the control group (p^C) was sampled from a uniform distribution U (0.3, 0.7). The probability of the event in the treatment group (p^T) was calculated from the equation logit (p^T)=logit (p^C)+δ+θ, where δ was the effect of study-specific characteristics on the log-OR, and θ was the mean effect size. The mean effect size equalled 0 because the effect of publication bias on the type I error rate for the test of the mean effect size was investigated. I sampled δ from a normal distribution N (0, τ²). For the between-study variability, τ², the values 0.02, 0.12 and 0.9 were considered. These values are the 10th, 50th and 90th centiles of the predictive distribution of the between-study variability in the meta-analyses of clinical trials from the Cochrane database.³⁰ The size of the between-study variability is often expressed in terms of I², defined as the proportion of the total variability due to heterogeneity.³¹ The considered values of τ² correspond to I²=17%, I²=56% and I²=90%. The number of events in the treatment and control group was sampled from a binomial distribution.

Selection process

The relative risk (RR) was defined as the ratio of the probability of including statistically significant positive results to the probability of including other results. However, the conclusions of the study are equally applicable to the case of a higher publishing probability for statistically significant negative outcomes. A conventional two-sided significance level of 0.05 was assumed. Three values of RR were considered: 1, 4 and 10. For RR=1, no publication bias was present. A value of four was chosen because multiple studies on publication bias estimated the ratio of the probability of publishing studies showing statistically significant positive results to the probability of publishing other results as close to four.^13–15 A value of 10 represents a strong publication bias and is still relevant in the light of the empirical research on publication bias in the medical literature.^{13 16 32}

Publication bias detection

A meta-analysis is often accompanied by an investigation of the presence of publication bias. Therefore, publication bias tests were incorporated in the simulations. The funnel plot is a scatter plot of effect estimates against some measure of precision. In the absence of a bias, the effect estimates from smaller studies scatter widely at the bottom of the funnel plot, with the spread narrowing among larger studies, so that the plot resembles a symmetrical inverted funnel.³³ If there is a bias, funnel plots are often asymmetrical.^{33 34} Since a funnel plot asymmetry is commonly used to investigate the presence of publication bias,³⁵ the funnel plots were inspected visually and using the following formal tests:

• the Egger's test, ‘Egger’;³⁴

• the rank correlation test, ‘Rank’;³⁶

• a modified Egger's test based on the efficient score, ‘Harbord’;²⁸

• a regression test based on sample size, ‘Peters’;²⁶

• a rank correlation test for binary data, ‘Schwarzer’;³⁷

• the Egger's test based on the arcsine transformation, ‘Arc-Egger’;³⁸

• a rank correlation test based on the arcsine transformation, ‘Arc-rank’;³⁸

• the trim and fill method, ‘Trim’.³⁹

For all tests, a significance level of 0.05 was used. For ‘Egger’, ‘Rank’, ‘Harbord’, ‘Peters’, ‘Schwarzer’, ‘Arc-Egger’ and ‘Arc-rank’ two-sided tests were used. For the trim and fill method, the presence of publication bias was indicated when the number of missing studies estimated by the R estimator in the first step of the algorithm was greater than 3.³⁹

Meta-analysis

The mean log-OR was estimated using the random effects model proposed by DerSimonian and Laird, which is a widely used approach to conduct a meta-analysis.⁴⁰ Four sizes of meta-analyses were considered: N=10, N=20, N=50 and N=100. Meta-analyses including less than 10 studies were not considered because publication bias tests were not recommended for use in this case due to a low power.³³

Simulations

Four sample sizes (N=10, N=20, N=50 and N=100), three sizes of the between-study variability (τ²=0.02, τ²=0.12 and τ²=0.9), and three levels of publication bias were considered (RR=1, RR=4 and RR=10), resulting in 36 simulation scenarios. For each scenario, the estimates of the mean effect size were evaluated in terms of the bias and the mean squared error. The effect of publication bias on the type I error rate for the test of the mean effect size was estimated for a grid of values within the considered ranges of the level of publication bias and the size of between-study variability. A two-sided significance level of 0.05 was assumed.

For each scenario, the power and the type I error rate for the publication bias tests were also investigated. Additionally, I estimated the type I error rate for the test of the mean effect size using only those samples where no publication bias was found. The purpose of this analysis was to investigate the effect of a one-sided selection process based on the statistical significance on the false-positive rate in meta-analysis in situations where publication bias detection methods cannot not identify the bias. All reported estimates are based on 10 000 simulations. The analysis was conducted in R (V.2.15.0). The R code used to perform the simulations is available online (see data sharing statement).

Validity of the mean effect size estimates

Figure 1 shows the type I error rates for the test of the mean effect size for the range of the level of publication bias and the amount of between-study variability considered in the study. In the presence of a selection process characterised by a higher probability of including statistically significant positive results, the meta-analyses frequently concluded that the mean effect size differed from zero when it in fact equalled zero. The magnitude of the effect of publication bias increased with an increasing number of studies and the amount of between-study variability. When statistically significant positive results were four times more likely to be included than other results, the type I error rate was between 11% and 100%. When statistically significant positive results were 10 times more likely to be included, between 25% and 100% of the meta-analyses concluded that the mean effect size differed from zero when it in fact equalled 0 (figure 1).

A higher probability of including statistically significant positive outcomes led to a drastic increase of the bias and the mean squared error, especially when a large between-study variability was present (table 1). When statistically significant positive results were four times more likely to be included than other results and 90% of the variability was due to between-study differences, the random-effects meta-analysis overestimated the mean log-OR approximately by 0.5 on average. When statistically significant positive results were 10 times more likely to be included and the same amount of between-study variability was present, the random-effects meta-analysis overestimated the mean log-OR by 0.83 on average. The mean squared error was especially large when the between-study variability was large (table 1).

Publication bias detection

Next, I investigated whether a one-sided selection process based on the statistical significance (which caused a drastic increase of the false-positive rate of the meta-analyses, as described in the previous section) was detectable by different publication bias methods.

Figure 2 shows data from simulations without publication bias (A and B) and simulations in which statistically significant positive results were 10 times more likely to be included than other results (C and D). A visual examination of the funnel plots indicated that a one-sided selection process based on the statistical significance introduced little asymmetry to the funnel plot both when the between-study variability was small (compare figure 2A, C) and large (compare figure 2B, D). In other words, the funnel plot provided no evidence of publication bias when positive statistically significant results were 10 times more likely to be included than other results.

Table 2 gives the proportions of the meta-analyses in which the presence of publication bias was indicated by formal tests. The scenarios with publication bias (RR=4 and RR=10) provide estimates of the power of different tests to detect a one-sided selection process based on the statistical significance. The scenarios without publication bias provide estimates of the type I error rate (the rate at which publication bias was indicated when no publication bias was present). When statistically significant positive results were four times more likely to be included than other results, all methods indicated the presence of publication bias in not more than 15% of the meta-analyses for all simulation settings (table 2). When statistically significant positive results were 10 times more likely to be included, the power of publication bias detection methods did not exceed 30% for any simulation setting. The type I error rates for the ‘Egger’, ‘Harbord’ and ‘Arc-Egger’ tests substantially exceeded 0.05 for some simulation settings, especially when a large between-study variability was present

False-positive rate in meta-analyses in which no publication bias was found

For the completeness of the study, I repeated the investigation of the effect of a selection process based on the statistical significance on the type I error rate for the test of the mean effect size using only those samples in which a certain publication bias test did not show evidence of publication bias. The aim of this analysis was to study whether a one-sided selection process based on the statistical significance threatened the validity of those meta-analyses where no evidence of publication bias was apparent. For example, meta-analyses were simulated until 10 000 samples were identified in which the ‘Egger’ test did not show any evidence of publication bias. Next, those samples were used to estimate the rate at which the meta-analysis led to the conclusion that the mean effect size differed from 0 when it actually did not, under a selection process based on the statistical significance that could not be detected by the ‘Egger’ test. Table 3 compares the proportion of meta-analyses incorrectly showing that the mean effect size differed from zero among all samples (column ‘All’) and among samples where no publication bias was found. There was little difference in the type I error rate for the test of the mean effect size between the meta-analyses without evidence of publication bias and all meta-analyses.