Version classiqueVersion mobile

Proceedings of the Paris Open Science European Conference

 | 
Open Science European Conference

Health Research Transparency

Publication and reporting bias: a long history towards open science

Tianjing Li

Texte intégral

1Both previous speakers have used the terms reporting bias and publication bias many times. I hope to bring them into the historical context and provide a review of the history of the research into reporting biases.

  • 1 The James Lind Initiative, « Reporting Bias », The James Lind Library, accessed 16 March 2022, htt (...)

2The problem of selective reporting has been recognised for hundreds of years. Reporting bias refers to or occurs when the direction and nature of the findings influence how research findings are being disseminated. Early in the 17th Century Francis Bacon noted that, ‘The human intellect is more moved by affirmatives than by negatives’. Robert Boyle, the chemist, lamented the common tendency among scientists not to publish their results until they had a system worked out with the result that ‘many excellent notions or experiments are, by sober and modest men, suppressed’. Many other scientists across many different fields have recognised the problem of selective reporting over the years1.

  • 2 Austin Bradford Hill, « Discussion of a paper by DJ Finney », Journal of the Royal Statistical Soc (...)
  • 3 Seymour Kety, « Comment », in Psychopharmacology. Problems in Evaluation, by JO Cole and RW Gerard (...)

3More recently, in 1956, the father of medical statistics in Britain, Austin Bradford Hill2, wrote, ‘A negative result may be dour, but often it is no less important than the positive and in view of that importance it must surely be established by adequate publication of the evidence’. American psychiatrist Seymour Kety3 wrote, ‘A positive result is exciting and interesting and gets published quickly. A negative result or one, which is inconsistent with current opinion, is either unexciting or attributed to some error and is not published, so that at first in the case of a new therapy there is a clustering towards positive results with fewer negative results being published. Then some brave or naive or non‑comformist soul, like the little child who said that the emperor had no clothes, comes up with a negative result which he dares to publish. That starts the pendulum swinging in the other direction and now negative results become popular and important’.

4These historical quotes provided you the background of how long we have recognised this problem. Systematic research into selective reporting began in education and psychology in the 1950s. Selective reporting of healthcare research was identified in the early 1980s, but the importance and the consequence of that was not recognised until a decade later. It has gained a lot of importance as a result of rapid development of research synthesis methods, such as systematic review. Systematic reviews are based on a complete survey of the literature on a given topic. They inform healthcare decisions and policies. When the systematic reviews are based on a selective subset of studies rather than all the studies out there, the findings could be biased. The need to have access to complete information about the methods and results of research is strong motivation for study into selective reporting.

  • 4 S. Swaroop Vedula, Asbjørn Hróbjartsson, Matthew J. Page, « Section 8. Introduction to Systematic (...)

5Reporting bias arises when the dissemination of research is influenced by the nature and direction of findings4. The nature and direction of findings refers to the statistical significance of the results, the magnitude of the effect and whether the results are in favour of a particular hypothesis. Statistically significant results are results that are unlikely to be due to chance. Instead, some kind of cause is more likely. Selective reporting can manifest in many different ways. To broadly categorise selective reporting, it may concern not reporting studies at all, also known as publication bias; reporting studies in part, also known as selective outcome reporting; reporting in a manner that is difficult for others to access or reporting without transparency, such as duplicate publication or spinning. Spinning means undermining or overemphasising certain results.

  • 5 Mary Lee Smith, « Responses of counselors to case materials of clients differentiated by gender an (...)
  • 6 Christine Schmucker et al., « Extent of Non-Publication in Cohorts of Studies Approved by Research (...)

6Mary Lee Smith5 likely first mentioned the term ‘publication bias’ in a manuscript where she compared the meta‑analytical results based on published studies with those that were not published. A comparison of the protocols of studies before they actually initiated with the publication – we call those inception cohort studies – provides a reliable estimate of the association between the nature and direction of findings with the publication status. Based on two cohort studies approved by research ethics committees or included in trial registrations, only 49% to 60% of randomised trials were published and the findings were three times more likely to be published when the results were statistically significant6. This is a huge problem.

  • 7 Dirk Eyding et al., « Reboxetine for Acute Treatment of Major Depression: Systematic Review and Me (...)

7What are the consequences of publication bias? First of all, it is not scientific. It may alter the evidence on the benefit and harms of treatment. One example is a comparison of reboxetine with placebo7. If you compare the published results with the unpublished results for both the efficacy and safety outcomes, the published results overemphasised the beneficial effect of reboxetine, which may not exist at all, and underestimated the harmful effects of reboxetine. As an example, with regard to the outcome ‘remission’, only one published trial showed that reboxetine is 2.5 times more likely to reduce remission, whereas six other unpublished trials showed no evidence of effect. In other words, published results are more than two times more likely to show the efficacy of reboxetine, which the efficacy does not exist.

8Failure to publish is also unethical. Participants enter research studies with the understanding that their involvement will contribute to generating new knowledge and to science. Failure to publish breaches the trust of the participants. Lastly, failure to publish research studies constitutes research waste. It is not informative because they never made it to the public domain. It was estimated that on average about 12 000 randomised control trials that should have been published each year were not published.

  • 8 See for example An-Wen Chan et al., « Empirical Evidence for Selective Reporting of Outcomes in Ra (...)

9The second form of reporting bias is outcome reporting bias. Outcome reporting bias manifests in different forms when the reporting of the outcome is influenced by the nature and direction of the findings. An‑Wen Chan and his colleagues have conducted quite a few studies8, which definitely characterised outcome-reporting bias in healthcare research. Outcome reporting bias manifests in different forms, for example, not reporting pre‑specified data in the protocols, reporting primary outcomes as secondary outcomes and vice-versa, introducing new outcomes or reporting outcomes such that they are differentially accessible. In other words, outcome reporting bias occurs when the authors cherry pick what they decided to show to the public.

  • 9 S. Swaroop Vedula et al., « Outcome Reporting in Industry-Sponsored Trials of Gabapentin for Off-L (...)

10I was involved in a study that examined outcome reporting bias in industry‑sponsored trials of gabapentin for off‑label use. We had access to the internal company documents as well as the publicly-available documents for 21 randomised trials that compared gabapentin with placebos for off‑label use. We compared protocol to publication for the statement of primary outcome9. We identified 21 primary outcomes in the protocol. Those are pre‑specified primary outcomes. Eleven were reported without changes in the final publication. Ten were not reported or reported as secondary outcomes. In addition, 17 new primary outcomes were introduced. In terms of the p‑values for the protocol‑defined primary outcomes in those research reports, nearly all of them are non‑statistically significant at the 0.05 significance level. However, through selective outcome reporting, by switching outcomes, introducing new outcomes, or omitting outcomes, in the primary publications the p‑value became statistically significant. In other words, regarding the efficacy of gabapentin, dependent upon where you look, you may find very different results.

  • 10 Kerry Dwan et al., « Systematic Review of the Empirical Evidence of Study Publication Bias and Out (...)

11That was one case study. There are many studies like this. When you take all studies together in a systematic review10, it yields empirical evidence for publication bias and outcome reporting bias. In summary, statistically significant outcomes were more likely to be reported in full for both efficacy and harm outcomes. About 40% to 62% of studies had at least one primary outcome changed, newly introduced or omitted when trial publications were compared with protocols.

  • 11 Roberta W. Scherer et al., « Full Publication of Results Initially Presented in Abstracts », The C (...)

12The third form of selective reporting is reporting in a manner that is difficult for others to access. One study conducted by a colleague of mine11 looked at full publication of results initially presented in conference abstracts. We all go to conferences. Disseminating your findings at conferences is a great way to reach a broad audience. However, the conference abstracts are usually limited by the word limit, and they do not provide a full depiction of the study methods or the results. This systematic review examined over 307 000 abstracts presented at conferences. The authors found that only about 37% were subsequently published in full, such as in a journal article, and that number was slightly better for randomised trials, again definite evidence for outcome reporting bias and publication bias. Significant results were 1.3 times more likely to be published in full. Positive results, defined as a result favouring the experiment treatment, were 1.2 times more likely to be published in full.

  • 12 Kay Dickersin et al., « Publication Bias and Clinical Trials », Controlled Clinical Trials 8, no 4 (...)

13Now you may start wondering whose problem is this? Who contributes to biased reporting of clinical trials12? Numerous surveys and other types of studies of researchers have found that investigators’ failure to submit trial results for publication was the primary reason for unpublished clinical trials. The frequently quoted reasons are they are not interested in the results and they do not have time to submit results for publication. The reasons for selective outcome reporting mirror those of publication bias. Investigators’ misconception that the editors favour positive results was the main reason for not publishing the results. Also, there is an unawareness of the seriousness of underreporting. On the other hand, there is no evidence that peer reviewer suggestions or the editors’ decisions are influenced by the nature and direction of the findings. In summary, it was mainly the researchers’ problem. We should always submit our results for publication regardless of the direction and the nature of the finding.

14To end on a high note, how do we address reporting biases? Both speakers have touched upon trial registration and a result database. I will discuss data sharing and access to study documents. There are also reporting guidelines and core outcome sets that people can follow.

15Beginning with the early studies, it became quite clear that to address reporting and publication bias we need to have a full account of all the studies from its beginning. The study of publication reporting biases and their consequences has led to legislation and policy mandating trial registration.

Fig. 2: A timeline of trial registration seminal events

Fig. 2: A timeline of trial registration seminal events

Source: Li, “Publication and reporting bias: a long history towards open science”. Media used during the presentation (available on the conference's website).

16In 1997, the Food and Drug Administration Modernisation Act required that all the drug trials of severe and life‑threatening conditions must be registered. As a result, in response to that legislation, the National Library of Medicine of the National Institutes of Health in the US launched ClinicalTrials.gov, the trial registration database in 2000. However, the registration had not taken off until the International Committee of Medical Journal Editors (ICMJE) made a requirement of prospective registration of trials before the trial results can be published by the ICMJE journals.

17In 2006, the World Health Organisation launched an international initiative to establish the International Clinical Trials Registry Platform. The platform provides one‑point access to different registries across countries. In 2007, the FDA Amendments Act (FDAAA) expanded the registration requirements – for all drugs, biologics and devices subject to FDA regulation, the FDAAA requires that the registration be done as soon as the trial is initiated. The results should be posted within 12 months of the completion of the trial or within one month of approval of the product. In response to that requirement, in 2008 and 2009 respectively, ClinicalTrials.gov released the results database and also added an adverse events module.

18In the European Union (EU), the European Medicines Agency (EMA) took a more open policy compared with the FDA. In 2011, EMA required that all the trials conducted in the EU should be registered and make their results available within 12 months. As mentioned earlier, the journals also play a very critical role in the transparency of clinical trial research. In 2013, The British Medical Journal put in place a requirement for a statement about anonymised participant level data availability at the time of manuscript submission. In 2016, the final rule made by the US Department of Health and Human Services further clarifies the types of trials that need to be registered on ClinicalTrials.gov and the National Institutes of Health (NIH) also issued a policy to promote the sharing of clinical trial data, requiring that any trials that received funding entirely or partly from the NIH should be registered before the start of the trial and have their results posted in a timely fashion.

  • 13 Evan Mayo-Wilson et al., « Harms Are Assessed Inconsistently and Reported Inadequately Part 2: Non (...)
  • 14 M. Backonja et al., « Gabapentin for the Symptomatic Treatment of Painful Neuropathy in Patients w (...)

19Access to study documents is also very critical. In a study where we examined adverse events for gabapentin13, we compared the number of different types of adverse events that appeared in the public domain with the number of different types of adverse events in the hidden documents, the documents not publicly available, such as clinical study reports. In the first trial14, there were only seven different types of adverse events mentioned in the publication, whereas 100 different types of adverse events were mentioned in those hidden documents. It does not look better for the serious adverse events. Public access to protocols, including the statistical analysis plan and clinical study reporting, is essential for preserving the societal value of clinical trials.

  • 15 Center for Global Clinical Research Data, « Why Share Your Clinical Research Data », Vivli, access (...)

20Publication was traditionally the way of sharing the results from clinical trials. Increasingly the clinical trial increases our knowledge and influences our decision‑making through both the summary level and the individual participant level data being shared and reused by the public. Data sharing increases the scientific value. Evidence has shown that it also increases citation. Furthermore, there is a moral obligation to share the data. Participants of trials become more vocal demanding that their data be shared and reused amongst the broader scientific community in order to leverage the data to maximise the knowledge and value that can be gained from their contributions to the trials. Surveys of patients also showed that most people welcome the idea of data sharing if safeguards are in place for their data safety15.

21Increasingly, funders are mandating data sharing. For example, the Gates Foundation and the Wellcome Trust require their awardees to share their clinical trial data and individual participant level data. Journals are also making requirements of making a statement about data availability and data sharing at the time of trial registration, and those statements will be published when the trial reports are accepted for publication.

22In summary, the move towards clinical trial data sharing is part of a wider movement towards open science. Reporting of medical research is biased when it is influenced by the nature and direction of the findings. That is the definition for publication bias and outcome reporting bias. They can take many different forms. Reporting bias typically involves spurious exaggeration of beneficial effects and suppression of harmful effects of interventions. Clinical trial registration, access to protocols and statistical analysis plans, guidelines for transparent and complete reporting, adoption of core outcome sets and data sharing are critical to prevent reporting biases.

To go a step further

23“In several studies, including one of our own, we found that the data collected in ClinicalTrials.gov is a minimal requirement and it does not capture everything you would like to know about the trial, especially with regard to the methodology and with regard to all the results data. It is definitely a first step, and how reliably they can be used for data analysis in systematic review and meta‑analysis, I believe we still have a long way to go, but at least we know the existence of trials, so we know if we are missing results or the publication of certain results, we have a place to look. We are getting there.”

24“The funders have a very important role. In the US, for example, when you are submitting a new grant if your previous publications are not in the public domain, you cannot even submit the grant. It comes down to how different stakeholders play together. The institution has a huge role. The funders have a huge role. However, there are so many studies that are not funded, so that is becoming more problematic, and the public has to get involved and demand transparency in research and demand open science. All the players need to work together to make this happen.”

25“Negative results are as important, if not more important, than positive results in the sense that if they are hidden away, information in the public domain is cherry picked. It looks rosier than it actually does, which is very problematic. Negative results will help us to use fewer medications.”

26“Most people would agree that open science is a great idea, but it requires resources to get there. For most research funders, you could build data sharing into the costing of your study; so there are resources available for researchers to share their data with the public.”

Notes

1 The James Lind Initiative, « Reporting Bias », The James Lind Library, accessed 16 March 2022, https://www.jameslindlibrary.org/research-topics/biases/reporting-bias/.

2 Austin Bradford Hill, « Discussion of a paper by DJ Finney », Journal of the Royal Statistical Society 119 (1956): 19‑20.

3 Seymour Kety, « Comment », in Psychopharmacology. Problems in Evaluation, by JO Cole and RW Gerard, 583 (Washington DC: National Academy of Sciences, 651).

4 S. Swaroop Vedula, Asbjørn Hróbjartsson, Matthew J. Page, « Section 8. Introduction to Systematic Reviews », in Principles and Practice of Clinical Trials, ed. Steven Piantadosi and Curtis L. Meinert, 1st ed. (Springer, 2022).

5 Mary Lee Smith, « Responses of counselors to case materials of clients differentiated by gender and ethnic group » (Unpublished Doctoral Dissertation, University of Colorado, 1970).

6 Christine Schmucker et al., « Extent of Non-Publication in Cohorts of Studies Approved by Research Ethics Committees or Included in Trial Registries », PLOS ONE 9, no 12 (23 December 2014): e114023, https://doi.org/10.1371/journal.pone.0114023.

7 Dirk Eyding et al., « Reboxetine for Acute Treatment of Major Depression: Systematic Review and Meta-Analysis of Published and Unpublished Placebo and Selective Serotonin Reuptake Inhibitor Controlled Trials », BMJ 341 (12 octobre 2010): c4737, https://doi.org/10.1136/bmj.c4737.

8 See for example An-Wen Chan et al., « Empirical Evidence for Selective Reporting of Outcomes in Randomized Trials: Comparison of Protocols to Published Articles », JAMA 291, no 20 (26 mai 2004): 2457‑65, https://doi.org/10.1001/jama.291.20.2457; An-Wen Chan and Douglas G. Altman, « Identifying Outcome Reporting Bias in Randomised Trials on PubMed: Review of Publications and Survey of Authors », BMJ (Clinical Research Ed.) 330, no 7494 (2 April 2005): 753, https://doi.org/10.1136/bmj.38356.424606.8F.

9 S. Swaroop Vedula et al., « Outcome Reporting in Industry-Sponsored Trials of Gabapentin for Off-Label Use », New England Journal of Medicine 361, no 20 (12 November 2009): 1963‑71, https://doi.org/10.1056/NEJMsa0906126.

10 Kerry Dwan et al., « Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias — An Updated Review », PLOS ONE 8, no 7 (5 July 2013): e66844, https://doi.org/10.1371/journal.pone.0066844.

11 Roberta W. Scherer et al., « Full Publication of Results Initially Presented in Abstracts », The Cochrane Database of Systematic Reviews 11 (20 November 2018): MR000005, https://doi.org/10.1002/14651858.MR000005.pub4.

12 Kay Dickersin et al., « Publication Bias and Clinical Trials », Controlled Clinical Trials 8, no 4 (December 1987): 343‑53, https://doi.org/10.1016/0197-2456(87)90155-3; Kay Dickersin, Yuan-I. Min, Curtis L. Meinert, « Factors Influencing Publication of Research Results. Follow-up of Applications Submitted to Two Institutional Review Boards », JAMA 267, no 3 (15 January 1992): 374‑78; Kay Dickersin and Yuan-I. Min, « NIH Clinical Trials and Publication Bias », The Online Journal of Current Clinical Trials Doc No 50 (28 April 1993); Kay Dickersin and Iain Chalmers, « Recognizing, investigating and dealing with incomplete and biased reporting of clinical research: from Francis Bacon to the WHO », Journal of the Royal Society of Medicine 104, no 12 (December 2011): 532‑38, https://doi.org/10.1258/jrsm.2011.11k042; Marc Mehu and Klaus R. Scherer, « The Appraisal Bias Model of Cognitive Vulnerability to Depression », Emotion Review 7, no 3 (1 July 2015): 272‑79, https://doi.org/10.1177/1754073915575406.

13 Evan Mayo-Wilson et al., « Harms Are Assessed Inconsistently and Reported Inadequately Part 2: Nonsystematic Adverse Events », Journal of Clinical Epidemiology 113 (September 2019): 11‑19, https://doi.org/10.1016/j.jclinepi.2019.04.020.

14 M. Backonja et al., « Gabapentin for the Symptomatic Treatment of Painful Neuropathy in Patients with Diabetes Mellitus: A Randomized Controlled Trial », JAMA 280, no 21 (2 December 1998): 1831‑36, https://doi.org/10.1001/jama.280.21.1831.

15 Center for Global Clinical Research Data, « Why Share Your Clinical Research Data », Vivli, accessed 16 March 2022, https://vivli.org/why-share-your-clinical-research-data/; Institute of Medicine of the National Academies, Sharing Clinical Trial Data: Maximizing Benefits, Minimizing Risk (Washington, DC: The National Academies Press, 2015), https://doi.org/10.17226/18998.

Table des illustrations

Titre Fig. 2: A timeline of trial registration seminal events
Crédits Source: Li, “Publication and reporting bias: a long history towards open science”. Media used during the presentation (available on the conference's website).
URL http://books.openedition.org/oep/docannexe/image/16117/img-1.png
Fichier image/png, 110k

CC-BY-4.0

Le texte seul est utilisable sous licence CC BY 4.0. Les autres éléments (illustrations, fichiers annexes importés) sont « Tous droits réservés », sauf mention contraire.

Lire

Open access

Acheter

Volume papier

i6doc.com
Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search