Detecting (even more) outcome reporting bias in clinical trials by digging deep(er)

In clinical trials, outcomes need to be prespecified to ensure that investigators do not cherry pick whatever data look most impressive after a trial has been completed.

Post hoc outcome changes in clinical trials can undermine the credibility and validity of trial evidence and lead to biased or misleading conclusions for health care decision-making if they are not clearly flagged in scientific publications.

Outcome switching in German trials

However, such changes are common and often not transparently reported.

In a recent study, we analysed primary outcome discrepancies between trial registry entries and publications of clinical trials completed at German University Medical Centres between 2009 and 2017.

More than two in five trials (41%) in the German cohort had primary outcome discrepancies between the latest registry entry and the publication.

These findings are consistent with other studies that found frequent primary outcome discrepancies between study registries or study protocols on the one hand, and publications on the other (see our preprint for a full discussion of the literature).

However, our study dug deeper.

Unearthing 'hidden' discrepancies

We also looked at ‘hidden’ discrepancies.

Often, the primary outcome definition in the paper and in the latest version of the trial registry entry are identical. But when you dig deeper and look at older versions of the registry entry, you discover that the outcome was changed at some point after the first patient was enrolled.

In our cohort, 14% of trials had such ‘hidden’ discrepancies in primary outcomes.

Only 1% of discrepancies were transparently reported in publications.

Our findings suggest that previous assessments may have underestimated the extent of outcome reporting bias in clinical trials.

These ‘hidden’ discrepancies cannot be detected by simply comparing publications to the latest registry entry.

How can this be fixed?

• Clinical trial registries should provide easy access to the history of changes for each trial record, and flag any major discrepancies between different versions of the same record.

• Journal editors and peer reviewers should check for outcome consistency between all versions of registry entries and manuscripts.

• Principal investigators should adhere to ethical and reporting standards that include clear pre-specification of primary outcomes, and transparent disclosure of any outcome changes and the reasons for these in trial registries and publications.

The results from our study highlight the need for more rigorous scrutiny of the medical literature by journal editors, peer reviewers and readers. This guest blog was written by Martin Haslberger and Martin Holst. Both are researchers at the BIH-QUEST Center for Responsible Research in Berlin. Their preprint “Hidden changes to prespecified primary outcomes of clinical trials completed between 2009 and 2017 in German University Medical Centres: A meta-research study” is available online.