Detecting errors, fabrication, and misconduct in clinical trials: Audit Clinical Evidence

In this guest blog, Matthew B. Jané and Kutenda F. Mvududu explain how their initiative Audit Clinical Evidence (ACE) is checking for errors, fabrication, and misconduct in published clinical trial results. According to ACE’s website, their first two audits found 139 research integrity violations. In total, 26 clinical guidelines, 94 systematic reviews and 514 articles had cited the flawed trials.

We started the Audit Clinical Evidence (ACE) initiative to identify and correct errors, and ultimately preserve the credibility of the scientific record and trust in medical research.

Meta-analysts, reviewers, and stakeholders typically take clinical trial results at face value. They do not consider whether errors, fabrication, or misconduct may have contaminated the design, methods, analysis, and reporting of a study.

Peer review is the most common and conventional safeguard used in clinical research, yet reviewers rarely use forensic tools to check whether the reported results are accurate or plausible.

As a result, clinicians, scientists, and decision-makers may be misled and misinformed about treatments, interventions and their real-world outcomes. 

How do we audit clinical trials?

At ACE, we routinely audit published clinical research for serious errors or integrity concerns. We report our findings to journal editors to give them the opportunity to correct the affected articles. We also publish every report, so the public becomes aware of the problems.

Each audit asks whether an article’s reported results could be true:

• We reproduce its statistical analyses from the reported values and test whether its summary statistics are consistent with real data by using tools such as GRIM, GRIMMER, SPRITE, and DEBIT.

• We screen figures and images for duplication and manipulation with ImageTwin.

• We also check each article against the wider public record, including trial registrations, ethics documentation, and related publications.

All applications of these methods are made transparent so that anyone can reproduce the results of our reports.

First audit: Implausibly large treatment effects

Our first audit examined three randomized trials (all published in 2019) about exercise for depression in patients with heart failure. Two were published in Medicine and one in Clinics.

The three articles appear to report overlapping arms of the same trial, as they share duplicated baseline characteristics and outcome statistics. However, they list three different ethics approval numbers and incompatible author contribution statements, and none of them cites the others.

In the table displaying the results of the primary outcome (i.e., the PHQ-9 depression scale), 7 of the 9 reported PHQ-9 mean and SD pairs fail the GRIM or GRIMMER test. The implied standardized mean differences in the outcome come out to be about −3 which is unrealistically large for a clinical trial.

These results were then cited by 17 meta-analyses and review articles, propagating these problems throughout the literature.

Second audit: Detecting deviant errors

As the studies we audit have already entered systematic reviews and meta-analyses, an error in one article can spread to every review that includes it.

Our second audit report shows how this happens based on a small randomized controlled trial of resistance training for depression in a small sample of older adults. Every table in the article labels the values in parentheses as standard deviations, yet they are actually standard errors.

A standard error is the standard deviation divided by the square root of the sample size, so with groups of 15 to 17 people it is about a quarter of the standard deviation. A meta-analyst who mistakes the standard errors as standard deviations will divide the treatment effect by a number four times too small, which makes the effect about four times too large.

Of the 24 meta-analyses located that report an effect for this trial, 19 assign it an artificially large value, thus biasing their data sets. This goes to show how a simple error can have major downstream effects on the literature.

Flagging concerns with scientific journals

We send each report to the journal and log the correspondence publicly until the journal makes a decision.

Our reports, code, and handbook are open to the public. Reports are open access, archived on Zenodo with version control and a corresponding DOI.

Findings within audit reports are organized systematically. Each integrity concern found in a report is classified under the ACE taxonomy of research integrity violations. This taxonomy has 27 categories divided into four domains: authorship and publication, statistical and data integrity, ethical oversight and transparency, and indicators and minor issues.

By making every finding public, ACE aims to help clinical research correct itself before errors reach patients.

Are you concerned about a published trial?

If you have found a published trial that contains errors or other concerns, write to us at contact@auditclinicalevidence.org. Priority goes to concerns that are specific, reproducible, and relevant to clinical research.

This guest blog was written by Matthew B. Jané and Kutenda F. Mvududu, the founders of Audit Clinical Evidence (ACE). TranspariMED did not fact check any of the claims made in this blog. All images used are screenshots taken from the ACE website.