Automated search for clinical trial publications: two new tools

Two new tools aim to make the search for clinical trial results in scientific journals easier, faster and more accurate.

The tools are likely to be useful to systematic reviewers, guideline developers, meta-researchers, trial registry managers, and research funders and institutions seeking to keep track of their trial portfolios.

At present, both tools only cover trials that are listed on the world’s largest trial registry, ClinicalTrials.gov.

Tool 1: Trials to Publications

The first tool, “Trials to Publications”, has been around for several years already but was recently upgraded. According to the accompanying publication:

“Trials to Publications employs a machine-learning model based on title, abstract, and other metadata features to predict which publications are likely to present clinical outcome results from a given registered trial in ClinicalTrials.gov."

"We have now updated and expanded the scope of the tool, by extracting mentions of ClinicalTrials.gov registry numbers from the full-text of 3 online biomedical article collections, as well as retrieving biomedical publications that are mentioned within the ClinicalTrials.gov registry itself. These mentions greatly increase the number of linked publications identified by the tool and should assist those carrying out evidence syntheses as well as those studying the metascience of clinical trials.”

Tool 2: TrialScout

The second tool, “TrialScout”, is novel and links published results to trial registrations using a large language model. According to the accompanying publication:

“[TrialScout] uses a large language model to match clinical trials registered on ClinicalTrials.gov with corresponding result publications indexed in PubMed. TrialScout's performance was evaluated through comparison to human-coded matches from previous studies of results reporting rates. TrialScout had a sensitivity of 92.5% and a specificity of 81.2% compared to human coders. Manual review of 200 cases where TrialScout disagreed with human researchers showed that a majority (123/200, 61.5%; 95% CI, 54.4%–68.3%) of disagreements were due to human errors.”

How do the two tools compare?

Trials to Publications co-author Neil Smalheiser commented that:

“It would be interesting to do a detailed comparison of TrialScout and Trials to Publications, however, we have not done so, and do not have plans to do so in the near future. We would certainly cooperate with anyone who wants to do a comparison.”

TrialScout lead author Love von Schreeb commented that:

“TrialScout screens the actual abstract and bases its prediction on that rather than explicit trial of references or metadata comparisons. That makes it more accurate. On the other hand this comes at a higher cost.”

“Right now TrialScout replies on expensive proprietary models that makes running at scale just that - expensive. [Trials to Publications] is also transparent in nature, we know the weights and what they correspond to. The interpretability is high. In contrast, LLMs are black boxes essentially.”

How can the two tools be used in practice?

Trials to Publications co-author Neil Smalheiser wrote that:

“I will say that a number of users have requested that we send them the entire dataset of trials and predicted publications, rather than querying via the web, and we have accommodated such requests. Our pre-computed dataset is continually updated.”

TrialScout’s Love von Schreeb has already demonstrated his tool’s capacity at scale by searching for the results of a sample of 9,600 trials. (He found results for 63.6% of these trials.)

He wrote that:

“TrialScout is open source and easy to integrate into workflows of potential users, including automated workflows. We also host a public API, though this is currently poorly documented.”

“Going forward, we plan to further use and update the tool, provided there are interested users and funding. TrialScout can quite easily be adapted to other trial registries. The website itself has many potential areas for improvements such as bulk upload. The accuracy can be further improved by validating the tool with more recent models.”

What comes next?

Give both tools a test drive, links here:

• Trials to Publications tool and publication; co-author: Neil Smalheiser 

• TrialScout tool and publication; lead author: Love von Schreeb

Both researchers welcome enquiries from people who want to use or adapt their tools.