A novel end-to-end clinical trial matching system called TrialGPT can reduce patient recruitment screening time by 42.6%, according to researchers at the U.S. National Institutes of Health (NIH), where the algorithm was developed and tested.
The zero-shot large language models (LLMs)-driven system is designed to discern clinical trial inclusion and exclusion criteria and analyze a person's medical history to ensure that a match satisfies both patient needs and trial requirements, according to Drs. Qiao Jin at the NIH, Zifeng Wang from the University of Illinois Urbana-Champaign, and colleagues who led a user study.
"In this study, we focus on the patient-centric 'patient-to-trial' scheme because such a model can empower individual patients as well as referral offices to explore a large set of potentially eligible clinical trials," the authors wrote for an open access article published November 18 in Nature Communications.
Eligibility assessment
The concept of TrialGPT consists of three key components for streamlining the process of clinical trial patient matching:
- Retrieval by filtering out most clinical trials that would be irrelevant to the patient, using keyword generation and hybrid-fusion retrieval.
- Matching by predicting a person's clinical trial eligibility at the criterion level, with natural-language explanations showing the relevance of the patient to the criterion, locations of relevant sentences in the patient note that are relevant to the target criterion, and the eligibility classification indicating whether the patient meets this criterion.
- Ranking by a trial-level scoring system based on patient eligibility
Researchers evaluated the system using patient summaries and clinical trials from three publicly available cohorts. Experimental results showed that TrialGPT retrieval recalled over 90% of relevant clinical trials using less than 6% of the candidate clinical trials, Jin and colleagues noted. The model also generated better keywords for trial retrieval than those produced by four human clinicians, they added.
TrialGPT matching achieved a prediction accuracy of 0.873, close to the expert performance (0.887–0.900), the authors added. They also reported that the most effective features of GPT-4-based TrialGPT for ranking were the LLM-aggregated eligibility scores. They achieved a normalized discounted cumulative gain at rank 10 (NDCG@10) of 0.725 and precision at rank 10 (P@10) of 0.428.
In addition, a pilot user study (though of limited scope in sample size) mimicked the actual clinical trial matching task at the U.S. National Cancer Institute. That evaluation measured the performance of TrialGPT with two experts against the performance the experts alone. The researchers found that when clinicians used TrialGPT, they spent at least 40% less time screening patients but maintained the same level of accuracy, according to the NIH.
The pilot "offers insights into the potential benefits of LLMs for assisting clinical trial matching and provides impetus to conduct larger-scale prospective evaluations regarding the impact of LLM-assisted clinical workflows in future studies," the authors wrote. The study was also co-authored by collaborators from Albert Einstein College of Medicine in New York City, the University of Pittsburgh, and the University of Maryland in College Park, MD.
Fewer failed trials?
It has been estimated that one in five clinical trials don’t recruit the required number of people, and nearly all trials exceed their expected recruitment timelines. A system such as TrialGPT could be more patient-centric and reduce the number of clinical trials that fail due to insufficient enrollment, according to the authors.
“This study shows we can responsibly leverage AI technology so physicians can connect their patients to a relevant clinical trial that may be of interest to them with even more speed and efficiency,” stated NIH National Library of Medicine Acting Director Stephen Sherry, PhD, in an NIH blog post.
TrialGPT relies on OpenAI’s GPT series LLMs such as GPT-3.5 and GPT-4 as its backbone model. However, GPT-4 is currently the most capable LLM that can only be accessed through commercial applications or API, Jin and colleagues noted. Future studies should explore using and fine-tuning other open-source LLMs as alternatives, Jin and colleagues wrote.
A potentially more effective clinical trial matching system from the NIH along with commercial investments in software designed with similar goals come as decentralized clinical trials (DCTs) have emerged in hopes of shortening research and development (R&D) time for regulatory approval of new drugs and drug combinations and devices.
New trial designs
In one step, the U.S. Food and Drug Administration (FDA) released draft guidance for comment related to integrating research into routine clinical practice (as part of its real-world evidence [RWE] program), saying healthcare professionals (HCPs) are collecting data during routine clinical practice interactions that researchers may be able to use to satisfy trial data requirements.
"Such trials have sometimes been referred to as point of care trials or large simple trials," the FDA said in an overview of the guidance document, titled "Integrating Randomized Controlled Trials [RCTs] for Drug and Biological Products Into Routine Clinical Practice."
"Like decentralized clinical trials, which aim to bring trial-related activities to patients’ homes or other convenient locations, such RCTs may improve convenience and accessibility for participants and allow for enrollment of more representative populations, resulting in more generalizable trial results," added the FDA. "Leveraging established health care institutions and existing clinical expertise in the medical community can reduce startup times and speed up enrollment."
In the past, traditional clinical trials were confined to specific geographic locations. However, the FDA has enabled new trial designs for investigational products and medical devices -- incorporating mobile research units, community healthcare practices, and even remote and virtual participation -- that has created momentum for decentralized trials and may accelerate access to potential trial participants.
By enabling remote participation, DCTs may enhance convenience for trial participants, reduce the burden on caregivers, expand access to more patient populations, improve trial efficiency, and facilitate research on rare diseases and diseases affecting populations with limited mobility, the FDA said.
Clinical laboratory demands
The FDA's recent nonbinding guidances (one a draft and one a final summarized in a September 17 FDA roundup) suggest that clinical laboratory teams may need to be prepared for changing testing, data collection, and training needs related to trial-specific activities that could soon be integrated into community healthcare practices, facilitated by mobile research units, and provided through home-collection kits designated for clinical laboratory processing.
