CT-based multimodal deep learning for non-invasive overall survival prediction in advanced hepatocellular carcinoma patients treated with immunotherapy

Yujia Xia; Jing Zhou; Xiaolei Xun; Ting Wei; Ruitian Gao; Bobby Reddy; Chao Liu; Geoffrey Kim; Zhangsheng Yu

doi:10.1186/s13244-024-01784-8

journal article Open Access Aug 26, 2024

CT-based multimodal deep learning for non-invasive overall survival prediction in advanced hepatocellular carcinoma patients treated with immunotherapy

Yujia Xia Jing Zhou

Xiaolei Xun Ting Wei

Ruitian Gao Bobby Reddy

Chao Liu

Geoffrey Kim Zhangsheng Yu

Insights into Imaging Vol. 15 No. 1 · Springer Science and Business Media LLC

View at Publisher Save 10.1186/s13244-024-01784-8

Abstract

Abstract
Objectives
To develop a deep learning model combining CT scans and clinical information to predict overall survival in advanced hepatocellular carcinoma (HCC).

Methods
This retrospective study included immunotherapy-treated advanced HCC patients from 52 multi-national in-house centers between 2018 and 2022. A multi-modal prognostic model using baseline and the first follow-up CT images and 7 clinical variables was proposed. A convolutional-recurrent neural network (CRNN) was developed to extract spatial-temporal information from automatically selected representative 2D CT slices to provide a radiological score, then fused with a Cox-based clinical score to provide the survival risk. The model’s effectiveness was assessed using a time-dependent area under the receiver operating curve (AUC), and risk group stratification using the log-rank test. Prognostic performances of multi-modal inputs were compared to models of missing modality, and the size-based RECIST criteria.

Results
Two-hundred seven patients (mean age, 61 years ± 12 [SD], 180 men) were included. The multi-modal CRNN model reached the AUC of 0.777 and 0.704 of 1-year overall survival predictions in the validation and test sets. The model achieved significant risk stratification in validation (hazard ratio [HR] = 3.330, p = 0.008), and test sets (HR = 2.024, p = 0.047) based on the median risk score of the training set. Models with missing modalities (the single-modal imaging-based model and the model incorporating only baseline scans) can still achieve favorable risk stratification performance (all p < 0.05, except for one, p = 0.053). Moreover, results proved the superiority of the deep learning-based model to the RECIST criteria.

Conclusion
Deep learning analysis of CT scans and clinical data can offer significant prognostic insights for patients with advanced HCC.

Critical relevance statement
The established model can help monitor patients’ disease statuses and identify those with poor prognosis at the time of first follow-up, helping clinicians make informed treatment decisions, as well as early and timely interventions.

Key Points

An AI-based prognostic model was developed for advanced HCC using multi-national patients.

The model extracts spatial-temporal information from CT scans and integrates it with clinical variables to prognosticate.

The model demonstrated superior prognostic ability compared to the conventional size-based RECIST method.

Graphical Abstract

Topics

No keywords indexed for this article. Browse by subject →

References

40

[1]

Epidemiology of Hepatocellular Carcinoma

Katherine A. McGlynn, Jessica L. Petrick, Hashem B. El-Serag

Hepatology 2020 10.1002/hep.31288

[2]

Siegel RL, Miller KD, Wagle NS, Jemal A (2023) CA Cancer J Clin. https://doi.org/10.3322/caac.21763 10.3322/caac.21763

[3]

Llovet JM, Castet F, Heikenwälder M et al (2021) Immunotherapies for hepatocellular carcinoma. Nat Rev Clin Oncol 19:151–172. https://doi.org/10.1038/s41571-021-00573-2 10.1038/s41571-021-00573-2

[4]

Bruix J, Gores GJ, Mazzaferro V (2014) Hepatocellular carcinoma: clinical frontiers and perspectives. Gut 63:844–855. https://doi.org/10.1136/gutjnl-2013-306627 10.1136/gutjnl-2013-306627

[5]

Pinter M, Scheiner B, Peck-Radosavljevic M (2021) Immunotherapy for advanced hepatocellular carcinoma: a focus on special subgroups. Gut 70:204–214. https://doi.org/10.1136/gutjnl-2020-321702 10.1136/gutjnl-2020-321702

[6]

Qin S, Ren Z, Meng Z et al (2020) Camrelizumab in patients with previously treated advanced hepatocellular carcinoma: a multicentre, open-label, parallel-group, randomised, phase 2 trial. Lancet Oncol 21:571–580. https://doi.org/10.1016/s1470-2045(20)30011-5 10.1016/s1470-2045(20)30011-5

[7]

Jin W, Luo Q (2022) When artificial intelligence meets PD-1/PD-L1 inhibitors: population screening, response prediction and efficacy evaluation. Comput Biol Med 145:105499. https://doi.org/10.1016/j.compbiomed.2022.105499 10.1016/j.compbiomed.2022.105499

[8]

RECIST — learning from the past to build the future

Saskia Litière, Sandra Collette, Elisabeth G. E. de Vries et al.

Nature Reviews Clinical Oncology 2016 10.1038/nrclinonc.2016.195

[9]

Eisenhauer E, Therasse P, Bogaerts J et al (2009) New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1). Eur J Cancer 45:228–247. https://doi.org/10.1016/j.ejca.2008.10.026 10.1016/j.ejca.2008.10.026

[10]

Bruix J (2021) Endpoints in clinical trials for liver cancer and their value in evidence-based clinical decision making: an unresolved Gordian knot. J Hepatol 74:1483–1488. https://doi.org/10.1016/j.jhep.2021.01.033 10.1016/j.jhep.2021.01.033

[11]

Coudray N, Tsirigos A (2020) Deep learning links histology, molecular signatures and prognosis in cancer. Nat Cancer 1:755–757. https://doi.org/10.1038/s43018-020-0099-2 10.1038/s43018-020-0099-2

[12]

Jiang C, Chen K, Teng Y et al (2022) Deep learning–based tumour segmentation and total metabolic tumour volume prediction in the prognosis of diffuse large B-cell lymphoma patients in 3D FDG-PET images. Eur Radiol 32:4801–4812. https://doi.org/10.1007/s00330-022-08573-1 10.1007/s00330-022-08573-1

[13]

Saillard C, Schmauch B, Laifa O et al (2020) Predicting survival after hepatocellular carcinoma resection using deep learning on histological slides. Hepatology 72:2000–2013. https://doi.org/10.1002/hep.31207 10.1002/hep.31207

[14]

Shi J, Wang X, Ding G et al (2020) Exploring prognostic indicators in the pathological images of hepatocellular carcinoma based on deep learning. Gut 70:951–961. https://doi.org/10.1136/gutjnl-2020-320930 10.1136/gutjnl-2020-320930

[15]

Liang J, Zhang W, Yang J et al (2023) Deep learning supported discovery of biomarkers for clinical prognosis of liver cancer. Nat Mach Intell 5:408–420. https://doi.org/10.1038/s42256-023-00635-3 10.1038/s42256-023-00635-3

[16]

Chaudhary K, Poirion O, Lu L (2018) Deep learning-based multi-omics integration robustly predicts survival in liver cancer. Clin Cancer Res 24:1248–1259. https://doi.org/10.1158/1078-0432.ccr-17-0853 10.1158/1078-0432.ccr-17-0853

[17]

Isensee F, Jaeger PF, Kohl S et al (2020) nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods 18:203–211. https://doi.org/10.1038/s41592-020-01008-z 10.1038/s41592-020-01008-z

[18]

Antonelli M, Reinke A, Bakas S et al (2022) The medical segmentation Decathlon. Nat Commun 13:4128. https://doi.org/10.1038/s41467-022-30695-9 10.1038/s41467-022-30695-9

[19]

Shao J, Zhong B (2003) Last observation carry-forward and last observation analysis. Stat Med 22:2429–2441. https://doi.org/10.1002/sim.1519 10.1002/sim.1519

[20]

Tan M, Le QV (2019) EfficientNet: rethinking model scaling for convolutional neural networks. In: Proceedings of the 36th International Conference on Machine Learning, ICML, Long Beach, 9–15 June 2019.

[21]

Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber

Neural Computation 1997 10.1162/neco.1997.9.8.1735

[22]

Harrell FE, Lee KL, Califf RM, Pryor DB, Rosati RA (1984) Regression modelling strategies for improved prognostic prediction. Stat Med 3:143–152. https://doi.org/10.1002/sim.4780030207 10.1002/sim.4780030207

[23]

Chambless LE, Diao G (2006) Estimation of time-dependent area under the ROC curve for long-term risk prediction. Stat Med 25:3474–3486. https://doi.org/10.1002/sim.2299 10.1002/sim.2299

[24]

Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization

Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das et al.

2017 IEEE International Conference on Computer Vis... 10.1109/iccv.2017.74

[25]

He X, Li K, Wei R et al (2023) A multitask deep learning radiomics model for predicting the macrotrabecular-massive subtype and prognosis of hepatocellular carcinoma after hepatic arterial infusion chemotherapy. Radiol Med 128:1508–1520. https://doi.org/10.1007/s11547-023-01719-1 10.1007/s11547-023-01719-1

[26]

Meng Y, Zhang X, Zhang B et al (2023) Deep learning nomogram based on Gd-EOB-DTPA MRI for predicting early recurrence in hepatocellular carcinoma after hepatectomy. Eur Radiol 33:4949–4961. https://doi.org/10.1007/s00330-023-09419-0 10.1007/s00330-023-09419-0

[27]

Xu Y, Zhao J, Chen Q et al (2023) Contrastive learning for preoperative early recurrence prediction of hepatocellular carcinoma with liver CT image and tumor mask. In: 2023, 45th Annual international conference of the IEEE engineering in medicine & biology society (EMBC), IEEE, Piscataway. https://doi.org/10.1109/embc40787.2023.10340893 10.1109/embc40787.2023.10340893

[28]

Zhang L, Xia W, Zhang Y et al (2020) Deep learning predicts overall survival of patients with unresectable hepatocellular carcinoma treated by transarterial chemoembolization plus sorafenib. Front Oncol. https://doi.org/10.3389/fonc.2020.593292 10.3389/fonc.2020.593292

[29]

MRI radiomics based on deep learning automated segmentation to predict early recurrence of hepatocellular carcinoma

Hong Wei, Tianying Zheng, Xiaolan Zhang et al.

Insights into Imaging 2024 10.1186/s13244-024-01679-8

[30]

Pfister D, Núñez NG, Pinyol R et al (2021) NASH limits anti-tumour surveillance in immunotherapy-treated HCC. Nature 592:450–456. https://doi.org/10.1038/s41586-021-03362-0 10.1038/s41586-021-03362-0

[31]

Forner A, Llovet JM, Bruix J (2012) Hepatocellular carcinoma. Lancet 379:1245–1255. https://doi.org/10.1016/s0140-6736(11)61347-0 10.1016/s0140-6736(11)61347-0

[32]

Shimada M, Takenaka K, Gion T et al (1996) Prognosis of recurrent hepatocellular carcinoma: A 10-year surgical experience in Japan. Gastroenterology 111:720–726. https://doi.org/10.1053/gast.1996.v111.pm8780578 10.1053/gast.1996.v111.pm8780578

[33]

Wada Y, Nakashima O, Kutami R, Yamamoto O, Kojiro M (1998) Clinicopathological study on hepatocellular carcinoma with lymphocytic infiltration. Hepatology 27:407–414. https://doi.org/10.1002/hep.510270214 10.1002/hep.510270214

[34]

Wang H, Liu Y, Xu N et al (2022) Development and validation of a deep learning model for survival prognosis of transcatheter arterial chemoembolization in patients with intermediate-stage hepatocellular carcinoma. Eur J Radiol 156:110527. https://doi.org/10.1016/j.ejrad.2022.110527 10.1016/j.ejrad.2022.110527

[35]

Preoperative CT-based Deep Learning Model for Predicting Disease-Free Survival in Patients with Lung Adenocarcinomas

Hyungjin Kim, Jin Mo Goo, Kyung Hee Lee et al.

Radiology 2020 10.1148/radiol.2020192764

[36]

Parmar C, Velazquez ER, Leijenaar RTH et al (2014) Robust radiomics feature quantification using semiautomatic volumetric segmentation. PLoS One 9:e102107. https://doi.org/10.1371/journal.pone.0102107 10.1371/journal.pone.0102107

[37]

Deep Learning Predicts Lung Cancer Treatment Response from Serial Medical Imaging

Yiwen Xu, Ahmed Hosny, Roman Zeleznik et al.

Clinical Cancer Research 2019 10.1158/1078-0432.ccr-18-2495

[38]

De Frutos JP, Pedersen A, Pelanis E et al (2023) Learning deep abdominal CT registration through adaptive loss weighting and synthetic data generation. PLoS One 18:e0282110. https://doi.org/10.1371/journal.pone.0282110 10.1371/journal.pone.0282110

[39]

Hofmanninger J, Prayer F, Pan J, Röhrich S, Prosch H, Langs G (2020) Automatic lung segmentation in routine imaging is primarily a data diversity problem, not a methodology problem. Eur Radiol Exp 4:50. https://doi.org/10.1186/s41747-020-00173-2 10.1186/s41747-020-00173-2

[40]

Schütte K, Schinner R, Fabritius MP et al (2020) Impact of extrahepatic metastases on overall survival in patients with advanced liver dominant hepatocellular carcinoma: a subanalysis of the SORAMIC trial. Liver Cancer 9:771–786. https://doi.org/10.1159/000510798 10.1159/000510798

Cited By

20

Machine learning methods for histopathological image analysis: Updates in 2024

Daisuke Komura, Mieko Ochi · 2025

Computational and Structural Biotec...

Metrics

20

Citations

40

References

Details

Published: Aug 26, 2024
Vol/Issue: 15(1)
License: View

Authors

Chinese Academy of Sciences

Funding

National Natural Science Foundation of China Award: 12171318

Science and Technology Innovation Plan Of Shanghai Science and Technology Commission Award: 21ZR1436300

Shanghai Jiao Tong University STAR Grant Award: 20190102

Cite This Article

Yujia Xia, Jing Zhou, Xiaolei Xun, et al. (2024). CT-based multimodal deep learning for non-invasive overall survival prediction in advanced hepatocellular carcinoma patients treated with immunotherapy. Insights into Imaging, 15(1). https://doi.org/10.1186/s13244-024-01784-8

CT-based multimodal deep learning for non-invasive overall survival prediction in advanced hepatocellular carcinoma patients treated with immunotherapy

You May Also Like