Survival Prediction Algorithm for Gastric Cancer Patients Based on Multi-Modal Multi-Instance Learning

Survival prediction is vital for treating gastric cancer patients. To overcome the shortcomings of classic survival prediction algorithms based on histopathology images, such as pixel-level label missing, high information volume, and single modality, a survival prediction algorithm for gastric cancer patients based on multi-modal multi-instance learning is proposed. Firstly, hidden features behind the clinical data and histopathological image are extracted using MLP and self-supervised learning method SimCLR, respectively. Subsequently, a globally-aware multi-instance learning approach is employed to extract bag-level embeddings at high resolution and instance-level embeddings of histopathological images at low resolution are obtained using an average pooling method. Finally, a multi-modal fusion approach is utilized to integrate bag-level embeddings, instance-level embeddings, and clinical data features, thus facilitating information interaction between different modalities and effective utilization of image information at various resolutions. The experimental results on the pathological database of gastric cancer patients from Yunnan Cancer Hospital show that, compared to traditional multi-instance learning methods, the proposed algorithm improves the prediction accuracy by 5.6-10.0 percentage points and 4.1-7.5 percentage points from 5×and 20×histopathological images, respectively. Furthermore, compared to conventional multi-modal fusion methods, the proposed algorithm exhibits an accuracy improvement of 3.5-4.9 percentage points.