Attention2Minority: A salient instance inference-based multiple instance learning for classifying small lesions in whole slide images

2.1. Multiple instance learning in WSIs

Multiple instance learning (MIL) represents a weakly supervised learning approach in which labels are assigned to collections of instances, referred to as “bags”, as opposed to the conventional machine learning practice of assigning labels to individual instances [26]. Typically, it performs binary classification on a bag by predicting the presence of “positive” instances. In 2018, Courtiol et al. conducted a pioneering study on predicting slide-level outcomes of Whole Slide Images (WSI) using MIL [19]. In their work, the focus was on predicting the presence of tumor metastasis in sentinel lymph nodes. Within this context, a slide was defined as a bag and patches containing tumor lesions were defined as positive instances. Their MIL approach addressed the GPU memory limitation issue in analyzing giga-pixel WSIs by partitioning WSI into small patches, and then embedding the patches into vectors. Following this work, multiple MIL methodologies were introduced for WSIs. These approaches can be categorized into two main groups: critical-instances-based approaches [8, 9, 19, 24, 27] and attention-based approaches [10, 20–22, 28]. Critical-instances-based methods usually predict slide-level outcome from a few selected patches with high confidence on-the-fly. For example, Campanella et al. [8] selected patches with top-k highest probability predicted by their model and aggregated the selected patches using recurrence neural network, and Li et al. [9] predicted one critical patch and then aggregated other patches based on their similarity to the critical patch. On the other hand, attention-based approaches learn attention weights to highlight important patches during aggregation without selecting patches. Ilse et al. initially presented attention-based MIL (ABMIL) for natural images and tabular data. Lu et al. [22] extended the use of ABMIL to WSIs by adding clustering constraint. Zhang et al. [20] further partitioned a slide into multiple pseudo-bags of patches and performed ABMIL on each pseudo-bag separately. Other studies [10, 28] emphasized the importance of contextual relationship among the patches. As a result, these studies proposed to aggregate patches within a WSI using Transformer models [29]. Overall, it is challenging to accurately select critical-instances in weakly supervised learning, hence most of the recent studies utilize attention-based approach. However, the sensitivity of the existing MIL models are still moderate, especially for those WSIs with small lesions (i.e., corresponding to a few positive patches). In this study, we propose a salient instance inference-based MIL to alleviate this issue.

2.2. Deep learning in medical image analysis

In recent years, the application of deep learning has gained prominence in addressing similar demands for medical image analysis across various domains such as CT, X-ray, and MRI. Wen et al. introduced an attention capsule sampling network to enhance key slice in chest CT images for COVID-19 diagnosis [30]. Zhang et al. presented a double-branch attention network to perform tumor segmentation and metastasis classification simultaneously, in which two related tasks can share feature learning [31]. Leveraging CT imaging features for Osteoporotic Vertebral Fracture classification in X-rays, Wang et al. proposed a multi-modal neural network [32]. Besides, non-neural network methodologies like feature engineering and optimization algorithms have been explored in X-ray image segmentation tasks [33, 34]. Extending beyond radiology images, successful studies have emerged in clinical photographs and endoscopic images. For example, Su et al. devised an ensemble classifier to combine different neural network backbones for gastrointestinal disease from gastrointestinal endoscopic images [35]. Hu et al. proposed polyp extraction and reflection removal methods to highlight salient region in endoscopy images [36]. Exploring skin disease classification, Zhou et al. explore the impact of background color on convolutional neural networks in skin photograph analysis [37].

In contrast to these tasks, WSI classification poses a more challenging problem with exponentially higher image size and tiny portion of lesion areas, which makes common convolutional neural networks inadequate. Therefore, multiple instance learning has become the standard strategy for classifying WSIs.

3.1. Dataset

Our study is developed based on the Camelyon16 dataset [18]. Camelyon16 is a publicly available hematoxylin and eosin (H&E) whole slide image (WSI) dataset for breast cancer lymph node metastases classification. The training set includes 271 WSIs, and the hold-out testing set has 129 WSIs divided into two classes: normal and tumor. In this weakly supervised learning study, only the WSI-level labels (i.e., normal or tumor) are available.

3.2. Multiple instance learning for WSI classification

In multiple instance learning (MIL) formulation, a bag refers to the set of nlabelled instances in a WSI. Each bag’s label is determined by the presence of positive instances. In other words, a positive bag needs to contain at least one positive instance and a negative bag needs to exclusively contain negative instances. Therefore, we formulate the MIL problem as:

where $Y$ is the label of the bag, $y_i\in \{0,1\}$, for $i=1,\dots ,m$, is the label of instance inside the bag, and $m$ is the number of instances inside the bag [38]. To predict the bag-level label $Y$, the MIL models need to analyze and aggregate the instances, and then reach the conclusion in the form of:

where $f(\cdot )$ is the feature extraction function for instances, $\sigma (\cdot )$ is the aggregation operator, and $g(\cdot )$ is the bag-level classifier.

For a variety of reasons, the WSI classification inevitably develops into the MIL formulation. First, the raw images of WSIs with gigapixel resolution seldom fit in the available graphics processor memory. As a result, the typical paradigm entails cropping a WSI into numerous image patches and then extracting feature embeddings from the patches using a feature extractor (i.e., the $f(\cdot )$ function). This enables us to fit all embedding into a GPU memory. Second, in WSI classification, such as normal/tumor, bags containing embeddings from lesion areas might be deemed positive because they are only present in tumor WSIs. Third, as it is painstakingly difficult to annotate WSIs, we often have access to the WSI-level labels (i.e., $Y$) but not the instance-level labels (i.e., $y_i$). The ability of MIL to address all three problems makes it a suitable candidate for WSI-level classification.

In the case of tumor WSI with small lesions, the WSIs typically contain dominant regions of normal tissues, making it difficult for the MIL model to focus on the patches corresponding to small lesions [22]. For instance, in Camelyon16 dataset, the ratio of tumor region to the entire tissue region of a tumor WSI could be lower than 0.5%.

3.3. Motivation of salient instance inference

To depict the motivation of our method, we take tumor WSIs classification as an example. Consider a tumor WSI (positive bag) with a small number of tumor patch feature embeddings (positive instances). Let $\left\{{\text{u}}_1,{\text{u}}_2,\dots ,{\text{u}}_{\text{p}},{\text{v}}_1,{\text{v}}_2,\dots ,{\text{v}}_{\text{q}}\right\}$ denote $m=p+q$ instances of the WSI, where $u_i\in {ℝ}^D$ and $v_j\in {ℝ}^D$, for $i=1,2,\dots ,p$ and $j=1,2,\dots ,q$, denote the negative and positive instances, respectively. For the WSI with small lesions, we assume that $p>>q$.

Assuming that our MIL aggregation function is weighted average:

where $z\in {ℝ}^D$ is the bag-level representation and $a_i$ and $b_j\in ℝ$ for $i=1,2,\dots ,p$ and $j=1,2,\dots ,q$ are the weights emphasizing the positive instances and ${\sum }_{\text{i}=1}^{\text{p}}{\text{a}}_{\text{i}}+{\sum }_{\text{j}=1}^{\text{q}}{\text{b}}_{\text{j}}=1$. In other words, the bag-level representation is a convex combination of all negative and positive instances in a bag. Considering the morphological difference between tumor and normal tissues, the positive and negative instances should lay in different sub-spaces. However, since $p>>q$, if the attention scores of the negative instances are not sufficiently small, the resulting convex combination $z$ will easily fall into the negative subspace. As a result, the positive bag-level representation will be inseparable from the negative bag. This effect is demonstrated in Section 3.1. On the other hand, if we intentionally select possible positive instances in the bag, therefore increasing the density of positive instances in the positive bags, we can make the bag-level representation more separable from the negative bag and derive an easier classification problem. Please see Fig. 2 as a toy example when the instance dimension $D=2$.

For a naïve solution, if we can have an accurate instance-level positive instance prediction model, we can easily select high probability instances to achieve our goal. However, this type of prediction model usually requires fully supervised training with reliable instance labels, which is impossible for most WSI classification cases. Therefore, a weakly supervised patch selection methodology is crucial to tackling the MIL problem.

3.4. Salient instance inference

Our salient instance inference (Sii) is composed of the following three steps: (i) learning representative instances from negative (normal) WSIs in the training set; (ii) comparing similarities between the instances of an input WSI and the learned representative instances; (iii) selecting instances with low similarity scores (i.e., high saliency) from the input WSI to form a bag.

3.5. MIL classification

Given the salient bag derived from the Sii process, we will aggregate the bag using MIL aggregation and finally make WSI-level prediction. The MIL aggregation function we are using is the attention-based MIL (ABMIL) [21]. Given a salient bag $B=\left\{q_1,\dots ,q_t\right\}$ for a WSI, we have the formulation:

where:

where $V\in {ℝ}^{D\times L}$, $U\in {ℝ}^{D\times L}$ and $W\in {ℝ}^{L\times D}$ are the learnable weights of fully connection networks. Each instance $q_i\in {ℝ}^{D\times 1}$ is scaled by attention weight $a_i\in ℝ$ and summed to yield the representation vector $z\in {ℝ}^{D\times 1}$ for bag $B$. Finally, we apply a separate fully connection network on to $z$ to make WSI-level prediction. The overview of the complete SiiMIL is shown in Fig. 3.

3.6. Implementation details and evaluation metrics

In data preprocessing stage, we cropped each WSI into 224 × 224 image patches without overlapping under 20 × magnification. We applied color thresholding method to extract foreground tissue patches and discarded background patches that contained less than 75% foreground tissue. Then, we embedded each patch with a ResNet50 model [40] pre-trained on ImageNet dataset [41]. The ResNet50 was truncated after the third residual block which embedded patches into 1024-dimension feature embeddings.

During the training of our MIL model, we used Adam Optimizer [42] with 0.0002 learning rate and 0.00001 weight decay. To avoid overfitting, we applied early stopping strategy with the patience of 10 epochs and the maximum training of 100 epochs depending on the validation loss. In other words, the training was halted if the validation loss do not decrease for 10 epochs.

We conducted five-fold Monte Carlo cross-validation on the training set with random splits in the ratio of training: validation set = 90:10. Then, we selected the model with the best validation AUC and evaluated it on the official testing set of Camelyon16. Our model contains three primary hyperparameters: $t_i$, $K$ and $r$. To clarify, $t_i$ controls how many representative instances that we select from each normal WSI using CUR-decomposition, $K$ controls the $topK$ operator during the cosine similarity comparison, and $r$ controls the top $r\%$ salient instances selection. We tuned and selected the hyperparameters based on the validation AUC and decided to choose $t_i=100$, $K=150$, and $r=0.3$.

We used five evaluation metrics to evaluate the WSI-level classification performance, which are accuracy, AUC (of ROC), precision, recall, and F1-score. They are defined as follows:

where $TPR$ stands for true positive rate, $FPR$ stands for false positive rate, and $t$ indicates classification threshold changing from 0 to 1.

where $\text{TP}$, $\text{TN}$, $\text{FP}$, and $\text{FN}$ stand for true positives, true negatives, false positives, and false negatives.

Moreover, to evaluate the effect of Sii module, we defined the metric “tumor instance rate” ($\text{TIR}$) as:

for a bag of instances.

Our code is available at: https://github.com/JoeSu666/SiiMIL.

4.1. Results of tumor instance inference

To validate the effectiveness of Salient instance embeddings inference, we grouped the WSIs based on their TIR in four groups: (i) < 0.5%, (ii) 0.5–1%, (iii) 1–10%, (iv) >=10%. In Fig. 4, we exhibited the performance of salient instance inference across different groups by comparing the tumor instance rate (TIR) of original tumor WSIs and the tumor bags after applying Sii.

4.2. Results on WSI classification

In Table 1, we presented our performance on Camelyon16 dataset and compared them with a set of state-of-the-arts MIL-based WSI classification models [9, 10, 20–22]. The proposed SiiMIL achieved 0.9225 AUC which outperforms the comparison models. To be noticed, the ABMIL was the baseline method of SiiMIL that didn’t perform Sii and applied the attention module to all the instances of the WSIs. We observed more than 8 points improvement in AUC from the baseline to our SiiMIL.

Due to the remarkable bag TIR improvement by using the Sii, our SiiMIL also achieved the best recall compared to the comparison methods, which is crucial in medical domain. To demonstrate the effect of tumor instance rate on recall, we grouped the tumor WSIs in different TIRs: (i) < 0.5%, (ii) 0.5–1%, (iii) 1–10%, (iv) >=10%. Then, we evaluated the MIL models’ recall in every group of tumor WSIs (see Fig. 5). Here, we observed that our SiiMIL achieved outstanding accuracy in all of the groups.

4.3. Salient instance inference as an addition to MIL models

Other than an integrated MIL model, our Sii module can be also treated as a powerful addition to other MIL models. In Table 2, we presented the performance of some MIL models that used ResNet50 feature extractor as our model did. In parallel, we presented their performance where we applied Sii to form bags before MIL process. We observed that, by feeding salient bags yielded by our Sii, most of the MIL methods (i.e., ABMIL, CLAM, and DTFD-MIL) achieved improved performance. These methods, like ours, assume the permutation invariance property among instances, making them resilient to our instance selection strategy. Conversely, TransMIL, which leverages a Transformer architecture to capture contextual information, experienced a decline in performance. This outcome is reasonable, as our instance selection strategy potentially disrupts the contextual relationships within bags.

4.4. Ablations studies

We further conducted ablation studies to investigate the effects of three primary hyperparameters to our model: $t_i$, $k$, and $r$. To investigate the effect of numbers of CUR-based keys, we created key sets by setting $t_i=50,100,200,300$ and 400, and named the derived key sets as: (i) CUR-50, (ii) CUR-100, (iii) CUR-200, (iv) CUR-300, and (v) CUR-400. Then, we applied those key sets separately in our SiiMIL and evaluated the resulted mean validation AUCs based on a five-fold cross-validation. From Fig. 6a, we found that our method achieved best performance when we used the CUR-100 key sets.

In Fig. 6b, we presented the mean validation AUCs of our method when we used different $\left(k,r\right)$ settings. We observed that our method achieved the best performance when we chose $k=150$ and $r=0.3$.

4.5. Interpretability of SiiMIL

Fig. 7 depicts a normal WSI as well as the position of the representative negative instances (keys). Fig. 7a depicts a typical WSI from our dataset. The location of the 100 keys selected by the proposed representation learning methodology are depicted in Fig. 7b. It is clear that these 100 keys correspond to different anatomical structures within the example WSI. Fig. 7c depicts a heatmap in which each instance embedding is colored the same as its most similar key (measured using cosine similarity in the feature space).

Our SiiMIL is also of high interpretability which is important for a diagnostic tool. To demonstrate the interpretability of SiiMIL, we visualized some small tumor WSIs’ attention heatmaps that produced by the attention module of our model in Fig. 8 (middle column). As comparison, we also visualize the corresponding attention heatmaps produced by the MIL model without using our Sii (i.e., original attention-based MIL) in Fig. 8 (right column). In the heatmaps, the regions with high brightness correspond to the instances that received high attention weights. We can clearly observe that the SiiMIL is sensitive to the tumor regions during predictions, even on the small tumor WSIs.