Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2024 Oct 1.
Published in final edited form as: Med Image Anal. 2023 Aug 2;89:102918. doi: 10.1016/j.media.2023.102918

Segment Anything Model for medical image analysis: an experimental study

Maciej A Mazurowski a,b,c,d, Haoyu Dong b,*, Hanxue Gu b, Jichen Yang b, Nicholas Konz b, Yixin Zhang b
PMCID: PMC10528428  NIHMSID: NIHMS1927264  PMID: 37595404

Abstract

Training segmentation models for medical images continues to be challenging due to the limited availability of data annotations. Segment Anything Model (SAM) is a foundation model trained on over 1 billion annotations, predominantly for natural images, that is intended to segment user-defined objects of interest in an interactive manner. While the model performance on natural images is impressive, medical image domains pose their own set of challenges. Here, we perform an extensive evaluation of SAM’s ability to segment medical images on a collection of 19 medical imaging datasets from various modalities and anatomies. In our experiments, we generated point and box prompts for SAM using a standard method that simulates interactive segmentation. We report the following findings: (1) SAM’s performance based on single prompts highly varies depending on the dataset and the task, from IoU=0.1135 for spine MRI to IoU=0.8650 for hip X-ray. (2) Segmentation performance appears to be better for well-circumscribed objects with prompts with less ambiguity such as the segmentation of organs in computed tomography and poorer in various other scenarios such as the segmentation of brain tumors. (3) SAM performs notably better with box prompts than with point prompts. (4) SAM outperforms similar methods RITM, SimpleClick, and FocalClick in almost all single-point prompt settings. (5) When multiple-point prompts are provided iteratively, SAM’s performance generally improves only slightly while other methods’ performance improves to the level that surpasses SAM’s point-based performance. We also provide several illustrations for SAM’s performance on all tested datasets, iterative segmentation, and SAM’s behavior given prompt ambiguity. We conclude that SAM shows impressive zero-shot segmentation performance for certain medical imaging datasets, but moderate to poor performance for others. SAM has the potential to make a significant impact in automated medical image segmentation in medical imaging, but appropriate care needs to be applied when using it. Code for evaluation SAM is made publicly available at https://github.com/mazurowski-lab/segment-anything-medical-evaluation.

Keywords: Segmentation, Foundation Models, Deep Learning

Graphical Abstract

graphic file with name nihms-1927264-f0001.jpg

1. Introduction

Image segmentation is a central task in medical image analysis, ranging from the segmentation of organs Rister et al. (2019), abnormalities Anwar et al. (2018), bones Kayalibay et al. (2017), and others Pham et al. (2000), which has received significant advancements from deep learning Hesamian et al. (2019); Razzak et al. (2018). However, developing and training segmentation models for new medical imaging data and/or tasks is practically challenging, due to the expensive and time-consuming nature of collecting and curating medical images, primarily because trained radiologists must typically provide careful mask annotations for images.

These difficulties could be significantly mitigated with the advent of foundation models Zhou et al. (2023) and zero-shot learning Chen et al. (2021). Foundation models are neural networks trained on an extensive amount of data, using creative learning and prompting objectives that typically do not require traditional supervised training labels, both of which contribute towards the ability to perform zero-shot learning on completely new data in a variety of settings. Foundation models have shown paradigm-shifting abilities in the domain of natural language processing OpenAI (2023). The recently developed Segment Anything Model is a foundation model that has achieved promising zero-shot segmentation performance on a variety of natural image datasets Kirillov et al. (2023).

1.1. What is SAM?

Segment Anything Model (SAM) is designed to segment an object of interest in an image given certain prompts provided by a user. Prompts can take the form of a single point, a set of points (including an entire mask), a bounding box, or text. The model is asked to return a valid segmentation mask even in the presence of ambiguity in the prompt. The general idea behind this approach is that the model has learned the concept of an object and thus can segment any object that is pointed out. This results in a high potential for it to be able to segment objects of types that it has not seen without any additional training, i.e., high performance in the zero-shot learning regime. In addition to the prompt-based definition of the task, the SAM authors utilized a specific model architecture and a uniquely large dataset to achieve this goal, described as follows.

SAM was trained progressively alongside the development of the dataset of images with corresponding object masks (SA-1B). The dataset was developed in three stages. First, a set of images was annotated by human annotators by clicking on objects and manually refining masks generated by SAM, which at that point was trained on public datasets. Second, the annotators were asked to segment masks that were not confidently generated by SAM to increase the diversity of objects. The final set of masks was generated automatically by prompting the SAM model with a set of points distributed in a grid across the image and selecting confident and stable masks.

1.2. How to segment medical images with SAM?

SAM is designed to require a prompt or a set of prompts to produce a segmentation mask. Technically, the model can be run without a prompt to provide any visible object, but we do not expect this to be useful for medical images, where there are often many other objects in the image beside the one of interest. Given this prompt-based nature, in its basic form, SAM cannot be used the same way as most segmentation models in medical imaging where the input is simply an image and the output is a segmentation mask or multiple masks for the desired object or objects.

We propose that there are three main ways in which SAM can be used in the process of segmentation of medical images. The first two involve using the actual Segment Anything Model in the process of annotation, mask generation, or training of additional models. These approaches do not involve changes to the SAM. The third approach involves the process of training/fine-tuning a SAM-like model targeted for medical images. We detail each approach next. Note that we do not comment here on text-based prompting, as it is still in the proof-of-concept stage for SAM.

Semi-automated annotation (“human in the loop”).

The manual annotation of medical images is one of the main challenges of developing segmentation models in this field since it typically requires the valuable time of physicians. SAM could be used in this setting as a tool for faster annotation. This could be done in different ways. In the simplest case, a human user provides prompts for SAM, which generates a mask to be approved or modified by the user; this could be refined iteratively. Another option is where SAM is given prompts distributed in a grid across the image (the “segment everything” mode), and generates masks for multiple objects which are then named, selected, and/or modified by the user. This is only the start; many other possibilities could be imagined.

SAM assisting other segmentation models.

One version of this usage mode is where SAM works alongside another algorithm to automatically segment images (an “inference mode”). For example, SAM, based on point prompts distributed across the image, could generate multiple object masks which then could be classified as specific objects by a separate classification model. Similarly, an independent detection model, e.g., ViTDet Li et al. (2022), could generate object-bounding boxes of images to be used as prompts for SAM to generate precise segmentation masks.

Furthermore, SAM could be used in the training loop of some other semantic segmentation model. For example, the masks generated by a segmentation model on unlabeled images during training could be used as prompts to SAM to generate more precise masks for these images, which could be used as iteratively refined supervised training examples for the model being trained. One could conceptualize many other specific modes of including SAM in the process of training new segmentation models.

New medical image foundation segmentation models.

In this usage mode, the development process of a new segmentation foundation model for medical images could be guided by SAM’s own development process. The largest difficulty of this would be in the much lower availability of medical images and quality annotations, compared to natural images, but this is possible in principle. A more feasible option could be to fine-tune SAM on medical images and masks from a variety of medical imaging domains, rather than training from scratch, as this would likely require fewer images.

2. Methodology

In the previous section, we described various usage scenarios of SAM for medical image segmentation. These are conceptually promising but largely rely on the assumption that SAM can generate accurate segmentations of medical images. Here, we experimentally evaluate this claim and evaluate the performance of SAM within a variety of different realistic usage scenarios and datasets in medical imaging.

2.1. Datasets

We compiled and curated a set of 19 publicly available medical imaging datasets for image segmentation. While the phrase “medical imaging” is sometimes used to refer to all images pertaining to medicine, we focus on the common definition of radiological images. Our dataset includes planar X-rays, magnetic resonance images (MRIs), computed tomography (CT) images, ultrasound (US) images, and positron emission tomography (PET) images. The datasets are summarized in Tables 1 and 1. For datasets containing more than one type of object, we considered the segmentation of each object as a separate task.

2.2. Data Pre-processing

All images were normalized by dividing all pixel values by the maximum value in that image and multiplying all pixels by 255. There were some additional normalization steps applied to 3 of the 19 datasets by our group including volume-based linear transformation to the 0–255 range and removal of extreme values. For the MRI-Brain dataset, we used T1, T1Gd, and T2-FLAIR as three channels. For the PET-Whole-Body dataset, we presented the PET information in the red channel and the CT information in both the green and blue channels. For all the other datasets, we presented the gray-scale information in all three channels. We did not resize the images, but the SAM algorithm does automatic resizing before evaluation.

Algorithm 1.

Prompt Point Generation Scheme

Input: Image X∈RH×W, ground truth mask M∈{0,1}H×W, Segment Anything Model SAM, prompt count N, closest-zero-pixel distance function d = distance Transform() from OpenCV Bradski (2000).
1: Initialize first prompt point p1 as the point within the mask foreground farthest from the background:
2: 𝒫=argmax⁡(d[(i,j),(k,l)]) for all (i,j), (k,l) such that Mij=1,Mkl=0.
3: Choose randomly if multiple points satisfy this:
4: p1=random_choice(𝒫)
5: PredictmaskY1=SAM⁡X,p1
6: Get prediction error region E1=Y1∪M-Y1∩M
7: Subsequent prompt points are those farthest from the boundary of iteratively-updating error region En:
8: for n=2,…,N do
9: 𝒫n=argmax(i,j)(d[(i,j),(k,l)]) for all (i,j), (k,l) such that En-1ij=1,En-1kl=0.
10: pn=random_choice𝒫n
11: Yn=SAM⁡X,pn
12: En=Yn∪M-Yn∩M
13: end for
14: return Prompt points p1,…,pN∈N2
Open in a new tab

2.3. Experiments

We performed a thorough evaluation of SAM with both non-iterative prompts (generated prior to SAM being applied) and iterative prompts (generated after seeing the model’s predictions). We also explored the “segment everything” mode of SAM and analyzed the different outputs that SAM generates in response to ambiguity in the prompts.

2.3.1. Prompting Strategies

Non-iteratvie prompts.

In this primary mode of evaluation, prompts were simulated to reflect how a human user might generate them while looking at the objects. We focus on five modes of non-iterative prompting designed to capture the realistic usage cases of SAM for generating image masks, using either points or bounding boxes. An essential thing to consider is that a single “object” of interest / “ground truth” mask may consist of multiple disconnected parts, which is especially common in medical images. An example of this is a cross-sectional image of a liver (such as MRI or CT) where in one 3D slice, the liver is portrayed as two non-contiguous areas. Given this consideration, we introduce the following five prompting modes:

  • One prompt point is placed at the center of the largest contiguous region of the object of interest/ground truth mask.

  • A prompt point is placed at the center of each separate contiguous region of the object of interest (up to three points).

  • One box prompt is placed to tightly enclose the largest contiguous region of the object of interest.

  • A box prompt is placed to tightly enclose each separate contiguous region of the object of interest (up to three boxes)

  • A single box is placed to tightly enclose the entire object mask.

Juxtaposed examples of each prompting mode for the same image are shown in Figure 1. Modes 1 and 2 are equivalent if the object consists of only one contiguous region/part, and the same is true for modes 3, 4, and 5. The point prompts were generated as the point farthest from the boundary of the mask for the object or its part.

Fig. 1.

Fig. 1.

Open in a new tab

Examples of prompt(s) generated by the five modes respectively. Green contours show the ground-truth masks, and blue star(s) and box(es) indicate the prompts.

Iterative prompts.

We use a common, intuitive strategy for simulating realistic iterative point prompts, which reflects how those could be generated by a user in an interactive way Mahadevan et al. (2018). The details of the prompt generation are illustrated in Algorithm 1. Specifically, once the network makes a prediction, we compute an error map where both false positive and false negative predictions are marked as 1, i.e., the point furthest from 0. Then we can find the location of the next prompt as the central location of the largest component of the error mask. The label of the prompt is based on whether the new location is foreground or background.

Prompt ambiguity and oracle performance.

Prompts can be ambiguous in the sense that it may be unclear which object in the image the prompt is referring to. A typical scenario is when objects are nested within each other in the image. For example, when a user provides a point prompt within a necrotic component of a brain tumor, they could intend to segment that component, the entire tumor, one hemisphere of the brain, the entire brain, or the entire head. In response to this issue, SAM provides multiple outputs aiming at disambiguating the prompts. This is a very important and practical feature of SAM since in the interactive segmentation setting, multiple potential outputs could be presented to the user, from which they could select the one which is the closest to the object that they intended. In our experiments, we display some examples of the multiple outputs generated by SAM to illustrate how it deals with the ambiguity of the prompts.

Related to the ambiguity, in all experiments, we also present what the developers of SAM call “oracle performance”. This is the performance of the model when the prediction closest (in terms of IoU) to the true mask, i.e., oracle prediction, is always used, out of SAM’s three generated predictions. Note that this prediction may differ from SAM’s most confident prediction. While this assumes knowledge of the true mask and is a biased way to assess performance when there is no additional interaction with the user after providing the initial prompts, it is a practical reflection of performance in a setting where the user can select one of the masks generated by SAM. When prompts are generated iteratively, the oracle prediction is used to create the error map that guides the location of the next prompt.

Variability in prompt placement.

We evaluated the sensitivity of SAM’s performance to the location of the prompts within the object. To do so, we divided the object region into 4 subregions according to the distance of each pixel to the closest edge such that each sub-region had approximately the same area. For example, the most central area consisted of pixels in approximately the bottom 25% according to such distance. Then, for each evaluated dataset and task, we drew, with replacement, 10,000 images, and randomly placed a prompt in (1) the central location, (2) the most central sub-regions, (3) the union of the two most central sub-regions, (4) the union of the three most central sub-regions, (5) the entire object region. In rare circumstances where there was more than one location that qualified to be the central location in the object, we selected one randomly. We conducted this analysis for 3 datasets with 8 tasks.

2.3.2. Comparison with other methods

We compared SAM with three interactive segmentation methods, namely RITM Sofiiuk et al. (2021), SimpleClick Liu et al. (2022), and FocalClick Chen et al. (2022). RITM adopted HRNet-18 Wang et al. (2019) as the backbone segmentation model and an iterative mask correction approach based on the { previous mask, new click } set to achieve outstanding performance on multiple natural imaging datasets. Based on the framework proposed by RITM, SimpleClick replaced the HRNet-18 backbone in RITM with a plain-ViT Dosovitskiy et al. (2021) and included a “click embedding” layer symmetric to patch embedding. It allows for fewer clicks than RITM to achieve an above-threshold performance. FocalClick replaced the HRNet-18 backbone in RITM with a SegFormer Xie et al. (2021) and restricted the update responding to new clicks to occur only as local. The method proposed “progressive merge” that can exploit morphological information and prevent unintended changes far away from users’ clicks, resulting in faster inference time.

2.3.3. Performance evaluation metric

For each dataset, we evaluated the accuracy of the masks that SAM and aforementioned methods generate given prompts, with respect to the “ground truth” mask annotations for the given dataset and task. In the quantitative evaluation, we always use the mask with the highest confidence generated by SAM for a given prompt. We used IoU as the evaluation metric, similar to as in SAM’s original paper Kirillov et al. (2023). For datasets containing multiple types of objects, performance was reported independently for each type of object. Since SAM only supports 2D inputs, the IoUs were evaluated over each individual slice.

3. Results

3.1. Performance of SAM for different modes of use for 28 tasks

The performance of SAM for our five prompting modes, introduced in Section 2.3.1, of use is shown in Figure 2. We draw several conclusions. First, SAMs performance varies widely across different datasets. It ranges from an impressive IoU of 0.9118 to a very poor IoU of 0.1136.

Fig. 2.

Fig. 2.

Open in a new tab

Performance of SAM under 5 modes of use. Left: Performance of SAM across 28 segmentation tasks, with results ranked in descending order based on Mode 4. Oracle performance for each mode is indicated by the inverted triangle. Right: A summarized performance comparison of all five modes across all tasks, presented in a box and whisker plot format.

Comparing the performance for different prompting modes shows clear superiority of box prompts over point prompts. Moreover, as expected, prompts where each separate part of the object is indicated separately are generally superior to those where only one part is indicated, or all parts are outlined in one box. This was particularly pronounced for datasets where objects typically consist of more than one part. Following these two trends, Mode 4, where each part of an object is indicated by a separate box, showed the best performance with an average IoU of 0.6542.

Additionally, the oracle mode showed a moderate improvement over the default mode. The magnitude of this improvement was highly dependent on the dataset.

Figure 3 shows examples of segmentations generated in prompting Mode 2 (a point for each object part) and Mode 4 (a box around each object part) for 4 selected datasets. For each dataset, we provide examples of SAM’s segmentations in the 25th, 50th, and 75th percentile of IoU. Green contours represent the ground truth masks, red contours represent SAM’s prediction, and teal points or boxes represent the prompts given to SAM. This figure illustrates a high variability of SAM’s performance ranging from near-perfect for well-circumscribed objects with unambiguous prompts to very poor, particularly for objects with ambiguous prompts. A similar illustration for all datasets is provided in Appendix Section 1.

Fig. 3.

Fig. 3.

Open in a new tab

Visualization of SAM’s segmentation results in two different modes. Each dataset is shown in two sequential rows, with its name along the left side. For each dataset, it displays three examples from left to right, reflecting the 25th, 50th, and 75th percentiles of IoU across all images for that dataset. For each example, we visualize (top left) the raw image; (bottom left) the zoom-in image with the area of interest; (top right) the segmented results for mode 2: 1 point at each object region; (bottom right) the segmented results for mode 4: 1 box region at each object region. Additionally, the IoU is represented above each segmented result. Examples of all the datasets are shown in Appendix Figure 1–5.

3.2. Comparing SAM to other interactive segmentation methods

We compare SAM to other interactive methods in the non-iterative prompting setting in Figure 4. SAM performed better than all other methods on 24 out of 28 tasks, with a dramatic improvement in performance for some of them. When used in oracle mode, SAM was better than all other methods for 26 out of 28 tasks. The average performance across different datasets was 0.4595 IoU for SAM, 0.5137 IoU for SAM in oracle mode, 0.2240 IoU for FocalClick, 0.1910 IoU for SimpleClick, and 0.1322 IoU for RITM. Note that SAM exhibits notably better overall performance than all other methods even though it is not used in its best-performing mode of using boxes as prompts, as we used point prompts to have a fair comparison between different methods. If SAM used Mode 3 (single box), it outperformed all other methods in all but one task. The average performance for Mode 3 was 0.5891 IoU.

Fig. 4.

Fig. 4.

Open in a new tab

Comparison of SAM with three other competing methods, namely RITM, SimpleClick, and Focalclick, under the 1-point prompt setting. The results are presented in the form of the difference between SAM and other methods (Δ IoU), and ranked based on the descending order of the largest Δ IoU for each task.

3.3. Performance of SAM and other methods for iterative segmentation

We show the average performances of SAM, RITM, SimpleClick, and FocalClick in Figure 5. The detailed performance of the three competing methods over 28 tasks under the interactive prompt setting is shown in Appendix Section 2. As seen in the results above, for the scenario where prompts are provided prior to any interaction with the output of the models, SAM performs notably better than all other methods. However, when additional clicks are provided iteratively with the goal of refining the segmentations returned by the models, the superiority of SAM diminishes, and it is surpassed by two other methods (SimpleClick and RITM) for five or more points provided by the users. This is due to the fact that SAM does not appear to draw hardly any benefits from additional information provided through the interactive points after two or three points have been provided. Figure 6 illustrates this phenomenon.

Fig. 5.

Fig. 5.

Open in a new tab

Comparison of SAM and other methods under an interactive prompt setting. (Left) it presents the average performance of SAM and other methods across all tasks with respect to the number of prompt changes. (Right) it shows the detailed performance of SAM over each task.

Fig. 6.

Fig. 6.

Open in a new tab

Examples of SAM’s prediction under the interactive prompt setting. For each dataset, we display the results from 1-point prompts to 9-point prompts, respectively. The positive prompts are represented as green stars, and the negative prompts are represented as red stars.

We also see that SAM, when given a single prompt, has difficulty segmenting objects with multiple non-contiguous regions. It is more likely to segment one contiguous region instead of trying to find additional semantically similar regions in the entire image. Therefore, in the scenario where multiple regions of interest exist for an object, additional prompt points for SAM can be beneficial if they target additional regions, but beyond that, the benefit of further prompt points is negligible and in some cases such additional input is detrimental.

3.4. Performance of SAM in the presence of ambiguity of prompts

When applying SAM on medical images, we found a consistent tendency of SAM in interpreting the point prompts. As shown in Figure 7, the SAM’s highest-confidence map predictions tend to look similar to results generated by region-growing-based algorithms. In most cases, a connected region with similar intensity in the image bounded by regions with dissimilar intensities would be segmented. On the other hand, we found masks predictions generated with lower confidence scores may expand the highest-confidence predictions and tend to have more variety in intensity/texture.

Fig. 7.

Fig. 7.

Open in a new tab

Visualizations of examples with ambiguity based on SAM; the 1st, 2nd, and 3rd confident predictions are shown sequentially.

3.5. Performance of SAM given different locations of the prompt within the image

Figure 8 shows how the performance of SAM and its variability across different cases changes as the prompt is placed randomly within an increasingly large region around the center of the object. We observe that for some tasks there is only very small or no decrease in performance when the prompt is not placed in the center of the object while for other ones, there is generally a small decrease in median performance and sometimes an increase in variability of performance between cases when the prompts are placed randomly within a region with increasing size. The most notable decrease in performance due to variability in the prompt location was observed for the task where the original performance was the lowest.

Fig. 8.

Fig. 8.

Open in a new tab

Performance of SAM when prompts are placed randomly within certain regions.

3.6. Performance of SAM for objects of different sizes

In Figure 9(a) we show how object size relates to the performance of SAM. We describe object size as the ratio of the number of pixels in the object to the total in the image. We found object size to vary broadly across our different datasets, by up to over two orders of magnitude. Some datasets also showed high variability of object size within the dataset itself (e.g., kidneys in ultrasound images) while others showed relative consistency of object size (such as bones in hip X-ray images).

Fig. 9.

Fig. 9.

Open in a new tab

(Top) the relative size of the object in each dataset; (Bottom) the object size vs. detection performance for mode 2 and mode 4 separately; we also show a regression fitted curve each.

Figures 9(b) and 9(c) show the relationship between the average object size in a dataset and the performance of SAM. While the correlations are low, there is a trend towards a higher performance of SAM for larger objects. Note that the correlation analysis looks at average object sizes in the image and does not consider the variation of object sizes in the individual datasets.

3.7. Segment-everything mode for medical images

In Figure 10, we provide an example of using SAM’s “segment everything mode” on medical images. Segment-everything mode uses a dense evenly-spaced mesh of prompt points, designed to direct SAM to segment the image into many different regions. We find that this mode provides mixed usefulness and that the results are somewhat dependent on the number of prompts. While imperfect, this setup could potentially be useful in some applications, e.g., as a “starting point” for creating segmentation annotations for different objects in a new image. Selecting masks from the output of segment everything mode is effectively a special case of the prompting mode 2 in our experiments except with additional post-processing. We hence would not elaborate on this mode in our paper.

Fig. 10.

Fig. 10.

Open in a new tab

Examples of segment everything mode. For each example, we sampled a different number of grid points at each side as 25,26 and 27.

4. Conclusions and discussion

In this study, we evaluated the new Segment Anything Model for the segmentation of medical images. We reached the following conclusions:

  • SAM’s accuracy for zero-shot medical image segmentation is moderate on average and varies significantly across different datasets and different images within a dataset.

  • The model performs best with box prompts, particularly when one box is provided for each separate part of the object of interest.

  • SAM outperforms RITM, SimpleClick, and FocalClick in the vast majority of the evaluated settings where a single non-iterative prompt point is provided.

  • In the setting where multiple iteratively-refined point prompts are provided, SAM obtains very limited benefit from additional point prompts, except for objects with multiple parts. On the other hand, the other algorithms improve notably with additional point prompts, to the level of surpassing SAM’s performance. However, the point prompting modes are inferior to SAM’s box prompting modes.

  • We find a small but non-statistically significant correlation between the average object size in a dataset and SAM performance.

One of the contributions of our study is that we identified five different modes of use for interactive segmentation methods. This is of particular importance for models which have multiple components, a common feature in medical imaging. These modes also showed different performances in such scenarios. While these modes demonstrate the variety of uses, future work could focus on prompt engineering, both non-iterative and iterative, which could potentially reach even higher performance.

A notable limitation of SAM in the context of medical imaging is that it operates in 2D only, while many medical images are 3D. The algorithm could be extended to 3D in several ways. A simple way would be to first generate the mask in a given slice based on the provided prompt and then automatically generate prompts in neighboring slices based on the generated mask. Another one would be to have SAM generate masks in different orthogonal places based on the prompt provided in a given slice. This could be repeated in different slices, and the results could be fused Jun et al. (2021); Jang and Hwang (2022). The image encoder and mask decoder could also be extended to 3D. These and other possible extensions will certainly be associated with notable challenges and a significant amount of algorithmic development, but it might be a worthwhile endeavor to improve the speed at which medical images could be annotated.

The segment anything model, the associated preprint, and the code illustrate the strengths of open science and the activity of machine learning and machine learning in medical imaging communities. Since we made the first version of this manuscript available, approximately 2 weeks after the release of the SAM paper (or even some before our release), multiple preprints have appeared evaluating SAM in broadly understood medical imaging and radiological imaging Deng et al. (2023); Huang et al. (2023) Cheng et al. (2023). Some preprints already showed extensions of SAM to medical imaging Ma and Wang (2023); He et al. (2023) and one paper showed integration of SAM into 3D Slicer Liu et al. (2023). This demonstrates a high likelihood that SAM will become an important part of image segmentation in medical imaging.

Overall, SAM shows promise for use in medical images, as long as suitable prompting strategies are used for the dataset and task of choice. Future work will include the development of different ways to adapt it to construct medical imaging-specific models as well as extend to 3D segmentation.

Supplementary Material

1

Table 1.

All datasets evaluated in this paper. “num. masks” refers to the number of images with non-zero masks. For 3D modalities, 2D slices are used as inputs.

Abbreviated dataset name Full dataset name and citation Modality Num. classes Object(s) of interest Num. masks
MRI-Spine Spinal Cord Grey Matter Segmentation Challenge (Prados et al. (2017)) MRI 2 Gray matter, spinal cord 551
MRI-Heart Medical Segmentation Decathlon (Simpson et al. (2019)) MRI 1 Heart 1,301
MRI-Prostate Initiative for Collaborative Computer Vision Benchmarking (Lemaître et al. (2015)) MRI 1 Prostate 893
MRI-Brain The Multimodal Brain Tumor Image Segmentation Benchmark (BraTS) (Menze et al. (2014)) MRI 3 GD-enhancing tumor, Peritumoral edema, necrotic and nonenhancing tumor core 12,591
MRI-Breast Duke Breast Cancer MRI: Breast + FGT Segmentation (Saha et al. (2018); Hu et al. (2022)) MRI 2 Breast, fibroglandular tissue 503
Xray-Chest Montgomery County and Shenzhen Chest X-ray Datasets (Jaeger et al. (2014)) X-ray 1 Chest 704
Xray-Hip X-ray Images of the Hip Joints (Gut (2021)) X-ray 2 Ilium, femur 140
US-Breast Dataset of Breast Ultrasound Images (Al-Dhabyani et al. (2020)) Ultrasound 1 Breast 647
US-Kidney CT2US for Kidney Segmentation (Song et al. (2022)) Ultrasound 1 Kidney 4,586
US-Muscle Transverse Musculoskeletal Ultrasound Image Segmentations (Marzola et al. (2021)) Ultrasound 1 Muscle 4,044
US-Nerve Ultrasound Nerve Segmentation Identify (Anna et al. (2016)) Ultrasound 1 Nerve 2,323
US-Ovarian-Tumor Multi-Modality Ovarian Tumor Ultrasound (MMOTU) (Zhao et al. (2022)) Ultrasound 1 Ovarian tumor 1,469
CT-Colon Medical Segmentation Decathlon (Simpson et al. (2019)) CT 1 Colon cancer primaries 1,285
CT-HepaticVessel Medical Segmentation Decathlon (Simpson et al. (2019)) CT 1 Vessels, tumors 13,046
CT-Pancreas Medical Segmentation Decathlon (Simpson et al. (2019)) CT 1 parenchyma and mass 8,792
CT-Spleen Medical Segmentation Decathlon (Simpson et al. (2019)) CT 1 spleen 1,051
CT-Liver The Liver Tumor Segmentation Benchmark (LiTS) (Bilic et al. (2023)) CT 1 Liver 5,501
CT-Organ CT Volumes with Multiple Organ Segmentations (CT-ORG) (Rister et al. (2019)) CT 5 Liver, bladder, lungs, kidney, bone 4,776
PET-Whole-Body A FDG-PET/CT dataset with annotated tumor lesions (Gatidis et al. (2022)) PET/CT 1 Lesion 1,015
Open in a new tab

Highlights.

  • Segment Anything Model (SAM) is a new algorithm for interactive image segmentation

  • Performance of SAM varies widely on the 19 evaluated medical imaging datasets

  • SAM performs best when box annotations are provided for each component of object

  • SAM outperforms other methods in interactive but non-iterative modes

  • SAM is likely to be a valuable tool in medical image segmentation if used correctly

Acknowledgments

Research reported in this publication was supported by the National Institute Of Biomedical Imaging And Bioengineering of the National Institutes of Health under Award Number R01EB031575 and by the National Heart Lung and Blood Institute of the National Institutes of Health under Award Number R44HL152825. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Footnotes

Declaration of interests

The authors declare that they have no known competng fnancial interests or personal relatonships that could have appeared to influence the work reported in this paper.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain

References

  1. Al-Dhabyani W, Gomaa M, Khaled H, Fahmy A, 2020. Dataset of breast ultrasound images. Data in brief 28, 104863. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Anna M, Hasnin, kaggle446, shirzad, Will C., yffud, 2016. Ultrasound nerve segmentation. URL: https://kaggle.com/competitions/ultrasound-nerve-segmentation. [Google Scholar]
  3. Anwar SM, Majid M, Qayyum A, Awais M, Alnowami M, Khan MK, 2018. Medical image analysis using convolutional neural networks: a review. Journal of medical systems 42, 1–13. [DOI] [PubMed] [Google Scholar]
  4. Bilic P, Christ P, Li HB, Vorontsov E, Ben-Cohen A, Kaissis G, Szeskin A, Jacobs C, Mamani GEH, Chartrand G, et al. , 2023. The liver tumor segmentation benchmark (lits). Medical Image Analysis 84, 102680. Bradski G., 2000. The OpenCV Library. Dr. Dobb’s Journal of Software Tools [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Chen J, Geng Y, Chen Z, Horrocks I, Pan JZ, Chen H, 2021. Knowledge-aware zero-shot learning: Survey and perspective, in: International Joint Conference on Artificial Intelligence. [Google Scholar]
  6. Chen X, Zhao Z, Zhang Y, Duan M, Qi D, Zhao H, 2022. Focalclick: Towards practical interactive image segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1300–1309. [Google Scholar]
  7. Cheng D, Qin Z, Jiang Z, Zhang S, Lao Q, Li K, 2023. Sam on medical images: A comprehensive study on three prompt modes. ArXiv abs/2305.00035. [Google Scholar]
  8. Deng R, Cui C, Liu Q, Yao T, Remedios LW, Bao S, Landman BA, Wheless L, Coburn LA, Wilson KT, Wang Y, Zhao S, Fogo AB, Yang H, Tang Y, Huo Y, 2023. Segment anything model (sam) for digital pathology: Assess zero-shot segmentation on whole slide imaging. ArXiv abs/2304.04155. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Uszkoreit J, Houlsby N, 2021. An image is worth 16×16 words: Transformers for image recognition at scale. ICLR. [Google Scholar]
  10. Gatidis S, Hepp T, Früh M, La Fougère C, Nikolaou K, Pfannenberg C, Schölkopf B, Küstner T, Cyran C, Rubin D, 2022. A whole-body fdg-pet/ct dataset with manually annotated tumor lesions. Scientific Data 9, 601. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Gut D, 2021. X-ray images of the hip joints 1. URL: https://data.mendeley.com/datasets/zm6bxzhmfz, doi: 10.17632/zm6bxzhmfz. 1. publisher: Mendeley Data. [DOI] [Google Scholar]
  12. He S, Bao R, Li J, Grant PE, Ou Y, 2023. Accuracy of segmentanything model (sam) in medical image segmentation tasks. arXiv preprint arXiv:2304.09324. [Google Scholar]
  13. Hesamian MH, Jia W, He X, Kennedy P, 2019. Deep learning techniques for medical image segmentation: achievements and challenges. Journal of digital imaging 32, 582–596. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Hu S, Park C, Lew CO, Grimm LJ, Baker JA, Taylor-Cho MW, Mazurowski MA, 2022. Fully automated deep learning method for fibroglandular tissue segmentation in breast mri. [Google Scholar]
  15. Huang Y, Yang X, Liu L, Zhou H, Chang A, Zhou X, Chen R, Yu J, Chen J, Chen C, et al. , 2023. Segment anything model for medical images? arXiv preprint arXiv:2304.14660. [DOI] [PubMed] [Google Scholar]
  16. Jaeger S, Candemir S, Antani S, Wáng YXJ, Lu PX, Thoma G, 2014. Two public chest x-ray datasets for computer-aided screening of pulmonary diseases. Quantitative imaging in medicine and surgery 4, 475. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Jang J, Hwang D, 2022. M3t: three-dimensional medical image classifier using multi-plane and multi-slice transformer, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20718–20729. [Google Scholar]
  18. Jun E, Jeong S, Heo DW, Suk HI, 2021. Medical transformer: Universal brain encoder for 3d mri analysis. arXiv preprint arXiv:2104.13633. [DOI] [PubMed] [Google Scholar]
  19. Kayalibay B, Jensen G, van der Smagt P, 2017. Cnn-based segmentation of medical imaging data. arXiv preprint arXiv:1701.03056. [Google Scholar]
  20. Kirillov A, Mintun E, Ravi N, Mao H, Rolland C, Gustafson L, Xiao T, Whitehead S, Berg AC, Lo WY, et al. , 2023. Segment anything. arXiv preprint arXiv:2304.02643. [Google Scholar]
  21. Lemaître G, Martí R, Freixenet J, Vilanova JC, Walker PM, Meriaudeau F, 2015. Computer-aided detection and diagnosis for prostate cancer based on mono and multi-parametric mri: a review. Computers in biology and medicine 60, 8–31. [DOI] [PubMed] [Google Scholar]
  22. Li Y, Mao H, Girshick RB, He K, 2022. Exploring plain vision transformer backbones for object detection. ArXiv abs/2203.16527. [Google Scholar]
  23. Liu Q, Xu Z, Bertasius G, Niethammer M, 2022. Simpleclick: Interactive image segmentation with simple vision transformers. ArXiv abs/2210.11006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Liu Y, Zhang J, She Z, Kheradmand A, Armand M, 2023. Samm (segment any medical model): A 3d slicer integration to sam. ArXiv abs/2304.05622. [Google Scholar]
  25. Ma J, Wang B, 2023. Segment anything in medical images. arXiv preprint arXiv:2304.12306. [Google Scholar]
  26. Mahadevan S, Voigtlaender P, Leibe B, 2018. Iteratively trained interactive segmentation, in: British Machine Vision Conference. [Google Scholar]
  27. Marzola F, van Alfen N, Doorduin J, Meiburger KM, 2021. Deep learning segmentation of transverse musculoskeletal ultrasound images for neuromuscular disease assessment. Computers in Biology and Medicine 135, 104623. [DOI] [PubMed] [Google Scholar]
  28. Menze BH, Jakab A, Bauer S, Kalpathy-Cramer J, Farahani K, Kirby J, Burren Y, Porz N, Slotboom J, Wiest R, et al. , 2014. The multimodal brain tumor image segmentation benchmark (brats). IEEE transactions on medical imaging 34, 1993–2024. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. OpenAI, 2023. Gpt-4 technical report. arXiv:2303.08774. [Google Scholar]
  30. Pham DL, Xu C, Prince JL, 2000. Current methods in medical image segmentation. Annual review of biomedical engineering 2, 315–337. [DOI] [PubMed] [Google Scholar]
  31. Prados F, Ashburner J, Blaiotta C, Brosch T, Carballido-Gamio J, Cardoso MJ, Conrad BN, Datta E, Dávid G, De Leener B, et al. , 2017. Spinal cord grey matter segmentation challenge. Neuroimage 152, 312–329. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Razzak MI, Naz S, Zaib A, 2018. Deep learning for medical image processing: Overview, challenges and the future. Classification in BioApps: Automation of Decision Making, 323–350. [Google Scholar]
  33. Rister B, Shivakumar K, Nobashi T, Rubin DL, 2019. CT-ORG: A Dataset of CT Volumes With Multiple Organ Segmentations. URL: https://wiki.cancerimagingarchive.net/x/OgWkAw, doi: 10.7937/TCIA. 2019.TT7F4V7O. version Number: 1 Type: dataset. [DOI] [Google Scholar]
  34. Saha A, Harowicz MR, Grimm LJ, Kim CE, Ghate SV, Walsh R, Mazurowski MA, 2018. A machine learning approach to radiogenomics of breast cancer: a study of 922 subjects and 529 dce-mri features. British journal of cancer 119, 508–516. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Simpson AL, Antonelli M, Bakas S, Bilello M, Farahani K, Van Ginneken B, Kopp-Schneider A, Landman BA, Litjens G, Menze B, et al. , 2019. A large annotated medical image dataset for the development and evaluation of segmentation algorithms. arXiv preprint arXiv:1902.09063
  36. Sofiiuk K, Petrov IA, Konushin A, 2021. Reviving iterative training with mask guidance for interactive segmentation. 2022 IEEE International Conference on Image Processing (ICIP), 3141–3145. [Google Scholar]
  37. Song Y, Zheng J, Lei L, Ni Z, Zhao B, Hu Y, 2022. Ct2us: Cross-modal transfer learning for kidney segmentation in ultrasound images with synthesized data. Ultrasonics 122, 106706. [DOI] [PubMed] [Google Scholar]
  38. Wang J, Sun K, Cheng T, Jiang B, Deng C, Zhao Y, Liu D, Mu Y, Tan M, Wang X, Liu W, Xiao B, 2019. Deep high-resolution representation learning for visual recognition. TPAMI. [DOI] [PubMed] [Google Scholar]
  39. Xie E, Wang W, Yu Z, Anandkumar A, Alvarez JM, Luo P, 2021. Segformer: Simple and efficient design for semantic segmentation with transformers, in: Neural Information Processing Systems (NeurIPS). [Google Scholar]
  40. Zhao Q, Lyu S, Bai W, Cai L, Liu B, Wu M, Sang X, Yang M, Chen L, 2022. A multi-modality ovarian tumor ultrasound image dataset for unsupervised cross-domain semantic segmentation. arXiv preprint arXiv:2207.06799 [Google Scholar]
  41. Zhou C, Li Q, Li C, Yu J, Liu Y, Wang G, Zhang K, Ji C, Yan Q, He L, et al. , 2023. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. arXiv preprint arXiv:2302.09419 [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

1
NIHMS1927264-supplement-1.pdf (17.9MB, pdf)

RESOURCES