Abstract
Segmentation in medical images plays a crucial first step for the quantitative analysis of organs of interest and disease conditions that may affect them. In the 2000s, we released an open-source medical image processing and analysis software system called CAVASS (Computer-Assisted Visualization and Analysis Software System), whose lineage goes back to 1980 as the first software for medical image processing. CAVASS supports multiple types of medical images and numerous usage scenarios/use cases. To allow ready access to advanced deep learning models within the easy-to-use graphical user interface (GUI) of CAVASS to greatly facilitate ground truth segmentation generation and production mode image segmentation, we created an enhanced version of CAVASS, called CAVASS-DL, by integrating the Segment Anything Model (SAM) within our software, which can extract and fuse the features from images and interactive prompts to enhance the segmentation performance. Firstly, we modified the GUI to enable intuitive region-of-interest (ROI) prompting via bounding boxes. Secondly, we implemented and trained the SAM network using PyTorch platform. Lastly, we bundled the SAM network as a standalone executable interface for CAVASS. The illustrative and quantitative results on 20 dynamic MRI scans of neck and thorax demonstrate that our approach achieves high accuracy and efficiency for semi-automatic anatomy object segmentation via CAVASS-DL in a convenient manner.
Keywords: Medical image processing software, Segment Anything Model, Anatomy segmentation, CAVASS-DL
1. INTRODUCTION
CAVASS (Computer-Assisted Visualization and Analysis Software System) [1] is a longstanding open-source platform for medical image processing, whose lineage goes back to the earliest software packages developed in this field called DISPLAY and DISPLAY82, image visualization and processing systems for 3D computed tomography (CT) images developed by Medical Image Processing Group (MIPG) at the State University of New York, Buffalo, and then University of Pennsylvania, Philadelphia, in 1980 and 1982, respectively [2, 3]. DISPLAY82 was distributed with source code to 100s of sites before the term “open source” came into common usage. An extensively enhanced Unix version, termed 3DVIEWNIX [4] was released in 1993, with a GUI developed from scratch based on X-windows and a standardized image and object representation protocol which was a 3D (actually nD) generalization [5] of the ACR-NEMA standards existing at that time prior to DICOM. This system was installed at hundreds of sites worldwide with source codes distributed. It integrates visualization, processing, virtual surgery on objects, and quantitative analysis functionalities while supporting standard formats like DICOM and MATLAB across multiple imaging modalities including CT, magnetic resonance imaging (MRI), and positron emission tomography (PET). The software’s core strengths lie in its comprehensive processing pipeline, enabling end-to-end workflows from data import to 3D construction for applications such as surgical planning and lesion quantification. Its open-source architecture and plug-in extensibility further facilitate customized adaptation in research and clinical environments. CAVASS is the latest system in its lineage generalizing 3DVIEWNIX from Unix/Linux platforms to also include Windows and Mac platforms with a user interface developed based on wxWidgets. The current CAVASS platform inherits all of the rich functionalities of 3DVIEWNIX including various image processing operations via multitudes of algorithms like interpolation, filtering, segmentation, registration, 2D/3D/4D visualization, object manipulation for surgery planning, various image analysis operations for object quantification, etc.
Deep learning-based models have been widely used and have provided great potential for object segmentation. Segment Anything Model (SAM) is a foundational semi-automatic segmentation model that is widely applied in medical image processing [6, 7], which enables zero-shot generalization across diverse objects using interactive prompts (e.g., points, boxes, and masks). By combining user guidance with automated segmentation, SAM achieves high-precision boundary delineation in complex scenarios while minimizing manual effort. This synergistic approach balances automation with expert control, making it ideal for clinical applications requiring both speed and accuracy [8]. The current CAVASS platform has not integrated any deep learning (DL) models yet. CAVASS allows efficient interactive operations, such as refining any available segmentation masks input from different tools to obtain ground truth masks.
This study proposes augmenting CAVASS with the DL segmentation model of SAM [9] for efficient 2D/3D segmentation. Instead of resetting prompts in SAM, the enhanced version of CAVASS, called CAVASS-DL, allows for direct modification on the segmentation results. The integration aims to modernize its segmentation workflow while preserving its established strengths in multimodal data integration, analysis, and visualization. The features of the enhanced CAVASS-DL include: (1) all CAVASS operations inherited, (2) efficient implementation and interface of SAM using Python and C/C++ languages, (3) an easy-to-use GUI for interactive image segmentation, and (4) a flexible framework that allows users to integrate any other DL-based segmentation models.
2. MATERIALS & METHODS
As shown in Figure 1, the enhanced CAVASS inherits the architecture of the previous version, which employs a modular, pipeline-driven architecture designed for extensible medical image processing tool kit. Its core framework integrates four layered components: a data interface inherited from 3DVIEWNIX [4] for handling medical images input/output with metadata preservation, a processing interface for image processing operations, a graphical interface built on wxWidgets for image visualization and user interaction, and a group of computer-assisted visualization and analysis (CAVA) functions for computational processing. The processing interface utilizes a directed acyclic graph (DAG) structure, where nodes represent algorithms and edges define data flow between modules. This pipeline model allows users to reconfigure workflows dynamically. The CAVA functions can be divided into four groups: (1) image processing for enhancing and extracting object information (e.g., interpolation, filtering, segmentation, registration, intensity standardization), (2) various forms of 2D (slice) and 3D (surface and volume) visualization; (3) manipulation for altering 3D objects for surgery planning and other applications, and (4) analysis for quantifying object information. To integrate SAM into the CAVA functions, we utilized Python libraries (e.g., PyTorch [10], NumPy). Then, we bundled the model and all its dependencies into a single executable file (.exe) on the Windows operating system using the PyInstaller package. In addition, we altered the GUI for setting the region-of-interest (ROI) for the object of interest required by SAM as prior knowledge or prompt.
Figure 1.
Depiction of the enhanced CAVASS (CAVASS-DL) software.
2.1. Medical Image Datasets
To evaluate the difference in anatomy segmentation performance between manual ground truth (GT) segmentation and our semi-automatic method, we collected thoracic dynamic magnetic resonance imaging (dMRI) scans from 10 subjects, and upper airway neck dMRI scans from another 10 subjects, as summarized in Table 1. The thoracic dMRI dataset was acquired on a 3.0 T MRI scanner (Siemens Healthcare, Erlangen, Germany), under an IRB-approved research protocol at the Children’s Hospital of Philadelphia, including 46 3D frames with 1450 2D slices, while the neck dMRI dataset was acquired on a 3.0T MRI scanner (Philips Healthcare, Best, Netherlands), under an IRB-approved research study protocol at the Albert Einstein College of Medicine, New York, containing 50 3D frames with 2300 2D slices. All dMRI scans were annotated with the anatomy object of interest using the manual segmentation tool of CAVASS as the ground truth (GT). Each dataset was divided into training and testing sets with ratio of 4:1 for training and testing the DL models. We note that these two segmentation tasks are some of the most challenging segmentation problems unlike many other tasks particularly involving CT and static imagery.
Table 1.
Summary of the dMRI data sets.
| Dataset name | Thoracic dMRI dataset | Neck dMRI dataset |
|---|---|---|
| Anatomy object | Lung | Upper airway |
| Number of subjects | 23 | 17 |
| Number of 3D frames (2D slices) in training set | 36 (1092 slices) | 40 (1840 slices) |
| Number of 3D frames (2D slices) in testing set | 10 (358 slices) | 10 (460 slices) |
| Voxel size (mm3) | 1.46×1.46×6.00 | 1.50×1.50×1.50 |
| Image size | 224×256×33~224×256×38 | 112×264×46 |
2.2. Implementation details
As shown in Figure 2, SAM integrates four core components into a transformer-based architecture for prompt-driven segmentation: (1) an image encoder, built upon a Vision Transformer model pre-trained with masked autoencoding for extracting multi-scale contextual features and outputting a spatially structured 64×64 feature map with 256-channel embeddings that preserve critical anatomical relationships, (2) a prompt encoder for processing diverse sparse geometric prompts (points, bounding boxes) via positional encodings with foreground/background differentiation and projecting all prompt features to 256-dimensional vectors aligned with the image feature space, (3) a feature fusion module for combining the image and prompt features together through transformer decoder layers, yielding a 64×64×256 feature volume encoding task-specific semantics essential for image segmentation, and (4) a mask decoder that receives the fused feature maps and generates high-quality segmentation result through convolutional and transposed convolutional layers.
Figure 2.
The architecture of SAM.
To train the SAM network, we integrated the Focal loss [11] and Dice loss functions to define the loss function of SAM as below:
| (1) |
where S(X) and G(X) denote the predicted segmentation and GT, respectively, W and Ω represent the parameters of the network and image spatial domain, respectively, P(l = G(X)|X) represents the probability value of pixel X classified as ground truth G(X), E is a bias term (0.000001) that prevents the denominator from going to 0, γ is the exponent of the dynamic weight value, and λ1 is a trade-off parameter. In our experiments, the Adam optimizer was utilized to optimize the parameters of SAM network with the following parameters: number of iterations = 300, batch size = 10, and learning rate = 0.00001, and we utilized the GT mask to generate the bounding boxes for anatomic objects. In addition, we designed a script to automatically generate the bounding boxes of the objects for training data, speeding up the training efficiency. In this study, we utilized the PyTorch platform to implement the network architecture of SAM and trained the network using the annotated training data and Adam optimization algorithm. However, embedding Python within a C/C++ program introduces several challenges, primarily revolving around interpreter lifecycle management, data conversion, error handling, and thread synchronization. One major difficulty is initializing the Python interpreter correctly. Improper initialization can cause crashes, while neglecting to release resources during shutdown may lead to memory leaks. To solve this problem, we used the PyInstaller software to bundle the SAM code into standalone executables by analyzing the script’s dependencies and packaging everything needed for execution into a single distributable file. It starts by tracing all imported modules, libraries, and data files referenced in the main Python script, including the Python interpreter itself. Using this dependency map, PyInstaller copies these components into a temporary folder structure, along with a bootloader executable that initializes the environment. For single-file mode, it compresses all collected files into an archive appended to the bootloader. When the CAVASS program launches the generated .exe file, the bootloader extracts these archived resources into a secure temporary directory, sets up the Python runtime environment, including module paths, and executes the script.
2.3. Interactive segmentation
To ensure that SAM can correctly segment the anatomy object of interest, the SAM algorithm requires users to input prompts (such as bounding boxes) into the model. To address this, we re-designed the layout of the interactive 2D image segmentation GUI in CAVASS and placed the widgets of the SAM algorithm at the dialog window of the GUI, as shown in Figure 3. Our semi-automatic anatomy segmentation method includes the following four steps:
Figure 3.
Example of semi-automatic anatomy segmentation by CAVASS-DL.
Step 1 – Choose the path of model parameters. Click the “Choose” button with the left mouse button and select the stored file containing the SAM network parameters. This operation provides users with the option of including other DL models in the future. The users can re-train the SAM network using more datasets for improving segmentation accuracy or include other datasets for segmenting other objects.
Step 2 – Set the bounding box of the object. Utilize the left mouse button to draw the ROI (Figure 3). This operation has good flexibility due to the tolerance of the SAM algorithm to variable ROI size and position.
Step 3 – Click the “Run” button with the left mouse button to launch SAM. This operation would construct the SAM network on the CPU, load the selected network parameters, and input the medical image with the coordinates of the bounding box into the interface for outputting a segmentation of the object (blue mask in Figure 3).
Step 4 – Confirm the segmentation result. Click the “Add” button to save the segmentation result (green mask in Figure 3) or the “Clear” button to remove the segmentation result. The user can utilize the manual tools of the CAVASS Interactive-2D command to manually refine the segmentation (as shown in the right-bottom corner of the dialog window section in Figure 3).
3. RESULTS AND DISCUSSION
In this study, we utilized the enhanced CAVASS-DL platform to segment the upper airway structure and lungs in dMRI scans on the testing set and computed the 3D Dice index (DC) and average Hausdorff distance (HD) for evaluating the segmentation performance, as shown in Table 2. We observe that the enhanced CAVASS achieves high mean values of 0.93 and 0.95 and low standard deviations of 0.05 and 0.06 in terms of Dice index for lung and upper airway segmentation in the testing dataset, respectively, demonstrating excellent segmentation accuracy and stability. Similarly, the low HD values indicate that the proposed method obtains good shape agreement of the anatomy objects of interest with manual GT segmentations. For visual comparison, we show two examples in Figure 4.
Table 2.
DC and HD expressed as mean ± standard deviation.
| 3D DC | 2D HD (mm) | |
|---|---|---|
| Lung | 0.93 ± 0.05 | 1.48 ± 0.80 |
| Upper Airway | 0.95 ± 0.06 | 1.20 ± 0.37 |
Figure 4.
Examples of lung and upper airway segmentation in dMRI. The yellow rectangles indicate the bounding box ROIs of the anatomy objects of interest. The green and orange color overlays represent semi-automatic (SA) and ground truth (GT) segmentations, respectively.
Experiments were conducted on a PC with an Intel i7-6700 CPU without GPUs. To assess the efficiency of our system, we recruited two trainees to test the enhanced CAVASS-DL software, and computed the average human time required per dMRI scan via the manual GT method and our method, see Table 3. We find that the average processing time per subject of our method was 383.6 and 479.7 secs for segmenting 3D lung and 3D upper airway, respectively, which are 90.2 and 141.8 secs (i.e., 21% and 24%) less than the times required via the manual segmentation method, indicating good potential of our method for clinical and research application. Please note that these are extremely challenging segmentation tasks.
Table 3.
Average processing time in seconds (per dMRI scan) for manual segmentation and our segmentation approach.
| Manual (CAVASS) | Enhanced CAVASS-DL | |
|---|---|---|
| Lung | 413.8 ± 37.6 | 323.6 ± 35.8 |
| Upper Airway | 581.5 ± 92.3 | 439.7 ± 61.2 |
4. CONCLUSIONS
In this paper, we combined the SAM network with CAVASS software to construct an enhanced semi-automatic segmentation system called CAVASS-DL for medical image processing and analysis. The interactive segmentation tool includes four main operations: (1) choose the path of network parameters for loading the pre-trained DL model, (2) set the ROIs of the anatomy object of interest in 2D medical images using the GUI, (3) launch the inference to segment the specified anatomy object, and (4) confirm the segmentation result or edit as needed. The quantitative and qualitative experimental results on 20 test dMRI datasets of the neck and thorax demonstrate that our approach achieves good agreement with manual segmentation results for lung and upper airway anatomy objects, while significantly improving segmentation efficiency.
Acknowledgement
This work is supported by an NIH grant R01 CA255748.
REFERENCES
- [1].Grevera G, Udupa JK, Odhner D, Zhuge Y, Souza A, Iwanaga T, Mishra S: CAVASS: a computer-assisted visualization and analysis software system. Journal of Digital Imaging 2007, 20:101–18. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [2].Udupa JK: DISPLAY-A system of programs for two-and three-dimensional display of medical objects from CT data: Technical Report MIPG41, Medical Image Processing Group, Department of Computer Science, SUNY/Buffalo, Buffalo, New York, 1980. (112 pages). [Google Scholar]
- [3].Udupa JK: DISPLAY82 - A system of programs for the display of 3D information in CT data: Technical Report MIPG67, Medical Image Processing Group, Department of Radiology, University of Pennsylvania, Philadelphia, April, 1983. (205 pages). [Google Scholar]
- [4].Udupa JK, Odhner D, Samarasekera S, Goncalves RJ, Iyer K, Venugopal KP, Furuie SS: 3DVIEWNIX: An open, transportable, multidimensional, multimodality, multiparametric imaging software system. Medical Imaging 1994: Image Capture, Formatting, and Display 1994, 2164:58–73. [Google Scholar]
- [5].Udupa JK, Hung H-M, Odhner D, Goncalves R: Multidimensional data format specification: a generalization of the American College of Radiology-National Electric Manufacturers Association standards. Journal of Digital Imaging 1992, 5:26–45. [DOI] [PubMed] [Google Scholar]
- [6].Mazurowski MA, Dong H, Gu H, Yang J, Konz N, Zhang Y: Segment anything model for medical image analysis: an experimental study. Medical Image Analysis 2023, 89:102918. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [7].Zhang L, Deng X, Lu Y: Segment anything model (sam) for medical image segmentation: A preliminary review. 2023 IEEE International Conference On Bioinformatics and Biomedicine (BIBM): IEEE, 2023. pp. 4187–94. [Google Scholar]
- [8].Ali M, Wu T, Hu H, Luo Q, Xu D, Zheng W, Jin N, Yang C, Yao J: A review of the segment anything model (sam) for medical image analysis: Accomplishments and perspectives. Computerized Medical Imaging and Graphics 2025, 119:102473. [DOI] [PubMed] [Google Scholar]
- [9].Kirillov A, Mintun E, Ravi N, Mao H, Rolland C, Gustafson L, Xiao T, Whitehead S, Berg AC, Lo W-Y: Segment anything. Proceedings of the IEEE/CVF International Conference On Computer Vision, 2023. pp. 4015–26. [Google Scholar]
- [10].Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L: Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems 2019, 32. [Google Scholar]
- [11].Lin T-Y, Goyal P, Girshick R, He K, Dollár P: Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision, 2017. pp. 2980–8. [Google Scholar]