Toward a standard for the evaluation of PET‐Auto‐Segmentation methods following the recommendations of AAPM task group No. 211: Requirements and implementation

2.A. Usability and accessibility

In order for a standard to be usable, it is essential that the software is easy to use and quick to learn, although it is safe to assume some level of prior knowledge in the field (e.g., PET image analysis and segmentation) from the users. In particular, the user interface is required to be intuitive and accompanied by comprehensive documentation to guide the user through common useful cases or specific tasks. In addition, the software is required to be accessible to the public and understandable by the targeted user‐base. It may be desirable also that the design allows the software to be further extended and used for other applications in the future.

2.B. Application areas

The areas of application of the benchmark relate to the field of oncology. Image types are expected to reflect the state‐of‐the‐art in diagnostic imaging and treatment management and to adopt the most recent digitized histopathology methods and bespoke phantoms. The benchmark should be easily extendable to satisfy the needs of more application areas according to the availability of new data and new technology. The types of Volumes of Interest (VOI) included in a standardized evaluation protocol should at a minimum include disease sites established for using PET in radiotherapy treatment planning.

Best estimates of reference contours (RC) may originate from various sources depending on the image type included in the dataset. We distinguish between the following types of RC:

1. Absolute truth: only available for simulated images.

2. Single ‘best’ estimates: surrogate of truth provided for physical phantom images and in the special case of patient images for which histopathology data are available. In physical phantom images, the optimal threshold in simultaneous CT images provides a uniquely best estimate but cannot be considered the absolute truth because of threshold uncertainty arising from partial volume effects and potential misalignments between PET and CT datasets. The accuracy of RC data for patient scans provided by pathology examination of excised lesions is limited due to specimen deformation during processing and possible metabolic changes between the time of scan and the time of excision.7

3. Multiple equally ‘best’ estimates: they can be provided in the form of consensus manual expert delineations when no single delineation can be considered to be the best.

2.C. Performance criteria

This section describes the outputs expected from a benchmark in terms of both segmentation results and subsequent analysis using quantitative metrics extracted from the images. The benchmark is required to evaluate the agreement of PET‐AS results with the best available ‘truth’ estimate, as well as their robustness and the clinical implications of segmentation inaccuracies. The term ‘agreement’ relates to both volumetric and geometric properties. This is in line with the end‐points defined in the TG211 report, which includes “the spatial distribution of the tracer obtained from the PET image after correcting for physical artifacts” (cf. Ref. 6, section 4.A). Performance criteria for segmentation methods can include:6, 8

1. Accuracy: ability to recover the true tumor contour

2. Reproducibility: ability to provide the same result when used multiple times on the same image

3. Efficiency: ability to minimize computational complexity and workflow

4. Robustness: ability to provide similar results under varying acquisition and image reconstruction conditions

In the case of PET‐AS methods that rely on a pure automatic approach without user intervention, the reproducibility is expected to be 100%, and the efficiency including human and computational resources required for the segmentation is expected to be high due to the automatic process. Hence, it is suggested that the benchmark evaluation tools should focus on accuracy and robustness of the PET‐AS methods. Following these requirements, the accuracy metrics to be included in the benchmark are grouped into three categories, corresponding to increasing degrees of complexity:

• Level I: metrics that assess the agreement in terms of volumetric properties such as the number of voxels in the VOI and the statistics of PET signal integrated over that volume

• Level II: metrics that quantify the geometric agreement including spatial matching between a particular PET‐AS contour and the RC

• Level III: metrics that evaluate the clinical relevance of the disagreement between PET‐AS contours and RCs. These metrics describe the “knock‐on” effect that segmentation inaccuracy has on parameters used in treatment selection and planning and, in the case of external‐beam radiotherapy, dose delivery. This functionality is expected to ultimately be related to treatment outcome and is not implemented in the first release of this benchmark.

The robustness metrics should include, as suggested by Hatt et al., 9 the analysis of the sensitivity to the following variations:

1. across datasets, governed by differences in anatomy and physiology as reflected by the image characteristics,

2. within a dataset, resulting from natural differences in gross tumor volume shape/size between different patients, and

3. within an image, according to differences in image reconstruction and noise levels across different realizations of that image.

2.D. Recommendation for standard requirements

The following components and functionalities are desirable for the benchmark software:

• A1: Open access to the software and functionalities that are understandable by both developers and users of the segmentation methods.

• A2: Collection of datasets representing the clinical applications requiring validation of PET-AS methods.

• A3: Carefully selected images and RCs checked to only include cases that can provide meaningful metric values.

• A4: Capability to allow future extensions by adding new datasets.

• A5: Implementation of PET-AS methods representing the current state-of-the-art, as described in ref. 6.

• A6: Capability to facilitate the addition of an algorithm that is developed outside the application's framework and that can be used to segment images and produce contours in a format compatible with the benchmark analysis routines.

• A7: Metrics to quantify the agreement between PET-AS volumes and RCs and to evaluate the clinical implications of segmentation inaccuracies.

• A8: Functionalities allowing the evaluation of robustness of PET-AS methods.

• A9: Ability to directly compare the accuracy of the segmentation of:

  1. the same image using different PET‐AS methods
  2. different images using the same PET‐AS method.

2.E. Evaluation of the benchmark

The evaluation of the benchmark aims at addressing the following question: how good are the chosen datasets and metrics at quantifying and comparing the performance of PET‐AS methods? The implicit hypothesis is that the collection of images and metrics provided with the benchmark is appropriate and allows comparing PET‐AS methods with enough accuracy and precision to distinguish between the methods and identify their specific strengths and weaknesses.

To validate this hypothesis, we need to test both the Discriminative power (DP) and specificity of the benchmark for differentiating PET‐AS methods. DP, which here relates to the ability to distinguish between two PET‐AS methods that are close in performance, can be tested by evaluating the range of variation of the performance metrics across the range of PET‐AS methods and images. Specificity, defined here as the ability of the software to detect changes in performance that are linked to the difficulty of the segmentation, can be tested by comparing reference contours with a modified version of these contours which were prepared to be less accurate.

3.A. Data

3.B. Workflow and analysis

The workflow implemented in PETASset is shown in Fig. 3 and includes:

1. Image and VOI visualization

2. Image segmentation

3. Data analysis and evaluation of performance metrics

4. Structured reporting

The input to PETASset is a set of contours the accuracy of which has to be evaluated. The contours can be generated using the (a) default PET‐AS methods provided with PETASset, (b) the segmentation module available in CERR or (c) custom Matlab code (cf. Fig. 1).

A number of different analyses can be carried out in PETASset as shown in Fig. 4. Depending on the used dataset and evaluation metric, a given PET‐AS method can be tested in terms of absolute accuracy and/or in terms of robustness to a specific reconstruction parameter. For instance, testing a PET‐AS method on UCLPTLU data with Level I and II metrics provides a quantitative measure of its performance on clinical lung tumor data. In addition, the robustness of a PET‐AS method to different image acquisitions of the same subject can be assessed using the MILPABB dataset as the standard deviation of results obtained for the same lesions over the different acquisition instances available.

All evaluation strategies, represented by Level I‐II metrics, operate on three types of data:

• PET image

• User‐generated contours

• Reference contour

It is worth noting that in PETASset all RCs and PET‐AS contours in each study are defined in the same frame of reference and in the same co‐ordinate grid as the image space and resolution of the PET data are used. The evaluation strategies implemented in PETASset are described in Section 3.C.2.

3.C. Code

With reference to Fig. 1, the PETASset code system is saved in the “Code” folder. All the code files (or functions), written in the Matlab language, are accessible from the PETASset drop‐down menu and are separated into three categories (a) Segmentation, (b) Analysis, and (c) Report described in detail in the next sections.

4.A. Design and implementation

PETASset was designed and built following AAPM TG211 report which identified the need for developing a standard evaluation framework designed for the assessment of both existing and future PET‐AS algorithms including those derived from supervised machine learning methods.6, 23, 24 Figure 4 shows the workflows available in PETASset and the large range of information that can be extracted from the current version of the benchmark. It is important to note that, although a number of datasets and evaluation metrics are included, PETASset is not required, and therefore not designed, to evaluate each PET‐AS method on all datasets using all metrics. Such a requirement is unrealistic due to different assumptions made about the datasets by different PET‐AS methods and metrics. For example, a PET‐AS method may operate on hybrid data, which assumes the availability of both PET and CT datasets. This assumption may hold for a patient dataset but not for numerical or physical phantoms. The imaging and RC data available in the current version of PETASset are intended to represent the state‐of‐the‐art and have been assembled with the contribution of leading clinical and research institutions in the field. The PETASset datasets include RCs for homogeneous (physical and numerical phantoms) as well as heterogeneous tracer uptake (numerical phantom and clinical data). However, we emphasize that in order to be meaningful, the comparison of PET‐AS methods developed with the same goal should be performed on the same datasets and according to the same metrics.

We therefore recommend that the overall performance of a PET‐AS method is evaluated on all the PET‐ASset data applicable, for a given segmentation method. This can be done via the automated analysis tool embedded in the software, which can select the analysis to be carried out according to the segmentation type. For more specific tests, such as robustness to a particular image reconstruction method, users can follow the process shown in Fig. 4.

The design of PETASset allows users to evaluate segmentation methods themselves. This can be done either by importing segmentation contours produced externally, or by adding a segmentation tool to the benchmark software.

Level II analysis data are the primary output of PETASset since they are easy to interpret and compare between PET‐AS methods and since spatial accuracy is a main concern. Level I metrics provide essential information which put Level II results into context and allow users to make additional considerations regarding the relevance of the observed contouring errors.

PETASset can provide ALs for all evaluation metrics included in the package. This can be used to compare the performance of new and well established PET‐AS methods. The quality and usefulness of these ALs will depend on the quality of data available in PETASset, which will need to be regularly updated. It should be noted that the ALs provided in this document are not recommended for the evaluation of new PET‐AS, since they were derived using a small number of PET‐AS methods and a limited set of images. Further work is needed to produce task‐dependent and reliable ALs for PET image segmentation.

The RC is hidden via encryption in PETASset when the software is distributed to the user. Even then, there is a risk that users optimize their segmentation method blindly to increase the accuracy score of certain algorithms. This optimization may lead to the development of tools that may not perform well outside the package. This risk could be limited by restricting in PETASset the recording of results and the generation of reports to the PET‐AS methods that are tested on all datasets and considering all metrics. Increasing number and diversity of reference images and contours will also help reducing this risk.

Constant improvement and maintenance of the PETASset is needed in order to deal with these challenges.

4.B. Future work

The current version of PETASset is a research tool that can be reliably used to evaluate the performance of PET‐AS methods against reference RC data. The following additional functionalities are expected to enhance impact of PETASset in clinical practice:

1. Web access: provide web access to PETASset data, tools and statistics including reports for selected datasets and segmentation methods.

2. Level III Analysis: design and implement metrics to evaluate the clinical implications of contour accuracy in radiotherapy treatment planning.25 It is envisaged that Level III metrics will operate on reference dose maps calculated using PETASset's RC and distributed with the benchmark.

3. Reference data: the value of PETASset will be enhanced by adding more test data including 4D PET/CT scans and expert consensus VOIs.26 In particular, including images with highly varying degrees of tumor size, activity, contrast and resolution will enable thorough robustness studies in fulfillment of requirement A8. It is envisaged that synthetic datasets generated with PETSTEP will also help growing the PETASset database, in particular to include data specifically designed for testing robustness to the partial volume effect.

4. Imaging modalities: PETASset should evolve to include the next generation of auto‐contouring methods that combine information from different imaging modalities such as CT and MRI.

5. Unified performance score: performance metrics are specific and limited to only certain image or contour parameters. PETASset could be used to combine more metrics in a unified score reporting a single performance value. This would be a desirable feature and research toward such a metric is encouraged.

6. Knowledge‐based PET segmentation: continuously adding to the PETASset database standardized data on the performance of different PET‐AS methods will enable us to start building models and ALs to use as a baseline for the assessment of new PET‐AS algorithms and for the optimal segmentation of virtually every type of PET image.