A deep learning based automatic segmentation approach for anatomical structures in intensity modulation radiotherapy

1.   Introduction

Patients with intensity-modulated radiotherapy need to go through several steps, including positioning, target contour, planning, plan design, plan verification, plan execution, etc. The contour of target volumes is one of the key steps in the preparation of a radiotherapy treatment plan of intensity modulated radiotherapy ^[1,2]. Nevertheless, manual contour is time taking process and conditional on intra- and inter-observer differences although established on standard guidelines. Multiple studies have shown that the contouring consensus among different oncologists at various hospitals is poor, to solve this radiotherapeutic problem, multiple studies have focused on automatic contouring ^[3,4].

With the improvement in the speed of computer calculations and the improvement of the images processing makes it possible to develop approach for delineation automatically for radiation therapy. In recent years, automatic segment has become one of the most popular research in the field of radiotherapy, atlas and deep learning based automatic contouring methods are the two mainly technologies at present. As early as 2009, Mahdavi et al. ^[5] reported and characterized a semi-automatic prostate segmentation method for prostate brachytherapy.

Coupled with the large number of patients in radiotherapy, the research can acquire much more the learning data. In radiation therapy automated methods for sub-volume oriented dose-escalation plans can assist medical examination ^[6,7], the precise contour can reduce the toxic effect of normal tissues and benefit patients with longer survival periods such as nasopharyngeal carcinoma (NPC), breast and rectal cancer patients.

The objective of this study is to investigate the automatic techniques into NPC, breast and rectal cancer patients radiotherapy planning practice through automatic segmentation approach, which based on the deep learning method. The manual contours were acted as a reference for the study.

2.   Materials and methods

2.1.   Patient data

Thirteen NPC, ten breast cancer and ten rectal cancer patients, who received IMRT between Oct 1^st 2019 to Oct 1^st 2020 and underwent contrast-enhanced planning computed tomography (CT), were selected for this study. Written informed consent was obtained from each patient, and all studies involving people, medical records were approved by the Clinical Research Ethics Committee.

2.2.   Contour and planning

2.2.1.   Manual contour

The organs at risk (OARs) were contoured by one radiation oncologist who has more than 10 years' experience in the field of contouring (e.g., NPC was contoured by a H & N oncologist, etc.) according to the recommendations of the European Society of Therapeutic Radiation Oncology Guidelines on planning CT. The same radiotherapist contoured OARs for three sequences on Multiplan2.4.1 system (Accuray Inc, USA), the OARs in NPC mainly contain: Lens, Optic-Nerves, Parotids, BrainStem and Spinal Cord; Lung, Heart and Spinal Cord for breast cancer while Bladder and Femoral heads for rectal cancer ^[8].

2.2.2.   Automatic segment approach

The OARs were automatically contoured on the same images using automatic segment approach. The approach based on deep learning and traditional algorithm model, which divided into three parts: data collection, annotation, model training and prediction ^[9]. The module builds loss function pool, image segmentation model pool and selectable parameter pool after obtaining training set data. The training model is constructed by randomly selecting the corresponding loss function, image segmentation model and training parameter value based on the selectable parameter pool, and the training model is trained based on the training set data to obtain the trained training model ^[10]. Then, the DSC value of each trained training model is calculated on the corresponding verification set. Finally, the corresponding training model with the highest DSC value is selected as the final OAR automatic.

2.2.2.1.   The training network

The default network in deep learning network is an adaptive structure strategy similar to 2D-UNet ^[11], it contains 5 maximum pooling layers and 5 upper adoption layers, The basic convolutional core channels are 32, size 3 × 3, Behind each convolutional layer are batch normalized (Batch Norm) layers and a linear rectifier function (ReLU), the jump connection is cascaded between the encoder and the decoder, combine high-resolution details and deep image semantic information to improve the effect of sketching the model. The network architecture is shown in Figure 1.

2.2.2.2.   Loss function

Furthermore, the loss function is set to Dice Loss, learning rate to 3e-4, optimizer set to Adam, Batch Size to 16 and the total number of iterations is 100 epochs.

2.2.2.3.   Assessment

There are two evaluation methods in approach: cross-validation and training/test separation validation. The first method divides the dataset into the same number of k subsets. Use k-1 subsets for training each time and the other as a test set. Repeat the training for k times and evaluate the results. The second method divides the dataset into two parts for training and testing, which applies to those with a large number of cases. For small dataset, we selected k = 5 for cross-validation while with big dataset we chose 80% for the training and 20% for the test.

For evaluation, we adopted two quantitative evaluation metrics while the manual contouring served as the reference standard for contouring time and DVH parameters comparisons^[12].

2.3.   Evaluations of automatic contours

The quality of the automatic is critical, numerous measures are utilized to compare the volumes between automatic and manual contouring including Time, Dice Similarity Cofficient (DSC), Hausdorff Distances (HD), Jaccard Distance (JD) and DVH parameters ^[13].

2.3.1.   Dice similarity coefficient

A and B represented automatically contoured and manually contoured OARs, respectively. As the contours approach agreement, the DSC value approaches 1; as the volumes diverge into two nonoverlapping structures, the DSC value goes to 0. The calculation of DSC in commercially available registration systems is not standard, although its inclusion in commercial software is a recommendation of TG132. However many professional literature reported that the values of DSC > 0.700 shows optimum overlap of contours ^[14,15].

2.3.2.   Hausdorff Distances (HD)

where h(A, B) is the fixed HD from A to B. where h(A, B) = max(min(||a-b||)), ${\rm{a}} \in A, b \in B$, when the HD near to zeros, the difference between the auto contouring and manual contouring get smaller.

2.3.3.   Jaccard Distance(JD)

JD is used to describe the dissimilarity between sets. The larger the JD, the higher the sample similarity. Where A and B represented automatically contoured and manually contoured OARs, respectively.

2.3.4.   Time evaluation

The time needed by the approach to generate contours was measured by when the CT was imported until the end of the entire generation process.

2.3.5.   Evaluation of DVH Parameters

Once the automatic completed, the automatic contours were sent back to the Pinnacle Treatment Planning System (TPS, version 9.8, Philips Radiation Oncology systems, Madison, WI). Then the planning dose based on manual contouring were copied to the automatically contours for planning parameters evaluation and comparison. The prescribe dose are 60Gy/30fx for NPC, 50Gy/25fx for breast cancer and 50Gy/25fx for rectal cancer.

We mainly use the max dose (D_max), the Dose of the certain percentage of the volume (D_x), and the volume of receiving the certain dose of organs (V_x) for DVH Parameters. Figure 2 shows the whole workflow of the evaluation. Dose assessment of OARs reference RTOG0615 (Radiation Therapy Oncology Group, RTOG0615) ^[16].

2.4.   Statistical analysis

All statistical comparison were performed using Origin2019. A comparison of the time required to contour and DVH parameters were performed using paired t test, with significance set at p < 0.05.

Here, X_1, X₂ represented automatically contoured and manually contoured. S² represented the total variance, d₀ represented the average difference between the two samples.

3.   Results

3.1.   Evaluation of the time

The time required to manually contour the OARs were 56.5 ± 9.0 minutes (mean ± standard deviation) for NPC, 23.12 ± 4.23min for breast cancer and 45.23 ± 2.39 min (mean ± standard deviation) for rectal cancer. And auto contours required 1.5 ± 0.23minutes, 1.45 ± 0.78 and 1.8±0.56 respectively (Table 1). The difference of requiring time savings of 70%, which were all highly significant (p < 0.0001).

3.2.   Contours and DVH parameter evaluation

Regarding the clinical demand and the clinical use of automatic software, we only selected the partial OARs for comparison. We just selected the Eyes, Lens, Optic path, BrainStem, Spinal Cord, Parotids for NPC, while selected Lung, Heart, Spinal Cord and Bladder, Femoral heads for breast and rectal cancer respectively.

Mean values and standard deviations of parameters that evaluate the contours generated by the automatic software, which use manual contours as a reference, for each organ of the NPC, breast and rectal cancer.

Tables 5, 8, 10 showed the OAR geometrical quantitative evaluation parameters of 33 patients between automatic contours with manual contours. Automatic contours overlapped with manual contours of Eye with the highest degree DSC of 0.907 ± 0.02 while with the poorest DSC of 0.459 ± 0.112 of Spinal Cord for NPC. Among the HD of OARs, the maximum HD was (23.0 ± 1.5) mm in Spinal Cord, and the minimum was (2.45 ± 0.31) mm in lens. The maximum JD in eye was 0.89 ± 0.03 while the minimum in Spinal Cord was 0.38 ± 0.2; And Lung with the highest degree DSC of 0.944 ± 0.03 while with the poorest DSC of 0.709 ± 0.1 of Spinal Cord for breast. Among the HD of Lung-L, Lung-R, Heart and Spinal Cord, the maximum HD was (20.7 ± 5.3) mm in Lung-L, and the minimum was (3.15 ± 0.8) mm in Spinal Cord. The JD in Lung and Heart were greater than or equal to 0.8 while that in Spinal Cord was 0.7 ± 0.1; And Bladder with the highest degree DSC of 0.91 ± 0.04 while with the poorest DSC of 0.43 ± 0.1 of Femoral heads for rectal cancer. Among the HD of Bladder and femurs, the maximum HD was 35.3 ± 6.9 in Femur, and the minimum was (2.87 ± 1.1) mm in Bladder. JD in Bladder was the greater than 0.9 while the mean value in Femur was 0.4. The contours achieved good contour level, however the Dose parameters were not similarity with the contours. Even if the contours reach the high level, the DVH Parameters of automatic contours had the significantly difference with the manual contours from Tables 2–4, 6, 7 and 9.

We generated dose volume histograms dosimetric parameters for each patient from the automatic contours and the reference contours. Examples of these dosimetric endpoints include the mean dose (D_mean), the minimum dose to the hottest x% of a structure (D_x). The full list of parameters investigated in this study is given in Tables 2 to 9. Studies of Tables gives qualitative representation of the best match contours in our cohort, where the Eyes had best geometric overlap, yet the doses had significantly difference with the manual contours.

4.   Discussion

Medical image segmentation using deep learning methods has been widely studied at present. All deep learning algorithms are designed to improve the accuracy of automatic segmentation ^[17,18]. The classical deep learning techniques are mostly based on convolutional neural network (CNN), which can be trained on patched that belong or not the segmentation class ^[19]. A problem with this approach is that the patches may overlap with the same convolution calculated multiple times. A series of image segmentation models are derived from the CNN, but the improved accuracy depends on an increase in the number of network layers which aggravates the over-fitting problem ^[20]. In addition, the output image size is far less than the input, and cannot meet the end-to-end medical image segmentation needs. The U-net ^[21] is based on two-dimensional convolutions on slices of volumetric medical data, it consists of two stages, one contracting (reduction in image size) and one expanding (restoration of the original image size). The approach used in the paper is a flexible deep learning platform and modular back-end architecture which embedded U-net. The approach backend consists of four components, including a data manager, model trainer, model evaluation, and model deployment. The model has been extensively tested previously. Christ et al. ^[22] segments the liver, U-Net1 segments the liver from the input image and then enters the results into U-Net2 for segmenting the liver tissue, U-Net2 also segments the liver tissue directly from the input image. Two U-Net form complex connections in this way, forming cascading fully convolutional neural networks, achieving results with both liver and lesion segmentation DSC values higher than 94%. By connecting the deep learning network in string and parallel, the network depth and the segmentation specificity are increased.

In our research, we used the clinical automatic segment approach for contouring OARs in NPC, breast and rectal cancer. The automatic system will be widely application in clinical radiotherapy while manual contouring proved to time-consume. From Table 1, the deep-learning automatic contouring approach, allowed significantly time-saving work for the radiation oncologists. And many methods exist to compare the automatic segmentation and manual contour. Ciardo et al. ^[23] also investigated 47 patients treated with breast cancer with Atlas based segmentation in the supine position during breast conservation surgery or non-conservation surgery, editing atlas-based volumes save time (12%) compared with manual contouring (44%). Henry et al. ^[24] assessed atlas-based segment (atlas-based segment, ABS) in 9 various OARs, and lymph node volumes in 30 patients with breast cancer through the software result in a 40% decrease in structures contour time. Eldesoky et al. ^[8] reported efficacy and precision by developing clinically acceptable contours faster compared to manual contours and also declared the average time for manual contour was reduced to 94% for all structure after lumpectomy. Macchia et al. ^[25] compared 3 commercial software (ABAS, MIM Maestro, VelocityAI) for atlas-based segmentation applying them to a number of the prostate, Head and Neck, and pleura. In this paper, the physician can save one hour for the H & N site, the time saved was about twenty minutes for a mesothelioma patient, and 40 minutes for a prostate patient.

The most distinction advantage of the automatic contour methods compared to the manual contour is saving the time. However, we need to think whether our research can meet the requirement of clinical demand by obtaining faster contouring time. Deep learning framework ^[26,27] can be utilized for medical image diagnosis, accurate diagnosis is the foundation of accurate contour. Based on prior knowledge, radiotherapy requires accurate radiation delivery and precise contouring of the OARs. CT images typically have high resolution and provide detailed anatomical information. However, limited physiological information can be captured by CT imaging, in order to evaluate target volumes effectively in automatic segment, medical imaging modalities such as the CT and positron emission tomography (PET) are widely used. Song et al. ^[28] proposed a novel approach for the segmentation of the PET-CT images, which makes use of the strength from both modalities: the functionality information of PET and the anatomical structure information of CT, both Lung and N & head tumor are well validated. Rundo ^[29] also proposed fully automatic multimodal PET/MRI segmentation method to help the clinicians to define a CTV that includes both metabolic and morphologic information. The precise of the dose delivery with relate to the precise contours of Targets and OARs, and the toxicity correlated with the risk of the OARs dose received. Therefore, we discussed the DVH parameters of each contour which was delineated by automatic segment approach. The Gemetric measures did not predict the accuracy of dosimetric parameters determined by automatic software in our research. The results were similar to those of Robert et al. ^[30].

This work represents the investigation of the impact of an automated contouring on estimates of radiation dose to the whole body patients. It demonstrates that dosimetric information of automatic contours from CT images is difference with the DSC. We deduced that the result with a perfect Dice-similarity coefficient one would expect to have perfect accuracy with dosimetric endpoints ^[31]. However, on the contrary we investigated that geometric parameters evaluation such as DSC which was heavily influenced by the volume of a structure, but may not consistently reflect whether an automatic contour accurately represents actual radiation dose.

For the NPC, all the OARs have good DSC (>0.7) besides Lens, Optic Nerve and Spinal Cord which due to the small size. Small size of organs is more likely to contoured error such as Lens. In addition, for small size organs such as OpticNerves with unclear tissue boundaries, there were still a certain deviation between automatic and manual. On the other hand, we can learn from Tables 5 and 8, the volume and the exposed dose of Spinal Cord had big distinction in Head and Neck, we can obtain the differences of Spinal Cord from Figure 3, which because of the precise contour of the boundary between the Spinal Cord and BrainStem. In manual contouring, the Spinal Cord contoured to C2 while contoured to upper C1 by Automatic segment approach. So the dose of Spinal Cord contoured by automatic higher than the manual. In Figure 4, we found that the smaller the size of the Organ the less difference of the dose. But there are great difference of the contouring of small organs, many scholars tried to develop standard model to improve contouring accuracy, Yang et al. ^[32] proposed an automated structure nomenclature standardization framework simulates clinicians' domain knowledge and recognition mechanisms to identify small-volume organs at risk (OARs) with heavily imbalanced data better than other methods.

For breast cancer, the Lung and the Heart which had big size no matter of the volume (p > 0.05) and DVH distribution (p > 0.05) had no difference between two methods. At the same time, we can see that the Spinal Cord of thorax is more accurate than the Neck. Multiple studies with automatic contouring come from the breast cancer, which mainly because of the boundary of OARs was clear and density contrast was obvious. In the studies of Robert et al. ^[33], Xu et al. ^[34] and Henry ^[24] et al., which are all based on breast cancer patients, their research all have achieved well clinical validation, and the same trend of clinical validation was also found. Therefore good correlation was obtained in terms of DSC, volume or dose parameters.

For rectal cancer, it can meet the clinical requirement and also have the similar dose of the Bladder because of the obvious boundary. Other studies have investigated the automatic contouring of rectal cancer using different methods, Song et al. ^[3] using deep learning for rectal cancer postoperative radiotherapy, it concluded that convolutional neural networks at various feature resolution levels well delineated rectal cancer CTVs and OARs, displaying high quality and requiring shorter computation and manual correction time. Because the femoral head has obvious bone markers, it should be well delineated automatically. However in Table 10, we found the DSC and JD was lower than the clinical predict, the contour and the dose all had the differences. From Figure 3, we can clearly see that manual contour the partial Femoral heads. After communicating with radiation oncologist, the contouring method of different department present small difference according to the guideline.

5.   Conclusions

In conclusion, we investigated the contour of NPC, breast and rectal cancer by automatic segment approach. The value of DSC does not fully reflect the accuracy of dose distribution. For the obvious boundary of the Heart, Lung, Bladder, they can be completely dependent on automatic contouring. However, the strict validation is also required for small size OARs. In addition, for the Femoral heads, Spinal Cord and BrainStem contour guideline can be obtained using the semi-automatic method and validation when required. In the future, we will take more effort at improving the contouring accuracy of small volume organs and establishing the standardization of contouring. Furthermore, we also will extend our work to the segmentation of the tumor from multi-modalities.

Acknowledgments

The authors would like to express their sincere gratitude to the technicians from MANTEIA Inc. and radiotherapists at the Radiotherapy Department of Jinling Hospital for their assistance in preparing the image scans.

Conflict of interest

The authors declare no conflicts of interest.