Skip to main content

Segmentation-guided multi-modal registration of liver images for dose estimation in SIRT

Abstract

Purpose

Selective internal radiation therapy (SIRT) requires a good liver registration of multi-modality images to obtain precise dose prediction and measurement. This study investigated the feasibility of liver registration of CT and MR images, guided by segmentation of the liver and its landmarks. The influence of the resulting lesion registration on dose estimation was evaluated.

Methods

The liver segmentation was done with a convolutional neural network (CNN), and the landmarks were segmented manually. Our image-based registration software and its liver-segmentation-guided extension (CNN-guided) were tuned and evaluated with 49 CT and 26 MR images from 20 SIRT patients. Each liver registration was evaluated by the root mean square distance (RMSD) of mean surface distance between manually delineated liver contours and mass center distance between manually delineated landmarks (lesions, clips, etc.). The root mean square of RMSDs (RRMSD) was used to evaluate all liver registrations. The CNN-guided registration was further extended by incorporating landmark segmentations (CNN&LM-guided) to assess the value of additional landmark guidance. To evaluate the influence of segmentation-guided registration on dose estimation, mean dose and volume percentages receiving at least 70 Gy (V70) estimated on the 99mTc-labeled macro-aggregated albumin (99mTc-MAA) SPECT were computed, either based on lesions from the reference 99mTc-MAA CT (reference lesions) or from the registered floating CT or MR images (registered lesions) using the CNN- or CNN&LM-guided algorithms.

Results

The RRMSD decreased for the floating CTs and MRs by 1.0 mm (11%) and 3.4 mm (34%) using CNN guidance for the image-based registration and by 2.1 mm (26%) and 1.4 mm (21%) using landmark guidance for the CNN-guided registration. The quartiles for the relative mean dose difference (the V70 difference) between the reference and registered lesions and their correlations [25th, 75th; r] are as follows: [− 5.5% (− 1.3%), 5.6% (3.4%); 0.97 (0.95)] and [− 12.3% (− 2.1%), 14.8% (2.9%); 0.96 (0.97)] for the CNN&LM- and CNN-guided CT to CT registrations, [− 7.7% (− 6.6%), 7.0% (3.1%); 0.97 (0.90)] and [− 15.1% (− 11.3%), 2.4% (2.5%); 0.91 (0.78)] for the CNN&LM- and CNN-guided MR to CT registrations.

Conclusion

Guidance by CNN liver segmentations and landmarks markedly improves the performance of the image-based registration. The small mean dose change between the reference and registered lesions demonstrates the feasibility of applying the CNN&LM- or CNN-guided registration to volume-level dose prediction. The CNN&LM- and CNN-guided registrations for CTs can be applied to voxel-level dose prediction according to their small V70 change for most lesions. The CNN-guided MR to CT registration still needs to incorporate landmark guidance for smaller change of voxel-level dose estimation.

Introduction

Selective internal radiation therapy (SIRT) or radioembolization is increasingly applied for the treatment of surgically unresectable primary liver malignancies and secondary metastases. During this treatment, microspheres loaded with β-emitting radionuclides, including yttrium-90 (90Y) or holmium-166 (166Ho), are infused into selected branches of the hepatic artery according to the vascular anatomy mapped by angiography [1]. Since the selected branches dominate the blood supply to tumors [2], these radioactive microspheres are trapped within the tumors. The high energy, small tissue penetrating range, and concentration in tumors allow these microspheres to deposit higher energy per mass in tumors, while preventing healthy liver parenchyma dysfunction by limiting their irradiation..

In the SIRT planning, the absorbed dose is used to measure the amount of energy per mass (in Gy or J/kg) from ionizing radiation deposited in a volume of interest (VOI), including tumors and healthy liver parenchyma. It serves as a toxicity indicator for tumors and normal tissues and a criterion for determining the amount of injected activity. There are several different methods to determine the injected activity, including mono-compartment [3] and multi-compartment [4] methods and voxel-based approaches [3]. The dose calculation better reflects the underlying biology as the VOI changes from the volume level (the whole liver, tumors, and non-tumoral parts) to the voxel level.

During the pre- and post-treatment studies, multi-modality images are acquired for VOI delineation and dose calculation. In the pre-treatment study, a SPECT/CT scan is performed shortly after administration of 99mTc-labeled macro-aggregated albumin (99mTc-MAA) particles into selected branches of hepatic artery to mimic the activity distribution of 90Y microspheres in the liver [5]. Cone beam CT (CBCT) images are used for delineation of liver perfusion territories (LPTs) [6]. 18F-fluorodeoxyglucose ([18F]FDG) or [68Ga]DOTATATE PET/CT scans are performed for [18F]FDG- or [68Ga]DOTATATE-avid tumors. Contrast-enhanced and diffusion-weighted MR images are acquired for tumors that are not [18F]FDG- or [68Ga]DOTATATE-avid. Following the administration of 90Y-microspheres, a post-treatment study is performed to obtain the PET/CT or PET/MR images of the actual activity distribution inside the liver [7]. This PET image can be used to compute the absorbed dose in different VOIs for evaluation of the treatment irradiation distribution.

Registration of multi-modality images plays an essential role in information integration for SIRT dosimetry. Different methods of liver registration for intra-modality (CT or MR) and inter-modality images (CT and MR) have been studied. These methods include surface-based [8,9,10,11], vessel-based [9, 11, 12], intensity-based [10, 13, 14], and segmentation-based registrations [15, 16]. Some studies combine both liver surfaces and vascular structures for better registration of tumors inside the liver [9, 11], since most tumors are found near vessels. Most intensity-based methods adopt mutual information for image similarity measurement. The segmentation-based methods use liver segmentations obtained either with histogram-based thresholding [15] or with a convolutional neural network (CNN) [16] for guidance of rigid or affine registration. For the application of liver registration in SIRT, Alsultan et al. used a rigid registration method in Simplicit90Y (Mirada Medical Ltd, Oxford, UK) to register contrast-enhanced CT images to low-dose CT images for efficacy evaluation of coil embolization to acquire intrahepatic redistribution [17]. Spahr et al. implemented a registration framework based on normalized gradient fields for liver registration of multi-modality images and evaluated their algorithm through landmarks and deformation field analyses [18]. Nodari et al. used a multi-modality deformable registration algorithm in MIM SurePlan (v7.0.1; MIM software Cleveland, USA), performed by a trained medical physicist, to register the liver tumor contours from MR images to 99mTc-MAA-SPECT/CT and 90Y-PET/CT [19]. Their study indicates that different tumor contours from anatomical and scintigraphic images have no significant impact on mean dose, and registering anatomical tumor contours to scintigraphic images is feasible for improving therapeutic strategy [19]. Besides, our in-house non-rigid liver registration regularized by a spring model was applied in our previous studies on the development of pre-treatment dosimetry [6] and evaluation of the predictive value of 99mTc-MAA-based dose planning [20].

This study describes a (semi-)automatic segmentation-guided registration algorithm and evaluates its performance for registering liver contours and anatomic landmarks and its influence on dose estimation for SIRT. Our in-house image-based registration algorithm was modified for guidance by only CNN liver segmentations and by both CNN liver segmentations and manually delineated landmarks. (Semi-)automatic registration algorithms have the advantage of providing the physicians with integrated image information for more precise dosimetry, while not creating much cumbersome and time-consuming work in the clinical workflow. In our experience, the current lack of automation impedes the full use of multi-modality information. The registration can be challenging, because in the 99mTc-MAA SPECT/CT protocol, the CT is acquired without contrast enhancement, and some MR images from the 90Y-PET/MR studies have severe artefacts. In this study, we wanted to assess (semi-)automatic liver registration and demonstrate its feasibility and value in a clinical context, so that it can contribute to a personalized and precise SIRT treatment with fewer manual interactions.

Material and methods

Data

The training datasets for the CNN model contain 119 CT images from Liver Tumor Segmentation (LiTS17) and SLIVER07 challenges, 30 MR T1 images from the CHAOS challenge, and 50 SIRT CT and 110 MR T1 images from our hospital.

For the registration experiment, 49 CT and 26 MR images from 20 SIRT patients were selected according to several criteria. The selection procedure is presented in Fig. 1. Since the algorithm is designed for registering both CT and MR images in the pre- and post-treatment studies to the 99mTc-MAA CT, there should be at least one CT and one MR from these studies in addition to the CT image from the 99mTc-MAA-SPECT/CT for each patient. Since thresholding a 99mTc-MAA SPECT for lesions may result in overestimation of the tumor targeting performance, landmark delineation (including lesions) for the 99mTc-MAA study was performed on its CT image. 99mTc-MAA CTs in our clinical routine are not contrast enhanced. Therefore, it was important to select patients with visible landmarks in 99mTc-MAA CTs. After patient selection, landmarks visible in all CT and MR images for each patient were chosen and manually delineated by a nuclear medicine physician. Most landmarks were lesions, and some of them were vessel knots and metal clips. For evaluation of liver registration, liver contours were manually delineated by a trained researcher and then corrected by a radiographer with over 10 years of experience in liver delineation. For the hyper-parameter tuning of the registration algorithm, 25 CT and 14 MR images from 10 patients were randomly selected for training, 24 CT and 12 MR images from the other 10 patients were used for testing. The characteristics of the 20 SIRT patients are presented in Table 1.

Fig. 1
figure 1

SIRT patient selection for the registration experiment

Table 1 Characteristics of the SIRT patients for registration experiment

All SIRT datasets for this research were evaluated at KU Leuven after approval by the Ethics Committee Research of UZ / KU Leuven.

CNN structure for liver segmentation

The CNN model used in the paper adopts a U-net structure [21], which was developed for biomedical image segmentation. The U-net structure (see Fig. 2), adapted for 3D images, consists of four resolution hierarchies with three skip connections to combine the high-level liver features from low-resolution hierarchies with the detailed liver features from high resolution hierarchies. Our previous research demonstrated that the CNN segmentations resulted in good segmentation quality without consuming much time and work [22].

Fig. 2
figure 2

Overview of the CNN structure. The model consists of four resolution hierarchies with three skip connections. The input image has the voxel size of \(3\times 3\times 3\) mm3. The input image size is 189 \(\times 162\times 162\) , and the output image size is 163 \(\times 136\times 136\) . The up- and down-sampling rate is three, and the up-sampling is implemented by repetition. The convolutional kernel size for all layers is 3 × 3 × 3

Segmentation-guided registration

The in-house image-based registration algorithm (see Fig. 3) consists of affine registration followed by non-rigid registration. The image similarity metric for affine registration is computed by mutual information. The 12 affine parameters, generated after affine registration, are converted into an initial displacement field for non-rigid registration, where each element represents the displacement of the corresponding voxel in the floating image. The image similarity metric for non-rigid registration is also mutual information. To avoid topology-violating deformations during non-rigid registration, the voxel displacement is regularized through a spring model [23]. Each pair of neighboring voxels is assumed to be connected through a spring, which opposes distance changes. The spring rigidity is used to adjust the regularization power. Therefore, the image-based non-rigid registration algorithm minimizes the weighted sum of the image similarity loss (LI) and the regularization loss (LR). The computation and optimization of LI and LR are explained in detail in [23].

Fig. 3
figure 3

Overview of the in-house image-based registration algorithm

This image-based registration algorithm was extended through guidance by CNN liver segmentations (see Fig. 4). CNN liver segmentations for the reference and floating images are registered via affine transformation to provide initialization of the non-rigid registration. In the next stage, images and their CNN liver segmentations are simultaneously non-rigidly registered. The segmentation similarity loss (LS) is computed as the sum of squared differences between the reference and warped segmentations, representing the segmentations as binary images. The computation and optimization of LS are the same as that of LI when using sum of squared distance as the similarity measurement, as described in [23]. The final loss of the non-rigid registration guided by CNN liver segmentations (CNN-guided) is the weighted sum: \({w}_{I}{L}_{I}+{w}_{S}{L}_{S}+{L}_{R}\).

Fig. 4
figure 4

Overview of the registration algorithm guided by CNN liver segmentations

To explore the possibility of landmark guidance for better lesion registration, a labeled map containing both CNN liver segmentations and landmarks as two classes replaces the CNN liver segmentations in the CNN-guided algorithm with other settings unchanged. This algorithm guided by both CNN liver segmentations and landmarks (CNN&LM-guided) is compared with the image-based and CNN-guided algorithms in the following experiments.

Experiments

CNN liver segmentation

The CNN model trained on CT and MR images from the public challenges, and our hospital was implemented to generate automatic liver segmentations for the 49 CT and 26 MR images from 20 SIRT patients selected for the registration experiment. The results were evaluated through the dice similarity coefficient (DSC) between the CNN liver segmentation and the ground truth liver segmentation. The DSC quantifies the overlap between the segmentations, its definition and computation are explained in [24].

Registration of multi-modality images

For affine registration, the results from using either images or CNN liver segmentations were compared. For non-rigid registration, 1D (\({w}_{S}=0\)) and 2D grid searches were implemented to find the optimal weights without and with guidance of CNN liver segmentations. The non-rigid registration was initialized by either image-based or CNN-based affine registration. To evaluate the value of landmark guidance for better lesion registration, 2D grid searches were implemented to find the optimal weights for the CNN&LM-guided non-rigid registration initialized by the CNN&LM-based affine registration. Each liver registration was evaluated by the root mean squared distance (RMSD) of mean surface distance (MSD) between liver contours and mass center distance (MCD) between landmarks: \(\mathrm{RMSD}=\sqrt{({\mathrm{MSD}}^{2}+{\sum }_{i=1}^{{N}_{\mathrm{landmarks}}}{\mathrm{MCD}}_{i}^{2})/(1+{N}_{\mathrm{landmarks}})}\). Mean surface distance is designed to measure the contour difference between two segmentations [25]. Since a landmark can appear with different shapes and volumes in images of different modality, mass center distance, which is independent of shape and volume changes, is used to evaluate landmark registration. The root mean square of RMSDs (RRMSD) from all liver registrations was used to evaluate each pair of weights. The optimal weights, with the lowest RRMSD, were found through grid searches on the training datasets for CT to CT and MR to CT registrations, respectively. After that, different registration settings were compared on the test datasets through RMSD and RRMSD.

Dose estimation

The CNN- and CNN&LM-guided registrations using the optimal weights were implemented to generate registered landmarks (including lesions) for the floating CT and MR images (registered floating landmarks) from the test datasets for comparison with the landmarks delineated on 99mTc-MAA CTs (reference landmarks). A five-scale Likert score, with its criteria presented in Table 2, was used for grading the floating landmarks registered to the reference landmarks. Dose estimation using the registered floating lesions from the CNN- and CNN&LM-guided algorithms was compared with dose estimation using the reference lesions through the mean dose and the volume percentage receiving at least 70 (V70) and 100 Gy (V100) in the lesion. The absorbed doses of 70 and 100 Gy illustrate an intermediate and high tumor response probability. The injected activities prescribed for the left and right LPTs for each patient were used for dose estimation. LPTs are delineated on CBCTs in our clinical workflow. Dose estimation computed on the left and right LPTs requires registration of CBCT to 99mTc-MAA-SPECT/CT, which introduces extra potential sources of registration errors. To focus on evaluating the influence of CT and MR registration on dose estimation in this study, the injected activities for the left and right LPTs are summed and distributed in a fractional uptake map generated by normalizing all counts of the 99mTc-MAA SPECT within the manually delineated liver contour. The computation of the absorbed dose was based on the local deposition model.

Table 2 Likert score criteria for scoring landmark registration

Results

CNN liver segmentation

The DSCs between CNN liver segmentations of CT and MR images from 20 SIRT patients for registration and manual liver segmentations are presented in Fig. 5. The median DSCs for CT and MR images are around 0.95 and 0.93, respectively. The DSCs for MR images have an outlier with a very low value of 0.72. Through visual inspection, the CNN segmentation for this MR image misses a large part of the liver volume and is incapable of registration guidance. Therefore, the case involving this MR image was excluded for MR to CT registration.

Fig. 5
figure 5

DSCs between manual liver segmentations and CNN liver segmentations of 49 CT and 26 MR images from the 20 SIRT patients selected for the registration experiment. The orange line, green triangle, and red stars represent median, mean, and outliers, respectively. The box corresponds to the first and third quartiles, and the whiskers give the range (except for the outliers)

Registration of multi-modality images

The results of CT to CT registrations are presented in Fig. 6, using the optimal weights for each deformable registration. The RRMSD for CNN-based affine registration is 3.1 mm (27%) smaller than that for image-based affine registration. The image-based non-rigid registration increases the RRMSD for CNN-based affine registration by 0.4 mm (5%). The RRMSD for CNN-guided affine and non-rigid registrationFootnote 1 is 1.0 mm (11%) smaller than that for image-based affine and non-rigid registration. The CNN&LM-guided affine and non-rigid registrationFootnote 2 has a RRMSD 2.1 mm (26%) smaller than the CNN-guided one.

Fig. 6
figure 6

RMSDs and RRMSDs of the test datasets (14 floating CT images) for comparison of image-based, CNN-guided, and CNN&LM-guided CT to CT registrations

The results of MR to CT registrations are presented in Fig. 7, using the optimal weights for each deformable registration. The CNN-based affine registration has a RRMSD 3.4 mm (34%) smaller than the image-based one does. The optimal wI and wS for CNN-guided non-rigid registration initialized by CNN-based affine registration were both zero. Therefore, the optimal performance of CNN-guided affine and non-rigid registration is the same as that of the CNN-based affine registration. This indicates that both image-based and CNN-guided non-rigid registration degrade the results of CNN-based affine registration. The RRMSD for CNN&LM-guided affine and non-rigid registration is 1.4 mm (21%) smaller than that for CNN-based affine registration.

Fig. 7
figure 7

RMSDs and RRMSDs of the test datasets (12 floating MR images) for comparison of image-based, CNN-guided, and CNN&LM-guided MR to CT registrations

Dose estimation

According to the results in the registration experiment, the optimal performance for CNN&LM- and CNN-guided CT to CT registration and CNN&LM-guided MR to CT registration were achieved when using both affine and non-rigid registrations, while the optimal performance for CNN-guided MR to CT registration was achieved when using only affine transformation. These settings using the optimal weights were used to generate the registered landmarks for dose estimation.

Registration of CT to 99mTc-MAA CT

The Likert scores for registration of all landmarks (lesions, vessels, clips, etc.) are presented in Fig. 8a. Over two thirds of registered landmarks had a score of at least 4, which means little or almost no misalignment and insignificant impact on dosimetry. There were in total 29 lesions from 10 SIRT patients used for dose estimation. The relative difference of mean dose and the difference of V70 and V100 between the reference and registered floating lesions using the CNN&LM- and CNN-guided registrations are shown in Fig. 9. There were 59% and 45% of lesions having an absolute relative difference of mean dose smaller than 10% for the CNN&LM- and CNN-guided registrations, respectively. Around 79% (76%) and 83% (69%) of lesions have an absolute V70 (V100) difference smaller than 10% for the CNN&LM- and CNN-guided registrations, respectively. As shown in Fig. 10, the mean dose, V70, and V100 for the floating lesions registered by both the CNN&LM- and CNN-guided algorithms had a strong correlation (\(r\ge 0.95\)) with the mean dose, V70, and V100 for the reference lesions.

Fig. 8
figure 8

Likert scores for the registered landmarks of floating CT (a) and MR (b) images using the CNN&LM-guided and CNN-guided registrations

Fig. 9
figure 9

Relative difference of mean dose (a) and difference of V70 (b) and V100 (c) between the reference and registered floating lesions using the CNN&LM-guided or CNN-guided CT to CT registration. MAA: the reference lesions. CNN&LM: the floating lesions registered by the CNN&LM-guided registration. CNN: the floating lesions registered by the CNN-guided registration. The relative difference of mean dose is computed by (mean dose (floating) – mean dose (reference)/mean dose (reference)). The difference of V70 and V100 is computed by (V70 or V100 (floating) – V70 or V100 (reference))

Fig. 10
figure 10

Passing-Bablok plots for the mean dose, V70, and V100 estimated on the floating CT lesions registered by the CNN&LM- or CNN-guided algorithm versus the mean dose, V70, and V100 estimated on the reference lesions. MAA: the reference lesions. CNN&LM: the floating lesions registered by the CNN&LM-guided registration. CNN: the floating lesions registered by the CNN-guided registration

Registration of MR to 99mTc-MAA CT

According to the Likert scores for all registered landmarks shown in Fig. 8b, around 92% and 63% of lesions registered by the CNN&LM- and CNN-guided algorithms had a score equal to or better than “moderate misalignment and impact on dosimetry”, respectively. There were in total 23 lesions from 10 SIRT patients for dose estimation. The relative difference of mean dose and the difference of V70 and V100 between the reference and registered floating lesions are presented in Fig. 11. Around 70% and 43% of lesions had an absolute difference of mean dose smaller than 10% for the CNN&LM- and CNN-guided registrations, respectively. The CNN&LM- and CNN-guided registrations have around 70% (70%) and 61% (61%) of lesions with an absolute V70 (V100) difference smaller than 10%, respectively. The correlation of mean dose, V70, and V100 between the reference and registered floating lesions is presented in Fig. 12. A weaker correlation (\(r<0.90\)) of V70 and V100 for the CNN-guided registration is observed than the correlation of mean dose for the CNN&LM- and CNN-guided registrations and of V70 and V100 for the CNN&LM-guided registration.

Fig. 11
figure 11

Relative difference of mean dose (a) and difference of V70 (b) and V100 (c) between the reference and registered floating lesions using the CNN&LM-guided or CNN-guided MR to CT registration. MAA: the reference lesions. CNN&LM: the floating lesions registered by the CNN&LM-guided registration. CNN: the floating lesions registered by the CNN-guided registration. The relative difference of mean dose is computed by (mean dose (floating) – mean dose (reference)/mean dose (reference)). The difference of V70 and V100 is computed by (V70 or V100 (floating) – V70 or V100 (reference))

Fig. 12
figure 12

Passing-Bablok plots for the mean dose, V70, and V100 estimated on the floating MR lesions registered by the CNN&LM- or CNN-guided algorithm versus the mean dose, V70, and V100 estimated on the reference lesions. MAA: the reference lesions. CNN&LM: the floating lesions registered by the CNN&LM-guided registration. CNN: the floating lesions registered by the CNN-guided registration

Some examples of the reference and floating lesions registered by the CNN&LM- and CNN-guided methods and their dose volume histograms (DVHs) are presented in Fig. 13.

Fig. 13
figure 13

Examples of the reference and floating lesions registered by the CNN&LM- and CNN-guided methods and their dose volume histograms from three patients. The red, blue, and green contours represent the reference lesion (MAA), the floating lesion registered by the CNN&LM-guided method (CNN&LM), and the floating lesion registered by the CNN-guided method (CNN). a, b the registered floating lesions from the CT of the DOTATATE (DOT) study; CNN&LM: score is 5, relative difference of mean dose is − 0.1%, the differences of V70 and V100 are 0.3% and − 0.1%, CNN: score is 5, relative difference of mean dose is − 0.9%, the differences of V70 and V100 are 0.2% and − 0.4%. c, d the registered floating lesions from the radiology MR (radMR); CNN&LM: score is 3, relative difference of mean dose is − 1.3%, the differences of V70 and V100 are − 0.7% and − 1.1%, CNN: score is 2, relative difference of mean dose is − 5.2%, the differences of V70 and V100 are − 1.0% and − 8.4%. e, f the registered floating lesions from radMR; CNN&LM: score is 3, relative difference of mean dose is 28.4%, the differences of V70 and V100 are 2.5% and 11.1%, CNN: score is 2, relative difference of mean dose is − 32.5%, the differences of V70 and V100 are − 11.1% and − 26.5%

Discussion

Registration of multimodality images

The CNN-based affine registration of CT to CT and MR to CT improves the RRMSD by 3.1 mm (27%) and 3.4 mm (34%), respectively, compared with the image-based affine registration. This substantial decrease shows the advantage of using CNN liver segmentations for affine registration without introducing non-affine deformation. The RRSMD increases by 0.4 mm (5%) when the CNN-guided affine registration is followed by the image-based non-rigid registration for CT images. The CNN-guided non-rigid registration of CT images decreases the RRMSD for the CNN-based affine registration by 0.1 mm (1%). This indicates the slightly negative influence of using only images (including non-contrast-enhanced 99mTc-MAA CT) and the limited improvement of using CNN liver segmentations for non-rigid registration of CT images, given the good initialization provided by the CNN-based affine registration. The optimal weights found for the CNN-guided affine and non-rigid MR to CT registration were both zero indicating that using both images and CNN liver segmentations for non-rigid MR to CT registration could not improve the results from the CNN-based affine registration. This might be caused by relatively poorer CNN liver segmentations for MR images than for CT images. Some CNN liver segmentations for MR images were found to miss some low-intensity lesion regions. This can cause big misalignment in these regions due to liver surface matching guided by CNN liver segmentations during the non-rigid registration. Nevertheless, the CNN-guided registration improves the RRMSD by 1.0 mm (11%) and 3.4 mm (34%) for the floating CT and MR images compared with the image-based registration. Even if there might be errors in CNN liver segmentations compared to the “ground truth”, these unedited CNN segmentations are still helpful for improving the image-based registration. This enables the automation of the liver-segmentation-guided registration without the need of extra manual correction for CNN liver segmentations.

Through landmark guidance, the RRMSD is decreased by 2.1 mm (26%) and 1.4 mm (21%) compared with the CNN-guided registration for the floating CT and MR images, respectively. Since we currently don’t have an automatic lesion segmentation tool for CT and MR images, manually delineated landmarks were used for both registration guidance and evaluation. This might cause a self-fulfilling effect for registration evaluation. However, it does not change the feasibility of using landmarks for better registration, since manually delineated landmarks can be taken as perfect automatic lesion segmentations. This indicates that developing automatic lesion segmentation would be beneficial for registration guidance. Besides, manually delineated lesions approved by the physician are usable in the clinical context.

Dose estimation

A strong correlation (\(r>0.9\)) of mean dose estimation existed between the reference and floating lesions registered by the CNN&LM- and CNN-guided registrations. Landmark guidance for the CNN-guided registration resulted in a smaller difference of mean dose for CT to CT registration than for MR to CT registration. Since mean dose is computed on the volume level, it appears less sensitive to contour changes than the voxel-level dosimetry.

For the voxel-level dosimetry, the CNN&LM- and CNN-guided registrations had around 79% (76%) and 83% (69%) of lesions with an absolute V70 (V100) difference between the reference and floating CT lesions smaller than 10%, respectively. A very strong correlation (\(r\ge 0.95\)) of V70 and V100 existed between the reference and floating CT lesions for the two methods. Landmark guidance for CT to CT registration made small improvement for the voxel-level dosimetry, which was also reflected in the Likert scores given by the physician. Around 70% (70%) and 61% (61%) of MR lesions had an absolute V70 (V100) difference smaller than 10% for the CNN&LM- and CNN-guided registrations, respectively. A weaker correlation (\(r<0.82\)) of V70 and V100 was observed between the reference lesions and the floating MR lesions for the CNN-guided registration than for the CNN&LM-guided registration. Landmark guidance for MR to CT registration helped decrease the discrepancy of the voxel-level dose estimation caused by lesion registration.

It was found that the relative difference of mean dose and the difference of V70 and V100 were smaller than 10% for most lesions with a volume over 50 cc. It is reasonable that small lesions with a small shift can create a relatively large voxel change. Besides, lesions delineated on images of different modality can appear with diverse shape and volume, due to different lesion information expressed in different images or tumor development. Small volumes and large shape and volume differences accounted for a V70 (V100) difference over 10% for 4 (3) out of 6 (7) and 3 (4) out of 5 (9) floating CT lesions registered by the CNN&LM- and CNN-guided algorithms and for 4 (2) out of 7 (7) and 5 (3) out of 9 (9) floating MR lesions registered by the CNN&LM- and CNN-guided algorithms. Good lesion registration does not ensure small difference of dose estimation, since the small size and large shape and volume difference are the other two critical factors with significant impact on dose estimation. It is difficult to eliminate the shape and volume difference between lesions delineated on different images, since each modality reflects a different aspect of lesion appearance. Therefore, it is beneficial to co-register all multi-modality images for joint lesion delineation by the physician, to approach the ground truth delineation by making full use of all information.

Poor lesion registration does not necessarily lead to significant changes of dose estimation. As presented in Fig. 13c, d, the lesion (green) of radiology MR registered by the CNN-guided method, scored with 2, does not have a good overlap with the reference lesion (red), while both lesion contours include most of the high-uptake region. The relative mean dose difference and the V70 and V100 difference between the reference and registered lesions are − 5.2%, − 1.0%, and − 8.4%, respectively. As long as the reference and registered lesions include a similar area of high- and low-uptake regions, the difference of dose estimation can be insignificant despite of poor registration. This indicates that the volume percentage does not necessarily reflect the true energy deposition for each voxel.

In our standard workflow, the liver and lesions are manually delineated on the anatomic image by using the delineation tools from a clinical software package used by the physician for SIRT planning. After that, the delineated volumes of interest (VOIs) are mapped to the 99mTc-MAA SPECT/CT by using the manual or semi-automatic registration tools of the software to register the anatomic image to the 99mTc-MAA CT. To shorten the processing time, the physician delineates the VOIs directly on the 99mTc-MAA SPECT in selected cases, obviating the need for registration. However, proper registration of anatomical images to the 99mTc-MAA SPECT/CT allows lesion delineation on the anatomical images and correlation to the SPECT findings, as recommended in recent international guidelines for SIRT [26]. The standard workflow requires the physician’s interaction during the entire process. In general, the time to complete the standard workflow is around 30 to 45 min. The segmentation-guided workflow consists of liver segmentation, registration, and lesion delineation. To facilitate clinical application and evaluation of these new tools, we have incorporated the entire workflow into the clinical software platform. Liver segmentation is fully automated by the CNN. It takes no more than 5 min for the trained CNN model to generate one liver segmentation by a CPU-based computation server. After that, the CNN liver segmentations are checked and corrected, if necessary, by the physician to ensure its usability for registration guidance, which takes 1 to 5 min. The segmentation-guided registration algorithm is performed by a CPU-based server without parallel computation, which takes around 15 min for each registration in general. Since the registration workflow is fully automated without manual interaction needed, the processing time is acceptable for routine clinical use. It could be speeded up by implementing parallel computation. Lesion delineation is manually performed by using the delineation tools of the clinical software, which takes around 10 min and needs to be automated in the future. In total, the segmentation-guided workflow can take around 30 min. The automated processing takes 20 min, which does not need the physician’s interaction. This makes the segmentation-guided workflow a useful tool for the physician.

In summary, the performance of the CNN&LM- and CNN-guided registrations makes them useful tools for SIRT treatment planning and verification. The deployment of these semi- and automatic registration tools would allow for dose prediction and measurement based on multi-modality images without introducing much manual interaction and workload, which currently impede the application of image analysis tools in the clinical workflow. The pre- and post-treatment studies contain many images with relatively poor quality, including non-contrast-enhanced CTs from the 99mTc-MAA study and MR with severe shading or bias artifacts from the 90Y-PET/MR. Nevertheless, these registration algorithms can produce reasonably good results for these low-quality images giving them practical value for clinical application. Based on these results, we will study the development of an automatic liver lesion segmentation method for fully automatizing the CNN&LM-guided registration. The clinical influence of these registration methods remains to be fully evaluated in a daily SIRT workflow from a volume-level and voxel-level perspective.

Conclusion

Registration guidance using CNN liver segmentations and landmarks greatly improved the performance of the in-house image-based registration. The CNN&LM- and CNN-guided registrations for CT and MR images can be used for the volume-level dosimetry, since the mean doses obtained from the reference and floating lesion contours were very similar. A small V70 and V100 change for most lesions of the floating CT images using the CNN&LM- and CNN-guided registrations demonstrates the feasibility of their application to the voxel-level dosimetry with the physician’s checkup. Landmark guidance is needed for the CNN-guided MR to CT registration to be applied to the voxel-level dosimetry. As a result, the CNN&LM- and CNN-guided registration algorithms could become valuable semi- and automatic tools applicable for SIRT dosimetry based on integration of multi-modality information.

Availability of data and materials

Due to GDPR, we cannot make the data publicly available.

Code availability

The code used in this study may be made available upon request from the corresponding author.

Notes

  1. “The CNN-guided affine and non-rigid registration” always refers to the CNN-based affine and CNN-guided non-rigid registration.

  2. “The CNN&LM-guided affine and non-rigid registration” always refers to the CNN&LM-based affine and CNN&LM-guided non-rigid registration.

References

  1. Padia SA, Lewandowski RJ, Johnson GE, Sze DY, Ward TJ, Gaba RC, Baerlocher MO, Gates VL, Riaz A, Brown DB, et al. Radioembolization of hepatic malignancies: background, quality improvement guidelines, and future directions. J Vasc Interv Radiol. 2017;28(1):1–15. https://doi.org/10.1016/j.jvir.2016.09.024.

    Article  PubMed  Google Scholar 

  2. Breedis C, Young G. The blood supply of neoplasms in the liver. Am J Pathol. 1954;30:969–77.

    CAS  PubMed  PubMed Central  Google Scholar 

  3. Bastiaannet R, Kappadath SC, Kunnen B, Braat AJAT, Lam MGEH, de Jong HWAM. The physics of radioembolization. EJNMMI Phys. 2018;5:22. https://doi.org/10.1186/s40658-018-0221-z.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Ho S, Lau WY, Leung TW, et al. Partition model for estimating radiation doses from Yttrium-90 microspheres in treating hepatic tumours. Eur J Nucl Med. 1996;23:947–52. https://doi.org/10.1007/bf01084369.

    CAS  Article  PubMed  Google Scholar 

  5. De Gersem R, Maleux G, Vanbilloen H, et al. Influence of time delay on the estimated lung shunt fraction on 99mTc-labeled MAA scintigraphy for 90Y microsphere treatment planning. Clin Nucl Med. 2013;38:940–2. https://doi.org/10.1097/rlu.0000000000000241.

    Article  PubMed  Google Scholar 

  6. Jafargholi Rangraz E, Coudyzer W, Maleux G, Baet K, Deroos CM, Nuyts J. Multi-modal image analysis for semi-automatic segmentation of the total liver and liver arterial perfusion territories for radioembolization. EJNMMI Res. 2019;9(1):19. https://doi.org/10.1186/s13550-019-0485-x.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  7. Lhommel R, van Elmbt L, Goffette P, Den Eynde MV, Jamar F, Pauwels S, Walrand S. Feasibility of 90Y TOF PET-based dosimetry in liver metastasis therapy using SIR-Spheres. Eur J Nucl Med Mol Imaging. 2010;37(9):1654–62. https://doi.org/10.1007/s00259-010-1470-9.

    Article  PubMed  Google Scholar 

  8. Lee WCC, Tublin ME, Chapman BE. Registration of MR and CT images of the liver: comparison of voxel similarity and surface based registration algorithms. Comput Methods Programs Biomed. 2005;78:101–14. https://doi.org/10.1016/j.cmpb.2004.12.006.

    Article  Google Scholar 

  9. Foruzan AH, Motlagh HR. Multimodality liver registration of Open-MR and CT scans. Int J Comput Assist Radiol Surg. 2015;10:1253–67. https://doi.org/10.1007/s11548-014-1139-0.

    Article  PubMed  Google Scholar 

  10. Gunay G, Luu MH, Moelker A, van Walsum T, Klein S. Semiautomated registration of pre- and intraoperative CT for image-guided percutaneous liver tumor ablation interventions. Med Phys. 2017;44:3718–25. https://doi.org/10.1002/mp.12332.

    Article  PubMed  Google Scholar 

  11. Cazoulat G, Elganainy D, Anderson BM, et al. Vasculature-driven biomechanical deformable image registration of longitudinal liver cholangiocarcinoma computed tomographic scans. Adv Radiat Oncol. 2020;5:269–78. https://doi.org/10.1016/j.adro.2019.10.002.

    Article  PubMed  Google Scholar 

  12. Charnoz A, Agnus V, Malandain G, Forest C, Tajine M, Soler L. Liver registration for the follow-up of hepatic tumors. Med Image Comput Comput Assist Interv. 2005;8(2):155–62. https://doi.org/10.1007/11566489_20.

    Article  PubMed  Google Scholar 

  13. Luu HM, Klink C, Niessen W, Moelker A, van Walsum T. Non-rigid registration of liver CT images for CT-guided ablation of liver tumors. PLoS ONE. 2016;11(9):e0161600. https://doi.org/10.1371/journal.pone.0161600.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  14. Huang X, Wang B, Liu R, Wang X, Wu Z. CT-MR image registration in liver treatment by maximization of mutual information. IEEE Int Symp IT Med Educ. 2008;2008:715–8. https://doi.org/10.1109/ITME.2008.4743959.

    Article  Google Scholar 

  15. Rieder C, Wirtz S, Strehlow J, et al. Automatic alignment of pre- and post-interventional liver CT images for assessment of radiofrequency ablation. Proc SPIE. 2012;8316:83163E. https://doi.org/10.1117/12.911188.

    Article  Google Scholar 

  16. Hasenstab KA, Cunha GM, Higaki A, et al. Fully automated convolutional neural network-based affine algorithm improves liver registration and lesion co-localization on hepatobiliary phase T1-weighted MR images. Eur Radiol Exp. 2019;3(1):43. https://doi.org/10.1186/s41747-019-0120-7.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Alsultan AA, van Roekel C, Barentsz MW, et al. The efficacy of coil embolization to obtain intrahepatic redistribution in radioembolization: qualitative and quantitative analyses. Cardiovasc Intervent Radiol. 2020;43:391–401. https://doi.org/10.1007/s00270-019-02351-1.

    Article  PubMed  Google Scholar 

  18. Spahr N, Thoduka S, Abolmaali N, Kikinis R, Schenk A. Multimodal image registration for liver radioembolization planning and patient assessment. Int J Comput Assist Radiol Surg. 2019;14(2):215–25. https://doi.org/10.1007/s11548-018-1877-5.

    Article  PubMed  Google Scholar 

  19. Nodari G, Popoff R, Riedinger JM. Impact of contouring methods on pre-treatment and post-treatment dosimetry for the prediction of tumor control and survival in HCC patients treated with selective internal radiation therapy. EJNMMI Res. 2021;11(1):24. https://doi.org/10.1186/s13550-021-00766-x.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  20. Jafargholi Rangraz E, Tang X, Van Laeken C, et al. Quantitative comparison of pre-treatment predictive and post-treatment measured dosimetry for selective internal radiation therapy using cone-beam CT for tumor and liver perfusion territory definition. EJNMMI Res. 2020;10(1):94. https://doi.org/10.1186/s13550-020-00675-5.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  21. Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. 2015. https://arxiv.org/abs/1505.04597.

  22. Tang X, Jafargholi Rangraz E, Coudyzer W, et al. Whole liver segmentation based on deep learning and manual adjustment for clinical use in SIRT. Eur J Nucl Med Mol Imaging. 2020;47:2742–52. https://doi.org/10.1007/s00259-020-04800-3.

    Article  PubMed  Google Scholar 

  23. De Moor K, Nuyts J, Plessers L, Stroobants S, Maes F, Dupont P. Non-rigid registration with position dependent rigidity for whole body PET follow-up studies. In: 2006 IEEE nuclear science symposium conference record, vol. 6. 2006. p. 3502–3506. https://doi.org/10.1109/nssmic.2006.353755.

  24. Zou KH, Warfield SK, Bharatha A, et al. Statistical validation of image segmentation quality based on a spatial overlap index1. Acad Radiol. 2004;11(2):178–89.

    Article  Google Scholar 

  25. Heimann T, van Ginneken B, Styner MA, et al. Comparison and evaluation of methods for liver segmentation from CT datasets. IEEE Trans Med Imaging. 2009;28(8):1251–65. https://doi.org/10.1109/tmi.2009.2013851.

    Article  PubMed  Google Scholar 

  26. Levillain H, Bagni O, Deroose CM, et al. International recommendations for personalised selective internal radiation therapy of primary and metastatic liver diseases with yttrium-90 resin microspheres. Eur J Nucl Med Mol Imaging. 2021;48(5):1570–84. https://doi.org/10.1007/s00259-020-05163-5.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

This project is funded by the H2020-ITN (MSCA 764458) project Hybrid and by the Research Foundation Flanders (FWO) project G082418N. We thank Jeroen Bertels and David Robben for providing the code of the CNN structure. We thank Georg Schramm for providing the tool for image format conversion. We thank Wies Deckers for providing the technical support.

Funding

This project is funded by the H2020-ITN (MSCA 764458) project Hybrid and by the Research Foundation Flanders (FWO) project G082418N.

Author information

Authors and Affiliations

Authors

Contributions

We declare that all authors have contributed to the performed research. Conception and design: XT, EJR, KB, MJG, CMD, JN; Data acquisition: XT, EJR, WC, GM, CV, CMD; Data delineation: XT, RH, WC, CMD; Data analysis: XT, MJG, CMD, JN; Writing manuscript, including contributing to, revising, and approving the manuscript: all. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Xikai Tang.

Ethics declarations

Ethics approval and consent to participate

All procedures performed in studies involving human participants were in accordance with the ethical standards of the Ethics Committee Research of UZ / KU Leuven and with the 1964 Declaration of HELSINKI and its later amendments or comparable ethical standards. We obtained the EC approval for use of all patient data involved in this retrospective study.

Consent for publication

All authors of this article give their consent for the publication of identifiable details, which can include figure(s) and/or videos and/or case history and/or details within the text (“Material”) to be published in the above Journal.

Competing interests

Esmaeel Jafargholi Rangraz is employed by Quirem Medical B.V., Deventer, Netherlands. Christophe M. Deroose is a Senior Clinical Investigator at the Research Foundation Flanders (FWO). Mark Gooding is employed by Mirada Medical Ltd, Oxford, UK, a medical software company. The department of nuclear medicine at KU Leuven receives support from GE for image reconstruction research. No other potential conflicts of interest relevant to this article exist.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Tang, X., Jafargholi Rangraz, E., Heeren, R. et al. Segmentation-guided multi-modal registration of liver images for dose estimation in SIRT. EJNMMI Phys 9, 3 (2022). https://doi.org/10.1186/s40658-022-00432-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s40658-022-00432-8

Keywords

  • Selective internal radiation therapy (SIRT)
  • Liver registration
  • Convolutional neural network (CNN)
  • Internal dosimetry
  • Multi-modality images