Registration of multimodality images
The CNN-based affine registration of CT to CT and MR to CT improves the RRMSD by 3.1 mm (27%) and 3.4 mm (34%), respectively, compared with the image-based affine registration. This substantial decrease shows the advantage of using CNN liver segmentations for affine registration without introducing non-affine deformation. The RRSMD increases by 0.4 mm (5%) when the CNN-guided affine registration is followed by the image-based non-rigid registration for CT images. The CNN-guided non-rigid registration of CT images decreases the RRMSD for the CNN-based affine registration by 0.1 mm (1%). This indicates the slightly negative influence of using only images (including non-contrast-enhanced 99mTc-MAA CT) and the limited improvement of using CNN liver segmentations for non-rigid registration of CT images, given the good initialization provided by the CNN-based affine registration. The optimal weights found for the CNN-guided affine and non-rigid MR to CT registration were both zero indicating that using both images and CNN liver segmentations for non-rigid MR to CT registration could not improve the results from the CNN-based affine registration. This might be caused by relatively poorer CNN liver segmentations for MR images than for CT images. Some CNN liver segmentations for MR images were found to miss some low-intensity lesion regions. This can cause big misalignment in these regions due to liver surface matching guided by CNN liver segmentations during the non-rigid registration. Nevertheless, the CNN-guided registration improves the RRMSD by 1.0 mm (11%) and 3.4 mm (34%) for the floating CT and MR images compared with the image-based registration. Even if there might be errors in CNN liver segmentations compared to the “ground truth”, these unedited CNN segmentations are still helpful for improving the image-based registration. This enables the automation of the liver-segmentation-guided registration without the need of extra manual correction for CNN liver segmentations.
Through landmark guidance, the RRMSD is decreased by 2.1 mm (26%) and 1.4 mm (21%) compared with the CNN-guided registration for the floating CT and MR images, respectively. Since we currently don’t have an automatic lesion segmentation tool for CT and MR images, manually delineated landmarks were used for both registration guidance and evaluation. This might cause a self-fulfilling effect for registration evaluation. However, it does not change the feasibility of using landmarks for better registration, since manually delineated landmarks can be taken as perfect automatic lesion segmentations. This indicates that developing automatic lesion segmentation would be beneficial for registration guidance. Besides, manually delineated lesions approved by the physician are usable in the clinical context.
Dose estimation
A strong correlation (\(r>0.9\)) of mean dose estimation existed between the reference and floating lesions registered by the CNN&LM- and CNN-guided registrations. Landmark guidance for the CNN-guided registration resulted in a smaller difference of mean dose for CT to CT registration than for MR to CT registration. Since mean dose is computed on the volume level, it appears less sensitive to contour changes than the voxel-level dosimetry.
For the voxel-level dosimetry, the CNN&LM- and CNN-guided registrations had around 79% (76%) and 83% (69%) of lesions with an absolute V70 (V100) difference between the reference and floating CT lesions smaller than 10%, respectively. A very strong correlation (\(r\ge 0.95\)) of V70 and V100 existed between the reference and floating CT lesions for the two methods. Landmark guidance for CT to CT registration made small improvement for the voxel-level dosimetry, which was also reflected in the Likert scores given by the physician. Around 70% (70%) and 61% (61%) of MR lesions had an absolute V70 (V100) difference smaller than 10% for the CNN&LM- and CNN-guided registrations, respectively. A weaker correlation (\(r<0.82\)) of V70 and V100 was observed between the reference lesions and the floating MR lesions for the CNN-guided registration than for the CNN&LM-guided registration. Landmark guidance for MR to CT registration helped decrease the discrepancy of the voxel-level dose estimation caused by lesion registration.
It was found that the relative difference of mean dose and the difference of V70 and V100 were smaller than 10% for most lesions with a volume over 50 cc. It is reasonable that small lesions with a small shift can create a relatively large voxel change. Besides, lesions delineated on images of different modality can appear with diverse shape and volume, due to different lesion information expressed in different images or tumor development. Small volumes and large shape and volume differences accounted for a V70 (V100) difference over 10% for 4 (3) out of 6 (7) and 3 (4) out of 5 (9) floating CT lesions registered by the CNN&LM- and CNN-guided algorithms and for 4 (2) out of 7 (7) and 5 (3) out of 9 (9) floating MR lesions registered by the CNN&LM- and CNN-guided algorithms. Good lesion registration does not ensure small difference of dose estimation, since the small size and large shape and volume difference are the other two critical factors with significant impact on dose estimation. It is difficult to eliminate the shape and volume difference between lesions delineated on different images, since each modality reflects a different aspect of lesion appearance. Therefore, it is beneficial to co-register all multi-modality images for joint lesion delineation by the physician, to approach the ground truth delineation by making full use of all information.
Poor lesion registration does not necessarily lead to significant changes of dose estimation. As presented in Fig. 13c, d, the lesion (green) of radiology MR registered by the CNN-guided method, scored with 2, does not have a good overlap with the reference lesion (red), while both lesion contours include most of the high-uptake region. The relative mean dose difference and the V70 and V100 difference between the reference and registered lesions are − 5.2%, − 1.0%, and − 8.4%, respectively. As long as the reference and registered lesions include a similar area of high- and low-uptake regions, the difference of dose estimation can be insignificant despite of poor registration. This indicates that the volume percentage does not necessarily reflect the true energy deposition for each voxel.
In our standard workflow, the liver and lesions are manually delineated on the anatomic image by using the delineation tools from a clinical software package used by the physician for SIRT planning. After that, the delineated volumes of interest (VOIs) are mapped to the 99mTc-MAA SPECT/CT by using the manual or semi-automatic registration tools of the software to register the anatomic image to the 99mTc-MAA CT. To shorten the processing time, the physician delineates the VOIs directly on the 99mTc-MAA SPECT in selected cases, obviating the need for registration. However, proper registration of anatomical images to the 99mTc-MAA SPECT/CT allows lesion delineation on the anatomical images and correlation to the SPECT findings, as recommended in recent international guidelines for SIRT [26]. The standard workflow requires the physician’s interaction during the entire process. In general, the time to complete the standard workflow is around 30 to 45 min. The segmentation-guided workflow consists of liver segmentation, registration, and lesion delineation. To facilitate clinical application and evaluation of these new tools, we have incorporated the entire workflow into the clinical software platform. Liver segmentation is fully automated by the CNN. It takes no more than 5 min for the trained CNN model to generate one liver segmentation by a CPU-based computation server. After that, the CNN liver segmentations are checked and corrected, if necessary, by the physician to ensure its usability for registration guidance, which takes 1 to 5 min. The segmentation-guided registration algorithm is performed by a CPU-based server without parallel computation, which takes around 15 min for each registration in general. Since the registration workflow is fully automated without manual interaction needed, the processing time is acceptable for routine clinical use. It could be speeded up by implementing parallel computation. Lesion delineation is manually performed by using the delineation tools of the clinical software, which takes around 10 min and needs to be automated in the future. In total, the segmentation-guided workflow can take around 30 min. The automated processing takes 20 min, which does not need the physician’s interaction. This makes the segmentation-guided workflow a useful tool for the physician.
In summary, the performance of the CNN&LM- and CNN-guided registrations makes them useful tools for SIRT treatment planning and verification. The deployment of these semi- and automatic registration tools would allow for dose prediction and measurement based on multi-modality images without introducing much manual interaction and workload, which currently impede the application of image analysis tools in the clinical workflow. The pre- and post-treatment studies contain many images with relatively poor quality, including non-contrast-enhanced CTs from the 99mTc-MAA study and MR with severe shading or bias artifacts from the 90Y-PET/MR. Nevertheless, these registration algorithms can produce reasonably good results for these low-quality images giving them practical value for clinical application. Based on these results, we will study the development of an automatic liver lesion segmentation method for fully automatizing the CNN&LM-guided registration. The clinical influence of these registration methods remains to be fully evaluated in a daily SIRT workflow from a volume-level and voxel-level perspective.