Improving Mask R-CNN for Nuclei Instance Segmentation in Hematoxylin & Eosin-Stained Histological Images
Benjamin Bancher, Amirreza Mahbod, Isabella Ellinger, Rupert Ecker, Georg Dorffner
Keywords: Mask R-CNN, Nuclei Segmentation, Medical Image Analysis, Deep Learning, Computational Pathology
Abstract: Digital pathology is an emerging topic in the analysis of pathologic tissue samples. It includes providing the tools towards more automated workflows to derive clinically relevant information. Digitization and storage of whole slide images have become commonplace and allow modern image analysis methods to be used. In recent years, computer-based segmentation of cell nuclei has gathered considerable attention in the development of highly specialized algorithms. Currently, most of these algorithms are based on performing semantic segmentation of all cell nuclei and separating overlapping instances in a post-processing step. Recently, instance-aware segmentation methods such as Mask R-CNN have been proposed to enable unified instance detection and segmentation, even in overlapping cases. In this work, we propose a modified Mask R-CNN-based approach by incorporating distance maps of instances and hematoxylin-stain intensities as extra input channels to the model. Moreover, we explore the impact of three well-known inference strategies, namely test-time augmentation, ensembling, and knowledge transfer through pre-training on the segmentation performance. We perform extensive ablation experiments across multiple runs to quantitatively define the most optimal inference strategy in the proposed Mask R-CNN algorithm. Our results show that average instance segmentation improvements of up to 3.5% and 4.1% based on Aggregate Jaccard Index and Panoptic Quality score can be obtained, respectively, using the proposed techniques in comparison to a standard Mask R-CNN model. Our findings confirm the effectiveness of aggregating information at the network input stage and optimizing inference workflows using minimal effort. Implemented modifications and codes are publicly available through a GitHub repository under: github.com/bbanc/Improved-Mask-R-CNN-for-nuclei-segmentation.