Effect of Gaussian filtered images on Mask RCNN in detection and segmentation of potholes in smart cities

1.   Introduction

Several faults such as potholes, cracks and uneven surfaces on roads cause a significant amount of loss of lives in the world. The roads as backbones of transformations contribute to the nation's economic growth. Since 2011, drivers have spent more than $\$$3 billion on car repairs due to pothole damage ^[1]. From the fiscal year 2013 through the fiscal year 2021, the yearly number of potholes fixed in San Antonio, Texas was over 100,520, with COVID-19 accounting for 80,937 of those. This is a significant toll, and the paper suggests that many cases go undetected. The United States' road network is extensive: there are roughly 746,100 miles of road in the country ^[2]. Deep learning has produced several advances and successes in complicated feature extraction and target recognition in recent years. Simultaneously, the combination of deep learning models and computer vision has advanced Artificial Intelligence (AI) in recent years ^{[3,4,5,6,7,8,9]}.

To improve the single-stage object detector's speed and accuracy, YOLOV3 was introduced, which is a combination of improved ResNet as backbone and object box prediction ^[10]. Convolutional neural networks (CNNs) are used to teach computers how to detect and segment objects with a high complexity but little contour and edge information. Maskrcnn is a good model for industrial image identification because it has a clear network topology, high resilience, and simultaneous recognition and segmentation processing ^[11].

Techniques such as sensor-based ^[12,13], image processing and machine learning-based ^{[14,15,16,17,18,19,20,21]}, laser-based 3D reconstruction and stereo vision-based ^{[22,23,24,25,26,27,28]}, are some of the approaches used to automate the pothole detection process on roads. Sensor-based systems employ vibration sensors to detect potholes. The accuracy of detecting potholes may be impacted by false-positive and false-negative readings from the vibration sensor interpreting joints on roads as potholes or not detecting potholes in the centre of a lane, respectively.

Finding an efficient model that will detect and segment potholes with high accuracy and speed is very important, several studies were carried out to perform pothole detection but are based on classification models, Fast RCNN, Faster RCNN and so on. In this study, the following contributions were made to improve pothole detection and segmentation at the same time with higher accuracy and speed of detection.

1) Effect of Gaussian Smoothing Filter on Mask RCNN with different backbone was studied.

2) New Mask RCNN with ResNet-50 and MobileNetv1 backbone was proposed.

3) Gaussian smoothing filter was employed to improve the training images as well as the model's performance.

4) The performance of the models was compared using mean average precision and recall to find the best performing model.

5) The ResNet-50 trained using the Gaussian filtered images outperformed the remaining models employed in the study.

2.   Related works

With the current availability of low-cost cameras and improved image processing techniques, interest have grown in developing new pothole detection models. Traditional image processing algorithms are accurate, but they need time-consuming tasks like manually extracting features and changing image processing parameters. To restore the pavement surface, 3D reconstruction methods acquire 3D road data. A trained model for detecting potholes in 2D digital pictures was created using machine learning techniques. In order to enhance the accuracy of ML techniques, experts must manually extract attributes. Deep convolutional neural network operations are used in DL techniques to automate data extraction and categorization in real time ^[29,30].

DeepMask which produces a mask on the target object instance segmentation found to have low boundary segmentation accuracy is recommended by ^[31]. The first end-to-end instance segmentation framework proposed by ^[32], Full Convolutional Instance Segmentation (FCIS) improve the position-sensitive score map by predicting both the bounding box and instance segmentation, the drawback of this model was, that it was not able to predict the boundaries of occluded objects efficiently ^[33]. Mask RCNN framework, which is an algorithm with relatively improved instance segmentation results among existing segmentation algorithms was proposed ^[7,34]. A mask R-CNN outperforms a faster R-CNN in terms of performance ^[7]. Cucumbers with a similar colored body and leaves are properly detected by a mask R-CNN ^[35]. It also distinguishes between six distinct types of culinary equipment and photovoltaic plant borderlines ^[36,37]. A mask R-CNN has a higher average precision (AP), allowing it to quickly evaluate accident indemnities by detecting the level of vehicle damage ^[38]. It has an AP of 98% for sorting various-sized hardwood planks ^[39], hence it is utilized to categorize the planks for manufacturers development of enhanced image processing methods and the availability of cheap camera sensors have fueled the development of model-based pothole detection systems ^[40]. Traditional image processing algorithms are accurate, but they need time-consuming tasks like manually extracting features and changing image processing parameters. To restore the pavement surface, 3D reconstruction methods acquire 3D road data. A trained model for detecting potholes in 2D digital pictures was created using machine learning techniques. In order to enhance the accuracy of ML techniques, experts must manually extract attributes. Deep convolutional neural network operations are used in DL techniques to automate data extraction and categorization in real time ^[29].

3.   Pothole detection

3.1.   Mask RCNN

Mask RCNN, an instance segmentation algorithm model that can segment objects at the pixel level while identifying targets, was used in this work to detect potholes. Faster RCNN, Region of Interest alignment algorithm (ROIAlign) and Feature Pyramid Networks (FPN) made up the Mask RCNN topology ^[41]. The Mask R-CNN was not meant to coordinate network inputs and outputs pixel-by-pixel. The way RoIPool, the defacto basic process for attending to instances, conducts coarse spatial quantization for feature extraction exemplifies this. The network structure is depicted schematically in Figure 1.

The FPN and RPN of the backbone network execute multi-dimensional feature extraction and information fusion, while the RPN also produces and provides target candidate areas based on extracted feature maps and classifications. Finally, the ROIAlign is used to correct the target region, which is then integrated with FCN to perform target instance segmentation ^[42]. RoIAlign is a straightforward, quantization-free layer that accurately stores precise spatial positions. Under more stringent localization measures, it increases mask accuracy by 10 to 50%. To detect the category, we decouple mask and class prediction and depend on the network's RoI classification branch. Mask RCNN outperforms all state-of-the-art single-model results on the COCO instance segmentation challenge based on ablation experiments conducted by ^[7].

3.2.   MobileNet

The MobileNet ^[44] model is a network model in which the basic unit is depthwise separable convolution. It contains two levels: depthwise and point convolutions, which are regarded separately as convolution layers. Each dense block layer's input feature maps are a superposition of the preceding convolution layer's output feature maps. Between two dense blocks in DenseNet, there is a transition layer. The transition layer uses a 1×1 convolution kernel to decrease the number of input feature maps. MobileNet does not have a transition layer and instead relies on a convolution layer rather than a pooling layer. To minimize the size of the feature map, the convolution layer directly convolutes the output feature map of the preceding point convolution layer with stride 2. The architecture is presented in Figure 2.

3.3.   ResNet

Machine learning experts add extra layers while working with deep convolutional neural networks to tackle an issue in computer vision. These extra layers aid in the faster resolution of complicated issues since separate layers may be taught for different tasks to produce highly accurate outcomes. While the number of stacked layers might enhance the model's characteristics, a deeper network can reveal the degradation issue. Overfitting has not caused this deterioration. It might be caused by the network's setup, optimization algorithm, or, more crucially, the issue of vanishing or exploding gradients. To increase the accuracy of the models, deep residual nets incorporate residual blocks. They solve the problem of disappearing gradients by creating an alternate path for the gradient to follow. They also allow the model to learn an identity function, which assures that the model's upper layers perform equally well as the lower levels. ResNet was built specifically to address this issue ^[8,9]. The architecture is presented in Figure 3.

The ResNet-50 design is based on the Resnet34 model, however, each building block is made up of a stack of three layers rather than two. This model generates 3.8 billion FLOPS and is substantially more accurate than the 34-layer ResNet model. A 50-layer design was created by replacing each of the previous 2-layer blocks with a 3-layer bottleneck block ^[46].

3.4.   Gaussian filtering (smoothing)

Gaussian filters are a class of linear smoothing filters with weights chosen according to the form of the Gaussian function ^[47]. Eq (1) is a very good filter for eliminating noise taken from a normal distribution the Gaussian smoothing filter.

Determines the width of the Gaussian. The two-dimensional discrete Gaussian zero mean function, Eq (2) is used as a smoothing filter for image processing.

3.5.   Dataset and data preprocessing

665 pothole images from ^[48] were used in this study, image annotation was carried out using VGG Image Annotator (VIA) ^[49], and polygons were drawn around the potholes on each image. Before the annotation, the images were made into sets, a Gaussian smoothing filter was applied to one set of the images. 80% of the images were used for training, and 20% for validation. Samples of smoothening filtered and original potholes are presented in Figure 4.

3.6.   Model training

In this study, Mask-RCNN with two different backbones ResNet-50 and MobileNetv1 was employed to detect and segment potholes and to compare the performance of the proposed detection model, weight of previously trained models was adopted using the transfer learning technique, The models were trained for two different scenarios, in the first scenario the model was trained on the original images using ResNet-50 and MobileNetv1 as the backbone, while in the second scenario, the model was trained on images which were smoothen using Gaussian smoothing filter with above-mentioned backbones. Before training the model, the data was augmented to increase the number of images and also the model performance. The data augmentation will also prevent the model from overfitting. The models' hyperparameters were tuned to learning rate = 0.001, learning-momentum = 0.9, weight-decay = 0.0001, detection min confidence = 0.9, steps pre epoch num_classes = 100, maskpool_size pool_size = 14, validation_steps = 100, epoch = 50. The training was carried out using a system with GeForce Nvidia GTX1080, RAM 16 Gb and processor i7. Anaconda environment with jupyter notebook tensorflow = 1.4.0 and keras = 2.0.8 was used for the training environment. The performance evaluation used for the models is mean Average Precision (mAP) and Recall at a threshold of 50 and 75% detection levels. The framework for the employed pothole detection and segmentation is presented in Figure 5.

4.   Results and discussion

In this study, Mask RCNN was employed with different backbone networks and enhanced images using a Gaussian filter to detect and segment potholes, from the study, models' performance based on AP and Recall were compared to find the best performing model. In Table 1, it was observed that the best performing model which is ResNet-50 with Gaussian filtered images achieves the highest performance at 94.43, 96.26 and 98.10% with respect to mAP, mAP-75 and mAP-50, the results also show 2% increase in performance compared to the ResNet-50 trained with original images. The least performing model is MobileNet with 59.38, 70.30 and 75.75% mAP, mAP-75 and mAP-50 respectively. The highest recall was achieved by the ResNet-50 trained using Gaussian Filtered images as presented in Table 2. The results are presented graphically in Figures 6 and 7.

With the modification of the Mask-RCNN by changing the backbone, it was observed that the MobileNet is very easy to train but the performance is not as good as the ResNet-50, in most of the studies carried out using Mask-RCNN, most of the adopted backbones were Resnet101. Looking at the size of our data and the aim of having a model that is lightweight to fit into edge devices and mobile phones, the ResNet-50 can serve as an alternative to ResNet-101 as a backbone, the MobileNet can still be improved to match the performance of the ResNet-50. The Gaussian filter also shows that its effect can improve the performance of the models. Figure 8 shows a test image with the results after detection, Figure 9 shows an example of region proposals and Figure 10 shows segmented pothole.

5.   Conclusions

As discussed potholes are major causes of accidents in the world, and it is very important to develop a model that will detect potholes efficiently in real-time, Also it is very important to develop models that can fit into edge and mobile devices. In this study Mask-RCNN was employed with different backbones, ResNet-50 and MobileNetv1 to detect potholes, also, to improve the model's performance, Gaussian filtering was applied to the images to reduce noise and improve the visibility of the edges of the potholes, it was observed that the ResNet-50 trained with the smoothening filtered images outperformed all the other models employed in the study, also the techniques proves that the Gaussian filtering can improve the performance of models when it comes to detection of the target object.

So far, the different backbones of ResNet-50 and MobileNetv1employed in this study show an acceptable level of performance in pothole detection and segmentation of potholes, the performance of the models can be increased by using larger datasets. In the future, we will employ different backbones and improved backbone structures to improve performance.

Acknowledgments

We will like to acknowledge the International Research Centre for AI and IoT and the AI and Robotics Institute, Near East University for the provided support in this research.

Conflict of interest

The authors declare there is no conflict of interest.