Urban scene segmentation model based on multi-scale shuffle features

1.   Introduction

Remotely-sensed image maps have been widely utilized in various fields such as urban planning due to their real-time characteristics and rich spectral and location information. Semantic segmentation, which involves the recognition and classification of features, has now become an important method for remote-sensing image analysis, providing invaluable data support for regulating urban development, preserving state-owned land and coordinating urban spatial layout. Unfortunately, remotely-sensed images commonly possess distinct target scales, an irregular number of targets and obscured targets, posing tremendous challenges to their interpretation. Therefore, the quality of remote sensing image segmentation is crucial to the further development of this discipline ^[1,2,3,4].

Image segmentation refers to the process of categorizing each pixel and segmenting it into regions that indicate differences between regions and similarities within regions, rendering the image more comprehensible and analyzable ^[5]. Threshold segmentation is done by thresholding the image grayscale values and dividing the image into regions according to the threshold value ^[6]. Region segmentation is to select seed points based on certain similar properties and cluster them according to that property ^[7]. Also, edge segmentation is to find out the image edge pixel points and join the found pixel points together to form the desired region boundary ^[8]. These methods are the traditional methods of image segmentation that are commonly applied.

Presently, thresholding techniques using meta-heuristic equalization algorithms are easily used for image segmentation ^[9]. Region segmentation can be achieved by using a combination of Multiresolution Segmentation (MRS) and Simple Linear Iterative Clustering (SLIC) superpixel algorithms ^[10], and from mathematical and geometric perspectives ^[11], among other approaches to achieve target regions. Edge segmentation is more widely used in medical and remote sensing images. By performing recursive iterations on pre-defined categories based on regions, it enables regions with under-grown high-ranking memberships to be associated with predicted categories in time to improve the accuracy of remote sensing images ^[12]. Furthermore, the features of motion cracks are extracted from medical CT images by an edge extraction-based algorithm as a way to improve the segmentation accuracy after image enhancement ^[13]. Most of the above segmentation is based on features such as grey scale, color, texture and shape of the image, which is mainly based on expert cognition and understanding and manual design. However, this approach has limited generalizability for remotely-sensed images due to the high variability in image features and changing shots over time that make feature-based segmentation less effective ^[14].

In recent years, the use of convolutional neural networks (CNNs) has become prevalent in image segmentation due to the remarkable advancements in deep learning techniques ^[15,16,17]. Various CNN-based semantic segmentation networks have been proposed, including FCN ^[18], SegNet ^[19], PSPNet ^[20], UNet ^[21] and DeepLabV3 ^[22]. A large number of studies have been conducted to improve the network to achieve higher recognition accuracy for the object of study. Adding other modules to the above high-precision network or modifying the structure of the network itself can achieve better understanding and recognition of the model. For example, Xie et al. ^[23] introduced DUSegNet, a distinctive semantic segmentation model for open-pit mines at a pixel level. DUSegNet combines SegNet's pyramidal model and upsampling method of pooling indices with UNet's convolutional skip connection architecture and dilated convolution's intensifier. Wang et al. ^[24] introduced a PSPNet based semantic segmentation network for coal gangue images (SSNet_CG), which extracts feature information by embedding feature fusion channels, an attention mechanism and a three-layer pyramid pooling module, providing a new idea for fast coal gangue recognition. Su et al. ^[25] designed an improved UNet network model that combines the benefits of DenseNet, UNet, Dilated Convolution and DeconvNet to perform remote sensing image segmentation. Liu et al. ^[26] presented a Context-Transfer-UNet (CT-UNet) network to solve the problem of blurred building map boundaries after segmentation and inconsistencies within classes caused by the similarity between buildings and backgrounds. Yang et al. ^[27] addressed the untimely acquisition of wheat inversion information and facilitated the identification of inversion losses in wheat seed selection through a DeepLabV3+ wheat inversion detection model based on multi-headed self-attention. Lastly, Belhadi et al. ^[28] developed a medical segmentation model based on the UNet network to train complex medical data for Internet of Medical Things (IoMT) scenarios.

Although the above deep learning methods show promising segmentation performance, their classification targets are relatively single, mostly scene-specific and of low generalization ability. To address the problems of uneven number of target distribution, difficulty in feature extraction and loss of image information in remote sensing images a network model of GSEPNet is proposed in this paper. The model is based on UNet for remote sensing image segmentation, and introduces shuffle feature extraction module (SF Block), which consists of BlurPool, GSSConv convolution and SE module. First, MaxPool in the routine UNet model is replaced with BlurPool to make up for the missing translation invariance due to data enhancement across blocks and improve segmentation accuracy. Second, in order to solve the problem of difficult feature extraction among multiple features with high similarity in urban remote sensing images, the GSSConv and SE module are introduced. They enhance the interaction of information between parcels, filter redundant information and improve network segmentation performance. Due to the lack of boundary information caused by the sliced image map, a multi-scale attention (MsCA) is constructed to improve the adaptive capture of global contextual information and further enrich the deep semantic information features.

The organization of this paper is as follows: In Section 2, the process of building the GSEPNet network is described in detail and significance modules in remote sensing image segmentation is introduced. In Section 3, the process of making the image dataset is described and the feasibility of the model is verified through a large number of comparison experiments. Section 4 discusses the advantages and limitations of the algorithm. Section 5 concludes the study and provides an outlook for the future.

2.   The proposed Methods

In this paper, the input remote sensing images are scaled to 512×512×3 and passed into the model based on the encoder-decoder UNet network model. The encoder part consists of 4 blocks, where each block consists of 1 BlurPool layer ^[29], GSSConv convolution and channel attention SE ^[30], to reconstruct the SF Block. The number of feature maps is doubled after each downsampling, and MsCA is inserted after block 4, and its output features are fused with the block 3 output feature channel splicing as the input features in the decoding stage, then decoding is performed with bilinear interpolation upsampling after completing two ordinary convolutions. The segmentation of remote sensing images is achieved by repeating the decoding four times and then going through ordinary convolution. The final network model GSEPNet is shown in Figure 1.

In this paper, the segmentation model was proposed to solve the problems of uneven number of remote sensing image targets, difficulties in feature extraction and loss of image information.

(1) In order to solve the problem of missing translation invariance caused by the input of data-enhanced remote sensing images into the network, the missing translation invariance of the MaxPool layer with a step size of 2 can be better mitigated by using the BlurPool layer in the Encoder stage

(2) Due to the segmentation difficulties caused by the existence of highly similar features to parcels in urban remote sensing images, the interaction information between channels is preserved by GSSConv. Then, the SE module is introduced to differentiate the importance of different channel features and to locate and identify target areas more accurately. GSSConv and SE module introduce spatial location relations for the decoder stage not only help the model to detect and extract the same image features, but also improve the model to obtain finer segmentation results.

(3) The processing of remote sensing images in a slice leads to the destruction of the overall structure of the image and increases the difficulty of image category segmentation. The final stage of encoder uses MsCA to aggregate boundary information and various categories of contextual information, in this way, the feature information between different channels can be dynamically interconnected to achieve the ability of accurate feature positioning and improve the performance of network segmentation.

2.1.   SF Block

Because of the large variety and uneven distribution of land parcels in urban remote sensing images, and the high similarity of interlaced distribution between individual parcels, especially for roads and background, it is likely to confuse target feature areas with non-target feature areas during feature extraction, leading to category misclassification. In the encoder stage of the UNet model, the feature extraction network is stacked with two 3×3 Standard Convolution (SC). Although the feature extraction capability and fusion capability of SC are efficient, the lack of information flow between groups and the retention of redundant information will limit the network segmentation performance.

In this paper, we reconstruct the feature extraction network of the model to alleviate the translation invariance of lost remote sensing images, enhance the information interaction between various types of parcels, and filter the redundant information to improve the classification accuracy of the network. The SF Block proposed in this paper consists of the following structures: First, the feature map is guaranteed to have translational invariance through BlurPool. Second, GSSConv is used to maintain the information interconnection between channels and space, which enhances the information transfer between channels and improves the training performance of the network. Finally, the SE module is embedded to weight the importance of the feature extracted images and enhance the classification of parcels with differential information extraction.

2.1.1.   BlurPool

The original dataset was enhanced to solve the problem of uneven distribution of parcels across the remote sensing imagery, resulting in large differences among the accuracy of parcel segmentation. In most semantic segmentation networks, convolutional neural network and pooling are used to provide translation invariance. However, when the step size of the downsampling operation is greater than 2, the output will change drastically upon a small translation or transformation of the input. In this paper, BlurPool is used to alleviate the missing translation invariance of this downsampling and facilitate the system to produce identical responses.

MaxPool is most widely used because it can effectively reduce the amount of data in neural networks, simplify data and speed up data processing. Selecting the maximum value of the original data covered by the convolutional kernel to compress the feature map helps the network to remove redundant information, reduce the number of network parameters, enhance local information acquisition and retain texture information. As shown in formula (1) and formula (2).

Where H_in and H_out represent the input and output lengths respectively. W_out and W_in are input and output widths respectively. padding represents the feature image completion operation, K_S is the window size for MaxPool, D is the element step size in the control window and S is the step length.

The paper introduces BlurPool to proposed module, which combines low-pass filtering with anti-aliasing to retrieve to a certain extent the translation invariance lost by the convolution operation of MaxPool(stride > 1), as shown in Figure 2, where the difference in MaxPool's results is huge for just one pixel translation.

The MaxPool (stride = 2) can be decomposed into two parts, MaxPool with stride = 1 and BlurPool with stride = 2. First, the shift isotropy is maintained by the Max operation being evaluated intensively in a sliding window. Second, an anti-aliasing filter with a kernel of m×m is added in combination with atomic sampling as a better representation of the intermediate signal, as shown in Figure 3.

2.1.2.   GSSConv

The high degree of similarity between parcels inevitably leads to the accuracy of image segmentation, and most segmentation networks focus on contextual information linking of various categories of channel or spatial information, neglecting the fusion of information between categories in space or between channels, resulting in information lag between categories. This paper mainly uses GSSConv in the feature extraction network to make the information of each type of channel or space intermingled. First, the convolution compresses the input feature information to generate dense channel information. Then, the information is infiltrated into the feature information proposed by Depth-wise Separable Convolution (DSC) to achieve complete mixing of SC information into the output of DSC. Next, the channel shuffle is used to achieve dense mixed information of each category. Finally, the dense mixed information is compressed by 3×3 SC to expand the perceptual field and improve the generalization ability of the model, as shown in Figure 4.

The specific steps are as follows, input the feature map R^W×H×C1 from the original image or the output of the pooling layer into GSSConv, where W×H is the spatial dimensional size of the feature map and C1 represents the feature map tensor.

First, the input feature map tensor C1 is halved to C_, and the dense feature information O^W×H×C_ is extracted by taking the feature map R^W×H×C_ and passing it into the SC with a convolution kernel size of 1. Second, the output feature information is input into the DSC to obtain the output feature information P^W×H×C_. Then, the feature information P^W×H×C_ and R^{W×H×C_ are} stitched in dimension, so that the DSC and SC information are intermingled, achieving the output information M^W×H×C2. Next, the features of each group are dispersed to different groups by channel shuffle, which enables the output features to contain features from each group. Finally, the feature information is compressed by a 3×3 SC to extract more semantic information and enhance the network segmentation performance. The main formula for its GSSConv is shown below.

where K_n represents the size of the convolution kernel for each convolution and g represents the number of input feature map subgroups.

2.1.3.   SE Block

When using deep learning methods for semantic segmentation of images, the loss of space or channel information in the feature layer can limit the network segmentation performance. Channel attention SE is introduced in this model, which can assign the importance of information on the channel, focus on the information that is more critical to the current task among the numerous input information, suppress useless information, and improve the representation ability of the network model.

The SE module structure is shown in Figure 5, with the following steps.

The SE module includes three parts: Squeeze, Excitation and Reweight. The feature map of the input module is Q, whose dimension is W×H×C1. W×H is the spatial dimension of the feature map, and C1 represents the feature map tensor. The SE module is implemented through the following. First, through the convolution operation L1, the feature tensor of the input Q is converted from C1 to C2. Then, through the Squeeze operation L_sq, by global average pooling and the feature map Q^W×H×C2 is compressed into X^1×1×C2. Next, the Excitation operation L_ex is performed, using the fitted correlation between the channels of the two fully connected layers. Finally, the Reweight operation multiplies the weight value of each channel calculated by the SE module with the two-dimensional matrix of the corresponding channel of the original feature map, and the result is outputted. The mapping relationship is shown in formula (4) to formula (6).

β_c represents each feature channel. δ is the Relu function. W1 and W2 are the parameters of full-connections operation. The sigmoid activation function represented by η.

2.2.   MsCA Module

Slicing of remote sensing images can cause loss of contextual semantics and boundary information of the sliced images. Most of the semantic segmentation networks obtain contextual semantic information by fusing multi-scale feature information, but they can't extract features adaptively to link contextual information. This paper embeds the MsCA module to associate network context through the attention to assign weights to different scales of information. The MsCA module is shown in Figure. 6. First, the feature map m output from Block4 is subjected to convolution operation to generate feature map q. Then the output feature q is subjected to multi-scale feature extraction, i.e. the feature map q is pooled according to 3 different convolutional kernel sizes. By this method, three different scale feature maps are obtained. Finally, the multi-scale feature map is implemented through an attention mechanism to aggregate boundary information and contextual information between spaces and channels.

The specific steps are as follows. First, the feature map m^W×H×C output from Block4 is halved in dimension to C_ and the 3×3 convolution operation is used to output the feature information as m₁^W×H×C_. Different scale pooling operations are used on the output feature map to obtain feature maps of different scales. The pooling module is 3 layers, each of size 5×5, 7×7 and 9×9, respectively, and MaxPool is applied to the feature maps to obtain P1, P2 and P3, 3 pooled feature maps of different sizes with the same dimension size as C_. Then, pooled feature maps are stacked with m₁^W×H×C_ on the channel to output the feature m₂^W×H×C1. Next, m₂^W×H×C1 is sequentially subjected to a Squeeze operation with global pooling and an Excitation operation consisting of two full joins, and the channel attention weights are extracted using the Sigmoid function for the output features, then the weights are mapped to the feature information m₂^W×H×C1 to obtain the output feature information. Finally, the spatial information of all channels is blended by 3×3 SC compression of the feature information. The MsCA module relationship is as following Eq(7).

where $\varphi ()$ denotes a one-dimensional convolutional channel interaction, Avgpool represents the global average pooling operation, δ is the Relu function, W₁ and W₂ are the parameters of full-connections operation.

3.   Experimental Analysis and Discussion

3.1.   Experimental platform construction

The model was trained on an 16G RAM, AMD R5-3600 CPU and NVIDIA GeForce RTX 2080Ti 12GB GPU provided by Ubuntu 18.04 OS. The deep learning framework is Pytorch 1.6.0, using CUDA Toolkit 10.0 and CUDNN V 7.6.5 as the model training acceleration toolkit.

3.2.   Dataset

The dataset utilized in this paper was obtained from the AI Classification and Recognition of Satellite Images competition held during the 2017 CCF Big Data and Computational Intelligence Conference (BDCI 2017). The dataset consists of land cover samples visually interpreted from high-resolution remote sensing imagery depicting a southern Chinese region, captured using sub-meter resolution and visible spectral bands (R, G, B).

3.2.1.   Data acquisition

The BDCI2017 dataset comprises of three remote sensing images that have been annotated. The dimensions of the images are 4011×2470 pixels, 5664×5142 pixels and 3357×6116 pixels, respectively. The training samples are categorized into five classes: vegetation (marker 1), buildings (marker 2), water bodies (marker 3), roads (marker 4) and others (marker 0). Cropland, forest land and grassland are classified as vegetation. The labels were given pseudo-colors to help visualize subsequent experiments. Figure 7 illustrates a part of the schematic diagram.

The dataset labeling follows a color scheme where red areas represent vegetation, green areas reflect buildings, yellow areas denote water bodies, blue areas indicate roads and black areas serve as the background. Refer to Figure 8 for illustration.

3.2.2.   Data pre-processing

The dataset has many challenges related to image extraction. For instance, direct processing of remote sensing images can cause memory shortages, direct cutting of images can lead to loss of contextual semantics. Additionally, the clipped image samples have the disadvantage of insufficient data volume. Consequently, the samples need to be processed for sliding segmentation and data enhancement. The image samples are pre-processed as follows.

(1) Sliding segmentation: the original image is sliced to save internal storage and enhance contextual semantic information. Select a sliding window of 512×512 and cut one piece per sliding 128 pixels, where the image size is 512×512. The sliding spacing of 128 pixels can effectively eliminate the loss of semantic information between features due to image segmentation. Total 874 remote sensing images were segmented and stored in the format of "jpg". The segmentation effect is shown in Figure 9.

(2) Data enhancement: at the beginning of the model training, image data are enhanced by rotating, panning and mirroring the data set to prevent underfitting of the model while expanding the sample size of remote sensing, so that the model has high detection capability of the target and generalization of the model. The enhanced sample size reaches 3104. These enhanced urban remote sensing images were divided into training and validation sets (at 9:1 ratio), and datasets were produced in PASCAL VOC format to facilitate subsequent experiments.

3.3.   Evaluation indicators

The segmentation report of the proposed model is generated through the evaluation of several measures, including Overall Accuracy (OA), Mean IoU (mIoU), Mean Recall (mRecall), Mean Precision (P) and Dice coefficient (Dice).

OA measures the overall classification accuracy of all samples. On the other hand, mIoU computes the average ratio of the intersection of the predicted and true values of all classes to their concurrent sets. mRecall, on the other hand, represents the probability that the prediction is positive among all positive samples. Furthermore, mPrecision captures the probability that the true class is positive among all samples with positive predictions. Finally, Dice measures the similarity between the sets of predicted positive classes and true positive classes. Meanwhile, the memory (GPU) used by the training model is counted. In summary, these metrics are described as follows:

where S_correct represents all correctly classified samples and S represents the total number of samples. Moreover, C_Sn, P_Sn and N represent the true sample of a class, the sample predicted to be that class and all classes, respectively. TN, TP, FN and FP as shown in Table 1.

3.4.   Experimental parameter setting

In this paper, UNet was chosen as the base model and the network was trained for 100 epochs. We train the model from scratch in the urban image segmentation phase. In order for the network to train the dataset correctly, the images were resized to 512×512 and normalized uniformly before being fed in the model. To ensure the validity of the experimental results, the training and validation sets were randomly shuffled while multiple cross-validations were performed before entering the network. An iterative reduction algorithm was applied to the gradient of this network to decrease the learning rate and improve segmentation performance.

The stochastic gradient descent (SGD) optimizer was used to update the model parameters, with an initial learning rate of 0.00001. This learning rate was adjusted for each step using the "Warmup" method, with a weight decay of 1e-4, momentum of 0.9 and dampening of 0. Due to computer hardware limitations and graphics card memory, the batch size was set to 4. Moreover, we used the dice loss function to mitigate the negative impact of foreground-background imbalance in the sample.

3.5.   Analysis of experimental results

3.5.1.   Ablation experiments

To demonstrate that the GSEPNet model designed in this paper improves translation invariance of feature images, interactivity of feature information and dynamic capture of spatial location information in remote sensing segmentation of urban images, the SF Block feature extraction module and MsCA module were used for ablation experiments under the framework of the network, respectively. To ensure the validity of the experimental results, we conducted all experiments on the same dataset and experimental environment built in Section 3.1 of this paper. The experimental results are shown in Table 2, and a comparison of the loss rate and OA for each model is shown in Figure 10.

Table 2 shows the performance improvement when replacing the feature extraction module of the original UNet network with the SF Block module. The SF Block module effectively addresses segmentation challenges arising from the lack of translation invariance and highly similar features of remote sensing images. The improved OA, mIoU, mRecall, P and Dice are 81.6%, 69.2%, 80.7%, 81.9% and 81.1%, respectively. To further address the original network's bias towards local information extraction, MsCA is introduced to enhance contextual information fusion. The refined network improves OA, mIoU, mRecall, P and Dice by 2.3%, 3.4%, 3.3%, 2.2% and 2.6%, respectively, over the UNet model. Figure 10 shows that the GSEPNet network has a better segmentation performance than the UNet network. Both the SF Block and MsCA modules positively impact the segmentation effect. In summary, compared to the original network model, GSEPNet improves OA, mIoU, mRecall, P and Dice by 4.1%, 5.7%, 5.3%, 3.6% and 4.3%, respectively.

3.5.2.   Shuffle feature extraction Block experiment

To validate the effect of different positions and numbers of SF Block features on network segmentation accuracy, this paper investigates how the position of the shuffle feature extraction module in the network can affect segmentation performance. The feature extraction Blocks of the original UNet network were replaced with various numbers and positions of SF Blocks to conduct the experiments, and the results are presented in Table 3. The positions of SF Blocks in Table 3 are indicated with 0 and 1, representing whether the respective four positions in the network were replaced with SF Blocks. The value of 0 indicates no replacement, while the value of 1 indicates replacement.

The findings presented in Table 3 demonstrate that the stacked replacement feature extraction network has a positive impact on network segmentation, particularly when deploying a higher number of SF Blocks. Adopting a fully SF Block-constructed feature extraction network yields significant improvements, with OA increasing to 81.2% and mIoU to 68.3%, alongside mRecall, P and Dice enhancements of 3.3%, 2.8% and 2.9%, respectively. Our experiments confirm that the SF Block-reconstructed feature extraction network employed in the paper not only addresses the missing translational invariance of data expansion, but also boosts information intermingling and differential information extraction between different locations. Therefore, employing SF Block as the feature extraction module enables optimal feature information acquisition, leading to improved network accuracy.

3.5.3.   Experimental analysis of multiscale channel attention mechanisms

In this study, we evaluate the performance of two models, GS-SPPNet and GSEPNet, which incorporate the SPP ^[31] and MsCA modules, respectively, at the end of the SF Block feature extraction network in a typical encode-decode structural framework for network segmentation. Our experimental comparison, detailed in Table 4, shows that incorporating the multiscale structure has a positive impact on segmentation performance. Specifically, the GSEPNet model outperforms the GS-SPPNet model in terms of OA, mIoU, mRecall, P and Dice by 0.5%, 0.6%, 0.3%, 0.8% and 0.4%, respectively. These results confirm that the MsCA module used in this paper effectively extracts feature-linked contextual information, fuses local features with global information and enhances the aggregation of local object-to-boundary information. Ultimately, our findings suggest that multiscale image fusion using the MsCA module can significantly improve the model's accuracy for network segmentation.

3.5.4.   Comparison tests of different models

To evaluate the effectiveness of our proposed GSEPNet model for segmentation on the BDCI2017 dataset, we compared it against several other models, including FCN ^[18], PSPNet ^[20], DeepLabV3 ^[22], DeepLabV3+ ^[32], LR-ASPP ^[33], MobileNetV2-DeepLabV3+ ^[34] and UNet ^[21]. All experimental comparisons were conducted on the experimental environment built in Section 3.1 of this paper to ensure the validity of the results. The experimental outcomes are reported in Table 5.

As can be seen from Table 5, with the same sample pool for model training, the GSEPNet model proposed in this paper achieves an overall classification accuracy of 83.1%, an mIoU of 71.0%, an mRecall of 82.7%, an P of 82.7% and an Dice of 82.5%, which are higher than other encode-decode segmentation networks, indicating that the GSEPNet model designed in this paper can be effectively used for urban parcel classification and recognition scenarios.

GSSNet occupies the best index scores in more categories, especially in the vegetation category, where it has the highest segmentation IoU of more than 80% compared with the other six models; for the water body category, the proposed model in this paper has only 0.7% lower IoU than the MobileNetV2-DeepLabV3+ model, and also achieves higher segmentation results compared with the remaining models. Analysis of the reasons for this shows that the dataset has a concentrated distribution of vegetation and water bodies, with more samples collected. Roads have a low IoU among all parcel segmentation due to their scattered distribution and high similarity to other targets, where GSEPNet reduces the classification error by introducing the SF Block feature extraction module, which increases the IoU of road segmentation to 68.5%. The building plots have the lowest segmentation IoU because they contain more scattered small targets and the lack of edge information makes it difficult to identify the targets.

As shown in Figure. 11, the urban remote sensing image segmentation m odels of FCN, PSPNet, DeepLabV3, DeepLabV3+, LR-ASPP, MobileNetV2-DeepLabV3+, UNet and GSEPNet were obtained based on the training models, and 4 different urban remote sensing image object block segmentations were selected for analysis, where scenes 1 to 4 are shown from left to right respectively. Comparing the overall segmentation results, the former six networks are less effective than the GSEPNet model and show misclassification in different cases, while the boundary information is handled more ambiguously. GSEPNet uses the SF Block feature extraction module to enhance differential feature extraction and reduce feature errors, and the MsCA module can effectively aggregate multi-scale feature information and capture boundary information.

In scenes 1 and 2, the vegetation and water targets are more widely distributed, GSEPNet handles the target outline and other target boundary information more clearly. In scenes 3 and 4, there are too many small targets in the building category and the targets are not evenly distributed. GSEPNet effectively retains the detail information of small-scale targets compared with other networks. In Scene 4, the road plots is scattered, which makes it difficult to identify each target and the boundary contours are difficult to be segmented, while GSEPNet performs well and is capable of handling complex segmentation scenes.

3.5.5.   Comparison tests of models in other urban remote sensing datasets

To verify the segmentation performance and model generalization ability of GSEPNet on other urban remote sensing images, the HD-Maps dataset is selected for model validation ^[35]. The dataset contains a total of 15 labeled images, including parking (marker 0), road (marker 1), building (marker 2), sidewalk (marker 3) and background (marker 4). In this experiment, the original and labeled maps of the dataset are cut into 512×512 size for training and validation using the sliding segmentation in Section 3.2.2, and the training and validation sets are divided in the ratio of 9:1. The GSEPNet model is compared with PSPNet, DeepLabV3+, ResUNet and UNet segmentation models. To ensure the validity of the experimental results, these experiments were conducted in the experimental environment constructed in Section 3.1 of this paper, and the experimental results are shown in Table 6. The segmentation effect of GSEPNet model is shown in Figure 12. Due to the uneven distribution of the number of categories in the HD-Maps dataset and the sporadic distribution of some categories, it is difficult to segment the tiny targets. The training results show that the GSEPNet model can segment better than the other four segmentation models without any processing of the dataset categories, so the GSEPNet model has higher generalization in the segmentation of urban remote sensing images.

4.   Discussion

In semantic segmentation, the effectiveness of segmentation is related to the selection of networks, the optimization of parameters and the construction of models. This study is focused on the recognition and segmentation of urban features. The recognition of features in remote sensing images relies on spatial location relationships and textures, and requires a combined shallow and deeper network layer for feature extraction, so UNet is chosen as the basis of the segmentation model.

The advantages and disadvantages of the method are discussed. The advantages of this study's approach to urban feature segmentation are as follows.

(1) The image dataset is more informative as it uses publicly available datasets.

(2) The algorithm is also computationally inexpensive as the equipment used to train it is simple, and most computer devices are capable of supporting the model.

(3) GSEPNet has excellent performance in segmenting urban features.

The proposed algorithm also has the following limitations:

(1) The training set has only 2130 images, a small sample of the dataset. Expansion of the dataset can improve the network performance.

(2) Due to the limitation of experimental equipment, the segmentation effect of the model will be improved if the network parameters, such as batch_size and epoch, are further optimized.

5.   Conclusion

This paper proposes a deep learning-based urban remote sensing image segmentation network named GSEPNet. It addresses the challenge of uneven distribution of quantities, undifferentiability among parcels and loss of boundary information by incorporating a shuffle feature extraction module and a multi-scale attention mechanism into the UNet network. The GSEPNet model proposed in this paper, which yielded OA of 83.1%, mIoU of 71.0%, mRecall of 82.7%, P of 82.7% and Dice of 82.5% achieves relatively high segmentation results on the BDCI2017 public dataset. Compared to other models such as FCN, PSPNet, DeepLabV3, DeepLabV3+, Iraspp, MobileNetV2-DeepLabV3+ and UNet, GSEPNet achieved the highest segmentation accuracy and average overlap among the eight networks. Further, it is proved experimentally that using shuffle feature extraction module and multi-scale attention in the GSEPNet model can improve the image segmentation effect of the network. The segmentation model proposed in this paper can better segment the feature classes in remote sensing images, and is highly feasible. The feature segmentation in urban remote sensing images mainly relies on spatial location information, and the image information is constantly increasing. Therefore, future research needs to consider how to further increase the depth of the network, reduce the amount of computation in the network and challenge more complex datasets.

Acknowledgments

This research was funded by Project supported by the National Natural Science Foundation, China, grant number 52065033

Conflict of interest

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this paper.