Lightweight network study of leather defect segmentation with Kronecker product multipath decoding

1.   Introduction

Leather is one of the most essential raw materials for household items in people's lives today, for example, fur clothing, leather shoes, leather seats, etc. Most studies on leather defect detection algorithms are based on computer vision and mainly focus on finished leather ^{[1,2,3,4,5,6,7]}, but there are fewer studies on wet blue leather. Compared with finished leather, wet blue leather is still in a semi-processed state, its leather is moister and its surface is relatively uneven. So the captured images of wet blue leather are also more likely to have uneven light and dark and are more difficult to detect.

Depending on the detection effect, leather defect detection algorithms are mainly classified into classification and segmentation. Classification algorithms detect defects in the form of square wireframes and include principal component analysis ^[8], support vector machines ^[9,10], and multilayer perceptrons ^[11,12]. The segmentation algorithm is used to segment the defects based on the edges of the defective part so that it can calculate the exact utilization of wet blue leather. This algorithm includes frequency analysis ^[13], statistics ^[14,15], modeling methods ^[16], and deep learning methods.

For defects with black lines and wrinkles, the classification and segmentation detection algorithms are proposed in ^[17]. In which, the AlexNet is used for the image classification algorithm and the U-Net is used for the segmentation algorithm. They achieved a $95\%$ classification detection performance and a $99.84\%$ crossover rate in their experiments with 250 defective samples and 125 non-defective samples. In ^[18], the authors presented a texture defect detection algorithm based on a deep convolutional generative adversarial network, which reconstructs the input image so that it is different from the original input image to achieve initial defect segmentation. In addition, the linear discriminant analysis (LDA) sub-module for error elimination detection has been added to the model. However, the leather detected by the algorithm belongs to the finished leather, while the wet blue leather has not been experimented with. The detection objects are mainly items with regular texture on the surface, which is not conducive to practical applications.

For the surface detection of five types of wet blue leather defects as decayed surfaces, open wounds, puncture wounds, insect bites, and rotten grains, Chen et al. ^[19] introduced a leather defect segmentation algorithm based on hyperspectral technology. The algorithm first uses hyperspectral to image the leather and then uses a 1D-convolutional neural network (CNN), 2D-UNet, and 3D-UNet to segment the defect areas, and the three networks have different advantages for different leather defects. Shen et al. ^[20] constructed a kronecker product matching (KPM) module using the Kronecker product to find the associated features by running the features of two images through the Kronecker product to achieve the character of recognition. Xiao et al. ^[21] constructed a new fully-connected way of a separable layer using the Kronecker product and verified that a separable layer is better than the fully-connected layer for classification by replacing the fully-connected layer in the classical classification recognition network.

Existing deep learning algorithms detect fewer types of wet blue leather defects, and there is still room for improvement in detection speed. To detect more defects and improve the detection speed, a lightweight semantic segmentation network with multi-path decoding is proposed in this paper. First, a module that can extract global semantic information from the image is constructed using the Kronecker product. Then, the module is applied between different levels of encoding networks. Finally, a multi-path decoding network is formed by upsampling. The semantic segmentation network thus constructed can perform accurate segmentation recognition of $10$ different leather defects. After extensive training, a leather defect segmentation network with more satisfactory results is obtained.

The rest of this paper is arranged as follows. An introduction of related methods is presented in Section 2. Section 3 provides the network structure regarding encoding and decoding networks. The descriptions of the dataset used, the principal image processing explanation, the experimental configurations, and the experiment results are provided in Section 4. Finally, the conclusion is drawn in Section 5, accompanied by methodological recommendations for future work.

2.   Methods

The Kronecker product is a theorem proposed by mathematician Leopold Kronecker and is a special form of the tensor product ^[22]. For any matrices $A = [a_{ij}]\in\mathbb{R}^{m_{1}\times n_{1}}$ and $B = [b_{ij}]\in\mathbb{R}^{m_{2}\times n_{2}}$, their Kronecker product, i.e., the direct product or tensor product, denoted as $A \otimes B$, is defined by

and $M\in \mathbb{R}^{m \times n} \, (m = m_{1}m_{2}, \, n = n_{1}n_{2})$ is a block matrix whose $ij$-th block is $a_{ij}B$. More specifically, it can be expressed as follows:

The application of the Kronecker product in convolutional neural networks is initially the Kronecker convolution, which employs the Kronecker product to expand the standard convolutional kernel so that feature vectors neglected by atrous convolutions can be captured ^[23]. It is similar in idea to hole convolution ^[24]. The Kronecker convolution uses an additional transformation matrix as shown:

where the upper left corner $I$ is a $r_2\times r_2$ square matrix which has all the element values of $1$, and the lower right corner $O$ is a zero matrix. $r_1$ is the inter-dilating factor that controls the dilation rate of the convolutions. $r_2$ is denoted as the intra-sharing factor to control the size of subregions to capture feature vectors and share filter vectors. Then, the kernel $K$ of Kronecker convolution can be enlarged by computing the Kronecker product of $F$ and $K$, that is, the new kernel $K'$ can be formulated as

Moreover, the use of the Kronecker product can greatly reduce the complexity of the calculation and also save computer storage space. After performing the Kronecker product with the traditional convolution kernel $K$, a new convolution kernel $K'$ with an expansion factor is obtained. When convolving with the input, not only the perceptual field can be greatly expanded, but also the expansion size of the convolution kernel can be controlled.

In general, the structure for feature extraction in CNN is the convolutional layer. When extracting global semantic information, the size of the convolutional kernel and the number of parameters are often large and not easy to train. Alternatively, the fully connected layer can be used to extract global semantic information. The fully-connected layer is usually placed at the end of the network, connecting the feature map and the labels. Each pixel of a label represents the probability that the input image belongs to that location class. The fully connected layer also connects all pixels of the feature map, which means that all pixels of the feature map are used to determine the probability that the input image belongs to a certain category.

Therefore, the extraction of global semantic information with a fully connected layer requires the idea shown in Figure 1. Each feature plane of the feature map is expanded into a one-dimensional form, and then a fully connected layer is added to extract semantic information, and the output is recombined into the form of a three-dimensional matrix. Assuming that the size of the feature map is $\mathbb{R}^{m_{1}\times n_{1}}$, the feature map is first expanded in one dimension. Then the expanded feature map is compressed with the fully connected layer to obtain the size $\mathbb{R}^{m_{2}\times n_{2}}$. Finally, the result is recombined into the form of a two-dimensional matrix. In this process, the number of fully connected parameters is $W = m_{1}\; \times\; n_{1}\; \times\; m_{2}\; \times\; n_{2}$. If $m_{1} = n_{1} = 256$ and $m_{2} = n_{2} = 16$, the number of parameters of the fully connected layer is $10^{7}$ levels, which is several times of the classical semantic segmentation network.

Here, we replace the parameters of the fully connected layer with two small matrices of the same size to obtain the parameters of the fully connected layer of the same size according to the Kronecker product, as shown in Figure 2, where orange indicates the original feature map and cyan is the output. The matrices $A$ and $B$ represent the small parameter matrices that constitute the fully connected parameters, and the operation between $A$ and $B$ is also the Kronecker product. This new fully connected method is called the Kronecker product connected layer (KPCL). The large parameter matrix formed by matrices $A$ and $B$ is multiplied with each layer of the feature map, i.e., the KPCL layer is multiplied with each layer of the feature map, and the KPCL layer does not change the position relationship of channel directions in the feature map.

In Figure 3, some details of the two small matrices $A$ and $B$ are shown when replacing the parameter matrix of the fully connected layer, and the comparison of the number of parameters before and after is shown below

If $w_{0} = w_{1}\times w_{1}$ and $h_{0} = h_{1}\times h_{1}$, Eq (2.5) can be simplified to obtain

In the case, where $w_1$ and $h_1$ are large, applying the Kronecker product can greatly reduce the number of fully connected parameters of the parameterization.

3.   The network structure

3.1.   Encoding network

The encoding network uses the first four layers of the VGG16 network feature extraction module ^[25], and the size of the feature map is reduced by $1/2$ after each layer feature extraction. Each convolutional kernel size is $3 \times 3$. The first pooling layer of the encoding network is used for global feature extraction using the Kronecker product connectivity layer. The size of the first pooling layer is $H\; \times\; W\; \times\; C$, and after extending its feature space plane, two-dimensional features of size $(H\; \times\; W)\; \times\; C$ are obtained. The weight parameters constructed from the Kronecker product perform a "fully connected" operation on the first pooling layer, as shown in Figure 4.

The 8-fold compressed feature maps are obtained after two Kronecker product-connected layers. The feature maps extracted by the Kronecker product and the layer-by-layer convolution are combined and filtered with two convolutional kernels of sizes $3\times 3$ and $1\times1$ to obtain the output of the coded network, see Figure 5.

In addition, each convolutional layer contains a convolutional layer, a batch normalization layer, and a pooling layer, and the whole feature extraction structure is pruned to reduce the encoding network parameters and improve the network training speed.

3.2.   Decoding network

The decoding network uses three decoding paths, as shown in Figure 5. Of these, the first decoding path is a continuous 2-fold bilinear upsampling of the result of the encoding network, which yields a segmentation map that is $1/2$ the size of the input. In addition, each small magnification of upsampling allows for better segmentation of small-sized defects. The other two decoding paths both rely on KPCL layers composed of the Kronecker product and target the feature maps before the second and third pooling layers in the encoding network. Both decoding paths use two KPCL layers, where the role of the first KPCL layer is to extract global semantic information from the feature map while performing 2-fold downsampling of the feature map, and the second KPCL layer and the subsequent $3\times3$ convolutional layers are responsible for optimizing the results of the first KPCL layer.

Then, the segmentation information extracted from the two decoding paths is directly amplified by bilinear upsampling and combined with the main decoding path to form three decoding paths. Finally, the results obtained from the three decoding paths are fused in the channel direction to obtain the output of KMDNet.

The three decoding paths scale the feature map to $1/2$ the input size to reduce the parameters. The result of the network is still further scaled to the input size by additional bilinear interpolation. Figures 6 and 7 show the defect segmentation information after the Kronecker product decoding path and the main decoding path, of size $128\times 128\times 32$, which are obtained by stitching the defect segmentation information in the channel direction. Figure 6 shows the recognition effect of the main decoding path on the defective parts. It can be seen that most of the decoding results have a distinct black dot-like region in the lower-left corner, which is the same as the defective part, but the position in the original defective image is a normal texture region. Figure 7 shows the results of the Kronecker product decoding path. As can be seen from the resulting image on the right, the result contains only the semantic information of the defect, and the lower-left corner does not contain additional defect information. Therefore, after combining the main decoding path and the Kronecker product decoding path, the wrong segmentation defect can be avoided in the lower-left corner of the segmentation result.

4.   Experiment

This experiment uses Tensorflow and Keras as the deep learning development frameworks. The training platform is given below:

● CPU uses Intel (R) Weon (R) W-2102 CPU @2.9GHz (4-core).

● GPU uses Nvidia RTX TITAN, and the Video Memory is 24G.

● The system environment is Windows10.

and the testing platform is as follows:

● CPU uses AMD Ryzen5 1400 Quad-Core Processor @3.20GHz (4-core).

● The system environment is Windows10.

4.1.   Dataset

The dataset for this experiment was collected by a related leather manufacturing plant in Jinjiang city using a high-precision industrial camera. The types of defects in the leather dataset include a hole, psoriasis, slip, brand, fester, open wound, suture, wart, scratch, and healing wound, as shown in Figure 8. The number of the wet leather datasets is $1944$, and the ratio of defective samples to non-defective samples is $1603:524$.

To speed up the training time and reduce the testing time, the size of the original image is compressed to $256\; {\mathrm{dp}}\; \times\; 256\; {\mathrm{dp}}$ and the defective regions of the image are labeled by the labeling procedure. Since the size of the network output is $1/2$ of the input, the image labels are also downsampled and the raw data labels are reduced to $128\; {\mathrm{dp}}\; \times\; 128\; {\mathrm{dp}}$ to meet the needs of the model, see Figure 9.

4.2.   Image preprocessing

Due to the large variety of defects in this training, some defects with inconspicuous features can greatly affect the effectiveness of the training. Therefore, appropriate image preprocessing is required to highlight the inconspicuous features of these defects while ensuring the details of the image texture. The preprocessing includes the following steps.

$1)$ Gamma transform

The gamma transform improves the detail of leather textures and imperfections by dividing the pixel value of the input image by $255$, normalizing it to the interval $[0, 1]$, and then calculating its gamma squared value, as follows:

where $I_{(r, c)}$ is the normalized pixel value. When $\gamma = 1$, the original pixel value is not affected; when $0 < \gamma < 1$, the more the pixel value, the higher the contrast of the picture; and $\gamma > 1$, the contrast of the picture is reduced.

$2)$ Histogram equalization

Histogram equalization can improve the contrast of an image. In this experiment, the grayscale histogram of the image needs to be counted after the gamma transformation of the image to improve the contrast.

4.3.   Analysis of results

4.3.1.   Evaluation criteria

In deep learning, the evaluation criteria for binary classification of machine learning models are based on the confusion matrix, see Table 1. In artificial intelligence, the confusion matrix is mainly used to compare the classification results with the actual measurements, and the accuracy of the classification results can be shown inside the confusion matrix. In Table 1, $TP$ denotes the value where both the true category and the model prediction are $1$; $TN$ denotes the value where both the true category and the model prediction are $0$; $FP$ denotes the value where the true category is $0$ and the model prediction is $1$, and $FN$ denotes the value where the true category is $1$ and the model prediction is $0$.

The confusion matrix yields a number of model evaluation metrics. For example, Eq (4.2) shows the mean intersection over union ($mIoU$), it is a measure of the similarity between the predicted segmentation region and the true segmentation region in an image,

where $k$ is the number of classification categories without background, and $k+1$ includes background categories. When $i = 0$, positive examples in the confusion matrix are the background, and negative examples are other target categories. When $i$ is greater than $0$, positive examples represent class $i$ targets. In addition, the metric accuracy ($ACC$) measuring the proportion of correct classifications by the model is shown in Eq (4.3),

Equation (4.4) displays a measure of the recall of the model in covering positive samples in the prediction,

Eq (4.5) gives the exponential $F1$ score,

which measures the accuracy of the dichotomous classification model.

The performance of the model was examined by using $mIoU$, $ACC$, $Recall$, and $F1$ score as comprehensive evaluation criteria, where the larger the values of $mIoU$ and $ACC$, the better the model.

4.3.2.   Results analysis of the KPCL layer

To verify the role of the KPCL layer, the data results for the model without and with the KPCL layer are given in Table 2. It can be seen that the model with the KPCL layer removed decreases in all metrics, especially in $mIoU$ and $Recall$, which are severely attenuated. The metric $Recall$ represents whether the segmentation of leather defects in the image is complete, so the Kronecker product multi-decoding path improves the ability to segment leather defects completely without an increase in detection speed.

The KMDNet improved in all four metrics, especially the $mIoU$ and $Recall$ metrics, which improved by 7.45 and $19.19\%$, respectively. As can be seen in Figure 10, the KMDNet can segment the true defect areas on unimportant scratches, while the network without the KPCL layer cannot segment the same defects.

4.3.3.   Comparison between different models

KMDNet is compared with models such as SegNet, U-Net, PSPNet, and DFANet, where U-Net ^[26] has been applied to leather defect detection and segmentation in recent years, SegNet ^[27] and PSPNet ^[28] are classical semantic segmentation models that have emerged recently, and DFANet ^[29] is a lightweight semantic segmentation network for road scenes that emerged in 2019. The segmentation data of each model under all defect types are shown in Table 3.

In order to use the network structure as the only variable, the same preprocessing process was performed on the data before the network training, and the sample weights and number of batches for the comparison model were kept consistent with KMDNet during training. Compared to DFANet, KMDNet has a $9.34\%$ more $mIoU$, $5.53\%$ more $ACC$, $9.64\%$ more $Recall$, and $13.58\%$ more $F1$ score, but the detection time is only half of the DFANet. Compared with U-Net, the $mIoU$ of KMDNet increased by $0.36\%$, but the $ACC$ decreased by $0.36\%$, and the two models were very close. However, for $Recall$ and $F1$ score, KMDNet has more advantages and is significantly lower than U-Net in terms of detection speed and model parameters. This demonstrates that KMDNet has a shorter defect segmentation time and better recall compared to U-Net, which has been applied in the field of leather defect segmentation. As shown in Figure 11, KMDNet has a higher overlap between the segmented defective part and the real defective part compared to U-Net. Moreover, KMDNet has many advantages over SegNet and PSPNet in all six metrics. Based on these metrics, KMDNet is the optimal solution for segmenting leather defects compared with SegNet, U-Net, DFAnet, and other networks.

Figure 11 shows the different segmentation results of KMDNet and U-Net for the scratch defects; U-Net cannot completely segment the scratch defects, but KMDNet does a better job of segmenting the scratch defects. The difference between the two can also be reflected in the normal leather texture area.

Figure 12 illustrates the different segmentation results of KMDNet and U-Net in the normal region. It can be seen that KMDNet accurately segmented the defect-free region, while U-Net showed segmentation errors.

Table 4 presents the segmentation results of KMDNet for different types of defects. As can be seen from it, defects such as the hole, psoriasis, brand, fester, and wart have the best segmentation results, with $F1$ scores above $80\%$. Defects such as open wounds, sutures, and healing wounds have the second-highest segmentation effect with an $F1$ score of over $73\%$, while slip and scratch have a poor segmentation effect with an $F1$ score of about $60\%$. This result indicates that there is a strong correlation between the feature saliency of defects and the segmentation effect. For defects with distinct features, the semantic segmentation network tends to learn better feature maps and thus achieve higher segmentation results. However, for defects with unclear features, it is difficult for the network to learn better feature maps.

5.   Conclusions

In this paper, a multi-decoding segmentation network using the Kronecker product, KMDNet, is proposed in order to identify defects in different shapes of wet blue leathers. The defect detection of wet blue leathers is more challenging than that of general rawhide leathers. This paper is an analytical study using a multi-decoding segmentation network with the Kronecker product. The following list is a summary of the major contributions of this paper.

$1)$ Based on ensuring the network segmentation effect, our proposed KMDNet can reduce the number of parameters per convolution and the network output size; it can also reduce the number of network parameters and improve the detection speed.

$2)$ KMDNet uses the Kronecker product to construct a new connectivity layer instead of a fully connected layer. The new connectivity layer will decode the feature map of the first layer of the network, extract global semantic information, and downsample it to fuse with the feature map of the regular feature extraction network in the channel direction.

$3)$ The decoding network uses the Kronecker product connectivity layer to construct multiple decoding paths to decode the small-sized feature map to $1/2$ of the original image size and then obtains the final segmentation result by bilinear upsampling outside the network.

$4)$ Trained with/through the collected wet blue leather defect data, our proposed KMDNet reveals a much better detection effect than the original segmentation network, and the $mIoU$ is improved by $7.45\%$ after KMDNet is added to the KPCL layer.

$5)$ KMDNet has advantages in segmentation effect compared with other networks such as U-Net, and at the same time, the number of network parameters is less and the detection speed is faster, which enables further practical application in the industry.

The model proposed in this paper is for the binary segmentation of wet blue leather defects. The coded network of KMDNet uses the KPCL layer for global semantic information extraction only for the first-level feature extraction unit. Our future work is to achieve the distinction of different types of defects while segmenting them, which requires enriching the network structure during the coding phase while keeping the network lightweight. The goal is to enable inspectors to effectively diagnose different defects faster. For example, convolutional layers are added to the encoding network to extract features that can represent the differences between different types of defects. For poorly segmented defects, the segmentation effect can be improved by increasing the amount of data for such defects. In addition, techniques such as near-infrared imaging can also be tried in the image preprocessing stage, allowing the segmentation network to extract richer defect information from the wet blue leather images. In the meantime, comparing KMDNet with the newly improved segmentation networks is also part of our next work.

Acknowledgments

This research of F.R. is partially supported by Qing Lan Project of Jiangsu Province, NSFC (11601226), Project of Philosophy and Social Science Research in Colleges and Universities in Jiangsu Province (2021SJB0081), Postgraduate Research & Practice Innovation Program of Jiangsu Province (SJCX22_0406) and Postgraduate Education Reform Project of Nanjing Tech University (YJG2212). This research of J. H. is partially supported by Fujian Provincial Science and Technology Project Fund (2021T3060, 2021T3032) and Program of Quanzhou Science and Technology (2021C063L).

Conflict of interest

The authors declare there is no conflict of interest.