Feature extraction of face image based on LBP and 2-D Gabor wavelet transform

1.   Introduction

As a specific application in biometric identification technologies, there have emerged a lot of algorithms for face recognition technology after nearly half a century of research. Chan and Bledsoe presented a research paper on automatic face recognition for the first time ^[1]. They hold that face recognition could be realized based on geometric structural features. In the 1990s, the research on automatic face recognition ushered in a new leaping development. Among numerous methods for face recognition, the face recognition based on facial features was undoubtedly the most representative ^[2]. The feature extraction from face image data was carried out by using discrete K-L transformation, also known as principal component analysis (PCA).

Since the beginning of this century, more in-depth studies on face recognition have been carried out. In order to improve the performance of face recognition algorithms, many scholars have done a fruitful work, which is summed up in the following areas:

a. Original face recognition algorithms have been improved. For example, in order to reduce the calculated amount of feature analysis required by PCA, Professor Yang et al. in Nanjing University of Science and Technology proposed 2-D PCA method, followed by (2D) 2PCA, MatPCA, G2D-PCA and UPCA later on; There have also been various improved versions for LDA method, such as 2D-LDA, NLDA, RKDA and NKDA ^[3].

b. New description of face image data has been searched. Image texture description has many methods such as local binary pattern(LBP)^[4], gray level co-occurrence matrix, histogram of oriented gradients ^[5]; New descriptions can also be produced when some integral transformation is conducted on images. These transformations include Fourier transformation, Gabor wavelet transform, Harr wavelet transform, discrete cosine transform.

c. More innovative face recognition methods have been proposed; Such innovative face recognition methods include: Face recognition based on support vector machine ^[6], face detection and recognition based on Adaboost algorithm ^[7,8], face recognition based on manifold learning ^[9], face recognition based on compressed sensing ^[10].

d. Face recognition algorithms have been integrated and optimized ^[11]. Currently, it is very difficult to achieve better recognition performance only relying on a certain kind of approach for face recognition. The optimization and combination of different descriptions, different feature extractions, and different classifiers on face images will achieve face recognition algorithms with higher recognition rate and better robustness.

Under the influence of light, gesture and facial expression, the performance of face recognition algorithms are generally affected to a certain extent. For how to effectively reduce the impact of various factors on the performance of face recognition algorithms, combined with LBP and 2-D Gabor wavelet transform, this paper described the features of face image, and carried on tests on the performance of face recognition described by LBP and 2-D Gabor wavelet in different scales and directions. Through synthesizing and analyzing the experimental results, this paper proposed a method of extracting local and global features from face image.

2.   Local binary pattern representation of face image

LBP was first proposed by Ojala et al.. It is a nonparametric operator which describes local image structure. The simplest LBP made a binary encoding on the pixel gray difference between center pixel and the pixel within its 8-neighborhood, then converted it into a decimal number which was used for the expression of texture feature of center pixel. Such description was invariable to the overall change of gray level in images. In order to eliminate the pattern change caused by image rotation, Ojala et al. further presented a new definition for rotation invariant local binary pattern. In addition to this, Marko et al. put forward center-symmetric local binary pattern ^[12], Xu et al. proposed dual local binary pattern, Tan et al. proposed local triple pattern. In 2004, Ahonen et al. first described face images by using LBP description operator and used the techniques such as histogram statistic and chi-square statistical measurement for face recognition. Experiments showed that this method could effectively eliminate the impact of factors such as illumination on the performance of face recognition ^[13,14].

2.1.   Local binary pattern

Image texture feature generally means the characteristics of each pixel reflecting within its neighborhood, so the pixel neighborhood must first be defined generally when describing image texture features. The neighborhood used by LBP to describe an image is an annular area which can be written as $(P, R)$, wherein, R is the radius of annular neighborhood, P is the pixels which are uniformly selected in annular neighborhood ^[15].

The feature information of any pixel in an image is determined by a combination of pixel itself and its neighborhood. The feature information of each pixel can be measured with joint probability distribution $T({g_c})$. $T({g_c})$ is defined as shown in formula (1):

Where, g_c is the gray value of central pixel, $\left\{g_{i} | i = 0, \ldots, P-1\right\}$ is the gray value of pixel on annular neighborhood. A new feature measure can be achieved when subtracting the gray value of center pixel from the gray value of each pixel within the neighborhood. Suppose any $\left\{\left(g_{i}-g_{c}\right) | i = 0, 1, \ldots, P-1\right\}$ is independent of g_c, it is defined as follows:

Where, $\bar t({g_c})$ represents the grey information of image, $\bar t({g_0} - {g_c}, {g_1} - {g_c}, \cdots, {g_{P - }}_1 - {g_c})$ represents the texture information of image. Under the influence of illumination variation, $\bar t({g_c})$ generally changes greatly, while $\left\{ {{g_i} - {g_c}\left| {i = 0, 1, {\kern 1pt} \cdots } \right.} \right., \left. {P - 1} \right\}$ is also affected by illumination variation to some extent. If $\bar t({g_c})$ is abandoned, the measure of gray difference $\left\{ {{g_i} - {g_c}\left| {i = 0, 1, {\kern 1pt} \cdots } \right.} \right., \left. {P - 1} \right\}$ between the pixel within annular neighborhood and center pixel is transformed into a new measure $\left\{ {s({g_i} - {g_c})\left| {i = 0, 1, {\kern 1pt} \cdots } \right.} \right., \left. {P - 1} \right\}$ which is not sensitive to light, the texture feature information which is not sensitive to illumination variation at any one pixel in the image can be further defined as formula (3):

The values of $\left\{s\left(g_{i}-g_{c}\right) | i = 0, 1, \ldots, P-1\right\}$ generally do not change in the case of uniform illumination variation. LBP pattern is defined as follows:

2.2.   Local binary pattern of face image

The facial texture feature information can be effectively extracted by using local binary pattern operator to describe face image. Figure 1 shows the face description images by means of the operators such as traditional local binary pattern ($LBP8, 1$(b)), rotation invariant local binary pattern ($LBP_{8, 1}^{ri}$(c)), consistent local binary pattern ($LBP_{24, 3}^{riu2}$(d)), center-symmetric local binary pattern ($CS\_LBP16, 2$(e)) and dual local binary pattern ($DLBP8, 1$(f1, f2)).

3.   2-D Gabor transform of face image

In 1946, Dennis Gabor discovered the uncertainty principle of one-dimensional signal and provided the original form of Gabor function ^[16]. He pointed out that any complex sinusoidal signal modulated by Gaussian function could reach the lower limit of combined uncertainty relations in time domain and frequency domain. In 1980, Daugman proposed a two-dimensional Gabor (2-D Gabor) filter function and used it for image characterization ^[17]. In recent years, researchers have used 2-D Gabor filters for face image description and also for face recognition resulting in a lot of new research achievements.

3.1.   2-D Gabor filter

The new image description obtained from image filtering by using 2-D Gabor filters in different scales and directions can effectively reduce the impact of the changes of illumination, facial expression and gesture on face recognition accuracy. 2-D Gabor filter is defined as follows:

Where, $\sigma$ reflects the size of Gaussian window, ${\bf{x}} = (x, y)$ is the coordinate location of a given image pixel, ${\bf{k}}_{u, v} = k_{v}\left(\cos \left(\phi_{u}\right), \sin \left(\phi_{u}\right)\right)$ is the center frequency of filter. Here, $k_{v} = k_{\max } / f^{v}$, $k_{\max }$ is the maximum frequency, v determines the wavelength of 2-D Gabor wavelet; $\phi_{u} = \pi u / K$ is the direction of 2-D Gabor wavelet function. Figure 2a, b shows the real part and imaginary part of a set of 2-D Gabor filter functions. Figure 2c shows the amplitude values of a set of 2-D Gabor filter functions, wherein the preferences are: $k_{\max } = \pi / 2, \quad f = \sqrt{2}, \quad \sigma = \sqrt{2} \pi, \quad u = 0, 1, \ldots, 7, \quad v = 0, 1, \ldots, 4$.

3.2.   2-D Gabor wavelet face image

Different image characterizations can be obtained from image filtering by using 2-D Gabor filters in different scales and directions. Each description can be calculated by formula (6):

Where, * represents convolution operation.

Figure 3 shows the filter output and amplitude output at real and imaginary parts of 2-D Gabor filter for face image. It is thus clear that the new images obtained from image filtering by using 2-D Gabor filter can not only carry out effective global description but also effectively extract the detail features of images. Relative to single original grayscale image, the features at different levels of image can be extracted through 2-D Gabor filter. These levels of features can be separated for face classification ^[15].

4.   LBP and Gabor wavelet transform based face image local and global feature extraction

4.1.   Face image based on LBP

Suppose that face image can be represented by I, the steps for face recognition by using simplest LBP pattern $LB{P_{8, 1}}$ are as follows:

a. Use $LB{P_{8, 1}}$ to describe face image I for a texture description $I\_LB{P_{8, 1}}$ on face image. Figure 4b shows LBP description $I\_LB{P_{8, 1}}$ in Figure 3a.

b. Divide $I\_LB{P_{8, 1}}$ into several pieces ($I\_LB{P_{8, 1}}$ in Figure 4b is divided into 3 × 3, nine pieces). Use histograms to count all pieces to obtain a histogram description $I\_LBP\_his{t_i}$ of each small piece, and connect the histograms to form a new histogram description of face image $I\_LBP\_Hist = [I\_LBP\_his{t_1}, I\_LBP\_his{t_2},$$..., I\_LBP\_his{t_n}]$. Figure 5 shows the histograms of each small piece and the intuitive connection of histograms of each small piece.

c. $I\_LBP\_Hist$ is a vector expression for original face image I. However, the dimension of $I\_LBP\_Hist$ is generally higher. For the partition mode as shown above. The dimension of $I\_LBP\_Hist$ reaches up to $256 \times 9 = 2304$. Such high-dimensional data are not conducive to classification and recognition, especially in small samples. Direct at solving the problem, $I\_LBP\_Hist$ is generally treated with dimension reduction, then classified and identified. The commonly used methods of dimension reduction include PCA, independent component analysis (ICA) and so on. Figure 6b shows the performance for the face database of ORL was treated with dimension reduction by using PCA method, and the face recognition was conducted by using NN (Nearest Neighbor) of Euclidean distance metric; Figure 6a shows the performance for the face images in ORL face database were first treated with $LB{P_{8, 1}}$ description, then described with histograms, done with dimension reduction by PCA method, and the face recognition was conducted by using NN of Euclidean distance metric. Each person had four face images involving in the training, and the remaining six images were used for testing. Compare Figure 6a with Figure 6b, it shows that the method of LBP + PCA has achieved better recognition effect.

Similarly, we compare the number of recognition samples and the average time consumed by different methods, and the results are shown in Table 1.

Table 1 shows the first hit recognition rate value (rank1) of three different algorithms and the tenth hit recognition rate value (rank10) when the final recognition rate tends to be stable after many training, as well as the time-consuming. The experimental data of the table show that with the increase of training times, the recognition rates of the three algorithms are improved. The recognition rates of LBP algorithm are low, while PCA algorithm takes a long time because of the high dimension of feature vectors. In this paper, the algorithm first uses LBP to process the histogram of the encoding image and then uses PCA to reduce the dimension of the extracted features, which greatly improves the recognition rate while reducing the computational complexity and improving the recognition speed.

4.2.   Facial feature extraction based on LBP and Gabor

Gabor transform not only has an astonishing similarity to the human retinal cell receiving field model, but also obtains the minimum uncertainty of frequency and time, and has strong robustness to external disturbances such as illumination, posture, expression and so on. The traditional face feature extraction method based on Gabor transform firstly uses convoluted face images and Gabor multi-direction and multi-scale kernels, and then cascades and down sampling to obtain Gabor face features. However, this method can get a higher dimension, which takes a lot of time and has no rotation invariance. When the face image rotates in the plane, the recognition performance decreases. LBP is an operator for texture analysis. It has many advantages such as simple algorithm, not too complicated calculation process, strong discrimination ability and so on. It has been widely studied and used in recent years. LBP operator has strong gray scale invariance and rotation invariance characteristics, which can reduce the influence of rotation shift and illumination unevenness.

Therefore, this paper uses 2-D Gabor wavelet to extract the original image features, including multi-angle feature extraction and multi-scale feature extraction, in order to effectively reduce the interference of illumination and rotation on image extraction. The obtained image is encoded and histogram is constructed by LBP algorithm, so that the image details are adequately information. Then PCA method is used to reduce the image dimension. It's named GBLBP here.

From the feature selection method, dimension reduction method and classifier introduced above, the corresponding algorithm is obtained.

Step 1: According to (6), 40 subetaaphs of face image $I$ are obtained by Gabor wavelet transform, in which μ∈{0, 1, 2, 3, 4, 5, 6, 7} and V∈{0, 1, 2, 3, 4};

Step 2: All subetaaphs are divided into M blocks R0, R1, …, Rm − 1;

Step 3: For the local points (a, b) in each specific region Rk, according to (4), take the GLLBP feature with the scale ν0 and the direction µc as the center, and get the corresponding histogram $I\_LBP\_his{t_i}$;

Step 4: The final feature representation vector $I\_LBP\_Hist$ of I is obtained by integrating each feature;

Step 5: Dimension reduction according to PCA;

Step 6: Perform steps 1–5 for all training samples;

Step 7: For each test sample $I'$, get the eigenvector $I'\_LBP\_Hist$ of the test sample according to steps 1–4;

Step 8: The dimension of $I'\_LBP\_Hist$ is reduced according to the dimension reduction matrix of PCA;

Step 9: Get test sample label by NN classifier.

Since face description in each scale and direction contains some facial feature information, this paper carried on the face recognition performance test on each description separately, and investigated the face recognition performance of 2-D Gabor wavelet description in different scales and directions. This paper carried on test on ORL face database by using PCA + NN respectively. Figure 7a shows the output result of the input face image at real part of small-scale 2-D Gabor wavelet filter; Figure 7b shows the output of the input face images at real part of large-scale 2-D Gabor wavelet filter.

The experiment showed that the recognition performance was poor when the wavelet scale of 2-D Gabor wavelet filter was smaller; when the wavelet scale was larger, the recognition performance of new description became better. This paper carried on the test on the face description in both scale and direction by using PCA + NN respectively. The variation diagrams in which face recognition rate increased with the number of features are shown in Figure 8a, b.

As can be seen from the two pictures in Figure 8, large-scale 2-D Gabor wavelet filter could be better used for the classification and identification of face image. Actually, the description of face image by using small-scale 2-D Gabor wavelet filter reflected the high frequency in the image, corresponding to the local feature information, including contour information of each organ on human face, but also the noise information; while the description of face image by using large-scale 2-D Gabor wavelet filter reflected the low frequency in the image, corresponding to the global feature information.

Another experiment was conducted in the paper. The face descriptions in the same scale but different directions were set as a group, and the face descriptions of the same group in different directions were superimposed each other to form a face description of 2-D Gabor wavelet in a certain scale. Figure 9 shows a number of face description of 2-D Gabor wavelet in different scales about the same face, wherein, Figure 9a corresponded to the smallest wavelet scale, Figure 9e corresponded to the maximum wavelet scale. The classification and recognition performance test on face descriptions corresponding to scales in Figure 9a, e by using PCA + NN were conducted respectively. The test results are shown in Figure 10a, b; The classification and recognition performance test on face descriptions corresponding to scales in Figure 9a, e by using LBP + PCA + NN were conducted respectively. The test results are shown in Figure 10c, d.

As can be seen from the comparison of Figure 10a, c, for the face description of 2-D Gabor wavelet in small scale, the texture description was first carried on by using LBP, then it was described with blocks, histograms, PCA feature extraction, which could get better recognition performance compared to PCA feature extraction directly from the data via 2-D Gabor wavelet transform. This is due to the use of PCA for feature extraction, wherein the corresponding features were the low frequency information of the data; while the face description of 2-D Gabor wavelet in small scale corresponded to the contour information of face organ, i.e. high frequency; In the case of face description of 2-D Gabor wavelet in small scale, we hoped to take advantage of our face contour information, namely high-frequency to recognize faces. So the performance of feature extraction directly from the data via 2-D Gabor wavelet transform by using PCA as well as classification and identification was poor; First, the expression of texture pattern about the converted data through LBP could extract the contour information of face image data, and secondly a new vector expression of face image data was formed through the description of blocks and histograms. The vector expression still retained sufficient contour information, so it was reasonable to obtain higher recognition performance for feature extraction and recognition through PCA.

As can be seen from Figure 10b, d, for the face description of 2-D Gabor wavelet in large scale, the performance of face classification and recognition by direct use of "PCA + NN" and use of "LBP + PCA + NN " were all good. This is because that the face description of 2-D Gabor wavelet in large scale corresponded to the information of low frequency in the original face image. The corresponding extraction features using PCA were the information of low frequency about the data. However, the face description of large-scale 2-D Gabor wavelet did not lose too much low-frequency information in face images, so the direct use of "PCA + NN" could achieve better recognition performance for the face description of large-scale 2-D Gabor wavelet; while the use of "LBP + PCA + NN " could also achieve better recognition performance, for the contour information of large-scale 2-D Gabor wavelet description was extracted for face recognition.

Thus an method for extracting global and local features can be drawn in this paper: (1) Global feature extraction–large-scale 2-D Gabor wavelet description on face images, data dimension reduction with principal component analytical method, thereby obtaining effective global features; (2) Local feature extraction—different-scale 2-D Gabor wavelet description on face images, then use LBP for texture description, then image block, histogram description, PCA data dimension reduction, thereby obtaining effective local features of face images.

In order to better reflect the recognition rate comparison between LBP and 2-D Gabor expression recognition algorithm and 2D-DCT, CBP, PCA algorithm to be compared. Ten-fold cross validation method was used to test the accuracy of the algorithm. The data set was divided into ten parts, nine of which were used as training data and one as test data in turn. The corresponding accuracy will be obtained in each test. The average accuracy of the results of 10 times is used to estimate the accuracy of the algorithm. For the facial expression database, the recognition results are shown in Table 2.

Experimental results show that the GBLBP algorithm has a higher recognition rate in facial expression recognition than other commonly used algorithms.

4.2.   Performance comparison of algorithms

In order to have objective and scientific comparison results, hypothesis testing is used on the experimental results. Let the variable ${X_1}, {X_2}, {X_3}, {X_4}$ denote the classification error rate of algorithms proposed in this paper, Gabor, LBP and PCA, respectively. Since the value of ${X_1}, {X_2}, {X_3}, {X_4}$ is subject to many random factors, we assume that they submit to normal distribution, ${X_i}\sim N({\mu _i}, \sigma _i^2)$, $i = 1, 2, 3, 4$. Now, we compare the random variable mean of these algorithms, ${\mu _i}$ ($i = 1, 2, 3, 4$). The smaller the ${\mu _i}$ is, the lower the expected classification error rate is, and the higher the efficiency is. Because the sample variance is the unbiased estimation of the overall variance, the sample variance value is used as an estimate of the generality variance. In this experiment the significance level $\alpha$ sets as 0.01.

Table 3 shows the comparison process on ${\mu _i}$ and other parameters. We can see from Table 1 that, the expectations of classification error rate in this paper is far below than other algorithms.

In order to verify the feasibility of this algorithm, this section compares the algorithm with other common algorithms under the same hardware and software conditions, in which LBP, Gabor and GBLBP use PCA to reduce dimension. In the experiment, Gabor selected 5 scales, 8 directions and 35 × 35 window size. Face image block mode is 5 × 3 and dimension is reduced to 75. ORL face database contains 400 samples, a total of 40 groups, each group of 10 images, and each group is the same person under different postures, expressions of images, size 112*92. In ORL face database, the training set sample images are randomly extracted for experiment. When the sample data are 80 to 320, 30 experiments are carried out and their mean values are calculated. If the number of samples is 360, the recognition rate can be obtained by ranking method, and the CPU running time of different algorithms under different number of training samples can be calculated by ranking method. The experimental results are shown in Tables 4 and 5.

Compared with ORL face database, Table 4 shows that the proposed algorithm has higher recognition rate than Gabor and LBP only, and the average recognition rate of PCA is the lowest. Because the proposed algorithm is compatible with the advantages of two algorithms: 2-D Gabor wavelet can extract multi-directional and multi-scale spatial frequency features of local images; LBP algorithm can represent the details of adjacent image regions, and extract more comprehensive and effective feature information. Compared with 2-D Gabor transform and LBP method, the algorithm used in this paper is better. The PCA-based feature extraction method is to extract the overall features of face images from the overall contour of face images. It cannot effectively describe the local details of face images, and cannot represent the relationship between regions. In the process of dimension reduction, it is easy to ignore some effective feature information. The recognition rate is the lowest.

From Table 5, it can be seen that the algorithm in this paper has high computational complexity and takes longer time to recognize a single image than the other three methods. If the size of a face image is 112 × 92, 40 images of the same size will be generated after Gabor transformation, and then after LBP, the dimension will increase many times. Therefore, the recognition of a face image will be done. It takes longer than the other three feature extraction methods, but in terms of recognition rate, it is more effective than the other three feature extraction algorithms. For face recognition, correct recognition is the key, so the algorithm in this paper is more feasible.

5.   Conclusion

This paper described face images in different scales and directions by using 2-D Gabor wavelet, and extracted features from various descriptions respectively combined with PCA method. The recognition performance tests on face description in various scales and directions were separately conducted by using NN classifier. The experiment results showed that higher classification accuracy could be obtained for large-scale face image description was used for separately pattern classification; then 2-D Gabor wavelet descriptions of face images in various scales and directions were treated with LBP description, and classified with blocks, histograms, PCA data dimension reduction and NN. The experiment results showed that this algorithm could obtain a relatively good recognition performance of 2-D Gabor wavelet description in a variety of scales and describe. Finally, this paper proposed a method of extracting global and local features from face images based on LBP and Gabor wavelet transform through the analysis of experimental data.

Acknowledgments

This work was jointly supported by National Natural Science Fund (NO. 61876185) and the Fundamental Research Funds for Central Universities (NO. 2015QNB21).

Conflict of interest

We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.