An intelligent recognition framework of access control system with anti-spoofing function

1.   Introduction

With Covid-19 spreading across the world, epidemic prevention has become an essential part of public life and more and more people begin to be concerned about their health of themselves and turn to adapt to a new lifestyle. Covid-19 is a kind of respiratory infectious disease that spreads mainly through droplets and close contact. To protect people in public places from infection, one practical suggestion for people is to wear masks as possible, keep social distance, and so on. However, most public facilities cannot be directly compatible with these new normal actions under epidemic. Taking the entry scene for example, most public entries are equipped with ACS to effectively prevent idle personnel, recognize people and check their identities. In order to prevent entry procedures causing the spread of the epidemic, it is not suggested people to taking off masks or sojourn on the ACS. Therefore, it needs to enable ACS with high-level biometric recognition capability and passage efficiency. As one popular direction of computer vision research, biometric recognition is widely used in security systems due to its safety and convenience. Nowadays common biometric recognition methods include face recognition, fingerprint recognition, iris recognition, voice recognition, etc. Among these methods, face recognition is supposed to be the most convenient compared with others, especially its non-contact characteristics match the requirement of Covid-19 epidemic prevention. In most public entry conditions, a face recognition module is implemented based on a monocular camera. In practice, this design has to confront several challenges, such as mask-wearing reorganization and Face Anti-Spoofing Detection (FASD). The article will discuss the solutions we think of for the above two problems.

In terms of mask-wearing recognition, this paper focuses on the fusion between mask detection and ACS based on face recognition in order to remind users to wear masks correctly. Qin et al. ^[1] proposed a facemask-wearing condition identification method, which is divided into three conditions for face mask detection and includes a face with masks correctly, faces with masks incorrectly and faces without masks. This method achieves relatively satisfying results in experiments but lacks robustness under complex background. Ejaz et al. ^[2] proposed another face masks detection method based on Principal Component Analysis (PCA). This method meets the application requirement on the embedded devices, but the accuracy of face-mask detection is not good. In order to deal with the lack of datasets, Jiang et al. ^[3] designed one kind of mask detector which uses transfer learning to enhance networks robustness which increases the capability of image feature extraction in the Backone. Through using feature pyramid networks (FPN) ^[4] and Convolutional Block Attention Module (CBAM) ^[5], this method increases the accuracy of classification results and enables the frame compatible with embedded systems too.

Completing face recognition is divided into two tasks: Face identification task and face verification task. The face identification task compares one face with face datasets from which the most possible identity will be found. The face verification task is a binary classifier, it makes use of a similarity metric algorithm to get confidence whether this identity indeed matches the tested face. Thanks to the recent developments of Convolutional Neural Networks (CNNs), face recognition has achieved great progress, but there are many challenges, such as illumination, pixel resolution, and so on. There are two ideas for face recognition in the era of deep learning: Metric learning ^[6] and margin-based classification ^[9,10,11,12]. Metric learning is also called distance metric learning, which classifies input images by calculating the similarity between two input images. However, the model training relies on so large data that the efficiency of the fitting is very slow. Margin-based classification is margin limited to feature layer by modifying softmax formula indirectly. This method makes the network obtain a feature that is more discriminative than the feature obtained after training. But the model has the instrict of overfitting because of harsh constraints. Yi Sun et al. ^[7] proposed a DeepID model that achieves 99.15% face verification accuracy on the LFW dataset by designing deep CNNs and using both face identification and verification signals as supervision. The advantage of DeepID is its high accuracy, but its training process is also prone to be overfitted. Dung Nguyen et al. ^[8] proposed a framework called FaceNet that directly learns a mapping through transferring face images to a compact Euclidean distance which directly corresponds to face similarity. This method achieves an accuracy of 99.63% on the LFW and accuracy of 95.12% on YouTube FacesDB respectively. However, there are many facial features lost in the process, which makes it easy to focus on the local feature rather than the global feature. Besides, another algorithm called ArcFace Jiankang was proposed by Deng et al ^[9]. It has a geometric interpretation to obtain highly discriminative features for face recognition. The verification accuracy of this method is 99.83% on the LFW dataset and 98.02% on the YTF dataset. ArcFace 9has high performance, but its weight model is so large that it is much more difficult to apply in transfer learning. MV-softmax ^[10,11] integrates the feature margin and feature mining into one unified loss function, it implements the discriminative learning by adaptively coding the feature vector of false classification. The main disadvantage of MV-softmax is to stress hard samples or half-hard samples, this may cause a negative influence on the model coverage. CurricularFace ^[12] adopts one kind of adaptive strategy to train loss function using margin samples and hard samples mining, which has great value in practice.

Generally speaking, although face recognition technology has been widely studied, there still exists a balance among the network size, computation cost and generalization. In addition, new face recognition also needs to complete tasks when people wear masks as possible, this is also the focus in the research field.

Compared with mask-wearing recognition, FASD is a long-term and widely-discussed problem because nowadays the biometric information of the face is easier to capture and be utilized for spoofing. Usually, Face spoofing attacks method includes photo attacks ^[13,14], video attacks ^[15,16] driven by reinforcement learning model, and 3D mask attacks ^[17,18,19] within low cost, hyper-real 3D color masks. Compared with widely studied 2D face presentation attacks, 3D face spoofing attacks are more challenging because present 3D face recognition systems are difficult to balance the effectiveness and cost. A typical 3D anti-spoofing system based on holographic imaging has little availability for most public entry scenes due to its high expense. Consequently, conventional face recognition systems are vulnerable to the above proofing attacks, most widely-applied face anti-spoofing methods up to now are still Human-computer interaction (HCI) methods that verify the user through some behavior interaction, such as winking, head moving, and lip language. Therefore, it is necessary to endow ACS with the capability of verifying the authenticity of facial information, and reject illegal users who present an imitation or fake face of the enrolled user.

The core task of the FASD is to find the intrinsic difference between the sampled information and fake spoofing information. With the development of the reinforcement learning technique, it has not been difficult to produce dynamic face movement completely driven by data, which presents a challenge to the anti-spoofing method based on human-machine interaction. Up to now, the main attention is increasingly drawn on the research involved in color texture analyzing ^{[13,14,20,21]} and adaptive recognition based on deep learning ^[22,23], etc. Määttä et al. ^[19] utilized multi-scale LBP to analyze the texture of face images. Bouklenafe et al. ^[13,14] proposed a face anti-spoofing algorithm based on color texture. This algorithm considered the three-color space, RGB, YCbCr and HSV, which improves the model reliability in various conditions. After that, Bouklenafe et al. ^[13,14] started to focus on the luminance and chrominance channels and utilized the fusion mechanism of color and texture. Although FASD based on texture has drawn much attention, its performance is easy to be affected by illumination and image resolution. When the above factor decreases in quality, this method may be difficult to maintain self-stability. Another category of FASD is to monitor face movement. Usually, a genuine face has lots of movement characteristics, such as blinking, mouth opening and head-turning, while fake faces are difficult to formulate similar movements. Pan et al. ^[24] proposed a real-time liveness detection approach by recognizing spontaneous eyeblinks to prevent photos attack in face recognition. Zadeh et al. ^[25] also proposed a method utilizing eye and mouth movements to conduct liveness detection. Despite these ways tend to have high accuracy, it is still necessary to highly interact with users. On the other hand, the time cost of this detection is relatively long and not friendly to users. In recent years, there has appeared simulated dynamic faces driven by the reinforcement learning model ^[26], which presents challenges to FASD research. It is also found that the CNNs have great potential for learning faced with various spoofing ways and improved the robustness of anti-spoofing algorithms. Atoum et al. ^[22] proposed a two-stream CNNs-based approach for face anti-spoofing. Liu et al. ^[23] assumed liveness detection as a binary classification problem to be directly learned by Deep Neural Networks (DNNs). Despite CNNs ^[27,28] having potential for face liveness detection, it has to face the problem of overfitting and needs further studies.

Based on the above discussion, it is not difficult to find that the liveness detection research has competed with a variety of spoofing approaches all the time. In this paper, we focus on these issues and present an intelligent access control framework with two improvements:

1) A novel authentication system fusing the mask detection algorithm and face authentication algorithm with face anti-spoofing algorithm is proposed. The mask detection algorithm employs the mask detector called RetinaFaceMask, and uses MobileNetV3 as the feature extractor; the face authentication algorithm makes use of Tripletloss for training the neural network model;

2) In order to realize the anti-spoofing function, this paper makes use of a lightweight neural network and uniform Local Binary Patterns (LBP) method to extract the color texture of the image frame. The accuracy and training efficiency of the above model is tested on the CASIA-FASD dataset and Replay-Attack dataset.

This paper is organized as follows. Sections Ⅱ and Ⅲ discuss the design of the network framework in detail, the selection of benchmark datasets, and experiment setups respectively. Then the analysis and discussion of experimental results are presented in Section Ⅳ.

2.   Design of intelligent access control framework

The proposed framework of the intelligent access control framework is composed of three parts: Mask detection module, face detection and recognition module and face anti-spoofing module (as shown in Figure 1). During the operation process, the passenger is sampled using videos and flagged the mask-wearing status by the mask detection module. Then it makes use of a multi-task convolutional neural network (MTCNN) ^[29] to process captured video frames and align them within the user's face. Through MTCNN the detected face is centrally located in alignment pictures with a size of 160^*160 pixel). The alignment pictures are then filled into the face anti-spoofing module in which the liveness detection algorithm can judge its authenticity. If one face is judged as real, it will enter the face recognition module for identity matching. In the face recognition module, if the real face matches one in the database, this face and its identity will be flagged by this system. Otherwise, the face recognition will end and present an alarm message.

2.1.   Mask and face detection

This paper designs mask and face detection module, which includes mask detection unit and face detection unit. The mask detection unit is designed based on a kind of face-mask detector called RetinaFaceMask (as shown in Figure 2). In order to apply to embedded applications, the feature extractor is replaced with MobileNetV3 which is featured with its low parameter quantity and lightweight architecture. On the other hand, the high-level semantic information is extracted by FPN from the feature maps of different sizes while fusing this information into previous layers' feature maps by adding operation with a coefficient. In Figure 2, the head part of mask detection architecture represents classifiers, predictors, estimators which complete inference function and indicate the mask-wearing status. The test image input into networks model in the testing, which outputs two results are face confidence and mask confidence by removing proposal which is less than the set threshold. The remaining results are filtered using Non-Maximum Suppression (NMS). The prediction results of faces and masks are retained, whose weighted logistic loss is defined as (1). In the Eq (1), ${d}_{i}$ denoted a detection, ${y}_{i}\in \left\{-\mathrm{1, 1}\right\}$ indicate whether or not ${d}_{i}$ was successfully matched to an object, and let f denote the scoring function (RetinaFaceMask) that jointly scores all detections on an image $f\left({\left[{d}_{i}\right]}_{i = 1}^{n}\right) = {\left[{s}_{i}\right]}_{i = 1}^{n}$. For a variety potential detection results, here loss per detections is coupled to the other detections through the marching produces ${y}_{i}$. By this way, the prediction results of faces and masks are retained.

Due to the limited data set of face masks, it is difficult for the network to be trained through the training set to get good results so that we train a more robust network model through transfer learning. In RetinaFaceMask, only the parameters of the backbone and neck are transferred from Wider Face, a dataset consisting of 32,203 images and 393,703 annotated faces. RetinaFaceMask proposed a new contextual attention module as Heads in order to improve the prediction performance of face masks. The context attention module has three branches, each branch has a 3×3, two 3×3, and three 3×3 convolution kernels, and using different numbers of 3×3 convolution kernels to obtain different receptive fields from the same feature map. The original context module does not consider the face or mask, so it needs to feature an abstraction component put it into retraining. Here the abstraction component is designed to be one simplified convolutional block attention module (CBAM). Given an intermediate feature map from the original RetinaFaceMask architecture, the CBAM can sequentially infer a 1D channel attention map and a 2D spatial attention map. Although this structure is suitable to save the memory cost, the spatial attention module in the CBAM still can occupy a variety of computation costs ^[30,31]. Therefore, in this paper, we replace the attention module of CBAM with on Maxpool-Convolution-AveragePool component. This simplification is equivalent to cascading the shared spatial perception of the feature abstraction. After it, the RetinaFaceMask is prompted to pay attention to the face and mask features.

Another module of the mask and face detection module is the face detection unit, which uses the MTCNN model. This model is built as a three-layer neural network, which is P-Net, R-Net, O-Net. In the operation, the obtained images are processed by these three networks to detect faces and face landmarks. When input images come, they will be resized continuously to produce the image pyramid. The pyramid is filled into P-Net to obtain lots of candidates that are filtered by R-Net more precisely afterward. Finally, the filtered data is filled into O-Net to calculate the landmarks' position of potential faces. This model realizes face detection and alignment functions.

2.2.   Face anti-spoofing

Boulkenafet et al. ^[14] propose that face anti-spoofing could be distinguished by analyzing color texture. The color is analyzed by utilizing uniform LBP in the luminance and chrominance channels. Uniform LBP extracts the histogram from a single image band and concatenates these histograms into the final descriptor which be calculated by Support Vector Machine (SVM) to distinguish real face and fake face.

LBP is a feature descriptor of local texture for gray-scale images proposed by Ojala et al. ^[32]. The traditional extraction approach of the LBP feature descriptor makes the original image be divided into blocks of n^*n and each block is divided into cells of a^*a. Then, the feature of each cell is extracted by LBP. Considering the center pixel value as a threshold to compare with 8 pixels from the yield of threshold around. The LBP descriptor is defined as Eq (2):

(${x}_{c}$, ${y}_{c}$) is the center point in the 3^*3 area, which pixel value is ${i}_{c}$. ${i}_{p}$ represents the value of other pixels in the area. $p$ is the number of pixel points. $s\left(x\right)$ is a symbolic function which is defined as Eq (3):

$s\left(x\right) = 1$, if $x\ge 0$, otherwise $s\left(x\right) = 0$. The values of each pixel point be calculated by exploiting LBP descriptor. The result is connected to a binary value and converted to a decimal value which is the value of the LBP descriptor. Generating situations are 256 by this approach, but lots of situations are very complex so it is not good for the extraction of facial features. Thereby, Ojala, et al. ^[33] proposed Uniform LBP to optimize the traditional LBP descriptor for implementing dimensionality reduction through the transition approach. The uniform LBP descriptor is defined as Eq (4):

$(x , y)$ is the center point of the neighborhood, ${r}_{n}\left(n = 0, \cdots , P-1\right)$ is the pixel value of surrounding neighborhood, ${r}_{c}$ is the pixel value of center point, where $R$ is the radius of the surrounding neighborhood, $P$ is the number of points in the surrounding neighborhood, where ${U}^{\left(i\right)}$ is defined as Eq (5):

where, $\delta \left(x\right)$ is the symbolic function which is defined as Eq (6):

$\delta \left(x\right) = 1$, if $x\ge 0$, otherwise $\delta \left(x\right) = 0$.

Uniform LBP extracts the histogram from a single image band, and then concatenates these histograms into the final descriptor which be calculated by SVM ^[34] to distinguish real face and fake face. The whole process is shown as Figure 3. The Figure 3 contains two options to complete liveness detection task. The option 1 is face anti-spoofing based on color texture analysis. It extracts the faces in the dataset and crops them to the unified format of 224 ^* 224 pixels. These unified face frames are converted into the color spaces of HSV and YCbCr. Then the LBP descriptor extracts the feature histogram from color spaces, and connects them in series. In the end, serialized feature histogram is input in SVM to classify as real or fake face.

Option 2 in Figure 3 completes face anti-spoofing based on a lightweight neural network. The advantages of the lightweight neural network ^{[35,36,37,38,39]} are high accuracy, smaller parameter, detection faster and can be well applied to intelligent access control systems. Face liveness detection is defined as a binary classification question. Using the face data set of size 224×224 to train a lightweight neural network, and change the linear activation function of the last layer of the network structure to the Tanh activation function. Because the weights of multi-type image classification and living detection two classifications are different, the two-classification model suitable for living detection is used for pre-training. In terms of hyperparameter selection, Epoch is 150, Batch Size is 64, the cross-entropy loss function is used, the optimizer selects random gradient descent, the weight attenuation is 4×10^-5, and the Momentum is 0.9. According to the above conditions, the lightweight neural network is trained.

2.3.   Identity verification through face recognition

The function of face detection and verification is a hot spot in the field of security. In this paper, we propose a neural network called Face-net. This neural network encodes face images into digital vectors (or called face-net vectors). In order to calculate the confidence between the face image and preset identity, this paper uses Euclidean distance to compare two face-net vectors and determine whether these two vectors belong to the same person. As shown in Figure 4(b) represents distances between images. The distance between the same person's images is basically below one threshold which is demarcated a visa experiment.

As shown in the part (a) of Figure 4 In this paper, Face-net model is designed to realize the face verification and recognition, architecture of Face-net model is shown in Figure 4(a), batch refers to the input face image sample which is found and then is cropped to a fixed size of 160×160 pixel. Deep architecture refers to a deep learning architecture, the deep architecture of Face-net uses the GoogleNet network, which combines the traditional convolutional neural network with Inception structure. This configuration can get the balance between the size and over-fitting characteristics of the deep neural network. In practice, the main disadvantage of this method is the over-high dimensions of output features. So, optimization is conducted by replacing the last layer of the softmax classifier with one layer of L2 feature normalization. Through the normalization of the L2 layer, the revised module will output a 128-dimensional feature vector which is then optimized with triple loss. The triplet loss is one kind of loss function in Figure 4(c). The input of the triplet loss function is three groups of images. These three image groups are called Achor(A), Negative(N) and Positive(P) respectively. The A and P are captured from the same person with different imaging angles. The N is captured by other people. The similarity between image groups is quality by Euclidean distance. When training face-net, the Euclidean distance between A and P will gradually decrease, and the distance between A and N will gradually increase. Through this mechanism, the face-net learns the separability between features, and builds connections between every single image from a specific person. This is the basis for identity verification. The definition of triplet loss is shown as Eq (7):

In the Eq (6), ${x}_{i}^{A}$ (Anchor) represents the image of A, ${x}_{i}^{P}$ (Positive) represents the different image of A, ${x}_{i}^{N}$ (Negative) represents the image of B. $\alpha$ represents constant, and T represents the set of triplets in the training set. The loss function minimization $L$ is defined as Eq (8).

In the Eq (7), $f\left(x\right)\in {\mathbb{R}}^{d}$ represents a face image $x$ that is embedded in $d$-dimensional European space. Among them, the second norm on the left represents the intraclass distance, the second norm on the right represents the distance between classes, and $\alpha$ is a constant. The "$+$" at bottom right represents that when the value in the brackets is greater than zero, the loss is taken, and when it is less than zero, the loss is zero. $M$ is the base. The optimization process is to use the gradient descent method to make the loss function continuously decrease which means the distance within the class continuously decreasing, and the distance between classes continuously increased.

To draw a conclusion, aiming at the possibility of infection and negligence in human supervision in public places, at the meantime, in order to meet the needs of embedded device platform, this paper proposes an intelligent access control framework composed of the mask detection algorithm, MTCNN algorithm, face anti-spoofing based on color texture analysis and Face-net identity verification. This framework can meet the requirements of embedded device applications and effectively solve the security problem of the public place access control system under the background of the spreading coronavirus.

3.   Experimental setup and results analysis

In this section, we present the experimental evaluation of the proposed scheme for the intelligent access control system. The two data sets used in our experimental evaluation are introduced in detail. Experiments are performed on different anti-spoofing datasets. Details are given in the subsequent sections.

3.1.   Benchmark datasets selection

In order to evaluate proposed system objectively, we have chosen widely accepted CASIA-FASD ^[40] and Replay Attack ^[41] Datasets for experiments, both of which have well-described evaluation protocols. The CASIA-FASD datasets consist of 600 videos including 50 objects, each of the objects has 12 videos including 3 videos of legal access and 9 videos of illegal access. There are 3 different types of videos in the datasets which is low, normal, high, respectively. Illegal access videos have 3 different types of attack methods including warped photo attacks, cut photo attacks, and video attacks. The CASIA-FASD datasets are shown as Figure 5. The Reply-Attack datasets include 1200 videos which have 50 objects. Each of the objects has 24 videos which include 4 videos of legal access and 20 videos of illegal access, each of the videos has about 10 seconds in length. All videos are classified to 3 different backgrounds and 2 different illuminations. Illegal access videos have 3 different types of attack methods including print attack, digital images attack, videos attack. The datasets have 3 parts of datasets including train datasets, development datasets, text datasets.

3.2.   Experimental setup

In order to obtain a face recognition security system with a low calculation amount and a high recognition rate, the live detection is more user-friendly. The experiment verifies the performance of the algorithm by comparing the size of the model parameters, the recognition accuracy, the time required for the system to process a frame of pictures through the entire process, and the CPU occupancy.

This section focuses on the experiments to verify the anti-spoofing performance of the algorithm, using the CASIA-FASD and Replay-Attack datasets to conduct experiments. The datasets are shown in Table 1.

a) The videos contained in the CASIA-FASD and Replay-Attack datasets are read and saved by OpenCV for 30 frames. After each frame is obtained, the image will be aligned and cropped by MTCNN and normalized to a 224×224 RGB picture.

b) This chapter uses datasets for training and testing separately. In the quantitative analysis, the experiment uses accuracy rate, EER, HTER, and receiver operating characteristic (ROC) curve as the evaluation indicators for evaluating the algorithm. When quantitatively analyzing the real-time performance of the face security system, we verify the performance of the algorithm through the time required for the system to process a frame of pictures and CPU occupancy rate.

c) All experiments have been conducted on a computer with PyCharm® installed in Windows 10. The processor model is Intel Core i5-8250U@ 1.60 GHz, and the memory is 8 GB RAM. The algorithm in this chapter is based on PyTorch 1.5 and TensorFlow 1.12 deep-learning framework, using Python 3.7 language implementation.

3.3.   Results analysis

We did experiments on CASIA and Replay-Attack datasets to compare with the EER, HTER, ROC curves of different algorithms, the time required for the system to process a frame of pictures, and the experimental results of the CPU occupancy rate. The advantages and disadvantages of the algorithm are analyzed.

The CASIA dataset has rich attack methods and complex datasets. The Replay-Attack dataset has a total of 25,200 pictures, the attack method is video attack, the control variable is light, and the data set attack method is relatively simple. From Figures 6 and 7, we can see that under the Replay-Attack dataset with a single attack method, the algorithm can perform better classification, while under the complex CASIA data set, the LBP algorithm can still maintain good classification performance.

The lightweight neural network is limited in the size of the dataset. On the one hand, it is subject to fewer model parameters. Even if it is pre-trained under the live detection model, the network convergence process is still slow; on the other hand, under the multi-image classification It can achieve good accuracy, but live detection pays more attention to the image texture, spectral information, motion information, depth information and other nuances between the real face and the fraudulent face. This is a lightweight neural network for live detection under higher requirements.

It can be seen from Figures 6 and 7, and Table 2 that the size of the model parameters also affects the accuracy of live detection to a certain extent. Figures 8 and 9 verify that the LBP algorithm takes the least time when using SVM for facial feature classification, and the CPU usage is low. Because SVM has a better classification effect for small sample data and has a faster calculation speed.

The overall results of the experiments show that the system fused with the LBP algorithm and the FaceNet algorithm can effectively identify the attacks of forged faces on the face recognition security system while the real-time detection efficiency is high, and the lightweight neural network exists in the real-time live detection feasibility.

4.   Conclusions

In this paper, we have proposed a novel security classification framework based on face recognition which will possibly contribute to ACS in public condition. This framework is composed of a mask detection algorithm and face authentication algorithm with an anti-spoofing function. According to the epidemic prevention problem, the mask detection algorithm is integrated with a dexterous structure which contains a novel context attention head module to focus on the face and mask features, and a series of lightweight units to remove object cross-class. This configuration enables the mask detection module to meet the operation requirement of embedded devices. Besides, we proposed one novel liveness detection method that analyzes the color and texture of image frames using the uniform LBP method. In order to test the performance of the framework, this paper employs CASIA-FASD and Reply-Attack datasets as a benchmark. The test indicates that the HTER is 9.7%, the EER is 5.5%. The average process time of a single frame is 0.12 seconds. The above results demonstrate that this framework has a high anti-spoofing capability and can be employed on the embedded system to complete the mask detection and face authentication task in real-time, which can effectively support the public entry task under an epidemic background.

Acknowledgments

This work is supported by the National Key Research and Development Program of China (No. 2020YFB1313600).

Conflict of interest

All authors declare no conflicts of interest in this paper.