A lightweight double-channel depthwise separable convolutional neural network for multimodal fusion gait recognition

Xiaoguang Liu; Meng Chen; Tie Liang; Cunguang Lou; Hongrui Wang; Xiuling Liu; Xiaoguang Liu; Meng Chen; Tie Liang; Cunguang Lou; Hongrui Wang; Xiuling Liu

doi:10.3934/mbe.2022055

Mathematical Biosciences and Engineering

2022, Volume 19, Issue 2: 1195-1212. doi: 10.3934/mbe.2022055

Previous Article Next Article

Research article

A lightweight double-channel depthwise separable convolutional neural network for multimodal fusion gait recognition

1.
College of Electronic and Information Engineering, Hebei University, Baoding, Hebei, China
2.
Key Laboratory of Digital Medical Engineering of Hebei Province, Hebei University, Baoding Hebei, China
academic editor: Ivan Tyukin

Received: 30 September 2021 Accepted: 23 November 2021 Published: 30 November 2021

Gait recognition is an emerging biometric technology that can be used to protect the privacy of wearable device owners. To improve the performance of the existing gait recognition method based on wearable devices and to reduce the memory size of the model and increase its robustness, a new identification method based on multimodal fusion of gait cycle data is proposed. In addition, to preserve the time-dependence and correlation of the data, we convert the time-series data into two-dimensional images using the Gramian angular field (GAF) algorithm. To address the problem of high model complexity in existing methods, we propose a lightweight double-channel depthwise separable convolutional neural network (DC-DSCNN) model for gait recognition for wearable devices. Specifically, the time series data of gait cycles and GAF images are first transferred to the upper and lower layers of the DC-DSCNN model. The gait features are then extracted with a three-layer depthwise separable convolutional neural network (DSCNN) module. Next, the extracted features are transferred to a softmax classifier to implement gait recognition. To evaluate the performance of the proposed method, the gait dataset of 24 subjects were collected. Experimental results show that the recognition accuracy of the DC-DSCNN algorithm is 99.58%, and the memory usage of the model is only 972 KB, which verifies that the proposed method can enable gait recognition for wearable devices with lower power consumption and higher real-time performance.
- wearable devices,
- gait recognition,
- gait cycles,
- multimodal fusion,
- DC-DSCNN
Citation: Xiaoguang Liu, Meng Chen, Tie Liang, Cunguang Lou, Hongrui Wang, Xiuling Liu. A lightweight double-channel depthwise separable convolutional neural network for multimodal fusion gait recognition[J]. Mathematical Biosciences and Engineering, 2022, 19(2): 1195-1212. doi: 10.3934/mbe.2022055

Related Papers:

Abstract

Gait recognition is an emerging biometric technology that can be used to protect the privacy of wearable device owners. To improve the performance of the existing gait recognition method based on wearable devices and to reduce the memory size of the model and increase its robustness, a new identification method based on multimodal fusion of gait cycle data is proposed. In addition, to preserve the time-dependence and correlation of the data, we convert the time-series data into two-dimensional images using the Gramian angular field (GAF) algorithm. To address the problem of high model complexity in existing methods, we propose a lightweight double-channel depthwise separable convolutional neural network (DC-DSCNN) model for gait recognition for wearable devices. Specifically, the time series data of gait cycles and GAF images are first transferred to the upper and lower layers of the DC-DSCNN model. The gait features are then extracted with a three-layer depthwise separable convolutional neural network (DSCNN) module. Next, the extracted features are transferred to a softmax classifier to implement gait recognition. To evaluate the performance of the proposed method, the gait dataset of 24 subjects were collected. Experimental results show that the recognition accuracy of the DC-DSCNN algorithm is 99.58%, and the memory usage of the model is only 972 KB, which verifies that the proposed method can enable gait recognition for wearable devices with lower power consumption and higher real-time performance.

References

[1]	J. Zhou, Z. F. Cao, X. L. Dong, X. D. Lin, Security and privacy in cloud-assisted wireless wearable communications: challenges, solutions and, future directions, IEEE Wireless Commun., 22 (2015), 136–144. doi: 10.1109/MWC.2015.7096296. doi: 10.1109/MWC.2015.7096296
[2]	K. Bayoumy, M. Gaber, A. Elshafeey, O. Mhaimeed, E. H. Dineen, F. A. Marvel, et al., Smart wearable devices in cardiovascular care: where we are and how to move forward, Nat. Rev. Cardiol., 18 (2021), 581–599. doi: 10.1038/ s41569-021-00522-7. doi: 10.1038/s41569-021-00522-7
[3]	R. Mungovan, Face recognition: fighting the fakes, Biom. Technol. Today, 2021 (2021), 5–7. doi: 10.1016/S0969-4765(21)00021-7. doi: 10.1016/S0969-4765(21)00021-7
[4]	G. Jeon, S. Lee, S. H. Lee, J. Shim, J. Ra, K. W. Park, et al., Highly sensitive active-matrix driven self-capacitive fingerprint sensor based on oxide thin film transistor, Sci. Rep., 9 (2019), 3216–3226. doi: 10.1038/s41598-019-40005-x. doi: 10.1038/s41598-019-40005-x
[5]	M. Kumar, N. Singh, R. Kumar, S. Goel, K. Kumar, Gait recognition based on vision systems: A systematic survey, J. Visual Commun. Image Representation, 75 (2021), 103052–103064. doi: 10.1016/j.jvcir.2021.103052. doi: 10.1016/j.jvcir.2021.103052
[6]	H. J. Ailisto, M. Lindholm, J. Mäntyjärvi, E. Vildjiounaite, S. Mäkelä, Identifying people from gait pattern with accelerometers, Biom. Technol. Hum. Identif. II., 5779 (2005), 7–14. doi: 10.1117/12.603331. doi: 10.1117/12.603331
[7]	L. Rong, J. Zhou, M. Liu, X. Hou, A wearable acceleration sensor system for gait recognition, in 2007 2nd IEEE Conference on Industrial Electronics and Applications., 2007. Available from: https://ieeexplore.ieee.org/document/4318894.
[8]	F. M. Sun, C. F. Mao, X. M. Fan, Y. Li, Accelerometer-based speed-adaptive gait authentication method for wearable IoT devices, IEEE Int. Things. J., 6 (2018), 820–830. doi: 10.1109/JIOT.2018.2860592. doi: 10.1109/JIOT.2018.2860592
[9]	M. Ahmad, A. K. Bashir, A. M. Khan, M. Mazzara, S. Distefano, S. Sarfraz, Multi sensor-based implicit user identification, preprint, arXiv: 1706.01739v3.
[10]	S. Choi, I. H. Youn, R. LeMay, S. Burns, J. H. Youn, Biometric gait recognition based on wireless acceleration sensor using k-nearest neighbor classification, Int. Conf. Comput., 2014. Available from: https://ieeexplore.ieee.org/document/6785491.
[11]	M. Gadaleta, M. Rossi, IDNet: smartphone-based gait recognition with convolutional neural networks, Pattern Recognit., 74 (2018), 25–37. doi: 10.1016/j.patcog.2017.09.005. doi: 10.1016/j.patcog.2017.09.005
[12]	R. Delgado-Escano, F. M. Castro, J. R. Cozar, M. J. Marin-Jimenez, N. Guil, An end-to-end multi- task and fusion CNN for inertial-based gait recognition, IEEE Access., 7 (2018), 1897–1908. doi: 10.1109/ACCESS.2018.2886899. doi: 10.1109/ACCESS.2018.2886899
[13]	Q. Zou, Y. L. Wang, Q. Wang, Y. Zhao, Q. Q. Li, Deep learning-based gait recognition using smartphones in the wild, IEEE Trans. Inf. Forensics Secur., 15 (2020), 3197–3212. doi: 10.1109/TIFS.2020.2985628. doi: 10.1109/TIFS.2020.2985628
[14]	L. Tran, T. Hoang, T. Nguyen, H. Kim, D. Choi, Multi-model long short-term memory network for gait recognition using window-based data segment, IEEE Access., 9 (2021), 23826–23839. doi: 10.1109/ACCESS.2021.3056880. doi: 10.1109/ACCESS.2021.3056880
[15]	A. I. Middya, S. Roy, S. Mandal, R. Talukdar, Privacy protected user identification using deep learning for smartphone-based participatory sensing applications, Neural Comput. Appl., 33 (2021), 17303–17313. doi: 10.1007/s00521-021-06319-6. doi: 10.1007/s00521-021-06319-6
[16]	H. H. Huang, P. Zhou, Y. Li, F. M. Sun, A lightweight attention-based CNN model for efficient gait recognition with wearable IMU sensors, Sensors, 21 (2021), 2866–2879. doi: 10.3390/s21082866. doi: 10.3390/s21082866
[17]	M. Paulich, M. Schepers, N. Rudigkeit, G. Bellusci, Xsens MTw Awinda: miniature wireless inertial-magnetic motion tracker for highly accurate 3D kinematic applications, XSens: Enschede, The Netherlands, (2018), 1–9. doi: 10.13140/RG.2.2.23576.49929.
[18]	L. F. Mo, L. J. Zeng, Running gait pattern recognition based on cross-correlation analysis of single acceleration sensor, Math. Biosci. Eng., 16 (2019). 6242–6256. doi: 10.3934/mbe.2019311. doi: 10.3934/mbe.2019311
[19]	B. Auvinet, G. Berrut, C. Touzard, L. Moutel, N. Collet, D. Chaleil, et al., Reference data for normal subjects obtained with an accelerometric device, Gait Posture., 16 (2002), 124–134. doi: 10.1016/S0966-6362(01)00203-X. doi: 10.1016/S0966-6362(01)00203-X
[20]	H. Prasanth, M. Caban, U. Keller, G. Courtine, A. Ijspeert, H. Vallery, et al., Wearable sensor-based real-time gait detection: a systematic review, Sensors, 21 (2021), 2727–2755. doi: 10.3390/s21082727. doi: 10.3390/s21082727
[21]	M. Muller, Dynamic time warping, Information Retrieval for Music and Motion, (2007), 69–84. doi: 10.1007/978-3-540-74048-3_4.
[22]	Z. G. Wang, T. Oates, Imaging time-series to improve classification and imputation, in Proceeding of 24th International Joint Conference on Artificial Intelligence, preprint, arXiv: 1506.00327v1.
[23]	A. G. Howard, M. L. Zhu, B. Chen, D. Kalenichenko, W. J. Wang, T. Weyand, et al., MobileNets: efficient convolutional neural networks for mobile vision applications, preprint, arXiv: 1704.04861v1.
[24]	F. Chollet, Xception: deep learning with depthwise separable convolutions, in 2017 IEEE CVPR, Honolulu, HI, USA, (2017), 1800–1807. doi: 10.1109/CVPR.2017.195.
[25]	M. N. Chong, Q. M. Li, J. Li., Parameter estimation via deep learning for camera localization, IOP Conf. Ser.: Mater. Sci. Eng., 569 (2019). doi: 10.1088/1757-899X/569/5/052101. doi: 10.1088/1757-899X/569/5/052101
[26]	S. H. Wang, Z. Q. Zhu, Y. D. Zhang, PSCNN: PatchShuffle convolutional neural network for COVID-19 explainable diagnosis, Front. Public Health, 9 (2021), 768278–768304. doi: 10.3389/fpubh.2021.768278. doi: 10.3389/fpubh.2021.768278
[27]	D. A. Clevert, T. Unterthiner, S. Hochreiter, Fast and accurate deep network learning by exponential linear units (ELUs), preprint, arXiv: 1511.07289v5.
[28]	V. Nair, G. E. Hinton, Rectified linear units improve restricted boltzmann machines, in Proceedings of the 27th International Conference on Machine Learning, Haifa, Israel, (2010). 807–814.

Reader Comments

Your name:*

Email:*
© 2022 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)