Human motion recognition based on Nano-CMOS Image sensor

Shangbin Li; Yu Liu; Shangbin Li; Yu Liu

doi:10.3934/mbe.2023444

Mathematical Biosciences and Engineering

2023, Volume 20, Issue 6: 10135-10152. doi: 10.3934/mbe.2023444

Previous Article Next Article

Research article

Human motion recognition based on Nano-CMOS Image sensor

Shangbin Li ,
Yu Liu ^,

Physical Education Department, Harbin Engineering University, Harbin 150001, China

Academic Editor: Yang Kuang

Received: 27 January 2023 Revised: 19 March 2023 Accepted: 22 March 2023 Published: 29 March 2023

Human motion recognition is of great value in the fields of intelligent monitoring systems, driver assistance system, advanced human-computer interaction, human motion analysis, image and video processing. However, the current human motion recognition methods have the problem of poor recognition effect. Therefore, we propose a human motion recognition method based on Nano complementary metal oxide semiconductor (CMOS) image sensor. First, using the Nano-CMOS image sensor to transform and process the human motion image, and combines the background mixed model of pixels in the human motion image to extract the human motion features, and feature selection is conducted. Second, according to the three-dimensional scanning features of Nano-CMOS image sensor, the human joint coordinate information data is collected, the state variables of human motion are sensed by the sensor, and the human motion model is constructed according to the measurement matrix of human motions. Finally, the foreground features of human motion images are obtained by calculating the feature parameters of each motion gesture. According to the posterior conditional probability of human motion images, the recognition objective function of human motion is obtained to realize human motion recognition. The results show that the human motion recognition effect of the proposed method is good, the extraction accuracy is high, the average human motion recognition rate is 92%, the classification accuracy is high, and the recognition speed is up to 186 frames/s.
- recognition,
- human motion,
- Nano-CMOS Image sensor,
- feature extraction,
- gray image
Citation: Shangbin Li, Yu Liu. Human motion recognition based on Nano-CMOS Image sensor[J]. Mathematical Biosciences and Engineering, 2023, 20(6): 10135-10152. doi: 10.3934/mbe.2023444

Related Papers:

Abstract

Human motion recognition is of great value in the fields of intelligent monitoring systems, driver assistance system, advanced human-computer interaction, human motion analysis, image and video processing. However, the current human motion recognition methods have the problem of poor recognition effect. Therefore, we propose a human motion recognition method based on Nano complementary metal oxide semiconductor (CMOS) image sensor. First, using the Nano-CMOS image sensor to transform and process the human motion image, and combines the background mixed model of pixels in the human motion image to extract the human motion features, and feature selection is conducted. Second, according to the three-dimensional scanning features of Nano-CMOS image sensor, the human joint coordinate information data is collected, the state variables of human motion are sensed by the sensor, and the human motion model is constructed according to the measurement matrix of human motions. Finally, the foreground features of human motion images are obtained by calculating the feature parameters of each motion gesture. According to the posterior conditional probability of human motion images, the recognition objective function of human motion is obtained to realize human motion recognition. The results show that the human motion recognition effect of the proposed method is good, the extraction accuracy is high, the average human motion recognition rate is 92%, the classification accuracy is high, and the recognition speed is up to 186 frames/s.

References

[1]	G. Yan, M. Hua, Z. Zhong, Multi-derivative physical and geometric convolutional embedding networks for skeleton-based action recognition. Comput. Aided Gemo. D., 86 (2021), 101964. https://doi.org/10.1016/j.cagd.2021.101964 doi: 10.1016/j.cagd.2021.101964
[2]	H. Su, W. Qi, Y. Schmirander, S. E. Ovur, S. Cai, X. Xiong, A human activity-aware shared control solution for medical human-robot interaction, Assembly Auton., 42 (2022), 388–394. https://doi.org/10.1108/AA-12-2021-0174 doi: 10.1108/AA-12-2021-0174
[3]	Y. Xue, Y. Yu, K. Yin, P. F. Li, S. X. Xie, Z. J. Ju, Human In-Hand motion recognition based on multi-modal perception information fusion, IEEE Sens. J., 22 (2022), 6793–6805. https://doi.org/10.1109/JSEN.2022.3148992 doi: 10.1109/JSEN.2022.3148992
[4]	Y. Liu, H. Zhang, D. Xu, K. J. He, Graph transformer network with temporal kernel attention for skeleton-based action recognition, Knowl.-Based Syst., 240 (2022), 108146. https://doi.org/10.1016/j.knosys.2022.108146 doi: 10.1016/j.knosys.2022.108146
[5]	Y. Yuan, B. Yu, W. Wang, B. H. Yu, Multi-filter dynamic graph convolutional networks for skeleton-based action recognition, Procedia Comput. Sci., 183 (2021), 572–578. https://doi.org/10.1016/j.procs.2021.02.099 doi: 10.1016/j.procs.2021.02.099
[6]	D. Feng, Z. C. Wu, J. Zhang, T. Ren, Multi-Scale Spatial Temporal Graph Neural Network for Skeleton-Based Action Recognition, IEEE Access, 9 (2021), 58256–58265. https://doi.org/10.1109/ACCESS.2021.3073107 doi: 10.1109/ACCESS.2021.3073107
[7]	Y. Q. Hong, Visual human action recognition based on deep belief network, Comput. Sci., 48 (2021), 400–403. https://doi.org/10.11896/jsjkx.210200079 doi: 10.11896/jsjkx.210200079
[8]	H. Su, W. Qi, J. Chen, D. Zhang, Fuzzy approximation-based Task-Space control of robot manipulators with remote center of motion constraint, IEEE Trans. Fuzzy Syst., 30 (2022), 1564–1573. https://doi.org/10.1109/TFUZZ.2022.3157075 doi: 10.1109/TFUZZ.2022.3157075
[9]	X. Y. Li, X. W. Hao, J. G. Jia, Y. F. Zhou, Human action recognition method based on multi-attention mechanism and spatiotemporal graph convolution networks, J. Comput. Aided Des. Comput. Graph., 33 (2021), 1055–1063. https://doi.org/10.3724/SP.J.1089.2021.18640 doi: 10.3724/SP.J.1089.2021.18640
[10]	Z. T. Xiao, L. Zhang, W. Wang, Human action recognition based on kinematic dynamic image, J. Tianjin Polytechnic Univer., 40 (2021), 53–59. https://doi.org/10.3969/j.issn.1671-024x.2021.01.010 doi: 10.3969/j.issn.1671-024x.2021.01.010
[11]	Q. Liu, Human motion state recognition based on MEMS sensors and Zigbee network, Comput. Commun., 181 (2022), 164–172. https://doi.org/10.1016/j.comcom.2021.10.018 doi: 10.1016/j.comcom.2021.10.018
[12]	P. Chen, S. Guo, H. Li, X. Wang, G. L. Cui, C. S Jiang, et al., Through-wall human motion recognition based on transfer learning and ensemble learning, IEEE Geosci. Remote. S., 19 (2022), 1–5. https://doi.org/10.1109/LGRS.2021.3070374 doi: 10.1109/LGRS.2021.3070374
[13]	Z. Tao, Z. Hao, Y. Lei, Human motion mode recognition based on multi-parameter fusion of wearable inertial module unit and flexible pressure sensor, Sensor. Mater., 34 (2022), 1017–1031. https://doi.org/10.18494/SAM3755 doi: 10.18494/SAM3755
[14]	W. Luo, B. Ning, High-dynamic dance motion recognition method based on video visual analysis, Sci. Programming-Neth, 2022 (2022), 1–9. https://doi.org/10.1155/2022/6724892 doi: 10.1155/2022/6724892
[15]	H. Lee, J. K. Mandivarapu, N. Ogbazghi, Y. S. Li, Real-time interface control with motion gesture recognition based on non-contact capacitive sensing, preprint, arXiv: 2201.01755.
[16]	S. Chen, K. Xu, Z. Mi, X. H. Jiang, T. F. Sun, Dual-domain graph convolutional networks for skeleton-based action recognition, Mach. Learn., 111 (2022), 2381–2406. https://doi.org/10.1007/s10994-022-06141-8 doi: 10.1007/s10994-022-06141-8
[17]	X. Ji, Q. Zhao, J. Cheng, C. F. Ma, Exploiting spatio-temporal representation for 3D human action recognition from depth map sequences, Knowl.-Based Syst., 227 (2021), 107040. https://doi.org/10.1016/j.knosys.2021.107040 doi: 10.1016/j.knosys.2021.107040
[18]	W. Ding, C. Ding, G. Li, K. Liu, Skeleton-based square grid for human action recognition with 3D convolutional neural network, IEEE Access, 9 (2021), 54078–54089. https://doi.org/10.1109/ACCESS.2021.3059650 doi: 10.1109/ACCESS.2021.3059650
[19]	M. F. Tsai, C. H. Chen, Spatial temporal variation graph convolutional networks (STV-GCN) for skeleton-based emotional action recognition, IEEE Access, 9 (2021), 13870–13877. https://doi.org/10.1109/ACCESS.2021.3052246 doi: 10.1109/ACCESS.2021.3052246
[20]	H. Xia, X. Gao, Multi-scale mixed dense graph convolution network for skeleton-based action recognition. IEEE Access, 9 (2021), 36475–36484. https://doi.org/10.1109/ACCESS.2020.3049029 doi: 10.1109/ACCESS.2020.3049029
[21]	K. B. de Carvalho, V. T. Basílio, A. S Brando, Action recognition for educational proposals applying concepts of social assistive robotics, Cogn. Syst. Res., 71 (2022), 1–8. https://doi.org/10.1016/j.cogsys.2021.09.002 doi: 10.1016/j.cogsys.2021.09.002
[22]	Y. Kong, Y. Wang, A. Li, Spatiotemporal saliency representation learning for video action recognition, IEEE T. Multimedia, 24 (2022), 1515–1528. https://doi.org/10.1109/TMM.2021.3066775 doi: 10.1109/TMM.2021.3066775
[23]	H. Wang, B. Yu, K. Xia, J. Q. Li, X. Zuo, Skeleton edge motion networks for human action recognition, Neurocomputing, 423 (2021), 1–12. https://doi.org/10.1016/j.neucom.2020.10.037 doi: 10.1016/j.neucom.2020.10.037
[24]	R. D. Brehar, M. P. Muresan, T. Marita, C. Vancea, M. Negru, S. Nedevschi, Pedestrian street-cross action recognition in monocular far infrared sequences, IEEE Access, 9 (2021), 74302–74324. https://doi.org/10.1109/ACCESS.2021.3080822 doi: 10.1109/ACCESS.2021.3080822
[25]	J. Xie, W. Xin, R. Liu, L. Sheng, X. Liu, X. Gao, et al., Cross-channel graph convolutional networks for skeleton-based action recognition, IEEE Access, 9 (2021), 9055–9065. https://doi.org/10.1109/ACCESS.2021.3049808 doi: 10.1109/ACCESS.2021.3049808
[26]	A. Avp, B. Apa, A. Iao, Comparison of action recognition from video and IMUs, Procedia Comput. Sci., 186 (2021), 242–249. https://doi.org/10.1016/j.procs.2021.04.144 doi: 10.1016/j.procs.2021.04.144
[27]	R. Xia, Y. Li, W. Luo, LAGA-Net: Local-and-global attention network for skeleton based action recognition, IEEE T. Multimedia, 24 (2022), 2648–2661. https://doi.org/10.1109/TMM.2021.3086758 doi: 10.1109/TMM.2021.3086758

Reader Comments

Your name:*

Email:*
© 2023 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)