Human activity recognition: an approach 2D CNN-LSTM to sequential image representation and processing of inertial sensor data

Wallace Camacho Carlos; Alessandro Copetti; Luciano Bertini; Leonard Barreto Moreira; Otávio de Souza Martins Gomes; Wallace Camacho Carlos; Alessandro Copetti; Luciano Bertini; Leonard Barreto Moreira; Otávio de Souza Martins Gomes

doi:10.3934/bioeng.2024024

AIMS Bioengineering

2024, Volume 11, Issue 4: 527-560. doi: 10.3934/bioeng.2024024

Previous Article Next Article

Research article Special Issues

Human activity recognition: an approach 2D CNN-LSTM to sequential image representation and processing of inertial sensor data

1.
Science and Technology Institute, Fluminense Federal University (ICT-UFF), Rio das Ostras, RJ, Brazil
2.
Institute of Systems Engineering and Information Technology, Federal University of Itajubá (IESTI-UNIFEI), Itajubá, MG, Brazil

Received: 27 May 2024 Revised: 28 October 2024 Accepted: 15 November 2024 Published: 27 November 2024

The field of human activity recognition, abbreviated as HAR, benefits significantly from deep learning by addressing the complexity of human behavior and the vast volume of data produced by sensors. This work adopted the strategy of converting inertial data, such as accelerometer and gyroscope signals, into 2D images through recurrence plots. This approach facilitated the effective exploration of data input and neural network architectures. By utilizing the recent history of movements as input for the models, this study evaluated the impact of this methodology on HAR using two adapted architectures: 2D convolutional neural networks combined with long short-term memory layers (2D CNN-LSTM) and standalone 2D convolutional neural networks (2D CNN). Their performances were compared with other state-of-the-art deep learning models. The contributions of this study were threefold: the handling of input data, the development of the two network architectures for HAR, and the high accuracy achieved, ranging from 97% to 98%, on the public University of California, Irvine human activity recognition dataset (UCI-HAR). These results highlighted the benefit of incorporating temporal data to enhance accuracy in activity classification.
- activity recognition,
- recurrence plot,
- 2D-DL,
- CNN,
- LSTM
Citation: Wallace Camacho Carlos, Alessandro Copetti, Luciano Bertini, Leonard Barreto Moreira, Otávio de Souza Martins Gomes. Human activity recognition: an approach 2D CNN-LSTM to sequential image representation and processing of inertial sensor data[J]. AIMS Bioengineering, 2024, 11(4): 527-560. doi: 10.3934/bioeng.2024024

Related Papers:

Abstract

The field of human activity recognition, abbreviated as HAR, benefits significantly from deep learning by addressing the complexity of human behavior and the vast volume of data produced by sensors. This work adopted the strategy of converting inertial data, such as accelerometer and gyroscope signals, into 2D images through recurrence plots. This approach facilitated the effective exploration of data input and neural network architectures. By utilizing the recent history of movements as input for the models, this study evaluated the impact of this methodology on HAR using two adapted architectures: 2D convolutional neural networks combined with long short-term memory layers (2D CNN-LSTM) and standalone 2D convolutional neural networks (2D CNN). Their performances were compared with other state-of-the-art deep learning models. The contributions of this study were threefold: the handling of input data, the development of the two network architectures for HAR, and the high accuracy achieved, ranging from 97% to 98%, on the public University of California, Irvine human activity recognition dataset (UCI-HAR). These results highlighted the benefit of incorporating temporal data to enhance accuracy in activity classification.

Acknowledgments

The authors thank FAPERJ for partially funding this work.

Conflict of interest

The authors declare no conflict of interest, ensuring the objectivity and transparency of this research.

References

[1]	Ferrari A, Micucci D, Mobilio M, et al. (2021) Trends in human activity recognition using smartphones. J Reliable Intell Environ 7: 189-213. https://doi.org/10.1007/s40860-021-00147-0
[2]	Ordóñez FJ, Roggen D (2016) Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors 16: 115. https://doi.org/10.3390/s16010115
[3]	Archana R, Jeevaraj PSE (2024) Deep learning models for digital image processing: a review. Artif Intell Rev 57: 11. https://doi.org/10.1007/s10462-023-10631-z
[4]	Wang CY, Bochkovskiy A, Liao HYM YOLOv7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors (2023)2023: 7464-7475. https://doi.org/10.1109/CVPR52729.2023.00721
[5]	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need (2017)2017: 6000-6010. https://doi.org/10.5555/3295222.3295349
[6]	Li T, Zhang Y, Wang T (2021) SRPM–CNN: a combined model based on slide relative position matrix and CNN for time series classification. Complex Intell Syst 7: 1619-1631. https://doi.org/10.1007/s40747-021-00296-y
[7]	Hussain A, Khan SU, Khan N, et al. (2024) AI-driven behavior biometrics framework for robust human activity recognition in surveillance systems. Eng Appl Artif Intell 127: 107218. https://doi.org/10.1016/j.engappai.2023.107218
[8]	An G, Zheng Z, Wu D, et al. (2019) Deep spectral feature pyramid in the frequency domain for long-term action recognition. J Vis Commun Image R 64: 102650. https://doi.org/10.1016/j.jvcir.2019.102650
[9]	Torse DA, Khanai R, Desai VV Classification of epileptic seizures using recurrence plots and machine learning techniques (2019).IEEE2019: 0611-0615. Available from: https://ieeexplore.ieee.org/document/869798
[10]	Hurezeanu B, Ungureanu GM, Digulescu A, et al. Fetal heart rate variability study with recurrence plot analysis (2013).IEEE2013: 1-4. Available from: https://ieeexplore.ieee.org/document/6707310
[11]	San-Um W, Potiwanna C, Jakborvornphan S Characterizations of critical heart disease in ECG signal features through recurrence plots as for medical imaging diagnostics (2018)2018: 183-188. Available from: https://ieeexplore.ieee.org/document/8391189
[12]	Tian Y, Huang J, Sun Y Fault diagnosis for rolling bearings based on recurrence plot and convolutional neural wetwork (2023)2023: 335-340. Available from: https://ieeexplore.ieee.org/document/10109955
[13]	LeCun Y, Boser B, Denker JS, et al. (1989) Backpropagation applied to handwritten zip code recognition. Neural comput 1: 541-551. https://doi.org/10.1162/neco.1989.1.4.541
[14]	Krohn J, Beyleveld G, Bassens A (2019) Deep Learning Illustrated: A Visual, Interactive Guide to Artificial Intelligence. USA: Addison-Wesley Professional. Available from: https://www.deeplearningillustrated.com
[15]	LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521: 436-444. https://doi.org/10.1038/nature14539
[16]	Kang X, Song B, Sun F (2019) A deep similarity metric method based on incomplete data for traffic anomaly detection in IoT. Appl Sci 9: 135. https://doi.org/10.3390/app9010135
[17]	Li Z, Liu F, Yang W, et al. (2021) A survey of convolutional neural networks: analysis, applications, and prospects. IEEE T Neur Net Lear 33: 6999-7019. https://doi.org/10.1109/TNNLS.2021.3084827
[18]	Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neur Comput 9: 1735-1780. https://doi.org/10.1162/neco.1997.9.8.1735
[19]	Gers FA, Schmidhuber J, Cummins F (2000) Learning to forget: continual prediction with LSTM. Neur Comput 12: 2451-2471. https://doi.org/10.1162/089976600300015015
[20]	Arif S, Wang J, Ul Hassan T, et al. (2019) 3D-CNN-based fused feature maps with LSTM applied to action recognition. Future Int 11: 42. https://doi.org/10.3390/fi11020042
[21]	Ercolano G, Rossi S (2021) Combining CNN and LSTM for activity of daily living recognition with a 3D matrix skeleton representation. Intel Serv Robot 14: 175-185. https://doi.org/10.1007/s11370-021-00358-7
[22]	Deng F, Chen Z, Liu Y, et al. (2022) A novel combination neural network based on convlstm-transformer for bearing remaining useful life prediction. Machines 10: 1226. https://doi.org/10.3390/machines10121226
[23]	Shi X, Chen Z, Wang H, et al. (2015) Convolutional LSTM network: A machine learning approach for precipitation nowcasting. Advances in Neural Information Processing Systems. NY: Curran Associates 802-810. Available from: https://proceedings.neurips.cc/paper/2015/hash/07563a3fe3bbe7e3ba84431ad9d055af-Abstract.html
[24]	Yu C, Ma X, Ren J, et al. Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction (2020).Springer-Verlag, United Kingdom, Springer International Publishing 507-523. https://doi.org/10.1007/978-3-030-58610-2_30
[25]	Gupta S (2021) Deep learning based human activity recognition (HAR) using wearable sensor data. Int J Inf Manage Data Insights 1: 100046. https://doi.org/10.1016/j.jjimei.2021.100046
[26]	Marwan N (2008) A historical review of recurrence plots. Eur Phys J Spec Top 164: 3-12. https://doi.org/10.1140/epjst/e2008-00829-1
[27]	Marwan N, Carmen Romano M, Thiel M, et al. (2007) Recurrence plots for the analysis of complex systems. Phys Rep 438: 237-329. https://doi.org/10.1016/j.physrep.2006.11.001
[28]	Daniel N, Klein I (2021) INIM: inertial images construction with applications to activity recognition. Sensors 21: 4787. https://doi.org/10.3390/s21144787
[29]	Wang Z, Oates T Imaging time-series to improve classification and imputation. Proceedings of the 24th International Conference on Artificial Intelligence (2015).AAAI Press2015: 3939-3945. https://doi.org/10.48550/arXiv.1506.00327
[30]	Pandey A, Kumar P, Prasad S 2d convolutional lstm-based approach for human action recognition on various sensor data (2023).Springer Nature Singapore327: 405-417. https://doi.org/10.1007/978-981-19-7524-0_36
[31]	Koşar E, Barshan B (2023) A new CNN-LSTM architecture for activity recognition employing wearable motion sensor data: enabling diverse feature extraction. Eng Appl Artif Intel 124: 106529. https://doi.org/10.1016/j.engappai.2023.106529
[32]	Yang C-L, Yang C-Y, Chen Z-X, et al. Multivariate time series data transformation for convolutional neural network (2019).IEEE2019: 188-192. https://doi.org/10.1109/SII.2019.8700425
[33]	Bisong E (2019) Google Colaboratory. Building Machine Learning and Deep Learning Models on Google Cloud Platform. Berkeley: Apress 59-64. https://doi.org/10.1007/978-1-4842-4470-8_7
[34]	Tan M, Le Q Efficientnet: Rethinking model scaling for convolutional neural networks (2019).PMLR97: 6105-6114. Available from: https://proceedings.mlr.press/v97/tan19a.html
[35]	Qiao H, Wang T, Wang P, et al. (2018) A time-distributed spatiotemporal feature learning method for machine health monitoring with multi-sensor time series. Sensors 18: 2932. https://doi.org/10.3390/s18092932
[36]	Blunck H, Bhattacharya S, Prentow T, et al. (2015) Heterogeneity activity recognition. UCI Mach Learn Repos 10: C5689X. https://doi.org/10.24432/C5689X
[37]	Barshan B, Altun K (2013) Daily and sports activities. UCI Mach Learn Repos 10: C5C59F. https://doi.org/10.24432/C5C59F

Reader Comments

Your name:*

Email:*
© 2024 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)