Face emotion recognition based on infrared thermal imagery by applying machine learning and parallelism

Basem Assiri; Mohammad Alamgir Hossain; Basem Assiri; Mohammad Alamgir Hossain

doi:10.3934/mbe.2023042

Mathematical Biosciences and Engineering

2023, Volume 20, Issue 1: 913-929. doi: 10.3934/mbe.2023042

Previous Article Next Article

Research article Special Issues

Face emotion recognition based on infrared thermal imagery by applying machine learning and parallelism

Basem Assiri ,
Mohammad Alamgir Hossain ^,

Department of Computer Science, College of CS & IT, Jazan University, Kingdom of Saudi Arabia

Received: 16 April 2022 Revised: 28 July 2022 Accepted: 04 September 2022 Published: 18 October 2022

Over time for the past few years, facial expression identification has been a promising area. However, darkness, lighting conditions, and other factors make facial emotion identification challenging to detect. As a result, thermal images are suggested as a solution to such problems and for a variety of other benefits. Furthermore, focusing on significant regions of a face rather than the entire face is sufficient for reducing processing and improving accuracy at the same time. This research introduces novel infrared thermal image-based approaches for facial emotion recognition. First, the entire image of the face is separated into four pieces. Then, we accepted only four active regions (ARs) to prepare training and testing datasets. These four ARs are the left eye, right eye, and lips areas. In addition, ten-folded cross-validation is proposed to improve recognition accuracy using Convolutional Neural Network (CNN), a machine learning technique. Furthermore, we incorporated a parallelism technique to reduce processing-time in testing and training datasets. As a result, we have seen that the processing time reduces to 50%. Finally, a decision-level fusion is applied to improve the recognition accuracy. As a result, the proposed technique achieves a recognition accuracy of 96.87 %. The achieved accuracy ascertains the robustness of our proposed scheme.
- facial emotion recognition,
- infrared image,
- active region, machine learning,
- parallelism
Citation: Basem Assiri, Mohammad Alamgir Hossain. Face emotion recognition based on infrared thermal imagery by applying machine learning and parallelism[J]. Mathematical Biosciences and Engineering, 2023, 20(1): 913-929. doi: 10.3934/mbe.2023042

Related Papers:

Abstract

Over time for the past few years, facial expression identification has been a promising area. However, darkness, lighting conditions, and other factors make facial emotion identification challenging to detect. As a result, thermal images are suggested as a solution to such problems and for a variety of other benefits. Furthermore, focusing on significant regions of a face rather than the entire face is sufficient for reducing processing and improving accuracy at the same time. This research introduces novel infrared thermal image-based approaches for facial emotion recognition. First, the entire image of the face is separated into four pieces. Then, we accepted only four active regions (ARs) to prepare training and testing datasets. These four ARs are the left eye, right eye, and lips areas. In addition, ten-folded cross-validation is proposed to improve recognition accuracy using Convolutional Neural Network (CNN), a machine learning technique. Furthermore, we incorporated a parallelism technique to reduce processing-time in testing and training datasets. As a result, we have seen that the processing time reduces to 50%. Finally, a decision-level fusion is applied to improve the recognition accuracy. As a result, the proposed technique achieves a recognition accuracy of 96.87 %. The achieved accuracy ascertains the robustness of our proposed scheme.

References

[1]	M. A. Hossain, B. Assiri, Facial emotion verification by infrared image, in International Conference on Emerging Smart Computing and Informatics (ESCI), (2020). https://doi.org/10.1109/ESCI48226.2020.9167616
[2]	M. A. Hossain, B. Assiri, Emotion specific human face authentication based on infrared thermal image, in 2020 2nd International Conference on Computer and Information Sciences (ICCIS), (2020). https://doi.org/10.1109/ICCIS49240.2020.9257683
[3]	M. Vollmer, K. P. Möllmann, Infrared Thermal Imaging: Fundamentals, Research and Applications, 2nd edition, WILEY-VCH Verlag, 2018. https://doi.org/10.1002/9783527693306
[4]	M. A. Hossain, G. Sanyal, Tracking humans based on interest point over span-space in multifarious situations, Int. J. Software Eng. Appl., 10 (2016), 175–192. https://doi.org/10.14257/ijseia.2016.10.9.15 doi: 10.14257/ijseia.2016.10.9.15
[5]	C. Myeon-gyun, A study on the obstacle recognition for autonomous driving RC car using lidar and infrared thermal camera, in 2019 Eleventh International Conference on Ubiquitous and Future Networks (ICUFN), (2019). http://doi.org/10.1109/ICUFN.2019.8806152
[6]	Q. Wan, S. P. Rao, A. Kaszowska, V. Voronin, K. Panetta, H. A, Taylor, et al., Face description using anisotropic gradient: infrared thermal to visible face recognition, in Mobile Multimedia Image Processing, Security, and Applications, (2018). https://doi.org/10.1117/12.2304898
[7]	T. Bae, K. Youngchoon, A. Sangho, IR-band conversion of target and background using surface temperature estimation and error compensation for military IR sensor simulation, Sensors, 19 (2019), 2455. https://doi.org/10.3390/s19112455 doi: 10.3390/s19112455
[8]	Y. Abdelrahman, P. Knierim, P. W. Wozniak, N. Henze, A. Schmidt, See through the fire: evaluating the augmentation of visual perception of firefighters using depth and thermal cameras, in Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing, (2017), 693–696. https://doi.org/10.1145/3123024.3129269
[9]	M. A. Hossain, D. Samanta, G. Sanyal, Eye diseases detection based on covariance, Int. J. Comput. Sci. Inf. Technol. Sec., 2 (2012), 376–379.
[10]	E. Sousa, R. Vardasca, S. Teixeira, A. Seixas, J. Mendes, A. Costa-Ferreira, A review on the application of medical infrared thermal imaging in hands, Infrared Phy. Technol., 85 (2017), 315–323. https://doi.org/10.1016/j.infrared.2017.07.020 doi: 10.1016/j.infrared.2017.07.020
[11]	M. A. Hossain, B. Assiri, Facial expression recognition based on active region of interest using deep learning and parallelism, Peer. J. Comput. Sci., 8 (2022), e894. https://doi.org/10.7717/peerj-cs.894 doi: 10.7717/peerj-cs.894
[12]	N. M. Moacdieh, N. Sarter, The effects of data density, display organization, and stress on search prformance: An eye tracking study of clutter, IEEE Trans. Hmman Mach. Syst., 47 (2017), 886–895. https://doi.org/10.1109/THMS.2017.2717899 doi: 10.1109/THMS.2017.2717899
[13]	M. A. Hossain, B. Assiri, An enhanced eye tracking approach using pipeline computation, Arab. J. Sci. Eng., 45 (2020), 1–14. https://doi.org/10.1007/s13369-019-04322-7 doi: 10.1007/s13369-019-04073-5
[14]	S. U. Mahmood, F. Crimbly, S. Khan, E. Choudry, S. Mehwish, Strategies for rational use of personal protective equipment (PPE) among healthcare providers during the COVID-19 crisis, Cureus, 12 (2020), e8248. http://doi.org/10.7759/cureus.8248 doi: 10.7759/cureus.8248
[15]	C. Filippini, D. Perpetuini, D. Cardone, A. M. Chiarelli, A. Merla, Thermal infrared imaging-based affective computing and its application to facilitate human robot interaction: A review, Appl. Sci., 10 (2020), 2924. https://doi.org/10.3390/app10082924 doi: 10.3390/app10082924
[16]	M. A. Eid, N. Giakoumidis, A. El Saddik, A novel eye-gaze-controlled wheelchair system for navigating unknown environments: Case study with a person with ALS, IEEE Access, 4 (2016), 558–573. https://doi.org/10.1109/ACCESS.2016.2520093 doi: 10.1109/ACCESS.2016.2520093
[17]	M. A. Hossain, D. Samanta, G. Sanyal, Extraction of panic expression depending on lip detection, in 2012 International Conference on Computing Sciences, (2012), 137–141, https://doi.org/10.1109/ICCS.2012.35
[18]	M. A. Hossain, D. Samanta, Automated smiley face extraction based on genetic algorithm, Comput. Sci. Inf. Technol., 2012 (2012), 31–37. https://doi.org/10.5121/csit.2012.2304 doi: 10.5121/csit.2012.2304
[19]	S. S. Alam, R. Jianu, Analyzing eye-tracking information in visualization and data space: From where on the screen to what on the screen, IEEE Trans. Visualization Comput. Graphics, 23 (2017), 1492–1505. https://doi.org/10.1109/TVCG.2016.2535340 doi: 10.1109/TVCG.2016.2535340
[20]	W. Zhang, H. Liu, Toward a reliable collection of eye-tracking data for image quality research: Challenges, solutions, and applications, IEEE Trans. Image Process., 26 (2017), 2424–2437. https://doi.org/10.1109/TIP.2017.2681424 doi: 10.1109/TIP.2017.2681424
[21]	A. Torabi, G. Massé, G. A. Bilodeau, An iterative integrated framework for thermal–visible image registration, sensor fusion, and people tracking for video surveillance applications, Comput. Vision Image Understanding, 116 (2012), 210–221. https://doi.org/10.1016/j.cviu.2011.10.006 doi: 10.1016/j.cviu.2011.10.006
[22]	Y. Liu, Y. Cao, Y. Li, M. Liu, R. Song, Y. Wang, et al., Facial expression recognition with PCA and LBP features extracting from active facial patches, in 2016 IEEE International Conference on Real-time Computing and Robotics (RCAR), (2016). https://doi.org/10.1109/RCAR.2016.7784056
[23]	W. R. Almeida, F. A. Andaló, R. Padilha, G. Bertocco, W. Dias, R. da S. Torres, et al., Detecting face presentation attacks in mobile devices with a patch-based CNN and a sensor aware loss function, Plos One, 4 (2020), 1–24. https://doi.org/doi.org/10.1155/2020/6385281 doi: 10.1155/2020/6385281
[24]	F. Khan, Facial expression recognition using facial landmark detection and feature extraction via neural networks, preprint, arXiv: 1812.04510.
[25]	M. A. Hossain, H. Zogan, Emotion tracking and grading based on sophisticated statistical approach, Int. J. Adv. Electron. Comput. Sci., 5 (2018), 9–13. https://doi.org/12-451-152482928314-18
[26]	W. Zhang, X. Sui, G. Gu, Q. Chen, H. Cao, Infrared thermal imaging super-resolution via multiscale Spatio-Temporal feature fusion network, IEEE Sensors J., 21 (2021), 19176–19185. https://doi.org/10.1109/JSEN.2021.3090021 doi: 10.1109/JSEN.2021.3090021
[27]	H. Mady, S. M. S. Hilles, Face recognition and detection using Random forest and combination of LBP and HOG features, in 2018 International Conference on Smart Computing and Electronic Enterprise (ICSCEE), (2018). https://doi.org/10.1109/ICSCEE.2018.8538377
[28]	K. T. Islam, R. G. Raj, A. Al-Murad, Performance of SVM, CNN, and ANN with BoW, HOG, and image pixels in face recognition, in 2017 2nd International Conference on Electrical & Electronic Engineering (ICEEE), (2017). https://doi.org/10.1109/CEEE.2017.8412925
[29]	M. Sajjad, S. Zahir, A. Ullah, Z. Akhtar, K. Muhammad, Human behavior understanding in big multimedia data using CNN based facial expression recognition, Mobile Networks Appl., 25 (2020), 1611–1621. https://doi.org/10.1007/s11036-019-01366-9 doi: 10.1007/s11036-019-01366-9
[30]	P. Liu, S. Han, Z. Meng, Y. Tong, Facial expression recognition via a boosted deep belief network, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2014). https://doi.org/10.1109/CVPR.2014.233
[31]	A. T. Lopes, E. Aguiar, A. F. D. Souza, T. O. Santos, Facial expression recognition with convolutional neural networks: coping with few data and the training sample order, Pattern Recognit., 61 (2017), 610–628. https://doi.org/10.1016/j.patcog.2016.07.026 doi: 10.1016/j.patcog.2016.07.026
[32]	S. Rajan, P. Chenniappan, S. Devaraj, N. Madian, Novel deep learning model for facial expression recognition based on maximum boosted CNN and LSTM, IET Image Process., 14 (2020), 1373–1381. https://doi.org/10.1049/iet-ipr.2019.1188 doi: 10.1049/iet-ipr.2019.1188
[33]	M. A. Hossain, G. Sanyal, A new improved tactic to extract facial expression based on genetic algorithm and WVDF, Int. J. Adv. Inf. Technol., 2 (2012), 37. https://doi.org/10.5121/ijait.2012.2504 doi: 10.5121/ijait.2012.2504
[34]	M. A. Hossain, D. Samanta, G. Sanyal, A novel approach for panic-face extraction based on mutation, in 2012 IEEE International Conference on Advanced Communication Control and Computing Technologies (ICACCCT), (2012), 473–477. https://doi.org/10.1109/ICACCCT.2012.6320825
[35]	J. Lee, S. Kim, S. Kim, K. Sohn, Multi-Modal recurrent attention networks for facial expression recognition, IEEE Trans. Image Process., 29 (2020), 6977–6991. https://doi.org/10.1109/TIP.2020.2996086 doi: 10.1109/TIP.2020.2996086
[36]	Z. Kang, S. J. Landry, An eye movement analysis algorithm for a multielement target tracking task: Maximum transition-based agglomerative hierarchical clustering, IEEE Trans. Hmman Mach. Syst., 45 (2015), 13–24. https://doi.org/10.1109/THMS.2014.2363121 doi: 10.1109/THMS.2014.2363121
[37]	M. A. Hossain, D. Samanta, G. Sanyal, Statistical approach for extraction of panic expression, in 2012 Fourth International Conference on Computational Intelligence and Communication Networks, (2012), 420–424. https://doi.org/10.1109/CICN.2012.189
[38]	R. Janarthanan, E. A. Refaee, K. Selvakumar, M. A. Hossain, S. Rajkumar, K. Marimuthu, Biomedical image retrieval using adaptive neuro-fuzzy optimized classifier system, Math. Biosci. Eng., 19 (2022), 8132–8151. https://doi.org/10.3934/mbe.2022380 doi: 10.3934/mbe.2022380
[39]	F. Bu, T. Pu, W. Huang, L. Zhu, Performance and evaluation of five-phase dual random SVPWM strategy with optimized probability density function, IEEE Trans. Ind. Electron., 66 (2019), 3323–3332. https://doi.org/10.1109/TIE.2018.2854570 doi: 10.1109/TIE.2018.2854570
[40]	B. Manda, P. Bhaskare, R. Muthuganapathy, A convolutional neural network approach to the classification of engineering models, IEEE Access, 9 (2021), 22711–22723. https://doi.org/10.1109/ACCESS.2021.3055826 doi: 10.1109/ACCESS.2021.3055826
[41]	M. A. Hossain, G. Sanyal, A stochastic statistical approach for tracking human activity, IJITMC, 1 (2013), 33–42. https://doi.org/10.5121/ijitmc.2013.1304 doi: 10.5121/ijitmc.2013.1304
[42]	A. J. A. AlBdairi, Z. Xiao, M. Alghaili, Identifying ethnics of people through face recognition: A deep CNN approach, Sci. Prog., 2020 (2020), 6385281. https://doi.org/10.1155/2020/6385281 doi: 10.1155/2020/6385281
[43]	N. Alay, H. H. Al-Baity, Deep learning approach for multimodal biometric recognition system based on fusion of iris, face, and finger vein traits, Sensor, 20 (2020), 5523–5539. https://doi.org/10.3390/s20195523 doi: 10.3390/s20195523

Reader Comments

Your name:*

Email:*
© 2023 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)