A sophisticated Drowsiness Detection System via Deep Transfer Learning for real time scenarios

Amina Turki; Omar Kahouli; Saleh Albadran; Mohamed Ksantini; Ali Aloui; Mouldi Ben Amara; Amina Turki; Omar Kahouli; Saleh Albadran; Mohamed Ksantini; Ali Aloui; Mouldi Ben Amara

doi:10.3934/math.2024156

AIMS Mathematics

2024, Volume 9, Issue 2: 3211-3234. doi: 10.3934/math.2024156

Previous Article Next Article

Research article Special Issues

A sophisticated Drowsiness Detection System via Deep Transfer Learning for real time scenarios

1.
Control & Energies Management Laboratory, National Engineering School of Sfax, University of Sfax, Tunisia
2.
Department of Electronics Engineering, Community College, University of Ha'il, 2440, Ha'il, Saudi Arabia
3.
Department of Electrical Engineering, College of Engineering, University of Ha'il, Ha'il 2440, Saudi Arabia

Received: 05 September 2023 Revised: 14 November 2023 Accepted: 20 November 2023 Published: 03 January 2024
MSC : 00A06, 68T01, 68T07

Driver drowsiness is one of the leading causes of road accidents resulting in serious physical injuries, fatalities, and substantial economic losses. A sophisticated Driver Drowsiness Detection (DDD) system can alert the driver in case of abnormal behavior and avoid catastrophes. Several studies have already addressed driver drowsiness through behavioral measures and facial features. In this paper, we propose a hybrid real-time DDD system based on the Eyes Closure Ratio and Mouth Opening Ratio using simple camera and deep learning techniques. This system seeks to model the driver's behavior in order to alert him/her in case of drowsiness states to avoid potential accidents. The main contribution of the proposed approach is to build a reliable system able to avoid false detected drowsiness situations and to alert only the real ones. To this end, our research procedure is divided into two processes. The offline process performs a classification module using pretrained Convolutional Neural Networks (CNNs) to detect the drowsiness of the driver. In the online process, we calculate the percentage of the eyes' closure and yawning frequency of the driver online from real-time video using the Chebyshev distance instead of the classic Euclidean distance. The accurate drowsiness state of the driver is evaluated with the aid of the pretrained CNNs based on an ensemble learning paradigm. In order to improve models' performances, we applied data augmentation techniques for the generated dataset. The accuracies achieved are 97 % for the VGG16 model, 96% for VGG19 model and 98% for ResNet50 model. This system can assess the driver's dynamics with a precision rate of 98%.
- Driver Drowsiness Detection (DDD) system,
- deep transfer learning,
- Convolutional Neural Networks (CNNs),
- the Chebyshev distance
Citation: Amina Turki, Omar Kahouli, Saleh Albadran, Mohamed Ksantini, Ali Aloui, Mouldi Ben Amara. A sophisticated Drowsiness Detection System via Deep Transfer Learning for real time scenarios[J]. AIMS Mathematics, 2024, 9(2): 3211-3234. doi: 10.3934/math.2024156

Related Papers:

Abstract

Driver drowsiness is one of the leading causes of road accidents resulting in serious physical injuries, fatalities, and substantial economic losses. A sophisticated Driver Drowsiness Detection (DDD) system can alert the driver in case of abnormal behavior and avoid catastrophes. Several studies have already addressed driver drowsiness through behavioral measures and facial features. In this paper, we propose a hybrid real-time DDD system based on the Eyes Closure Ratio and Mouth Opening Ratio using simple camera and deep learning techniques. This system seeks to model the driver's behavior in order to alert him/her in case of drowsiness states to avoid potential accidents. The main contribution of the proposed approach is to build a reliable system able to avoid false detected drowsiness situations and to alert only the real ones. To this end, our research procedure is divided into two processes. The offline process performs a classification module using pretrained Convolutional Neural Networks (CNNs) to detect the drowsiness of the driver. In the online process, we calculate the percentage of the eyes' closure and yawning frequency of the driver online from real-time video using the Chebyshev distance instead of the classic Euclidean distance. The accurate drowsiness state of the driver is evaluated with the aid of the pretrained CNNs based on an ensemble learning paradigm. In order to improve models' performances, we applied data augmentation techniques for the generated dataset. The accuracies achieved are 97 % for the VGG16 model, 96% for VGG19 model and 98% for ResNet50 model. This system can assess the driver's dynamics with a precision rate of 98%.

References

[1]	World Health Organization, Road traffic injuries, 2022. Available from: https://www.who.int/news-room/fact-sheets/detail/road-traffic-injuries.
[2]	S. Singh, Critical reasons for crashes investigated in the national motor vehicle crash causation survey, National Highway Traffic Safety Administration (NHTSA), 2015.
[3]	M. M. R. Komol, M. M. Hasan, M. Elhenawy, S. Yasmin, M. Masoud, A. Rakotonirainy, Crash severity analysis of vulnerable road users using machine learning, PLoS ONE, 16 (2021), e0255828. https://doi.org/10.1371/journal.pone.0255828 doi: 10.1371/journal.pone.0255828
[4]	National Safety Council, Drivers are falling asleep behind the wheel. 2022. Available from: https://www.nsc.org/road/safety-topics/fatigued-driver?
[5]	J. M. Owens, T. A. Dingus, F. Guo, Y. Fang, M. Perez, J. McClafferty, et al., Prevalence of drowsy driving crashes: Estimates from a large-scale naturalistic driving study, AAA foundation for traffic safety, Washington, 2018.
[6]	Euro NCAP 2025 Roadmap. Available from: https://cdn.euroncap.com/media/30700/euroncap-roadmap-2025-v4.pdf.
[7]	Official Journal of the European Union, Document 32019R2144—Regulation (EU) 2019/2144 of the European Parliament and of the Council. 2019. Available from: https://eur-lex.europa.eu/eli/reg/2019/2144/oj.
[8]	M. Ramzan, H. U. Khan, S. M Awan. A. Ismail, M. Ilyas, A. Mahmood, A survey on state-of-the-art drowsiness detection techniques, IEEE Access, 7 (2019), 61904–61919. https://doi.org/10.1109/ACCESS.2019.2914373 doi: 10.1109/ACCESS.2019.2914373
[9]	C. N. Watling, M. M. Hasan, G. S. Larue, Sensitivity and specificity of the driver sleepiness detection methods using physiological signals: A systematic review, Accident Anal. Prev., 150 (2021), 105900. https://doi.org/10.1016/j.aap.2020.105900 doi: 10.1016/j.aap.2020.105900
[10]	M. M. Hasan, C. N. Watling, G. S. Larue, Physiological signal-based drowsiness detection using machine learning: Singular and hybrid signal approaches, J. Safety Res., 80 (2022), 215–225. https://doi.org/10.1016/j.jsr.2021.12.001 doi: 10.1016/j.jsr.2021.12.001
[11]	C. C. Liu, S. G. Hosking, M. G. Lenné, Predicting driver drowsiness using vehicle measures: Recent insights and future challenges, J. Safety Res., 40 (2009), 239–245. https://doi.org/10.1016/j.jsr.2009.04.005 doi: 10.1016/j.jsr.2009.04.005
[12]	P. M. Forsman, B. J. Vila, R. A. Short, C. G. Mott, H. P. A. Van Dongen, Efficient driver drowsiness detection at moderate levels of drowsiness, Accident Anal. Prev., 50 (2013), 341–350. https://doi.org/10.1016/j.aap.2012.05.005 doi: 10.1016/j.aap.2012.05.005
[13]	X. Zhang, X. Wang, X. Yang, C. Xu, X. Zhu, J. Wei, Driver drowsiness detection using mixed-effect ordered logit model considering time cumulative effect, Anal. Methods Accid. Res., 26 (2020), 100114. https://doi.org/10.1016/j.amar.2020.100114 doi: 10.1016/j.amar.2020.100114
[14]	E. Ouabida, A. Essadike, A. Bouzid, Optical correlator based algorithm for driver drowsiness detection, Optik, 204 (2020), 164102. https://doi.org/10.1016/j.ijleo.2019.164102 doi: 10.1016/j.ijleo.2019.164102
[15]	Y. Sun, P. Yan, Z. Li, J. Zou, D. Hong, Driver fatigue detection system based on colored and infrared eye features fusion, Comput. Mater. Con., 63 (2020), 1563–1574. https://doi.org/10.32604/cmc.2020.09763 doi: 10.32604/cmc.2020.09763
[16]	M. K. Kamti, R. Iqbal, Evolution of driver fatigue detection techniques—A review from 2007 to 2021, Transport. Res. Rec., 2676 (2022), 485–507. https://doi.org/10.1177/03611981221096118 doi: 10.1177/03611981221096118
[17]	Y. Albadawi, M. Takruri, M. Awad, A review of recent developments in driver drowsiness detection systems, Sensors, 22 (2022), 2069. https://doi.org/10.3390/s22052069 doi: 10.3390/s22052069
[18]	A. A. Bamidele, K. Kamardin, N. S. N. A. Aziz, S. M. Sam, I. S. Ahmed, A. Azizan, et al., Non-intrusive driver drowsiness detection based on face and eye tracking. Int. J. Adv. Comput. Sci. Appl., 10 (2019), 549–569. https://doi.org/10.14569/IJACSA.2019.0100775 doi: 10.14569/IJACSA.2019.0100775
[19]	S. T. Lin, Y. Y. Tan, P. Y. Chua, L. K. Tey, C. H. Ang, Perclos threshold for drowsiness detection during real driving. J. Vision, 12 (2012), 546. https://doi.org/10.1167/12.9.546 doi: 10.1167/12.9.546
[20]	A. Rosebrock, Eyeblink detection with OpenCV, Python, and Dlib, PyImageSearch, 2017. Available from: https://pyimagesearch.com/2017/04/24/eye-blink-detection-opencv-python-dlib/.
[21]	V. Pradeep, Namratha, T. Nisha, Shravya, M. Vshker, A review on eye aspect ratio technique, IJARSCT, 3 (2023), 98–100. https://doi.org/10.48175/IJARSCT-7843 doi: 10.48175/IJARSCT-7843
[22]	R. C. Chen, C. W. Chang, C. Dewi, Determining the driver's mental state using detecting the eyes, 2022 IET International Conference on Engineering Technologies and Applications (IET-ICETA), 2022. https://doi.org/10.1109/IET-ICETA56553.2022.9971619 doi: 10.1109/IET-ICETA56553.2022.9971619
[23]	S. Thiha, J. Rajasekera, Efficient online engagement analytics algorithm toolkit that can run on edge, Algorithms, 16 (2023), 86. https://doi.org/10.3390/a16020086 doi: 10.3390/a16020086
[24]	A. Moujahid, F. Dornaika, I. Arganda-Carreras, J. Reta, Efficient and compact face descriptor for driver drowsiness detection, Expert Syst. Appl., 168 (2021), 114334. https://doi.org/10.1016/j.eswa.2020.114334 doi: 10.1016/j.eswa.2020.114334
[25]	A. C. Huang, C. Yuan, S. H. Meng, T. J. Huang, Design of fatigue driving behavior detection based on circle hough transform, Big Data, 11 (2023), 1–17. https://doi.org/10.1089/big.2021.0166 doi: 10.1089/big.2021.0166
[26]	M. T. Khan, H. Anwar, F. Ullah, A. Ur Rehman, R. Ullah, A. Iqbal, et al., Smart real-time video surveillance platform for drowsiness detection based on eyelid closure. Wirel. Commun. Mob. Com., 2019 (2019), 2036818. https://doi.org/10.1155/2019/2036818 doi: 10.1155/2019/2036818
[27]	C. B. S. Maior, M. J. das Chagas Moura, J. M. M. Santana, I. D. Lins, Real-time classification for autonomous drowsiness detection using eye aspect ratio, Expert Syst. Appl., 158 (2020), 113505. https://doi.org/10.1016/j.eswa.2020.113505 doi: 10.1016/j.eswa.2020.113505
[28]	A. S. Zandi, A. Quddus, L. Prest, F. J. E. Comeau, Non-intrusive detection of drowsy driving based on eye tracking data, Transport. Res. Rec., 2673 (2019), 247–257. https://doi.org/10.1177/0361198119847985 doi: 10.1177/0361198119847985
[29]	M. Hashemi, A. Mirrashid, A. B. Shirazi, Driver safety development: Real-time driver drowsiness detection system based on convolutional neural network, SN Comput. Sci., 1 (2020), 289. https://doi.org/10.1007/s42979-020-00306-9 doi: 10.1007/s42979-020-00306-9
[30]	N. Alioua, A. Amine, M. Rziza, Driver's fatigue detection based on yawning extraction, Int. J. Veh. Technol., 2014 (2014), 678786. https://doi.org/10.1155/2014/678786 doi: 10.1155/2014/678786
[31]	X. Ma, L. P. Chau, K. H. Yap, Depth video-based two-stream convolutional neural networks for driver fatigue detection, 2017 International Conference on Orange Technologies (ICOT), 2017. https://doi.org/10.1109/ICOT.2017.8336111 doi: 10.1109/ICOT.2017.8336111
[32]	B. K. Savasx, Y. Becerikli, Real time driver fatigue detection system based on multi-task ConNN, IEEE Access, 8 (2020), 12491–12498. https://doi.org/10.1109/access.2020.2963960 doi: 10.1109/access.2020.2963960
[33]	A. Celecia, K. Figueiredo, M. Vellasco, R. González, A portable fuzzy driver drowsiness estimation system, Sensors, 20 (2020), 4093. https://doi.org/10.3390/s20154093 doi: 10.3390/s20154093
[34]	N. Alioua, A. Amine, M. Rziza, D. Aboutajdine, Driver's fatigue and drowsiness detection to reduce traffic accidents on road, In: Computer analysis of images and patterns, Berlin, Heidelberg: Springer, 2011. https://doi.org/10.1007/978-3-642-23678-5_47
[35]	L. Alzubaidi, J. Zhang, A. J. Humaidi, A. Al-Dujaili, Y. Duan, O. Al-Shamma, et al. Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions, J Big Data, 8 (2021), 53. https://doi.org/10.1186/s40537-021-00444-8 doi: 10.1186/s40537-021-00444-8
[36]	Z. Li, F. Liu, W. Yang, S. Peng, J. Zhou, A survey of convolutional neural networks: Analysis, applications, and prospects, IEEE T. Neur. Net. Lear. Syst., 33 (2022), 6999–7019. https://doi.org/10.1109/TNNLS.2021.3084827 doi: 10.1109/TNNLS.2021.3084827
[37]	F. Chollet, Transfer learning & fine-tuning, 2020. Available from: https://keras.io/guides/transfer_learning/
[38]	E. Magán, M. P. Sesmero, J. M. Alonso-Weber, A. Sanchis, Driver drowsiness detection by applying deep learning techniques to sequences of images, Appl. Sci., 12 (2022), 1145. https://doi.org/10.3390/app12031145 doi: 10.3390/app12031145
[39]	N. Ho, Y. C. Kim, Evaluation of transfer learning in deep convolutional neural network models for cardiac short axis slice classification, Sci Rep., 11 (2021), 1839. https://doi.org/10.1038/s41598-021-81525-9 doi: 10.1038/s41598-021-81525-9
[40]	A. Kensert, P. J. Harrison, O. Spjuth, Transfer learning with deep convolutional neural networks for classifying cellular morphological changes, SLAS Discov., 24 (2019), 466–475. https://doi.org/10.1177/2472555218818756 doi: 10.1177/2472555218818756
[41]	A. Aytekin, V. Mençik, Detection of driver dynamics with VGG16 model, Appl. Comput. Syst., 27 (2022), 83–88. https://doi.org/10.2478/acss-2022-0009 doi: 10.2478/acss-2022-0009
[42]	M. Dua, Shakshi, R. Singla, S. Raj, A. Jangra, Deep CNN models-based ensemble approach to driver drowsiness detection, Neural Comput. Applic., 33 (2021), 3155–3168. https://doi.org/10.1007/s00521-020-05209-7 doi: 10.1007/s00521-020-05209-7
[43]	J. Yu, S. Park, S. Lee, M. Jeon, Driver drowsiness detection using condition-adaptive representation learning framework, IEEE T. Intell. Transp. Syst., 20 (2018), 4206–4218. https://doi.org/10.1109/TITS.2018.2883823 doi: 10.1109/TITS.2018.2883823
[44]	P. Sanghyuk, P. Fei, S. Kang, C. D. Yoo, Driver drowsiness detection system based on feature representation learning using various deep networks. In: Computer Vision–ACCV 2016 Workshops. Springer, Cham, 2017. https://doi.org/10.1007/978-3-319-54526-4_12
[45]	Q. Abbas, HybridFatigue: A real-time driver drowsiness detection using hybrid features and transfer learning, Int. J. Adv. Comput. Sci. Appl., 11 (2020), 585–593. https://doi.org/10.14569/IJACSA.2020.0110173 doi: 10.14569/IJACSA.2020.0110173
[46]	D. Lee, Which deep learning model can best explain object representations of within-category exemplars? J Vision, 21 (2021), 12. https://doi.org/10.1167/jov.21.10.12 doi: 10.1167/jov.21.10.12
[47]	A. Shabnam, M. Omidyeganeh, S. Shirmohammadi, B. Hariri, YawDD: A yawning detection dataset, MMSys '14: Proceedings of the 5th ACM Multimedia Systems Conference, 2014, 24–28. https://doi.org/10.1145/2557642.2563678 doi: 10.1145/2557642.2563678
[48]	K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition. Proceedings of the 3rd International Conference on Learning Representations (ICLR).
[49]	K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. https://doi.org/10.1109/CVPR.2016.90 doi: 10.1109/CVPR.2016.90
[50]	Dlib C++ toolkit, Available from: http://dlib.net/.
[51]	P. Viola, M. Jones, Rapid object detection using a boosted cascade of simple features, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, 2001, https://doi.org/10.1109/CVPR.2001.990517 doi: 10.1109/CVPR.2001.990517
[52]	V. Kazemi, J. Sullivan, One millisecond face alignment with an ensemble of regression trees, 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014. https://doi.org/10.1109/CVPR.2014.241 doi: 10.1109/CVPR.2014.241
[53]	R. Potolea, S. Cacoveanu, C. Lemnaru, Meta-learning framework for prediction strategy evaluation, International Conference on Enterprise Information Systems, Berlin, Heidelberg: Springer, 2011. https://doi.org/10.1007/978-3-642-19802-1_20
[54]	R. Dillmann, J. Beyerer, J.D. Hanebeck, T. Schultz, Advances in artificial intelligence, Proceedings of the Springer 33rd Annual German Conference on AI. Karlsruhe, 2010.
[55]	C. Dewi, R. C. Chen, X. Jiang, H. Yu, Adjusting eye aspect ratio for strong eye blink detection based on facial landmarks, PeerJ Comput. Sci., 8 (2022), e943. https://doi.org/10.7717/peerj-cs.943 doi: 10.7717/peerj-cs.943
[56]	C. Dewi, R. C. Chen, C. W. Chang, S. H. Wu, X. Jiang, H. Yu, Eye aspect ratio for real-time drowsiness detection to improve driver safety, Electronics, 11 (2022), 3183. https://doi.org/10.3390/electronics11193183 doi: 10.3390/electronics11193183
[57]	N. Kadri, A. Ellouze, M. Ksantini, S. H. Turki, New LSTM deep learning algorithm for driving behavior classification, Cybernet. Syst., 54 (2023), 387–405. https://doi.org/10.1080/01969722.2022.2059133 doi: 10.1080/01969722.2022.2059133
[58]	F. You, Y. Gong, H. Tu, J. Liang, H. Wang, A fatigue driving detection algorithm based on facial motion information entropy, J. Adv. Transport, 2020 (2020), 8851485. https://doi.org/10.1155/2020/8851485 doi: 10.1155/2020/8851485
[59]	W. Deng, R. Wu, Real-time driver-drowsiness detection system using facial features, IEEE Access, 7 (2019), 118727–118738. https://doi.org/10.1109/ACCESS.2019.2936663 doi: 10.1109/ACCESS.2019.2936663
[60]	J. Bai, W. Yu, Z. Xiao, V. Havyarimana, A. C. Regan, H. Jiang, et al., Two-stream spatial-temporal graph convolutional networks for driver drowsiness detection, IEEE T. Cybernetics, 52 (2022), 13821–13833. https://doi.org/10.1109/TCYB.2021.3110813 doi: 10.1109/TCYB.2021.3110813
[61]	F. Liu, D. Chen, J. Zhou, F. Xu, A review of driver fatigue detection and its advances on the use of RGB-D camera and deep learning, Eng. Appl. Artif. Intel., 116 (2022), 105399. https://doi.org/10.1016/j.engappai.2022.105399 doi: 10.1016/j.engappai.2022.105399
[62]	N. Triki, M. Karray, M. Ksantini, A real-time traffic sign recognition method using a new attention-based deep convolutional neural network for smart vehicles, Appl. Sci., 13 (2023), 4793. https://doi.org/10.3390/app13084793 doi: 10.3390/app13084793

Reader Comments

Your name:*

Email:*
© 2024 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)