R2DS: A novel hierarchical framework for driver fatigue detection in mountain freeway

Feng You; Yunbo Gong; Xiaolong Li; Haiwei Wang; Feng You; Yunbo Gong; Xiaolong Li; Haiwei Wang

doi:10.3934/mbe.2020190

Mathematical Biosciences and Engineering

2020, Volume 17, Issue 4: 3356-3381. doi: 10.3934/mbe.2020190

Previous Article Next Article

Research article Special Issues

R2DS: A novel hierarchical framework for driver fatigue detection in mountain freeway

1.
School of Civil Engineering and Transportation, South China University of Technology, Guangzhou 510640, China
2.
School of transportation and economic management, Guangdong Communication Polytechnic, Guangzhou 510650, China

Received: 26 February 2020 Accepted: 20 April 2020 Published: 29 April 2020

Fatigue driving is one of the main factors which affect the safety of drivers and passengers in mountain freeway. To improve the driving safety, the application of fatigue driving detection system is a crucial measure. Accuracy, speed and robustness are key performances of fatigue detection system. However, most researches pay attention to one of them, instead of taking care of them all. It has limitation in practical application. This paper proposes a novel three-layered framework, named Real-time and Robust Detection System. Specifically, the framework includes three modules, called facial feature extraction, eyes regions extraction and fatigue detection. In the facial feature extraction module, the paper designs a deep cascaded convolutional neural network to detect the face and locate eye key points. Then, a face tracking sub-module is constructed to increase the speed of the algorithm, and a face validation submodule is applied to improve the stability of detection. Furthermore, to ensure the orderly operation of each sub-module, we designed a recognition loop based on the finite state machine. It can extract facial feature of the driver. In the second module, eyes regions of the driver were captured according to the geometric feature of face and eyes. In the fatigue detection module, the ellipse fitting method is applied to obtain the shape of driver's pupils. According to the relationship between the long and short axes of the ellipse, eyes state (opening or closed) can be decided. Lastly, the PERCLOS, which is defined by calculating the number of closed eyes in a period, is used to determine whether fatigue driving or not. The experimental results show that the comprehensive accuracy of fatigue detection is 95.87%. The average algorithm rate is 32.29 ms/f in an image of 640×480 pixels. The research results can serve the design of a new generation of driver fatigue detection system to mountain freeway.
- fatigue detection,
- mountain freeway,
- DCCNN,
- finite state machine,
- PERCLOS
Citation: Feng You, Yunbo Gong, Xiaolong Li, Haiwei Wang. R2DS: A novel hierarchical framework for driver fatigue detection in mountain freeway[J]. Mathematical Biosciences and Engineering, 2020, 17(4): 3356-3381. doi: 10.3934/mbe.2020190

Related Papers:

Abstract

Fatigue driving is one of the main factors which affect the safety of drivers and passengers in mountain freeway. To improve the driving safety, the application of fatigue driving detection system is a crucial measure. Accuracy, speed and robustness are key performances of fatigue detection system. However, most researches pay attention to one of them, instead of taking care of them all. It has limitation in practical application. This paper proposes a novel three-layered framework, named Real-time and Robust Detection System. Specifically, the framework includes three modules, called facial feature extraction, eyes regions extraction and fatigue detection. In the facial feature extraction module, the paper designs a deep cascaded convolutional neural network to detect the face and locate eye key points. Then, a face tracking sub-module is constructed to increase the speed of the algorithm, and a face validation submodule is applied to improve the stability of detection. Furthermore, to ensure the orderly operation of each sub-module, we designed a recognition loop based on the finite state machine. It can extract facial feature of the driver. In the second module, eyes regions of the driver were captured according to the geometric feature of face and eyes. In the fatigue detection module, the ellipse fitting method is applied to obtain the shape of driver's pupils. According to the relationship between the long and short axes of the ellipse, eyes state (opening or closed) can be decided. Lastly, the PERCLOS, which is defined by calculating the number of closed eyes in a period, is used to determine whether fatigue driving or not. The experimental results show that the comprehensive accuracy of fatigue detection is 95.87%. The average algorithm rate is 32.29 ms/f in an image of 640×480 pixels. The research results can serve the design of a new generation of driver fatigue detection system to mountain freeway.

References

[1]	A. Amodio, M. Ermidoro, D. Maggi, S. Formentin S. M. Savaresi, Automatic Detection of Driver Impairment Based on Pupillary Light Reflex, IEEE Trans. Intell. Transp. Syst., 20 (2019), 3038-3048. doi: 10.1109/TITS.2018.2871262
[2]	X. Li, X. Lian, F. Liu, Rear-End Road Crash Characteristics Analysis Based on Chinese In-Depth Crash Study Data, 16th Cota International Conference of Transportation Professionals, 2016, 1536-1545. Available from: https://ascelibrary.org/doi/abs/10.1061/9780784479896.140.
[3]	G. Zhang, K. K. W. Yau, X. Zhang, Y. Li, Traffic accidents involving fatigue driving and their extent of casualties, Accid. Anal. Prev., 87 (2016), 34-42. doi: 10.1016/j.aap.2015.10.033
[4]	D. Mollicone, K. Kan, C. Mott, R. Bartels, S. Bruneau, M. Wollen, et al., Predicting performance and safety based on driver fatigue, Accid. Anal. Prev., 126 (2019), 142-145. doi: 10.1016/j.aap.2018.03.004
[5]	P. Chen, Research on driver fatigue detection strategy based on human eye state, 2017 Chinese Automation Congress (CAC), 2017,619-623. Available from: https://ieeexplore.ieee.org/abstract/document/8242842.
[6]	G. Sikander, S. Anwar, Driver Fatigue Detection Systems: A Review, IEEE Trans. Intell. Transp. Syst., 20 (2019), 2339-2352. doi: 10.1109/TITS.2018.2868499
[7]	A. Anund, C. Fors, C. Ahlstrom, The severity of driver fatigue in terms of line crossing: A pilot study comparing day- and night time driving in simulator, Eur. Transp. Res. Rev., 9 (2017), 31. doi: 10.1007/s12544-017-0248-6
[8]	Y. Chang, Y. Feng, O. T. Chen, Real-time physiological and facial monitoring for safe driving, Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2016, 4849-4852. Available from: https://ieeexplore.ieee.org/abstract/document/7591813.
[9]	Z. Zhou, Y. Zhou, Z. Pu, Y. Xu, Simulation of pedestrian behavior during the flashing green signal using a modified social force model, Transportmetrica A Transp. Sci., 15 (2019), 1019-1040. doi: 10.1080/23249935.2018.1559895
[10]	Z. You, Y. Gao, J. Zhang, H. Zhang, M. Zhou, C. Wu, A study on driver fatigue recognition based on SVM method, International Conference on Transportation Information & Safety, 2017,693-697. Available from: https://ieeexplore.ieee.org/abstract/document/8047842.
[11]	J. Hu, Automated Detection of Driver Fatigue Based on AdaBoost Classifier with EEG Signals, Front. Comput. Neurosci., 11 (2017), 72. doi: 10.3389/fncom.2017.00072
[12]	W. Kong, W. Lin, F. Babiloni, S. Hu, G. Borghini, Investigating Driver Fatigue versus Alertness Using the Granger Causality Network, Sensors, 15 (2015), 19181-19198. doi: 10.3390/s150819181
[13]	Z. Zhou, Y. Cai, R. Ke, J. Yang, A collision avoidance model for two-pedestrian groups: Considering random avoidance patterns, Phys. A, 475 (2017), 142-154. doi: 10.1016/j.physa.2016.12.041
[14]	D. Ma, X. Luo, S. Jin, D. Wang, W. Guo, F. Wang, Lane-based saturation degree estimation for signalized intersections using travel time data, IEEE Intell. Transp. Syst. Mag., 9 (2017), 136-148.
[15]	D. M. Morris, J. J. Pilcher, F. S. Switzer Ⅲ, Lane heading difference: An innovative model for drowsy driving detection using retrospective analysis around curves, Accid. Anal. Prev., 80 (2015), 117-124. doi: 10.1016/j.aap.2015.04.007
[16]	Z. Li, S. Li, R. Li, B. Cheng, J. Shi, Online Detection of Driver Fatigue Using Steering Wheel Angles for Real Driving Conditions, Sensors, 17 (2017), 495. doi: 10.3390/s17030495
[17]	A. D. McDonald, C. Schwarz, J. D. Lee, T. L. Brown, Real-Time Detection of Drowsiness Related Lane Departures Using Steering Wheel Angle, Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 56 (2016), 2201-2205. Available from: https://journals.sagepub.com/doi/abs/10.1177/1071181312561464.
[18]	D. Ma, X. Luo, W. Li, S. Jin, W. Guo, D. Wang, Traffic demand estimation for lane groups at signal-controlled intersections using travel times from video-imaging detectors, IET Intell. Transp. Syst., 11 (2017), 222-229. doi: 10.1049/iet-its.2016.0233
[19]	D. Ma, X. Luo, S. Jin, W. Guo, D. Wang, Estimating maximum queue length for traffic lane groups using travel times from video-imaging data, IEEE Intell. Transp. Syst. Mag., 10 (2018), 123-134. doi: 10.1109/MITS.2018.2842047
[20]	B. Mandal, L. Li, G. S. Wang, J. Lin, Towards Detection of Bus Driver Fatigue Based on Robust Visual Analysis of Eye State, IEEE Trans. Intell. Transp., 18 (2017), 545-557. doi: 10.1109/TITS.2016.2582900
[21]	N. Alioua, A. Amine, M. Rziza, Driver's Fatigue Detection Based on Yawning Extraction, Int. J. Veh. Technol., 2014 (2014), 1-7.
[22]	K. J. Raman, A. Azman, V. Arumugam, S. Z. Ibrahim, S. Yogarayan, M. F. A. Abdullah, et al., Fatigue Monitoring Based on Yawning and Head Movement, International Conference on Information Communication Technology, 2018,343-347. Available from: https://ieeexplore.ieee.org/abstract/document/8528759/.
[23]	X. Sun, H. Zhang, W. Meng, R. Zhang, K. Li, T. Peng, Primary resonance analysis and vibration suppression for the harmonically excited nonlinear suspension system using a pair of symmetric viscoelastic buffers, Nonlinear Dyn., 94 (2018), 1243-1265. doi: 10.1007/s11071-018-4421-9
[24]	C. Xu, Y. Yang, S. Jin, Z. Qu, L. Hou, Potential risk and its influencing factors for separated bicycle paths, Accid. Anal. Prev., 87 (2016), 59-67. doi: 10.1016/j.aap.2015.11.014
[25]	J. J. Yan, H. H. Kuo, Y. F. Lin, T. L. Liao, Real-Time Driver Drowsiness Detection System Based on PERCLOS and Grayscale Image Processing, 2016 International Symposium on Computer, Consumer and Control (IS3C) IEEE, 2016,243-246. Available from: https://ieeexplore.ieee.org/abstract/document/7545182.
[26]	G. Niu, C. Wang, Driver Fatigue Features Extraction, Math. Probl. Eng., 2014 (2014), 860517.
[27]	J. Shen, G. Li, W. Yan, W. Tao, G. Xu, D. Diao, et al., Nighttime Driving Safety Improvement via Image Enhancement for Driver Face Detection, IEEE Access, 6 (2018), 45625-45634. doi: 10.1109/ACCESS.2018.2864629
[28]	A. Liu, Z. Li, L. Wang, Y. Zhao, A practical driver fatigue detection algorithm based on eye state, 2010 Asia Pacific Conference on Postgraduate Research in Microelectronics and Electronics (PrimeAsia), 2010,235-238. Available from: https://ieeexplore.ieee.org/abstract/document/5604919.
[29]	X. Ma, L. Chau, K. Yap, Depth video-based two-stream convolutional neural networks for driver fatigue detection, 2017 International Conference on Orange Technologies, 2017,155-158. Available from: https://ieeexplore.ieee.org/abstract/document/8336111.
[30]	W. Di, R. Wang, P. Ge, F. Cao, Driver eye feature extraction based on infrared illuminator, 2009 IEEE Intelligent Vehicles Symposium, 2009,330-334. Available from: https://ieeexplore.ieee.org/abstract/document/5164299.
[31]	B. Cyganek, S. Gruszczyński, Hybrid computer vision system for drivers' eye recognition and fatigue monitoring, Neurocomputing, 126 (2014), 78-94. doi: 10.1016/j.neucom.2013.01.048
[32]	F. You, Y. Li, L. Huang, K. Chen, R. Zhang, J. Xu, Monitoring drivers' sleepy status at night based on machine vision, Multimedia Tools Appl., 76 (2017), 14869-14886. doi: 10.1007/s11042-016-4103-x
[33]	L. Geng, X. Liang, Z. Xiao, Y. Li, Real-time driver fatigue detection based on polymorphic infrared features and depth learning, Infrared Laser Eng., 47 (2018), 69-77.
[34]	H. Xiong, X. Zhu, R. Zhang, Energy Recovery Strategy Numerical Simulation for Dual Axle Drive Pure Electric Vehicle Based on Motor Loss Model and Big Data Calculation, Complexity, 2018 (2018), 4071743.
[35]	L. M. Bergasa, J. Nuevo, M. A. Sotelo, R. Barea, M. E. Lopez, Real-Time System for Monitoring Driver Vigilance, IEEE Trans. Intell. Transp., 7 (2006), 63-77. doi: 10.1109/TITS.2006.869598
[36]	R. Gupta, K. Aman, N. Shiva, Y. Singh, An improved fatigue detection system based on behavioral characteristics of driver, 2017 2nd IEEE International Conference on Intelligent Transportation Engineering (ICITE), 2017,227-230. Available from: https://ieeexplore.ieee.org/abstract/document/8056914.
[37]	Z. Zheng, J. Yang, L. Yang, A robust method for eye features extraction on color image, Pattern Recognit. Lett., 26 (2005), 2252-2261. doi: 10.1016/j.patrec.2005.03.033
[38]	X. P. Zhao, C. M. Meng, M. K. Feng, S. J. Chang, Fatigue detection based on cascade convolutional neural network, J. Optoelectron.·Laser, 28 (2017), 497-502.
[39]	F. Zhang, J. Su, L. Geng, Z. Xiao, Driver Fatigue Detection Based on Eye State Recognition, International Conference on Machine Vision & Information Technology, 2017,105-100. Available from: https://ieeexplore.ieee.org/abstract/document/7878723.
[40]	K. Zhang, Z. Zhang, Z. Li, Y. Qiao, Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks, IEEE Signal Proc. Lett., 23 (2016), 1499-1503. doi: 10.1109/LSP.2016.2603342
[41]	X. Qu, Y. Yu, M. Zhou, C. T. Lin, X. Wang, Jointly Dampening Traffic Oscillations and Improving Energy Consumption with Electric, Connected and Automated Vehicles: A Reinforcement Learning Based Approach, Appl. Energy, 257 (2020), 114030. doi: 10.1016/j.apenergy.2019.114030
[42]	S. Qiu, G. Wen, Z. Deng, J. Liu, Y. Fan, Accurate non-maximum suppression for object detection in high-resolution remote sensing images, Remote Sens. Lett., 9 (2018), 237-246. doi: 10.1080/2150704X.2017.1415473
[43]	M. Zhou, Y. Yu, X. Qu, Development of an Efficient Driving Strategy for Connected and Automated Vehicles at Signalized Intersections: A Reinforcement Learning Approach, IEEE Trans. Intell. Transp. Syst., 21 (2020), 433-443. doi: 10.1109/TITS.2019.2942014
[44]	J. Tabor, P. Spurek, Cross-entropy clustering, Pattern Recognit., 47 (2014), 3046-3059. doi: 10.1016/j.patcog.2014.03.006
[45]	H. Lin, X. Tang, G. Ou, An Open Loop with Kalman Filter for Intermittent GNSS Signal Tracking, IEEE Commun. Lett., 21 (2017), 2634-2637.
[46]	Z. Y. Lu, D. M. Wang, J. Wang, Y. Wang, A tracking algorithm based on orthogonal cubature Kalman filter with TDOA and FDOA, Acta Phys. Sin., 64 (2015), 150502.
[47]	K.Y. Liu, Y. Li, S. Li, L. Tang, L. Wang, A new parallel particle filter face tracking method based on heterogeneous system, J. Real Time Image Process., 7 (2012), 153-163. doi: 10.1007/s11554-011-0225-6
[48]	W. Pairo, P. Loncomilla, J. R. Del Solar, A Delay-Free and Robust Object Tracking Approach for Robotics Applications, J. Intell. Robot. Syst., 95 (2018), 99-117.
[49]	K. Chen, C. L. Liu, Y. Xu, Face Detection and Tracking Based on Adaboost CamShift and Kalman Filter Algorithm, Communications in Computer and Information Science, 2014. Available from: https://link.springer.com/chapter/10.1007/978-3-662-45261-5_16.
[50]	J. Lee, H. Jung, J. Yoo, A Real-time Face Tracking Algorithm using Improved CamShift with Depth Information, J. Electr. Eng. Technol., 12 (2017), 2067-2078.
[51]	R. Zhang, Z. He, H. Wang, F. You, K. Li, Study on self-tuning tyre friction control for developing main-servo loop integrated chassis control system, IEEE Access, 5 (2017), 6649-6660. doi: 10.1109/ACCESS.2017.2669263
[52]	S. Wei, S. Lai, Fast Template Matching Based on Normalized Cross Correlation with Adaptive Multilevel Winner Update, IEEE Transp. Image Process., 17 (2018), 2227-2235.
[53]	X. LI, N. Tan, T. Wang, S. Su, Detecting driver fatigue based on nonlinear speech processing and fuzzy SVM, International Conference on Signal Processing, 2014,510-515. Available from: https://ieeexplore.ieee.org/abstract/document/7015057.
[54]	F. J. R. Ruiz, I. Valera, L. Svensson F. Perez-Cruz, Infinite Factorial Finite State Machine for Blind Multiuser Channel Estimation, IEEE Trans. Cognit. Communi. Networking, 4 (2018), 177-191. doi: 10.1109/TCCN.2018.2790976
[55]	L. Lin, C. Huang, X. Ni, J. Wang, H. Zhang, X. Li, et al., Driver fatigue detection based on eye state, Technol. Health Care, 23 (2015), S453-S463. doi: 10.3233/THC-150982
[56]	S. Junaedi and H. Akbar, Driver Drowsiness Detection Based on Face Feature and PERCLOS, J. Phys., 1090 (2018), 12037.
[57]	P. Zheng, Z. Song, Y. Zhou, PERCLOS-Based Recognition Algorithms of Motor Driver Fatigue, J. China Agric. Univ., 7 (2002), 104-109.
[58]	V. Jain, E. Learned-Miller, FDDB: A benchmark for face detection in unconstrained settings, UMass Amherst Technical Report January, 2010. Available from: https://people.cs.umass.edu/~elm/papers/fddb.pdf.
[59]	J. Hernándezorallo, ROC curves for regression, Pattern Recognit., 46 (2013), 3395-3411. doi: 10.1016/j.patcog.2013.06.014
[60]	L. Tychsen-Smith, L. Petersson, Improving Object Localization with Fitness NMS and Bounded IoU Loss, IEEE Conference on Computer Vision and Pattern Recognition, 2017, 6877-6885. Available from: http://openaccess.thecvf.com/content_cvpr_2018/html/Tychsen-Smith_Improving_Object_Localization_CVPR_2018_paper.html.
[61]	R. S. Salcido, Diabetic Diagnostic Dilemmas: Sensitivity and Specificity, Adv. Skin Wound Care, 31 (2018), 149. doi: 10.1097/01.ASW.0000531227.22496.c0
[62]	C. Zhang, X. Lu, Z. Huang, S. Xia C. Fu, A Driver Fatigue Recognition Algorithm Based on Spatio-Temporal Feature Sequence, 2019 12th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), 2019, 1-6. Available from: https://ieeexplore.ieee.org/abstract/document/8965990.
[63]	Z. Xiao, Z. Hu, L. Geng, F. Zhang, J. Wu, Y. Li, Fatigue driving recognition network: Fatigue driving recognition via convolutional neural network and long short-term memory units, IET Intell. Transp. Syst., 13 (2019), 1410-1416. doi: 10.1049/iet-its.2018.5392

Reader Comments

Your name:*

Email:*
© 2020 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)