TKRD: Trusted kernel rootkit detection for cybersecurity of VMs based on machine learning and memory forensic analysis

Xiao Wang; Jianbiao Zhang; Ai Zhang; Jinchang Ren; Xiao Wang; Jianbiao Zhang; Ai Zhang; Jinchang Ren

doi:10.3934/mbe.2019132

Mathematical Biosciences and Engineering

2019, Volume 16, Issue 4: 2650-2667. doi: 10.3934/mbe.2019132

Previous Article Next Article

Research article Special Issues

TKRD: Trusted kernel rootkit detection for cybersecurity of VMs based on machine learning and memory forensic analysis

1.
Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
2.
Beijing Key Laboratory of Trusted Computing, Beijing 100124, China
3.
Department of Computer Science & Engineering, UC San Diego, CA, USA
4.
Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow, U.K.

Received: 05 December 2018 Accepted: 13 March 2019 Published: 26 March 2019

The promotion of cloud computing makes the virtual machine (VM) increasingly a target of malware attacks in cybersecurity such as those by kernel rootkits. Memory forensic, which observes the malicious tracks from the memory aspect, is a useful way for malware detection. In this paper, we propose a novel TKRD method to automatically detect kernel rootkits in VMs from private cloud, by combining VM memory forensic analysis with bio-inspired machine learning technology. Malicious features are extracted from the memory dumps of the VM through memory forensic analysis method. Based on these features, various machine learning classifiers are trained including Decision tree, Rule based classifiers, Bayesian and Support vector machines (SVM). The experiment results show that the Random Forest classifier has the best performance which can effectively detect unknown kernel rootkits with an Accuracy of 0.986 and an AUC value (the area under the receiver operating characteristic curve) of 0.998.
- virtual machine,
- private cloud,
- kernel rootkit detection,
- memory forensic,
- machine learning
Citation: Xiao Wang, Jianbiao Zhang, Ai Zhang, Jinchang Ren. TKRD: Trusted kernel rootkit detection for cybersecurity of VMs based on machine learning and memory forensic analysis[J]. Mathematical Biosciences and Engineering, 2019, 16(4): 2650-2667. doi: 10.3934/mbe.2019132

Related Papers:

Abstract

The promotion of cloud computing makes the virtual machine (VM) increasingly a target of malware attacks in cybersecurity such as those by kernel rootkits. Memory forensic, which observes the malicious tracks from the memory aspect, is a useful way for malware detection. In this paper, we propose a novel TKRD method to automatically detect kernel rootkits in VMs from private cloud, by combining VM memory forensic analysis with bio-inspired machine learning technology. Malicious features are extracted from the memory dumps of the VM through memory forensic analysis method. Based on these features, various machine learning classifiers are trained including Decision tree, Rule based classifiers, Bayesian and Support vector machines (SVM). The experiment results show that the Random Forest classifier has the best performance which can effectively detect unknown kernel rootkits with an Accuracy of 0.986 and an AUC value (the area under the receiver operating characteristic curve) of 0.998.

References

[1]	Y. Ding, H. M. Wang, P. C. Shi, et al., Trusted cloud service, Chin. J. Comput., 38 (2015), 133–149.
[2]	M. Ali, S. U. Khan and A. V. Vasilakos, Security in cloud computing: Opportunities and challenges, Inform. Sciences., 305 (2015), 357–383.
[3]	Y. Q. Zhang, X. F. Wang, X. F. Liu, et al., Survey on cloud computing security, J. Software, 27 (2016), 1328−1348.
[4]	J. Wilhelm and T. C. Chiueh, A forced sampled execution approach to kernel rootkit identification, In: International Workshop on Recent Advances in Intrusion Detection; 2007 Sept 5–7; Gold Goast, Australia. Berlin: Springer; 2007: 219–235.
[5]	N. Zhang, R. Zhang, K. Sun, et al., Memory Forensic Challenges Under Misused Architectural Features, IEEE T. Inf. Foren. Sec., 13 (2018), 2345–2358.
[6]	A. Cohen, N. Nissim, L. Rokach, et al., SFEM: Structural feature extraction methodology for the detection of malicious office documents using machine learning methods, Expert Syst. Appl., 63 (2016), 324–343.
[7]	N. Nissim, R. Moskovitch, O. BarAd, et al., ALDROID:Efficient update of Android anti-virus software using designated active learning methods, Knowl. Inf. Syst., 49 (2016), 795–833.
[8]	N. Nissim, A. Cohen, C. Glezer, et al., Detection of malicious PDF files and directions for enhancements: A state-of-the art survey, Comput. Secur., 48 (2015), 246–266.
[9]	G. Hoglund and J. Butler, Rootkits: subverting the Windows kernel, Addison-Wesley Professional, New Jersey, 2006.
[10]	A. Case and G. G. Richard III, Advancing Mac OS X rootkit detection, Digit. Invest., 14 (2015), S25–S33.
[11]	H. Yang, J. Zhuge, H. Liu, et al., A tool for volatile memory acquisition from Android devices, In: IFIP International Conference on Digital Forensics; 2016 Jan 4-6; New Delhi, India. Cham: Springer; 2016: 365–378.
[12]	A. Kumara and C. D. Jaidhar, Execution time measurement of virtual machine volatile artifacts analyzers, In: 2015 IEEE 21st International Conference on Parallel and Distributed Systems (ICPADS);2015 Dec14-17; Melbourne, VIC, Australia. IEEE; 2015: 314–319.
[13]	Q. Hua and Y. Zhang, Detecting Malware and Rootkit via Memory Forensics, In:2015 International Conference on Computer Science and Mechanical Automation (CSMA); 2015 Oct 23–25; Hangzhou, China. IEEE; 2015: 92–96.
[14]	C. W. Tien, J. W. Liao, S. C. Chang, et al., Memory forensics using virtual machine introspection for Malware analysis, In:2017 IEEE Conference on Dependable and Secure Computing; 2017 Aug 7-10; Taipei, Taiwan. IEEE; 2017: 518–519.
[15]	A. Cohen and N. Nissim, Trusted detection of ransomware in a private cloud using machine learning methods leveraging meta-features from volatile memory, Expert Syst. Appl., 102 (2018), 158–178.
[16]	N. Nissim, Y. Lapidot, A. Cohen, et al., Trusted system-calls analysis methodology aimed at detection of compromised virtual machines using sequential mining, Knowl.-Based Syst., 153 (2018), 147–175.
[17]	A. Kumara and C. D. Jaidhar, Automated multi-level malware detection system based on reconstructed semantic view of executables using machine learning techniques at VMM, Future Gener. Comp. Sy., 79 (2018), 431–446.
[18]	H. Upadhyay, H. A. Gohel, A. Pons, et al., Windows Virtualization Architecture For Cyber Threats Detection. In:2018 1st International Conference on Data Intelligence and Security (ICDIS). 2018 Apr 8-10; South Padre Island, TX, USA.IEEE; 2018: 119–122.
[19]	R. Mosli, R. Li, B. Yuan, et al., Automated malware detection using artifacts in forensic memory images. In:2016 IEEE Symposium on Technologies for Homeland Security (HST). 2016 May 10-11; Waltham, MA, USA. IEEE; 2016: 1–6.
[20]	M. A. Kumara and C. D. Jaidhar, Leveraging virtual machine introspection with memory forensics to detect and characterize unknown malware using machine learning techniques at hypervisor, Digit. Invest., 23 (2017), 99–123.
[21]	J. Bai and J. Wang, Improving malware detection using multi view ensemble learning, Secur. Commun. Netw., 9 (2016), 4227–4241.
[22]	OpenStack. Available from: https://docs.openstack.org/rocky/.
[23]	Volatility. Available from: https://www.volatilityfoundation.org/.
[24]	M. H. Ligh, A. Case, J. Levy, et al., The art of memory forensics: detecting malware and threats in windows, linux, and Mac memory, John Wiley & Sons, New Jersey, 2014.
[25]	Malshare. Available from: http://www.malshare.com
[26]	T. Kim, B. Kang, M. Rho, et al., A Multimodal Deep Learning Method for Android Malware Detection Using Various Features. IEEE T. Inform. Foren. Sec., 14 (2019), 773–788.
[27]	Virustotal. Available from: https://www.virustotal.com/
[28]	M. Hall, E. Frank, G. Holmes, et al., The WEKA data mining software: an update. ACM SIGKDD explorations newsletter, 11 (2009): 10–18.
[29]	Z. Wang, J. Ren, D. Zhang, et al., A deep-learning based feature hybrid framework for spatiotemporal saliency detection inside videos, Neurocomputing, 287 (2018), 68–83.
[30]	J. Zabalza, J. Ren, J. Zheng, et al., Novel segmented stacked autoencoder for effective dimensionality reduction and feature extraction in hyperspectral imaging, Neurocomputing, 185 (2016), 1–10.
[31]	S. Md Noor, J. Ren, S. Marshall, et al., Hyperspectral Image Enhancement and Mixture Deep-Learning Classification of Corneal Epithelium Injuries, Sensors, 17 (2017), 2644.
[32]	J. Ren, D. Wang and J Jiang, Effective recognition of MCCs in mammograms using an improved neural classifier, Eng. Appl. Artif. Intel., 24 (2011), 638–645.

Reader Comments

Your name:*

Email:*
© 2019 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)