Learning from imbalanced data is a challenging task in the machine learning field, as with this type of data, many traditional supervised learning algorithms tend to focus more on the majority class while damaging the interests of the minority class. Stacking ensemble, which formulates an ensemble by using a meta-learner to combine the predictions of multiple base classifiers, has been used for solving class imbalance learning issues. Specifically, in the context of class imbalance learning, a stacking ensemble learning algorithm is generally considered to combine with a specific sampling algorithm. Such an operation, however, might suffer from suboptimization problems as only using a sampling strategy may make it difficult to acquire diverse enough features. In addition, we also note that using all of these features may damage the meta-learner as there may exist noisy and redundant features. To address these problems, we have proposed a novel stacking ensemble learning algorithm named MSFSS, which divides the learning procedure into two phases. The first stage combined multiple sampling algorithms and multiple supervised learning approaches to construct meta feature space by means of cross combination. The adoption of this strategy satisfied the diversity of the stacking ensemble. The second phase adopted the whale optimization algorithm (WOA) to select the optimal sub-feature combination from the meta feature space, which further improved the quality of the features. Finally, a linear regression classifier was trained as the meta learner to conduct the final prediction. Experimental results on 40 benchmarked imbalanced datasets showed that the proposed MSFSS algorithm significantly outperformed several popular and state-of-the-art class imbalance ensemble learning algorithms. Specifically, the MSFSS acquired the best results in terms of the F-measure metric on 27 datasets and the best results in terms of the G-mean metric on 26 datasets, out of 40 datasets. Although it required consuming more time than several other competitors, the increment of the running time was acceptable. The experimental results indicated the effectiveness and superiority of the proposed MSFSS algorithm.
Citation: Shuxiang Wang, Changbin Shao, Sen Xu, Xibei Yang, Hualong Yu. MSFSS: A whale optimization-based multiple sampling feature selection stacking ensemble algorithm for classifying imbalanced data[J]. AIMS Mathematics, 2024, 9(7): 17504-17530. doi: 10.3934/math.2024851
Learning from imbalanced data is a challenging task in the machine learning field, as with this type of data, many traditional supervised learning algorithms tend to focus more on the majority class while damaging the interests of the minority class. Stacking ensemble, which formulates an ensemble by using a meta-learner to combine the predictions of multiple base classifiers, has been used for solving class imbalance learning issues. Specifically, in the context of class imbalance learning, a stacking ensemble learning algorithm is generally considered to combine with a specific sampling algorithm. Such an operation, however, might suffer from suboptimization problems as only using a sampling strategy may make it difficult to acquire diverse enough features. In addition, we also note that using all of these features may damage the meta-learner as there may exist noisy and redundant features. To address these problems, we have proposed a novel stacking ensemble learning algorithm named MSFSS, which divides the learning procedure into two phases. The first stage combined multiple sampling algorithms and multiple supervised learning approaches to construct meta feature space by means of cross combination. The adoption of this strategy satisfied the diversity of the stacking ensemble. The second phase adopted the whale optimization algorithm (WOA) to select the optimal sub-feature combination from the meta feature space, which further improved the quality of the features. Finally, a linear regression classifier was trained as the meta learner to conduct the final prediction. Experimental results on 40 benchmarked imbalanced datasets showed that the proposed MSFSS algorithm significantly outperformed several popular and state-of-the-art class imbalance ensemble learning algorithms. Specifically, the MSFSS acquired the best results in terms of the F-measure metric on 27 datasets and the best results in terms of the G-mean metric on 26 datasets, out of 40 datasets. Although it required consuming more time than several other competitors, the increment of the running time was acceptable. The experimental results indicated the effectiveness and superiority of the proposed MSFSS algorithm.
| [1] |
P. Branco, L. Torgo, R. P. Ribeiro, A survey of predictive modeling on imbalanced domains, ACM Comput. Surv. (CSUR), 49 (2016), 1–50. https://doi.org/10.1145/2907070 doi: 10.1145/2907070
|
| [2] |
K. Oksuz, B. C. Cam, S. Kalkan, E. Akbas, Imbalance problems in object detection: A review, IEEE T. Pattern Anal., 43 (2021), 3388–3415. https://doi.org/10.1109/TPAMI.2020.2981890 doi: 10.1109/TPAMI.2020.2981890
|
| [3] |
M. Ghorbani, A. Kazi, M. S. Baghshah, H. R. Rabiee, N. Navab, RA-GCN: Graph convolutional network for disease prediction problems with imbalanced data, Med. Image Anal., 75 (2022), 102272. https://doi.org/10.1016/j.media.2021.102272 doi: 10.1016/j.media.2021.102272
|
| [4] |
Y. C. Wang, C. H Cheng, A multiple combined method for rebalancing medical data with class imbalances, Comput. Biol. Med., 134 (2021), 104527. https://doi.org/10.1016/j.compbiomed.2021.104527 doi: 10.1016/j.compbiomed.2021.104527
|
| [5] |
A. Abdelkhalek, M. Mashaly, Addressing the class imbalance problem in network intrusion detection systems using data resampling and deep learning, J. Supercomput., 79 (2023), 10611–10644. https://doi.org/10.1007/s11227-023-05073-x doi: 10.1007/s11227-023-05073-x
|
| [6] |
Z. Li, K. Kamnitsas, B. Glocker, Analyzing overfitting under class imbalance in neural networks for image segmentation, IEEE T. Med. Imaging, 40 (2021), 1065–1077. https://doi.org/10.1109/TMI.2020.3046692 doi: 10.1109/TMI.2020.3046692
|
| [7] |
V. Rupapara, F. Rustam, H. F. Shahzad, A. Mehmood, I. Ashraf, G. S. Choi, Impact of SMOTE on imbalanced text features for toxic comments classification using RVVC model, IEEE Access, 9 (2021), 78621–78634. https://doi.org/10.1109/ACCESS.2021.3083638 doi: 10.1109/ACCESS.2021.3083638
|
| [8] |
W. Zheng, Y. Xun, X. Wu, Z. Deng, X. Chen, Y. Sui, A comparative study of class rebalancing methods for security bug report classification, IEEE T. Reliab., 70 (2021), 1658–1670. https://doi.org/10.1109/TR.2021.3118026 doi: 10.1109/TR.2021.3118026
|
| [9] |
J. Kuang, G. Xu, T. Tao, Q. Wu, Class-imbalance adversarial transfer learning network for cross-domain fault diagnosis with imbalanced data, IEEE T. Instrum. Meas., 71 (2021), 1–11. https://doi.org/10.1109/TIM.2021.3136175 doi: 10.1109/TIM.2021.3136175
|
| [10] |
M. Qian, Y. F. Li, A weakly supervised learning-based oversampling framework for class-imbalanced fault diagnosis, IEEE T. Reliab., 71 (2022), 429–442. https://doi.org/10.1109/TR.2021.3138448 doi: 10.1109/TR.2021.3138448
|
| [11] |
Y. Aydın, Ü. Işıkdağ, G. Bekdaş, S. M. Nigdeli, Z. W. Geem, Use of machine learning techniques in soil classification, Sustainability, 15 (2023), 2374. https://doi.org/10.3390/su15032374 doi: 10.3390/su15032374
|
| [12] |
M. Asgari, W. Yang, M. Farnaghi, Spatiotemporal data partitioning for distributed random forest algorithm: Air quality prediction using imbalanced big spatiotemporal data on spark distributed framework, Environ. Technol. Inno., 27 (2022), 102776. https://doi.org/10.1016/j.eti.2022.102776 doi: 10.1016/j.eti.2022.102776
|
| [13] |
L. Dou, F. Yang, L. Xu, Q. Zou, A comprehensive review of the imbalance classification of protein post-translational modifications, Brief. Bioinform., 22 (2021), bbab089. https://doi.org/10.1093/bib/bbab089 doi: 10.1093/bib/bbab089
|
| [14] |
S. Y. Bae, J. Lee, J. Jeong, C. Lim, J. Choi, Effective data-balancing methods for class-imbalanced genotoxicity datasets using machine learning algorithms and molecular fingerprints, Comput. Toxicol., 20 (2021), 100178. https://doi.org/10.1016/j.comtox.2021.100178 doi: 10.1016/j.comtox.2021.100178
|
| [15] |
G. H. Fu, Y. J. Wu, M. J. Zong, J. Pan, Hellinger distance-based stable sparse feature selection for high-dimensional class-imbalanced data, BMC Bioinformatics, 21 (2020), 121. https://doi.org/10.1186/s12859-020-3411-3 doi: 10.1186/s12859-020-3411-3
|
| [16] |
N. V. Chawla, K. W. Bowyer, L. O. Hall, W. P. Kegelmeyer, SMOTE: Synthetic minority over-sampling technique, J. Artif. Intell. Res., 16 (2002), 321–357. https://doi.org/10.1613/jair.953 doi: 10.1613/jair.953
|
| [17] |
G. E. A. P. A. Batista, R. C. Prati, M. C. Monard, A study of the behavior of several methods for balancing machine learning training data, ACM SIGKDD Explor. Newslett., 6 (2004), 20–29. https://doi.org/10.1145/1007730.1007735 doi: 10.1145/1007730.1007735
|
| [18] | H. He, Y. Bai, E. A. Garcia, S. Li, ADASYN: Adaptive synthetic sampling approach for imbalanced learning, In: 2008 IEEE international joint conference on neural networks (IEEE world congress on computational intelligence), IEEE Press, 2008. https://doi.org/10.1109/IJCNN.2008.4633969 |
| [19] | M. Kubat, S. Matwin, Addressing the curse of imbalanced training sets: one-sided selection, In: International Conference of Machine Learning, Morgan Kaufmann, 1997. |
| [20] |
M. A. Tahir, J. Kittler, F. Yan, Inverse random under sampling for class imbalance problem and its application to multi-label classification, Pattern Recogn., 45 (2012), 3738–3750. https://doi.org/10.1016/j.patcog.2012.03.014 doi: 10.1016/j.patcog.2012.03.014
|
| [21] |
A. Zhang, H. Yu, Z. Huan, X. Yang, S. Zheng, S. Gao, SMOTE-RkNN: A hybrid re-sampling method based on SMOTE and reverse k-nearest neighbors, Inform. Sci., 595 (2022), 70–88. https://doi.org/10.1016/j.ins.2022.02.038 doi: 10.1016/j.ins.2022.02.038
|
| [22] |
R. Batuwita, V. Palade, FSVM-CIL: Fuzzy support vector machines for class imbalance learning, IEEE T. Fuzzy Syst., 18 (2010), 558–571. https://doi.org/10.1109/TFUZZ.2010.2042721 doi: 10.1109/TFUZZ.2010.2042721
|
| [23] |
H. Yu, C. Sun, X. Yang, S. Zheng, H Zou, Fuzzy support vector machine with relative density information for classifying imbalanced data, IEEE T. Fuzzy Syst., 27 (2019), 2353–2367. https://doi.org/10.1109/TFUZZ.2019.2898371 doi: 10.1109/TFUZZ.2019.2898371
|
| [24] |
H. Yu, C. Mu, C. Sun, W. Yang, X. Yang, X. Zuo, Support vector machine-based optimized decision threshold adjustment strategy for classifying imbalanced data, Knowl.-Based Syst., 76 (2015), 67–78. https://doi.org/10.1016/j.knosys.2014.12.007 doi: 10.1016/j.knosys.2014.12.007
|
| [25] |
H. Yu, C. Sun, X. Yang, W. Yang, J. Shen, Y. Qi, ODOC-ELM: Optimal decision outputs compensation-based extreme learning machine for classifying imbalanced data, Knowl.-Based Syst., 92 (2016), 55–70. https://doi.org/10.1016/j.knosys.2015.10.012 doi: 10.1016/j.knosys.2015.10.012
|
| [26] | J. Laurikkala, Improving identification of difficult small classes by balancing class distribution, In: Artificial Intelligence in Medicine: 8th Conference on Artificial Intelligence in Medicine in Europe, AIME 2001 Cascais, Portugal, Springer Berlin Heidelberg, 2001. https://doi.org/10.1007/3-540-48229-6_9 |
| [27] | F. S. Hanifah, H. Wijayanto, A. Kurnia, Smotebagging algorithm for imbalanced dataset in logistic regression analysis (case: Credit of bank X), Appl. Math. Sci., 9 (2015), 6857–6865. http://dx.doi.org/10.12988/ams.2015.58562 |
| [28] | C. Seiffert, T. M. Khoshgoftaar, J. Van Hulse, A. Napolitano, RUSBoost: Improving classification performance when training data is skewed, In: 19th international conference on pattern recognition, IEEE, 2008. https://doi.org/10.1002/abio.370040210 |
| [29] | Y. Zhang, G. Liu, W. Luan, C. Yan, C. Jiang, An approach to class imbalance problem based on Stacking and inverse random under sampling methods, In: 2018 IEEE 15th international conference on networking, sensing and control (ICNSC), IEEE, 2018. https://doi.org/10.1002/abio.370040210 |
| [30] | Y. Pristyanto, A. F. Nugraha, I. Pratama, A. Dahlan, L. A. Wirasakti, Dual approach to handling imbalanced class in datasets using oversampling and ensemble learning techniques, In: 2021 15th International Conference on Ubiquitous Information Management and Communication (IMCOM), IEEE, 2021. https://doi.org/10.1002/abio.370040210 |
| [31] |
Z. Seng, S. A. Kareem, K. D. Varathan, A neighborhood undersampling stacked ensemble (NUS-SE) in imbalanced classification, Exp. Syst. Appl., 168 (2021), 114246. https://doi.org/10.1016/j.eswa.2020.114246 doi: 10.1016/j.eswa.2020.114246
|
| [32] |
D. H. Wolpert, Stacked generalization, Neural Networks, 5 (1992), 241–259. https://doi.org/10.1016/S0893-6080(05)80023-1 doi: 10.1016/S0893-6080(05)80023-1
|
| [33] | Y. Shi, R. Eberhart, A modified particle swarm optimizer, In: Proceedings of 1998 IEEE international conference on evolutionary computation proceedings. IEEE world congress on computational intelligence (Cat. No. 98TH8360), IEEE, 1998, 69–73. https://doi.org/10.1109/icec.1998.699146 |
| [34] | K. V. Price, Differential evolution: A fast and simple numerical optimizer, In: Proceedings of North American fuzzy information processing, IEEE, 1996,524–527. https://doi.org/10.1109/nafips.1996.534790 |
| [35] |
E. Cuevas, M. Cienfuegos, D. Zaldívar, M. Pérez-Cisneros, A swarm optimization algorithm inspired in the behavior of the social-spider, Exp. Syst. Appl., 40 (2013), 6374–6384. https://doi.org/10.1016/j.eswa.2013.05.041 doi: 10.1016/j.eswa.2013.05.041
|
| [36] |
S. Mirjalili, A. Lewis, The whale optimization algorithm, Adv. Eng. Soft., 95 (2016), 51–67. https://doi.org/10.1016/j.advengsoft.2016.01.008 doi: 10.1016/j.advengsoft.2016.01.008
|
| [37] |
E. Cuevas, A. Rodríguez, M. Perez, J. Murillo-Olmos, B. Morales-Castañ eda, A. Alejo-Reyes, et al., Optimal evaluation of re-opening policies for COVID-19 through the use of metaheuristic schemes, Appl. Math. Model., 121 (2023), 506–523. https://doi.org/10.1016/j.apm.2023.05.012 doi: 10.1016/j.apm.2023.05.012
|
| [38] |
M. H. Nadimi-Shahraki, S. Taghian, S. Mirjalili, L. Abualigah, M. Abd Elaziz, D. Oliva, EWOA-OPF: Effective whale optimization algorithm to solve optimal power flow problem, Electronics, 10 (2021), 2975. https://doi.org/10.1007/978-981-16-9447-9_20 doi: 10.1007/978-981-16-9447-9_20
|
| [39] |
R. Kundu, S. Chattopadhyay, E. Cuevas, R. Sarkar, AltWOA: Altruistic Whale Optimization Algorithm for feature selection on microarray datasets, Comput. Biol. Med., 144 (2022), 105349. https://doi.org/10.1016/j.compbiomed.2022.105349 doi: 10.1016/j.compbiomed.2022.105349
|
| [40] |
M. S. Santos, P. H. Abreu, N. Japkowicz, A. Fernández, C. Soares, S. Wilk, et al., On the joint-effect of class imbalance and overlap: a critical review, Artif. Intell. Rev., 55 (2022), 6207–6275. https://doi.org/10.1007/s10462-022-10150-3 doi: 10.1007/s10462-022-10150-3
|
| [41] |
S. K. Pandey, A. K. Tripathi, An empirical study toward dealing with noise and class imbalance issues in software defect prediction, Soft Comput., 25 (2021), 13465–13492. https://doi.org/10.1007/s00500-021-06096-3 doi: 10.1007/s00500-021-06096-3
|
| [42] |
L. Breiman, Bagging predictors, Mach. Learn., 24 (1996), 123–140. https://doi.org/10.1007/BF00058655 doi: 10.1007/BF00058655
|
| [43] |
R E. Schapire, The strength of weak learnability, Mach. Learn., 5 (1990), 197–227. https://doi.org/10.1007/BF00116037 doi: 10.1007/BF00116037
|
| [44] | A. Krogh, J. Vedelsby, Neural network ensembles, cross validation, and active learning, Adv. Neural Inform. Proces. Syst., 7 (1995), 231–238. Available from: http://papers.nips.cc/paper/1001-neural-network-ensembles-cross-validation-and-active-learning. |
| [45] |
S. Zhang, X. Li, M. Zong, X. Zhu, R. Wang, Efficient kNN classification with different numbers of nearest neighbors, IEEE T. Neur. Net. Learn., 29 (2018), 1774–1785. https://doi.org/10.1109/TNNLS.2017.2673241 doi: 10.1109/TNNLS.2017.2673241
|
| [46] |
J R. Quinlan, Induction of decision trees, Mach. Learn., 1 (1986), 81–106. https://doi.org/10.1023/A:1022643204877 doi: 10.1023/A:1022643204877
|
| [47] |
C. Cortes, V. Vapnik, Support-vector networks, Mach. Learn., 20 (1995), 273–297. https://doi.org/10.1007/BF00994018 doi: 10.1007/BF00994018
|
| [48] |
T. Bayes, An essay towards solving a problem in the doctrine of chances, MD Comput. Comput. Med. Pract., 8 (1991), 376–418. https://doi.org/10.1002/abio.370040210 doi: 10.1002/abio.370040210
|
| [49] |
A. Tharwat, T. Gaber, A. Ibrahim, A. E. Hassanien, Linear discriminant analysis: A detailed tutorial, AI Commun., 30 (2017), 169–190. https://doi.org/10.3233/AIC-170729 doi: 10.3233/AIC-170729
|
| [50] |
X. Su, X. Yan, C. L. Tsai, Linear regression, WIRES Comput. Stat., 4 (2012), 275–294. https://doi.org/10.1002/wics.1198 doi: 10.1002/wics.1198
|
| [51] | C. Blake, E. Keogh, C. J. Merz, UCI repository of machine learning databases, Department of Information and Computer Science, University of California, Irvine, CA, USA, 1998. Available from: http://www.ics.uci.edu/mlearn/MLRepository.html. |
| [52] |
I. Triguero, S. González, J. M. Moyano, S. García, J. Alcalá-Fdez, J. Luengo, et al., KEEL 3.0: An open source software for multi-stage analysis in data mining international, J. Comput. Intell. Syst., 10 (2017), 1238–1249. https://doi.org/10.2991/ijcis.10.1.82 doi: 10.2991/ijcis.10.1.82
|
| [53] | J. Demsar, Statistical comparisons of classifiers over multiple data sets, J. Mach. Learn. Res., 7 (2006), 1–30. Available from: http://jmlr.org/papers/v7/demsar06a.html. |
| [54] |
S. García, A. Fernández, J. Luengo, F. Herrera, Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power, Inform. Sci., 180 (2010), 2044–2064. https://doi.org/10.1016/j.ins.2009.12.010 doi: 10.1016/j.ins.2009.12.010
|
Figures(8) / Tables(8)
Shuxiang Wang, Changbin Shao, Sen Xu, Xibei Yang, Hualong Yu. MSFSS: A whale optimization-based multiple sampling feature selection stacking ensemble algorithm for classifying imbalanced data[J]. AIMS Mathematics, 2024, 9(7): 17504-17530. doi: 10.3934/math.2024851
DownLoad: