A novel profit-based validity index approach for feature selection in credit risk prediction

1.   Introduction

Failure to accurately identify customers' credit risk before granting credit is the main reason for non-performing loans in commercial banks, and whether the construction of the feature system is reasonable directly affects the evaluation results' effectiveness. The construction of a credit risk feature system involves at least the following two aspects:

The first problem is feature combination selection. The discriminative ability of a single feature is strong, but a feature system consisting of individual features is not necessarily optimal. Furthermore, 2^m - 1 feature combinations exist for m features, and different feature combinations will yield different evaluation results and loan profits. Therefore, we aim to address an empirical issue: constructing the feature selection model and selecting the optimal feature combination with respect to discriminative ability and loan profit.

The second problem is the determination of feature weights. Different weights given to the same group of features will lead to different evaluation results, and reasonable feature weights can also provide an essential reference for bank credit approval. Therefore, we aim to answer another empirical question: determining optimal feature weights and ensuring the evaluation results are reasonable and effective.

Motivated by these reasons, a new feature system based on the validity index and maximum profit is proposed. The validity index determines which features enter the prediction model, and the loan profit is the criterion for evaluating the quality of selected features.

This paper is most related to those of Liu et al. ^[1] and Kozodoi et al. ^[2] regarding feature selection. In ^[1], the validity index was constructed based on the between-class coefficient. In ^[2], the loss of the customer predicted to be non-default is 0. The difference in this study is the construction of the objective functions. We take the minimum validity index constructed by the intra-class, between-class, and partition coefficients as the first objective function and the maximum difference between right income and wrong cost as the second objective. Regarding feature weights, the authors in ^[3,4,5] proposed to take maximum accuracy or minimum error as the objective function. In this paper, the feature weights are obtained by the influence of the features on loan profit.

The primary contributions of this work are as follows:

(1) A new feature combination selection method based on the validity index and loan profit to ensure that the selected feature combination considers both the discriminative ability and loan profit, which addresses the deficiency that existing research does not account for the actual business objectives of banks.

(2) The feature weights are determined by considering the impact of the feature on loan profit. Every change in profit to the sum of all change values is taken as the feature weight, which addresses the shortcoming of existing research that only considers classification accuracy.

(3) A novel evaluation function is proposed based on the loan profit. The profit function consists of the right income and wrong. Because of taking the real loan profit as an estimation method, the feature combination selected is more likely to obtain more profits than the feature combination obtained by the maximum accuracy or minimum error.

The features selected by the model proposed in this paper have better discriminative ability and can obtain the maximum profit than the other 14 feature selection methods. The feature selection method based on the validity index proposed in this paper is effective.

The subsequent chapters of this paper are organized as follows. Section 2 provides a literature review. Section 3 introduces the new feature selection method. Section 4 presents the results of comparing the dataset of Chinese listed companies, and the robustness of the proposed model is verified. Finally, Section 5 presents the conclusions and future work.

2.   Related work

Scholars have performed numerous studies on the construction of feature systems for credit risk evaluation, primarily focusing on feature selection and the determination of feature weights.

(1) Research on feature selection for credit risk evaluation.

The single feature selection method defines the importance degree of each feature and selects the features with high importance to compose the feature set. According to the different evaluation functions, it can be divided into four categories. First, relief and its extended feature selection algorithm ^[6] based on distance measurement. Second, mutual information, information gain, etc. ^[7] based on information measurement. Third, the Pearson correlation coefficient ^[8], Fisher score ^[9] and other methods are used based on the importance of features. Fourth, focus, LVF and other algorithms ^[10] based on consistency measures are used to find the smallest subset that has the same classification and discrimination ability as the whole set on the premise of ensuring classification ability.

The feature combination selection method has 2^m - 1 combinations for m features, considering the joint effect of features on default status. Xia et al. ^[11] proposed a feature selection model based on a gradient boosting decision tree, and a Bayesian network was used to optimize the parameters of the decision tree to select the optimal feature combination. Jadhav et al. ^[12] graded the features based on information gain, obtained the optimal feature combination through the embedded method, and obtained the credit rating through K-nearest neighbors, naive Bayes, etc. Arora and Kaur ^[13] proposed a feature combination selection method based on the Bolasso model and combined it with four models, such as random forest and decision tree, to prove its robustness and effectiveness. Liu et al. ^[1] proposed a new validity index and combined it with the support vector machine. The empirical results show that the feature combination selected by the validity index has higher accuracy. Kozodoi et al. ^[2] proposed a profit-based credit feature selection method and demonstrated that it can obtain the maximum profit in the case of a small number of features.

The existing studies focus more on the single feature selection, less on feature combination selection, and even less on bank profit. We construct a profit-maximization feature combination selection method based on the validity index proposed by Liu et al. ^[1]. Profit is defined as the difference between the right income and the wrong cost. The heuristic algorithm is used to obtain the optimal feature combination from 2^m - 1 feature combinations.

(2) Research on feature weighting for credit risk evaluation.

The weight is determined by the feature's degree of variation. On the one hand, the weight comes from the subjective weighting of experts, such as the analytic hierarchy process (AHP) and Delphi method. For example, Gu et al. ^[14] proposed a credit evaluation method based on the analytic hierarchy process (AHP) and data envelopment analysis (DEA). On the other hand, the feature weight is determined by the connection degree of each feature or the amount of information provided. For example, Li et al. ^[15] proposed a combination of the entropy weight method and fuzzy comprehensive method to construct a credit evaluation model and divide the credit ratings of microenterprises. Karaaslan and Özden ^[16] established a credit rating prediction model based on the grey model and grey correlation, which shows that the feature weights obtained by grey correlation can provide a reference for the variable selection of the grey model.

The weight is determined by the relationship between the feature and default status. Zhu et al. ^[17] take the minimum sum of the average distance between good and best applicants and the average distance between bad and worst applicants as the objective function to calculate the optimal feature weights. Chen et al. ^[18] applied the neighborhood rough set to an unbalanced data set and determined the feature weights according to the upper and lower boundaries. Panday et al. ^[19] proposed an imwk-means clustering algorithm to carry out unsupervised clustering, took the minimum weighted Euclidean distance within the class as the objective function, and adopted the Lagrange multiplier to solve for the optimal feature weights and the number of clusters.

The weight is given through the relationship between the feature and accuracy. Serrano-Silva et al. ^[20] proposed a combination of a heuristic algorithm with the NAC model to obtain feature weights. Through the comparison of 10 credit data sets, they found that the classification model after weighting has better classification ability. Mercadier and Lardy ^[21] took the reduced precision of random forest after random arrangement as the feature weight, and the mean square error proved the effectiveness of this method. Mateos-García et al. ^[5] proposed the SWAN model based on the K-nearest neighbour approach to improve the feature weights. The empirical results show that the setting of weights can significantly improve the model's accuracy.

According to the above studies, the existing feature weighting mainly focuses on three aspects, with little consideration of profit. Based on the optimal feature combination obtained with respect to profit maximization, the feature weights are determined following the approach of Mercadier and Lardy ^[21]. We calculate the feature weights by the change in profit after deleting each feature with replacement to the sum of all change values.

(3) Research on credit risk prediction methods.

In existing work for credit risk prediction, most searchers adopted popular two-way classification methods directly, such as mathematical statistics methods, artificial intelligence methods, and ensemble learning methods.

The most common mathematical statistics methods summarize a classification rule through many existing samples, including multivariate discriminant analysis ^[22], naive Bayes ^[23], linear discriminant analysis ^[24], and fuzzy logistic regression ^[25]. Multivariate discriminant analysis is a supervised learning method that aims to find a discriminant function that maximizes the separation between the groups ^[26]. Naive Bayes is a simple and efficient algorithm based on Bayes' theorem, assuming independence between the features ^[27]. The key idea behind LDA is to transform the original high-dimensional feature space into a lower-dimensional space while preserving the discriminatory information between the classes ^[28]. Fuzzy logistic regression is an extension of traditional logistic regression that incorporates fuzzy logic principles to handle uncertainty and ambiguity in data ^[29].

Credit risk prediction methods based on artificial intelligence show greater advantages. The traditional models include artificial neural networks ^[30], support vector machines ^[31], and extreme learning machines ^[32]. Deep learning has gained significant attention and popularity in recent years due to its remarkable performance in various applications. Zhang et al. ^[33] developed a fraud detection system using a deep learning architecture and an advanced feature engineering process based on homogeneity-oriented behavior analysis. Ala'raj et al. ^[34] proposed a model based on the bidirectional Long-Short Term Memory (LSTM) model to give each customer the probability of a missed payment during the next month. Zhao et al. ^[35] proposed a novel imputation method named Multiple Generative Adversarial Imputation Networks for accurate missing data prediction. The experimental results on real-world data demonstrate that the proposed method is superior to other popular imputation methods.

Ensemble learning methods refer to training multiple classification models and combining multiple results for better results. Currently, the main ensemble learning methods are bagging ^[36] and boosting ^[37], and the learning strategies are average, majority voting, and stacking ^[38]. Shen et al. ^[39] developed a new deep-learning ensemble credit risk evaluation model to deal with imbalanced credit data combined with the long-short-term-memory (LSTM) network and the adaptive boosting (AdaBoost) algorithm. Forough and Momtazi ^[40] proposed an ensemble model based on the sequential modeling of data using deep recurrent neural networks and a novel voting mechanism based on an artificial neural network to detect fraudulent actions. Belhadi et al. ^[41] proposed a hybrid ensemble machine learning approach to forecast the credit risk associated with SMEs' agriculture 4.0 investments. Two core approaches were used, i.e., the Rotation Forest algorithm and the Logit Boosting algorithm.

The existing credit risk prediction models, especially the ensemble models, have been proven to have higher classification accuracy. We construct an ensemble model, which is composed of six models: logistic regression ^[42], linear discriminant analysis ^[43], decision tree ^[44], support vector machine ^[45], backpropagation neural network ^[46] and KNN ^[47].

3.   Feature system construction

3.1.   Basic parameters of the validity index

The validity index constructed by the intra-class, between-class, and partition coefficients is used to test the feature combination's discriminative ability. The basic parameters of the validity index are calculated as follows.

(1) Calculation of the distance from each sample to the class centroid.

Let d_ic be the distance between the ith sample and the cth class centroid, h be the number of features, $x_{ij}^c$ be the jth feature data of the ith sample in the cth class, and p_cj be the jth feature data in the cth class centroid. The d_ic is defined as:

where the cth class centroid is a vector composed of the mean of all samples in class c.

(2) Calculation of the membership degree.

The membership degree μ_ic of the ith sample belonging to class c is defined as:

For any i and c, if d_ic > 0, then,

In Eq. (2), if there are i and c such that d_ic = 0, then μ_ic = 1; if there is k such that k ≠c, then μ_ic = 0.

(3) Calculation of the partition coefficient.

The partition coefficient (PC) ^[48] with h features is calculated as:

where r represents the number of classes, n represents the number of samples, and μ_ic represents the membership degree of the ith sample belonging to class c.

Each sample has two membership degrees of default and non-default. Considering the effect of the higher membership degree in each class in this paper on the distinction degree, we obtain a new partition coefficient PC′(h).

(4) Calculation of the intra-class coefficient and between-class coefficient.

The partition coefficient determines the features' distinction degree from the membership degree perspective, but insufficient attention is given to intra-class and between-class data. We construct intra-class and between-class coefficients from the perspective of data structure.

Let r be the number of classes, ${{\mathbf{\bar p}}_c}$ be a vector composed of the cth class centroid, ${{\mathbf{\bar p}}_k}$ be a vector composed of the cth class centroid, n be the number of all samples, and m_c be the number of the cth class. The between-class coefficient BC(c) with h features is calculated as:

Let r be the number of classes, m_c be the number of the cth class, m_c be the number of the cth class, μ_ic be the membership degree of the ith sample belonging to class c, ${\mathbf{x}}_i^c$ be the ith sample in the cth class, and ${{\mathbf{\bar p}}_c}$ be a vector composed of the cth class centroid. The intra-class coefficient IC(c) with h features is calculated as follows:

3.2.   Optimal feature combination selection

3.2.1.   First feature selection based on the validity index

The partition coefficient PC(h)′, intra-class coefficient IC(h), and between-class coefficient BC(h) are integrated to construct the following validity index VI(h).

The BC(h)/IC(h) in the first term represents the ratio of the between-class and intra-class coefficients. The larger the BC(h)/IC(h) value is, the better the discriminative ability of h features. For 0 < e^-BC(h)/IC(h)≤1, the smaller its value is, the better the discriminative ability of h features. According to Eq (4), the smaller the 1-PC'(h) in the second term is, the better the discriminative ability of h features. Therefore, the smaller the validity index is, the better the feature combination containing h features can distinguish between defaults and non-defaults.

Compared with existing work ^[1], the validity index constructed in this paper has the following two advantages:

First, the validity index proposed in this paper fully considers the separation degree within classes and the separation degree between classes. It can select the optimal feature combination by traversing all feature combinations when the number of features is determined.

Second, the validity index proposed in this paper adds a partition coefficient to measure the distinction degree of all samples, which avoids the deficiency that existing studies only focus on the separation degree of classes.

The amount of calculation is extremely large for dozens or even hundreds of features ^[49]. To reduce the computational burden, we select the optimal feature combination through the backward selection sequence ^[50] of the heuristic search.

Step 1: The validity index value VI(h) is computed for all h features.

Step 2: Each feature is deleted one at a time, h validity indices are computed for all subsets with h-1 features, and the feature combination with the minimum validity index is the optimal feature combination.

Step 3: Each feature among the remaining h-1 features is deleted one at a time, and the optimal feature combination is obtained to form a subset with h-2 features.

Step 4: Steps 2-4 are repeated until one feature is left.

Step 5: h feature combinations including 1, 2, ‧‧‧, h features are obtained.

3.2.2.   Second feature selection based on profit

To measure the discriminative ability of the feature combination, we should not only focus on the accuracy of the feature combination but also consider whether the discrimination results yield actual profit for banks ^[2].

If default customers are correctly classified, the bank will avoid loss by refusing to grant them credit, which can be regarded as an expected benefit of the bank. If default customers are misclassified, the bank will incur bad debt due to the loans. If non-default customers are misclassified, the bank will refuse to grant credit, lose possible high-quality customers and lose the interest generated by the loans. The bank will earn interest income from the loans if non-default customers are correctly classified.

(1) Calculation of direct income and indirect loss of non-default customers.

For non-default customers, the direct income comes from interest income, and the indirect loss comes from the interest loss caused by the loss of customers. It is generally believed that a company has many sources of loans, and its disclosure is limited, so it is difficult to measure the amount of loans from the company's financial data. We use the ID (interest-bearing debt) to determine the benefit and loss of granting credit to non-default customers. Interest-bearing debt requires interest payments, including the company's liabilities, long-term liabilities due within one year, long-term loans, bonds payable, etc. The interest is obtained by multiplying the interest-bearing debt by the loan interest rate.

For the ith sample, the value of "interest-bearing liabilities" is set to ID_i, and the loan interest rate is set to r. If non-default customers are correctly classified, the more interest-bearing liabilities, the more interest income will be earned. If non-default customers are misclassified, the more interest-bearing liabilities there are, the more interest income will be lost.

(2) Calculation of the direct loss and indirect income of default customers.

The direct loss of default customers comes from the bad debt, and the indirect income comes from the evaded bad debt. When default customers have cash flow difficulties and refuse to repay all their loans, they will choose to pay part of the loans, and the loss given default will not be 100%.

After confirming that the quick ratio has a significant impact on default ^[51], we take the difference between "current liability" and "quick assets" as the direct loss estimation when granting credit to default customers. Quick assets include rapidly realizable enterprise assets such as monetary funds and accounts receivable. Current liabilities include short-term loans, accounts payable, notes payable, and other corporate liabilities that need to be repaid within a short period. The difference between current liabilities and liquid assets is the loss that companies cannot repay in a short period, which may cause a loss for commercial banks.

For the ith sample, the value of "interest-bearing liabilities" is set to ID_i, the value of "quick ratio" is set to QA_i, and the "current liability" is set to CL_i. The profit function constructed in this paper is as follows:

The profit(h) in Eq (10) represents the difference between the right income and the wrong cost. The first item indicates the loss avoided after identifying default customers; the second item indicates the interest income after identifying non-default customers; the third item indicates the loan loss after lending to default customers; and the fourth item means the interest loss after refusing to lend to non-default customers.

Compared with the existing research ^[2] that only considers the profit generated by two results of TP and FP in the confusion matrix, the profit function in this paper considers four results in Table 1, which can more comprehensively characterize the composition of bank profits.

Based on obtaining h feature combinations in 3.2.1, selecting the best feature combination from h feature combinations is necessary. The idea of the second feature selection targeting profit maximization in this paper is as follows:

Step 1: Every feature combination is substituted into six classification models, such as the support vector machine and decision tree. Four parameters of each model in the confusion matrix are obtained.

Step 2: According to Eq (10), the profit value of each classification model is obtained, and the average profit value of six classification models is taken as the classification ability of each feature combination.

Step 3: The average profit values of h feature combinations are compared. In this paper, the feature combination corresponding to the maximum average profit value is the best.

3.3.   Determination of feature weights

The contribution of each feature to discriminative ability is different. Some features with larger contributions must be given higher weights, while some features with lower contributions must be given lower weights to ensure the feature system's overall accuracy.

If h features exist in the optimal feature combination, one feature is deleted with replacement, and subsequently, the h profits containing h-1 features are calculated. The absolute differences between the h profits containing h-1 features and the profit containing an optimal feature combination are Δx₁, Δx₂, Δx₃... Δx_m. The ratio of the absolute difference to the sum of all absolute differences Δx_i / $\sum\limits_{j = 1}^m {\Delta {x_j}}$ denotes the weight of the ith feature.

3.4.   Construction of credit score model

Banks consider not only whether customers default but also the default risk. Whether a customer defaults or not only indicates their ability to repay their loan on time, but it is not possible to sort the lending of customers and determine the priority lending customers. A credit score model is essential for evaluating customers' credit risk. A good credit score model should give customers with low default probability a higher credit score ^[52].

Let a be the offset, b be the factor, and P_i be the comprehensive probability. The credit score is:

The credit score in Eq (11) quantifies the customer's performance. The higher the credit score is, the lower the default probability. The comprehensive probability P_i is obtained by default probability P_i1 and the sum of weighted features P_i2, P_i = 0.5P_i1+0.5P_i2. The default probability P_i1 is obtained by the model with the best classification performance among the six classification models, and P_i2 is obtained by the optimal features and weights:

P_i2 in Eq (12) reflects the influence of features and feature weights on credit scores. As the credit score increases, the default probability decreases. We established a credit scorecard model ^[53]. When the credit score decreases by 20 points, the default rate is twice as high as that of $\frac{P_i}{1-P_i}$. When the default-to-default ratio is 1:20, the credit score is 600. The credit score of Eq (13)is redefined as follows:

The credit score in Eq (13) differs from the existing research ^[52] in at least two aspects:

First, the expression for the credit score is different. Existing studies use default probability or the sum of weighted features as the credit score without recognizing that the credit score should have discriminative ability rather than a simple high or low default probability. We draw on the idea of the credit scorecard model and builds a credit score model based on comprehensive probability. On the one hand, the model has good interpretability and can explain why the credit score is not high. On the other hand, by setting the score factor, the credit score has discriminative ability.

Second, the rationality of credit scores is different. Existing scholars consider default probability P_i1 or the sum of weighted features P_i2. However, it is important for the credit score to consider not only the influence of the asset-liability ratio, corporate profit, and other features but also the default probability. Only considering the default probability P_i1 easily causes the customer's credit score to fluctuate substantially, and there is a large difference between the previous year and the following year or even the opposite unreasonable situation. Only considering the sum of weighted features P_i2 cannot reflect the impact of events such as default on credit score.

4.   Empirical analysis

4.1.   Data sources

(1) Data description.

In this paper, we take ST(*ST) and non-ST companies in the Shanghai and Shenzhen stock markets as the research objects. The ST(*ST) companies are treated as defaulters and other listed companies are defined as non-defaulters ^[54]. The ST label indicates that the listed companies' stocks have investment risks, while *ST signifies that the company's stocks are at risk of being delisted. These two situations indicate that the listed companies involve substantial risk. Ultimately, this work selects 47, 171 observations of listed companies from 2000 to 2018, with a non-default sample (45, 317) that is 24.44 times the size of the default sample (1, 854).

Based on the classic high-frequency features of three foreign credit institutions such as Moody's, Standard & Poor's and Fitch ^[55], domestic financial institutions such as the Agricultural Bank of China and Bank of China, and authoritative references ^{[56,57,58,59]}, a credit risk feature system for listed companies is established. The feature system includes three first-level criterion layers (432 financial factors, 119 non-financial factors, and 153 macro factors) and nine second-level criterion layers, including solvency, profitability, and operating capability.

(2) Sample processing.

The original data are normalized, and each feature is converted into a value in [0, 1] ^[60,61]. To allow the established model to have multi-year predictive ability and to avoid the time point problem of financial statements identified by Ohlson ^[62], the observations at time t-m (m = 1, 2, ‧‧‧, 5) before the defaults at time t are adopted.

The data at times t-1 to t-5 are defined as follows:

For each ST(*ST) company, the year when the company was marked as ST(*ST) is taken as the base time t. The observations of m years before time t, that is, the observations at time t-m (m = 1, 2, 3, 4, 5) and the status at time t, are used to form five default data sets.

For each non-ST company, the number of non-ST companies with m continuous years is N₀. The number of non-ST companies in each year is determined by N₀/L (year span). The feature data x_ij^t-m of non-ST companies at time t-m (m = 1, 2, 3, 4, 5) and the default status y_j^t at time t are used to construct a data set of non-ST companies.

The default and non-default companies also form five new data sets. Each time window in this paper contains 3, 425 companies, consisting of 2, 949 non-default companies and 476 default companies. By using 10-fold cross-validation, the samples are randomly divided into ten parts: eight for the training set, one for the validation set and one for the test set. The average of the ten classification results is taken as the final result.

4.2.   Construction of the optimal feature system

In this paper, a 10-fold cross-validation method is used to test the classification performance of the feature system. According to the ratio of 8:1:1, the data are randomly divided into ten parts. Eight parts for the training set, one part for the validation set, and one part for the test set. The training set is used to adjust the model's parameters and optimize its performance. The validation set is used to obtain the optimal feature combination and weights. The test set is used for out-of-sample performance evaluation ^[63]. After obtaining the optimal features, they are applied to the classification model, which provides the out-of-sample default probabilities. By comparing the default probabilities with a threshold (0.5), the default status of customers can be determined, thus obtaining the accuracy of default prediction.

The model's classification performance is based on five classification criteria: Accuracy, Type-I error, Type-II error, G-mean and the Area Under Curve (AUC). In this paper, the six classification models are logistic regression ^[42], linear discriminant analysis ^[43], decision tree ^[44], support vector machine model ^[45], backpropagation neural network ^[46], and KNN ^[47], and the interest rate is 5%. The steps for obtaining the optimal feature system are shown in 3.2 and 3.3. The number of optimal features from t-1 to t-5 is shown in Table 2.

Through the 5-year feature frequency analysis, 43 features, such as monetary funds, notes receivable, intangible assets, and social contribution per share, have a 5-year predictive ability; 20 features, such as capital reserves, the executive shareholding ratio, whether shareholder protection is disclosed, and social donations, have a 4-year predictive ability; 15 features, such as the payout ratio before tax and whether environment and sustainable development is disclosed, have a 3-year predictive ability; and 14 features, such as the cash ratio, income tax and external guarantee balance, have a 2-year predictive ability.

4.3.   Comparison of classification performance

4.3.1.   Comparison of classification performance between models

The classification performance of the optimal feature combination is compared with features without profit consideration and without feature selection. The comparison results of the models' classification performance at time t-2 are shown in Table 3. The others have the same conclusions as time t-2, so only the results of time t-2 are listed. The five criterion values of each method are the average values of six classification models.

For the logit model, the objective function minimization is the stochastic gradient descent, the initial linear coefficient is 0, the learning rate is 0.05 and the observation weight is 1. For the LDA model, for the cost of misclassification, the default is 1, the discriminant type is linear, and the observation weight is 1/n. For the DT model, the maximum category level is 10, the maximum number of decision splits is (X, 1)-1, the minimum number of leaf node observations is 1, and the minimum number of branch node observations is 10. For the SVM model, the kernel function is the RBF, and the kernel scale is 2. The kernel scale is obtained through the genetic algorithm ^[12]. For the BPNN model, the number of network layers is 10, the learning rate is 0.1, the maximum training time is 100, the initial weight change is 0.05 and the loss function is a quadratic mean square function. For the KNN model, the number of nearest neighbours is 5, the distance is Euclidean distance, and the distance weighting is 1/distance.

1). The proposed validity index+Profit

2). The proposed validity index+AUC

3). Validity index proposed by Liu et al. ^[1]+Profit

4). Validity index proposed by Liu et al. ^[1]+AUC

5). Validity index proposed by Peña et al. ^[64]+Profit

6). Validity index proposed by Peña et al. ^[64]+AUC

7). Sequential Forward Selection ^[11]+Profit

8). Sequential Forward Selection ^[11]+AUC

9). Sequential Backward Selection ^[50]+Profit

10). Sequential Backward Selection ^[50]+AUC

11). LASSO ^[65]+Profit

12). LASSO ^[65]+AUC

13). Profit function proposed by Maldonado et al. ^[9]

14). Profit function proposed by Garrido et al. ^[66]

15). Profit function proposed by Kozodoi et al. ^[2]

Fifteen feature selection methods are compared in Table 3. Feature selection considering profit can significantly improve predictive ability and obtain the maximum profit. The predicted profit values at t-1 to t-5 are larger than those of the models that do not consider profit. Compared with the validity indices proposed by Liu et al. ^[1] and Peña et al. ^[64], the validity index proposed in this paper has higher classification accuracy and profit, which shows that the features selected by the proposed validity index have higher default identification ability to separate the two types of samples. Sequential backward selection ^[50] is used in the feature selection process. By comparison, the backward selection algorithm based on the validity index has higher classification accuracy and profit than the backward selection algorithm. By comparison with other feature selection methods, such as sequential forward selection ^[11] and LASSO ^[65], the effectiveness of the proposed feature selection method is further verified.

The following conclusions can be drawn from the comparative analysis of the model classification performance in Figure 1.

First, the order according to the average accuracy is t-1 > t-2 > t-3 > t-4 > t-5. The model proposed in this paper has the strongest predictive ability in the next year, followed by the next two years, and the worst in the next five years. The results fully conform to the law of "the longer the prediction time, the lower the prediction accuracy" ^[67]. The possible reason is that the longer the time span of feature data X and label Y, the weaker the correlation between them is, leading to worse model classification performance.

Second, after averaging the profits of the five-time windows, the order is t-1 > t-2 > t-3 > t-4 > t-5. The feature combination selected by the model proposed in this paper has the maximum profit-predicting ability in the next year, followed by the next two years, and the worst in the next five years. The results conform to the law that the longer the prediction time, the lower the prediction profit. There are two possible reasons: one is that the accuracy meets this standard due to features and samples, and the other is that lower Type-I and Type-II errors determine a higher profit.

Finally, by comparing the average accuracy and the average profit, it is found that the year with the high profit is not high in accuracy. For example, the profit at time t-3 ranks third, but the G-mean ranks last. This shows that the impact of classification performance on profit is not significantly positively correlated but is also affected by other factors, such as interest-bearing liabilities, current liabilities, quick assets, interest rate, etc. Type-II error has a more significant impact on profit. Once the customer defaults, the bank will suffer a more significant loss.

4.3.2.   Robustness tests

(1) Data sources.

In this paper, we use three credit datasets, namely, Chinese Farmer, Germany, and Taiwan ^[68,69,70], listed in Table 4, to verify the robustness of the proposed model. If the profit and accuracy proposed in this paper are also higher than those of the other models in the above three datasets, the feature selection model is robust.

Chinese Farmer dataset. There are 2, 044 observations and 43 features, including 1, 816 non-default customers (label 0) and 228 default customers (label 1).

Germany dataset. There are 1, 000 observations and 20 features, including 700 non-default customers (label 0) and 300 default customers (label 1).

Taiwan dataset. There are 30, 000 observations and 23 features, including 23, 364 non-default customers (label 0) and 6, 636 default customers (label 1).

The Chinese Farmer dataset contains the loan amount (A), interest rate (r), and outstanding principal and interest (O). The interest loss in Table 1 is -A×r, the interest income is -A×r, the evaded bad debt is O, and the bad debt is O. Because the Germany and Taiwan datasets only contain the loan amount (A) and interest rate (r) set as 5%, the interest loss is -A×r and the interest income is -A×r. For the evaded bad debt and bad debt, the evaded bad debt is O, and the bad debt is O ^[71,72].

(2) Comparison of accuracy and profit in different datasets.

The experimental results in Table 5 show that the accuracy of Model 1 (the proposed model) is not the highest, as the accuracy, Type-II error, and G-mean are lower than those of Model 3 in the Chinese Farmer dataset. The accuracy and Type-II error are lower than those of Model 2 in the Germany dataset. The accuracy is lower than that of Model 13, and the G-mean is lower than that of Model 2 in the Taiwan dataset. However, the model in this paper has the highest profit among all 15 models.

There is an inconsistency between accuracy and profit in Table 5, i.e., the model with the highest accuracy does not necessarily obtain the most loan profit. This result is related to the construction of the objective function. The classic classification models take the minimum error as the objective function, guaranteeing high accuracy. In contrast, our model takes the maximum profit as the objective function, ensuring a high-profit value.

4.4.   Comparative analysis of credit qualifications

The key to credit risk management is to measure the default risk, and the credit score model is an important means to quantify the default risk. The credit score model is directly related to the default risk, which can affect banks' bad debts. In this paper, we construct a credit score model to measure the credit risk of Chinese listed companies and compare and analyze the credit qualifications of different company sizes and industries.

4.4.1.   Analysis of companies' credit qualifications in different sizes

The proposed validity index+Profit is the model with the highest profit from t-1 to t-5. The samples at t-m (m = 1, 2, ‧‧‧, 5) are trained on. The samples of Chinese listed companies in 2018 are substituted into the five classification models trained by the decision tree to obtain the default probability from 2019 to 2023. The mean value of each customer's five-year default probability is taken as the final default probability P_i1. P_i2 is obtained by the mean of the sum of weighted features. The credit score of each customer is calculated by Eq (13), and the credit qualifications in different sizes are further shown in Table 6.

According to size, the companies are divided into large, medium and small and micro companies ^[73]. The average credit scores from high to low are medium-sized, small and micro-sized and large-sized. The Mann-Whitney U test is used to verify whether there is a significant difference between two different sizes of companies. It is found that there are significant differences between large companies and the other two types of companies, while there is no significant difference between medium-sized companies and small and micro companies.

4.4.2.   Analysis of credit qualifications in different industries

We divided the observations into 18 industries according to the China Securities Regulatory Commission's 2012 industry classification guidelines ^[74]. This article only shows the differences in credit qualifications among the ten industries with large samples (greater than 50).

1). Manufacturing Industry

2). Information Transmission, Software and Information Technology Service Industry

3). Wholesale and Retail Industry

4). Real Estate Industry

5). Electricity, Heat, Gas, Water Production and Supply Industry

6). Construction Industry

7). Transportation, Storage, and Postal Industry

8). Financial Industry

9). Mining Industry

10). Culture, Sports, and Entertainment Industry

The credit scores in column 3 in Table 7 show that the credit qualifications of the information transmission, software, information technology services industry & culture, sports, and entertainment industries are good. The manufacturing and electricity, heat, gas, water production and supply industries are mediocre. The financial industry and real estate industry are poor. A Mann-Whitney U test between different industries shows significant differences in credit qualifications between most industries, such as the financial industry and all other industries, the manufacturing industry and the wholesale and retail industry. However, there is no difference in credit qualifications among some industries, such as the manufacturing industry and the electricity, heat, gas, water production and supply industry.

4.4.3.   Analysis of credit qualifications based on different production factors

According to different production factors, we divide industries into labor-intensive, capital-intensive and technology-intensive industries ^{[75,76,77,78]}. This article presents the credit scores of these three industries and examines whether there are significant differences.

The credit scores in Table 8 from high to low are technology-intensive industries, labor-intensive industries, and capital-intensive industries. The Mann-Whitney U test shows that there are significant differences between any two of the three industries.

5.   Conclusions and future work

This work offers the following primary conclusions:

(1) By taking the minimum validity index and the maximum profit as the objective function, the features selected from 3, 425 observations of listed companies at times t-1 to t-5 are 91, 84, 79, 73 and 58, respectively. Forty-three features, such as monetary funds and social contribution per share, have a 5-year predictive ability; 20 features, such as capital reserve and social donation, have a 4-year predictive ability; 15 features, such as whether to disclose environment and sustainable development, have a 3-year predictive ability; and 14 features, such as cash ratio and external guarantee balance, have a 2-year predictive ability.

(2) Compared with 14 methods, the features selected by our proposed method can obtain the maximum profit to ensure classification performance. The processed t-1 to t-5 data can indicate whether there is a default in the next five years and the profits generated by bank decisions; that is, it can provide at least five years of predictions.

(3) The listed companies' difference test ranks companies' credit qualifications: medium companies > larger companies > small and micro companies. The credit qualifications of the information transmission, software, and information technology services industry and the culture, sports, and entertainment industry are good. The manufacturing industry and the electricity, heat, gas, water production and supply industry are mediocre. The financial industry and real estate industry are poor. The credit qualifications from high to low are technology-intensive industries, labor-intensive industries, and capital-intensive industries. The Mann-Whitney U test shows that there are significant differences between any two of the three industries.

The proposed two-stage feature selection method provides a new perspective to obtain the status of Chinese listed companies in a timely manner. This will help lending institutions better identify high-risk customers, reduce default risks and increase loan profitability, thereby enhancing the competitiveness of the entire banking industry.

However, the proposed framework still has some shortcomings, which also represent a direction for future research. (i) Further studies may include using a method for imbalanced data, such as undersampling and oversampling. (ii) Further studies may use fuzzy-based models because crisp data are insufficient to tackle uncertain situations.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

We gratefully acknowledge financial support from the General Program of National Social Science Foundation of China (21BJY065).

Conflict of interest

All authors declare no conflicts of interest in this paper.