Multi-label feature selection via constraint mapping space regularization

1.   Introduction

Feature selection is one of the important dimensionality reduction methods to deal with dimensional disasters ^[1,2], whether in single-label clustering, classification, or multi-label classification. It is also a hot topic in research on machine learning and data analysis ^[3].

As a type of dimensionality reduction technology, feature selection methods select the most representative feature subset from the original features of the sample by applying a certain strategy or model to remove redundant and irrelevant features and thus achieve the task of reducing data dimensionality ^[4]. In addition, feature selection, as a common dimensionality reduction technique, has a several advantages, such as the ability to reduce the calculation and storage pressure of the learning algorithm and improve its robustness and interpretability. Based on the interaction with the learning system, multi-label feature selection can be divided into filter ^[5,6,7], wrapper ^[8,9], or embedded methods^[10,11,12]. The embedded method is different from the filter method, which completely ignores the learning algorithm's influence on feature selection. The embedded method is also different from the wrapper method, which completely relies on a learning algorithm to guide feature selection. The embedded method embeds the feature selection process in the learning algorithm and makes it complete feature selection in the learning process.

By reviewing the research on the existing embedded models, we know that most of the existing models are based on linear mapping and information theory and are often combined with manifold learning and sparse regular terms to construct multi-label feature selection models. In linear mapping-based approaches, either a linear mapping is directly from samples to real labels, or a linear mapping is constructed by using pseudo-labels instead of real labels. However, the binary nature of real labels contradicts the nature of the continuous type of variables of linear mapping. In addition, the use of pseudo-labels increases the number of variables of the model, thus increasing the computational burden of the model. Fortunately, we demonstrate here that constraining the mapping space of linear mappings directly in the model can alleviate this problem well. Specifically, we construct a novel sparse multi-label feature selection model by introducing the Hilbert-Schmidt independence criterion ^[13] and sample manifold learning and combining the $L_{21}$ norm as a sparse constraint. Relative to the existing state-of-the-art models, the proposed model alleviates the problem of real label space being non-applicable to linear mapping by constraining the mapping space, it also more effectively reduces the computational burden of the model and improves the stability of the model.

The main research work of this paper is as follows:

1) We introduce the HSIC and sample manifold, which jointly constrain the fundamental properties of the mapping space in terms of both real-label and sample structure.

2) We introduce the $L_{21}$ norm as a sparse constraint, which allows for the construction of a sparse multi-label feature selection model for constrained mapping spaces, and an optimization algorithm with convergence has been designed to optimize the proposed method.

3) A comparative experiment was conducted with eight highly influential multi-label feature selection algorithms. Experimental results show that the proposed method is effective and feasible.

The rest of the paper is summarized as follows: in Section 2, notations and a brief overview of existing models are given. In Section 3, the model establishment, optimization, and convergence proofs for the proposed algorithm are introduced. In Section 4, the results of comparative tests are presented and analyzed, and the proposed algorithm's parameter sensitivity, convergence, and time complexity are tested and analyzed. Finally, the summary of this paper and the direction of future research are given in Section 5.

2.   Related work

2.1.   Notations of this paper

For any matrix $A \in R^{c\times d}$, $A^{T}$ is the transpose of $A$; $A_{ij}$ is a member of the $i$th row and $j$th column of $A$; the $i$th row vector of $A$ is denoted by $A_{i\ast}$; the $j$th column vector of $A$ is denoted by $A_{\ast j}$; the $L_{21}$ norm of $A$ is $\|A\|_{21}$; the Frobenius norm of $A$ is $\|A\|_{F}$; $S_{A}$ is the similarity matrix with respect to the matrix $A$; $L_{A}$ is the Laplacian matrix of the similar matrix $S_{A}$; $n$, $d$ and $m$ represent the number of samples, the number of features, and the number of labels, respectively. $X \in R^{n\times d}$ is the sample matrix; $Y \in R^{n\times m}$ is the label matrix; $Q \in R^{d\times m}$ is the weight matrix; $H \in R^{n\times n}$ is the center matrix.

2.2.   A review of multi-label learning

In this subsection, we briefly review the research status of embedded multi-label feature selection techniques by discussing many works of literature on multi-label feature selection.

Among the multi-label feature selection models based on information theory, the classic representative models include SCLS ^[14], MDMR ^[15], PMU ^[16], and FIMF ^[17]. Among them, MDMR, PMU, and FIMF use mutual information to quantify the importance of features and select features for their importance. However, these models may lose important information when processing higher-order label data. Moreover, the computational cost of high-order multivariate mutual information is prohibitive. However, SCLS has poor algorithm performance due to a combination of excessive labels and features, making feature selection impractical ^[18,19]. In addition, a multi-label feature selection method considering maximum correlation was proposed in ^[20]. In order to ensure the correlation between the selected features and different label groups, the model integrates the maximum correlation of high-order label correlation into the feature selection model. In addition, Gao et al. ^[21] designed a multi-label feature selection method that includes three low-order information theory terms and unified the framework of multi-label information theory methods.

In linear mapping-based multi-label feature selection models, some models directly apply a linear mapping between samples to real labels. For example, Li et al. ^[22] combined this penalty term with the low-dimensional labeled manifold and proposed a multi-label feature selection method based on robust, flexible, and sparse regularization (RFSFS). Similarly, Li et al. ^[23] devised a highly sparse paradigm and combined it with a linear mapping between samples to real labels, as well as low redundancy constraints. Thus, a multi-label feature selection technique with high-sparsity, personalized and low-redundancy shared common features is proposed. However, since the duality of real labels does not apply in linear mapping, most scholars have constructed pseudo-label space that is suitable for linear mapping through the use of samples or real labels. In addition, due to the rapid development of manifold learning, it has been widely applied in many fields, such as cooperative clustering ^[24], feature selection algorithms, and dimensionality reduction ^[25,26,27]. In a feature selection model, manifold learning can constrain the consistency of topological structures between two spaces, and scholars often introduce manifold learning into multi-label feature selection models to improve the performance of the models.

By combining linear mapping and manifold learning, Zhang et al. ^[28] developed a manifold regularized discriminative feature selection technique for multi-label learning (MDFS). The model utilizes the manifold structure of the sample manifold and the low dimensional manifold of the label. The Frobenius norm distance between the real-label matrix and the pseudo-label matrix is used to ensure that the pseudo-label does not lose the properties of the real label. Finally, the sparsity of the feature weight matrix is constrained by the $L_{21}$ norm. Huang et al. ^[29] constrained pseudo-label learning by combining the HSIC and sample manifold methodology in the linear mapping. A multi-label feature selection technique (MRDM) was proposed based on manifold regularization and dependence maximization.

Hu et al. ^[30] constructed the common structure of sample space and label space by utilizing multiple linear maps and a proposed multi-label feature selection technique (SCMFS) with a shared common mode. The model maps sample space and label space to pseudo-label space by using linear mapping technique to obtain pseudo-label space through the use of a common structure of sample space and label space; furthermore, combining this with the $L_{21}$ norm sparse constraint, multi-label feature selection is realized. The specific formula of this model is as follows:

where $V$ is the pseudo-label matrix. $Q$ and $B$ are the corresponding coefficient matrices, respectively. $W$ is the feature weight matrix. $\alpha$, $\beta$ and $\gamma$ are the regularized parameters.

Similar to SCMFS, to improve the model's performance, Gao et al. ^[31] introduced sample manifolds into the shared structure model to strengthen the constraints on pseudo-labels through the sample manifolds when the pseudo-label matrix and the sample matrix have the same geometric manifold structure. A multi-label feature selection technique (SSFS) with a constrained latent structure shared term is proposed. The specific formula of this model is as follows:

where $V$ is the pseudo-label matrix; $L$ is the Laplacian matrix of $X$; $Q$ and $M$ are the corresponding coefficient matrices.

In addition, Zhang and Ma ^[32] proposed a multi-label feature selection technique (NMDG) that utilizes dynamic graph constraints to improve the model's performance and generalization ability. In this model, the learning of pseudo-labels is constrained by linear maps and the label manifold, and the learning of the feature weight matrix is constrained by the feature manifold and low-dimensional dynamic manifold of the pseudo-labels to realize feature selection. The objective function of this model is as follows:

where $F$ is the pseudo-label matrix; $b$ is the bias vector. $L_{Y}$, $L_{F^{T}}$, and $L_{X^{T}}$ are the Laplacian matrices of the label similarity matrix, the dynamic Laplacian matrix of the pseudo-label low-dimensional similarity matrix, and the Laplacian matrix of the feature similarity matrix, respectively.

In summary, RFSFS and ERSFS directly apply a linear mapping of samples to real labels to construct the model. However, the performance of the model is degraded by the fact that the binary nature of real labels does not apply to linear mapping methodology. To address this problem, models such as MRDM and SSFS models use pseudo-labels instead of real labels to construct linear maps, which increases the computational burden of the model and reduces the stability of the model. Unlike the state-of-the-art models described above, we mitigate the problem of real labels not being applicable to linear maps by constraining the mapping space of linear maps. This also avoids the problems of large computational burden and instability that the use of pseudo-labels imposes on the model. The specific technical details of the proposed model will be given in Section 3.

3.   Proposed method

This section presents a sparse multi-label feature selection technique (CRMFS) based on a constrained mapping space and manifold regularization. Moreover, the optimization solution and convergence proof of the model are also given in this section.

3.1.   Problem description

Let $X^{T} = [X_{1 \ast}, X_{2 \ast}, \cdots, X_{n \ast}, ]$ be the transpose of the sample matrix $X \in R^{n\times d}$ and $X_{i \ast} \in R^{1\times d}$ represent the $i$th row sample vector of $X$. The label matrix of $X$ is $Y = [Y_{\ast 1}, Y_{\ast 2}, \cdots, Y_{\ast m}, ]$, $Y \in R^{n\times m}$, and $Y_{\ast i} \in R^{n\times 1}$ is the $j$th column label vector of $Y$. $X$ and $Y$ jointly form multi-label data set $D = \{(X_{i \ast}, Y_{i \ast})| i = 1, 2, \cdots, n\}$, where $Y_{i \ast} \in R^{1\times m}$ is the $i$th row vector of $Y$ and represents the label vector corresponding to the $i$th sample. In addition, $Y_{ij}\in \{0, 1\}$ is the $i$th row, $j$th column member of $Y$. When $Y_{ij} = 1$, it means that the $i$th sample $X_{i \ast}$ belongs to the $j$th label $Y_{\ast j}$. When $Y_{ij} = 0$, sample $X_{i \ast}$ does not belong to the label $Y_{\ast j}$. Multi-label data set $D$ is used to construct the corresponding model to realize feature selection.

3.2.   CRMFS

The brief review in subsection 2.2 shows that both the MRDM model and the SSFS model are types of multi-label feature selection sparse models. Sparse models are generally constructed by combining linear mapping methodology and sparse constraints because of the simplicity of least squares calculation and strong interpretability. The specific formula of the objective function of the sparse model is as follows:

where $\alpha$ is the penalty parameter and $\| \ast \|_{F}$ represents the Frobenius norm. $Q \in R^{d\times m}$ is the coefficient matrix for the linear mapping. Since $Q_{i \ast}$ can represent the importance of its corresponding feature $X_{i \ast}$, $Q$ is also called the feature weight matrix. $R(\ast)$ is the penalty function of $\ast$.

In sparse models, the $L_{1}$ norm and $L_{21}$ norm are often used as penalty functions. Different penalty functions have different properties, leading to different properties of the constrained variables. For example, the $L_{1}$ norm can simultaneously guide the inter-line and intra-line sparsity of the constrained variables. The $L_{21}$ norm can guide the constrained variable inter-row sparsity and intra-row stability. In multi-label feature selection, it is more suitable to use $L_{21}$ norm constraint $Q$ sparsity than the $L_{1}$ norm to better distinguish the importance of features. Let $R(Q) = \| Q \|_{2, 1}$ and construct a sparse model that is suitable for feature selection:

where $\| Q \|_{2, 1} = \sum_{i = 1}^{d} (\sum_{j = 1}^{m} Q_{ij}^{2})^{\frac{1}{2}}$ represents the norm of $Q$.

Considering that the linear mapping has a specific deviation, in order to improve the generalization ability of the model, we introduce a bias vector $b \in R^{m\times 1}$ into the linear mapping scheme and reconstruct the sparse model to be as follows:

where $b^{T}$ is the transpose of $b$ and $v \in R^{n\times 1}$ is the column vector of all ones.

In addition, since label space is binary and unsuitable for linear mapping, many scholars tend to construct non-binary pseudo-label space instead of real-label space to realize feature selection. However, the use of pseudo-label space will increase the number of variables in the model, affecting the model's efficiency, and may lead to the loss of important label information in the process of pseudo-label construction, which affects the performance of the model. In order to avoid the adverse effects of learning pseudo-label space, we try to solve the aforementioned problem by constraining the properties of the mapping space $XQ$.

As can be ascertained from Eq (3.3), linear mapping takes sample space as the initial mapping space, so in an ideal state, mapping space $XQ$ should retain the basic geometric structure of sample space $X$ and maintain the consistency of the topological structure with sample space $X$. According to the hypothesis, if $X_{i \ast}$ and $X_{j \ast}$ have a high degree of similarity, $X_{i \ast}Q$ and $X_{i \ast}Q$ should also have a high degree of similarity. Mapping space $XQ$ and sample space $X$ should have the same basic manifold structure.

where $L_{X} \in R^{n\times n}$ is the Laplacian matrix of $S_{X} \in R^{n\times n}$ and $L_{X} = D_{X}-S_{X}$. $D_{X} \in R^{n\times n}$ is the diagonal matrix and $(D_{X})_{ii} = \sum_{j = 1}^{n} (S_{X})_{ij}$. $(S_{X})_{ij}$ is the $i$th row $j$th column element of the sample similarity matrix $S_{X}$. In this study, we chose to use the Gaussian function to learn $S_{X}$:

where the parameter $\delta \in R$. $N_{k}(\ast)$ represents the k-nearest neighbor of $\ast$, and $k$ is the nearest neighbor threshold.

Combining Eqs (3.3) and (3.4), we can obtain a sparse multi-label feature selection model with manifold constraints:

where $\beta$ is the manifold regular term parameter.

In addition, it can be ascertained from Eq (3.3) that the target space of a linear mapping is label space, but due to the binarity of label space, sample space cannot be well mapped to label space. However, considering that the use of pseudo-label space will increase the model complexity and reduce the robustness of the model, the mapped space should have the primary information of the label space. It can remove irrelevant or redundant interference information. To solve this problem, the HSIC ^[13] is adopted to constrain the correlation between mapping space and label space. The specific formula is as follows:

where $H \in R^{n\times n}$ is the center matrix and $V_{\ast} \in R^{\ast \times \ast}$ is the identity matrix.

Combining Eqs (3.6) and (3.7), we can obtain the objective function of the CRMFS model:

where $\gamma$ is the regularization parameter. Since Eq (3.7) does not converge without any constraint and in Eq (3.8), the other terms have become the constraints of the fourth term, it is not required for constraint $(XQ)^{T}XQ = V$ to be added in Eq (3.8).

In addition, for any matrix $A$, $\| A\|_{F}^{2} = tr(A^{T}A)$. In addition, due to the non-smoothness of the $L_{21}$ norm, $tr(Q^{T}MQ)$ is used here to replace the $L_{21}$ norm and obtain the approximate solution of Eq (3.8); thus, the objective function of the CRMFS model can be rewritten as follows:

where $M \in R^{d\times d}$ is the diagonal matrix and $M_{ii} = \frac{1}{2\|Q_{i \ast}\|_{2}}$.

3.3.   Optimal solution

According to Eq (3.9), in the CRMFS model, the function of $b$ is given by

Take the partial derivative of the above equation with respect to $b$:

Let $\frac{\partial \Phi(b)}{\partial b} = 0$; then, we have

Combining Eqs (3.9) and (3.12), the objective function of the CRMFS model can be transformed as follows:

where $H = V_{n}-\frac{1}{n}vv^{T}$ is the center matrix.

It is easy to prove that the above equation is convex with respect to $Q$, so we find the optimal solution for $Q$ by taking its derivative. The derivative function of the above equation with respect to $Q$ is given by

Let $\frac{\partial \Phi(Q)}{\partial Q} = 0$; then, we can get the update formula for $Q$:

According to the solution process of appeal optimization, we designed and present the algorithm of the CRMFS model, as shown in Table 1.

3.4.   Proof of convergence

In this subsection, we give the theoretical proof of convergence of the CRMFS algorithm; however, before proving convergence, we need to know the following lemma:

Lemma 1 ^[33]: For any non-zero vectors $a \in R^{l \times m}$ and $c \in R^{l \times m}$, both make the following formula true:

In combination with Eq (3.12), Eq (3.8) can be transformed to be as follows:

According to Lemma 1, in iteration $t$, we have

The sum of Eq (3.18) can be derived as follows:

Further transformation yields

By combining Eqs (3.17) and (3.20), we can derive the following:

In conclusion, the convergence of the CRMFS algorithm is proved.

4.   Experiment

This section discusses a comparative experiment between CRMFS and seven advanced multi-label feature selection algorithms (RFSFS ^[22], SSFS ^[31], MRDM ^[29], SCLS ^[14], MDMR ^[15], PMU ^[16], FIMF ^[17], MFS-MCDM ^[34]). The experiment was conducted on ten classical real multi-label data sets. In the experiment, five indicators, including hamming loss, ranking loss, one-error, coverage, and average precision, were used as evaluation indicators, and the ML-KNN algorithm ^[35] was used as the representative algorithm for classification.

4.1.   Experimental settings

Ten classic multi-label data sets were collected from five fields of research, including biology, images, and text. These multi-label data sets were all taken from Mulan Library (http://mulan.sourceforge.net/datasets.html). In Table 2, we give the total number of samples, number of features, number of labels, domain, cardinality (Card), and other parameters for each experimental data set.

Experimental environment: All relevant experimental experiments included a Microsoft Windows 10 system; processor: ADM Ryzen 5 3600 6-core Processor 3.59 GHz; memory: 16.00 GB; and programming software: Matlab R2016a.

First, we discretized ^[36] all experimental data to have equal-width intervals. The smoothing and nearest-neighbor parameters in the ML-KNN algorithm were set to 1 and 10, respectively. Meanwhile, the parameters of the FIMF algorithm were also set to their default values: $b = 2$ and $Q = 10$. Second, since the Gaussian function is used in CRMFS, SSFS, MRDM, and other algorithms to learn the basic manifolds of samples or labels, the nearest neighbor parameter was set to 5; also, the label nearest-neighbor parameter for Image data was set to 3, and $\delta = 1$ was set. Additionally, we set the selected feature range to $[5, 10, 15, 20, 25, 30, 35, 40, 45, 50]$ and expressly set the selected feature range to $2\thicksim18$ for the Flags data set. Finally, we applied the grid search strategy to set all experimental algorithms' adjustment range of the regularization parameter to be $[0.001, 0.01, 0.1, 1, 10,100, 1000]$.

4.2.   Evaluation metric

In this subsection, we give the detailed meanings and formulas for hamming loss, ranking loss, one-error, coverage, and average precision measures in five multi-label classification indexes, where "$\downarrow$" means that the smaller the value of relevant indexes, the better the algorithm performance; "$\uparrow$" means that the larger the value of relevant indexes, the better the algorithm performance.

Let $D \in R^{n\times d}$ be the sample data of the training set and $Y \in R^{n\times m}$ be the corresponding label set data. $h(D_{i\ast})$ represents the binary label vector, and $rank_{i\ast}(q)$ represents the rank of the label prediction $Y_{q\ast}$.

1) Hamming loss $(\downarrow)$: Represents the percentage of misclassified labels.

where $HL \in [0, 1]$ and $\bigtriangleup$ is the symbol of symmetry difference.

2) Ranking loss $(\downarrow)$: Measure the gap between the predicted list and the actual sorted list.

where $RL \in [0, 1]$. $P = \delta(rank_{i\ast} (q)\geq rank_{i\ast} (q^{'}))$. $\delta(z)$ is the indicator function and $\overline{Y_{i\ast}}$ is the complement of $Y_{i\ast}$ on $Y$.

3) One-error $(\downarrow)$: Represents that there is no sample proportion of the predicted most relevant predicted label among the real labels.

where $OE \in [0, 1]$ and $q_{i} = arg min_{q\in [1, m]} rank_{i\ast} (q)$.

4) Coverage $(\downarrow)$: Represents the average number of steps required for the sorted labels to cover the real-label correlation set.

where $CV \in [0, m-1]$.

5) Average precision $(\uparrow)$: Represents the percentage of labels in the ranking that are more relevant than a particular label.

where $AP \in [0, 1]$.

4.3.   Results and discussion

In Tables 3–7, we show the optimal results of each experimental algorithm under the optimal parameters in the experimental range. Among them, SSFS, MRDM, and other algorithms contain multiple variables, so these algorithms results are presented as the mean values of 10 runs. Meanwhile, in Tables 3–7, we use bolding to mark the optimal results under the same index on the same data set, indicating that the experimental algorithm with the bolded results on the data has the optimal algorithm performance under this index. In addition, we denote its performance ranking under this indicator on this data set by including "()" next to each result in Tables 3–7; finally, in the last two rows of Tables 3–7, we also added two rows, i.e., "Rank" and "Average", where "Rank" represents the average ranking of the overall performance of each experimental algorithm under this index. "Average" means that on all data sets, each value is the mean value of the optimal result of the algorithm under this index.

As can be ascertained from Tables 3–7, although the performance of the CRMFS algorithm on the Scene data set is slightly inferior to that of the MRDM algorithm, the average and rank of the CRMFS algorithm are optimal under each index, which also indicates that the overall performance of the CRMFS algorithm is better than that of all comparison algorithms.

Specifically, Table 3 shows the optimal performance comparison for each experimental algorithm under the average precision index. As ascertained from Table 3, although, for the Scene data set, the performance of the CRMFS algorithm decreases by 0.023 relative to that of the MRDM algorithm, it is still superior to comparison algorithms such as SSFS and MFS-MCDM. In addition, the optimal performance of the CRMFS algorithm on other experimental data sets was always in first place, and the overall performance of the CRMFS algorithm under this index was improved by $3.2\% \thicksim 15.84\%$ relative to other comparison algorithms.

Table 4 shows the optimal performance of each experimental algorithm under the hamming loss index. As seen in Table 4, although the overall performance of the CRMFS algorithm was only improved by $0.86\% \thicksim 5.6\%$, it was still greatly improved on some data sets. For example, compared with the FIMF algorithm, the performance of the CRMFS algorithm on the Birds and Image data sets was improved by $10.76\%$ and $15.81\%$, respectively. Even on the Genbase data set, the performance of the comparison algorithm was improved by $16.67\% \thicksim 95.33\%$.

Tables 5 and 6 respectively show the optimal performance of each experimental algorithm under the one-error and ranking loss indicators. As shown in Tables 5 and 6, among the ten experimental data sets, the optimal results of the CRMFS algorithm are bolded for 8 data sets in Table 5 and 9 data sets in Table 6. From Table 6 only on the Scene data set was the performance slightly inferior to that of the MRDM algorithm. In addition, relative to that of other comparison algorithms, the overall performance of the CRMFS algorithm under the one-error and ranking loss indexes was improved by $10.17\% \thicksim 38.76\%$ and $5.14\% \thicksim 22.86\%$, respectively.

Table 7 compares the optimal performance of each experimental algorithm under the coverage index. As shown in Table 7, although the performance of the CRMFS algorithm was slightly worse than that of the MRDM algorithm on the Scene, Image, and Emotion data sets and slightly worse than that of the FIMF algorithm on the Flags data set, the optimal performance of the CRMFS algorithm consistently ranked first on other data sets. In addition, the overall performance of the CRMFS algorithm was improved by $1.38\% \thicksim 5.9\%$ for each comparison algorithm.

In order to more intuitively show the performance of each experimental algorithm in terms of the number of selected features and the performance comparison results when the same number of features is selected, we constructed plots, with the number of selected features as the abscissa and the value of each indicator as the ordinate. Figures 1–5 show the results for each evaluation metric.

Figures 1–5 show the performance comparison results for the indicators of average precision, hamming loss, one-error, ranking loss, and coverage. From the overall view of Figures 1–5, first, the CRMFS algorithm effectively solves the problem of multi-label feature selection. Second, when the number of selected features was 5, the performance of the CRMFS algorithm was in the optimal position in most cases. In addition, the performance curve for the CRMFS algorithm was always in the best or second-best position. Therefore, compared with MRDM, SSFS, and other comparison algorithms, the CRMFS algorithm has certain advantages and more effective at solving the problem of multi-label feature selection.

Specifically, it can be seen in Figure 1 that, in the cases of the data sets of Birds, Yeast, and Enron, the performance curves for the CRMFS algorithm are obviously above those for the MRDM and SSFS comparison algorithms. According to Figures 2–5, it can be seen that, in the cases of the data sets of Birds, Yeast, Image, and Enron, the performance curves for the CRMFS algorithm are lower than those for the MRDM and SSFS comparison algorithms. In addition, as shown in Figures 1–5, although the performance of the CRMFS algorithm was slightly inferior to that of the MRDM algorithm on the Scene data set, the performance of the CRMFS algorithm was still optimal on other data sets.

In conclusion, although the performance improvement of the CRMFS algorithm differed for different experimental data sets, in terms of the overall performance, the CRMFS algorithm is superior to MRDM, SSFS, and the other comparative algorithms.

4.4.   Statistical test and analysis

To further analyze and compare the overall performance of each experimental algorithm, the Bonferroni-Dunn test ($\alpha = 0.01$) ^[37] was adopted to conduct a significant difference analysis and comparison of each experimental algorithm. The comparison results are visually shown in Figure 6.

In Figure 6, the horizontal axis represents the overall performance ranking of all experimental algorithms. From left to right, the overall performance of the algorithm is getting better and better. A red line links the algorithms with no significant difference for convenience. There is a significant difference if the difference in the average ranking reaches the difference threshold ($CD$), where $CD = q_{\alpha}\sqrt{\frac{K(K+1)}{6N}}$. In this study, $q_{\alpha} = 3.590\; (K = 9, a = 0.01)$, and $CD = 4.3968\; (K = 9, N = 10)$. As shown in Figure 6, we can see that, although the CRMFS algorithm has no significant difference from the MRDM, MDMR, and MFS-MCDM algorithms, it is significantly different from other advanced comparison algorithms. Indeed, the CRMFS algorithm consistently ranked first on the far right in the overall performance ranking.

In addition, Friedman's non-parametric statistical test ($\alpha = 0.05$) ^[38] was used to analyze the experimental results of each algorithm on ten classical multi-label data sets, and the quantitative results were obtained. The specific formula of the Friedman test is as follows:

where $K$ and $N$ represent the number of algorithms and data sets in the experiment, respectively; $\chi_{F}^{2} = \frac{12N}{K(K+1)}(\sum_{i = 1}^{K}R_{i}^{2}-\frac{K(K+1)^{2}}{4})$; $R_{i}$ is the sorting value of the $i$th algorithm.

The quantitative results of the Friedman test are shown in Table 8. Friedman statistics and critical values for each indicator are given in Table 8. As seen in Table 8, Friedman's statistics for all indicators were far higher than the critical value, rejecting the null hypothesis of each evaluation indicator. It also shows that the overall performance of the CRMFS algorithm under different indicators is better than MRDM, SSFS, and the other advanced comparison algorithms.

In summary, the experimental results of the Bonferroni-Dunn test and Friedman test show that the overall performance of the CRMFS algorithm is superior to SSFS, MRDM, and the other seven advanced comparison algorithms for either a single indicator or all indicators.

4.5.   Ablation studies

In this subsection, we discuss some ablation studies that we conducted to explore the effectiveness of the modules in CRMFS. In this experiment, we set one parameter in CRMFS to 0, indicating that the module is not used; we also performed a grid search with the other two parameters in the range $[0.001, 0.01, 0.1, 1, 10,100, 1000]$ to capture the optimal results of CRMFS for the average precision metric when the first 50 features are selected. The experimental results are shown in Figure 7.

It can be seen that applying the $L_{21}$ norm constraint on the Flags data set led to a slight performance degradation of CRMFS. Upon analysis, we believe that this was caused by the fact that applying the $L_{21}$ norm constraint on the Flags data set does not result in satisfactory handling of the feature redundancy problem. In addition, we found that the performance of CRMFS under the average precision metric was significantly degraded when a module was not used, which suggests that all modules in CRMFS can effectively manage the multi-label feature selection problem.

4.6.   Parameter sensitivity analysis

After the analysis in subsections 4.3 and 4.4, we know that the CRMFS algorithm has certain advantages and better effectiveness than the seven advanced multi-label feature selection algorithms. In order to more comprehensively analyze the influence of parameter changes on the CRMFS algorithm, we designed and conducted parameter sensitivity analysis experiments on the CRMFS algorithm on the Scene data set. In the experiment, we set the range of the number of feature selections to be $[14, 28, 42, 56, 70, 84, 98]$. For the parameters $\alpha$, $\beta$, and $\gamma$ in the CRMFS algorithm, We fixed two of the parameters to equal one and made the third parameter search within the range of $[0.001, 0.01, 0.1, 1, 10,100, 1000]$ to observe the performance changes of the CRMFS algorithm for the average precision index. The experimental results are shown in Figure 8.

In Figure 8, $\#$ represents the number of features selected. According to Figure 8, we can see that the CRMFS algorithm's value for the average precision index increases with the number of selected features, so feature selection is practical. Second, we can see the optimal value range of each parameter in the CRMFS algorithm on the Scene data set, where the optimal value range of $\alpha$ was $[0.1, 0.01]$, the optimal value range of $\beta$ was $[0.01, 0.1, 1, 10]$, and the optimal value range of $\gamma$ was $[10, 100]$. In addition, the optimal range of parameters varies from data set to data set.

In conclusion, the CRMFS algorithm demonstrated strong robustness on the Scene data set, and the optimal value range of each parameter indicates that the constraint mapping space has a positive effect on the CRMFS algorithm.

4.7.   Convergence and time complexity

In order to analyze the effective convergence of the CRMFS algorithm, although we have provided the theoretical proof of the convergence of the CRMFS algorithm in Subsection 3.4, we have conducted a convergence analysis experiment for the CRMFS algorithm for more intuitive analysis.

The experiment was conducted by using six data sets: Emotion, Birds, Scene, Genbase, Image, and Enron. In the experiment, we set the value of all parameters in the CRMFS algorithm to equal 1. Additionally, we set the number of iterations to equal 50. We observed changes in the value of the objective function in each iteration. The experimental results are shown in Figure 9.

As shown in Figure 9, the CRMFS algorithm converges on all experimental data sets and proves the correctness of Section 3.4. In addition, for all experimental data sets, the convergence rate of the CRMFS algorithm was very fast, generally within ten iterations.

Finally, we performed time complexity analysis of the CRMFS algorithm. In general, on the multi-label data set $m < d$ and $m < n$. According to Table 1, in an iteration of the CRMFS algorithm, the time complexity of updating $Q$ is $O(d^{2}n)$; the time complexity of updating $M$ is $O(d^{2})$. So the total time complexity of the CRMFS algorithm is $O(td^{2}n+td^{2})$. It can be seen in Figure 9 that the CRMFS algorithm has a fast convergence speed and requires a small number of iterations. Therefore, it can be seen that the number of features $d$ and the number of samples $n$ of multi-label data sets significantly influence the time complexity of the CRMFS algorithm.

5.   Conclusions and future research

Regarding the multi-label feature selection model based on linear mapping, we have developed a multi-label feature selection model (CRMFS) based on constrained mapping space to solve the problem of poor algorithm performance; it is possible because the duality of labels does not apply to linear mapping. The sample manifold and HSIC constrain this model's mapping space. Among them, the sample manifold ensures the consistency of the mapping space and the topological structure of the sample space, and HSIC ensures high correlation between the mapping space and the label space. In addition, an optimization algorithm has been designed for the model, and the algorithm's convergence analysis, parameter sensitivity analysis, and time complexity analysis were performed. Finally, a comparative experiment comparing the CRMFS algorithm with seven advanced multi-label feature selection algorithms such as SSFS and MRDM, was conducted on ten classical multi-label data sets. Experimental results on five multi-label indexes prove the proposed algorithm's effectiveness and superiority.

In addition, it is known from the ablation studies that CRMFS has some limitations in terms of ability to manage feature redundancy or label redundancy. Therefore, in the next step of this research, we will address the redundancy problem in multi-label learning by using dynamic manifold learning and combining it with subspace learning to enhance the learning capability of the local sample structure and local label structure of the model in order to improve the generalization ability and overall performance of the model.

Use of AI tools declaration

The authors declare that they have not used artificial intelligence tools in the creation of this article.

Acknowledgments

This work was supported by the National Natural Science Foundation of China: Research on efficient cooperative intelligent optimization algorithm based on feature evaluation and dynamic complementarity (no. 12101477), Shaanxi Province Fund: Research on Super-multi-objective investment portfolio and algorithm based on fuzzy random theory (no. 2023-JC-YB-064).

Conflict of interest

The authors declare that there is no conflict of interest.