Robust safe semi-supervised learning framework for high-dimensional data classification

1.   Introduction

With the exponential growth of computing technology and increased access to diverse data sources, the availability of information has skyrocketed. To effectively extract the relevant data from this vast pool, machine learning has emerged as a crucial tool. It offers a solution to the challenging task of extracting the desired information amidst an abundance of data. In supervised learning, a multitude of data is utilized to train a model, which is then employed to make predictions on unlabeled data. However, if the model fails to generalize adequately and there is an excess of labeled data, it may lead to overfitting. In numerous practical scenarios, there exists a considerable amount of unlabeled data alongside a subset of labeled data. However, labeling data can be a laborious and costly process. Utilizing the limited labeled data to enhance performance through learning from the vast pool of unlabeled data poses a significant challenge. Over the past decade, semi-supervised learning (SSL) has emerged as a highly effective learning framework, showcasing remarkable success in both theory and practical applications ^[1,2]. Obtaining labeled samples is often a challenging and expensive task, while acquiring unlabeled samples tends to be easier and more cost-effective in many real-world problems. Semi-supervised learning (SSL) tackles this issue by leveraging different assumptions to establish connections between labeled and unlabeled samples. Among these assumptions, the manifold assumption ^[1] has gained significant traction and is widely adopted in practice. This assumption assumes that the data resides on a low-dimensional manifold, facilitating the exploration and utilization of unlabeled data for improved learning outcomes. For instance, Belkin et al. ^[1] introduced two algorithms, namely Laplacian regularized least squares (Lap-RLS) and support vector machines (Lap-SVM), which utilize manifold regularization to effectively harness information from unlabeled samples. These algorithms have demonstrated promising results. However, recent studies have highlighted a potential concern related to the reliance on unlabeled samples, suggesting that they could be compromised and potentially diminish the performance of SSL ^[3,4,5]. This raises limitations on the practical applicability of SSL to some extent ^[6,7]. Consequently, there is a need to develop a safe semi-supervised learning (SaSSL) method that guarantees never performing worse than its supervised learning (SL) counterpart utilizing only labeled samples ^{[8,9,10,11,12]}. In recent years, numerous outstanding safe semi-supervised learning methods have been proposed ^{[13,14,15,16]}.

Extreme learning machines (ELM) ^[17,18,19] have been extensively researched as single hidden layer feedforward networks (SLFNs). ELM has gained traction in the field of machine learning due to its simple architecture, low computational requirements, and wide applicability ^[20,21,22]. Additionally, ELM addresses drawbacks associated with conventional neural networks, including issues like local minima, imprecise learning, and slow convergence. Moreover, ELM offers a unified learning framework that caters to various applications such as regression, binary, and multiclass classification ^[23]. Recently, a semi-supervised ELM algorithm based on manifold regularization has been proposed to effectively leverage both labeled and unlabeled samples ^[24]. Although the results of ELM have been promising, it is worth mentioning that existing algorithms typically employ the mean square error (MSE) criterion as the cost function, while the impact of sample noise and outliers remain understudied. In the MSE-based criteria, equal penalty is assigned to all samples. However, samples with outliers or non-Gaussian noise tend to exhibit larger errors, leading to higher penalties for these samples. Consequently, the generalization performance suffers in the presence of non-Gaussian noise or outliers. This issue is particularly critical in semi-supervised learning, where misclassifications between labeled data easily affect nearby unlabeled data. Recently, the correlation coefficient has emerged as a novel criterion to address non-Gaussian noise and outliers ^{[25,26,27,28]}. Correntropy, a local similarity measure in kernel space, has been proposed to complement this approach. It offers an effective mechanism to mitigate the impact of noise and outliers by enhancing robustness through assigning lower weights to data outside the local neighborhood. In the areas of resilient learning and signal processing, such as adaptive filtering ^[26], state estimation ^[27], and principal component analysis ^[28], the correntropy-based criterion has shown promising results ^{[29,30,31,32]}. The correlation-based criterion has been used in a number of extreme learning machine (ELM) algorithms, including ^{[33,34,35,36]}. In order to obtain robust learning performance when addressing outliers, Chen et al. ^[29] introduced the kernel mean p-power error (KMPE), a non-second-order statistical metric in kernel space. The regularized correntropy criterion (RCC) had been developed by Xing and Wang in order to help the ELM handle noise or outliers in the training data ^[30]. In parallel, Yang et al. ^[37] proposed a brand-new semi-supervised ELM technique based on the maximum correntropy criterion (MCC), known as RCC-based SSELM (RC-SSELM). According to experimental findings, RC-SSELM performs impressively in semi-supervised learning scenarios with non-Gaussian noise and outliers. Chen et al. established the Maximum Mixed Correlation Criterion (MMCC) for ELM to achieve robust generalization performance in ^[32] and presented the mixed correlation approach, which employs a combination of two or more kernel functions, to enhance the performance of ELM. A new semi-supervised ELM algorithm based on the MMCC optimization strategy was proposed by Yang et al. ^[35], which increases the algorithm's effectiveness and adaptability for managing huge and complicated outliers. For the purpose of demonstrating the efficacy of MC-SSELM, experimental results from several benchmark datasets were collected. The algorithm's superiority in terms of performance and adaptability was also shown through comparisons with a number of cutting-edge semi-supervised learning methods.

It can be observed that none of the existing semi-supervised ELM learning algorithms based on correntropy has focused on the uncertainty and potential risks of the unlabeled samples in the semi-supervised learning process. Inspired by the above research, in this paper, first of all, a robust and safe semi-supervised learning framework is proposed. Then, on the basis of this learning framework, a robust safe semi-supervised extreme machine learning (RS3ELM) framework is proposed. In more specific terms, the main contributions of this paper are as follows:

(1) Targeting uncertainty and potential risk of unlabeled samples in semi-supervised learning, a risk regularization term is constructed, which can better handle the uncertainty and potential risks, thereby improving the security and reliability of the learning algorithm.

(2) Based on the theory of correntropy, a new robust loss function is defined in the kernel space, which is called the $p$-power $C$-loss. The influence of noise points and outliers on the performance of the model can be effectively suppressed by this loss function.

(3) A robust and safe semi-supervised learning framework is established based on the risk regularization term and the $p$-power $C$-loss.

(4) A robust safe semi-supervised extreme learning machine (RS3ELM) for pattern classification is proposed based on this learning framework.

(5) The generalizing bound of RS3ELM is given by using Rademacher complexity. The fixed-point iteration method is used to solve the RS3ELM, and the computational complexity and the convergence of the algorithm are analyzed from a theoretical point of view.

(6) Experimental results show that RS3ELM performs significantly better than existing semi-supervised ELM learning algorithms on multiple benchmark datasets.

The remaining work is organized as follows. In Section 2, we review related work, including supervised learning, extreme learning machines (ELM), semi-supervised learning, and semi-supervised extreme learning (SS-ELM). In Section 3, we detail our algorithm. Experimental results and analysis are presented in Section 4. Finally, the main conclusions of this work are summarized in Section 5, and we also discuss future developments.

2.   Related work

2.1.   ELM

Let $\mathcal{T}_{l} = \{\mathit{\boldsymbol{x}}_{i}, y_{i}\}_{i = 1}^{l}$ be the training set, where $l$ is the number of training samples, $x_{i}\in\mathbb{R}^{n}$, $y_{i}\in\{-1, +1\}(i = 1, \ldots, l)$. Suppose that $f(\cdot)$ represents the decision function. Let $\mathcal{H}_{K}$ denote the reproducing kernel Hilbert space (RKHS) associated with a Mercer kernel $K$. Suppose that the general decision function $f$, the basic supervised learning framework, can be expressed as

where $\rho_{loss}(\cdot)$ is any loss function, such as the hinge loss function $\max\{0.1-y_{i}f(x_{i})\}$; $\|f\|_{\mathcal{H}}^{2}$ is the RKHS norm penalty and represents the complexity of functions in RKHS $\mathcal{H}_{K}$.

Under the assumption that $L$ is the number of neurons in the hidden layer, the output function of ELM ^[17,18] is

where $\mathit{\boldsymbol{\beta}} = [\beta_{1}, \beta_{2}, \ldots, \beta_{L}]^{T}$ is the vector of output weights between the hidden layer of $L$ nodes and the output node, $\mathit{\boldsymbol{Y}} = [y_{1}, y_{2}, \ldots, y_{l}]^{T}$, and $\mathit{\boldsymbol{H}}$ is the output matrix of the hidden layer defined as

where $h_{i}(x) = G(\mathbf{a_{i}}, b_{i}, \mathbf{x}) = \mathbf{a_{i}}\cdot\mathbf{x}+b_{i}, i = 1, \dots, L$ ($\mathit{\boldsymbol{a}}$ and $b$ can be randomly generated according to a continuous probability distribution). The primal regularization ELM framework can be expressed as

where $C$ is a penalty coefficient for the errors in the training process. Then, the output weight vector $\mathit{\boldsymbol{\beta}}$ can obtained by

where $\mathit{\boldsymbol{I}}_{L}$ is an identity matrix of dimension $L$ and $\mathit{\boldsymbol{I}}_{l}$ is an identity matrix of dimension $l$.

2.2.   SS-ELM

Let $\mathcal{T} = \mathcal{T}_{l}\cup\mathcal{T}_{u} = \{x_{i}, y_{i}\}_{i = 1}^{l} \cup\{x_{i}\}_{i = l+1}^{l+u}$ be a semi-supervised learning training dataset, where $x_{i}\in\mathbb{R}^{n}$, $y_{i}\in\{-1, +1\}$, $\mathcal{T}_{l}$ denotes labeled samples set with $l$, $\mathcal{T}_{u}$ denotes unlabeled samples set with $u$; $n = l+u$. Suppose that the general decision function $f$, the general semi-supervised learning framework can be expressed as the following optimization problem:

where $\rho_{loss}(\cdot)$ is some loss function, $\|f\|_{\mathcal{H}}^{2}$ is the RKHS norm penalty and represents the complexity of functions in RKHS $\mathcal{H}_{K}$, $\gamma_{A}$ and $\gamma_{I}$ are the nonnegative regularization parameters, and $\|f\|_{I}^{2}$ is the manifold regularizer (MR) ^[1] whose empirical form takes

where $\mathit{\boldsymbol{L}} = \mathit{\boldsymbol{D}}-\mathit{\boldsymbol{W}}$ is the graph Laplacian; $\mathit{\boldsymbol{D}}$ is the diagonal degree matrix of $\mathit{\boldsymbol{W}}$ given by $D_{ii} = \sum_{j = 1}^{l+u}W_{ij}$, $D_{ij} = 0$ for $i\neq j$; and the normalizing coefficient $\frac{1}{(l+u)^{2}}$ is the natural scale factor for the empirical estimate of the Laplace operator. The weight matrix $W$ may be defined by $k$ nearest neighbors or graph kernels as follows

where $N_k(x_j)$ denotes the data sets of $k$ nearest neighbors of $x_i$.

Consequently, the primal problem of the semi-supervised extreme learning machine (SS-ELM) by introducing a manifold regularization term into (2.3), is

where $C$ and $\lambda$ are regularization parameters, $Tr(\cdot)$ denotes the trace of a matrix. Thus, we have

where $\mathit{\boldsymbol{I}}_{l}$ is an identity matrix of dimension $l$; and $\mathit{\boldsymbol{C}}$ is a diagonal matrix whose entries are $\mathit{\boldsymbol{C}}_{jj} = \frac{C}{l_{t_{j}}}$, where $l_{t_{j}}$ is the number of training samples belonging to class ${j}$, $j = 1, 2\ldots, l$; $\mathit{\boldsymbol{I}}_{n}$ is an identity matrix of dimension $n$.

2.3.   Correntropy

In this section, the definition of correntropy is derived ^[25]. Some of its properties are briefly introduced.

Definition 1. ^[25] Given two random variables $X$ and $Y$, the correntropy is defined as:

where $E[\cdot]$ is the expectation operator, $F_{XY}(x, y)$ is the joint distribution function, and $\Phi(x) = \kappa(x, \cdot)$ is a nonlinear mapping induced by a Mercer kernel $\kappa(\cdot)$, which transforms $x$ from the original space to a functional Hilbert space (or kernel space) $\mathcal{H}$ equipped with an inner product $\langle\cdot, \cdot\rangle\mathcal{H}$ which satisfies $\langle\Phi(x), \Phi(y)\rangle_{\mathcal{H}} = \kappa(x, y)$. It is obvious that we have $V(X, Y) = E[\kappa(X, Y)]$. Throughout this article, unless otherwise noted, the kernel function is a Gaussian kernel, given by

with $\sigma$ being the kernel bandwidth.

In practice, if the given sample is $\{(x_{1}, y_{1}), (x_{2}, y_{2}), ..., (x_{m}, y_{m})\}$, but the joint probability density function between the data is unknown, the correlation coefficient can be approximated as

Property 1. ^[25] Correntropy is symmetric, expressed as $V_{\sigma}(X, Y) = V_{\sigma}(Y, X)$.

Property 2. ^[25] The value of correntropy is positive and bounded, and it is proven that the upper bound is reached if and only if $X = Y$.

Property 3. ^[25] Assume that the probability density function of the sample $\{(x_{i}, y_{i}), i = 1, ..., m\}$ is $f_{X, Y}(x, y)$. $E = Y-X$ is defined as the error random variable. $f_{E, \sigma}(e)$ is the Parzen estimate of the error probability density function from the data sample $\{(e_{i} = x_{i}-y_{i}), i = 1, ..., N\}$. Afterwards $V_{m, \sigma}(X, Y)$ is equal to the value of $f_{E, \sigma}(e)$ evaluated at point $e = 0$.

Definition 2. ^[33,34] Given two random variables $X$ and $Y$, the C-Loss $C(X, Y)$ is defined by

It holds that $C(X, Y) = 1-V(X, Y)$, so that the minimization of C-Loss is equivalent to the maximization of correlation. The MCC has recently attracted increasing attention due to its robustness to large outliers ^{[33,34,35,36]}.

3.   Main contributions

3.1.   Motivation

In this section, we systematically describe our approach. Typically, semi-supervised learning leverages a large volume of unlabeled data alongside a sparse amount of labeled data. However, research indicates that the inclusion of unlabeled samples can sometimes degrade the performance of semi-supervised classifiers due to the inherent uncertainties and risks associated with unlabeled data. Moreover, in manifold regularization-based semi-supervised learning frameworks, often only the local manifold geometry of the samples is considered, while the global information is overlooked. Our core methodology involves constructing a risk degree regularization term to assess the uncertainty and potential risk of unlabeled data during the semi-supervised learning process. Intuitively, the risk associated with unlabeled samples is minimal when they contribute positively to semi-supervised learning; conversely, the risk is high, warranting the use of a supervised classifier to predict their labels. Additionally, we introduce a novel non-second-order statistical indicator, referred to as $C_{p}$-Loss, within the kernel domain. The $C_{p}$-Loss metric is symmetrical and bounded by non-negativity, which effectively reduces the impact of outliers and noise on the model's performance. Furthermore, we propose a robust, safe, semi-supervised extreme learning machine (RS3ELM) based on this framework. We derive the generalization boundary of RS3ELM using Rademacher complexity, and the optimization of the output weight matrix in RS3ELM is performed through a analysis also covers the convergence and computational complexity of RS3ELM.

3.2.   Risk estimation of unlabeled samples

In this section, we will provide a more detailed explanation of our algorithm. Our algorithm is based on two main methods: the SL method and the collaborative representation-based classification (CRC) method. Initially, we utilize the SL method and CRC method to reconstruct the unlabeled samples. This involves using the labeled samples to train a model that can reconstruct the features of the unlabeled samples. By doing this, we aim to capture the underlying structure and patterns in the unlabeled data. Furthermore, to assess the risk of the unlabeled samples, we compare the original and reconstructed versions of the unlabeled samples. By examining how well the reconstructed samples match the original ones, we obtain a measure of the risk associated with the unlabeled data. To incorporate these risk measures into our learning framework, we define risk-based regularization terms. These regularization terms serve as constraints that guide the learning process. By embedding these terms into the semi-supervised learning framework, we ensure that the resulting model strikes a balance between supervised and semi-supervised learning approaches. Therefore, the outputs of our learning framework represent a compromise between the information obtained from the available labeled samples and the reconstruction-based risk assessment of the unlabeled samples. This approach allows us to leverage the benefits of both supervised and semi-supervised learning, resulting in a more robust and effective learning algorithm.

To begin with, we start by training an ELM classifier using the labeled samples $\mathcal{X}_{l}$. Using this trained classifier, we can then make predictions $y_{u}$ for the unlabeled samples $x_{u}$. These predictions serve as an estimate of the risk associated with each unlabeled sample. Next, we employ the collaborative representation-based classification (CRC) method to reconstruct the unlabeled samples. This involves utilizing the labeled samples $\mathcal{X}^{y_{u}}_{l}$ from the same class $y_{u}$ to reconstruct the unlabeled samples. The objective function of CRC can be defined as follows:

where $\lambda$ is a regularization parameter.

Let $L_{1}(\delta_{u}) = \frac{\|x_{u}-\mathcal{X}^{y_{u}}_{l}\delta_{u}\|^{2}}{2\sigma^{2}}+\lambda\|\delta_{u}\|^{2}$ and $L_{2}(\delta_{j}) = \frac{\|x_{u}-\mathcal{X}^{y_{j}}_{l}\delta_{j}\|^{2}}{2\sigma^{2}}-1+\exp^{\frac{-\|x_{j}-\mathcal{X}^{y_{j}}_{l}\delta_{j}\|^{2}}{2\sigma^{2}}}$. Thus, the optimization problem (3.1) can be rewritten as

Clearly, the functions $L_{1}(\delta_{j})$ and $L_{2}(\delta_{j})$ are both convex. This means that the optimization problem given by Eq (3.2) is a DC (difference of convex) programming problem, where the differentiable convex component is $L_{2}(\delta_{j})$. By utilizing DC programming techniques, we can find the optimal value of $\delta_{j}$ denoted as $\delta_{j}^{\ast}$. With this optimal value, we can reconstruct the unlabeled samples using the following expression:

Moreover, we apply the ELM classifier to classify the reconstructed samples $\bar{x}_{j}$ and obtain the predicted label $\bar{y}_{j}$.

Definition 3. Denote $y_{j}$ and $\bar{y}_{j}$ as the predictions of $x_{j}$ and $\bar{x}_{j}$, respectively. The degree of risk (RD) of the unlabeled samples can be defined as the following:

Based on Eq (3.4), when the predictions are identical ($\hat{y} = \bar{y}$), it implies that the unlabeled samples might be considered safe. In this case, as the error increases, $r_{j}$ should be reduced to minimize any potential risks. On the other hand, if the predictions are not equal ($\hat{y}\neq\bar{y}$), it indicates that the unlabeled samples could be potentially risky. Therefore, in such instances, $r_{j}$ should be elevated with increasing error to effectively address these risks.

Definition 4. Let $g(x)$ and $f(x)$ be the outputs of the supervised classifier and the semi-supervised classifier, respectively. The decision function $\Xi_{R}(f)$ aims to find a trade-off between $g(x)$ and $f(x)$ by incorporating a risk-based term. It can be expressed as follows:

Here, $r_{j}$ represents the safety degree of the unlabeled samples $x_{j}$.

Based on the above analysis, it is evident that $\Xi_{R}(f)$ plays a crucial role in determining the degree of disparity between supervised learning and semi-supervised learning.

3.3.   $C_{p}$-Loss based on correntropy

Definition 5. The $C_{p}$-Loss between two random variables $X$ and $Y$ can be defined as follows:

Here, $p > 0$ is the power parameter.

Remark 1. If $p = 1$, then the $C_{p}$-Loss will degenerate into the $C$-Loss. In other words, $C$-Loss is a special case of $C_{p}$-Loss.

Remark 2. Given $N$ samples $\{x_{i}, y_{i}\}$, the empirical $C_{p}$-Loss can be easily obtained as

The curves of our proposed loss function under different parameters are shown in Figure 1.

Our $C_p(X, Y)$ loss has the following interesting properties:

Property 4. Symmetry: $C_p(X, Y) = C_p(Y, X)$.

Property 5. Non-negative boundedness: $0\leq C_p(X, Y) < 1$. The equal sign is true if and only if $X = Y$.

Property 6. When $\sigma\rightarrow \infty$, we have

Property 7. When $p\rightarrow0$, we have

3.4.   Robust safe semi-supervised learning framework

We can define the robust safe semi-supervised learning framework as follows:

In Eq (3.10), the regularization parameters $\gamma_{A} > 0$, $\gamma_{I} > 0$, and $\gamma_{R} > 0$ are used. The first three terms in Eq (3.10) are responsible for finding the semi-supervised classifier. The last term controls the trade-off between supervised and semi-supervised learning. The objective function in Eq (3.10) possesses the following characteristics:

(1) The empirical risk is defined in the first term of Eq (3.10), which evaluates how well the model fits the trained samples.

(2) The second term of Eq (3.10) represents the structural risk, which ensures the generalization capability of the model and prevents it from overfitting.

(3) The third term of Eq (3.10) introduces a joint regularization term. This term not only utilizes discriminative information more effectively but also explores the local geometric structure of new samples to enhance the classification performance. For this joint regularization, samples that are similar on the manifold should have the same class label if they belong to the same class, or different class labels if they do not.

(4) The fourth term of Eq (3.10) incorporates risk-based regularization. This term controls the trade-off between supervised and semi-supervised learning. The choice of risk degrees determines how the unlabeled samples are utilized.

Building upon our robust safe semi-supervised learning framework, we introduce a novel approach named robust safe semi-supervised extreme learning machine (RS3ELM), which is described as follows:

In this framework, we incorporate three regularization parameters: $\lambda$, $\gamma$, and $\alpha$. These parameters contribute to the definition of semi-supervised classifiers, while the last term is responsible for ensuring a suitable balance between ELM and SS-ELM.

In this scenario, we denote $\boldsymbol{\beta}_{ELM}$ as the optimal solution obtained from ELM, $\mathit{\boldsymbol{H}}_{u}$ as the hidden layer output matrix pertaining to unlabeled samples, and $\mathit{\boldsymbol{R}}$ as a diagonal matrix with the entry $\mathit{\boldsymbol{R}}_{jj} = r_{j+l}$. The derivative of Eq (3.11) with respect to $\boldsymbol{\beta}$ is

Thus, we can get

where $\hat{\lambda} = \frac{2\sigma^{2}l\lambda}{p}$, $\hat{\gamma} = \frac{2\sigma^{2}l\gamma}{p}$, $\hat{\alpha} = \frac{2\sigma^{2}l\alpha}{p}$, $\mathit{\boldsymbol{h}}_{i}$ is the $i$-th row of $\mathit{\boldsymbol{H}}$, and $\mathit{\boldsymbol{A}}$ is a diagonal matrix with diagonal elements $A_{ii} = \varphi(e_{i}) = (1-\kappa_{\sigma}(e_{i}))^{p-1}\kappa_{\sigma}(e_{i})$.

The obtained optimal solution $\mathit{\boldsymbol{\beta}}$ can be expressed as $\mathit{\boldsymbol{\beta}} = [\mathit{\boldsymbol{H}}^{T}\mathit{\boldsymbol{AH}}+\hat{\lambda}\mathit{\boldsymbol{I}}+\hat{\gamma}\mathit{\boldsymbol{H}}_{n}^{T}\mathit{\boldsymbol{L}}\mathit{\boldsymbol{H}}_{n}+\hat{\alpha}\mathit{\boldsymbol{H}}_{u}^{T}\mathit{\boldsymbol{R}}\mathit{\boldsymbol{H}}_{u}]^{-1}(\mathit{\boldsymbol{H}}^{T}\mathit{\boldsymbol{AT}}+\hat{\alpha}\mathit{\boldsymbol{H}}_{u}^{T}\mathit{\boldsymbol{R}}\mathit{\boldsymbol{H}}_{u}\boldsymbol{\beta}_{ELM})$. However, this equation does not have a closed-form solution as the right-hand side matrix depends on the weight vector $\mathit{\boldsymbol{\beta}}$ through the error term $e_{i} = t_{i}-\mathit{\boldsymbol{h}}_{i}\mathit{\boldsymbol{\beta}}$. Therefore, it is essentially a fixed-point equation, and the true optimal solution can be obtained using a fixed-point iterative algorithm.

Given a test set $\mathit{\boldsymbol{X}}_{new}$, we can calculate its corresponding hidden layer output matrix $\mathit{\boldsymbol{H}}_{new}$, and the prediction result can be obtained as follows:

Based on the above discussion, our algorithm will be presented in Algorithm 1.

3.5.   Convergence and computational complexity analysis

3.5.1.   Convergence analysis

Theorem 1. ^[38] Based on the robust estimation given by Huber, if the loss function $\Phi(Z)$ is called a robust loss function, then it needs to satisfy the following:

● $\Phi(Z)\geq0$ and $\Phi(Z) = 0$;

● $\forall Z\in \mathbb{R}$, $\Phi(Z) = \Phi(-Z)$;

● $\forall Z\geq0$, $\Phi^{\prime}(Z)\geq0$;

● $\Phi(Z)$ is second-order differentiable in $\mathbb{R}^{+}$, and $\Phi^{\prime\prime}(0^{+}) > 0$;

● $\Phi(\sqrt{Z})$ is concave in $\mathbb{R}^{+}$;

then there exists a convex function $\Psi(q)$ such that

When $Z$ is fixed, a minimum solution $q^{\ast}$ exists in the right-hand side of (3.16):

where $q^{\ast} = \left\{ \begin{aligned} \frac{\Phi^{\prime}(Z)}{Z} & & Z > 0, \\ \Phi^{\prime\prime}(0^{+}) & & Z = 0, \\ \frac{\Phi^{\prime}(-Z)}{-Z} & & Z < 0. \end{aligned} \right.$

Proposition 1. For the function $G(z) = \exp\left(\frac{-\|z\|^2}{2\sigma^2}\right)$, it is possible to find a convex function $\psi: \mathbb{R} \to \mathbb{R}$ such that the following equation holds:

In the above equation, $\mathbb{R}^-$ represents the set of negative real numbers, and $\sup(\cdot)$ denotes the operation of finding the supremum. Additionally, for a fixed $z$, the supremum is attained at $\vartheta = -G(z)$.

To simplify the expression, let us define the objective function of the proposed method as follows:

According to Proposition 1, we can rewrite (3.19) as

where $\vartheta_{i} = (\vartheta_{1}, \vartheta_{2}, \cdots, \vartheta_{l})$.

Thus, given a fixed $\mathit{\boldsymbol{\beta}}$, the local optimal solution of (3.20) is

where $e_{i} = t_{i}^{T}-h_{i}\mathit{\boldsymbol{\beta}}$, $i = 1, 2, \cdots, l$.

Then, we have $\Gamma(\mathit{\boldsymbol{\beta}}) = \min_{\vartheta}\Gamma(\mathit{\boldsymbol{\beta}}, \vartheta)$ and $\mathit{\boldsymbol{\beta}}$ can be derived as

$Const$ is a constant real number greater than zero, $\sum_{i = 1}^{l}(\varphi(e_{i})\mathit{\boldsymbol{h}}_{i}^{T}\mathit{\boldsymbol{h}}_{i}\mathit{\boldsymbol{\beta}})+\hat{\lambda}\mathit{\boldsymbol{\beta}}+\hat{\gamma}(\mathit{\boldsymbol{H}}_{n}^{T}\mathit{\boldsymbol{L}}\mathit{\boldsymbol{H}}_{n}\boldsymbol{\beta}) +\hat{\alpha}\mathit{\boldsymbol{H}}_{u}^{T}\mathit{\boldsymbol{R}}(\mathit{\boldsymbol{H}}_{u}\mathit{\boldsymbol{\beta}}) = \sum_{i = 1}^{l}(\varphi(e_{i})t_{i}\mathit{\boldsymbol{h}}_{i}^{T})+\hat{\alpha}\mathit{\boldsymbol{H}}_{u}^{T}\mathit{\boldsymbol{R}}\mathit{\boldsymbol{H}}_{u}\boldsymbol{\beta}_{ELM}, \nonumber$ where $\varphi(e_{i}) = (1-\kappa_{\sigma}(e_{i}))^{p-1}\kappa_{\sigma}(e_{i})$. Therefore, the output weight vector $\beta$ can be calculated by the following formula

where $\mathit{\boldsymbol{A}}$ is a diagonal matrix and diagonal elements for $A_{ii} = \varphi(e_{i}) = (1-\kappa_{\sigma}(e_{i}))^{p-1}\kappa_{\sigma}(e_{i})$.

Assume that at iteration $k$, there is

Therefore, we can obtain

For a fixed $\mathit{\boldsymbol{\beta}}^{k}$, $\hat{\Gamma}(\beta^{k}, \vartheta^{k+1})$ is the minimum value with respect to $\vartheta$. Therefore,

Therefore, we have

3.5.2.   Computational complexity analysis

In the RS3ELM algorithm, the computational complexity primarily resides in the matrix update ($\mathit{\boldsymbol{R}}$), weight vector update ($\beta$), graph Laplacian construction, and the computation of the nearest neighbors ($k$). The main computational cost of each iteration is determined by the $\mathcal{O}(L^{3})$ computation required to update $\beta$.

Thus, the approximate computational complexity of the RS3ELM algorithm is given by $\mathcal{O}(T\cdot(L^{3}+(l + u)^{2}\log(l + u)+2(l+u)^{2})))$. Based on our experimental results, a satisfactory choice for the iteration count $T$ is $10$.

3.6.   Generalization bound of RS3ELM

In this section, we provide theoretical bounds for the generalization error of the proposed method based on the Rademacher complexity. Let us define the empirical Rademacher complexity of function class $\mathcal{F}$, denoted by $\hat{E}_n(\mathcal{F})$, for a sample $\{x_1, \ldots, x_n\}$ generated by a distribution $D$, where $\mathcal{F}$ is a real-valued function class with domain $\mathcal{X}$.

The expectation is taken over $\mathit{\boldsymbol{\sigma}} = (\sigma_{1}, \sigma_{2}, \ldots, \sigma_{n})^{T}$, where $\sigma_i \in \{-1, +1\}$ are independent uniform random variables. The Rademacher random variables satisfy $\mathit{\boldsymbol{P}}\{\sigma_i = -1\} = \mathit{\boldsymbol{P}}\{\sigma_i = +1\} = \frac{1}{2}$.

Based on the above, we can express the Rademacher complexity of $\mathcal{F}$, denoted by $E_n(\mathcal{F})$, as follows:

Here, the expectation is taken over the joint distribution of $\mathit{\boldsymbol{x}}$ and $\mathit{\boldsymbol{\sigma}}$.

Theorem 2. ^[37] Let $\mathcal{F}:\mathcal{Z} = \mathcal{X}\times \mathcal{Y}\mapsto[-1, +1]$ be a class of functions. Let $n$ samples be drawn independently from a distribution $D$. Then, with a probability of at least $1-\theta$, for every $f\in\mathcal{F}$, we have

where $Err(f)$ and $\hat{Err}(f)$ denote the expected error and the empirical error of $f$, respectively.

If we can compute the empirical Rademacher complexity $\hat{E}_{n}(\mathcal{F})$ for the function class $\mathcal{F}$, then this bound is applicable to a wide range of learning algorithms. Additionally, for kernelized algorithms, it is relatively simple to bound the empirical Rademacher complexity using the trace of the kernel matrix.

Theorem 3. ^[37] For a kernel $\mathit{\boldsymbol{K}}:\mathcal{X}\times \mathcal{X}\mapsto\mathbb{R}$ and a sample $\{x_{1}, \ldots, x_{n}\}$ from $\mathcal{X}$, the empirical Rademacher complexity of the class $\mathcal{F}(B)$ satisfies the following condition: If the norm $\|f\|_{\mathcal{H}}\leq B$, then

If for all $x\in\mathcal{X}$, we have $\mathit{\boldsymbol{K}}(x, x)\leq T^{2}$ and $\mathit{\boldsymbol{K}}$ is a standard kernel, the inequality above can be rewritten as

Theorem 4. (Generalization bound of RS3ELM) Suppose $Err(f)$ and $\hat{Err}(f)$ are the expected error and the empirical error of RS3ELM, $l$ and $u$ are the sets of labeled and unlabeled examples, respectively, and $n = l+u$. If the unlabeled sample is risk-free and $\mathit{\boldsymbol{K}}(x, x)\leq T^2$, then for every $\theta\in(0, 1)$, with probability at least $1-\theta$, the generalization error of RS3ELM is given by

Proof. To utilize Theorem 3, we need to determine the value of $B$ in the inequality (3.33), which serves as an upper bound for $\|f\|^{2}_{\mathcal{H}}$. Assuming the unlabeled samples are safe, the objective function of RS3ELM in the regenerative Hilbert space can be reformulated as

Assuming that $f^{\ast} = \arg\min_{f\in\mathcal{H}}\Phi(f)$ is the solution to (3.36), we have $\Phi(f^{\ast})\leq\Phi(\mathit{\boldsymbol{0}})$. As a result, we can derive additional information

Let $\pi_1 < \pi_2 < \cdots < \pi_r$ be the non-zero eigenvalues of the Laplacian matrix $\mathit{\boldsymbol{L}}$, where $r$ is the rank of $\mathit{\boldsymbol{L}}$. In terms of the eigenvalues, we have the inequality $\pi_1 < \pi_2 < \cdots < \pi_r$. This inequality represents the relationship between the minimum eigenvalue $\pi_1$ and the maximum eigenvalue $\pi_r$ of the Laplacian matrix.

and

By combining the inequalities (3.37) and (3.39), we can derive the following resulting inequalities

Hence, to restrict the search range of $f^{\ast}$, we can confine it within a radius $R = \sqrt{\frac{\Phi(\mathit{\boldsymbol{0}})}{\left(\frac{\lambda}{2}+\frac{\gamma\pi_{1}}{2n^{2}}\right)}}$ inside the ball. Let $\mathcal{H}_R = \{f\in\mathcal{H}: \|f\|_{\mathcal{H}}\leq R\}$ denote the sphere with radius $R$ in the regenerative Hilbert space $\mathcal{H}$, and the generalized version of RS3ELM can be expressed as:

where

Suppose the output weight vector $\mathit{\boldsymbol{\beta}} = (0, 0, \ldots, 0)^{T}$ is chosen such that the last two terms in Eq (3.36) become zero. Then

By substituting (3.43) into (3.40), we can get

The inequality (3.44) can be equivalently written as

Obviously, the radius $R$ is

By substituting Eq (3.46) into Eq (3.42), we can derive the following inequality.

By substituting Eq (3.47) into Eq (3.41), we can easily obtain an approximation for the generalization error bound of RS3ELM:

□

4.   Experiments

4.1.   Experimental setup

In this section, we conducted experiments on nine benchmark datasets to validate the effectiveness of the RS3ELM algorithm. We compared it with the following algorithms: SS-ELM ^[24], Lap-SVM ^[1], Lap-RLS ^[2], RCSSELM ^[36], and MC-SSELM ^[35]. For the experiments, we used the following parameters:

● SS-ELM: The number of hidden nodes $L$ was chosen from the set $\{100,500, 1000, 2000\}$. The regularization parameters $C$ and $\lambda$ were chosen from the set $\{10^{-5}, 10^{-3}, 10^{-1}, 10^{1}, 10^{3}, 10^{5}\}$.

● RC-SSELM: The regularization parameters $C$, $\lambda$, and the number of hidden nodes $L$ were chosen within the same range as SS-ELM. The Gaussian kernel parameter $\sigma$ was chosen from the set $\{1, 3, 5, 7, 9\}$.

● Lap-SVM and Lap-RLS: The Gaussian kernel parameter $\sigma$ was chosen from the set $\{2^{-10}, 2^{-6}, 2^{-2}, 2^{2}, 2^{6}\}$. The regularization parameters $\gamma_{I}$ and $\gamma_{A}$ were chosen from the set $\{10^{-6}, 10^{-4}, 10^{-2}, 10^{-1}, 10^{0}, 10^{1}, 10^{2}\}$.

● MC-SSELM: The regularization parameters $C$, $\lambda$, and the number of hidden nodes $L$ were chosen within the same range as SS-ELM. The Gaussian kernel parameters $\sigma_{1}$ and $\sigma_{2}$ were chosen from the set $\{2^{-5}, 2^{-3}, 2^{-1}, 2^{1}, 2^{3}, 2^{5}\}$. The maximum number of iterations $K$ was set to 50, and the tolerable error $\varepsilon$ was set to $10^{-3}$. The variable center values were $\{-3, -1, 1, 3\}$, and the weight parameter $\alpha$ was chosen from the set $\{0.1, 0.2, 0.3, 0.4, 0.5\}$.

● RS3ELM: The maximum number of iterations $K$ was set to 50, and the tolerable error $\varepsilon$ was set to $10^{-3}$. The Gaussian kernel parameter $\sigma$ was chosen from the set $\{2^{-10}, 2^{-6}, 2^{-2}, 2^{2}, 2^{6}\}$. The regularization parameters $\gamma$, $\lambda$, and $\alpha$ were chosen from the set $\{10^{-6}, 10^{-4}, 10^{-2}, 10^{-1}, 10^{0}, 10^{1}, 10^{2}\}$.

To ensure a fair comparison, we performed a grid search to find the optimal parameters for all algorithms, and the algorithm with the best performance was selected.

To assess the classification performance of all the algorithms, we used the traditional accuracy index (ACC). The ACC is defined as follows:

In this equation, $TP$ represents the number of true positives, $TN$ represents the number of true negatives, $FN$ represents the number of false negatives, and $FP$ represents the number of false positives. A higher ACC value indicates a better model performance. Additionally, to compare the computational time of the algorithms, we recorded their running time, including both training and testing, on all the datasets used.

In the experiment, the main criterion for evaluating the performance of all the algorithms was the average classification accuracy, obtained through Monte Carlo cross-validation (MCCV) ^[39] and grid search. All the datasets used were approximately balanced, and the following steps were taken:

● Random division: The datasets were randomly divided into training sets and test sets, with a ratio of approximately 7:3.

● Repetition: This process of random division was repeated 10 times.

● Averaging: The results from the 10 repetitions were averaged to obtain a more reliable measure of performance.

● Normalization: To ensure fair comparison, all the datasets were normalized to the interval $[0, 1]$. Details of the datasets are presented in Table 1.

● Recording: The average classification accuracy, denoted by ACC, was obtained as the average of the 10 test results.

By following these steps, we aimed to obtain objective experimental results that accurately assessed the performance of the algorithms. We implemented the algorithms used in the experiments using MATLAB scripts. To ensure a fair comparison, we employed the MATLAB toolbox for quadratic programming (QP) to solve all the quadratic programming problems embedded in the algorithms. The implementation was carried out on a PC with the following specifications: Intel(R) Core(TM) i7-8700 processor (3.20 GHz) and 16 GB of RAM. The operating system used was Windows 10, and MATLAB version 2014a was employed for the implementation purpose.

4.2.   Experiments on synthetic dataset

In this section, we aim to demonstrate the robustness of our method by conducting experiments on a two-dimensional "XOR" dataset. The dataset is generated by perturbing points from two intersecting lines, $\mbox{Class 1:}\ y_{i} = 0.7x_{i}+\eta$ and $\mbox{Class 2:}\ y_{i} = -0.3x_{i}+\eta$. Each instance is randomly assigned Gaussian noise, following a normal distribution with mean 0 and standard deviation 0.2. Both classes consist of 1000 sample points. We evaluate our method on both the original "XOR" dataset and a contaminated version, which includes a few outliers. The experimental results are presented in Table 2.

The experimental results indicate that all methods perform exceptionally well in classifying the synthetic dataset without outliers. Among the algorithms, Lap-SVM exhibits the highest classification performance. However, the scenario changes when the dataset is contaminated with outliers. In this case, we observe that Lap-SVM, Lap-RSL, and SS-ELM demonstrate significantly poorer performance compared to the other algorithms. The classification accuracies of Lap-SVM, Lap-RSL, SS-ELM, RC-SSELM, MC-SSELM, and RS3ELM is presented in Table 2. It is evident from the experiments that our method, RS3ELM, outperforms the other five classifiers in terms of accuracy when classifying a synthetic dataset with outliers. However, in terms of training time, RS3ELM does not exhibit a notable advantage over the other classifiers. These results effectively highlight the robustness of RS3ELM. RS3ELM are more time-consuming: Despite having similar accuracy, RS3ELM requires more computational time compared to MC-SSELM. This could mean that the internal algorithms or processes involved in RS3ELM are more complex or require more iterations/operations for convergence.

4.3.   Experiment without outliers

In this section, we evaluate the performance of our proposed method in comparison with other algorithms in a noise-free environment. Table 3 presents the experimental results for all algorithms using the optimal parameters. For Lap-SVM, Lap-RSL, SS-ELM, RC-SSELM, and MC-SSELM algorithms, we refer to the literature ^[35] for their experimental settings and results. Please consult the literature ^[35] for more detailed information. From Table 3, it is evident that our proposed method achieves comparable performance to the other five algorithms in the noise-free experiment. In general, the ELM-based methods demonstrate better performance than the SVM-based methods in most cases.

4.4.   Robustness analysis

To assess the efficacy of our proposed approach, we utilized nine benchmark datasets sourced from the UCI machine learning repository: http://archive.ics.uci.edu/ml/datasets.html. In order to ensure consistency, we standardized these datasets, thereby constraining the feature values to the range of $[0, 1]$. Subsequently, we conducted two types of experiments on these standardized datasets.

In this section, we initially conducted experiments on datasets containing 10% and 30% outliers to assess the robustness of the proposed method. These outliers were generated by randomly selecting 10% and 30% of the labeled training samples and applying the label inversion technique. (In robustness testing, label inversion is used to test if a model can withstand label noise. For example, flipping a certain percentage of labels randomly in the training data helps to determine how well the model can generalize despite noisy conditions.) The experimental settings and results of Lap-SVM, Lap-RSL, SS-ELM, RC-SSELM, and MC-SSELM algorithms were adopted from the literature ^[35]. The optimal parameters were used in these experiments, and the results are presented in Tables 4 and 5. Observing the tables, it is evident that the performance of all the algorithms declined as the label noise increased. Moreover, the model utilizing the entropy loss function outperformed other methods, indicating its effectiveness in handling outliers.

In order to further validate the robustness of the proposed method, this study also conducted experiments in a characteristic noise environment. The characteristic noise dataset was generated by randomly selecting 30% of the characteristic values of each sample and assigning them a value of 0, thus creating a new experimental dataset. The experimental settings and results of Lap-SVM, Lap-RSL, SS-ELM, RC-SSELM, and MC-SSELM algorithms were obtained from the literature ^[35]. The experimental settings and parameter choices were consistent with the previous experiments. The results of this experiment are presented in Table 6 using the optimal parameters. From Table 6, it can be observed that the proposed method performs comparably to other related algorithms. To summarize, the proposed method in this paper demonstrates effectiveness in terms of robustness and significantly enhances the performance of the model.

4.5.   Convergence analysis

To evaluate the convergence of the proposed algorithm, experiments are conducted on multiple datasets including wine, breast cancer, diabetic, and G50C datasets. The optimal parameters are selected, and the experimental procedure is consistent with the noise experiment. The learning outcomes are presented in Figure 2. As observed from Figure 2, it is evident that the objective function value of RS3ELM progressively decreases with each iteration, converging to a stable value in fewer than 10 iterations. Hence, the proposed algorithm RS3ELM demonstrates convergence.

4.6.   Sensitivity analysis of parameters

The RS3ELM algorithm incorporates several parameters, namely $p$ and $\sigma$ in the C$_{p}$-Loss function, as well as the tradeoff coefficients for three terms in (3.11), which are $\lambda$, $\gamma$, and $\alpha$. For simplicity, we assume that $\gamma$, $\lambda$, and $\alpha$ are equal in value. To assist in parameter selection, we examine the impact of $p$, $\sigma$, $\lambda$, $\gamma$, and $\alpha$ on the performance of RS3ELM.

In our experimental analysis, we investigated the impact of $p$ and $\alpha$ on the performance of RS3ELM, while keeping $\sigma = 2^{-2}$, $\lambda = 10^{-2}$, and $\gamma = 10^{-1}$. We conducted the experiment by varying $\alpha$ and $p$ within the sets $\{10^{-4}, 10^{-2}, 10^{-1}, 10^{0}, 10^{1}, 10^{2}\}$ and $\{0.5, 1.0, 1.5, 2.0, 2.5, 3.0\}$, respectively. The accuracy results in relation to the parameters $p$ and $\alpha$ are presented in Figure 3. Although the classification performance of the wine, Breast Cancer, Diabetic, and g50c datasets were minimally influenced by changes in $p$ and $\alpha$, we can still make certain conclusions about the optimal values of these parameters based on Figure 3. Generally, outliers tend to affect the determination of hyperplanes. However, RS3ELM can effectively mitigate this influence by adjusting the parameter $p$, indicating that by controlling $p$, RS3ELM becomes less sensitive to outliers.

5.   Conclusions

In this research paper, we propose a robust safe framework for semi-supervised learning that ensures the reliable utilization of unlabeled samples and achieves resistance to noise and outliers. To implement this framework, we introduce a robust safe semi-supervised extreme learning machine (RS3ELM), which is solved using a fixed-point iterative algorithm. We theoretically analyze the computational complexity and convergence of the RS3ELM and provide a generalizing error bound based on the Rademacher complexity. Our experimental results on multiple datasets validate the robustness and classification accuracy of RS3ELM when compared to similar methods. Consequently, our proposed approach can be effectively applied to robust classification problems. It is important to note that our method focuses solely on binary classification tasks. Investigating the extension of this method to other learning tasks without compromising the model's classification accuracy and robustness is a valuable direction for future research.

Author contributions

Jun Ma and Xiaolong Zhu: Algorithm development, software creation, numerical example preparation, original draft writing, review and editing of the manuscript. All authors have read and approved the final version of the manuscript for publication.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

This work was supported in part by the Fundamental Research Funds for the Central Universities of North Minzu University (No.2021JCYJ07 and No.2023ZRLG01), in part by the National Natural Science Foundation of China (No.62366001 and No.12361062), in part by the Key Research and Development Program of Ningxia (Introduction of Talents Project) (No. 2022BSB03046), and in part by the Natural Science Foundation of Ningxia Provincial (No. 2023AAC02053).

Conflict of interest

The authors declare that they have no conflict of interest and no relevant financial or non-financial interests to disclose.