A new two-step inertial algorithm for solving convex bilevel optimization problems with application in data classification problems

1.   Introduction

Data classification is an important data mining technique with a wide variety of applications to classify the different kinds of data that exist practically in all aspects of our life. It has been recognized as a critical topic in machine learning and data mining.

We begin by reviewing the history of various mathematical models and related techniques used for this purpose. Convex bilevel optimization problem plays an important role in real-world applications. It can be applied to data classification, see for example ^[1,2,3,4]. The convex bilevel optimization problem consists of the constrained minimization problem known as the outer level,

where $\mathcal{H}$ is a real Hilbert space, $\phi : \mathcal{H} \to \mathbb{R}$ is a strongly convex differentiable function, and $\Lambda$ is a nonempty set of minimizers of the inner level given by

where $\varphi : \mathbb{R}^m \to \mathbb{R}$ is convex differentiable function such that $\nabla \varphi$ is $L_\varphi$-Lipschitzian and $\psi \in \Gamma_0(\mathcal{H})$, the set of proper lower semicontinuous convex functions from $\mathcal{H}$ to $\mathbb{R}$. Problems (1.1) and (1.2) are labeled as a bilevel optimization problem.

Furthermore, the solution of (1.2) can be restated as the problem of finding $\hat{u} \in \Lambda$ such that

Parikh and Boyd ^[5] introduced the proximal gradient technique for solving (1.3), that is, $\hat{u}$ is a solution of (1.3) if and only if $\hat{u} \in \mathfrak{F}(T)$ where $T$ is the prox-grad mapping defined by

for $t > 0$, and $\mathfrak{F}(T)$ is the set of fixed points of $T$. It is well-known that if $t \in (0, \frac{2}{L_\varphi})$, then $T$ is nonexpansive and $\mathfrak{F}(T) = \mathop {\rm{arg\;min}} \limits_{u \in \mathbb{R}^m} \{ \varphi(u) + \psi(u)\}$. We also note that the set of all common fixed points of $T_n = {\rm{prox}}_{c_n}\psi (I - s\nabla \varphi)$ is the set of minimizers of the inner level problem (1.2).

Furthermore, $u \in \mathfrak{F}(T)$ is a solution for problem (1.1) if $u$ satisfies the condition

Hereafter, we would like to give some background on iteration methods for finding a fixed point of the nonexpansive mapping $T$, that is, finding a point $u^* \in C$ such that $Tu^* = u^*$.

Let $\mathcal{H}$ be a real Hilbert space with norm $\| { \cdot } \|$ and inner product $\langle{\cdot, \cdot}\rangle$, and $C$ be a nonempty closed convex subset of $\mathcal{H}$. One of the most popular iterative methods for finding a fixed point of a nonexpansive mapping is the Mann iteration, which was first introduced by Mann ^[6]. Later, Reich ^[7] modified it to the general version

where $u_1 \in \mathcal{H}$ and $\{\lambda_n\}$ is a real sequence in $[0, 1]$. He proved the weak convergence of (1.4) under the condition $\sum_{n = 1}^\infty \lambda_n(1 - \lambda_n) = \infty.$

Later, Halpern ^[8] introduced an iterative method known as the Halpern iteration for finding a fixed point of nonexpansive mappings in real Hilbert spaces. His algorithm was given in the following form:

where $u_0, u_1 \in C$ and $\{\lambda_n\} \subset [0, 1]$. Under some condition on $\{\lambda_n\}$, he established a strong convergence theorem of (1.5) when $u_0 = 0$. Later, Reich ^[9] extended the Halpern iteration (1.5) to uniformly smooth Banach spaces.

In 1974, by modifying the Mann iteration, Ishikawa ^[10] introduced the Ishikawa iteration process as follows:

where $u_1 \in \mathcal{H}$ and $\{\lambda_n\}, \{\delta_n\} \subset [0, 1]$.

Moudafi ^[11] demonstrated a viscosity approximation method for a nonexpansive mapping in 2000, which was defined as

where $u_1 \in \mathcal{H}, \{\lambda_n\} \subset [0, 1]$, and $f$ is a contraction mapping. He proved that, under certain conditions, $\{u_n\}$ generated by (1.7) converges strongly to $x \in \mathfrak{F}(T)$.

By modification of the Ishikawa iteration, Agarwal et al. ^[12] presented the S-iteration process as follows:

where $\{\lambda_n\}, \{\delta_n\} \subset [0, 1]$ and $x_1$ is arbitrarily chosen. Furthemore, they demonstrated that the convergence behavior of the S-iteration is better than the iterations of Mann and Ishikawa.

Now, we would like to give some background on iteration methods to find a common fixed point of a countable family of a nonexpansive mapping $\{T_n\}$.

Aoyama et al. ^[13] demonstrated a Halpern type iteration

where $\{\lambda_n\} \subset [0, 1] \mbox{ and } u_1$ and $u \in C$ are arbitrarily chosen. Further, they showed that, under some condition on $\{\lambda_n\}, x_n \to x \in \bigcap\limits_{n = 1}^\infty \mathfrak{F}(T_n)$.

Thereafter, Takahashi ^[14] demonstrated the iteration process

where $\{\lambda_n\} \subset [0, 1]$, and established a strong convergence theorem of (1.10) under some constraint on $\{\lambda_n\}$.

In 2010, Klin-eam and Suantai ^[15] introduced the following algorithm:

where $\{\lambda_n\} \subset [0, 1]$ and $u_1 \in C$, and showed that, under certain conditions, $\{u_n\}$ generated by (1.11) converges strongly to a common fixed point of $T_n$.

Polyak ^[16] developed an inertial methodology for improving the convergence behavior of the method. From that time on, the inertial methodology was frequently employed to accelerate the convergence behavior of methods, such as the fast iterative shrinkage-thresholding algorithm (FISTA) defined as follows:

where $u_1 = v_0 \in \mathbb{R}^m, t_1 = 1, \mbox{ and }T = {\rm{prox}}_{\lambda g}(I-\lambda \nabla f)$ for $\lambda > 0$. FISTA was introduced by Beck and Teboulle ^[17], and they applied it to solve some image restoration problems where it was shown that the performance of FISTA was better than the existing methods in the literature.

A new accelerated viscosity algorithm (NAVA) was proposed by Puangpee and Suantai ^[18] for finding a common fixed point of $\{T_n\}$. It was defined as follows:

where $u_0, u_1 \in H, \mbox{ and } \{\sigma_n\}, \{\lambda_n\}, \{\delta_n\}$, and $\{\gamma_n\} \subset (0, 1).$ Moreover, they obtained a strong convergence theorem of (1.13) under certain control conditions.

Polyak ^[19] also highlighted how multi-step inertial methods can accelerate optimization approaches, despite the fact that neither the convergence nor the rate outcome of such multi-step inertial methods are proven in ^[19].

After that, Q. L. Dong et al. ^[20] presented the general inertial Mann algorithm as follows:

for each $n \geq 1$, where $\{\theta_n\} \subset [0, \theta], \{ \zeta_n \} \subset [0, \zeta]$ with $\theta_1 = \zeta_1 = 0$, and $\theta, \zeta \in [0, 1)$.

From here on, we would like to give a some direct methods to solve problem (1.1), namely, the Bilevel Gradient Sequential Averaging Method (BiG-SAM) and the inertial Bilevel Gradient Sequential Averaging Method (iBiG-SAM).

In 2017, Sabach and Shtern ^[21] presented the BiG-SAM process (Algorithm 1) as follows:

They showed that $u_n \to u$ where $u$ is a solution of (1.1) and (1.2).

Later, Shehu et al. ^[22] introduced iBiG-SAM (Algorithm 2) by utilizing an inertial technique with BiG-SAM as follows:

They proved a strong convergence theorem of Algorithm 2 under Assumption 1.1 as follows:

Assumption 1.1. Suppose $\{\lambda_n\}_{n = 1}^\infty \subset (0, 1)$ and $\{\epsilon_n\}_{n = 1}^\infty$ are positive sequences that satisfy the following conditions:

(1) $\lim_{n \to \infty} \lambda_n = 0$ and $\sum_{n = 1}^\infty \lambda_n = \infty.$

(2) $\epsilon_n = o(\lambda_n)$, i.e., $\lim_{n \to \infty} (\epsilon_n /\lambda_n) = 0$.

Motivated by ongoing research in this area, we are interested in introducing a new accelerated algorithm for solving convex bilevel optimization problems and applying it to solve data classification problems.

The following describes the way this paper is organized: Section 2 contains some fundamental definitions and helpful lemmas. The main results of this work are presented in Section 3. In this part, we provide a new accelerated algorithm for solving convex bilevel optimization problems and prove its strong convergence theorem. In Section 4, we also use our main finding to solve data classification problems. Finally, Section 5 contains a conclusion of our work.

2.   Materials and methods

Throughout this paper, let $C$ be a nonempty closed convex subset of real Hilbert space $\mathcal{H}$, and let $T: C \to C$ be a mapping. Let the strong and weak convergence of $\{u_n\}$ to $u \in \mathcal{H}$ be denoted by $u_n \to u$ and $u_n \rightharpoonup u$, respectively. A point $u \in C$ is said to be a fixed point of $T$ if $Tu = u$, and the set of all fixed points of $T$ is denoted by $\mathfrak{F}(T)$.

A set $C$ is said to be convex if $\alpha u +(1 - \alpha) v \in C$ for all $u, v \in C$ and $\alpha \in [0, 1]$.

Definition 2.1. Let $f : \mathcal{H} \to \bar{\mathbb{R}}$. Then, the function $f$ is convex on $C$ if

Definition 2.2. A function $f : \mathcal{H} \to \mathbb{R}$ is strongly convex with constant $\sigma > 0$ if for any $u, v \in \mathcal{H}$ and $\lambda \in[0, 1],$

Definition 2.3. For a scalar-valued function $f : \mathbb{R}^m \to \mathbb{R}$, the derivative of $f$ at $\bar{u}$ is denoted by $\nabla f(\bar{u}) \in \mathbb{R}^m$ and is defined as

A function $f$ is differentiable if it is differentiable at every $u \in \mathbb{R}^m$.

Definition 2.4. Let $f : \mathbb{R}^m \to \mathbb{R}$ be convex differentiable. The gradient of $f$ at $u$ denoted by $\nabla f(u)$, is defined by

Hereafter, we will recall some important definitions, lemmas, and propositions that will be used to prove our main results.

Definition 2.5. If there exists $\tau \geq 0$ such that

$T : C \to C$ is said to be Lipschitzian.

In the above inequality, if $0 \leq \tau < 1$, $T$ is called a contraction, and if $\tau = 1$, $T$ is called nonexpansive. It is known that $\mathfrak{F}(T)$ is closed and convex if $T$ is nonexpansive.

Definition 2.6. Let $u \in \mathcal{H}$. An element $u^* \in C$ is said to be a metric projection of $u$ on $C$ if

and $u^*$ is denoted by $P_C u$.

The function $P_C$ is called the metric projection of $\mathcal{H}$ onto $C$ and it is well-known that $P_C$ is nonexpansive. Moreover,

holds for all $u \in \mathcal{H}$ and $v \in C$. More information and properties of $P_C$ can be found in ^[23].

For finding a common fixed point of a family of nonexpansive $\{T_n\}$, we need some important conditions, one of which is the NST- condition introduced by Nakajo et al. ^[24].

Let $\{T_n\}$ and $\mathfrak{T}$ be two families of nonexpansive mappings of $\mathcal{H}$ into itself with $\varnothing \neq \mathfrak{F}(\mathfrak{T}) \subset \bigcap_{n = 1}^\infty \mathfrak{F}(T_n)$ where $\mathfrak{F}(T)$ is the set of all common fixed points of each $T \in \mathfrak{T}$. We say that $\{T_n\}$ satisfies NST- condition (Ⅰ) with $\mathfrak{T}$ if for each bounded sequence $\{u_n\}$

In particular, if $\mathfrak{T} = \{T\}$, then $\{T_n\}$ is said to satisfy NST- condition (Ⅰ) with $T$.

Definition 2.7. Let $\psi \in \Gamma_0(\mathcal{H})$ and $t > 0$. The proximitor of $t \psi$ at $v \in \mathcal{H}$, denoted by ${\rm{prox}}_{t \psi}(v)$, is defined as

The forward-backward operator $T$ of $\varphi$ and $\psi$ with respect to $t$ is denoted by $T := {\rm{prox}}_{t \psi} (I - t \nabla \varphi)$. Futhermore, if $t \in (0, 2/L_\varphi)$, where $L_\varphi$ is the Lipschitz gradient of $\varphi$, it is generally known that $T$ is nonexpansive.

The following lemma is required to prove our main results.

Lemma 2.8. ^[25,27] The following holds with $u, w \in \mathcal{H}$ and any arbitrary real number $\lambda \in [0, 1]:$

(1) $\| \lambda u + (1 - \lambda)w\|^2 = \lambda \|u\|^2 + (1 - \lambda)\|w\|^2 - \lambda(1 - \lambda)\|u - w\|^2;$

(2) $\|u \pm w\|^2 = \|u\|^2 \pm 2\langle u, w \rangle + \|w\|^2;$

(3) $\|u + w\|^2 \leq \|u\|^2 + 2\langle w, u + w\rangle.$

The following equality holds for all $u, v, w \in \mathcal{H}$ by utilizing Lemma 2.8 (1):

where $\alpha, \beta, \gamma \in [0, 1]$ with $\alpha + \beta +\gamma = 1.$

Lemma 2.9. ^[26] Let $\psi \in \Gamma_0(\mathcal{H})$, and $\varphi : \mathcal{H} \to \mathbb{R}$ be convex differentiable such that $\nabla \varphi$ is $L_\varphi$-Lipschitzian with $L_\varphi > 0$. Let $\{\mathfrak{c}_n\} \subset (0, 2/L_\varphi)$ and $\mathfrak{c} \in (0, 2/L_\varphi)$ such that $\mathfrak{c}_n \to \mathfrak{c}$. Define $T_n := {\rm{prox}}_{\mathfrak{c}_n \psi} (I - \mathfrak{c}_n \nabla \varphi)$, then $\{T_n\}$ satisfies NST-condition (I) with $T$, where $T := {\rm{prox}}_{\mathfrak{c} \psi} (I - \mathfrak{c} \nabla \varphi)$.

Lemma 2.10. ^[18] Let $T$ be a nonexpansive mapping, and $\{T_n\}$ be a family of nonexpansive mappings such that $\varnothing \neq \mathfrak{F}(T) \subset \bigcap_{n = 1}^\infty \mathfrak{F}(T_n)$. For any subsequences $\{k\}$ of $\{n\}$, if $\{T_n\}$ satisfies NST-condition (I) with $T$, then $\{T_k\}$ also satisfies NST-condition (I) with $T$.

Proposition 2.11. ^[21] Let $\phi$ be a strongly convex differentiable function from $\mathbb{R}^m$ into $\mathbb{R}$ with parameter $\sigma > 0$ such that $\nabla \phi$ is $L_\phi$-Lipschitzian. Define $T_s := I - s\nabla \phi$, where $I$ is the identity mapping. Then, $T_s$ is a contraction for all $s \leq \frac{2}{L_\phi + \sigma }$, that is

Lemma 2.12. ^[28] Let $T : \mathcal{H} \to \mathcal{H}$ be a nonexpansive mapping with $\mathfrak{F}(T) \neq \varnothing.$ Then, $I - T$ is demiclosed at zero, that is

for any sequences $\{u_n\} \in \mathcal{H}$ such that $u_n \rightharpoonup u \in \mathcal{H}$.

Lemma 2.13. ^[29,30] Let $\{p_n\}, \{\xi_n\}$ be sequences of nonnegative real numbers, $\{\alpha_n\}$ a sequence in $[0, 1]$, and $\{q_n\}$ a sequence of real numbers such that

for all $n \in \mathbb{N}.$ If the following conditions hold,

(1) $\sum_{n = 1}^\infty \alpha_n = \infty;$

(2) $\sum_{n = 1}^\infty r_n < \infty;$

(3) $\limsup_{n \to \infty} q_n \leq 0;$

then $\lim_{n \to \infty} p_n = 0.$

Lemma 2.14. ^[31] Let $\{\vartheta_n\}$ be a real sequence of numbers that does not decrease at infinity in such a way that there is a subsequence $\{\vartheta_{n_i}\}$ such that $\vartheta_{n_k} < \vartheta_{n_k +1}$ for all $k \in \mathbb{N}.$ Define the sequence $\{\pi(n)\}_{n \geq n_0}$ by

where $n_0 \in \mathbb{N}$ such that $\{j \leq n_0 : \vartheta_j < \vartheta_{j+1} \} \neq \varnothing.$ Then, the following hold:

(1) $\pi(n_0) \leq \pi(n_0 + 1) \leq \dots$ and $\pi(n) \to \infty;$

(2) $\vartheta_{\pi(n)} \leq \vartheta_{\pi(n) + 1}$ and $\vartheta_n \leq \vartheta_{\pi(n) + 1}$ for all $n \geq n_0$.

3.   Results

In this part, we propose a new accelerated algorithm for finding a common fixed point of a family of nonexpansive mappings in $\mathcal{H}$ by using the two-step inertial methodology with the viscosity approximation method. Second, we establish a strong convergence theorem under relevant conditions.

To do this, we start by introducing a new two-step inertial algorithm for estimating a solution for a common fixed point problem (Algorithm 3).

Throughout this section, let $\{T_n\}$ be a family of nonexpansive mappings on $\mathcal{H}$ into itself. Let $f$ be a $\tau$-contraction mapping on $\mathcal{H}$ with $\tau \in (0, 1)$, $\{\eta_n\} \subset (0, \infty)$, and $\{\lambda_n\}, \{\delta_n\}, \{\iota_n\} \subset (0, 1).$

Next, we prove a strong convergence theorem of Algorithm 3.

Theorem 3.1. Let $T : \mathcal{H} \to \mathcal{H}$ be a nonexpansive mapping with $\mathfrak{F}(T) \neq \varnothing$. Assume that $\varnothing \neq \mathfrak{F}(T) \subset \bigcap_{n = 1}^\infty \mathfrak{F}(T_n)$ such that $\{T_n\}$ satisfies NST-condition (I) with $T$. Let $\{u_n\}$ be a sequence generated by Algorithm 3 such that the following additional conditions hold:

(1) $\lim_{n \to \infty} \eta_n = 0,$

(2) $\lim_{n \to \infty} \iota_n = 0$ and $\sum_{n = 1}^\infty \iota_n = \infty,$

(3) $0 < a < \lambda_n$ for some $a \in \mathbb{R},$

(4) $0 < b < \delta_n < \lambda_n + \delta_n < c < 1$ for some $b, c \in \mathbb{R},$

then the sequence $\{u_n\} \to u \in \mathfrak{F}(T)$ such that $u = P_{\mathfrak{F}(T)}{f(u)}$.

Proof. Let $u \in \mathfrak{F}(T)$ such that $u = P_{\mathfrak{F}(T)}f(u).$ First, we show that $\{u_n\}$ is bounded. According to the definitions of $v_n$ and $w_n$, we obtain

and

We also know from (3.1) and (3.2) that

In accordance with Assumption (1) and the definition of $\theta_n$ and $\zeta_n$, we have

Then, positive constants $M_1, M_2$ exist such that

From (3.3), we have

where $\xi = \sup \{ \frac{1 - (1 - \tau) \lambda_n \iota_n}{\iota_n} \}$. As a result, $\{u_n\}$ is bounded. Moreover, $\{v_n\}$, $\{w_n\}$, $\{f(u_n)\}$, and $\{T_nv_n\}$ are all bounded.

Using Lemma 2.8 (2), we also have

Using Lemma 2.8 (3) and (3.4), we have

Since

and

as $n \to \infty$, there exist positive constants $M_3, M_4$ such that

It follows from (3.5) that

where $M_5 = \max\{ \sup \limits_{n} \|u_n - u\|, M_3, M_4\}$. From (3.6), we set

and

Hence, we obtain

After that, we examine the following two cases:

Case 1. Assume there is an $n_0 \in \mathbb{N}$ such that the sequence $\{\|u_n - u\|\}_{n \geq n_0}$ is nonincreasing. As a result, $\{\|u_n - u\|\}$ converges since it has boundaries from below by 0. We infer that $\sum_{n = 1}^\infty \alpha_n = \infty$, by using Assumptions (2) and (3). Then, using Lemma 2.13, we assert that

Indeed, by (3.2), we have

By Lemma 2.8 (1), (3.4), and (3.8), we have

This implies that

Assumptions (2) and (4), as well as the convergence of the sequences $\{\| {u_n - u} \|\}$ and the fact that $\theta_n \|u_n - u_{n-1}\| \to 0$ and $\left|{\zeta_n}\right| \! \cdot \! \|u_{n-1} - u_{n-2}\| \to 0$, imply that

Since $\{T_n\}$ satisfies NST-condition (Ⅰ) with $T$, we obtain

As a result of the definition of $v_n$ and $w_n$, we have

We can conclude from (3.11) and Assumption (2) that

By definition of $u_{n+1}$, we have

which implies

We can also conclude the following fact from the definition of $v_n$:

Hence,

Set

So, there is a subsequence $\{w_{n_k}\}$ of $\{w_n\}$ such that

Because $\{w_{n_k}\}$ is bounded, there must be a subsequence $\{w_{n_k'}\}$ of $\{w_{n_k}\}$ that satisfies $w_{n_k'} \rightharpoonup w \in \mathcal{H}$. We can assume $w_{n_k} \rightharpoonup w$ and (3.20) hold without losing generality.

We may conclude from (3.14) that $v_{n_k} \rightharpoonup w$, and we obtain that $w \in \mathfrak{F}(T)$ by using that fact and Lemma 2.12. Furthermore, we obtain the following fact by using $u = P_{\mathfrak{F}(T)} f(u)$ and (2.1):

Hence,

which implies $\limsup \limits_{n \to \infty} q_n \!\leq\! 0$ by using $\theta_n \|u_n - u_{n-1}\| \!\to\! 0$ and $\left|{\zeta_n}\right| \! \cdot \! \|u_{n-1} - u_{n-2}\| \!\to\! 0$.

Using Lemma 2.13, we can conclude that $u_n \to u$.

Case 2. Assume that for any $n_0$, the sequence $\{\|u_n - u\|\}_{n \geq n_0}$ is not monotonically nonincreasing. We define

So, there is a subsequence $\{\vartheta_{n_k}\}$ of $\{\vartheta_n\}$ such that $\vartheta_{n_k} < \vartheta_{n_k+1}$ for all $k \in \mathbb{N}$. We define $\pi : \{ n : n \geq n_0\} \to \mathbb{N}$, by

For any $n \geq n_0$, we have $\vartheta_{\pi(n)} \leq \vartheta_{\pi(n) + 1}$ by Lemma 2.14, that is

As in Case 1, by applying (3.23) we obtain $\delta_{\pi(n)} (1 - \lambda_{\pi(n)} - \delta_{\pi(n)}) \| v_{\pi(n)} - T_{\pi(n)} v_{\pi(n)} \|^2$

which implies

Similar to the proof in Case 1, we get

and

as $n \to \infty$, and so

As in Case 1, we then demonstrate that $\limsup \limits_{n \to \infty} \langle{f(u) - u, w_{\pi(n)} - u}\rangle \leq 0$. Set

There exists a subsequence $\{w_{\pi(t)}\}$ of $\{w_{\pi(n)}\}$ such that $w_{\pi(t)} \rightharpoonup w \in \mathcal{H}$ and

By Lemma 2.10, $\{T_{\pi(t)}\}$ satisfies NST-condition (Ⅰ) with $T$. Due to inequality (3.24), $\|v_{\pi(t)} - T_{\pi(t)}v_{\pi(t)}\| \to 0$, and we obtain

As in Case 1, we can conclude from (3.25) that $v_{\pi(t)} \rightharpoonup w$, and $w \in \mathfrak{F}(T)$. Using $u = P_{\mathfrak{F}(T)} f(u)$ and (2.1), we obtain

Then,

Since $\vartheta_{\pi(n)} \leq \vartheta_{\pi(n) + 1}$, and from (3.6) along with $(1 - \tau)\lambda_{\pi(n)}\iota_{\pi(n)} > 0$, we obtain

From $\frac{\theta_{\pi(n)}}{\lambda_{\pi(n)}}\| u_{\pi(n)} - u_{\pi(n)-1}\| \to 0, \frac{\left|{\zeta_{\pi(n)}}\right|}{\lambda_{\pi(n)}} \|u_{\pi(n)-1} - u_{\pi(n)-2}\| \to 0$, and (3.34), we obtain

and so $\|u_{\pi(n)} - u \| \to 0$ as $n \to \infty$.

This implies by (3.29) that $\|u_{\pi(n)+1} - u \| \to 0$ as $n \to \infty$. From Lemma 2.14 (2), we get $\vartheta_n \leq \vartheta_{\pi(n) + 1}$, that is,

Therefore, $u_n \to u$. □

For solving the problem (1.1), we assume the following assumptions:

Assumption 3.2. Let $\Phi$ be the set of all solutions of problem (1.1) where

(1) $\phi : \mathbb{R}^m \to \mathbb{R}$ is strongly convex with parameter $\sigma_\phi > 0,$

(2) $\phi$ is a continuously differentiable function such that $\nabla \phi$ is Lipschitz continuous with constant $L_\phi$.

For solving the problem (1.2), we assume:

Assumption 3.3. Let $\Lambda$ be a nonempty set of minimizer of problem (1.2).

(1) $\varphi : \mathbb{R}^m \to \mathbb{R}$ is convex and continuously differentiable, and $\nabla \varphi$ is Lipschitz continuous with constant $L_\varphi$,

(2) $\psi \in \Gamma_0(\mathbb{R}^m)$.

Next, we will present an algorithm (Algorithm 4) for solving problem (1.1).

Theorem 3.4. Let $\phi$ be a function satisfying Assumption 3.2, and $\varphi$ and $\psi$ be functions satisfying Assumption 3.3. Let $\{\mathfrak{c}_n\} \subset (0, \frac{2}{L_\varphi})$ and $\mathfrak{c} \in (0, \frac{2}{L_\varphi})$ such that $\mathfrak{c}_n \to \mathfrak{c}$ as $n \to \infty.$ Let $\{u_n\}$ be a sequence generated by Algorithm 4 with the same conditions as in Theorem 3.1. Then, $u_n \to u \in \Phi$.

Proof. Set $T_n = {\rm{prox}}_{\mathfrak{c}_n\psi}(I - \mathfrak{c}_n\nabla \varphi), T = {\rm{prox}}_{\mathfrak{c}\psi}(I - \mathfrak{c}\nabla \varphi)$ and $f = I - s\nabla \phi$. In addition, we know that $T_n$ and $T$ are nonexpansive mappings. We also know from Lemma 2.9 that $T_n$ satisfie NST-condition (Ⅰ) with $T$. According to Proposition 2.11, $f$ is contraction with constants $\tau = \sqrt{1 - \frac{2s\sigma L_\phi}{\sigma + L_\phi}}$ and $s \leq \frac{2}{L_\phi + \sigma }$. Theorem 3.1 clearly demonstrates that $u_n \to u \in \mathfrak{F}(T)$, where $u = P_{\mathfrak{F}(T)}f(u)$. We next claim that $u \in \Phi$. By using (2.1), we have for any $v \in \mathfrak{F}(T)$

Therefore, $u$ is a solution of problem (1.1). □

4.   Application in data classifications

In this section, we utilize our algorithm as a machine learning algorithm for data classification of Parkinson's disease and diabetes, and compare its effectiveness with BiG-SAM and iBiG-SAM.

Let $\{(x_k, t_k) \in \mathbb{R}^n \times \mathbb{R}^m : k = 1, 2, \dots, s\}$ be a training set with $s$ samples, with $x_k$ representing an input and $t_k$ representing a target. The mathematical model of single-layer feedforward neuron networks (SLFNs) is given by

where $o_k$ is an output of ELM for SLFNs, $h$ is the number of hidden nodes, $g$ is an activation function, $b_j$ is the bias, and $\alpha_j$ and $\omega_j$ are the weight vectors connecting the $j$-th hidden node with the output and input node, respectively.

The hidden-layer output matrix denoted by $\textbf{H}$, is given by

The target of standard SLFNs is to approximate these $s$ sample with zero means, that is, $\sum \limits_{k = 1}^s |o_k - t_k| = 0$. Then, there exists $\alpha_j, \omega_j$, and $b_j$ such that

We could derive the following simple equation from the $s$ equations above:

where $u = [\alpha_1^T, \cdots, \alpha_h^T]^T, \textbf{T} = [t_1^T, \cdots, t_s^T]^T$.

For solving ELM, it is necessary to calculate only the $u$ that satisfies (4.1) with random $\omega_j$ and $b_j$. If there is a pseudo-inverse $\textbf{H}^+$ of $\textbf{H}$, $u = \textbf{H}^+\textbf{T}$ is the solution of (4.1). If $H^+$ does not exists, we can obtain a solution in terms of the least squares problem, that is,

In machine learning, model fitness plays an essential role for training set accuracy. We cannot employ an overfitting model to predict unknown data; instead, we utilize the most common technique known as the least absolute shrinkage and selection operator (LASSO). It is formulated as

where $\| \cdot \|_1$ is the $l_1$-norm defined by $\|(x_1, \dots, x_n)\|_1 = \sum \limits_{i = 1}^n |x_i|$, and $\lambda > 0$ is a regularization parameter. We may simplify problem (4.3) to problem (1.2) by setting $\varphi(u) := \|\textbf{H} u - \textbf{T} \|_2^2$ and $\psi(u) := \lambda\| u \|_1$. For problem (1.1), we set $\phi := \frac{1}{2} \|u\|_2^2$ with $L_\phi = 1$ and $\sigma_\phi = 1$.

In this experiment, we aim to classify the datasets of the Parkinson's disease and diabetes from UCI and Kaggle, respectively.

Parkinson's disease dataset. ^[33] There are 195 examples in this dataset, all of which have 22 features. We classified two types of data in this dataset.

Diabetes dataset. ^[34] There are 768 examples in this set, all of which have 8 features. We classified two types of data in this dataset.

In this experiment, we establish the default settings by selecting the most advantageous choice for any parameter of each algorithm in order to reach the best level of performance, as follows:

$(1)$ For inner level: $\nabla \varphi (u) = 2\textbf{H}^T(\textbf{H}u - T)$ and $L_\varphi = \lambda_{\max}(\textbf{H}^*\textbf{H})$, the maximum eigenvalue of $\textbf{H}^*\textbf{H}$.

$(2)$ For Algorithm 1 (BiG-SAM) and Algorithm 2 (iBiG-SAM):

$(3)$ For Algorithm 4 (our algorithm):

$(4)$ For all algorithms:

● Regularization parameter: $\lambda = 10^{-5}$.

● Hidden nodes: $h = 30$.

● $n = 500, \alpha = 3$, and $s = 0.01$.

● $10$-fold cross-validation.

The following experiment uses the Parkinson's disease and diabetes disease datasets. We compare the effectiveness of Algorithms 1, 2, and 4 at the 500th iteration, as shown in Tables 1 and 2.

The results of Tables 1 and 2 reveal that Algorithm 4 provides a better accuracy for data classification than the others.

5.   Conclusions

We provide a new two-step inertial accelerated algorithm in this paper. First, we analyze the convergence behavior of this algorithm and establish the strong convergence theorem under relevant conditions. Next, we utilize our algorithm as a machine learning algorithm to solve data classification problems of some noncommunicable diseases and compare its efficacy with BiG-SAM and iBiG-SAM. We find that our algorithm outperforms BiG-SAM and iBig-SAM in terms of accuracy. In our future work, we would like to employ our proposed algorithm as a machine learning algorithm for prediction and classification of some noncommunicable diseases collected from the Sriphat Medical Center, Faculty of Medicine, Chiang Mai University, Chiang Mai, Thailand, and we also aim to build new innovations in the form of web applications/mobile applications/computer systems for data prediction and classification of noncommunicable diseases. These applications will have benefits for hospitals, communities, and citizens in terms of screening and preventing noncommunicable diseases.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

This work was partially supported by Chiang Mai University and Fundamental Fund 2024 (FF030/2567), Chiang Mai University.

The first author would like to thank CMU Presidential Scholarship for the financial support.

Conflict of interest

All authors declare no conflicts of interest in this paper.