A new accelerated algorithm with a linesearch technique for convex bilevel optimization problems with applications

Adisak Hanjing; Panadda Thongpaen; Suthep Suantai; Adisak Hanjing; Panadda Thongpaen; Suthep Suantai

doi:10.3934/math.20241088

AIMS Mathematics

2024, Volume 9, Issue 8: 22366-22392. doi: 10.3934/math.20241088

Previous Article Next Article

Research article

A new accelerated algorithm with a linesearch technique for convex bilevel optimization problems with applications

1.
Department of Science and Mathematics, Rajamangala University of Technology Isan Surin Campus, Surin 32000, Thailand
2.
Department of Mathematics, Faculty of Science, Chiang Mai University, Chiang Mai 50200, Thailand
3.
Research Center in Optimization and Computational Intelligence for Big Data Prediction, Department of Mathematics, Chiang Mai University, Chiang Mai 50200, Thailand

Received: 21 April 2024 Revised: 01 July 2024 Accepted: 09 July 2024 Published: 18 July 2024
MSC : 65K05, 90C25, 90C30

We considered a convex bilevel optimization problem when the outer level problem was to find a minimizer of a strongly convex function over the set of solutions of the inner level problem which was in the form of minimization of the sum of a convex differentiable function and a nonsmooth convex function. In this work, we proposed a novel accelerated algorithm by employing both linesearch and inertial techniques for solving a convex bilevel optimization problem. Then, we proved the strong convergence of the sequence generated by our proposed algorithm to an optimal solution of the convex bilevel optimization problems without the continuity assumption on the gradient of the objective function. Moreover, we presented the convergence behavior of the proposed method by some numerical experiments addressing image restoration problems and data classification problems with least squares constraints. Finally, the performances of the restorative image and the data classification of the proposed method were compared with other existing algorithms in the literature. According to the experiment, our proposed algorithm had a better convergence behavior than the others in the literature.
- accelerated algorithm,
- convex bilevel optimization problem,
- inertial step,
- linesearch technique,
- data classification,
- image restoration
Citation: Adisak Hanjing, Panadda Thongpaen, Suthep Suantai. A new accelerated algorithm with a linesearch technique for convex bilevel optimization problems with applications[J]. AIMS Mathematics, 2024, 9(8): 22366-22392. doi: 10.3934/math.20241088

Related Papers:

Abstract

We considered a convex bilevel optimization problem when the outer level problem was to find a minimizer of a strongly convex function over the set of solutions of the inner level problem which was in the form of minimization of the sum of a convex differentiable function and a nonsmooth convex function. In this work, we proposed a novel accelerated algorithm by employing both linesearch and inertial techniques for solving a convex bilevel optimization problem. Then, we proved the strong convergence of the sequence generated by our proposed algorithm to an optimal solution of the convex bilevel optimization problems without the continuity assumption on the gradient of the objective function. Moreover, we presented the convergence behavior of the proposed method by some numerical experiments addressing image restoration problems and data classification problems with least squares constraints. Finally, the performances of the restorative image and the data classification of the proposed method were compared with other existing algorithms in the literature. According to the experiment, our proposed algorithm had a better convergence behavior than the others in the literature.

References

[1]	P. Thongpaen, W. Inthakon, T. Leerapun, S. Suantai, A new accelerated algorithm for convex bilevel optimization problems and applications in data classification, Symmetry, 14 (2022), 2617. https://doi.org/10.3390/sym14122617 doi: 10.3390/sym14122617
[2]	P. Thongsri, B. Panyanak, S. Suantai, A new accelerated algorithm based on fixed point method for convex bilevel optimization problems with applications, Mathematics, 11 (2023), 702. https://doi.org/10.3390/math11030702 doi: 10.3390/math11030702
[3]	A. Dhara, J. Dutta, Optimality conditions in convex optimization: a finite-dimensional view, Boca Raton: CRC Press, 2011. https://doi.org/10.1201/b11156
[4]	S. Sabach, S. Shtern, A first order method for solving convex bilevel optimization problems, SIAM J. Optimiz., 27 (2017), 640–660. https://doi.org/10.1137/16M105592X doi: 10.1137/16M105592X
[5]	Y. Shehu, P. T. Vuong, A. Zemkoho, An inertial extrapolation method for convex simple bilevel optimization, Optim. Method. Softw., 2021 (36), 1–19. https://doi.org/10.1080/10556788.2019.1619729
[6]	B. T. Polyak, Some methods of speeding up the convergence of iteration methods, Comp. Math. Math. Phys., 4 (1964), 1–7. https://doi.org/10.1016/0041-5553(64)90137-5 doi: 10.1016/0041-5553(64)90137-5
[7]	H. K. Xu, Viscosity approximation methods for nonexpansive mappings, J. Math. Anal. Appl., 298 (2004), 279–291. https://doi.org/10.1016/j.jmaa.2004.04.059 doi: 10.1016/j.jmaa.2004.04.059
[8]	P. C. Duan, Y. Q. Zhang, Alternated and multi-step inertial approximation methods for solving convex bilevel optimization problems, Optimization, 72 (2023), 2517–2545. https://doi.org/10.1080/02331934.2022.2069022 doi: 10.1080/02331934.2022.2069022
[9]	L. O. Jolaoso, Y. Shehu, J. C. Yao, Strongly convergent inertial proximal point algorithm without on-line rule, J. Optim. Theory Appl., 200 (2024), 555–584. https://doi.org/10.1007/s10957-023-02355-5 doi: 10.1007/s10957-023-02355-5
[10]	Y. H. Yao, A. Adamu, Y. Shehu, Forward–reflected–backward splitting algorithms with momentum: weak, linear and strong convergence results, J. Optim. Theory Appl., 201 (2024), 1364–1397. https://doi.org/10.1007/s10957-024-02410-9 doi: 10.1007/s10957-024-02410-9
[11]	J. Y. B. Cruz, T. T. A. Nghia, On the convergence of the forward–backward splitting method with linesearches, Optim. Method. Softw., 31 (2016), 1209–1238. https://doi.org/10.1080/10556788.2016.1214959 doi: 10.1080/10556788.2016.1214959
[12]	K. Kankam, N. Pholasa, P. Cholamjiak, On convergence and complexity of the modified forward‐backward method involving new linesearches for convex minimization, Math. Method. Appl. Sci., 42 (2019), 1352–1362. https://doi.org/10.1002/mma.5420 doi: 10.1002/mma.5420
[13]	S. Suantai, K. Kankam, P. Cholamjiak, A novel forward-backward algorithm for solving convex minimization problem in Hilbert spaces, Mathematics, 8 (2020), 42. https://doi.org/10.3390/math8010042 doi: 10.3390/math8010042
[14]	W. Inthakon, S. Suantai, P. Sarnmeta, D. Chumpungam, A new machine learning algorithm based on optimization method for regression and classification problems, Mathematics, 8 (2020), 1007. https://doi.org/10.3390/math8061007 doi: 10.3390/math8061007
[15]	S. Suantai, M. A. Noor, K. Kankam, P. Cholamjiak, Novel forward–backward algorithms for optimization and applications to compressive sensing and image inpainting, Adv. Differ. Equ., 2021 (2021), 265. https://doi.org/10.1186/s13662-021-03422-9 doi: 10.1186/s13662-021-03422-9
[16]	D. Chumpungam, P. Sarnmeta, S. Suantai, A new forward–backward algorithm with line searchand inertial techniques for convex minimization problems with applications, Mathematics, 9 (2021), 1562. https://doi.org/10.3390/math9131562 doi: 10.3390/math9131562
[17]	A. Hanjing, P. Jailoka, S. Suantai, An accelerated forward-backward algorithm with a new linesearch for convex minimization problems and its applications, AIMS Mathematics, 6 (2021), 6180–6200. https://doi.org/10.3934/math.2021363 doi: 10.3934/math.2021363
[18]	H. H. Bauschke, P. L. Combettes, Convex analysis and monotone operator theory in Hilbert spaces, 2 Eds., Cham: Springer, 2017. https://doi.org/10.1007/978-3-319-48311-5
[19]	R. S. Burachik, A. N. Iusem, Set-valued mappings and enlargements of monotone operators, New York: Springer, 2008. https://doi.org/10.1007/978-0-387-69757-4
[20]	Y. Nesterov, Introductory lectures on convex optimization: A basic course, New York: Springer, 2004. https://doi.org/10.1007/978-1-4419-8853-9
[21]	H. K. Xu, Another control condition in an iterative method for nonexpansive mappings, B. Aust. Math. Soc., 65 (2002), 109–113. https://doi.org/10.1017/S0004972700020116 doi: 10.1017/S0004972700020116
[22]	P. E. Mainge, Strong convergence of projected subgradient methods for nonsmooth and nonstrictly convex minimization, Set-Valued Anal., 16 (2008), 899–912. https://doi.org/10.1007/s11228-008-0102-z doi: 10.1007/s11228-008-0102-z
[23]	R. Tibshirani, Regression shrinkage and selection via the Lasso, J. R. Stat. Soc. B, 58 (1996), 267–288. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x doi: 10.1111/j.2517-6161.1996.tb02080.x
[24]	P. Thongpaen, W. Inthakon, A. Kaewkhao, S. Suantai, Convex minimization problems based on an accelerated fixed point algorithm with applications to image restoration problems, J. Nonlinear Var. Anal., 7 (2023), 87–101. https://doi.org/10.23952/jnva.7.2023.1.06 doi: 10.23952/jnva.7.2023.1.06
[25]	G. B. Huang, Q. Y. Zhu, C. K. Siew, Extreme learning machine: theory and applications, Neurocomputing, 70 (2006), 489–501. https://doi.org/10.1016/j.neucom.2005.12.126 doi: 10.1016/j.neucom.2005.12.126
[26]	Z. K. Yao, X. L. Liang, G. P. Jiang, J. Y. Yao, Model-based reinforcement learning control of electrohydraulic position servo systems, IEEE-ASME T. Mech., 28 (2023), 1446–1455. https://doi.org/10.1109/TMECH.2022.3219115 doi: 10.1109/TMECH.2022.3219115
[27]	Z. K. Yao, F. Y. Xu, G. P. Jiang, J. Y. Yao, Data-driven control of hydraulic manipulators by reinforcement learning, IEEE-ASME T. Mech., 99 (2023), 1–12. https://doi.org/10.1109/TMECH.2023.3336070 doi: 10.1109/TMECH.2023.3336070
[28]	R. Detrano, A. Janosi, W. Steinbrunn, M. Pfisterer, J. J. Schmid, S. Sandhu, et al., International application of a new probability algorithm for the diagnosis of coronary artery disease, Am. J. Cardiol., 64 (1989), 304–310. https://doi.org/10.1016/0002-9149(89)90524-9 doi: 10.1016/0002-9149(89)90524-9
[29]	W. N. Street, W. H. Wolberg, O. L. Mangasarian, Nuclear feature extraction for breast tumor diagnosis, IS & T/SPIE's Symposium on Electronic Imaging: Science and Technology, San Jose, CA, United States, 1993,861–870. https://doi.org/10.1117/12.148698
[30]	M. D. Tissera, M. D. McDonnell, Deep extreme learning machines: supervised autoencoding architecture for classification, Neurocomputing, 174 (2016), 42–49. https://doi.org/10.1016/j.neucom.2015.03.110 doi: 10.1016/j.neucom.2015.03.110
[31]	B. Jia, D. Li, Z. S. Pan, G. Y. Hu, Two-dimensional extreme learning machine, Math. Probl. Eng., 2015 (2015), 491587. https://doi.org/10.1155/2015/491587 doi: 10.1155/2015/491587

Reader Comments

Your name:*

Email:*
© 2024 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)