Reservoir computing has emerged as a powerful and efficient machine learning tool especially in the reconstruction of many complex systems even for chaotic systems only based on the observational data. Though fruitful advances have been extensively studied, how to capture the art of hyper-parameter settings to construct efficient RC is still a long-standing and urgent problem. In contrast to the local manner of many works which aim to optimize one hyper-parameter while keeping others constant, in this work, we propose a global optimization framework using simulated annealing technique to find the optimal architecture of the randomly generated networks for a successful RC. Based on the optimized results, we further study several important properties of some hyper-parameters. Particularly, we find that the globally optimized reservoir network has a largest singular value significantly larger than one, which is contrary to the sufficient condition reported in the literature to guarantee the echo state property. We further reveal the mechanism of this phenomenon with a simplified model and the theory of nonlinear dynamical systems.
Citation: Bin Ren, Huanfei Ma. Global optimization of hyper-parameters in reservoir computing[J]. Electronic Research Archive, 2022, 30(7): 2719-2729. doi: 10.3934/era.2022139
Reservoir computing has emerged as a powerful and efficient machine learning tool especially in the reconstruction of many complex systems even for chaotic systems only based on the observational data. Though fruitful advances have been extensively studied, how to capture the art of hyper-parameter settings to construct efficient RC is still a long-standing and urgent problem. In contrast to the local manner of many works which aim to optimize one hyper-parameter while keeping others constant, in this work, we propose a global optimization framework using simulated annealing technique to find the optimal architecture of the randomly generated networks for a successful RC. Based on the optimized results, we further study several important properties of some hyper-parameters. Particularly, we find that the globally optimized reservoir network has a largest singular value significantly larger than one, which is contrary to the sufficient condition reported in the literature to guarantee the echo state property. We further reveal the mechanism of this phenomenon with a simplified model and the theory of nonlinear dynamical systems.
| [1] |
H. Jaeger, H. Haas, Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication, Science, 304 (2004), 78–80. https://doi.org/10.1126/science.1091277 doi: 10.1126/science.1091277
|
| [2] | H. Jaeger, The "echo state" approach to analysing and training recurrent neural networks-with an erratum note, German National Research Center for Information Technology GMD Technical Report, 34 (2001), 148. |
| [3] |
W. Maass, T. Natschläger, H. Markram, Real-time computing without stable states: A new framework for neural computation based on perturbations, Neural Comput., 14 (2002), 2531–2560. https://doi.org/10.1162/089976602760407955 doi: 10.1162/089976602760407955
|
| [4] |
J. Pathak, B. Hunt, M. Girvan, Z. Lu, E. Ott, Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach, Phys. Rev. Lett., 120 (2018), 024102. https://doi.org/10.1103/PhysRevLett.120.024102 doi: 10.1103/PhysRevLett.120.024102
|
| [5] |
G. Tanaka, T. Yamane, J. B. Héroux, R. Nakane, N. Kanazawa, S. Takeda, et al., Recent advances in physical reservoir computing: A review, Neural Networks, 115 (2019), 100–123. https://doi.org/10.1016/j.neunet.2019.03.005 doi: 10.1016/j.neunet.2019.03.005
|
| [6] |
Q. Zhu, H. F. Ma, W. Lin, Detecting unstable periodic orbits based only on time series: When adaptive delayed feedback control meets reservoir computing, Chaos, 29 (2019), 093125. https://doi.org/10.1063/1.5120867 doi: 10.1063/1.5120867
|
| [7] | H. Jaeger, Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the "echo state network" approach, GMD-Forschungszentrum Informationstechnik Bonn, 5 (2002). |
| [8] |
A. Haluszczynski, C. Räth, Good and bad predictions: Assessing and improving the replication of chaotic attractors by means of reservoir computing, Chaos, 29 (2019), 103143. https://doi.org/10.1063/1.5118725 doi: 10.1063/1.5118725
|
| [9] |
A. Griffith, A. Pomerance, D. J. Gauthier, Forecasting chaotic systems with very low connectivity reservoir computers, Chaos, 29 (2019), 123108. https://doi.org/10.1063/1.5120710 doi: 10.1063/1.5120710
|
| [10] |
M. Lukoševičius, H. Jaeger, Reservoir computing approaches to recurrent neural network training, Comput. Sci. Rev., 3 (2009), 127–149. https://doi.org/10.1016/j.cosrev.2009.03.005 doi: 10.1016/j.cosrev.2009.03.005
|
| [11] |
J. Jiang, Y. C. Lai, Model-free prediction of spatiotemporal dynamical systems with recurrent neural networks: Role of network spectral radius, Phys. Rev. Res., 1 (2019), 033056. https://doi.org/10.1103/PhysRevResearch.1.033056 doi: 10.1103/PhysRevResearch.1.033056
|
| [12] |
D. Verstraeten, B. Schrauwen, M. d'Haene, D. Stroobandt, An experimental unification of reservoir computing methods, Neural Networks, 20 (2007), 391–403. https://doi.org/10.1016/j.neunet.2007.04.003 doi: 10.1016/j.neunet.2007.04.003
|
| [13] |
H. Cui, X. Liu, L. Li, The architecture of dynamic reservoir in the echo state network, Chaos, 22 (2012), 033127. https://doi.org/10.1063/1.4746765 doi: 10.1063/1.4746765
|
| [14] |
B. Zhang, D. J. Miller, Y. Wang, Nonlinear system modeling with random matrices: echo state networks revisited, IEEE Trans. Neural Networks Learn. Syst., 23 (2011), 175–182. https://doi.org/10.1109/TNNLS.2011.2178562 doi: 10.1109/TNNLS.2011.2178562
|
| [15] |
M. Ji, Z. Jin, H. Tang, An improved simulated annealing for solving the linear constrained optimization problems, Appl. Math. Comput., 183 (2006), 251–259. https://doi.org/10.1016/j.amc.2006.05.070 doi: 10.1016/j.amc.2006.05.070
|
| [16] |
G. F. de Arruda, F. A. Rodrigues, Y. Moreno, Fundamentals of spreading processes in single and multilayer complex networks, Phys. Rep., 756 (2018), 1–59. https://doi.org/10.1016/j.physrep.2018.06.007 doi: 10.1016/j.physrep.2018.06.007
|
| [17] |
Z. Lu, J. Pathak, B. Hunt, M. Girvan, R. Brockett, E. Ott, Reservoir observers: Model-free inference of unmeasured variables in chaotic systems, Chaos, 27 (2017), 041102. https://doi.org/10.1063/1.4979665 doi: 10.1063/1.4979665
|
| [18] |
X. Dutoit, B. Schrauwen, J. Van Campenhout, D. Stroobandt, H. Van Brussel, M. Nuttin, Pruning and regularization in reservoir computing, Neurocomputing, 72 (2009), 1534–1546. https://doi.org/10.1016/j.neucom.2008.12.020 doi: 10.1016/j.neucom.2008.12.020
|
| [19] | D. Verstraeten, J. Dambre, X. Dutoit, B. Schrauwen, Memory versus non-linearity in reservoirs, in The 2010 International Joint Conference on Neural Networks (IJCNN), IEEE, (2010), 1–8. https://doi.org/10.1109/IJCNN.2010.5596492 |
| [20] |
X. Ying, S. Y. Leng, H. F. Ma, Q. Nie, Y. C. Lai, W. Lin, Continuity scaling: A rigorous framework for detecting and quantifying causality accurately, Research, 2022 (2022), 9870149. https://doi.org/10.34133/2022/9870149 doi: 10.34133/2022/9870149
|
| [21] |
J. W. Hou, H. F. Ma, D. He, J. Sun, Q. Nie, W. Lin, Harvesting random embedding for high-frequency change-point detection in temporal complex systems, Natl. Sci. Rev., 9 (2022), nwab228. https://doi.org/10.1093/nsr/nwab228 doi: 10.1093/nsr/nwab228
|
Figures(5) / Tables(1)
Bin Ren, Huanfei Ma. Global optimization of hyper-parameters in reservoir computing[J]. Electronic Research Archive, 2022, 30(7): 2719-2729. doi: 10.3934/era.2022139
DownLoad: