ODE-RU: a dynamical system view on recurrent neural networks

Pinchao Meng; Xinyu Wang; Weishi Yin; Pinchao Meng; Xinyu Wang; Weishi Yin

doi:10.3934/era.2022014

Electronic Research Archive

2022, Volume 30, Issue 1: 257-271. doi: 10.3934/era.2022014

Previous Article Next Article

Research article

ODE-RU: a dynamical system view on recurrent neural networks

Changchun University of Science and Technology, changchun, China

Academic Editor: Alain Miranville

Received: 12 October 2021 Revised: 03 December 2021 Accepted: 20 December 2021 Published: 28 December 2021

The core of the demonstration of this paper is to interpret the forward propagation process of machine learning as a parameter estimation problem of nonlinear dynamical systems. This process is to establish a connection between the Recurrent Neural Network and the discrete differential equation, so as to construct a new network structure: ODE-RU. At the same time, under the inspiration of the theory of ordinary differential equations, we propose a new forward propagation mode. In a large number of simulations and experiments, the forward propagation not only shows the trainability of the new architecture, but also achieves a low training error on the basis of main-taining the stability of the network. For the problem requiring long-term memory, we specifically study the obstacle shape reconstruction problem using the backscattering far-field features data set, and demonstrate the effectiveness of the proposed architecture using the data set. The results show that the network can effectively reduce the sensitivity to small changes in the input feature. And the error generated by the ordinary differential equation cyclic unit network in inverting the shape and position of obstacles is less than $ 10^{-2} $.
- ordinary differential equation,
- stability,
- obstacle inversion,
- recurrent neural network,
- gated cell structure
Citation: Pinchao Meng, Xinyu Wang, Weishi Yin. ODE-RU: a dynamical system view on recurrent neural networks[J]. Electronic Research Archive, 2022, 30(1): 257-271. doi: 10.3934/era.2022014

Related Papers:

Abstract

The core of the demonstration of this paper is to interpret the forward propagation process of machine learning as a parameter estimation problem of nonlinear dynamical systems. This process is to establish a connection between the Recurrent Neural Network and the discrete differential equation, so as to construct a new network structure: ODE-RU. At the same time, under the inspiration of the theory of ordinary differential equations, we propose a new forward propagation mode. In a large number of simulations and experiments, the forward propagation not only shows the trainability of the new architecture, but also achieves a low training error on the basis of main-taining the stability of the network. For the problem requiring long-term memory, we specifically study the obstacle shape reconstruction problem using the backscattering far-field features data set, and demonstrate the effectiveness of the proposed architecture using the data set. The results show that the network can effectively reduce the sensitivity to small changes in the input feature. And the error generated by the ordinary differential equation cyclic unit network in inverting the shape and position of obstacles is less than $ 10^{-2} $.

References

[1]	J. Collins, J. Sohl-Dickstein, D. Sussillo, Capacity and trainability in recurrent neural networks, Paper presented at 5th International Conference on Learning Representations, 2017.
[2]	S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural Comput., 9 (1997), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735 doi: 10.1162/neco.1997.9.8.1735
[3]	K. Cho, B. V. Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, et al., Learning phrase representations using RNN encoder-decoder for statistical machine translation, Comput. Sci., (2014), 1723–1734. https://doi.org/10.3115/v1/D14-1179 doi: 10.3115/v1/D14-1179
[4]	L. Bottou, F. E. Curtis, J. Nocedal, Optimization methods for large-scale machine learning, SIAM, 60 (2018), 223–231. https://doi.org/10.1137/16M1080173 doi: 10.1137/16M1080173
[5]	B. Chang, M. Chen, E. Haber, E. H. Chi, AntisymmetricRNN: a dynamical system view on recurrent neural networks, Paper presented at 7th International Conference on Learning Representations, 2019.
[6]	E. Haber, K. Lensink, T. Eran, L. Ruthotto, IMEXnet: A forward stable deep neural network, Paper ppresented at: Proceedings of the 36th International Conference on Machine Learning, 2019.
[7]	W. E, A proposal on machine learning via dynamical systems, Commun. Math. Stat., 1 (2017), 1–11. http://doi.org/10.1007/s40304-017-0103-z doi: 10.1007/s40304-017-0103-z
[8]	Y. Lu, A. Zhong, Q. Li, B. Dong, Beyond finite layer neural networks: bridging deep architectures and numerical differential equations, Ppaer pressented at: Proceedings of Machine Learning Research Proceedings of the 35th International Conference on Machine Learning, 2018.
[9]	R. T. Q. Chen, Y. Rubanova, J. Bettencourt, D. Duvenaud, Neural ordinary differential equations, Paper present at : Annual Conference on Neural Information Processing Systems, 2018.
[10]	P. Meng, L. Su, W. Yin, S. Zhang, Solving a kind of inverse scattering problem of acoustic waves based on linear sampling method and neural network, Alexandria Eng. J., 59 (2020), 1451–1462. http://doi.org/10.1016/j.aej.2020.03.047 doi: 10.1016/j.aej.2020.03.047
[11]	W. Yin, W. Yang, H. Liu, A neural network scheme for recovering scattering obstacles with limited phaseless far-field data, J. Comput. Phys., 417 (2020). http://doi.org/10.1016/j.jcp.2020.109594 doi: 10.1016/j.jcp.2020.109594
[12]	Y. Guo, D. Hoemberg, G. Hu, J. Li, H. Liu, A time domain sampling method for inverse acoustic scattering problems, J. Comput. Phys., 314 (2016), 647–660. http://doi.org/10.1016/j.jcp.2016.03.046 doi: 10.1016/j.jcp.2016.03.046
[13]	D. Zhang, F. Sun, L. Lu, Y. Guo, A harmonic polynomial method with a regularization strategy for the boundary value problems of Laplace's equation, Appl. Math. Lett, 79 (2018), 100–104. http://doi.org/10.1016/j.aml.2017.12.003 doi: 10.1016/j.aml.2017.12.003
[14]	H. Liu, M. Petrini, L. Rondi, J. Xiao, Stable determination of sound-hard polyhedral scatterers by a minimal number of scattering measurements, J. Differ. Equations, 262 (2018), 1631–1670. http://doi.org/10.1016/j.jde.2016.10.021 doi: 10.1016/j.jde.2016.10.021
[15]	H. Liu, X. Liu, Recovery of an embedded obstacle and its surrounding medium from formally determined scattering data, Inverse Probl., 33 (2017), 1–20. http://doi.org/10.1088/1361-6420/aa6770 doi: 10.1088/1361-6420/aa6770
[16]	H. Liu, X. Liu, X. Wang, Y. Wang, On a novel inverse scattering scheme using resonant modes with enhanced imaging resolution, Inverse Probl., 35 (2019). http://doi.org/10.1088/1361-6420/ab2932 doi: 10.1088/1361-6420/ab2932
[17]	W. Yin, J. Ge, P. Meng, F. Qu, A neural network method for the inverse scattering problem of impenetrable cavities, Electron. Res. Arch., 28 (2020), 1123–1142. http://doi.org/10.3934/era.2020062 doi: 10.3934/era.2020062
[18]	J. Xie, Y. chen, A numerical analysis method of fixed points and their stability in a complex dynamical systems, CCAMMS, (2019), 59–62.
[19]	W. Stephen, D. S. Mazel, Introduction to applied nonlinear dynamical systems and chaos, Name J., 33 (1991), 81. http://doi.org/10.1063/1.4822950 doi: 10.1063/1.4822950
[20]	G. Wang, Ordinary differential equations, Higher Education Press, (2013), 250–260.
[21]	E. Haber, L. Ruthotto, Stable architectures for deep neural networks, Inverse Probl., 34 (2018). http://doi.org/10.1088/1361-6420/aa9a90 doi: 10.1088/1361-6420/aa9a90
[22]	C. Bo, M. Chen, E. Haber, E. Chi, Antisymmetricrnn: a dynamical system view on recurrent neural networks, Paper present at: 7th International Conference on Learning Representations, 2019.
[23]	U. Ascher, L. Petzold, Computer methods for ordinary differential equations and differential-algebraic equations, in Science press, 2009.
[24]	D. Colton, Inverse acoustic and electromagnetic scattering theory, Inverse Prob., 47 (2003), 67–110. http://doi.org/10.1007/978-3-662-03537-5 doi: 10.1007/978-3-662-03537-5
[25]	D. Colton, J. Coyle, P. Monk, Recent developments in inverse acoustic scattering theory, SIAM, 42 (2000), 369–414. http://doi.org/10.1137/S0036144500367337 doi: 10.1137/S0036144500367337
[26]	A. Kirsch, New characterizations of solutions in inverse scattering theory, Appl. Anal., 76 (2000), 319–350. http://doi.org/10.1080/00036810008840888 doi: 10.1080/00036810008840888
[27]	A. Kirsch, N. Grinberg, The factorization method for inverse problems, Oxford University Press, 2008.
[28]	F. Zeng, P. Suarez, J. Sun, A decomposition method for an interior inverse scattering problem, Inverse Probl. Imaging, 7 (2013), 291–303. http://doi.org/10.3934/ipi.2013.7.291 doi: 10.3934/ipi.2013.7.291
[29]	J. Li, H. Liu, W. Tsui, X. Wang, An inverse scattering approach for geometric body generation: a machine learning perspective, Math. Eng., 1 (2019), 800–823. http://doi.org/10.3934/mine.2019.4.800 doi: 10.3934/mine.2019.4.800
[30]	J. Li, H. Liu, J. Zou, Locating multiple multiscale acoustic scatterers, Multiscale Model. Simul., 12 (2014), 927–952. http://doi.org/10.1137/13093409X doi: 10.1137/13093409X
[31]	H. Liu, J. Zou, Uniqueness in an inverse acoustic obstacle scattering problem for both sound-hard and sound-soft polyhedral scatterers, Inverse Probl., 22 (2016), 515–524. http://doi.org/10.1088/0266-5611/22/2/008 doi: 10.1088/0266-5611/22/2/008
[32]	Q. V. Le, N. Jaitly, G. E. Hinton, A simple way to initialize recurrent networks of rectified linear units, Comput. Sci., preprint, arXiv: 1504.00941.

Reader Comments

Your name:*

Email:*
© 2022 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)