Autoencoding for the "Good Dictionary" of eigenpairs of the Koopman operator

Neranjaka Jayarathne; Erik M. Bollt; Neranjaka Jayarathne; Erik M. Bollt

doi:10.3934/math.2024050

AIMS Mathematics

2024, Volume 9, Issue 1: 998-1022. doi: 10.3934/math.2024050

Previous Article Next Article

Research article Special Issues

Autoencoding for the "Good Dictionary" of eigenpairs of the Koopman operator

Neranjaka Jayarathne ^{1
,
,},
Erik M. Bollt ²

1.
Center for Complex Systems Science, Clarkson University, Potsdam, NY 13699-5815
2.
Department of Electrical and Computer Engineering and The Clarkson Center for Complex Systems Science, Clarkson University, Potsdam, New York 13699, USA

Received: 19 June 2023 Revised: 24 August 2023 Accepted: 31 August 2023 Published: 05 December 2023
MSC : 37M05, 68T07

Reduced order modelling relies on representing complex dynamical systems using simplified modes, which can be achieved through the Koopman operator(KO) analysis. However, computing Koopman eigenpairs for high-dimensional observable data can be inefficient. This paper proposes using deep autoencoders(AE), a type of deep learning technique, to perform nonlinear geometric transformations on raw data before computing Koopman eigenvectors. The encoded data produced by the deep AE is diffeomorphic to a manifold of the dynamical system and has a significantly lower dimension than the raw data. To handle high-dimensional time series data, Takens' time delay embedding is presented as a preprocessing technique. The paper concludes by presenting examples of these techniques in action.
- deep learning,
- autoencoders,
- data driven science,
- reduced order modelling,
- Koopman analysis
Citation: Neranjaka Jayarathne, Erik M. Bollt. Autoencoding for the 'Good Dictionary' of eigenpairs of the Koopman operator[J]. AIMS Mathematics, 2024, 9(1): 998-1022. doi: 10.3934/math.2024050

Related Papers:

Abstract

Reduced order modelling relies on representing complex dynamical systems using simplified modes, which can be achieved through the Koopman operator(KO) analysis. However, computing Koopman eigenpairs for high-dimensional observable data can be inefficient. This paper proposes using deep autoencoders(AE), a type of deep learning technique, to perform nonlinear geometric transformations on raw data before computing Koopman eigenvectors. The encoded data produced by the deep AE is diffeomorphic to a manifold of the dynamical system and has a significantly lower dimension than the raw data. To handle high-dimensional time series data, Takens' time delay embedding is presented as a preprocessing technique. The paper concludes by presenting examples of these techniques in action.

References

[1]	S. Brunton, J. Kutz, Data-driven science and engineering: Machine learning, dynamical systems, and control, Cambridge University Press, 2022. https://doi.org/10.1017/9781009089517
[2]	E. Bollt, N. Santitissadeekorn, Applied and computational measurable dynamics, SIAM, 2013. https://doi.org/10.1137/1.9781611972641
[3]	E. Bollt, Geometric considerations of a good dictionary for Koopman analysis of dynamical systems: Cardinality, "primary eigenfunction, " and efficient representation, Commun. Nonlinear Sci., 100 (2021), 105833. https://doi.org/10.1016/j.cnsns.2021.105833 doi: 10.1016/j.cnsns.2021.105833
[4]	M. Budišić, R. Mohr, I. Mezić, Applied koopmanism, Chaos: An Interdisciplinary J. Nonlinear Sci., 22 (2012), 047510. https://doi.org/10.1063/1.4772195
[5]	J. Kutz, S. Brunton, B. Brunton, J. Proctor, Dynamic mode decomposition: data-driven modeling of complex systems, SIAM, 2016.
[6]	Y. Lan, I. Mezić, Linearization in the large of nonlinear systems and Koopman operator spectrum, Physica D: Nonlinear Phenomena, 242 (2013), 42–53. https://doi.org/10.1016/j.physd.2012.08.017 doi: 10.1016/j.physd.2012.08.017
[7]	A. Avila, I. Mezić, Data-driven analysis and forecasting of highway traffic dynamics, Nat. Commun., 11 (2020), 1–16. https://doi.org/10.1038/s41467-020-15582-5 doi: 10.1038/s41467-020-15582-5
[8]	I. Mezić, Spectral properties of dynamical systems, model reduction and decompositions, Nonlin. Dynam., 41 (2005), 309–325. https://doi.org/10.1007/s11071-005-2824-x doi: 10.1007/s11071-005-2824-x
[9]	I. Mezić, Spectrum of the Koopman operator, spectral expansions in functional spaces, and state-space geometry, J. Nonlinear Sci., 30 (2020), 2091–2145. https://doi.org/10.1007/s00332-019-09598-5 doi: 10.1007/s00332-019-09598-5
[10]	I. Mezić, A. Banaszuk, Comparison of systems with complex behavior, Physica D, 197 (2004), 101–133. https://doi.org/10.1016/j.physd.2004.06.015 doi: 10.1016/j.physd.2004.06.015
[11]	C. Rowley, I. Mezić, S. Bagheri, P. Schlatter, D. Henningson, Spectral analysis of nonlinear flows, J. Fluid Mech., 641 (2009), 115–127. https://doi.org/10.1017/S0022112009992059 doi: 10.1017/S0022112009992059
[12]	P. Schmid, Dynamic mode decomposition of numerical and experimental data, J. Fluid Mech., 656 (2010), 5–28. https://doi.org/10.1017/S0022112010001217 doi: 10.1017/S0022112010001217
[13]	M. Jovanovic, P. Schmid, J. Nichols, Low-rank and sparse dynamic mode decomposition, Center Turbulence Res. Annual Res. Briefs, 2012 (2012), 139–152.
[14]	I. Kevrekidis, C. Rowley, M. Williams, A kernel-based method for data-driven Koopman spectral analysis, J. Comput. Dynam., 2 (2016), 247–265.
[15]	M. Williams, I. Kevrekidis, C. Rowley, A data–driven approximation of the koopman operator: Extending dynamic mode decomposition, J. Nonlinear Sci., 25 (2015), 1307–1346. https://doi.org/10.1007/s00332-015-9258-5 doi: 10.1007/s00332-015-9258-5
[16]	Q. Li, F. Dietrich, E. Bollt, I. Kevrekidis, Extended dynamic mode decomposition with dictionary learning: A data-driven adaptive spectral decomposition of the Koopman operator, Chaos: An Interdisciplinary J. Nonlinear Sci., 27 (2017), 103111. https://doi.org/10.1063/1.4993854 doi: 10.1063/1.4993854
[17]	E. Kaiser, J. Kutz, S. Brunton, Data-driven approximations of dynamical systems operators for control, The Koopman Operator In Systems And Control: Concepts, Methodologies, And Applications, (2020), 197–234. https://doi.org/10.1007/978-3-030-35713-9_8
[18]	I. Mezić, Analysis of fluid flows via spectral properties of the Koopman operator, Annual Rev. Fluid Mech., 45 (2013), 357–378. https://doi.org/10.1146/annurev-fluid-011212-140652 doi: 10.1146/annurev-fluid-011212-140652
[19]	P. Gaspard, Chaos, scattering and statistical mechanics, Chaos, 2005.
[20]	R. Abraham, J. Marsden, Foundations of mechanics, American Mathematical Soc., 2008. https://doi.org/10.1090/chel/364
[21]	A. Ackleh, E. Allen, R. Kearfott, P. Seshaiyer, Classical and modern numerical analysis: Theory, methods and practice, Crc Press, 2009. https://doi.org/10.1201/b12332
[22]	D. Floryan, M. Graham, Charts and atlases for nonlinear data-driven models of dynamics on manifolds, arXiv Preprint arXiv: 2108.05928, (2021).
[23]	C. Fefferman, S. Mitter, H. Narayanan, Testing the manifold hypothesis, J. Am. Math. Soc., 29 (2016), 983–1049. https://doi.org/10.1090/jams/852 doi: 10.1090/jams/852
[24]	H. Narayanan, S. Mitter, Sample complexity of testing the manifold hypothesis, Adv. Neural Inf. Process. Syst., 23 (2010).
[25]	A. Izenman, Introduction to manifold learning, Wires. Comput. Stat., 4 (2012), 439–446. https://doi.org/10.1002/wics.1222 doi: 10.1002/wics.1222
[26]	J. Tenenbaum, V. Silva, J. Langford, A global geometric framework for nonlinear dimensionality reduction, Science, 290 (2000), 2319–2323. https://doi.org/10.1126/science.290.5500.2319 doi: 10.1126/science.290.5500.2319
[27]	S. Roweis, L. Saul, Nonlinear dimensionality reduction by locally linear embedding, Science, 290 (2000), 2323–2326. https://doi.org/10.1126/science.290.5500.2323 doi: 10.1126/science.290.5500.2323
[28]	M. Balasubramanian, E. Schwartz, The isomap algorithm and topological stability, Science, 295 (2002), 7. https://doi.org/10.1126/science.295.5552.7a doi: 10.1126/science.295.5552.7a
[29]	M. Belkin, P. Niyogi, Laplacian eigenmaps and spectral techniques for embedding and clustering, Adv. Neural Inf. Process. Syst., 14, 2001. https://doi.org/10.7551/mitpress/1120.003.0080
[30]	Z. Ma, Z. Zhan, Z. Feng, J. Guo, Manifold learning based on straight-like geodesics and local coordinates, IEEE T. Neural Net. Lear., 32 (2020), 4956–4970. https://doi.org/10.1109/TNNLS.2020.3026426 doi: 10.1109/TNNLS.2020.3026426
[31]	W. Boothby, W. Boothby, An introduction to differentiable manifolds and Riemannian geometry, Revised, Gulf Professional Publishing, 2003.
[32]	X. Chen, J. Weng, W. Lu, J. Xu, J. Weng, Deep manifold learning combined with convolutional neural networks for action recognition, IEEE T. Neural Net. Lear., 29 (2017), 3938–3952. https://doi.org/10.1109/TNNLS.2017.2740318 doi: 10.1109/TNNLS.2017.2740318
[33]	R. Wang, X. Wu, J. Kittler, Symnet: A simple symmetric positive definite manifold deep learning method for image set classification, IEEE T. Neural Net. Lear., 33 (2021), 2208–2222. https://doi.org/10.1109/TNNLS.2020.3044176 doi: 10.1109/TNNLS.2020.3044176
[34]	K. Lee, K. Carlberg, Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders, J. Comput. Phys., 404 (2020), 108973. https://doi.org/10.1016/j.jcp.2019.108973 doi: 10.1016/j.jcp.2019.108973
[35]	J. Bakarji, K. Champion, J. Nathan Kutz, S. L. Brunton, Discovering governing equations from partial measurements with deep delay autoencoders, P Royal Soc. A, 479 (2023), 20230422. https://doi.org/10.1098/rspa.2023.0422 doi: 10.1098/rspa.2023.0422
[36]	Y. LeCun, PhD thesis: Modeles connexionnistes de l'apprentissage (connectionist learning models), (Universite P. et M. Curie (Paris 6), 1987.
[37]	J. Zhai, S. Zhang, J. Chen, Q. He, Autoencoder and its various variants, 2018 IEEE International Conference On Systems, Man, And Cybernetics (SMC), (2018), 415–419. https://doi.org/10.1109/SMC.2018.00080
[38]	S. Gu, B. Kelly, D. Xiu, Autoencoder asset pricing models, J. Econometrics, 222 (2021), 429–450. https://doi.org/10.1016/j.jeconom.2020.07.009 doi: 10.1016/j.jeconom.2020.07.009
[39]	C. Bishop, N. Nasrabadi, Pattern recognition and machine learning, Springer, 2006.
[40]	B. Karlik, A. Olgac, Performance analysis of various activation functions in generalized MLP architectures of neural networks, Int. J. Artif. Intell. Expert Syst., 1 (2011), 111–122.
[41]	P. Pant, R. Doshi, P. Bahl, A. Barati Farimani, Deep learning for reduced order modelling and efficient temporal evolution of fluid simulations, Phys. Fluids, 33 (2021), 107101. https://doi.org/10.1063/5.0062546 doi: 10.1063/5.0062546
[42]	Z. Bai, Krylov subspace techniques for reduced-order modeling of large-scale dynamical systems, Appl. Numer. Math., 43 (2002), 9–44. https://doi.org/10.1016/S0168-9274(02)00116-2 doi: 10.1016/S0168-9274(02)00116-2
[43]	D. Lucia, P. Beran, W. Silva, Reduced-order modeling: new approaches for computational physics, Prog. Aerosp. Sci., 40 (2004), 51–117. https://doi.org/10.1016/j.paerosci.2003.12.001 doi: 10.1016/j.paerosci.2003.12.001
[44]	N. Kazantzis, C. Kravaris, L. Syrou, A new model reduction method for nonlinear dynamical systems, Nonlinear Dynam., 59 (2010), 183–194. https://doi.org/10.1007/s11071-009-9531-y doi: 10.1007/s11071-009-9531-y
[45]	O. San, R. Maulik, Neural network closures for nonlinear model order reduction, Adv. Comput. Math., 44 (2018), 1717–1750. https://doi.org/10.1007/s10444-018-9590-z doi: 10.1007/s10444-018-9590-z
[46]	R. Fu, D. Xiao, I. Navon, F. Fang, L. Yang, C. Wang, et al., A non-linear non-intrusive reduced order model of fluid flow by auto-encoder and self-attention deep learning methods, Int. J. Numer. Meth. Eng., (2023). https://doi.org/10.1002/nme.7240
[47]	N. Aubry, P. Holmes, J. Lumley, E. Stone, The dynamics of coherent structures in the wall region of a turbulent boundary layer, J. Fluid Mech., 192 (1988), 115–173. https://doi.org/10.1017/S0022112088001818 doi: 10.1017/S0022112088001818
[48]	G. Berkooz, P. Holmes, J. Lumley, The proper orthogonal decomposition in the analysis of turbulent flows, Annu. Rev. Fluid Mech., 25 (1993), 539–575. https://doi.org/10.1146/annurev.fl.25.010193.002543 doi: 10.1146/annurev.fl.25.010193.002543
[49]	P. Holmes, J. Lumley, G. Berkooz, C. Rowley, Turbulence, coherent structures, dynamical systems and symmetry, Cambridge university press, 2012. https://doi.org/10.1017/CBO9780511919701
[50]	H. Hotelling, Analysis of a complex of statistical variables into principal components, J. Educ. Psychol., 24 (1933), 417–441. https://psycnet.apa.org/doi/10.1037/h0071325 doi: 10.1037/h0071325
[51]	E. Lorenz, Empirical orthogonal functions and statistical weather prediction, Massachusetts Institute of Technology, Department of Meteorology Cambridge, 1956.
[52]	M. Loeve, Probability theory: foundations, random sequences, New York, NY: Van Nostrand, 1955.
[53]	K. Taira, S. Brunton, S. Dawson, C. Rowley, T. Colonius, B. McKeon, et al., Modal analysis of fluid flows: An overview, Aiaa J., 55 (2017), 4013–4041. https://doi.org/10.2514/1.J056060 doi: 10.2514/1.J056060
[54]	P. Schmid, L. Li, M. Juniper, O. Pust, Applications of the dynamic mode decomposition, Theor. Comp. Fluid Dyn., 25 (2011), 249–259. https://doi.org/10.1007/s00162-010-0203-9 doi: 10.1007/s00162-010-0203-9
[55]	B. Brunton, L. Johnson, J. Ojemann, J. Kutz, Extracting spatial–temporal coherent patterns in large-scale neural recordings using dynamic mode decomposition, J. Neurosci. Meth., 258 (2016), 1–15. https://doi.org/10.1016/j.jneumeth.2015.10.010 doi: 10.1016/j.jneumeth.2015.10.010
[56]	E. Berger, M. Sastuba, D. Vogt, B. Jung, H. Amor, Dynamic mode decomposition for perturbation estimation in human robot interaction, The 23rd IEEE International Symposium On Robot And Human Interactive Communication, (2014), 593–600. https://doi.org/10.1109/ROMAN.2014.6926317
[57]	B. Koopman, Hamiltonian systems and transformation in Hilbert space, P. Natl. Acad. Sci., 17 (1931), 315–318. https://doi.org/10.1073/pnas.17.5.315 doi: 10.1073/pnas.17.5.315
[58]	E. Bollt, Q. Li, F. Dietrich, I. Kevrekidis, On matching, and even rectifying, dynamical systems through Koopman operator eigenfunctions, SIAM J. Appl. Dyn. Syst., 17 (2018), 1925–1960. https://doi.org/10.1137/17M116207X doi: 10.1137/17M116207X
[59]	T. Kanamaru, Van der Pol oscillator, Scholarpedia, 2007. Available from: http://www.scholarpedia.org/article/Van_der_Pol_oscillator
[60]	I. Triandaf, I. Schwartz, Karhunen-Loeve mode control of chaos in a reaction-diffusion process, Phys. Rev. E, 56 (1997), 204–212. https://doi.org/10.1103/PhysRevE.56.204 doi: 10.1103/PhysRevE.56.204
[61]	H. Goldstein, C. Poole, J. Safko, Classical mechanics, American Association of Physics Teachers, 2002.
[62]	F. Takens, Detecting strange attractors in turbulence, Dynamical Systems And Turbulence, Warwick 1980: Proceedings Of A Symposium Held At The University Of Warwick 1979/80, (2006), 366–381. https://doi.org/10.1007/BFb00919
[63]	D. Ruelle, F. Takens, On the nature of turbulence, Les Rencontres Physiciens-mathématiciens De Strasbourg-RCP25, 12 (1971), 1–44.
[64]	K. Falconer, Fractal geometry: Mathematical foundations and applications, John Wiley & Sons, 2004. 10.1002/0470013850
[65]	M. Adachi, Embeddings and immersions, American Mathematical Soc., 2012. https://doi.org/10.1090/mmono/124
[66]	A. Skopenkov, Embedding and knotting of manifolds in Euclidean spaces, London Math. Soc. Lecture Note Series, 347 (2008), 248. https://doi.org/10.1017/CBO9780511666315.008 doi: 10.1017/CBO9780511666315.008

Reader Comments

Your name:*

Email:*
© 2024 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)