Physics-informed neural networks (PINN) have lately become a research hotspot in the interdisciplinary field of machine learning and computational mathematics thanks to the flexibility in tackling forward and inverse problems. In this work, we explore the generality of the PINN training algorithm for solving Hamilton-Jacobi equations, and propose physics-informed neural networks based on adaptive weighted loss functions (AW-PINN) that is trained to solve unsupervised learning tasks with fewer training data while physical information constraints are imposed during the training process. To balance the contributions from different constrains automatically, the AW-PINN training algorithm adaptively update the weight coefficients of different loss terms by using the logarithmic mean to avoid additional hyperparameter. Moreover, the proposed AW-PINN algorithm imposes the periodicity requirement on the boundary condition and its gradient. The fully connected feedforward neural networks are considered and the optimizing procedure is taken as the Adam optimizer for some steps followed by the L-BFGS-B optimizer. The series of numerical experiments illustrate that the proposed algorithm effectively achieves noticeable improvements in predictive accuracy and the convergence rate of the total training error, and can approximate the solution even when the Hamiltonian is nonconvex. A comparison between the proposed algorithm and the original PINN algorithm for Hamilton-Jacobi equations indicates that the proposed AW-PINN algorithm can train the solutions more accurately with fewer iterations.
Citation: Youqiong Liu, Li Cai, Yaping Chen, Bin Wang. Physics-informed neural networks based on adaptive weighted loss functions for Hamilton-Jacobi equations[J]. Mathematical Biosciences and Engineering, 2022, 19(12): 12866-12896. doi: 10.3934/mbe.2022601
Physics-informed neural networks (PINN) have lately become a research hotspot in the interdisciplinary field of machine learning and computational mathematics thanks to the flexibility in tackling forward and inverse problems. In this work, we explore the generality of the PINN training algorithm for solving Hamilton-Jacobi equations, and propose physics-informed neural networks based on adaptive weighted loss functions (AW-PINN) that is trained to solve unsupervised learning tasks with fewer training data while physical information constraints are imposed during the training process. To balance the contributions from different constrains automatically, the AW-PINN training algorithm adaptively update the weight coefficients of different loss terms by using the logarithmic mean to avoid additional hyperparameter. Moreover, the proposed AW-PINN algorithm imposes the periodicity requirement on the boundary condition and its gradient. The fully connected feedforward neural networks are considered and the optimizing procedure is taken as the Adam optimizer for some steps followed by the L-BFGS-B optimizer. The series of numerical experiments illustrate that the proposed algorithm effectively achieves noticeable improvements in predictive accuracy and the convergence rate of the total training error, and can approximate the solution even when the Hamiltonian is nonconvex. A comparison between the proposed algorithm and the original PINN algorithm for Hamilton-Jacobi equations indicates that the proposed AW-PINN algorithm can train the solutions more accurately with fewer iterations.
| [1] |
K. Guo, Z. Yang, C. H. Yu, M. J. Buehler, Artificial intelligence and machine learning in design of mechanical materials, Mater. Horiz., 8 (2021), 1153–1172. https://doi.org/10.1039/D0MH01451F doi: 10.1039/D0MH01451F
|
| [2] |
R. Pestourie, Y. Mroueh, T. V. Nguyen, P. Das, S. G. Johnson, Active learning of deep surrogates for PDEs: Application to metasurface design, npj Comput. Mater., 6 (2020), 1–7. https://doi.org/10.1038/s41524-020-00431-2 doi: 10.1038/s41524-020-00431-2
|
| [3] |
H. Sasaki, H. Igarashi, Topology optimization accelerated by deep learning, IEEE Trans. Magn., 55 (2019), 1–5. https://doi.org/10.1109/TMAG.2019.2901906 doi: 10.1109/TMAG.2019.2901906
|
| [4] |
D. A. White, W. J. Arrighi, J. Kudo, S. E. Watts, Multiscale topology optimization using neural network surrogate models, Comput. Method. Appl. Mech. Eng., 346 (2019), 1118–1135. https://doi.org/10.1016/j.cma.2018.09.007 doi: 10.1016/j.cma.2018.09.007
|
| [5] |
M. Raissi, G. E. Karniadakis, Hidden physics models: Machine learning of nonlinear partial differential equations, J. Comput. Phys., 357 (2018), 125–141. https://doi.org/10.1016/j.jcp.2017.11.039 doi: 10.1016/j.jcp.2017.11.039
|
| [6] |
M. Raissi, P. Perdikaris, G. E. Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, J. Comput. Phys., 378 (2019), 686–707. https://doi.org/10.1016/j.jcp.2018.10.045 doi: 10.1016/j.jcp.2018.10.045
|
| [7] |
L. Lu, X. Meng, Z. Mao, G. E. Karniadakis, DeepXDE: A deep learning library for solving differential equations, SIAM Rev., 63 (2021), 208–228. https://doi.org/10.1137/19M1274067 doi: 10.1137/19M1274067
|
| [8] |
S. Wang, Y. Teng, P. Perdikaris, Understanding and mitigating gradient flow pathologies in physics-informed neural networks, SIAM J. Sci. Comput., 43 (2021), A3055–A3081. https://doi.org/10.1137/20M1318043 doi: 10.1137/20M1318043
|
| [9] |
S. Wang, X. Yu, P. Perdikaris, When and why PINNs fail to train: A neural tangent kernel perspective, J. Comput. Phys., 449 (2022), 110768. https://doi.org/10.1016/j.jcp.2021.110768 doi: 10.1016/j.jcp.2021.110768
|
| [10] |
W. Ji, W. Qiu, Z. Shi, S. Pan, S. Deng, Stiff-PINN: Physics-informed neural network for stiff chemical kinetics, J. Phys. Chem. A, 125 (2021), 8098–8106. https://doi.org/10.1021/acs.jpca.1c05102 doi: 10.1021/acs.jpca.1c05102
|
| [11] | C. Yu, Y. Tang, B. Liu, An adaptive activation function for multilayer feedforward neural networks, in 2002 IEEE Region 10 Conference on Computers, Communications, Control and Power Engineering. TENCOM'02. Proceedings, (2002), 645–650. https://doi.org/10.1109/TENCON.2002.1181357 |
| [12] |
S. Qian, H. Liu, C. Liu, S. Wu, H. S. Wong, Adaptive activation functions in convolutional neural networks, Neurocomputing, 272 (2018), 204–212. https://doi.org/10.1016/j.neucom.2017.06.070 doi: 10.1016/j.neucom.2017.06.070
|
| [13] |
A. D. Jagtap, K. Kawaguchi, G. E. Karniadakis, Adaptive activation functions accelerate convergence in deep and physics-informed neural networks, J. Comput. Phys., 404 (2020), 109136. https://doi.org/10.1016/j.jcp.2019.109136 doi: 10.1016/j.jcp.2019.109136
|
| [14] |
A. D. Jagtap, K. Kawaguchi, G. E. Karniadakis, Locally adaptive activation functions with slope recovery for deep and physics-informed neural networks, Proceed. R. Soc. A, 476 (2020), 20200334. https://doi.org/10.1098/rspa.2020.0334 doi: 10.1098/rspa.2020.0334
|
| [15] |
M. Raissi, A. Yazdani, G. E. Karniadakis, Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations, Science, 367 (2020), 1026–1030. https://doi.org/10.1126/science.aaw4741 doi: 10.1126/science.aaw4741
|
| [16] |
F. S. Costabal, Y. Yang, P. Perdikaris, D. E. Hurtado, E. Kuhl, Physics-informed neural networks for cardiac activation mapping, Front. Phys., 8 (2020), 42. https://doi.org/10.3389/fphy.2020.00042 doi: 10.3389/fphy.2020.00042
|
| [17] |
G. Kissas, Y. Yang, E. Hwuang, W. R. Witschey, J. A. Detre, P. Perdikaris, Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networks, Comput. Method. Appl. Mech. Eng., 358 (2020), 112623. https://doi.org/10.1016/j.cma.2019.112623 doi: 10.1016/j.cma.2019.112623
|
| [18] |
G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, L. Yang, Physics-informed machine learning, Nat. Rev. Phys., 3 (2021), 422–440. https://doi.org/10.1038/s42254-021-00314-5 doi: 10.1038/s42254-021-00314-5
|
| [19] |
S. Lin, Y. Chen, A two-stage physics-informed neural network method based on conserved quantities and applications in localized wave solutions, J. Comput. Phys., 457 (2022), 111053. https://doi.org/10.1016/j.jcp.2022.111053 doi: 10.1016/j.jcp.2022.111053
|
| [20] |
J. C. Pu, Y. Chen Data-driven vector localized waves and parameters discovery for Manakov system using deep learning approach, Chaos Solitons Fractals, 160 (2022), 112182. https://doi.org/10.1016/j.chaos.2022.112182 doi: 10.1016/j.chaos.2022.112182
|
| [21] |
Z. W. Miao, Y. Chen, Physics-informed neural networks method in high-dimensional integrable systems, Mod. Phys. Lett. B, 36 (2022), 2150531. https://doi.org/10.1142/S021798492150531X doi: 10.1142/S021798492150531X
|
| [22] |
L. Shen, D. Li, W. Zha, X. Li, X. Liu Surrogate modeling for porous flow using deep neural networks, J. Pet. Sci. Eng., 213 (2022), 110460. https://doi.org/10.1016/j.petrol.2022.110460 doi: 10.1016/j.petrol.2022.110460
|
| [23] |
D. Li, L. Shen, W. Zha, X. Liu, J. Tan Physics-constrained deep learning for solving seepage equation, J. Pet. Sci. Eng., 206 (2021), 109046. https://doi.org/10.1016/j.petrol.2021.109046 doi: 10.1016/j.petrol.2021.109046
|
| [24] |
M. Zhu, Y. Xu, J. Cao, The asymptotic profile of a dengue fever model on a periodically evolving domain, Appl. Math. Comput., 362 (2019), 124531. https://doi.org/10.1016/j.amc.2019.06.045 doi: 10.1016/j.amc.2019.06.045
|
| [25] |
G. J. Pettet, H. M. Byrne, D. L. S. Mcelwain, J. Norbury, A model of wound-healing angiogenesis in soft tissue, Math. Biosci., 136 (1996), 35–63. https://doi.org/10.1016/0025-5564(96)00044-2 doi: 10.1016/0025-5564(96)00044-2
|
| [26] |
T. Höfer, J. A. Sherratt, P. K. Maini, Cellular pattern formation during dictyostelium aggregation, Phys. D, 85 (1995), 425–444. https://doi.org/10.1016/0167-2789(95)00075-F doi: 10.1016/0167-2789(95)00075-F
|
| [27] |
J. King, R. Ahmadian, R. A. Falconer, Hydro-epidemiological modelling of bacterial transport and decay in nearshore coastal waters, Water Res., 196 (2021), 117049. https://doi.org/10.1016/j.watres.2021.117049 doi: 10.1016/j.watres.2021.117049
|
| [28] |
X. Wang, F. B. Wang, Impact of bacterial hyperinfectivity on cholera epidemics in a spatially heterogeneous environment, J. Math. Anal. Appl., 480 (2019), 123407. https://doi.org/10.1016/j.jmaa.2019.123407 doi: 10.1016/j.jmaa.2019.123407
|
| [29] |
Y. Wang, L. Cai, X. Luo, W. Ying, H. Gao, Simulation of action potential propagation based on the ghost structure method, Sci. Rep., 9 (2019), 10927. https://doi.org/10.1038/s41598-019-47321-2 doi: 10.1038/s41598-019-47321-2
|
| [30] |
Y. Wang, L. Cai, X. Feng, X. Luo, H. Gao, A ghost structure finite difference method for a fractional FitzHugh-Nagumo monodomain model on moving irregular domain, J. Comput. Phys., 428 (2021), 110081. https://doi.org/10.1016/j.jcp.2020.110081 doi: 10.1016/j.jcp.2020.110081
|
| [31] |
S. Bryson, D. Levy, High-order central WENO schemes for multidimensional Hamilton-Jacobi equations, SIAM J. Num. Anal., 41 (2003), 1339–1369. https://doi.org/10.1137/S0036142902408404 doi: 10.1137/S0036142902408404
|
| [32] |
C. L. Lin, E. Tadmor, High-resolution nonoscillatory central schemes for Hamilton-Jacobi equations, SIAM J. Sci. Comput., 21 (2000), 2163–2186. https://doi.org/10.1137/S1064827598344856 doi: 10.1137/S1064827598344856
|
| [33] |
S. Bryson, D. Levy, High-order semi-discrete central-upwind schemes for multi-dimensional Hamilton-Jacobi equations, J. Comput. Phys., 189 (2003), 63–87. https://doi.org/10.1016/S0021-9991(03)00201-8 doi: 10.1016/S0021-9991(03)00201-8
|
| [34] |
A. Kurganov, E. Tadmor, New high-resolution semi-discrete central schemes for Hamilton-Jacobi equations, J. Comput. Phys., 160 (2000), 720–742. https://doi.org/10.1006/jcph.2000.6485 doi: 10.1006/jcph.2000.6485
|
| [35] |
L. Cai, W. Xie, Y. Nie, J. Feng, High-resolution semi-discrete Hermite central-upwind scheme for multidimensional Hamilton-Jacobi equations, Appl. Num. Math., 80 (2014), 22–45. https://doi.org/10.1016/j.apnum.2014.02.002 doi: 10.1016/j.apnum.2014.02.002
|
| [36] |
S. Bryson, D. Levy, Mapped WENO and weighted power ENO reconstructions in semi-discrete central schemes for Hamilton-Jacobi equations, Appl. Num. Math., 56 (2006), 1211–1224. https://doi.org/10.1016/j.apnum.2006.03.005 doi: 10.1016/j.apnum.2006.03.005
|
| [37] |
F. Zheng, J. Qiu, Directly solving the Hamilton-Jacobi equations by Hermite WENO Schemes, J. Comput. Phys., 307 (2021), 423–445. https://doi.org/10.1016/j.jcp.2015.12.011 doi: 10.1016/j.jcp.2015.12.011
|
| [38] |
C. H. Kim, Y. Ha, H. Yang, J. Yoon, A third-order WENO scheme based on exponential polynomials for Hamilton-Jacobi equations, Appl. Num. Math., 165 (2021), 167–183. https://doi.org/10.1016/j.apnum.2021.01.020 doi: 10.1016/j.apnum.2021.01.020
|
| [39] |
P. J. Graber, C. Hermosilla, H. Zidani, Discontinuous solutions of Hamilton-Jacobi equations on networks, J. Differ. Equations, 263 (2017), 8418–8466. https://doi.org/10.1016/j.jde.2017.08.040 doi: 10.1016/j.jde.2017.08.040
|
| [40] |
J. Sirignano, K. Spiliopoulos, DGM: A deep learning algorithm for solving partial differential equations, J. Comput. Phys., 375 (2018), 1339-1364. https://doi.org/10.1016/j.jcp.2018.08.029 doi: 10.1016/j.jcp.2018.08.029
|
| [41] |
T. Nakamura-Zimmerer, Q. Gong, W. Kang, Adaptive deep learning for high-dimensional Hamilton-Jacobi-Bellman equations, SIAM J. Sci. Comput., 43 (2021), A1221-A1247. https://doi.org/10.1137/19M1288802 doi: 10.1137/19M1288802
|
| [42] |
J. Darbon, G. P. Langlois, T. Meng, Overcoming the curse of dimensionality for some Hamilton-Jacobi partial differential equations via neural network architectures, Res. Math. Sci., 7 (2020), 1–50. https://doi.org/10.1007/s40687-020-00215-6 doi: 10.1007/s40687-020-00215-6
|
| [43] |
J. Darbon, T. Meng, On some neural network architectures that can represent viscosity solutions of certain high dimensional Hamilton-Jacobi partial differential equations, J. Comput. Phys., 425 (2021), 109907. https://doi.org/10.1016/j.jcp.2020.109907 doi: 10.1016/j.jcp.2020.109907
|
| [44] | A. G. Baydin, B. A. Pearlmutter, A. A. Radul, J. M. Siskind, Automatic differentiation in machine learning: A survey, J. March. Learn. Res., 18 (2018), 1–43. http://jmlr.org/papers/v18/17-468.html |
| [45] | D. Kingma, J. Ba, Adam: A method for stochastic optimization, preprint, arXiv: 1412.6980. |
| [46] | J. Duchi, E. Hazan, Y. Singer, Adaptive subgradient methods for online learning and stochastic optimization, J. Mach. Learn. Res., 12 (2011), 2121–2159. http://jmlr.org/papers/v12/duchi11a.html |
| [47] |
D. C. Liu, J. Nocedal, On the limited memory BFGS method for large scale optimization, Math. Program., 45 (1989), 503–528. https://doi.org/10.1007/BF01589116 doi: 10.1007/BF01589116
|
| [48] |
R. van der Meer, C. W. Oosterlee, A. Borovykh, Optimally weighted loss functions for solving PDEs with neural networks, J. Comput. Appl. Math., 405 (2022), 113887. https://doi.org/10.1016/j.cam.2021.113887 doi: 10.1016/j.cam.2021.113887
|
| [49] |
F. Ismail, P. L. Roe, Affordable, entropy-consistent Euler flux functions II: Entropy production at shocks, J. Comput. Phys., 228 (2009), 5410–5436. https://doi.org/10.1016/j.jcp.2009.04.021 doi: 10.1016/j.jcp.2009.04.021
|
| [50] | X. Glorot, Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, J. Mach. Learn. Res., 9 (2010), 249–256. http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf |
| [51] |
S. Osher, C. W. Shu, High-order essentially nonoscillatory schemes for Hamilton-Jacobi equations, SIAM J. Numer. Anal., 28 (1991), 907–922. https://doi.org/10.1137/0728049 doi: 10.1137/0728049
|
| [52] |
A. A. Loya, D. Appelö, A Hermite method with a discontinuity sensor for Hamilton-Jacobi equations, J. Sci. Comput., 90 (2022), 1–31. https://doi.org/10.1007/s10915-022-01766-2 doi: 10.1007/s10915-022-01766-2
|
| [53] |
E. Rouy, A. Tourin, A viscosity solutions approach to Shape-From-Shading, SIAM J. Numer. Anal., 29 (1992), 867–884. https://doi.org/10.1137/0729053 doi: 10.1137/0729053
|
| [54] |
P. L. Lions, E. Rouy, A. Tourin, Shape-From-Shading, viscosity solutions and edges, Numer. Math., 64 (1993), 323–353. https://doi.org/10.1007/BF01388692 doi: 10.1007/BF01388692
|
| [55] |
G. Jiang, D. Peng, Weighted ENO schemes for Hamilton-Jacobi equations, SIAM J. Sci. Comput., 21 (2000), 2126–2143. https://doi.org/10.1137/S106482759732455X doi: 10.1137/S106482759732455X
|
Figures(17) / Tables(6)
Youqiong Liu, Li Cai, Yaping Chen, Bin Wang. Physics-informed neural networks based on adaptive weighted loss functions for Hamilton-Jacobi equations[J]. Mathematical Biosciences and Engineering, 2022, 19(12): 12866-12896. doi: 10.3934/mbe.2022601
DownLoad: