Developing robust nonlinear models through bootstrap aggregated deep belief networks

Changhao Zhu; Jie Zhang; Changhao Zhu; Jie Zhang

doi:10.3934/ElectrEng.2020.3.287

AIMS Electronics and Electrical Engineering

2020, Volume 4, Issue 3: 287-302. doi: 10.3934/ElectrEng.2020.3.287

Previous Article Next Article

Research article Special Issues

Developing robust nonlinear models through bootstrap aggregated deep belief networks

Changhao Zhu ^,,
Jie Zhang ^,

School of Engineering, Merz Court, Newcastle University, Newcastle upon Tyne NE1 7RU, UK

Received: 30 April 2020 Accepted: 30 June 2020 Published: 13 July 2020

Deep belief network (DBN) has recently emerged as a powerful tool in building nonlinear data driven models. However, a single DBN model can still lack reliability especially when the amount of data available for modelling is limited. This paper proposes a bootstrap aggregated deep belief network (BAGDBN) to improve model reliability and robustness. In the proposed method, bootstrap re-sampling with replacement is applied to the original modelling data to generate multiple replications. A DBN model is developed on each replication of the original modelling data. These individual DBN models are then combined to form a BAGDBN model. The proposed method is demonstrated on two application examples, modelling of a conic water tank and inferential estimation of polymer melt index in an industrial polypropylene polymerization process. The application results demonstrate that the proposed BAGDBN models can give more reliable estimation and prediction than single DBN models.
- machine learning,
- bootstrap aggregated deep belief network,
- robustness,
- soft sensor,
- polypropylene polymerization
Citation: Changhao Zhu, Jie Zhang. Developing robust nonlinear models through bootstrap aggregated deep belief networks[J]. AIMS Electronics and Electrical Engineering, 2020, 4(3): 287-302. doi: 10.3934/ElectrEng.2020.3.287

Related Papers:

Abstract

Deep belief network (DBN) has recently emerged as a powerful tool in building nonlinear data driven models. However, a single DBN model can still lack reliability especially when the amount of data available for modelling is limited. This paper proposes a bootstrap aggregated deep belief network (BAGDBN) to improve model reliability and robustness. In the proposed method, bootstrap re-sampling with replacement is applied to the original modelling data to generate multiple replications. A DBN model is developed on each replication of the original modelling data. These individual DBN models are then combined to form a BAGDBN model. The proposed method is demonstrated on two application examples, modelling of a conic water tank and inferential estimation of polymer melt index in an industrial polypropylene polymerization process. The application results demonstrate that the proposed BAGDBN models can give more reliable estimation and prediction than single DBN models.

References

[1]	Zhang J, Jin Q, Xu Y (2006) Inferential estimation of polymer melt index using sequentially trained bootstrap aggregated neural networks. Chemical Engineering & Technology: Industrial Chemistry-Plant Equipment-Process Engineering-Biotechnology 29: 442-448.
[2]	Shao W, Yao L, Ge Z, et al. (2018) Parallel computing and SGD-based DPMM for soft sensor development with large-scale semisupervised data. IEEE T Ind Electron 66: 6362-6373.
[3]	Shao W, Ge Z, Song Z (2019) Quality variable prediction for chemical processes based on semi-supervised Dirichlet process mixture of Gaussians. Chem Eng Sci 193: 394-410. doi: 10.1016/j.ces.2018.09.031
[4]	Yuan X, Ou C, Wang Y, et al. (2020) Deep quality-related feature extraction for soft sensing modeling: A deep learning approach with hybrid VW-SAE. Neurocomputing 396: 375-382. doi: 10.1016/j.neucom.2018.11.107
[5]	Yuan X, Zhou J, Huang B, et al. (2019) Hierarchical quality-relevant feature representation for soft sensor modeling: a novel deep learning strategy. IEEE T Ind Inform 16: 3721-3730.
[6]	Werbos P (1974) Beyond regression: new fools for prediction and analysis in the behavioral sciences. Harvard University.
[7]	Rosa E, Yu W (2016) Randomized algorithms for nonlinear system identification with deep learning modification. Inform Sciences 364: 197-212.
[8]	Hinton GE, Osindero S, Teh YW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18: 1527-1554. doi: 10.1162/neco.2006.18.7.1527
[9]	Yuan X, Huang B, Wang Y, et al. (2018) Deep learning-based feature representation and its application for soft sensor modeling with variable-wise weighted SAE. IEEE T Ind Inform 14: 3235-3243. doi: 10.1109/TII.2018.2809730
[10]	Yuan X, Ou C, Wang Y, et al. (2019) A layer-wise data augmentation strategy for deep learning networks and its soft sensor application in an industrial hydrocracking process. IEEE T Neur Net Lear.
[11]	Mnih A, Hinton GE (2009) A scalable hierarchical distributed language model. Advances in Neural Information Processing Systems, 1081-1088.
[12]	Li F, Zhang J, Shang C, et al. (2018) Modelling of a post-combustion CO2 capture process using deep belief network. Appl Therm Eng 130: 997-1003. doi: 10.1016/j.applthermaleng.2017.11.078
[13]	Shang C, Yang F, Huang D, et al. (2014) Data driven soft sensor development based on deep learning technique. J Process Contr 24: 223-233. doi: 10.1016/j.jprocont.2014.01.012
[14]	Wang Y, Pan Z, Yuan X, et al. (2020) A novel deep learning based fault diagnosis approach for chemical process with extended deep belief network. ISA Transactions 96: 457-467. doi: 10.1016/j.isatra.2019.07.001
[15]	Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9: 1735-1780. doi: 10.1162/neco.1997.9.8.1735
[16]	Yuan X, Li L, Wang Y (2019) Nonlinear dynamic soft sensor modeling with supervised long short-term memory network. IEEE T Ind Inform 16: 3168-3176.
[17]	Yuan X, Li L, Shardt Y, et al. (2020) Deep learning with spatiotemporal attention-based LSTM for industrial soft sensor model development. IEEE T Ind Electron.
[18]	Wöllmer M, Schuller B, Eyben F, et al. (2010) Combining long short-term memory and dynamic Bayesian networks for incremental emotion-sensitive artificial listening. IEEE J-STSP 4: 867-881.
[19]	Zhang J (1999) Developing robust non-linear models through bootstrap aggregated neural networks. Neurocomputing 25: 93-113. doi: 10.1016/S0925-2312(99)00054-5
[20]	Zhang J, Pantelelis NG (2011) Modelling and optimisation control of polymer composite moulding processes using bootstrap aggregated neural network models. 2011 International Conference on Electric Information and Control Engineering, 2363-2366.
[21]	Kaunga DL, Zhang J, Ferguson K (2013) Reliable modelling of chemical durability of high-level waste glass using bootstrap aggregated neural networks. 2013 Ninth International Conference on Natural Computation, 178-183.
[22]	Mohammed KJR, Zhang J (2013) Reliable optimisation control of a reactive polymer composite moulding process using ant colony optimisation and bootstrap aggregated neural networks. Neural Comput Appl 23: 1891-1898. doi: 10.1007/s00521-012-1273-y
[23]	Osuolale FN, Zhang J (2018) Exergetic optimization of atmospheric and vacuum distillation system based on bootstrap aggregated neural network models. Exergy for A Better Environment and Improved Sustainability 1: 1033-1046.
[24]	Mukherjee A, Zhang J (2008) A reliable multi-objective control strategy for batch processes based on bootstrap aggregated neural network models. J Process Contr 18: 720-734. doi: 10.1016/j.jprocont.2007.11.008
[25]	Zhou H, Huang GB, Lin Z, et al. (2015) Stacked extreme learning machines. IEEE T Syst Man Cy B 45: 2013-2025.
[26]	Low CY, Teoh ABJ (2017) Stacking-based deep neural network: Deep analytic network on convolutional spectral histogram features. 2017 IEEE International Conference on Image Processing (ICIP), 1592-1596.
[27]	Smolensky P (1986) Information processing in dynamical systems: Foundations of harmony theory. Parallel Distributed Processing: Explorations in the Microstructure of Cognition 1: 194-281.
[28]	Hinton GE (2012) A practical guide to training restricted Boltzmann machines. Neural networks: Tricks of the trade, 599-619.
[29]	Zhang J, Morris AJ (2000) Long range predictive control of nonlinear processes based on recurrent neurofuzzy network models. Neural Comput Appl 9: 50-59. doi: 10.1007/s005210070035
[30]	Soares JBP, Hamielec AE (1996) Kinetics of propylene polymerization with a non-supported heterogeneous Ziegler-Natta catalyst-effect of hydrogen on rate of polymerization, stereoregularity, and molecular weight distribution. Polymer 37: 4607-4614. doi: 10.1016/0032-3861(96)00286-8

Reader Comments

Your name:*

Email:*
© 2020 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)