Fault-tolerant control for nonlinear systems with a dead zone: Reinforcement learning approach

1.   Introduction

Optimal control theory originated in the 1960s and has become an important part of automatic control theory primarily owing to its spirit of seeking the optimal solution from all possible control schemes ^[1,2]. Optimal control problems for linear systems can generally be settled by solving Ricatti equations. However, when it comes to nonlinear systems, there are quite a few effective methods since the Hamilton-Jacobi-Bellman (HJB) equation should be addressed. The HJB equation is generally difficult to solve analytically. To overcome this bottleneck, a lot of remarkable approaches have been developed, such as adaptive dynamic ^[3,4], the use of actor-critic neural networks (ACNNs) ^[5,6], policy iteration, and so on. Reinforcement learning (RL), as a method that can solve optimal control problems in nonlinear systems to avoid solving the HJB equation, has received widespread attention in the past decades. In 1974, Werbos first applied the idea of RL to optimal control theory ^[7]. Since then, many outstanding outcomes have been subsequently discovered ^[8,9,10]. At present, RL theory is usually implemented by using ACNNs, where the critic neural network (CNN) provides policy evaluation and the actor neural network (ANN) updates the present policy. The RL algorithm can reduce energy consumption beyond that of other algorithms under the premise of achieving system stability; it became a significant method of modern control theory. In recent years, more and more approaches to RL have been presented in various fields, including online RL ^[11,12], integral RL ^[13,14], off-policy RL ^[15,16], etc.

1.1.   Motivation

General RL strategies utilize the gradient descent method to obtain the ideal weights of neural networks, which often fall into the local optimal problem ^[17], resulting in neural network estimation errors that cannot easily meet the requirements. To overcome this bottleneck, Bai et al. proposed the multigradient recursive (MGR) algorithm in ^[17] to obtain the global optimum solution. By updating pseudo-gradients, this state-of-the-art technique can settle down the local optimal problem and accelerate the convergence rate of the neural network weight. However, the MGR algorithm suffers because of a heavy computational burden. Thus, it is necessary to mention the minimal learning parameter (MLP) scheme. It can reduce the number of updated laws without prominently reducing the estimation accuracy. Many studies have been proposed to validate the effectiveness of the MLP. Nevertheless, the aforementioned papers adopt only one of the two algorithms and fail to combine both the MLP and MGR algorithms to combine their advantages.

The optimal control problem of strict-feedback nonlinear systems has been widely studied. However, none of these strategies can be extended into the field of nonstrict-feedback systems ^[18,19]. For this sake, some scholars have proposed their studies to overcome the bottleneck. For example, Tong et al. ^[20], presented a novel fuzzy tracking control design for a nonstrict-feedback SISO system. Bai et al. ^[21] utilized MLP-based RL theory to overcome optimal control problems for a class of nonstrict-feedback systems. However, all of the above results omitted the influence of actuator fault and dead zone input. These are common factors that affect the stability of the system. This negligence can lead to severe damage and must be taken seriously. Thus, many works are presented to offset their influence. However, no one has considered the situation that the dead zones and the actuator fault occur simultaneously. Thus, how to obtain an optimal controller for a nonstrict-feedback system with actuator faults and input dead zone with minimal computation and enough accuracy will be an important task, and it is the motivation for the current investigation.

1.2.   Related work

Combining with the back-stepping technique, the authors made a thorough investigation of the tracking control problem of the strict-feedback nonlinear systems ^[14,15,16]. At present, the back-stepping approach is introduced into the analysis for the high-order nonlinear systems. Li et al. ^[22], investigated the optimal control problem of a class of SISO strict-feedback systems via the fuzzy control method. Modares et al. ^[23] have developed an integral RL approach for the strict-feedback system with input constraints. Wang et al. ^[24] proposed an optimal fault-tolerant control strategy for a nonlinear strict-feedback system via adaptive critic design.

Besides, many papers have been proposed to study novel neural network weight updated algorithms. Li et al. ^[25] utilized the MLP technique to overcome the fault-tolerant problem of a class of multiagent systems. Liu et al. ^[26] designed an RL controller by applying a MLP scheme for classic MIMO systems with external disturbance. Bai et al. ^[27] developed an event trigger control scheme for the multiagent system based on the MLP technique.

Furthermore, many scholars are committed to investigating a tolerance strategy for the input dead zone and actuator fault. For instance, Wang and Yang ^[28] studied the fault detection problem for linear systems with disturbance. Tan et al. ^[29] developed a compensation control scheme for a class of discrete-time systems that has actuator failure. In addition, Na et al. ^[30] provided an adaptive dynamic control approach for a system with an unknown dead zone.

1.3.   Paper contribution and organization

Based on the above discussion, an RL optimal controller is built in this paper to deal with the fault tolerant control problem for a class of nonstrict-feedback nonlinear systems in discrete time with an unknown dead zone input and actuator fault. To deal with the dead zone and actuator fault issues, we propose two auxiliary systems to offset the influence. The ANN and CNN are utilized to approximate the unknown terms and long-time utility functions, respectively. We propose a novel approach to update the neural networks' weight. The stability of all signals in the closed-loop is rigorously proved and tracking errors are converged to a small compact set. The novelties of this paper can be concluded to be as follows

1) We propose a novel neural networks weigh-updated algorithm to eliminate the local optimal problem and reduce computational burden. Besides, compared with the ordinary gradient descent algorithm ^[11,26], the proposed approach can achieve a faster weight convergence rate.

2) We formulate a modified backstepping method with additional parameters to offset the influence of input dead zone, the actuator fault, and the algebraic loop problem. Besides that, the unified fault-tolerant control algorithms are developed based on the RL strategy.

The organization of this paper is given below. In Section 2, descriptions of the system and radial basis function neural network (RBF NN) theory are given. In Subsection 3.1, the CNN and our novel update law are presented. In Subsection 3.2, the design procedure of the adaptive RL controller is provided. In Section 4, we propose some simulation results to show the contributions of the scheme presented in this paper. The conclusion is provided in Section 5.

2.   Problem formulation and preliminaries

2.1.   Model description

The dynamics of a standard $n$-order strict-feedback nonlinear system ^{[31,32,33,34]} can be described as follows:

where $x_{i}(k)\in R$ for $i = 1, ..., n$ represents the state variable of the system. The notation $\bar{x}_{n}(k) = \left[x_{1}(k), x_{2}(k), ..., x_{n}(k)\right]^\top\in R^n$ denotes the vectors of the states. The notations $U(k) \in R$ and $y(k)\in R$ are the input and output signals, respectively. Notation $d(k)$ stands for the external disturbance. Notations $\varphi_{i}(\cdot)$ and $\phi_{i}(\cdot)$ represent unknown smooth nonlinear functions.

Motivated by the transformation proposed in ^[18,19], the nonstrict-feedback system (2.1) can be further expressed as

To proceed smoothly, an assumption is introduced in the following sequel.

Assumption 1: According to the contributions in ^[34,35], the functions $\varphi_{i}(\bar{x}(k))$ and $\phi_{i}(\bar{x}(k))$ satisfy $0 < \underline{\varphi} < \varphi_{i}(\bar{x}(k)) < \bar{\varphi}$ and $0 < \underline{\phi} < \phi_{i}(\bar{x}(k)) < \bar{\phi}$, where $\bar{\varphi}$ and $\underline{\varphi}$ are the unknown upper bound and the unknown lower bound of $\varphi_{i}(\bar{x}(k))$ and $\bar{\phi}$ and $\underline{\phi}$ are the unknown upper bound and unknown lower bound of $\phi_{i}(\bar{x}(k))$, respectively. The external disturbance is bounded and satisfies $\vert d(k) \vert \le \bar{d}$ with $\bar{d}$ being an unknown positive constant.

The control signal with the actuator fault and input dead zone can be described as $U(k) = \psi(k)u(k)+\delta(k)$, where $\psi(k)$ and $\delta(k)$ denote the efficiency factor and the unknown drift fault of the actuator, respectively. We assume that $\psi(k)$ is a positive constant with $\psi(k) < \bar{\psi} < 1$, where $\bar{\psi}$ is an unknown constant. Further, $\delta(k)$ satisfies $\delta(k) < \bar{\delta}$ with $\bar{\delta}$ being the upper bound. The dead zone can be defined as $u(k) = D(v(k))$, where $v(k)$ represents the input of the dead zone and $D(\cdot)$ is a function of v(k) which represents the output of the dead zone. According to propositions in ^[31], the dead zone is expressed as

where $b_{r}$ and $b_{l}$ denote the right slope of the dead zone and the left slope of the dead zone, respectively. The notations $f_{r}$ and $f_{l}$ are breakpoints of the input. For the purpose of simplifying the following calculation, the expression $D(v(k))$ can be converted into a new form, as follows

where $b(k)$ and $f(k)$ can be described as

We suppose that $b(k)$ and $f(k)$ satisfy $0 < \underline{b} < |b(k)| < \bar{b}$ and $0 < \underline{f} < |f(k)| < \bar{f}$, respectively. The control signal $U(k)$ can be reorganized as

In this paper, the RL controller is developed for the nonstrict-feedback nonlinear system (2.2), ensuring the semi-globally uniformly ultimately bounded (SGUUB) capability of all signals in the closed-loop system. Based on the ACNNs, the tracking error $\xi_{1}(k)$ is required to converge to the neighborhood of zero it will be specified subsequently.

2.2.   RBF NN

Note that, the RBF NN can approximate any smooth nonlinear functions over a compact set. That is to say, considering an unknown nonlinear function $F(N)$, there exists an RBF NN ${W^*}^\mathit{T}S(N)$ such that $F(N) = {W^*}^\mathit{T}S(N)+\sigma(N)$, where $W^* = [w_{1}, ..., w_{l}]^\mathit{T}\in R^l$ denotes the ideal weight vector, $l$ represents the node number in the hidden layer and $\sigma(N)$ is the estimation error. Both $W^*$ and $\sigma(N)$ satisfy $||W^*|| < \bar{W}$ and $||\sigma(N)|| < \bar{\sigma}$ with $\bar{W}$ and $\bar{\sigma}$ as unknown upper bounds. The notation $S(N) = [s_{1}(N), ..., s_{l}(N)]^\mathit{T}$ is the vector of the basis function and $s_{i}(N)$ is applied as Gaussian form $s_{i}(N) = \mathrm{exp} \left[-\frac{(N-c_{i})^\mathit{T}(N-c_{i})}{\eta_{i}^2} \right]$, where $c_{i}$ represents the kernel of the receptive field, $\eta_{i}(k)$ denotes the width of the function. Because $0 < s_{i}(N) < 1$, we can further derive that $0 < \sum_{i = 1}^l s_{i}(N) s_{i}(N) = S(N)^\mathit{T}S(N) < l$.

3.   Design of adaptive RL controller

3.1.   CNN

The utility function ^[35] can be chosen as

where $\varpi$ is a positive constant that denotes the threshold value of tracking performance. The tracking error is written as $\xi_{1}(k) = y(k)-x_{d}(k)$ $x_{d}(k)$ indicates the reference signal. The long term strategic utility function ^[21,27] is given by

where $a$ is a predefined positive parameter satisfying $a < 1$. According to RBF NN theory, the long term utility function $M(k)$ is defined below

where $W_{M}$ and $\delta_{M}(k)$ indicate the ideal weight vector and the error of the approximation, respectively. Let $S_{M}(k)$ be the RBF NN basis function. We define $\hat{M}(k) = \hat{W}_M^\mathit{T}(k)S_{M}(k)$; it denotes the estimation of function $M(k)$ with $\hat{W}_{M}(k)$ being the estimation of the ideal weight $W_{M}$.

On the basis of the MLP scheme, $\hat{M}(k)$ can be written in the form

where $\Vert \cdot \Vert$ indicates the Euclidean norm, $\hat{\Psi}_{M}(k) = \Vert \hat{W}_{M}(k) \Vert$ is true for $\hat{\Psi}\le\bar{\Psi}$ and $\bar{\Psi}$ is a positive unknown constant.

According to the scheme in ^[36], the equation of the Bellman error is designed as

Adopting the cost function in its quadratic form $\beta_{M}(k) = (1/2)(E_{M}(k))^2$, the gradient of $\hat{W}_M$ is obtained

Defining $\omega_{M}(k-j+1) = \hat{\Psi}_{M}(k)\Vert S_{M}(k-j+1)\Vert$, we can further get

where $\iota \ge 1$ is a positive predefined constant that indicates the step length of the gradient.

Together with (3.7), the updated law of $\hat{\Psi}_{M}$ can be obtained

where $\mu_{M}$ is the selected learning rate. The structure of the CNN is shown in Figure 1.

Remark 1: The neural networks in this paper are updated by our weight-updated algorithm. As compare to the classic gradient descent method, our updated algorithm has the following advantages: 1) Reduces the computational complexity; 2) Eliminates the local optimal problem; 3) Accelerates neural networks weight convergence speed;

3.2.   Design of $n$-step adaptive neural network controller

In this section, an ANN will be utilized to implement the $n$-step backstepping RL control strategy. Specifically, two auxiliary signals are introduced in the $n$ step to eliminate the impact of the dead zone and actuator fault.

Step 1$\colon$ The tracking error can be defined as $\xi_{1}(k+n) = x_{1}(k+n)-x_{d}(k+n)$ and $\xi_{2}(k+n-1) = x_{2}(k+n-1)-\alpha_{1}(k)$. According to System (2.2), the tracking error $\xi_{1}(k+n)$ can be further deduced as

where $\alpha_{1}(k)$ denotes the virtual controller. Let

With the universal approximation capability of the RBF NN, $\gamma_{1}(k) = W_{1}^\mathit{T}S_{1}(N_{1}(k))+\sigma_{1}(k)$ can be approximated, where $W_{1}$ represents the ideal weight vector. We define $N_{1}(k) = [\bar{x}_{n}(k+n-1), x_{d}(k+n)]$ and $\sigma_{1}(k)$ indicates the approximation error. Suppose that $W_{1}$ and $\sigma_{1}(k)$ satisfy $\Vert W_{1} \Vert < \bar{W}_{1}$ and $\Vert\sigma_{1}(k)\Vert < \bar{\sigma}_{1}$, respectively. Both $\bar{W}_{1}$ and $\bar{\sigma}_{1}$ are corresponding upper bounds.

Combining (3.9) and $\gamma_{1}(k)$, $\xi_{1}(k+n)$ can be further expressed as

For the purpose of solving the algebraic loop problem which will be mentioned later, the term $o_{1}$ is proposed

where $\epsilon_{1}(k) = [\bar{x}_{1}(k+n-1), x_{d}(k+n)]$ and $\Psi_{1} = \Vert W_{1}(k)\Vert$.

Adding and subtracting (3.11) into (3.10), one can easily derive

In order to further simplify (3.12), the virtual controller is designed as

where $\hat{\Psi}_{1}(k) = ||\hat{W}_{1}(k)||$ and $\hat{W}_{1}$ is the estimation of $W_{1}$.

Substituting (3.13) into (3.12), we gets

where $\tilde{\Psi}_{1}(k) = \hat{\Psi}_{1}(k)-\Psi_{1}$.

Transforming (3.14) with the $k+1$ time instant, one has

where $k_{1} = k+n-1$ represents the time instant.

On the basis of the RL control scheme, the strategic utility function can be defined as

where $M_{d}(k)$ represents the ideal strategic utility function and it is usually defined as "0" ^[37].

The cost function is derived as $\beta_{1}(k) = (1/2)(E_{1}(k))^2$ and the gradient of $\hat{\Psi}_{1}(k)$ is deduced as

The multigradient can be further obtained as

Define $\omega_{1}(k_{1}-j+1) = \hat{\Psi}_{1}(k_{1})||S_{1}(\epsilon_{1}(k_{1}-j+1))||$. Similar to step (3.8), the MGR updated law of $\hat{\Psi}_{1}(k)$ is derived

where $\mu_{1}$ stands for the chosen learning rate. The structure of the ANN is proposed in Figure 2.

Remark 2: It is necessary to emphasize that previous works usually designed the ANN basis function to have the form $S_{1}(N_{1}(k))$, which is not the function of $x_{1}(k)$. According to this design, $\alpha_{1}(k)$ and $\hat{\Psi}_{1}(k+1)$ are all built up as functions of $N_{1}(k) = [\bar{x}_{n}(k+n-1), x_{d}(k+n)]^\mathit{T}$. This results in the algebraic loop problem proposed in ^[38]. To settle this conundrum, the term $o_{1}(k)$ is presented in this paper and we adapt $\alpha_{1}(k)$ and $\hat{\Psi}_{1}(k+1)$ as functions of $\epsilon_{1}(k) = [\bar{x}_{1}(k+n-1), x_{d}(k+n))]^\mathit{T}$.

Step i: Define the tracking error $\xi_{i}(k+n-i+1) = x_{i}(k+n-i+1)-\alpha_{i-1}(k)$ and $\xi_{i+1}(k+n-i) = x_{i+1}(k+n-i)-\alpha_{i}(k)$. Notation $\alpha_{i-1}(k)$ and $\alpha_{i}(k)$ indicate the virtual controller at Step $i-1$ and Step $i$, respectively. Similar to the process in (3.9), one has

According to the definition of $\gamma_{1}(k)$, one has $\gamma_{i}(k) = -\bigg(\frac{\varphi_{i}(\bar{x}_{n}(k+n-i)}{\phi_{i}(\bar{x}_{n}(k+n-i))}+\alpha_{i-1}(k)-\frac{\alpha_{i-1}(k)}{\phi_{i}(\bar{x}_{n}(k+n-i))}\bigg)$. The unknown function can be approximated by the RBF NN which is given as $\gamma_{i}(k) = W_{i}^\mathit{T}S_{i}(N_{i}(k))+\sigma_{i}(k)$, where $W_{i}$ and $\sigma_{i}(k)$ are defined as the ideal weight vector and the approximation error, respectively. Futhermore, we let $N_{i}(k) = [x_{1}(k+n-i), ..., x_{n}(k+n-i), x_{d}(k+n)]^\mathit{T}$.

Substituting $\gamma_{i}(k)$ into (3.20), one has

The term $o_{i}(k)$ is given in the form below

where $\epsilon_{i}(k) = [\bar{x}_{i}(x+n-i), x_{d}(k+n)]^\mathit{T}$ and $\Psi_{i}$ denotes the Euclidean norm of the weight vector $W_{i}$.

Substituting (3.22) into (3.21) yields

The same as the previous process, the virtual controller is designed as

where $\hat{W}_{i}(k)$ is the estimation of $W_{i}$ and $\hat{\Psi}(k) = ||\hat{W}_{i}(k)||$.

Substituting (3.24) into (3.23), $\xi_{i}(k+n-i+1)$ expresses

Resembling Step (3.15), (3.25) can be further described as

where $k_{i} = k-n+i$.

Let the prediction error $E_{i}(k) = \hat{\Psi}_{i}(k)||S_{i}(\epsilon_{i}(k_{i}))||+\hat{M}(k)$. According to $E_{i}(k)$, the cost function is described in its quadratic form $\beta_{i}(k) = (1/2)(E_{i}(k))^2$. The gradient of $\hat{\Psi}_{i}$ is obtained according to the derivation:

On the basis of the MGR algorithm definition, the multigradient is expressed as

The updated law of $\hat{\Psi}_{i}(k)$ is deduced according to (3.28):

where $\mu_{i}$ is the $i$ step learning rate.

Step n$\colon$ The tracking error in the $n$-th subsystem can be described as $\xi_{n}(k+1) = x_{n}(k+1)-\alpha_{n-1}(k)$. Substitute (2.2) and (2.6) into the $n$-th tracking error equation:

For the purpose of simplifying 3.30, $\pi(k)$ is defined

Using the theory of the RBF NN to approximate (3.31), one gets

where the definitions of $W_{n}$ and $\sigma_{n}(k)$ are the same as those for the steps from $1$ to $n-1$ and $N_{n}(k) = [\bar{x}_{n}(k), x_{d}(k+n)]^\mathit{T}$.

Combining (3.32) and (3.29) we derive

From (3.33) we can acquire the dynamics of the actuator fault and dead-zone shown in (2.3) and (2.6), separately. It is easy to deduce that they have the following properties

where $\vartheta$ and $\tau$ are both unknown parameters. Define the estimation of both parameters as $\hat{\vartheta}$ and $\hat{\tau}$; so the estimation error $\tilde{\vartheta}$ and $\tilde{\tau}$ are given by

Based on estimation errors of (3.35), the actual controller is designed as

where $\hat{\Psi}_{n}(k) = ||\hat{W}_{n}(k)||$ and $\hat{W}_{n}(k)$ stands for the estimation of $W_{n}$. The time index $k_{n} = k$.

With (3.35) and (3.36), the estimation fault (3.33) can be further written as

where $\tilde{\Psi}_{n}(k) = \hat{\Psi}_{n}(k)-\Psi_{n}$ and $q(k) = \Psi_{n}||S_{n}(N_{n}(k_{n}))||-W_{n}^\mathit{T}S_{n}(N_{n}(k))-\sigma_{n}(k)$. Similar to Step $i$, the strategic utility function is defined

Further define the cost function as $\beta_{n}(k) = (1/2)(E_{n}(k))^2$; the gradient of $\hat{\Psi}_{n}(k)$ yields

The multigradient yields

where $\omega_{n}(k_{n}-j+1) = \hat{\Psi}_{n}(k_{n})||S_{n}(N_{n}(k_{n}-j+1))||$.

The weight updated law for $\hat{\Psi}_{n}(k)$ can be further obtained as

where $\mu_{n}$ is a chosen positive learning rate. Let

The cost functions of two auxiliary signals are chosen to have a form that is the same as $\beta_{i}(k)$ form

The gradients are deduced

Two MGR updated laws are obtained

where $\mu_{\vartheta}$ and $\mu_{\tau}$ are positive learning factors.

Figure 3 shows the structure of the proposed control strategy. The analysis of the stability and tracking performance are mentioned in the following theorem.

Theorem 1: Consider the nonstrict feedback nonlinear system (2.2). The adaptive RL strategy includes the updated laws (3.8), (3.19), (3.29) and (3.41), the virtual controllers (3.13) and (3.24) and the actual controller (3.36). If the selected parameters that

and Assumption 1 holds, our control strategy can ensure that all signals are SGUUB and the tracking error is tolerated. The proof of Theorem 1 is shown in the appendix.

4.   Simulation

In this section, some simulation results are presented to illustrate the effectiveness of the proposed approach.

The nonstrict-feedback nonlinear discrete time system is chosen as

where $x_{1}(k)$ and $x_{2}(k)$ are the states, $U(k)$ is the input and $y(k)$ is the output. The functions $\varphi_{1}(\bar{x}_{n}(k))$ and $\varphi_{2}(\bar{x}_{n}(k))$ are chosen as $x_{1}(k)$ and $x_{2}(k)$, respectively. We choose $\phi_{1}(\bar{x}_{n}(k))$ and $\phi_{1}(\bar{x}_{n}(k))$ as $[-x_{1}(k)+0.019(1.5-x_{1}(k))exp(4x_{2}(k)/(3.4+x_{2}(k)))]/20$ and $[-x_{2}(k)+3.1(0.4-x_{1}(k))exp(1.5x_{2}(k)/(3.4+x_{2}(k)))-4(x_{2}(k)-U(k))]$, respectively. The external disturbance $d(k)$ is chosen as $0.1$cos$(0.05k)$cos$(x_{1}(k))$. The desired signal $x_{d}(k)$ can be described as $x_{d}(k) = 0.013$sin$(\pi/8+0.6k\pi/38)$.

The parameters can be selected as follows: $\Psi_{2}(0) = 0.001$, $\mu_{M} = 0.005$, $\mu_{1} = 0.008$, $\mu_{2} = 0.006$, $\mu_{\tau} = 0.05$, $\mu_{\vartheta} = 0.05$, $a = 0.00001$, $\hat{\tau}(0) = 0.2$ and $\hat{\vartheta}(0) = 0.3$. The hidden layer node numbers of the ACNNs have been set as $l_{W_{1}} = 10$, $l_{W_{2}} = 10$ and $l_{W_{M}} = 10$. The pace of the multigradient is chosen as $\iota = 10$.

Figures 4 and 5 illustrate the trajectories of signals and the tracking error when $b_r = 0.5$, $b_l = -0.5$, $f_r = 0.3$, and $f_l = -0.3$, respectively. The control output achieved precise tracking of the reference signal and the tracking error was close to zero (almost 0.001). Figures 6 and 7 illustrate the trajectories of signals and the tracking error when $b_r = 0.4$, $b_l = -0.4$, $f_r = 0.25$ and $f_l = -0.25$, respectively. The tracking performance in this scenario was also great (the tracking error was almost 0.002). According to our results, it is true that our fault-tolerant approach can offset the influence of dead zones with different parameters.

Figure 8 describes the trajectories of the ANN weight, the CNN weight, and the control input. We make a comparison between the proposed scheme and the MLP-based strategy. According to the results, it is clear that the proposed scheme achieved a faster convergence rate with the weight parameters than the ordinary MLP scheme. Figures 9 and 10 show the tracking trajectory and the tracking error without utilizing two auxiliary systems. Affected by the input dead zone and the actuator fault, tracking became extremely inaccurate and the tracking error was up to 0.01. Comparing Figures 4 and 9, it is obvious that our the approach can successfully offset the influence of the input dead zone and the actuator fault.

To verify our novel updated algorithm can reduce the computational burden, an experiment was conducted as follows. We consider that the longer the computation the greater the computational burden. The total sample number was 1000. All of results were derived in the same environment by a computer with a 3.6 GHz CPU and 16 GB RAM.

From Table 1, we can derive that the computational time of the ordinary gradient descent algorithm is 0.5445 s but the proposed updated algorithm only needs 0.2862 s. The computational time is reduced by 47.44$\%$. Combined with the previous analysis, our approach not only alleviates the computational burden of the MGR algorithm but it also achieves a faster convergence rate for the weight parameters.

5.   Conclusions

The aim of this paper was to build up a fault-tolerant controller for a class of nonstrict feedback systems with input dead zone. We proposed a novel neural network-updated algorithm to achieve a faster computational speed and eliminate the local optimal problem. Two auxiliary parameters were presented to offset the influence of the dead zone and actuator fault. To eliminate the occurrence of the algebraic loop problem, an auxiliary term was introduced to overcome this difficulty. According to the Lyapunov theory, all signals in the closed-loop system were proven to be SGUUB and the tracking error converged to the neighborhood of zero. Finally, some simulation results were presented to illustrate the effectiveness of our approach.

There are still some other difficulties in the control area; for instance, the tracking control for stochastic systems via RL will be our future topic based on our current investigation.

Acknowledgments

This work was supported by the Natural Science Foundation Project of Chongqing under Grant cstc2019jcyj-msxmX036, and also in part by the open funding project of the Guangdong Provincial Key Laboratory of Intelligent Decision and Coordination Control under Grant F2021098.

Conflict of interest

The authors declare that they have no conflict of interest.

Appendix

Step 1: Defining $\theta_{\xi1}$, $\theta_{\tilde{\Psi}1}$ and $\theta_{M}$ as positive constants. The Lyapunov function is chosen as

where $V_{11}(k) = (\theta_{\xi1}/4)\xi_{1}^2(k)$, $V_{12}(k) = (\theta_{\tilde{\Psi}1}/\mu_{1})\sum_{s = 0}^{n-1}\tilde{\Psi}_{1}(k_{1}+i)^2$, $V_{13}(k) = (\theta_{M}/\mu_{M})\tilde{\Psi}_{M}(k)^2$ and $V_{14}(k) = 2\theta_{M}\sum_{j = 1}^{\iota}\left[\tilde{\Psi}_{M}(k)||S_{M}(k-j)|| \right]^2$.

The Cauchy-Schwarz inequality is expressed as

Young's inequality is also given below

where $\tilde{a}$ and $\tilde{b}$ are arbitrary vectors.

By utilizing the inequality (A.3) and property that $0 < S(N)^TS(N) < l$, the two terms shown in (3.12) have the following quality

where $i = 1, ..., n-1$.

According to (A.2) and (3.15), the first order difference of $V_{11}(k)$ is described as

Based on (A.2) and (3.19), the first order difference of $\Delta V_{12}(k)$ can be derived

Similar to the processes in (A.6), the first order difference of $\Delta V_{13}(k)$ is calculated as

Then, consider $\Delta V_{14}(k)$; we can obtain

Combining (A.5)–(A.8), the first-order difference of $\Delta V_{1}(k)$ is derived

where $B_{1} = 2\theta_{\Psi1}\iota (\bar{W}_{1}+\bar{W}_{M})^2+\theta_{\xi1}\bar{\Psi}_{1}^2\bar{\sigma}_{1}^2+\theta_{\xi1}\bar{\phi}_{1}^2(\bar{\Psi}_{1}^2+l)^2+2\theta_{M}\iota(\bar{\Psi}_{M}(1+a)+1)^2$.

Step i: The Lyapunov function in Steps $2$ to $n-1$ is designed as

where $V_{i1}(k) = (\theta_{\xi i}/4)\xi_{i}^2(k)$, $V_{i2}(k) = (\theta_{\Psi i}/\mu i)\sum_{s = 0}^{n-i}\tilde{\Psi}_{i}(k_{i}+s)^2$ and $\theta_{\xi i}$ and $\theta_{\Psi i}$ are both positive constants.

According to (3.26), the first order difference of $V_{i1}$ can be deduced

Similar to (A.6), one deduces $\Delta V_{i2}(k)$ as

Combining (A.11) and (A.12), $\Delta V_{i}(k)$ is derived, which shows that

where $B_{i} = \theta_{\xi i}\bar{\phi}_{i}^2(\bar{\Psi}_{i}^2+l)^2+2\theta_{\Psi i}\iota(\bar{\Psi}_{i}+\bar{\Psi}_{M})^2+\theta_{\xi i}\bar{\phi}_{i}^2\bar{\sigma}_{i}^2$.

Step n: The Lyapunov function in the $n$-th step is

where $V_{n1}(k) = (\theta_{\xi n}/2)\xi_{n}^2(k)$, $V_{n2}(k) = (\theta_{\Psi n}/\mu_{n})\tilde{\Psi}_{n}^2(k)$, $V_{n3}(k) = (\theta_{\vartheta}/\mu_{\vartheta})\tilde{\vartheta}^2(k)$ and $V_{n4}(k) = (\theta_{\tau}/\mu_{\tau})\tilde{\tau}^2(k)$.

Using the above inequality, (3.37) follows

where $q(k_{n})^2 < (\bar{\sigma}_{n}+2l^{1/2}\bar{\Psi}_{n}) = \bar{q}$.

Based on (3.41) and (A.2), $\Delta V_{n2}(k)$ is given as

Obviously, similar to (A.6), the first order difference of $V_{n3}(k)$ and $V_{n4}(k)$ is:

Combining (A.15)–(A.17), one has

where $B_{n} = \frac{16}{3}\theta_{\xi n}\bar{\phi}_{n}^2\bar{\psi}^2\bar{b}^2(2\tau^2+2\vartheta^2+\bar{q})+\frac{2}{3}\theta_{\xi n}\bar{d}^2+2\theta_{\tau}\iota(\tau+\bar{\Psi}_{M})^2+2\theta_{\Psi n}\iota (\bar{\Psi}_{n}+\bar{\Psi}_{M})^2+2\theta_{\vartheta}\iota(\vartheta+\bar{\Psi}_{M})^2$.

Combining the Lyapunov function from Step $1$ to Step $n$, we can obtain

Combining (A.9), (A.13) and (A.18), we finially get

where $B = \sum_{i = 1}^n B_i$.

Selecting the parameters as $0 < \mu_{M} < 1/(l\iota a^2), 0 < \mu_{i} < 1/l\iota, 0 < \mu_{\vartheta} < 1/\iota$ and $0 < \mu_{\tau} < 1/\iota$. Thus, the first order difference of $V(k)$ can be simplified as

In this paper, the parameters $\theta_{\Psi i}$, $\theta_{\Phi n}$, $\theta_{\xi i}$, $\theta_{\xi n}$, $\theta_{M}$, $\theta_{\vartheta}$ and $\theta_{\tau}$ are respectively designed as $\theta_{\vartheta} > \frac{16}{3}\theta_{\xi n}\bar{\phi}_{n}^2\bar{\psi}^2\bar{b}^2$, $\theta_{\tau} > \frac{16}{3}\theta_{\xi n}\bar{\phi}_{n}^2\bar{\psi}^2\bar{b}^2$, $\theta_{M} > \frac{1}{a^2}(2\theta_{M}+2\sum_{i = 1}^{n}\theta_{\Psi i}+2\theta_{\vartheta}+2\theta_{\tau})$, $\theta_{\Psi i} > \theta_{\Psi i}\bar{\phi}_{i}^2$, $\theta_{\Psi n} > \frac{16}{3}\theta_{\xi n}\bar{\phi}_{n}^2\bar{\psi}^2\bar{b}^2$, $\theta_{\xi i} > 4\theta_{\xi(i-1)}\bar{\phi}_{i-1}^2$ and $\theta_{\xi n} > 3\theta_{\xi(n-1)}\bar{\phi}_{n-1}^2$. On the basis of the method, we have that $\Delta V(k) < 0$ if the following inequalities hold

In this way, all signals in the closed-loop system are proven to be SGUUB.