Fault-tolerant control of a hydraulic servo actuator via adaptive dynamic programming

1.   Introduction

As the core of the contemporary control theory, optimal control has received wide attention from researchers and manufacturers during the last few decades. Werbos ^[1] first suggested ADP, a valuable approximation method, as an appropriate tool for solving optimal control tasks in nonlinear systems. In the past few years, many ADP-based methods have been well developed for discrete-time nonlinear systems ^[2,3,4], data-based control systems ^[5], intelligent systems ^[6,7] and unknown nonlinear systems subject to various constraints ^[8,9] or faults with the actuator ^[10,11].

With the rapid development of modern technologies, systems are becoming more and more complex and extensive. For this reason, the need for controlling complex systems has been increasing lately. As a result, unavoidable system faults can affect product quality, damage equipment or harm humans. It is well known that in existing engineering systems, the system components, such as actuators and sensors, may be severely damaged due to sudden faults that occur individually or simultaneously during operation, causing severe disasters ^[12,13]. From all types of faults, actuator faults significantly contribute to reducing control system performance. In addition, the actuators may suffer from faults during long-term operations owing to hardware ageing and other factors. Therefore, it is crucial to develop FTC approaches to handle such faults and keep the acceptable performance of the control system ^[14].

It is well known that there are passive and active FTC schemes. Passive FTC is a robust control technique in terms of a priori given set of faults. Passive FTC consists of two parts: a feedback control, which is exclusively controlled by a performance controller, while uncertainties of the model and different disturbances are controlled by the controller's robustness. Hence, passive FTC schemes are entirely restricted when handling significant faults. In contrast to that, active FTC schemes have a more vital ability to tolerate faults, which accomplish stability and requested performances by actively tuning control techniques according to the decision of a fault diagnosis unit. There are various methods for operating active FTCs, such as fault accommodation, fault compensation or fault reconfiguration. FTC has extensive usage and crucial importance in different research areas, particulary in the intelligent manufacturing industry. FTC is a progressive regulation method that provides secured work of the system in the case of component or parameter faults ^[15,16,17]. The general structure of the data-driven FTC systems is shown in Figure 1.

Recently, data-driven FTC has become one of the most vibrant research areas. A new online fault detection and isolation strategy based on the multi-model concept for aircraft jet engines is considered in ^[18], in which the linearization of nonlinear dynamics was performed on a set of linear models ^[19]. The fault detection and isolation problem of nonlinear systems using a symbolic-based linear multi-model concept is discussed in ^[20]. The intelligent optimal control strategy for unknown discrete-time nonlinear systems is discussed in ^[21,22]. This control strategy is ADP based iterative algorithm ^[23,24] belonging to machine learning for solving the appropriate Hamilton-Jacobi-Bellman (HJB) equation, which is very complicated to solve directly.

Many ADP based FTC approaches have been developed in recent years, in which actuator faults represent the most frequent faults. To operate with actuator gain and bias faults, the ADP method is employed to create a sliding-mode FTC algorithm ^[25]. In ^[26], a guaranteed cost FTC technique based on actor-critic networks is studied for highly complicated nonlinear systems with actuator efficiency loss. Estimated actuator faults from a fault observer is developed in ^[27] to derive FTC scheme for a class of nonlinear systems based on ADP. An adaptive fuzzy FTC method is developed in ^[28] for a strict-feedback nonlinear system.

A large number of hydraulic servo actuator (HSA) driven machines often operate with large loads in harsh and mostly outdoor environments. External influences affect that the HSA parameters cannot be determined accurately. As a result of unknown dynamics, high nonlinearities, unmeasurable states and high-quality control of HSA has always been a challenge in research. Moreover, changes in operating conditions during work, such as oil temperature, bulk modulus, fluctuating supply pressure and pipe volume, will lead to parameter changes that reduce existing control performance. The mentioned facts make challenging problems in achieving high-quality HSA control, which is unachievable without knowledge of the exact HSA model ^[29,30].

ADP is an efficient method for achieving high HSA optimal controller performance by relying on adaptive control, optimal control and reinforcement learning ^{[22,31,32,33,34]}. ADP is a data-driven control approach that can ensure the stability of a feedback-controlled system ^[31]. In the presence of unknown system dynamics and unmeasurable states, ADP approaches based on measured input/output data from linear systems, also known as output feedback, are of significant importance. One key advantage of using output feedback approaches is that no understanding of HSA dynamics is required. This indirect method creates a series of suboptimal controllers for an unknown HSA model, which with further iterations lead to the optimal control strategy. Since nonlinear systems may be accurately described by linear models with online estimated dynamics, this increases the practical value of control strategies ^[35,36]. Furthermore, measuring the entire HSA state vector directly is not practical in actual implementations and would need very costly instruments for measuring. Therefore, it is more practical when using control algorithms that employ state reconstruction approaches instead of taking direct measurements of the states ^[31].

ADP-based FTC for HSA is recommended as a result of the discussions above. The controller learns to account for unknown HSA dynamics, disturbances and faults in order to provide expected performances from the measured input/output data. ADP, state reconstruction and output feedback are used to iteratively solve FTC. After identifying the unknown HSA model, the algebraic Riccati equation (ARE) can be solved iteratively. To fulfill the criteria of the persistent excitation condition, some exploration noise should be introduced to control input to assure consistency of approximations and get unique solutions in every iteration step ^[37]. Application of ADP-based control methods makes data collecting for the discrete-time HSA model simpler than for the continuous-time model. As a result, the discretized HSA model's state vector may be reconstructed using the collected input and output data, allowing the implementation of ADP-based FTC control.

Here, a real-time fault compensation procedure based on ADP to enable optimal control of faulty HSA system is considered in this paper. When actuator faults occur, a ADP-based control algorithm may be biased or incapable of performing optimal control. This issue is fixed by reconfiguring the ADP controller using fault compensation. The Lyapunov theory provides a reliable guarantee for a closed-loop control system with actuator faults. Despite the classical ADP approach, the control action is no longer necessary in the proposed control procedure, which efficiently decreases the computational burden. The approach under consideration is divided into two parts: an ADP-based optimal control and a real-time fault compensation. Thus, it can be very suitably and easily applied to solve fault-tolerant control problems.

The remaining parts of this paper are arranged as follows. Section 2 discusses the difficulty of modeling HSA with uncertain dynamics. In Section 3, the FTC based on ADP is presented. The results of the simulation demonstrate the applicability and efficacy of the ADP-based FTC for the HSA in the face of total uncertainty in the model in Section 4. The final observations are provided in Section 5.

2.   HSA model

The HSA under consideration comprises of a servo valve and a hydraulic cylinder, see Figure 2. $V_a$, $p_a$, $A_a$ and $V_b$, $p_b$, $A_b$ are fluid volume, fluid pressure and effective area of the head and road piston side, respectively. Internal and external leakage flow are $q_{Li}$, $q_{Le}$, forward and return flows are $q_a$, $q_b$, respectively. Disturbance force is denoted as $F_{ext}$, load spring gradient is $K_e$, while $\beta_e$ is bulk modulus of the fluid. Supply and tank pressures are $p_S$ and $p_0$, respectively. Based on the component dynamics of the HSA, which consists of the dynamics of the motion of the piston, the dynamics of the pressure in the cylinder, and the dynamics of the servo valve, an analysis of the characteristics of the HSA can be derived. The HSA model is clearly built on complicated nonlinear equations that depend on a variety of factors that are impossible to identify precisely ^[30].

If the state and input variables are defined as

the HSA's nonlinear continuous-time dynamics may be described in state-space terms as follows

in which the state dynamics is $f(x(t))$, the input function is $g(x(t), u(t))$, output function $\eta (x(t)) = x_1(t)$, and disturbance function $h(t)$, piston displacement is $y$, $x_v$ is the spool valve displacement, $m_t$ is total mass which consists of piston mass $m_p$ and payload mass $m$.

This nonlinear HSA model will also be treated as a linearized model whose parameters were acquired experimentally at various operating points ^[30]. The mathematical equations are now more appropriately represented by means of the load pressure

leading to fewer complicated dynamic equations. Applying a new state vector $\begin{bmatrix} x_1(t) & x_2(t) & x_3(t) \end{bmatrix}^T \triangleq \begin{bmatrix} y(t) & \dot{y}(t) & p_L(t) \end{bmatrix}^T$ helps us to represent the HSA in a more succinct manner. The following is the reduced order's linearized continuous-time description.

where $A = \begin{bmatrix} 0 & 1 & 0 \\ 0 & -\frac{B_C}{m_t} & \frac{A_a}{m_t} \\ 0 & -K_d & K_p \end{bmatrix}$, $B = \begin{bmatrix} 0 & 0 & K_x \end{bmatrix}^T$, $C = \begin{bmatrix} 1 & 0 & 0 \end{bmatrix}$, in which $B_C$ represents viscous friction coefficient, while $K_d$, $K_p$ and $K_x$ are hydraulic cylinder gain, pressure and valve sensitivity coefficients, respectively, for more details see ^[30].

3.   Fault-tolerant controller design based on ADP

Let us consider the linear model of HSA with actuator faults as follows

where $x(t) \in \mathbb{R}^n$, $u(t) \in \mathbb{R}^m$, $y(t) \in \mathbb{R}^r$, $f_a(t) \in \mathbb{R}^m$, represent the system state, the control input, the output, the unknown actuator fault, respectively. $A \in \mathbb{R}^{n \times n}$, $B \in \mathbb{R}^{n \times m}$, $C \in \mathbb{R}^{r \times n}$ are unknown system matrices, assuming that $\left(A, B\right)$ is controllable and $\left(A, C\right)$ is observable.

For HSA system (3.1)-(3.2) with $f_a(t) = 0$ the performance function may be written as follows:

where $x_0 \in \mathbb{R}^n$ is an initial state, $Q = Q^T \ge 0$, $R = R^T > 0$, with $\left(A, Q^{1/2} C\right)$ observable.

The objective of the design is to create a linear optimal control as follows

which minimizes performance index (3.3), in which the optimal feedback gain matrix $K^*$ is

where $P^* = \left(P^* \right)^T > 0$ is a symmetric solution to the well-known algebraic Riccati equation (ARE) ^[38]

It should be noted that this optimal control design is mainly applicable to low-order simple linear systems. In fact, for high order large-scale systems, it is usually difficult to directly solve $P^*$ from (3.6), which is nonlinear in $P$.

Also, for practical implementation of the control system, it is easier to realize the data acquisition for the discrete-time systems compared with continuous-time systems. Consequently, we transform the continuous-time HSA into the following discrete-time HSA as follows

in which $A_d = e^{Ah}$, $B_d = \int_0^h \left(e^{A\tau} d\tau \right) B$, where $h > 0$ is a specific sampling period, assuming $\omega_h = 2\pi / h$ is non-pathological sampling frequency, whose existence is well known ^[39]. In other words, the controllability and observability of the original continuous-time HSA system is kept after discretization. Namely, if the state, input, and output vector at the sampled instant $kh$ are $x_k$, $u_k$, $y_k$, respectively, then $\left(A_d, C \right)$ and $\left(A_d, Q^{1/2} C \right)$ are observable while $\left(A_d, B_d \right)$ is controllable.

The ADP-based controller for HSA system is divided into three components: state reconstruction, critic, and actor. The state reconstruction establishes a link between the input and output data and HSA states solving the optimal control of HSA with unknown dynamics. The controller's critic component is designed to estimate the control strategy based on the input/output data. In order to enhance its effectiveness, the controller learns online. At last, the better control strategy is implemented by the actor.

Minimizing (3.3) can be approximated using the PI technique if the HSA system is fault-free (i.e., $f_a = 0$). The control goal is to provide a fault compensation control strategy for systems (3.1)-(3.2) that experience actuator faults in order to stabilize the closed-loop system.

Further, the performance index for discretized system (3.7)-(3.8) is:

where $Q_d = Qh$ and $R_d = Rh$. The optimal control low minimizing (3.9) is

where discrete optimal feedback gain matrix is $K_d^* = \left(R + B_d^T P_d^* B_d \right)^{-1} B_d^T P_d^* A_d$, in which $P_d^*$ is the unique symmetric positive definite solution to

Since equation (3.11) is nonlinear in $P_d^*$, it is difficult to directly solve $P_d^*$ for high order large scale systems. Nevertheless, many efficient algorithms have been developed to numerically approximate the solution of (3.11). One of these algorithms was developed by Hewer ^[40], and is introduced in the form of Lemma 3.1.

Lemma 3.1. Let $K_0 \in \mathbb{R}^{m \times n}$ be any stability feedback gain matrix and let $P_j$ be the symmetric positive definite solution of the Lyapunov equation

where $K_j$, $j = 1, 2, \ldots$ can be updated as follows

Then, it holds

1. $A_d - B_d K_j$ is a stability matrix;

2. $P_d^* \le P_{j+1} \le P_j$;

3. $\lim\limits_{j \to \infty} K_j = K_d^*$, $\lim\limits_{j \to \infty} P_j = P_d^*$.

The solution to the nonlinear equation (3.11) is numerically approximated by iteratively solving Lyapunov equations (3.12), which are linear in $P_j$, and recursively updating the control low $K_j$ by (3.13) ^[40]. The sequences $\{ P_j \}_{j = 0}^\infty$ and $\{ K_j \}_{j = 0}^\infty$, calculated by this method, converge to $P_d^*$ and $K_d^*$, respectively. Also, a Schur matrix is denoted as $A_d - B_d K_j$ for $j = 0, 1, \ldots$. It should be noted that the method by Hewers is an offline algorithm which is dependent on system characteristics. To apply this method online for the HSA model (3.7)-(3.8), it will be constructed an output feedback ADP-based control method, which does not reliant on knowing of HSA matrices.

Remark 3.1. For Hurwitz feedback matrix $A - B K$, $K \in R^{m \times n}$ is called stabilizing feedback gain matrix for a linear system $\dot{x} = A x + B u$.

Inspired by ^[41,42], HSA model (3.7)-(3.8) may be developed employing inputs and outputs sequences on the time horizon $\left[k-N, k-1\right]$ In the form of

where

and the observability index is $N = \max(\rho_u, \rho_v)$ in which $\rho_u$ is the minimum integer which guarantees that $U(\rho_u)$ is of full column rank and $\rho_v$ is the minimal integer which assures that $V(\rho_v)$ is of full row rank ^[41]. As a consequence, $U(N)$ has a left inverse that is expressed as $U^+(N) = \left[U^T(N) U(N) \right]^{-1} U^T(N)$. An ADP-based controller with output feedback can be used to deal with the optimal control issue of HSA. State reconstruction (3.14) is an essential part of this method, which uniqueness is expressed in the form of Lemma 3.2 ^[43].

Lemma 3.2. If the system's observability and controllability requirements (3.7)-(3.8) are satisfied, the states of HSA are uniquely obtained as a result of measurable inputs and outputs as shown below

where $\Theta = \begin{bmatrix} M_u & M_y \end{bmatrix}$ is full row rank, $M_u = V(N) - M_y T(N)$, $M_y = A_d^N U^+(N)$, $z_k = \begin{bmatrix} \bar{u}_{k-1, k-N}^T & \bar{y}_{k-1, k-N}^T \end{bmatrix}^T \in \mathbb{R}^q$, in which $q = N[\dim(u)+\dim(y)]$.

According to (3.12)-(3.13), an online output feedback learning approach may be implemented in the form of $u^*_k = - \overline{K}_d z_k$, resulting in a suboptimal feature of the closed-loop system. The discrete-time model (3.7) is shown below

where $A_j = A_d - B_d K_j$. Setting $\bar{K}_j = K_j \Theta$ and $\bar{P}_j = \Theta^T P_j \Theta$, from (3.12) and (3.16) it can be obtained

in which $\bar{H}_j^1 = B_d^T \bar{P}_j B_d$, $\bar{H}_j^2 = B_d^T \bar{P}_j A_d \Theta$, $\phi _K^1 = u_k^T \otimes u_k^T - (z_k^T \otimes z_k^T) (\bar{K}_j^T \otimes \bar{K}_j^T)$, $\phi _K^2 = 2 \left[(z_k^T \otimes z_k^T)(I_q \otimes \bar{K}_j^T) + (z_k^T \otimes u_k^T) \right]$.

The rank condition written in the form of Lemma 3.3 ensures the convergence of the online output feedback learning control ^[42,44].

Lemma 3.3. Let us suppose that for a sufficiently large $s \in \mathbb{Z}_+$ it holds:

where

where $k_0 < k_1 < \cdots < k_s \in \mathbb{Z}_+$ and $\eta_{kj} = \left[u_{kj}^T, z_{kj}^T\right]^T$, $j = \overline{0, s}$, then $\left(\bar{P}_j, \bar{H}_j^1, \bar{H}_j^2 \right)$ can be uniquely solved based on $\bar{K}_j$ and measurable online data during the period $k \in \left[k_0, k_s\right]$. Further, $\bar{K}_{j+1}$ is obtained as follows

To fulfill the condition of persistent excitation, some exploration noise $e_k$ must be included in the input signal without influencing the convergence of the online learning phase ^[45,46]. Applying the experimental design theory, a sum of sine waves with different frequencies is added to the input during the learning phase, which enables persistent excitation. Note that (3.17) is called policy evaluation, which is used to uniquely solve $\bar{P}_j$, and (3.20) is policy improvement (PI), which is used to update control gain $\bar{K}_{j+1}$. It should be noticed that solving (3.17) rather than (3.12), completely removes the initial need for precise knowledge of HSA dynamics. With the formulation for $z_k$ in mind, we can see that the control policy $\hat{u}_k = -\overline{K}_k^* z_k$ involves exclusively the previous measurement of inputs and outputs.

Control law is designed as $u_k = \hat{u}_{k} + \hat{f}_a$, where $\hat{f}_a$ is the fault compensation term suggested to prevent performance decline, which can be represented as $\hat{f}_{a, k+1} = l_a \left(2 \hat{u}_{k} R - x^T \hat{B}) \right)^T$. This fault approximation is employed to online compensate the effects of the actual actuator faults.

4.   Simulation results

Consider the linear HSA (2.5)-(2.6) with actuator fault $f_a(t)$:

The term $f_a(t)$ denotes an unknown additive actuator fault as:

for the purpose of simulation. A comparative view of actual and estimated actuator fault, made up of a step and sine-type faults, is shown in Figure 3.

The weight matrices $Q$ and $R$ are set to be identity matrices, the observability index is $N = 3$ and the initial state vector is $x_0 = \left[-5, 0, 5 \right]$ in order to illustrate the effectiveness of the proposed ADP-based control of HSA while $10^{-4}$ is chosen as the convergence threshold $\varepsilon$.

To illustrate the efficacy of the proposed ADP based FTC, the system input and output as well as system states of HSA, without the fault compensation are shown in Figure 4 and Figure 5.

As a result of the unknown actuator fault, the control law $u_k$ cannot provide system stability. It is clear that for good trajectory tracking, online compensation of the actuator fault must be included in the control low.

The system input and output of HSA as well as the system states under ADP-based control with fault compensation, are shown in Figure 6 and Figure 7. It is evident that the system states quickly reach equilibrium once a fault arises, which is used to learn the fault online.

In spite of the existence of system uncertainties and measurement noises, the tracking capabilities are good. It can be seen that despite the presence of system uncertainties and measurement noises, the tracking performances are good.

Finally, it can be shown that the designed controller has good flexibility and robustness. In order to further quantify the results of the comparison of the fault-tolerant controller using adaptive dynamic programming versus the ADP-based controller without fault compensation, the following three performance indices are given:

The index MSE is used to evaluate the tracking accuracy. The index IAE can evaluate the overshoot of the whole process of the system. The index ITAE is employed to evaluate the speed of the transient response. The numerical results of these three indices are shown in Table 1.

For MSE, the proposed fault-tolerant controller based on ADP is about 99.98% lower than the ADP-based controller without fault compensation. For IAE, the proposed fault-tolerant controller based on ADP control is about 97.42% lower than the ADP-based controller without fault compensation. Moreover, for ITAE, the proposed fault-tolerant controller based on ADP is about 98.52% lower than the ADP-based controller without fault compensation. As a result, it can be obtained from the three indices in Table 1 that the proposed scheme can achieve better tracking performance with smaller tracking error, smaller overshoot and faster response than the ADP-based controller without fault compensation.

5.   Conclusions

We develop a new data-driven FTC scheme for HSA with completely unknown dynamics and the presence of actuator faults based on ADP with an enhanced performance index. The provided FTC strategy has the primary benefit of avoiding knowledge of full system dynamics and faults using output feedback, the state reconstruction method and adaptive dynamic programming, which is extremely significant in real-world settings. As a result, data-driven FTC for discretized HSA was iteratively developed, and very efficiently ensures online solutions to control HSAs, which are regarded as highly nonlinear dynamic systems. Future research may involve event-triggered ADP-based control for industrial systems in order to reduce the interaction between the controller and the actuators.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgment

This research has been supported in part by the Serbian Ministry of Education, Science and Technological Development (451-03-47/2023-01/200108).

Conflict of interest

The authors declare that there are no conflicts of interest in this paper.