Reinforcement learning-based adaptive tracking control for flexible-joint robotic manipulators

1.   Introduction

In recent decades, automation has flourished, leading to the widespread integration of robots across various sectors, including industrial production ^[1], healthcare ^[2], defense ^[3], aerospace engineering ^[4], and numerous other domains ^[5,6,7]. Robots used in industrial production are typically made of rigid materials, which results in high manufacturing costs and limited degrees of freedom. Furthermore, because of their relatively rigid structure, they are not well-suited for complex environments and may struggle to efficiently complete tasks in situations that involve interacting with unpredictable environments or objects. Therefore, the control problem of flexible-joint robotic manipulators with high adaptability and an extensive range of degrees of freedom has received much attention, and various approaches have been developed(e.g., ^{[8,9,10,11,12]}), among which the backstepping-based strategy would be the commonly used only due to the advantages in handling nonlinearities ^{[13,14,15,16,17,18,19]}.

The backstepping controller, which utilizes a sampled-data extended state observer (SD-ESO), was proposed in ^[17] as a methodology to optimize the transient response of a flexible-joint robotic manipulator. This methodology is devised to minimize estimation inaccuracies and other constraints, thereby enhancing the overall performance of the robotic system. In ^[18], an explicit state feedback controller has been designed to solve the problem of practical tracking control of a flexible-joint robotic manipulator in the presence of actuator saturation by cleverly combining an inverse stepping scheme, an adaptive technique and a method of constructing a command filter and an actuator saturation assist system. In the study presented in ^[19], an adaptive control scheme is introduced to ensure the convergence of tracking deviations in a flexible-joint robotic manipulator. The methodology employs a backstepping control strategy to ensure that the deviation converges within a specified timeframe to a predetermined range. While the tracking accuracy and convergence rate can be well improved with the existing backstepping-based control schemes such as those mentioned above, they overlook the energy consumption of the controller. Considering that flexible manipulators require more energy for deformation and adjustment compared to rigid manipulators, optimizing energy consumption becomes crucial to enhance system performance and reduce operational costs. Therefore, it is crucial to implement control methods to optimize energy consumption.

Bellman in ^[20] and Pontryagin in ^[21] proposed the optimal control. This control approach aims to find control strategies for dynamical systems and to optimize the structured cost metric, thus achieving a harmonious balance between the available resources and required performance. However, since the optimal control is typically determined by solving the Hamilton-Jacobi-Bellman (HJB) equation ^[22], its inherent nonlinearity and complexity make it challenging to solve directly using analytical methods. Fortunately, the adaptive dynamic programming (ADP) or reinforcement learning (RL) proposed by Werbos et al. ^[23,24,25] provides an efficient technique for learning solutions to the HJB equation. The fundamental concept underlying this methodology is to modify the action step-by-step through feedback from the environment. This is generally achieved through the interactive learning of two neural networks (NNs): the actor and the critic. The critic plays a pivotal role in evaluating the actor's actions and providing feedback that guides the actor's policy optimization and subsequent action execution. Therefore, the energy consumption problem of the flexible-joint robotic manipulator can be managed by incorporating optimal control based on RL into the backstepping control. It should be pointed out that, integrating optimized control into the backstepping control of a flexible-joint robotic manipulator remains challenging due to the complexity of system control and convergence analysis.

In this paper, we propose a trajectory tracking control approach for flexible-joint robotic manipulators. By integrating optimization techniques into the backstepping control framework, we formulate each controller as an optimal solution tailored for its respective subsystem. This approach enhances the overall control efficacy of the flexible-joint robotic manipulator system. Concurrently, we employ RL grounded in the NN-based actor-critic architecture to tackle the intricate challenge posed by the HJB equation. In summary, the contributions of this paper are as follows:

(1) By constructing the performance index function with an error term and controller input, the controller is designed to minimize energy consumption and achieve the desired trajectory tracking task of the flexible-joint robotic manipulator.

(2) In the optimal backstepping control of a flexible-joint robotic manipulator, RL based on a NN actor-critic architecture is utilized. In this setup, the critic evaluates performance and provides feedback to the actor, which then executes the actor. This simplifies the design of the controller for the higher-order nonlinear flexible-joint robotic manipulator model.

The rest of this paper is organized as follows. In Section 2, we formulate the control problem, and give some fundamentals for design and analysis. In Section 3, a complete procedure is presented to show how an optimized controller is constructed, and the closed-loop stability is established. In Section 4, simulation results are collected to illustrate the effectiveness of our scheme. The whole paper is concluded in Section 5.

2.   Problem statement and preliminaries

2.1.   Problem description

Disregarding the viscous damping effects, as referenced in ^[26], we obtain the dynamic equations for the single-link flexible-joint robotic manipulator depicted in Figure 1.

where $q_1$ and $q_2$ are the angular positions of the link and motor shaft, and $u$ is the torque generated by the driving motor. The inertia $I$ and $J$, the link mass $M$, the gravity acceleration $g$, the position of the link's center of gravity $l$, and the coefficient of strength of the spring $k$ can be obtained by the identification system, so all of them are regarded as known parameters.

By selecting the state variables, $x_1 = q_1$, $x_2 = \dot{q}_1$, $x_3 = q_2$, $x_4 = \dot{q}_2$, the dynamic equation of system (2.1) becomes

System (2.2) is equivalent to the following nonlinear model

where $f_2(\bar{x}_2(t)) = -\frac{Mgd}{J_1}\sin (x_1(t))-\frac{k}{I}x_1(t)$, $g_{2} = \frac{k}{I}$, $f_4(\bar{x}_4(t)) = \frac{k}{J}\big(x_3(t)-x_1(t)\big)$, $g_{4} = \frac{1}{J}$. $y(t) \in R$ is the system output, $u(t) \in R$ is the control input, $f(\bar{x}_i(t)) \in R$ is a known and bounded continuous function, and $\dot{x}_i(t), i = 1, \dots, 4$, are assumed to exhibit stabilizability properties within the subsets that include the origin, and to satisfy the Lipschitz continuous.

Remark 2.1. The assumption that $\dot{x}_i$ satisfies Lipschitz continuous is made here to ensure that the system evolves smoothly over time, preventing sudden changes that could lead to instability or suboptimal performance to facilitate optimal control. Moreover, the system's seamless progression is ensured to remain within a defined boundary, subject to the confinement imposed by the Lipschitz continuity condition. In other words, the velocity of variation exhibited by the system's state variables is confined to a bounded region, dictated by a Lipschitz constant.

Definition 2.1. (Stable and ultimately uniformly bounded (SGUUB) ^[27]). For a nonlinear system with the state vector $x(t) \in R^n$

Its solution is said to be SGUUB if, for $x(0) \in \Omega_{x}$ where $\Omega_{x} \in R^n$ is a compact set, there exist two constants $\sigma$ and $T(\sigma, x(0))$, such that $\|x(t)\|\leq\sigma$ is held for all $t > t_{0}+T(\sigma, x(0))$.

The solution is characterized as SGUUB when, for any initial condition $x(0)$ within the compact subset $\Omega_{x} \in R^n$, there exist positive scalar constants $\sigma$ and $T(\sigma, x(0))$ that satisfy the inequality $\|x(t)\|\leq\sigma$ for all time instants t exceeding the initial time $t_0$ by a duration greater than $T(\sigma, x(0))$.

Lemma 2.1. Given $G(t)\in R$ with $G(0)$ bounded, if ${\dot{G}}(t)\leq-aG(t)+c$ for $a, c > 0$, then $G(t)\leq e^{-at}G(0)+\frac{c}{a}(1-e^{-at})$.

Control objectives: In developing a critic-actor RL-based optimal control strategy for the single-link manipulator system (2.3), our objective is to ensure the following:

P1) Within the closed-loop control framework, all error signals, designated as $z_i(t)$ for $i = 1, \cdots, 4$, and the weight estimation errors, expressed as $\tilde{W}_{ci}(t)$ and $\tilde{W}_{ai}(t)$ for $i = 1, \cdots, 4$, are assured to be SGUUB in a predictable and desirable fashion;

P2) The single-link manipulator joint angular position $q_1(t)$ exhibits the capability to follow the desired trajectory $y_r$ in a predictable and desirable manner.

2.2.   Basic knowledge of optimal control

To describe the optimal control strategy, consider the following nonlinear continuous-time dynamic system:

where $x(t) \in R^n$ represents the state variable, $f(x)\in R^n$ denotes a continuous function, $u(x) \in R^m$ signifies the input signal, and the term $g(x)\in R^{n\times m}$ is the continuous gain function. Assuming that the derivative $x(t)$ exhibits Lipschitz continuity within the set $\Omega$ encompassing the origin, it ensures the uniqueness of the solution for the nonlinear system (2.4) with bounded initial values. Furthermore, the stabilizability of the system (2.4) implies the availability of a continuous control function $u$ that can asymptotically stabilize the system, as referenced in ^[28].

Define the performance index of the dynamic system (2.4) as follows

where $r(x, u) = x^TP_1x+u^{T}P_2u$ is the cost function, $P_1 = P^T_1 \in R^{n\times n}$ and $P_2 = P^T_2 \in R^{m\times m}$ are two positive semi-definite matrices, and $P_2$ signifies the impact of control efforts on the total cost.

Definition 2.2. The control strategy $u(x)$ is considered acceptable on $\Omega$, denoted as $u(x) \in \Psi(\Omega)$, if $u(x)$ is continuous, $u(0) = 0$, $u$ is stable on $\Omega$, and $V(x)$ is finite.

When addressing the optimization of control strategies related to system (2.4), the primary objective is to determine a suitable control strategy, denoted as $u(x)$ and belonging to the set $\Psi(\Omega)$, that enables the minimization of the value function $V(x)$. Define the HJB function for system (2.4) as follows

where $V_{x}(x) = \partial V(x)/\partial x$ is the partial differentiation of the performance index function $V(x)$ with respect to the variable $x$.

To obtain optimal control, define the optimal function $V^*(x)$ for the dynamic system (2.4) mentioned above with the optimal input $u^{*}(x)$ as follows:

The HJB function is then obtained as follows:

where $V_{x}^*(x) = \partial V^*(x)/\partial x$ denotes the partial derivative of the optimal performance index function $V^*(x)$ with respect to $x$.

Assuming that (2.5) has, and only has, a unique solution, by solving the equation $\partial H(x, u^*, V_x^*)/\partial u^* = 0$, the expression of $u^{*}(x)$ is derived as

Substituting (2.6) into (2.5) gives the following result as

The optimal control policy $u^{*}(x)$ in (2.5) is unknown because the term $V_x^*(x)$ is unknown, but it can be obtained by solving (2.7) to find the gradient term $V_{x}^*(x)$, and then substituting $V_{x}^*(x)$ into (2.6). However solving (2.7) is difficult or even impossible, especially for some high-order systems. To tackle such a problem, the prevalent approach in the extant literature involves employing the technique of RL with an actor-critic architecture: see in ^[29].

2.3.   Neural networks and function approximation

Multiple use cases have formalized the strong function approximation and adaptive learning capabilities of NNs. Distinctly, for any given nonlinear and continuous function $F(z):R^n\to R^m$ that is defined over a compact domain $\Omega$, NNs of a specific configuration can serve as a proximate representation

where $W\in R^{p\times m}$ is the weight of the NN, $\Gamma(z) = [\gamma_1(z), \gamma_2(z), …, \gamma_p(z)]^T\in R^p$ represents the Gaussian basis function vector, and $p$ signifies the total number of neurons. Specifically, the expression for $\gamma_i$ where $i = 1, …, p$ is given as follows:

where $v_i = [v_{i1}, v_{i2}, \dots, v_{in}]$ are centers of the respective field, and $\varphi _i$ is the width of the Gaussian function.

In accordance with theoretical principles, there ought to exist an optimal weight matrix, denoted as $W^{*}$, which enables the accurate representation of $F(z)$ as follows

where $\varepsilon (z)\in R^m$ denotes the approximation error that when the number of neurons $p$ is large enough to satisfy $\|\varepsilon (z)\|\le \delta$, $\delta$ is an extremely small positive constant, and $W^*$ is the ideal weight used only for making stability analysis, denoted as

3.   Optimized backstepping design and stability analysis

Step 1: In this step, the tracking deviation vector is defined as $z_1(t) = x_1(t)-y_r(t)$. From (2.3), it can be deduced that its derivative is

The optimal virtual control for the first step is denoted by $\alpha^*_1(z_1)$, with the optimal value function being defined accordingly,

where $\alpha_{1}(z_1)$ is the virtual control, $\Omega_{z_1}$ is the admissible set of $\alpha_1^*$, and $r_1 = z_1^{2}(t)+\alpha_1^{2}(z_1)$ is the cost function in the first step. The optimal performance index function $V^*_1(z_1)$ is divided into two components as shown below to facilitate the construction of optimal tracking control,

where $\beta_1 > 0$ is a designable constant, and $V_1^o(z_1) = -\beta_1z_1^2(t)+V_1^*(z_1)$. By viewing $x_2(t)$ as $\alpha_1^*$, the HJB function can be obtained from tracking error (3.1) and the optimal function (3.3) as follows

The optimal virtual control $\alpha_{1}^{*}$ can be derived by solving $\partial H_{1}/\partial\alpha_{1}^{*} = 0$ as

Because solving $\partial V_{1}^{o}(z_{1})/\partial z_{1}$ is complex, but the term is continuous for $\Omega_{z_1}$, it can be approximated with an NN as

where $W_1^{*T}\in R^{m_1}$ represents the ideal weight in the NN, and the item $\Gamma_1(z_1)\in R^{m_1}$ signifies the basis function in the NN, and $\varepsilon _1(z_1)\in R$ is the bounded approximation error.

Remark 3.1. Note that both NNs and FLSs can be used to approximate uncertain functions: see ^[30,31,32] for examples. Nevertheless, compared with FLS, the NN approximator could have the following advantages: 1) NNs eliminate the need to formulate a rule base, as they can automatically learn the input-output mapping relationship through training, making the process less complex, and 2) NNs can effectively handle anomalous samples through an adaptive mechanism.

With the aid of (3.6), it can be derived from (3.3) and (3.5) that

Substituting (3.6) and (3.8) into (3.4), we can get the following expression:

where $\epsilon_{1}(t) = \varepsilon_{1}(z_{1})\big(-\dot{y}_{r}(t)+\alpha_{1}^{*}\big)+(1/4)\varepsilon_{1}^{2}(z_{1})$ is bounded.

Due to the uncertainty surrounding the ideal weight $W_{1}^{*}$, the optimal virtual control in (3.8) remains undetermined. Therefore, to achieve the desired tracking control, we employ an RL algorithm based on an actor-critic framework. In this framework, we use the critic module to assess the effectiveness of the control, while the actor component formulates the virtual control signal

where $\hat{V}_{1}^{*}$ is the estimation of $V_{1}^{*}$, $\hat{W}_{c1}\in R^{m1}$ represents the weight of critic NN, and $\hat{W}_{a1}\in R^{m1}$ is the actor NN weight.

Remark 3.2. It's worth noting that unlike the single NN approach for approximating unknown functions discussed in ^[31] and other works, this paper employs RL based on actor-critic NNs. In this framework, the critic evaluates performance and provides feedback to the participants, who then execute the suggested actions. Since the critic offers direct feedback on the policy, the actor can focus on optimizing the policy, resulting in a more stable and effective update. In contrast, a single NN typically updates its strategy based on direct returns to adjust the policy, which can result in greater variance and negatively impact the efficiency and stability of the learning process.

By incorporating Eqs (3.10) and (3.11) into the framework of (3.4), the HJB equation is derived as

Bellman residual error $e_1(t)$ can be derived from (3.9) and (3.12) as

Define the positive definite function of the Bellman residual error (3.13) as

To achieve the minimization of $E_1(t)$, the update law for the critic NN is derived by employing the method of gradient descent,

where $\mu_{c1} > 0$ is the learning rate of critic NN and $\omega_1 = \Gamma_1(z_1)\big(-\beta_1z_1(t)-(1/2)\hat{W}_{a1}^{T}\Gamma_1(z_1)-\dot{y}_r\big) \in R^{m_1}$.

Remark 3.3. The matrix $\omega_i(t)$ needs to satisfy the following equation for every $t$ within the interval $[t, t+\bar{t}_i]$:

where $\Lambda_i$, $\eta_i$, and $\bar{t}_i$ are all positive values, and $I_{m_i}\in R^{m_i\times m_i}$ is the identity matrix. Satisfying the aforementioned incentive persistence conditions enhances the robustness and adaptability of the system, which further ensures the stability and performance of the flexible-joint robotic manipulator system.

The actor NN weight is updated by the following law

where $\mu_{a1} > 0$ is the actor learning rate.

Designate the tracking discrepancy for the second step as $z_2(t) = x_2(t)-\hat{\alpha_{1}}(z_1)$. Replace $x_2(t)$ with $z_2(t)+\hat{\alpha_{1}}(z_1)$, then we can yield (3.1) as follows:

Taking into account the scalar quadratic Lyapunov function pertaining to the first step, its formulation is presented as follows:

where $\tilde{W}_{c1}(t) = \hat{W}_{c1}(t)-W_{1}^{*}$ is the critic NN weight error, and $\tilde{W}_{a1}(t) = \hat{W}_{a1}(t)-W_{1}^{*}$ is the NN weight error of the actor. The derivative of (3.19) is

Then, recalling the tracking error (3.18), the updating law (3.15) (3.17), and the virtual control (3.11), we have

By collating Eq (3.21), the following expression can be obtained:

The following results can be deduced because of the equation $\tilde{W}_{a1}(t) = \hat{W}_{a1}(t)-W_{1}^{*}$:

By inserting (3.23) and (3.24) into (3.22), $\dot{L}_1(t)$ is rewritten as

Utilizing Young's inequality $ab\leq(a^2/2)+(b^2/2)$, the following results are derived

By substituting (3.26), (3.27), and (3.28) into (3.25), we can get the following derivation:

There is the following fact:

then we can rewrite the inequality (3.29) as

Given the equation $\tilde{W}_{a1}(t) = \hat{W}_{a1}(t)-W_{1}^{*}$, it leads to the following equations:

Pursuant to Young's inequality, the subsequent consequence can be deduced

Adding (3.32) and (3.33) into (3.31) yields

Substituting the following equation

into (3.34), we have

Employing the principles of Young's inequality in conjunction with Cauchy's inequality, a series of inequalities can be formulated as follows:

By incorporating the aforementioned inequalities into (3.36), we have made the necessary substitution,

Rewrite (3.39) as follows:

where $\xi_1(t) = [z_1(t), \tilde{W}_{a1}^T(t), \tilde{W}_{c1}^T(t)]^T$, $C_1(t) = \frac12\dot{y}_r^2(t)+\frac{\mu_{a1}+1}2\big(W_1^{*T}\Gamma_1(z_1)\big)^2+\frac{\mu_{c1}}2\epsilon_1^2(t)$.

In accordance with the persistence of excitation (PE) assumption, the positivity of the definite matrix $A_1(t)$ can be ensured through the deliberate design of the parameters $\beta_{1}$, $\mu_{c1}$ and $\mu_{a1}$ in such a way that they satisfy the specified set of inequalities

where $\lambda_1$ is the maximal eigenvalue of $\Lambda_1 = W_1^{*T}\Gamma_1(z_1)\Gamma_1^T(z_1)W_1^*$. Then, (3.40) becomes

where $a_1$ is the lower bound on the minimum eigenvalue of $A_1(t)$ and $c_1$ is the maximum value of $C_1(t)$.

Step 2: According to the tracking error $z_2(t) = x_2(t)-\hat{\alpha_{1}}(z_1)$ in the second step, it yields that

The optimal value function $V_{2}^{*}(z_2)$ in second step can be defined with the dynamic error $z_2(t)$ and the optimal virtual control $\alpha ^*_2$ as

where $r_2 = z_2^{2}(t)+\alpha_2^{2}(z_2)$ is the cost function, and $\alpha_{2}(z_2)$ represents the virtual control. $\Psi(\Omega_{z_2})$ is the set of admissible control policies over $\Omega_{z_2}$, where $\Omega_{z_2}$ denotes a compact set that includes the origin of the system. To minimize the tracking error $z_2(t)$, we can rewrite the optimal value function $V_{2}^{*}$ as

where $\beta_2$ is a positive designable constant and $V_2^o(z_2) = -\beta_2z_2^2(t)+V_{2}^{*}(z_{2})$ is a scalar-valued function. According to both (3.43) and (3.45), the HJB equation of the second step is

Assuming that there is a solution and that it is unique, then by solving $\partial H_{2}/\partial\alpha_{2}^{*} = 0$, the optimal virtual control $\alpha_{2}^{*}$ is

Utilizing an NN approximator to estimate $\partial V_{2}^{o}(z_{2})/\partial z_{2}$ yields that

where $W_2^{*T}\in R^{m_2}$ signifies the ideal weight in the NN, and the item $\Gamma_2(z_2)\in R^{m_2}$ represents the basis function, and $\varepsilon _2(z_2)$ denotes the approximation error that is bounded. The gradient term $\partial V_{2}^{*}(z_{2})/\partial z_2$ and the optimal virtual control $\alpha _2^{*}(z_2)$ become

The optimal virtual control (3.50) cannot be used directly because the ideal weight vector $W_2^{*T}$ is unknown. To achieve an effective and optimized control strategy, we implement an RL based on actor-critic NNs for deriving practical optimization

where $\hat{V}_{2}^{*}$ is the estimation of $V_{2}^{*}$, $\hat{W}_{c2}\in R^{m2}$ represents the weight of critic NN, and $\hat{W}_{a2}\in R^{m2}$ denotes the actor NN weight. Upon inserting Eqs (3.51) and (3.52) into (3.46), we obtain the HJB equation

Remark 3.4. To ensure the boundedness of the HJB function $H_2$, here we prove the boundedness of $\dot{\hat{\alpha}}_1(z_1)$.

The expression for $\dot{\alpha_{1}}(z_1)$ is as follows:

Because the term $\dot{x}_1$ satisfies Lipschitz continuity, it is bounded. Obviously, $\dot{y}_r(t)$ and $\dot{\hat{W}}_{a1}^{T}\Gamma_{1}(z_{1})+\hat{W}_{a1}^{T}\dot{\Gamma_{1}}(z_{1})$ are also bounded. Consequently, the function $\dot{\alpha_{1}}(z_1)$, which consists of these bounded terms, is also bounded. Furthermore, $\dot{\alpha_{i}}(z_{i}), i = 1, \dots, 3$ is bounded at each step, although this will not be shown hereafter.

To optimize the function $E_2(t) = e_2^2(t)/2$, we employ the gradient descent methodology. Then we can derive the subsequent update law for the critic NN weight $\hat{W}_{c2}(t)$,

where $\mu_{c2} > 0$ is the learning rate and $\omega_2 = \Gamma_2(z_2)\big(f_2(\bar{x}_2)-\beta_2g_{2}^2z_2(t)-(g_{2}^2/2)\hat{W}_{a2}^{T}\Gamma_2(z_2)-\dot{\hat{\alpha}}_1(z_1)\big) \in R^{m_2}$. The renewal law of actor NN weight $\hat{W}_{a2}(t)$ is designed as

where $\mu _{a2} > 0$ is the learning rate of the actor NN.

By introducing the error variable in the third step as $z_3(t) = x_3(t)-\hat{\alpha _{2}}(z_2)$, we can rewrite (3.43) as

Design the Lyapunov function as

where $\tilde{W}_{c2}(t) = \hat{W}_{c2}(t)-W_{2}^{*}$ and $\tilde{W}_{a2}(t) = \hat{W}_{a2}(t)-W_{2}^{*}$. Its derivative is as follows:

Inserting (3.52), (3.54), (3.55), and (3.56) into (3.58), we have

Analogous to the first step, we can obtain the inequality shown as follows:

Rewrite (3.60) as follows:

with the matrix $\xi_2(t) = [z_2(t), \tilde{W}_{a2}^T(t), \tilde{W}_{c2}^T(t)]^T$ and the term $C_2(t) = \frac{1}{2}f_{2}^{2}(\bar{x}_{2})+\frac{1}{2}\dot{\hat{\alpha}}_1^2+\frac{\mu_{c2}}2\epsilon_2^2(t)+\frac{\mu_{a2}+g_{2}^2}2\bigg(W_2^{*T}\Gamma_2(z_2)\bigg)^2$.

In order to satisfy that the matrix $A_2(t)$ is positive definite, the parameters are designed as follows:

where $\lambda_2$ is the maximal eigenvalue of matrix $\Lambda_2 = W_2^{*T}\Gamma_2(z_2)\Gamma_2^T(z_2)W_2^*$. Consequently, we have

where $a_2$ is the minimum eigenvalue of $A_2(t)$ and $c_2$ is the maximum value of $C_2(t)$.

Step 3: Define the tracking error between $x_3(t)$ and $\hat{\alpha_{2}}(z_2)$ for the third step as $z_3(t) = x_3(t)-\hat{\alpha_{2}}(z_2)$. Its time derivative along the pure-feedback system (2.3) is

In the process, we first define the virtual control term $\alpha_3(z_3)$ and further introduce its optimal counterpart, denoted as $\alpha_3^*(z_3)$. Describe the performance index function $V_{3}^{*}(z_3)$ as

where $r_3 = z_3^{2}(t)+\alpha_3^{2}(z_3)$ is the cost function, and the set $\Omega_{z_3}$ represents a compact domain that encompasses the origin of the system. Rewrite the optimal value function $V_{3}^{*}$ as

where $\beta_3$ is a positive designable constant and $V_3^o(z_3) = -\beta_3z_3^2(t)+V_{3}^{*}(z_{3})$ is a scalar-valued function. Then, we can derive the HJB equation as follows:

By solving $\partial H_{3}/\partial\alpha_{3}^{*} = 0$, the optimal virtual control $\alpha_{3}^{*}$ is

By applying NN, the part $\partial V_{3}^{o}(z_{3})/\partial z_{3}$ can be approximated as

where $W_3^{*T}\in R^{m_3}$ represents the ideal weight, $\Gamma_3(z_3)\in R^{m_3}$ denotes the basis function in the NN, and $\varepsilon _3(z_3)$ signifies the bounded approximation error. With (3.69), the gradient term $\partial V_{3}^{*}(z_{3})/\partial z_3$ and the optimal virtual control $\alpha _3^{*}(z_3)$ are obtained:

Since $W^{*}_3$ is not directly available, an RL based on the actor-critic architecture is employed as

where $\hat{V}_{3}^{*}$ is the estimation of $V_{3}^{*}$, $\hat{W}_{c3}$ is the weight of critic NN, and $\hat{W}_{a3}$ is the weight of actor NN. Substituting (3.72) and (3.73) into (3.67), we can rewrite the HJB equation as

To minimize $E_3(t) = e_3^2(t)/2$, design the following updating laws for the weights in the critic and actor NNs

where $\mu_{a3} > 0$ and $\mu_{c3} > 0$ represent the designable learning rates of the actor NN and critic NN, respectively, and $\omega_3 = \Gamma_3(z_3)\big(-\beta_3z_3(t)-(1/2)\hat{W}_{a3}^{T}\Gamma_3(z_3)-\dot{\hat{\alpha}}_2(z_2)\big) \in R^{m_3}$.

The tracking error in the step 4 is written as $z_4(t) = x_4(t)-\hat{\alpha_{3}}(z_3)$, then (3.64) is replaced as

The Lyapunov function can be formulated as described below:

where $\tilde{W}_{c3}(t) = \hat{W}_{c3}(t)-W_{3}^{*}$ represents the estimation error of the critic NN, while $\tilde{W}_{a3}(t) = \hat{W}_{a3}(t)-W_{3}^{*}$ is the actor NN estimation error. The derivative of the Lyapunov quadratic scalar function (3.78) is

The equation along with (3.73), (3.75), (3.76), and (3.77) is

Applying the control (3.73), (3.75), and (3.76), similar with step 1, we have the result

Rewrite (3.81) as follows:

where $\xi_3(t) = [z_3(t), \tilde{W}_{a3}^T(t), \tilde{W}_{c3}^T(t)]^T$, $C_3(t) = \frac{1}{2}\dot{\hat{\alpha}}_2^2+\frac{\mu_{a3}+1}2\big(W_3^{*T}\Gamma_3(z_3)\big)^2+\frac{\mu_{c3}}2\epsilon_3^2(t)$.

Select parameters within the following intervals:

where $\lambda_3$ is the maximal eigenvalue of matrix $\Lambda_3 = W_3^{*T}\Gamma_3(z_3)\Gamma_3^T(z_3)W_3^*$. We have

where $a_3$ is the lower bound on the minimum eigenvalue of $A_3(t)$ and $c_3$ is the maximum value of $C_3(t)$.

Step 4: The actual input $u$ is obtained in the final step. The tracking error is $z_4(t) = x_4(t)-\hat{\alpha_{3}}(z_3)$, then

The performance index function in the final step is described as

where $u^*$ is the optimal actual input and $r_4 = z_4^{2}(t)+u^{2}(z_4)$ represents the cost function.

Without prejudice to generality, the actual controller $u(z_4)$ can be obtained as follows:

where $\hat{W}_{a4}$ is the weight of actor NN. With the critic and actor updating law

where $\mu_{c4} > 0$ and $\mu_{a4} > 0$ are the critic and actor learning rate, separately, and $\omega_4 = \Gamma_{4}(z_4)\big(f_4(\bar{x}_4)-\beta_4g_{4}^2z_4(t)-(g_{4}^2/2)\hat{W}_{a4}^{T}\Gamma_{4}(z_4)-\dot{\hat{\alpha}}_3(z_3)\big) \in R^{m_4}$.

In the final step, the Lyapunov quadratic scalar function is chosen as

where $\tilde{W}_{c4}(t) = \hat{W}_{c4}(t)-W_{4}^{*}$ is the critic NN estimation error, and $\tilde{W}_{a4}(t) = \hat{W}_{a4}(t)-W_{4}^{*}$ is estimation error of the actor NN. The derivative of (3.90) is

According to (3.87), (3.88), and (3.89), we have

Similar to the first step, we can also deduce the following result

Rewrite (3.93) as follows:

with the matrix $\xi_4(t) = [z_4(t), \tilde{W}_{a4}^T(t), \tilde{W}_{c4}^T(t)]^T$, and the term $C_4(t) = \frac{1}{2}f_{4}^{2}(\bar{x}_{4})+\frac{1}{2}\dot{\hat{\alpha}}_3^2+\frac{\mu_{a4}+g_{4}^2}{2}\big(W_4^{*T}\Gamma_{4}(z_4)\big)^2+\frac{\mu_{c4}}2\epsilon_4^2(t)$.

To ensure system stability, the design parameters $\beta_{4}$, $\mu_{c4}$, and $\mu_{a4}$ must satisfy

where $\lambda_4$ is the maximal eigenvalue of matrix $\Lambda_4 = W_4^{*T}\Gamma_{4}(z_4)\Gamma_{4}^T(z_4)W_4^*$.

The selection of $a_4$ as the infimum over $t\ge 0$ of the minimum eigenvalue of $A_4(t)$ and c4 as the supremum over $t\ge0$ of $C_4(t)$ allows Eq (3.94) to be reformulated as follows:

Based on the above derivation, we can achieve the objectives:

1) Within the closed-loop control framework, all error signals, designated as $z_i(t)$ for $i = 1, \cdots, 4$ and the weight estimation errors, expressed as $\tilde{W}_{ci}(t)$ and $\tilde{W}_{ai}(t)$ for $i = 1, \cdots, 4$, are assured to be SGUUB in anpredictable and desirable fashion;

2) The single-link manipulator joint angular position $q_1(t)$ exhibits the capability to follow the desired trajectory $y_r$ in a predictable and desirable manner.

Prove as follows:

1) The inequality (3.96) can be

where $a$ is the minimal of $a_k, k = 1, 2, \cdots, 4$ and $c$ is the sum of $c_k, k = 1, 2, \cdots, 4$. According to Lemma 2.1, we can clearly get the following result:

which can prove that that control objective 1 is valid.

2) Define $L_{z}(t) = (1/2)\sum_{k = 1}^{4}z_{k}^{2}(t)$. According to the Eqs (3.18), (3.56), (3.77), and (3.85), we have

Substituting (3.11), (3.52), (3.73), and (3.87) into (3.97), we have the following result:

Using Young's inequality, it is clear that we can get the following result:

where $D(t) = (1/2)f_2^2(\bar{x}_2)+(1/2)f_4^2(\bar{x}_4)+(1/2)\sum_{k = 1}^{3}\dot{\hat{\alpha}}_{k}^2+(1/2)\dot{\hat{y}}_{r}^2(t)$+$(1/2)(\hat{W}_{a1}^{T}(t)\Gamma_{1}(z_{1}))^{2}+(1/2)(\hat{W}_{a3}^{T}(t)\Gamma_{3}(z_{3}))^{2}$+$(g_2^2/2)(\hat{W}_{a2}^{T}(t)\Gamma_{2}(z_{2}))^{2}+(g_4^2/2)(\hat{W}_{a4}^{T}(t)\Gamma_{4}(z_{4}))^{2}$ is bounded. A constant $\rho$ exists, bounding $|D(t)|$. Hence, the above result can be described as

where $\beta$ is the minimal of $\{\beta_1-2, \beta_{2}g_{2}^2-g_{2}^2-1, \beta_3-2, \beta_{4}g_{4}^2-g_{4}^2-1\}$. Obviously, we can get the following result:

It implies that increasing $\beta$ sufficiently ensures desired tracking accuracy and control performance.

Ultimately, according to (3.11), (3.52), (3.73), and (3.87), we design an adaptive tracking control strategy for the flexible-joint manipulator. The details of this control method are illustrated in Figure 2.

4.   Simulation

To enhance the validation of the method's effectiveness in controlling a flexible-joint robotic manipulator, numerical simulations were conducted. Table 1 provides the key parameters relevant to the single-link manipulator. The initial conditions are set to $q_1(0) = 8\mathrm{deg}$, $\dot{q}_1(0) = 0\mathrm{deg/s}$, $q_2(0) = 10\mathrm{deg}$, and $\dot{q}_2(0) = 0\mathrm{deg/s}$, and we choose the desired trajectory as $y_r(t) = 28 \sin(3t/4)$, shown in Figure 3.

To achieve the tracking objectives, the design of the virtual controller for the first three steps and the input signal for the final step correspond to (3.11), (3.52), (3.73), and (3.87), respectively, where the designable parameters are set as $[\beta_{1}, \beta_{2}, \beta_{3}, \beta_{4}] = [6.00, 2.04, 11.00, 2.01]$. The NN at each step has $36$ neurons with centers uniformly distributed in the range $[-6, 6]$, and the widths $\varphi_i, i = 1, \cdots, 4$ of the Gaussian functions of the basis functions $\Gamma_{i}$ are all chosen to be $2$. The update rate of the critic weights at each step corresponds to (3.15), (3.54), (3.75), and (3.88), respectively, and the designable parameters learning rate and initial weights are $[\mu_{c1}, \mu_{c2}, \mu_{c3}, \mu_{c4}] = [0.4, 0.4, 0.4, 0.4]$, $W_ci(0) = [0.5]_{36\times 1}, i = 1, \cdots, 4$. The update rate of the actor weights at each step corresponds to (3.17), (3.55), (3.76), and (3.89), respectively, where the designable parameters learning rate and initial weights are $[\mu_{a1}, \mu_{a2}, \mu_{a3}, \mu_{a4}] = [300,300,300,300]$, $W_ai(0) = [0.4]_{36\times 1}, i = 1, \cdots, 4$.

Simulation result: The individual figures depict the results of the simulation process. The actual output $y(t)$ and the expected trajectory $y_r(t)$ are demonstrated in Figure 3, which it is clear to see that the actual output is able to better align with the desired output. Figure 4 shows the states $x_i, i = 1, \cdots, 4$. The weight's norm of critic NN $W_ci(t), i = 1, \cdots, 4$ is presented in Figure 5 and the weight's norm of actor NN $W_ai(t), i = 1, \cdots, 4$ is presented in Figure 6, which it is clear that all weights are bounded and converge to a certain value. The input $u(t)$ is illustrated in Figure 7, which observes that the input converges to the range of $[-5, 5]$. In addition, Figures 8 and 9 illustrate the tracking error $z_1(t)$ as $k$ varies within the range of $[100,200]$ and $i$ varies within the range of $[15, 30]$, demonstrating the robustness of this control method. In conclusion, it is observed that our control strategy enables the actual output $y(t)$ to track well on the expected trajectory $y_r(t)$ while optimizing the controller energy consumption. In order to better demonstrate the optimization of the energy consumption in this control scheme for a flexible robotic manipulator, we conduct a comparative experiment with the control scheme referenced in ^[19]. As illustrated in Figures 10 and 11, under conditions of similar tracking effectiveness, the control energy consumption of our scheme is significantly improved compared to that of the scheme in ^[19].

5.   Conclusions

In this paper, an optimal backstepping control scheme is proposed for trajectory tracking of a flexible manipulator by integrating optimal control into backstepping control. In this control scheme, each virtual controller, as well as the actual controller, is designed as an optimized solution in the corresponding inverse step. This approach achieves performance optimization for the entire flexible robotic manipulator system. RL is built on a critic-actor architecture, where the critic assesses performance then provides feedback to the actor. The actor then controls the system, and the two NNs collaborate to learn. Since the RL update law is derived from the negative gradient of a simple function, we simplify the design of the controller compared to existing optimal control methods for flexible robotic manipulators. Finally, the effectiveness of the control method for solving the trajectory tracking problem of flexible robotic manipulators is demonstrated through both theoretical analysis and simulation studies.

Author contributions

Huihui Zhong: Methodology, Validation, Writing-original draft; Weijian Wen: Formal analysis, Supervision; Jianjun Fan: Conceptualization, Investigation; Weijun Yang: Writing – Review and Editing, Visualization. All authors have read and approved the final version of the manuscript for publication.

Acknowledgments

This work was supported by the Special projects in key fields of colleges and universities in Guangdong Province, China (No.2024ZDZX1070, No.2024ZDZX3094), and the Guangdong University research and innovation team project (No.2024KCXTD075).

Conflict of interest

All authors declare no conflicts of interest in this paper.