Optimal control problems with time inconsistency

1.   Introduction

Bellman dynamic programming principle ^[1] and Pontryagin maximum principle ^[2] serve as two of the most important tools in solving optimal control problems with time consistency (i.e., classical optimal control problems). However, as times goes by, the cost functional and control systems of optimal control problems with time inconsistency is changing, which makes these two methods are not suitable for such problems. Thus, to explore optimal control problems with time inconsistency, we adopt a game theoretic approach. The notion of "equilibrium" are therefore considered instead of "optimization".

Actually, the research on time-inconsistent problems has a long history. The study of qualitative analysis on time-inconsistent behavior was carried out by Hume ^[3] in 1739 and by Smith ^[4] in 1759. These were then alluded to by Malthus in 1828, Pareto ^[5] in 1909, and Samuelson ^[6] in 1937. But until 1955, Strotz ^[7] in his milestone paper presented the mathematical formulation of the time-inconsistent problems. After that, the research on time-inconsistent problems is mainly divided into empirical research and theoretical research. The classical literature of empirical research includes the endowment theory of Thaler ^[8], the dynamic inconsistent theory of Kydland and Prescott ^[9] and the prospect theory of Kahneman and Tversky ^[10], who were Nobel Prize winners of economics in 2017, 2004 and 2002, respectively.

In the literature on optimal control problems with time inconsistency, a large number of mathematicians and behavioral finance scientists have carried out research on theoretical research and obtained rich research results. The authors first introduced the definition of feedback equilibrium control in ^[11] and characterized differentiable sub-game perfect equilibria in a continuous time inter-temporal decision optimization problem with non-constant discounting function. In Bj$\ddot{o}$rk and Murgoci ^[12], the authors generalized the extended HJB equation for a class of general controlled Markov process and a fairly general cost functional. They also proved that for every time-inconsistent optimal control problem, there is an associated time-consistent optimal control problem such that the optimal control for the time-consistent problem coincides with the equilibrium control for the time-inconsistent optimal control problem. In particular, we refer the reader to ^[13,14] and the references therein. In the series of works carried out by Yong and cooperative authors ^{[16,17,18,19]}, the authors performed an alternative method by partition of time interval and then regarded the time-inconsistent problem as the limit of the time-consistent problem, and achieved a number of research results. Motivated from optimal control problems with time inconsistency, Hamaguchi ^[20] researched the local solvability of a flow of forward-backward stochastic differential equations by using contraction mapping principle. We refer the reader to ^{[21,22,23,24]} for some relevant results.

Different from all the literature above listed, Hu et al. ^[25,26] researched the time-inconsistent stochastic linear quadratic (LQ, for short) control problems. They introduced the concept of equilibrium control by the local comparison made between open-loop controls. Using variational method, they derived the existence and uniqueness for equilibrium control, through a flow of forward-backward stochastic differential equations. In particular, they found an explicit equilibrium control and proved its uniqueness when the state is one dimensional and the coefficients in the problem are all deterministic. Some recent researched devoted to the open-loop equilibrium control can be found in ^[27,28,29] and the references therein.

The necessary and sufficient conditions are the essential characterization of mathematical problems including optimal control problems. The classical LQ optimal control problems are the equivalent relationship between control problem, two-point boundary value problem, and Riccati equation. In particular, we would like to mention the work of Peng et al. in a very recent paper ^[30], they studied the equivalent relationship between equilibrium control, two-point boundary problem, and Riccati equation for the time-inconsistent deterministic LQ control. Additionally, He and Jiang ^[31] formally acquired a necessary and sufficient condition by a method of extended HJB equations on the equilibrium strategies for time-inconsistent problems in continuous time.

Inspired by the above works, using the method of duality analysis, we study the necessary and sufficient conditions of optimal control problems with time inconsistency in the framework of open-loop equilibrium control in this paper. Furthermore, under the assumption of the solvability for the Riccati type equation, we investigate the existence of explicit equilibrium control by proving the solvability of two-point boundary value problem.

The remainder of this paper is organized as follows. In the second section, we formulate the mathematical model for a general class of optimal control problem with time inconsistency and introduce the definition of equilibrium control in the sense of open-loop. Section 3 is devoted to present the main results for a general class of optimal control problem with time inconsistency. In Section 4, we consider LQ optimal control problem with time inconsistency.

2.   Problem setting

Starting with an optimal control problem with time inconsistency. Let $T > 0$ be the end of a finite time horizon and $U\subset R^m$.

For any initial pair $(t, x)\in[0, T]\times R^{n}$, the following controlled system can be considered.

where $h:[0, T]\times R^n\times U\rightarrow R^n$ is a given map, $u(\cdot)$, a function valued in $U$, represents control, and $Y(\cdot)\in R^n$ represents the control trajectory. The control processes which are essentially bounded with respect to $t$, i.e.,

Subsequently, the following cost functional can be introduced.

for some given maps $\phi:[0, T]\times R^n \times [0, T]\times R^n\times U\rightarrow R$ and $\psi:[0, T]\times R^n\times R^n \rightarrow R$. Under some mild conditions, for any $(t, x)\in [0, T]\times R^n$ and $u(\cdot)\in \mathcal{U}[t, T]$, the state equation (2.1) admits a unique solution $Y(\cdot)\equiv Y^u_{t, x}(\cdot)$, and the cost functional $K\left(t, x;u(\cdot)\right)$ is well-defined. Thus, the following problem can be introduced.

$\textbf{Problem (TIP)}$. For $(t, x)\in[0, T]\times R^{n}$, find a control $\bar u(\cdot)\in\mathcal{U}[t, T]$ such that

Problem (TIP) is an optimal control problem with time inconsistency. With the time inconsistency, the notion "optimality" needs to be defined in an appropriate way. We adopt the concept of equilibrium solution within the framework of open-loop in this paper.

$\bf{Definition\; 2.1.}$ ^[25] Let $\bar u(\cdot)\in \mathcal{U}[0, T]$ be a given control and $\bar Y(\cdot)$ be the control trajectory corresponding to $\bar u(\cdot)$. The control $\bar u(\cdot)$ is called an equilibrium control if the following inequality holds

where

In this paper, $\mid \cdot \mid$ represents an Euclidean norm.

Next, we make the following assumptions.

$[H]$ $1)$ The map $h(s, y, u):[0, T]\times R^n\times U \rightarrow R^n$ be continuous, and $h(s, y, u)$ be also continuous differential with respect to $y$. There exists a constant $L > 0$ such that

and

for any $s\in [0, T]$, $y_1, y_2\in R^n$, and $u, u_1, u_2\in U$.

$2)$ The maps $\phi(t, x;s, y, u):[0, T]\times R^n\times [0, T]\times R^n\times U \rightarrow R$ and $\psi:[0, T]\times R^n\times R^n\rightarrow R$ are continuous, and $\phi$ and $\psi$ are also continuous differential with respect to $y$. There exists a constant $L > 0$ such that

and

for any $s\in [t, T]$, $y_1, y_2\in R^n$, and $u\in U$.

3.   Equilibrium control

In this section, we present a general necessary and sufficient condition of equilibrium control for the $\text{Problem (TIP)}$.

Let $\bar u(\cdot)$ is a given control and consider the perturbation control $u^{\varepsilon, t, v}(\cdot)$ defined by (2.5). $Y^\varepsilon(\cdot)$ and $\bar Y(\cdot)$ be corresponding control processes to the control system (2.1) with $u^{\varepsilon, t, v}(\cdot)$ and $\bar u(\cdot)$, respectively. We then introduce the following notations.

for any $0\leq \tau \leq s\leq T$. And

for any $0\leq \tau\leq T$.

$\bf{Proposition\; 3.1.}$ Let [H] hold. Then

1) $\Phi_\varepsilon(\cdot, \cdot)$ and $\Phi(\cdot, \cdot)$ are uniform bounded in $C\left([0, T]\times[0, T]\right)$.

2) $\Phi_\varepsilon(\cdot, \cdot)\longrightarrow\Phi(\cdot, \cdot)$ a.e. in $C\left([0, T]\times[0, T]\right)$ as $\varepsilon\rightarrow 0$.

Proof. Let $Y^\varepsilon(\cdot)$ and $\bar Y(\cdot)$ be the corresponding solutions to the control system (2.1) with $u^{\varepsilon, t, v}(\cdot)$ and $\bar u(\cdot)$, respectively, where $u^{\varepsilon, t, v}(\cdot)$ is defined by (2.5) and $\bar u(\cdot)$ is a given control. Define

Then,

By Gronwall's inequality, one has

Similarly, we obtain that

Moreover,

Because

Since $u(\cdot)\in L^\infty\left([t, T];U\right)$ for any $t\in[0, T]$, there is a constant $M > 0$ such that

Gronwall's inequality yields

Passing to the limit in above as $\varepsilon\rightarrow 0$ we obtain

Therefore, we have

Combining (3.3) and (3.4), we then have

This is implies that $\Phi_\varepsilon(\cdot, \cdot)$ is uniform bounded in $C\left([0, T]\times[0, T]\right)$. Similarly, we can easy prove $\Phi(\cdot, \cdot)$ is uniform bounded in $C\left([0, T]\times[0, T]\right)$. We thus completes the proof of (1).

We now claim that $\Phi_\varepsilon(\cdot, \cdot)\rightarrow \Phi(\cdot, \cdot)$ a.e. in $C\left([0, T]\times[0, T]\right)$ as $\varepsilon\rightarrow 0$. Since

which suggests that

It follows from (3.6) and uniform boundedness of $\Phi(\cdot, \cdot)$ that

This implies that (2) holds. We thus complete the proof.

$\bf{Proposition\; 3.2.}$ Let [H] hold. Then

Proof. By (3.2), we have

which implies that

It follows from Proposition 3.1, (3.3) and (3.6) that

This completes the proof.

$\bf{Theorem\; 3.3.}$ Let $[H]$ hold. Then Problem(TIP) admits an equilibrium control by Definition 2.1 if and only if the following quasi-variational problem

have a solution in $C\left([0, T];R^n\right)\times C\left([0, T];R^n\right)$, where

for any $(t, x;s, y, u, p)\in[0, T]\times R^n\times[0, T]\times R^n\times U\times R^n.$

Proof. Let $\bar u(\cdot)$ is a given control and $u^{\varepsilon, t, v}(\cdot)$ is defined by (2.5), $Y^\varepsilon(\cdot)$ and $\bar Y(\cdot)$ are the corresponding control processes to the control system (2.1) with $u^{\varepsilon, t, v}(\cdot)$ and $\bar u(\cdot)$, respectively. Then we have

Define

This implies that $Z^\varepsilon(0) = 0$ and

Now, we evaluate $\lim_{\varepsilon\searrow 0} \frac{K\left(t, \bar Y(t); u^{\varepsilon, t, v}(\cdot)\right)-K\left(t, \bar Y(t); \bar u(\cdot)\right)}{\varepsilon}$. It follows from (2.2) and (2.5) that

Plugging (3.10) into the above, we obtain that

Combining (3.8), (3.6) and Proposition 3.2, we then have

On the one hand, if $\bar u(\cdot)$ is an equilibrium control, then

This implies that

This completes the proof of necessary.

Conversely, assume $\left(\bar Y(\cdot), \omega (\cdot)\right)\in C\left([0, T];R^n\right)\times C\left([0, T];R^n\right)$ be the solution of (3.7). Let $\bar u(\cdot)$ given by (3.7), we claim $\bar u(\cdot)$ is an equilibrium control. We consider the perturbation control $u^{\varepsilon, t, v}(\cdot)$ given by (2.5). Similarly to the calculation of (3.11), we can obtain that

By (3.12), we have

Therefore,

We thus complete the proof.

4.   LQ optimal control problem with time inconsistency

As application, we study an LQ optimal control problem with time inconsistency in this section.

We use the following notations is this section.

Now, we make the following standard assumptions:

$(Q1)$ $A\in L^1\left([0, T];R^{n\times n}\right)$, $B\in L^2\left([0, T];R^{n\times m}\right)$.

$(Q2)$ $M\in C\left(D[0, T];\mathcal{S}^m\right)$ is symmetry and positive definite.

$(Q3)$ $L\in C\left(D[0, T]; \mathcal{S}^n\right)$, $N\in C\left([0, T]; \mathcal{S}^n\right)$ are symmetry and positive semi-definite.

$(Q4)$ For $0\leq t \leq s \leq T$, $\dot N(t)$ and $L_t(t, s)$ are symmetry and positive semi-definite. Here

We consider the following LQ optimal control system from the situation $(t, x)\in [0, T]\times R^n$.

with the following cost functional

One could introduce the following control problem.

$\bf{{Problem}\; {(TILQ)}.}$ For $(t, x)\in [0, T]\times R^n$. Find a control $\bar u(\cdot)\in \mathcal{U}[0, T]$ such that

Problem (TILQ) is an LQ optimal control problem with time inconsistency.

$\bf{Theorem\; 4.1.}$ Let $(Q1)$–$(Q4)$ hold. Then Problem (TILQ) admits an equilibrium control by Definition 2.1 if and only if the following two-point boundary value problem

have a solution in $C\left([0, T];R^n\right)\times C\left([0, T];R^n\right)$ and equilibrium control $\bar u(\cdot)$ is given by

Proof. Let $\bar Y(\cdot)$ and $Y^\varepsilon(\cdot)$ be the corresponding state trajectory to the control system (4.1) with $\bar u(\cdot)$ and $u^{\varepsilon, t, v}(\cdot)$, respectively, where $\bar u(\cdot)$ is a given control and $u^{\varepsilon, t, v}(\cdot)$ is defined by (2.5). It is easy to prove that

Let

Then $Z^\varepsilon(0) = 0$ and

Now, we evaluate the variation of the cost functional. It follows from (4.2) and (2.5) that

Plugging (4.8) into the above, we can have

It follows from (4.6) and (4.4) that

This implies that

for any $(t, v)\in [0, T]\times R^m.$ We thus prove that $\bar u(t) = -M^{-1}(t, t)B^\top(t)\varpi(t)$ is an equilibrium control by using the positive definiteness of the matrix-value function M.

Conversely, suppose that Problem (TILQ) have an equilibrium control $\bar u(\cdot)$ and $u^{\varepsilon, t, v}(\cdot)$ is defined by (2.5). Let $\bar Y(\cdot)$ and $Y^\varepsilon(\cdot)$ be the corresponding state trajectory to the control system (4.1) with $\bar u(\cdot)$ and $u^{\varepsilon, t, v}(\cdot)$, respectively. Define

Similarly to the computation of the (4.9), we have

Define

It follows from (4.12) that $\tilde {\mathbb{K}}\left(t, \bar Y(t); v\right)$ is strictly convex in $v$. Therefore, the definition of an equilibrium control yields that $\tilde {\mathbb{K}}\left(t, \bar Y(t); v\right)\geq0$, which, together with (4.12), we obtain that $\tilde {\mathbb{K}}\left(t, \bar Y(t); v\right)$ admits a unique minimum point $\tilde v$ given by

Then the uniqueness of the minimum point $\tilde v$ yields that

Combining (4.13) and (4.1), we then have

Thus, the differential equation (4.14) admits a unique solution $\bar Y(\cdot)$ given by

Plugging (4.15) into (4.11), we have

Combining (4.15) and (4.16), we completes the proof.

We now consider the solvability of the two-point boundary value problem (4.4). We introduce the following Riccati type equation:

where

Here

for any $0\leq t \leq \tau \leq T$. Observe that $\hat\Phi(\cdot, \cdot)$ depends on the unknown term $\Gamma(\cdot)$. Moreover, the Riccati type equation (4.17) is different from that of ^[30]. Therefore the results in ^[30] cannot be applied directly to (4.17) for its well posedness.

It is also worth pointing out that (4.17) does not have a symmetric structure. Thus $\Gamma(\cdot)$ is not expected to be symmetric.

$\bf{Theorem \; 4.2.}$ Let Assumptions $(Q1)-(Q4)$ hold. Suppose that the Riccati type equation (4.17) admits a unique solution $\Gamma(\cdot)\in C\left([0, T];R^{n\times n}\right)$. Then the two-point boundary value problem (4.4) admits a solution $\left(\bar Y(\cdot), \varpi(\cdot)\right)\in C\left([0, T];R^n\right)\times C\left([0, T];R^n\right)$. Furthermore, for any $t\in[0, T]$, we have

where $\hat \Phi (t, 0)$ is as introduced in (4.18).

Proof. Suppose that $\Gamma(\cdot)\in C\left([0, T];R^{n\times n}\right)$ is a solution of (4.17) and define $\left(\bar Y(\cdot), \varpi(\cdot), \bar u(\cdot)\right)$ as in (4.19)–(4.21). We are going to show that $\bar Y(\cdot)$ and $\varpi(\cdot)$ satisfy the two-point boundary value problem (4.4). First, observe that $\bar Y(\cdot)$ satisfies the following ODE,

On the other hand, differentiating both sides of (4.20) with respect to $t$, we obtain that

which implies that

Observe that

Invoking this into (4.23), we obtain that

It follows from (4.22) and (4.24) that the two-point boundary value problem (4.4) admits a solution in $C\left([0, T];R^n\right)\times C\left([0, T];R^n\right)$.This completes the proof.

Combining Theorem 4.1 with Theorem 4.2, we obtain the following result.

$\bf{Corollary\; 4.3.}$ Let Assumptions $(Q1)$–$(Q4)$ hold. Suppose that the Riccati type equation (4.17) admits a unique solution $\Gamma(\cdot)\in C\left([0, T];R^{n\times n}\right)$. Then Problem (TILQ) has an equilibrium control that can be represented by the state feedback form

Acknowledgments

The author gratefully acknowledges to the anonymous reviewers and editor for their careful review and valuable comments and supported by National Natural Science Foundation of China grant No. 12061021.

Conflict of interest

The author declares there is no conflict of interest.