An optimal control problem without control costs

1.   Introduction

We consider a two-dimensional controlled diffusion process $(X_1(t), X_2(t))$ defined by the following system of stochastic differential equations:

where $f_i(\cdot)$ is a real function, $b_i(\cdot) \neq 0$, $u(t)$ is the control variable, $v_i(\cdot) > 0$ and $\{B_i(t), t \ge 0\}$ is a standard Brownian motion, for $i = 1, 2$. The two Brownian motions are assumed to be independent. The functions $f_i(\cdot)$ and $v_i(\cdot)$ are respectively the infinitesimal mean and variance of the uncontrolled process, for $i = 1, 2$. The functions $b_1(\cdot)$ and $b_2(\cdot)$ are control coefficients or parameters.

Let

where $D$ is a subset of $\mathbb{R}^2$. The random variable $T$ is called a first-passage time in probability theory. The aim is to minimize the expected value of the cost function

where $q(\cdot, \cdot) \ge 0$, $\lambda$ is a real constant and $K(\cdot, \cdot)$ is a general termination cost function. This type of stochastic optimal control problem is known as a homing problem; see Whittle [1, p. 289] or Whittle [2, p. 222]. Notice however that there are no control costs. Therefore, the above problem is actually an extension of the classic homing problem. Moreover, we see in Eqs (1.1) and (1.2) that the control variable does not have the same effect on each component of the two-dimensional diffusion process $(X_1(t), X_2(t))$. Notice also that the problem is time-invariant, because the functions $f_i(\cdot), b_i(\cdot)$ and $v_i(\cdot)$, for $i = 1, 2$, as well as $q(\cdot, \cdot)$ and $K(\cdot, \cdot)$ do not depend explicitly on $t$.

Recent papers on homing problems include the following ones: Kounta and Dawson ^[3], Makasu ^[4] and Lefebvre ^[5]. The original homing problem has been extended in various ways: Lefebvre and Kounta ^[6] replaced the diffusion processes by discrete-time Markov chains, Lefebvre and Moutassim ^[7] considered the problem for jump-diffusion processes, and Lefebvre ^[8] treated the case of controlled autoregressive processes.

There are some papers on optimization problems for which the final time is random. However, this final time is not a first-passage time, as in homing problems. Such problems were considered, in particular, in Yan and Koo ^[9], Rodosthenous and Zhang ^[10], Yun and Choi ^[11], Khatab {et al.} ^[12] and Yu ^[13].

Homing problems are sometimes expressed as dynamical games; see Lefebvre ^[14]. It is possible to find papers on differential games with a random time horizon; see, for instance, Marín-Solano and Shevkoplyas ^[15] and Zaremba {et al.} ^[16]. However, in these papers, the final time is again not a first-passage time.

Next, we define the value function by

That is, $F(x_1, x_2)$ is the expected cost obtained by using the optimal control in the interval $[0, T]$. In Section 2, we will make use of dynamic programming to find the differential equation it satisfies. This differential equation is a non-linear second-order partial differential equation (PDE). We will see that the optimal control $u^*$ can be expressed in terms of the value function as follows:

In Section 3, we will find explicit solutions to the non-linear PDE satisfied by the value function, subject to the appropriate boundary conditions, in important particular cases. The method of similarity solutions will be used. Finally, some final remarks will be made in Section 4.

2.   Dynamic programming

Bellman's principle of optimality states that "an optimal policy has the property that, whatever the initial state and the initial decision, it must constitute an optimal policy with regards to the state resulting from the first decision". Hence, any remaining part of an optimal policy is also optimal. Therefore, we can write that

We have

Moreover, a standard Brownian motion $\{B(t), t \ge 0\}$ is such that

It follows, assuming that $F(x_1, x_2)$ is twice differentiable with respect to $x_1$ and $x_2$ and making use of Taylor's formula, that

Finally, dividing each side of the previous equation by $\Delta t$ and letting $\Delta t$ decrease to zero, we obtain the following dynamic programming equation:

Differentiating Eq (2.5) with respect to $u(0)$, we find, as mentioned in the Introduction section, that the optimal control is

Then, substituting the above expression into Eq (2.5), we can state the following proposition.

Proposition 2.1. The value function $F(x_1, x_2)$ satisfies the second-order, non-linear PDE

subject to the boundary condition

In the next section, explicit solutions to (2.7), (2.8) will be obtained in important particular cases. The method of similarity solutions will be used.

3.   Explicit solutions

{Case I}. The first particular case that we consider is the one for which $f_i(\cdot) \equiv 0$, $b_i(\cdot) \equiv 1$, $v_i(\cdot) \equiv 1$, for $i = 1, 2$, $q(\cdot, \cdot) \equiv 0$, $\lambda = 1$, $K(\cdot, \cdot) \equiv 0$ and we choose the first-passage time

where $x_i = X_i(0)$ for $i = 1, 2$. The diffusion process $(X_1(t), X_2(t))$ is then defined by the stochastic differential equations

Thus, $(X_1(t), X_2(t))$ is a controlled two-dimensional standard Brownian motion. This case is arguably the simplest non-degenerate two-dimensional problem that can be examined. Equation (2.7) reduces to

subject to the boundary conditions

To solve (3.4), (3.5), we will make use of the method of similarity solutions. We look for a solution of the form

where $w : = x_1-x_2$ is the similarity variable. For the method to work, we must be able to express both the Eq (3.4) and the boundary conditions (3.5) in terms of $w$. We find that Eq (3.4) is transformed into the second-order linear ordinary differential equation

while the boundary conditions become

The general solution of Eq (3.7) can be expressed as follows:

The particular solution that satisfies the boundary conditions (3.8) is

for $k_1 \le w \le k_2$. Let us choose $k_1 = 0$ and $k_2 = 1$. Then, the above solution reduces to

It follows that the value function $F(x_1, x_2)$ is given by

for $(x_1, x_2) \in \mathbb{R}^2$ such that $0 \le x_1-x_2 \le 1$.

Next, we deduce from Eq (2.6) and the fact that $F_{x_1} = H'(w) = -F_{x_2}$ that the optimal control in this particular problem is actually a constant:

Hence, the optimally controlled diffusion process satisfies

That is, $\{X_1^*(t), t\ge 0\}$ (respectively $\{X_2^*(t), t\ge 0\}$) is a Wiener process with drift parameter $1/4$ (resp. $1/2$) and variance parameter $1$. Since the two processes are independent, we can state that the one-dimensional process $\{X^*(t), t\ge 0\}$ defined by

is a Wiener process with drift parameter $\mu = -1/4$ and variance parameter $\sigma^2 = 2$.

Remarks. (i) With the choices $q(\cdot, \cdot) \equiv 0$, $\lambda = 1$ and $K(\cdot, \cdot) \equiv 0$ that we made above, the cost function $J(x_1, x_2)$ defined in Eq (1.4) reduces to $T_1(x_1, x_2)$. Therefore, the aim is to make the two-dimensional controlled process leave the continuation region as soon as possible. Even though there are no control costs, we saw that the optimal solution consists in choosing a (finite) constant control.

(ii) Let $T_1^*(x_1, x_2)$ be the first-passage time when we use the optimal control. We may write that $F(x_1, x_2) = E[T_1^*(x_1, x_2)]$. The function $m(w) : = E[T_1^*(w = x_1-x_2)]$ satisfies the second-order linear ordinary differential equation

subject to the boundary conditions $m(0) = m(1) = 0$. We then deduce from Eqs (3.7) and (3.8) (with $k_1 = 0$ and $k_2 = 1$) that the functions $H(w)$ and $m(w)$ are the same.

Case II. Assume now that $f_i(\cdot) \equiv 0$, $b_i[X_i(t)] = X_i(t)$, $v_i[X_i(t)] = X_i^2(t)$, for $i = 1, 2$, $q(\cdot, \cdot) \equiv 0$, $\lambda = 1$ and $K(\cdot, \cdot) \equiv 0$. Moreover, we define

where $k_1 > 0$. The controlled diffusion process $(X_1(t), X_2(t))$ is such that

This time, $(X_1(t), X_2(t))$ is a controlled two-dimensional geometric Brownian motion. A geometric Brownian motion $\{Y(t), t\ge 0\}$ can be expressed as the exponential of a Wiener process. Therefore, if we assume that $Y(0) > 0$, then we can state that $Y(t) > 0$ for any $t \ge 0$.

Equation (2.7) takes the form

and is subject to the boundary conditions

Based on the boundary conditions, we now look for a solution of the form $F(x_1, x_2) = H(w = x_1^2/x_2^2)$. We have

and

Substituting these expressions into Eq (3.21), we find that it becomes

The boundary conditions are simply $H(k_1) = H(k_2) = 0$, as in Case I.

The general solution of Eq (3.26) is

With $k_1 = 1$ and $k_2 = 2$, we find that

Finally, from the expressions in Eq (3.23), we calculate

Thus, the optimal control is again a constant. It follows that

The optimally controlled process $\{X_i^*(t), t\ge 0\}$ is also a geometric Brownian motion, for $i = 1, 2$. We can write that $X_1^*(t) = {{\rm{e}}}^{Z_1(t)}$, where $\{Z_1(t), t\ge 0\}$ is a Wiener process with drift parameter $-1/4$ and variance parameter $1$. Similarly, $X_2^*(t) = {{\rm{e}}}^{Z_2(t)}$, where $\{Z_2(t), t\ge 0\}$ is a Wiener process with drift parameter $0$ and variance parameter $1$. Hence, by independence,

where $\{Z(t), t\ge 0\}$ is a Wiener process with drift parameter $-1/2$ and variance parameter $8$. The infinitesimal parameters of $\{W(t), t \ge 0\}$ are given by $7 \quad w/2$ and $8 \quad w^2$. Therefore, we may write that the function $m(w) : = E[T_2^*(w = x_1^2/x_2^2)]$ satisfies the second-order linear ordinary differential equation

subject to $m(1) = m(2) = 0$, from which we may conclude that the functions $m(w)$ and $H(w)$ coincide, as required.

Case III. To conclude this section, we will present a case when the optimal control is not a constant. Assume, in Case II, that $b_1[X_1(t)] = X_1^2(t)$, $b_2[X_2(t)] = X_2^{3/2}(t)$, $\lambda = 0$ and $K(X_1(T_2), X_2(T_2)) = X_1^2(T_2)/X_2^2(T_2)$. Hence, there is only a termination cost. The aim is now to make the controlled process $(X_1(t), X_2(t))$ leave the continuation region through a given part of its boundary. Indeed, the optimizer must try to make $X_1^2(t)/X_2^2(t)$ take on the value $k_1$ before $k_2$ ($> k_1)$.

We find that Eq (3.26) becomes

subject to $H(k_i) = k_i$, for $i = 1, 2$. The general solution of the above equation is

where ${\rm Ei}_1$ is an exponential integral function defined by

The particular solution that satisfies the boundary conditions $H(1) = 1$ and $H(2) = 2$ is

We can now calculate the optimal control. We find that

We notice that not only the optimal control is not a constant, it is not a function of $w: = x_1^2/x_2^2$ either. The optimally controlled process $(X_1^*(t), X_2^*(t))$ satisfies the following stochastic differential equations:

Remark. Another case for which the optimal control is not a constant is the one when we replace $b_1[X_1(t)]$ by $1$ and $b_2[X_2(t)]$ by $\sqrt{X_2(t)}$ in Case III. This time, the value function is

for $x_1 > 0$ and $x_2 > 0$ such that $1 \le x^2_1/x_2^2 \le 2$. Finally, the optimal control is given by

4.   Conclusions

In this paper, a stochastic optimal control problem for a two-dimensional diffusion process $(X_1(t), X_2(t))$ has been considered. This problem is an extension of the so-called homing problems, in which the final time, rather than being either a fixed constant or infinity, is a random variable. The optimizer stops controlling the processes the first time a certain event occurs. Here, the cost function was modified: there were no control costs. However, the control variable $u(t)$ was assumed to influence each part of the controlled process differently; namely, the state dynamics are quadratic in $u(t)$ for $X_1(t)$, while they are linear in the case of $X_2(t)$.

In Section 2, we gave the PDE satisfied by the value function in the general case. Then, in Section 3, we presented various particular cases for which we were able to obtain explicit and exact solutions to the problems considered. The method of similarity solutions was used to solve the appropriate equations. Although there are no control costs, the optimal control was never either identical to zero or infinite.

When the method of similarity solutions fails, we could of course at least try to obtain numerical solutions to any particular problem. However, the aim of this paper was to present exact analytical solutions to important problems.

Acknowledgments

This research was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC). The author also wishes to thank the anonymous reviewers of this paper for their constructive comments.

Conflict of interest

The author reports that there are no competing interests to declare.