$(s, S)$ Inventory policies for stochastic controlled system of Lindley-type with lost-sales

1.   Introduction

This paper considers a discrete-time, single-item, infinite-stock inventory system in which excess demand is not backlogged. The dynamics of the system is determined by a random walk with a lower barrier; in particular, the state zero is considered as a barrier. Also, it is assumed that the control variable is affected by a random noise in the dynamic of the system; see (2.1). Furthermore, the following costs are considered: ordering/production, holding, and shortage. Then, the objective function is the infinite-horizon expected total discounted cost. The goal of the paper is to guarantee the existence of stationary policies and characterize the optimal stationary policies as $(s, S)$ policies (see ^[5,18,23,28]). The $(s, S)$-policies operate in the following manner: If the inventory level falls below a minimum $s\geq 0$, the controller will request a replenishment demand to restore the inventory stock to a maximum number $S\geq s$.

As mentioned earlier, a crucial component of the inventory model is the dynamics of the system; in this paper is considered a controlled version of a Lindley random walk. This random walk has been used to model waiting times in a queuing system with a server (see ^[20]). A controlled version of this dynamic was introduced in ^[14] to illustrate the convergence of the value iteration algorithm for Markov control processes with average costs. Furthermore, this dynamic has been used in several contexts of Markov control processes; see, e.g., ^[16,27]. Despite this, in these works, the optimal policy is not characterized, except by ^[11], where it is considered as a compact action set and bounded cost. The characterization of $(s, S)$ policies is an interesting problem in inventory theory, this problem can be traced back to the papers of ^[13,22,28]. One advantage of guaranteeing the existence of $(s, S)$ policies is their ease of implementation. However, it is not always possible to guarantee the existence of a $(s, S)$ inventory policy; see, for instance, ^[2]. Therefore, it is necessary to establish conditions in the stochastic control model that guarantee the existence of $(s, S)$ inventory policies. In addition, there is recent work in applied areas where strategies (s, S) are implemented in the real world, such as in healthcare ^[1,29] and in machine learning ^[8], to name a few. A survey work presenting applications of inventory theory in the real world can be found in ^[22]. On the other hand, recent research from the theoretical point of view of the model is presented in ^[6], where the optimality of strategies (s, S) is proved as a particular case of their setup concave linear piecewise ordering costs. The perishable inventory with backlogging demand was studied in ^[30]. In contrast to the work described in this manuscript, there is an extensive literature on continuous-time inventories, e.g., the recent work of ^[4], which studies perishable inventories.

Based on the review conducted, the main objectives of this work can be described as follows:

● To provide conditions that guarantee the optimality of inventory strategies for a system with lost sales (see Theorem 4.10).

● To propose approximation procedures for the strategies described in the previous point (see Theorem 5.3).

It is important to note that the goal of the first point is a more complex challenge, as opposed to a backlog inventory system as described in ^[22].

The methodology of the paper is as follows. First, the dynamic programming approach is validated, and the existence of optimal stationary policies is verified. Second, results of convex analysis are applied to guarantee that the minimizers of the value iteration functions and the optimal policies of the inventory system are $(s, S)$-policies. This characterization is achieved under assumptions of continuity and monotonicity in the components of the inventory control model; these conditions guarantee that the cost function is convex. The convexity property has been used previously in the inventory systems literature, see for example ^{[9,10,13,18,23,25,26,28,31]}.

The paper is organized as follows. Section 2 introduces the inventory control model and the assumptions about the components of the inventory control model. Section 3 introduces the dynamic programming approach. Section 4 presents the characterization of $(s, S)$ policies. Section 5 contains an analysis of the convergence of minimizers of value iteration. Finally, a numerical example is given in Section 6.

2.   An inventory control model with a Lindley dynamic system

Consider a discrete-time inventory control system. If $X_{t}$ denotes the inventory at time $t = 0, 1, \ldots$, the evolution of the system is modeled by the following Lindley-type dynamical system:

with $X_{0} = x \in X: = [0, \infty)$ known, where

i) $\left\lbrace \eta_{t} \right\rbrace$ is a sequence of independent and identically distributed ($i.i.d.$) Bernoulli random variables with parameter $p$, $0 < p < 1$, in this case the event $\left\lbrace \eta_{t} = 1 \right\rbrace$ means that an order placed at instant $t$ has been supplied.

ii) $a_{t}$ denotes the control or decision applied at time $t$ and represents the quantity ordered by the inventory manager (or decision maker).

iii) The sequence $\left\lbrace D_{t}\right\rbrace$ is conformed by $i.i.d.$ non-negative random variables with a common distribution $F$. $D_{t}$ denotes the demand within period $t$. It is assumed that sequence $\{D_t\}$ is independent of the sequence $\{\eta_t\}$.

Note that the difference equation given in (2.1) induces a stochastic kernel that can be expressed on $X$ by

is defined as follows

with $y, x, a\in [0, \infty)$ and

if $x, a\in [0, \infty)$ and $y < 0$.

Suppose further that it is associated with a one-step cost function $C:\mathbb{K}\longrightarrow [0, \infty)$, defined as follows:

where $K \geq 0$ is a fixed order price, $c > 0$ is the order price per unit, $h:[0, \infty) \longrightarrow [0, \infty)$ denotes the holding cost per period, $l:[0, \infty) \longrightarrow [0, \infty)$ indicates the shortage cost for unfilled demand and $\mathbb{E}$ denotes the expectation with respect to the joint distribution of the random vector $(\eta, D)$, where $(\eta, D)$ is a generic element of the sequence $\{(\eta _{t}, D_t)\}$. In all the following sections, the cost function is determined by the sum of two cost functions:

where $g(a) = KI_{\left\lbrace a: a > 0 \right\rbrace }(a)+ca$ and

with $(x, a) \in \mathbb{K}$.

Then, the inventory control model is identified with a Markov control process ^[17], in short, MCP. The components of the associate Markov control model are as follows: $X: = [0, \infty)$ is the state space, $A: = [0, \infty)$ is the action space, the dynamic and the cost function are given by (2.2) and (2.3), respectively. Consequently, the inventory system evolves as follows: If the stock of the inventory systems occupies state $X_{t} = x$ at time $t$ and a controller (inventory manager) requests the quantity of product $a_{t} = a$. Then, a cost $C(x, a)$ is incurred and the system jumps into a state $X_{t+1}$ according to the transition law $Q(\cdot|x, a)$. Once the transition into the new state has occurred, a new order (control) is requested and the process is repeated. Thus, for each $t\geq 1$ an admissible history $h_{t}$ of the inventory system up to the transition $t$ given by, $h_{t} = (X_{0}, a_{0}, \ldots, X_{t-1}, a_{t-1}, X_{t})$. Let $\mathbb{H}_{t}$, $t\geq 1$, be the set of all admissible histories of the system up to the transition $t$. The following definition will be used to characterize optimal strategies in the inventory control model.

Definition: A policy $\pi = \left\lbrace \pi _{t} \right\rbrace$ is a sequence of stochastic kernels $\pi _{t}$ on $A$ given $\mathbb{H}_{t}$, satisfying the constraint: $\pi_{t}(A|h_{t}) = 1$, for each $h_{t} \in\mathbb{H}_{t}$, $t\geq 1$. The collection of all policies is denoted by $\Pi$. Define $\mathbb{F}$ as the set of all measurable functions $f:X \longrightarrow A$. Thus, a Markov policy is a sequence $f_{t}$ such that $f_{t} \in \mathbb{F}$, for $t\geq 1$. In particular, a Markov policy $\pi = \left\lbrace f_{t} \right\rbrace$ is said to be stationary if $f_{t}$ is independent of $t$, i.e., $f_t = f \in \mathbb{F}$, for all $t\geq 1$, in this case, $f_t$ is denoted by $f$ and refers to $\mathbb{F}$ as the set of stationary policies.

In the subsequent, for each $x\in X$ and $\pi\in \Pi$ will be denoted by $\mathbb{P}_{x}^\pi$ the measure defined on the measurable space $\Omega: = ((X \times A)^\infty, \mathcal{F})$, where $\mathcal{F}$ is the corresponding product $\sigma$-algebra. The measure $\mathbb{P}_{x}^\pi$ is induced by the theorem of Ionescu Tulcea ^[3]. The expectation operator with respect to $\mathbb{P}_{x}^\pi$ is denoted by $\mathbb{E}_{x}^\pi$.

Inventory control problem

The goal of this subsection is to introduce the inventory control problem, then consider $\pi \in \Pi$, $x\in X$ and define the following objective function:

where $\alpha \in (0, 1)$ denotes a discount factor. The performance criterion defined in (2.5) is called expected total discounted cost (see ^[17]). Hence, the optimal inventory control problem consists in determining a policy $\pi^* \in \Pi$ such that,

for each $x\in X$, in this case $\pi ^*$ is denominated optimal policy. The function $V$ defined by

for each $x\in X$, it will be called the optimal value function.

In subsequent sections, the following assumption will be considered.

Basic Assumption (BA):

i) $h$ is a non-decreasing convex function such that $h(0_{+}) = h(0)$, where $h(0_{+}): = \inf \{h(y):y > 0\}$.

ii) $l$ is a convex function such that $l(0) = 0$ and $l'(u)$ exists and is non-negative for each $u\geq 0$.

iii) $\mathbb{E}[h((u-D)^+)] < \infty$ and $\mathbb{E}[l((D-u)^+)] < \infty$, for each $u\geq 0$ and $D$ is a generic element of the sequence $\{D_t\}$.

iv) If $D$ is a generic element of the sequence $\{D_t\}$, it is assumed that $D$ has a continuous density denoted by $\Delta$.

BA will not be mentioned in each lemma or theorem throughout the paper, and we will assume that it holds.

Remark 2.1. i) Observe that the cost function (2.4) is convex, due to the convexity and monotonicity of functions $h$ and $l$ (see BA i) and ii)). Therefore, the cost function (2.3) is a convex function, this property will be used in the proof of Lemma 4.3.

ii) Moreover, under BA i) and ii), $h$ and $l$ are continuous functions. This characterization will help us apply to demonstrate the validity of Lemma 3.1.

3.   Dynamic programming approach

Assuming BA, the inventory control model in the MCP literature is identified as a semi-continuous model ^[17]. Moreover, the following lemmas guarantee the existence of optimal stationary policies and the dynamic programming approach; see Theorem 3.7 below.

Lemma 3.1. The cost function $C$, is a lower semi-continuous ($l.s.c.$) function, i.e., the set $\left\lbrace (x, a)\in \mathbb{K} : C(x, a)\leq \lambda \right\rbrace$ is closed for all $\lambda \in \mathbb{R}$.

Proof. First, we can see directly that the cost $g$ is a continuous function for $a > 0$ and $l.s.c.$ at $a = 0$. We now want to prove that the function $H$ is continuous for any $(x, a)\in \mathbb{K}$. For this purpose, we fix $x\in X$ and $a\in A$. Take $x_{n}\in X$, $a_{n}\in A$, $n\geq 1$, such that $x_{n}\longrightarrow x$, $a_{n}\longrightarrow a$, when $n$ goes to infinity. Consequently, Eq (2.4) gives $H(x_{n}, a_{n})\longrightarrow H(x, a)$ as $n\longrightarrow \infty$, by applying the dominated convergence theorem ^[3] and continuity of the functions $h$ and $l$; the latter by Remark 2.1. Therefore, the cost function $C$ is a $l.s.c.$. □

Corollary 3.2. The cost $C$ is an inf-compact function, namely the set $\left\lbrace a\in A: C(x, a)\leq \lambda \right\rbrace$ is compact for each $x\in X$ and $\lambda \in \mathbb{R}$.

Proof. Note that $\{a\in A: ca\leq \lambda\} = [0, \lambda/c]$, for any $\lambda \in \mathbb{R}$. On the other hand, by Lemma 3.1, for all $\lambda \in \mathbb{R}$ and $x \in X$ the set defined as $M: = \left\lbrace a\in A: C(x, a) \leq \lambda \right\rbrace$ is closed set. Consequently, since $M$ is contained in the compact set $\left\lbrace a\in A: ca \leq \lambda \right\rbrace$, it is concluded that the cost function $C$ is inf-compact. □

Lemma 3.3. Transition law $Q$ (2.2) induced by the difference equation (2.1) is strongly continuous, i.e., $w(x, a): = \int_{X} u(y)Q(dy|x, a)$ is continuous and bounded on $\mathbb{K}$ for every measurable bounded function $u$ on $X$.

Proof. Let $u$ be a measurable and bounded function defined on $X$ and let $\left\lbrace (x_{n}, a_{n})\right\rbrace$ be a sequence on $\mathbb{K}$ convergent to $(x, a)\in \mathbb{K}$. Then, from (2.2), we have that

Observe by applying an adequate change of variable, the following identity holds:

On the other hand, it is easy to prove that $\{I_{(-\infty, x_n+a_n]}\}$ converges to $\{I_{(-\infty, x+a]}\}$ almost everywhere (a.e.) with respect to the Lebesgue measure $m$ on $\mathbb{R}$. Moreover, let $\theta \in (0, \infty)$ such that $\left| u(x)\right|\leq \theta$, for all $x\in X$ and consider the following functions:

with $s\in [0, \infty)$. Then, notice that the following statements hold:

i) $\{r_n\}$ converges to $r$ a.e. with respect to $m$.

ii) $\{g_n\}$ converges to $g$ a.e. with respect to $m$.

iii) For each $n = 1, 2, ...$, $\left| r_n(\cdot)\right|\leq g_n(\cdot)$.

iv) $\int g_n(s)ds = \int g(s) = \theta$.

Statements i)–iv) guarantee the hypothesis of the Dominated Convergence Theorem (see^[3]), in consequence, it is obtained that $\lim_{n \to +\infty}\int r_n(s)ds = \int r(s)ds$, i.e.,

Similarly, it is possible to prove that:

Besides, due to the continuity of the distribution function $F$, it yields that

In consequence, from (3.1)–(3.3), it is obtained that

Therefore, $w$ is a continuous function on $\mathbb{K}$. □

The demonstration of Lemma 3.4 is based on a result of renewal processes used in classical theory of inventories (see, e.g., ^[4,12]). For this purpose, define for each $x\in X$, the renewal process

where $W_0: = 0$ and $W_t: = \sum_{j = 1}^{t}D_j$. Observe that $\mathbb{E}[N(x)] < \infty$ for each $x\geq0$, a proof of this fact can be consulted in ^[15]. Furthermore, consider the residual lifetime defined as

Lemma 3.4. For all $x\in X$, $\mathbb{E}[l(R(x))] < \infty$.

Proof. Let $x\in X$ be fixed. Observe that the tail of the distribution of the residual lifetime is given for $z\geq 0$ as follows

where $U: = \mathbb{E}[N(x)]$ is the renewal function. On the other hand, from assumption BA i), it is obtained that

where $l(\infty): = lim_{u\rightarrow \infty}l(u)$. In consequence, by substituting the tail distribution of function $R$ in the last equation, it yields that

Now, observe that

and from Fubini's theorem ^[3], it is obtained that

Therefore,

Since state $x$ is arbitrary, the result follows. □

Lemma 3.5. There exists $\tilde{\pi}\in \Pi$ such that $v(\tilde{\pi}, x) < \infty$, for all $x\in X$.

Proof. Consider $x\in X$ fixed and the stationary policy $\tilde{\pi} = \{\tilde{g}, \tilde{g}, \ldots\}\subseteq \mathbb{F}$ with $\tilde{g}(y) = 0$ for all $y\in X$. Hence, the stochastic path $\{X_t^{\tilde{g}}\}$ generated by $\tilde{\pi}$ is given by

with $X^{\tilde{g}}_0 = x$. Consequently, observe that

Then, due to (2.3) and (2.4), it yields that

Now, from (3.6) and the renewal process (3.4), it may be found that

On the other hand, observe that

where the second inequality was obtained since $h$ is a non-decreasing function (see BA $i)$). Finally, both inequalities follow from (3.4). In consequence, since $l(0) = 0$, it is obtained that

By applying Lemma 3.4 to the last relation and due to $\mathbb{E}[l(D)] < \infty$ as a consequence of BA $iii)$, it is obtained that $v(\tilde{\pi}, x) < \infty$. Since $x\in X$ is arbitrary, the result holds. □

Remark 3.6. Note that Lemma 3.5 guarantees that $V(x) < \infty$, for all $x\in X$, due to $V(x)\leq v(\tilde{\pi}, x)$, for all $x\in X$.

As a consequence of the previous results, the following theorem is valid (see Theorem 4.2.3 and Lemma 4.2.8 in ^[17]).

Theorem 3.7. The following statements hold:

i) For each $x\in X$, the optimal value function satisfies the dynamic programming equation:

ii) There exists an optimal stationary policy $f\in \mathbb{F}$ such that for each $x\in X$ the following equation holds

iii) The value iteration functions, defined as $V_{0}(x): = 0$ and

$n\geq 1, x\in X$, converge monotonically increasing to the optimal value function $V$.

Remark 3.8. Minimizers of the value iteration functions will be denoted by $f_{n}$, $n\geq 0$. These minimizers satisfy the following, $f_{0}(x) = 0$ and for each $n\geq1$: $V_{n}(x) = C(x, f_{n}(x))+\alpha \mathbb{E}[V_{n-1}((x+\eta f_{n}(x)-D)^+)]$, for all $x\in X$.

4.   Characterization of $(s, S)$ policies

This section deals with the characterization of the minimizers of the value iteration functions and the optimal policies by $(s, S)$ policies.

Definition 4.1. Let $f \in \mathbb{F}$ be a stationary policy, if there exists $(s, S)\in \mathbb{R}^2$ such that $0\leq s\leq S$ and

$f$ is called a $(s, S)$ stationary policy.

Define the following functions

and

for $u\in X$, where $\hat{H}(u) = H(u, 0)$. In consequence, value iteration functions $\{V_n\}$ can be expressed for each $n\geq 1$ and $x \in X$ as follows,

Making the change of variable $y: = x+a$, the previous equation is equivalent to

for $n\geq 1$ and $V_0(x) = 0$, $x\in X$.

The following definition will be applied to characterize the optimal policy ^[21].

Definition 4.2. A function $\vartheta: [0, \infty)\longrightarrow [0, \infty)$ is called norm-like if $\vartheta(x) \to \infty$ as $x \to \infty$; this means that the sub-level sets $\{x: \vartheta(x)\leq r\}$ are precompact for each $r > 0$.

Lemma 4.3. For each $n = 1, 2, \ldots$, the following statements hold

i) $V_n$ is a non-decreasing convex function,

ii) $G_n$ is a convex function on $X$,

iii) $G_n$ is norm-like.

Proof. $i)$ The proof is by induction. It will be proved that $V_1$ is a non-decreasing function,

if $x, y \in X$ and $x\leq y$, due to $C(\cdot, a)$ is a non-decreasing function for each $a\in [0, \infty)$] (see (2.3) and AIM). Then, $V_1$ is a non-decreasing function. On the other hand, using Lemma 1 in ^[19] together with the fact that $C$ is a convex function, it is obtained that $V_1$ is a convex function. Now, suppose that $V_n$ is a non-decreasing convex function. To prove that $V_{n+1}$ is a non-decreasing function, observe that, as a consequence of $V_n$ is non-decreasing, the following inequality holds

for all $x, y\in X$, with $x\leq y$ and $a, s\in [0, \infty)$. In consequence,

for all $x, y\in X$, with $x\leq y$ and $a, s\in [0, \infty)$. Consequently, taking the minimum with respect to $a$, in each side of inequality (4.5), it results that $V_{n+1}(x)\leq V_{n+1}(y)$, $x, y \in X$ with $x\leq y$. Now, it will be proved that $V_{n+1}$ is a convex function. To this end, note that $C(x, a)+\alpha \mathbb{E}[V_{n}((x+\eta a-D)^+)]$ is a convex function for each $(x, a)\in \mathbb{K}$, this statement is a consequence of the induction hypothesis and the convexity of the cost function. Then, from Lemma 1 of ^[19], it may be concluded that $V_{n+1}$ is a convex function.

$ii)$ The previous statement implies that $\hat{V}_{n}$ is a convex function for each $n\geq 1$, as the following relations evidence it:

for each $u_{1}, u_{2}\in X$, $\lambda \in [0, 1]$. Consequently, since $\hat{H}$ is convex, (4.3) implies that $G_{n}$ is a convex function for each $n\geq 1$.

$iii)$ Observe that $G_n(u)\geq cu$, for $n \geq 1$ with $c > 0$. Hence, for each $n\geq 0$, $G_n(u)\to \infty$, when $u \to \infty$. This implies that $G_n$ is a norm-like function for each $n\geq 1$. □

Remark 4.4. Observe that for each $n \geq 1$, we have that

$i)\; G_n$ (4.3) is a continuous function on $(0, \infty)$, as a consequence of convexity of function $G_{n}$.

$ii)$ Given that $G_{n}$ is a norm-like function it follows that

is a non-empty set.

Lemma 4.5 is a modified version of Lemma 2.1 of ^[7], the proof of this lemma is presented here for the completeness of the paper.

Lemma 4.5. For each $n\geq 1$, let $S_{n} \in \underset{{y\geq 0}}{\operatorname{arg}\, \operatorname{min}}\; G_{n}(y)$ and $s_n: = \inf\{0 \leq x\leq S_n:G_n(x) \leq K+G_n(S_{n}) \}$ then the following statements hold:

$i) \; G_n(s_n) = K+G(S_n)$, if $s_n > 0$.

$ii) \; G_n(x) \leq K+G_n(S_n)$, $0\leq s_n\leq x\leq S_n$.

$iii) \; K+G_n(S_n)\leq G_n(x)$, $0\leq x < s_n$.

$iv) \; G_n(x)$ is a decreasing function on $[0, s_n)$.

Proof. Let $n\geq 1$. $i)$ Suppose that $s_n > 0$ then $G_n(0) > K+G_n(S_n)\geq G_n(s_n)$. Hence, there exists $u\in(0, s_{n})$ such that $G_n(u) = G_n(S_n)+K$, because of the continuity of function $G_{n}$ (see Remark 4.4). Now, observe that $s_n\leq u$, due to the definition of $s_n$, then $s_n = u$ and the result holds.

$ii)$ This statement follows directly from the definition of $s_n$.

$iii)$ Consider $0\leq x < s_n$, by Lemma 4.3 $ii)$, it follows that

then

Thus,

Now, due to $K+G_n(S_n)-G_n(s_n) = 0$ given that and $x\leq s_n\leq S_n$, (4.6) implies that $G_n(s_n)\leq G_n(x)$. Therefore, $K+G_n(S_n) = G_n(s_n)\leq G_n(x)$ with $0\leq x < s_n$.

$iv)$ Let $x_1, x_2 \in [0, s_{n})$ with $x_{1}\leq x_{2}$. Hence, by Lemma 4.3 $ii)$,

Now, from statement $iii)$, it is obtained that $G_{n}(x_{2})\geq K+G_{n}(S_{n})$, this relation and (4.7) together lead to $0\geq G_n(x_2)-G_n(x_1)$ and the result follows. □

A consequence of Lemma 4.5 is the following result.

Theorem 4.6. Consider $\{(s_{n}, S_{n}):n = 1, 2, \ldots\}$ in Lemma 4.5. Then, the minimizers of the value iteration functions are given by

with $0 < s_{n}\leq S_{n}$ and $n = 1, 2, ...$

Proof. Let $n\geq 1$ and $x\in X$. The proof will proceed by considering three cases depending on the order relation between state $x$, $s_n$, and $S_n$. The following simple but important claim is established.

Claim 1: $u \in \underset{{y\geq x}}{\operatorname{arg}\, \operatorname{min}}\; \{K+G_n(y)\}$ if and only if $u-x \in \underset{{a\geq 0}}{\operatorname{arg}\, \operatorname{min}}\; \{K+G_n(x+a)\}$.

Case 1. Suppose that $x < s_n$, Lemma 4.5 $iii)$ yields that $K+G_{n}(S_{n})\leq G_{n}(x)$. Consequently, by (4.4) and Claim, it follows that $f_n(x) = S_n-x$.

Case 2. Consider $s_n \leq x \leq S_n$, then by Lemma 4.5 $ii)$, it is obtained that $G_{n}(x)\leq K+G_{n}(S_{n})$. In consequence, $(4.4)$ implies that $f_n(x) = 0$.

Case 3. Finally, assume that $S_n < x$, since $G_{n}$ is a convex function, we have that $G_{n}$ is a non-decreasing function on $(S_{n}, \infty)$, this fact implies that $x \in \underset{{y\geq x}}{\operatorname{arg}\, \operatorname{min}}\; \{K+G_n(y)\}$. Now, from Claim 1, it follows that $0 \in \underset{{a\geq 0}}{\operatorname{arg}\, \operatorname{min}}\; \{K+G_n(x+a)\}$. Therefore, $f_{n}(x) = 0$.

The previous cases guarantee the truth of Theorem 4.6. □

Remark 4.7. Observe that Theorem 3.7 and (4.4) imply that the optimal value function satisfies the following equation

with

and

Since $V_n \uparrow V$ because of Theorem 3.7 and by the Dominated Convergence Theorem, it yields that

Lemma 4.8. $G$ is a norm-like and convex function.

Proof. Observe that $G(x)\geq cx$ for each $x\geq 0$. Hence, $G(x)\to \infty$ as $x$ goes to infinity. Thus, by Definition 4.2, $G$ is a norm-like function. Furthermore, (4.11) implies that $G$ is a convex function since $G_n$ is a convex function for each $n\geq 1$ (see Lemma 4.3). □

The proofs of Lemma 4.9 and Theorem 4.10 are in similar lines with the proofs of Lemma 4.5 and Theorem 4.6, respectively. Thus, the proofs will be omitted.

Lemma 4.9. Let $S \in \underset{{y\geq 0}}{\operatorname{arg}\, \operatorname{min}}\; G(y)$ and $s: = \inf\{0 \leq x\leq S:G(x) \leq K+G(S) \}$ then the following statements are valid.

$i) \; G(s) = G(S)+K$, if $s > 0$.

$ii) \; G(x) \leq K+G(S)$, $0\leq s\leq x\leq S$.

$iii) \; G(S)+K\leq G(x)$, $0\leq x < s$.

$iv) \; G(x)$ is a decreasing function on $[0, s)$.

Theorem 4.10. Consider $(s, S)$ as in Lemma 4.9. Hence, the optimal policy for the inventory system is an $(s, S)$ policy given by

with $0 < s\leq S$.

Remark 4.11. Observe that in Theorem 4.10, if $s = 0$ then $G(x)\leq G(S)+K$, for all $x\geq 0$, which implies that $f(x) = 0$, for all $x\geq 0$. A similar argumentation guarantees the following assertion: $f_{n}(x) = 0$, $x\in X$, if $s_{n} = 0$ for $n\geq 1$.

5.   Convergence of minimizers of the value iteration functions

In this section, the convergence of the minimizers of the value iteration function will be analyzed. First, the next auxiliary results are exposed.

Lemma 5.1. Let $\hat{C} \subset (0, \infty)$ be a compact set. Then, the following statements hold.

$i) \; \{G_n\}$ converges uniformly to $G$ on $\hat{C}$.

$ii)$ For each $\{u_n\} \subset \hat{C}$ such that $u_n \to u \in \hat{C}$, $\lim_{n\to \infty} G_n(u_n) = G(u)$.

Proof. $i)$ This statement is a direct consequence of (4.11) and Theorem 7.13 in ^[24].

$ii)$ Applying $i)$ of Lemma 5.1, it follows that for all $\epsilon > 0$ there exists $N_1\geq 1$ such that for each $x\in \hat{C}$,

if $n\geq N_1$. Furthermore, by continuity of function $G$ on $\hat{C}$, there exists $\delta > 0$ such that

if $|y-x| < \delta$. Now, since $u_n$ converges to $u$ when $n$ goes to infinity, there exists $N_2\geq 1$ such that

if $n\geq N_2$. Let $N = \max\{N_1, N_2\}$. Thus, (5.2) and (5.3) together leads to

if $n\geq N$. Then, taking $x = u_n$ in (5.1), it yields that

Therefore, by (5.4) and (5.5), $|G_n(u_n)-G(u)| < \epsilon$. This last relation implies that $G_n(u_n)$ converges to $G(u)$, when $n$ goes to infinity. □

Lemma 5.2. Let $B: = \{x > 0| G_1(x) \leq \inf_{y\geq 0}G(y)+ K\}$ and $\left\lbrace (s_{n}, S_{n}) :n\geq 1\right\rbrace$ as in Lemma 4.5. Then, the following statements hold

$i) \; B$ is a compact set in $\mathbb{R}$.

$ii) \; s_n, S_n \in B$, for each $n = 1, 2, ...$.

$iii)$ There exists a subsequence $\{(s_{n_k}, S_{n_k})\}$ of $\{(s_n, S_n)\}$, which converges to $(s^*, S^*)$ such that $S^* \in \underset{{y\geq 0}}{\operatorname{arg}\, \operatorname{min}}\; G(y)$ and $G(s^*) = K+G(S^*)$ with $0 < s^* \leq S^*$.

Proof. $i)$ First, observe that $B$ is a closed set, since $G_1$ is a continuous function on $(0, \infty)$. Now suppose that $B$ is not bounded, then there exists a sequence $\{b_n\}\subset B$ such that $b_n$ converges to infinity. Thus,

The previous relation is a contradiction since $\inf_{y\geq 0}G(y)+K < \infty$. Therefore $B$ is a compact set.

$ii)$ Observe that for each $n = 1, 2, ...$ the following inequalities are valid

since $\{G_n\}$ is a non-decreasing sequence whose limit is $G$ and the equalities are valid due to Lemma 4.5. Then, for each $n = 1, 2, ...$, $s_n$, $S_n \in B$.

$iii)$ The previous statements $i)$ and $ii)$ imply that there exists a subsequence $\{(s_{n_k}, S_{n_k})\}$ convergent to $(s^*, S^*)\in B^2$, with $0 < s^*\leq S^*$. Furthermore, due to Lemma 5.1 $ii)$, it yields that

Now, Lemma 4.5 implies that $G_{n_k}(S_{n_k})\leq G_{n_k}(x)$, $x\geq 0$. Then, when $k$ goes to infinity in the last inequality, it is obtained that $G(S^*)\leq G(x)$, $x\geq 0$. In consequence, $S^* \in \underset{{y \geq 0}}{\operatorname{arg}\, \operatorname{min}}\; {G(y)}$. On the other hand, as a consequence of Lemma 4.5 $ii)$, (5.6) and (5.7), it follows that $G(s^*) = K+G(S^*)$. □

Theorem 5.3. Let $\{f_n\}$ be the sequence of minimizers of value iteration functions. Then, there exists a subsequence $\{f_{n_k}\}$ such that converges uniformly on $X$ to an $(s^*, S^*)$ optimal policy.

Proof. First, in view of Theorem 4.6, for each $n\geq 1$, $f_{n}(x) = (S_{n}-x)I_{\{x:x\leq s_{n}\}}(x)$ with $0 < s_{n}\leq S_{n}$. Then, using $iii)$ of Lemma 5.2, it is obtained a subsequence $(s_{n_{n}}, S_{n_{k}})$ such that $(s_{n_{n}}, S_{n_{k}})\longrightarrow (s^*, S^*)$ when $k$ goes to infinity and $(s^*, S^*)$ satisfies the statements $i)-iv)$ of Lemma 4.9. Thus, consider

and observe that,

when $k\longrightarrow \infty$. Furthermore, $f^*$ is an optimal policy due to Theorem 4.10. This concludes the proof of the theorem. □

6.   A numerical example

Consider an inventory control system with an exponential distribution on the demand with parameter $\lambda = 0.1$ and suppose that the parameter of the variable $\eta$ is $p = 0.5$. On the other hand, for each $u\geq 0$, suppose that $h(u) = \gamma_1 u$, $l(u) = \gamma_2 u$, where $\gamma_1$, $\gamma_2$ are non negative constants. In this case, $\gamma_1$ represents the cost per unit shortage and $\gamma_2$ represents the cost per unit demand not supplied.

Furthermore, observe that for each $u\geq 0$,

Then by (4.3) for $n = 1$, it yields that

Lemma 6.1. For each $n = 1, 2, ...$, $G_n$ is a strictly convex function.

Proof. First, observe that the derivative of $G_{1}$ is given by $G_1^{\prime }(u) = p\gamma_1+c-p(\gamma_1+\gamma_2)e^{-\lambda u}$ and the second derivative $G_1^{\prime \prime }(u) = \lambda p(\gamma_1+\gamma_2)e^{-\lambda u} > 0, \; u\geq 0$. Consequently, the function $G_1$ is strictly convex. Then, since $G_{n}(u) = G_{1}(u)+\alpha p\hat{V}_{n-1}(u)$, $u\geq 0$ (see 4.3), it is concluded that for each $n = 1, 2, ...$, $G_{n}$ is a strictly convex function, due to $\hat{V}_{n}$ is a convex function for each $n\geq 1$ (see proof of Lemma 4.3) and $G_{1}$ is a strictly convex function. Then, the result follows. □

Remark 6.2. A consequence of Lemma 6.1 is the uniqueness of minimizers of the value iteration functions (see (4.4)). Furthermore, observe that (4.10) can be rewritten for each $u\geq 0$ as follows $G(u) = G_1(u)+\alpha pV^*(u)$, which is a strictly convex function, as a consequence of Lemma 6.1 and Remark 4.7. Hence, the optimal policy is unique due to the Eq (4.9).

Lemma 6.3. The unique minimizer of function $G_1$ is $y^* = -\lambda^{-1}(ln(\gamma_1+c/p)-ln(\gamma_1+\gamma_2) > 0,$ if $c/p < \gamma_2$.

Proof. Observe that, due to Lemma 6.1, the minimizer $y^*$ is characterized by the first order condition:

Thus, $y^* = -\lambda^{-1}ln((\gamma_1+c/p)(\gamma_1+\gamma_2)^{-1})$. In particular, since $c/p < \gamma_2$, it is obtained that $(\gamma_1+c/p)(\gamma_1+\gamma_2)^{-1} < 1$, in consequence, $y^* > 0$. □

The following algorithm is applied to approximate the optimal value function $V$ and the optimal policy $f$, for each state $x$ given as initial condition. The validity of the next algorithm is sustained in the following results: Lemma 4.9, Theorem 4.10, Lemmas 5.2 and 6.3.

Consider the following parameters: $\gamma_1 = \gamma_2 = 30$, $K = 1.5$, $c = 2.5$, $\epsilon = 0.01$, $\alpha = 0.2$ and $x = 40$, observe that $\gamma_2 > c/p$. Table 1 illustrates the numerical results obtained by applying the previous Algorithm 1.

Observe that the convergence of the optimal value function and $(s, S)$ policy are illustrated, see Theorem 4.10 and Lemma 5.2, respectively. Therefore, $f(x) = 15.75$ and $V(x) = 1498.92$ for $x = 40$ with an error $\epsilon = 0.01$.

7.   Conclusions

In this paper, a discrete-time inventory system was presented. In the manuscript, conditions for convexity and monotonicity in the components of the Markov control model were proposed. In this inventory system, the existence of stationary policies (s, S) was proved. To achieve this goal, the methodology of dynamic programming in discrete time was applied. Moreover, the existence of a subsequence of minimizers of the value iteration functions converging to a (s, S) optimal policy of the inventory system was proved. Finally, a numerical algorithm for approximating the optimal inventory cost and policy is presented and applied in a numerical example. Future work in this direction includes the following:

● Incorporating Markovian demand into the inventory model.

● Investigating the lost-sales inventory system under other performance measures, e.g., considering the long average or risk-sensitive criteria.

● Implementing the manuscript's proposal on real-world data by finding a suitable database for the assumptions presented in this manuscript.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

The authors would like to thank the referees for their valuable comments, corrections and suggestions, which lead to an improvement of the original paper.

Conflict of interest

The authors declare that they have no competing interests.