The optimal probability of the risk for finite horizon partially observable Markov decision processes

1.   Introduction

Analyzing the risk performance of a stochastic dynamic system is an important optimization control problem. Additionally, both theoretical and applied aspects are observed in relation to financial insurance ^[1], communication networks ^[2] and queuing systems ^[3]. Since the conventional expectation criterion could not effectively reflect the risk performance of the system, the criteria of risk probability were first proposed by Sobel ^[4], and implemented in Markov decision processes (MDPs). Afterwards, many scholars focused on the research of optimization problems of risk probability in MDPs. According to the characteristics of the sojourn time of the system state, these existing studies can be roughly divided into four categories: (i) Discrete-time Markov decision processes (DTMDPs)^[5,6,7,8]; (ii) Semi-Markov decision processes (SMDPs) ^[9,10,11]; (iii) Continuous-time Markov decision processes (CTMDPs)^[12,13,14]; and (iv) Piecewise deterministic Markov decision processes (PDMDPs)^[15]. A common feature of these existing literatures is that the system state is completely observable. However, in practical applications such as machine maintenance and finance, the traditional models of MDPs cannot effectively depict these practical problems because the information of the decision environment cannot be completely observed or perceived. Therefore, it is necessary to establish partially observable Markov decision processes (POMDPs) to optimize the risk probability of the control system.

Compared with completely observable Markov decision processes (COMDPs), the model of POMDPs is a more extensive stochastic control model with important theoretical significance and practical application values, and is widely used in fields such as industry, computational science, finance, and artificial intelligence. Therefore, many scholars began to focus on the problem of expected optimal for POMDPs. More specifically, Drake ^[16] established the POMDPs model, which attracted the attention of many experts and scholars. Regarding the expected optimal problem, Hinderer ^[17] discussed the finite state situation. Rhenius ^[18] and Hern$\acute{a}$ndez-Lema ^[19] discussed a more general state situation. Smallwood and Sondik ^[20] further expanded the algorithm to calculate the optimal strategies and value function (VF) by employing the dynamic programming method. Sawaki and Ichikawa ^[21] proposed a successive approximation method to calculate the optimal strategy and VF. White and Scherer ^[22] solved the infinite discounted optimization problem by modifying the reward function and employing the iterative approximation algorithm. B$\ddot{a}$uerle and Rieder ^[1] established the optimality equation by equivalently converting POMDPs into a filter MDPs model and proved the existence of an optimal strategy. Feinberg et al. ^[23] established some sufficient conditions to assure the existence of optimal strategies and an optimality equation for more general state and action spaces. In addition, many scholars have focused on the research of computational algorithms for the POMDPs ^[24]. However, these criteria mainly focus on the expected value of the total rewards, which could not effectively describe the risk situation faced by the control system. Therefore, it is necessary to introduce the criteria of the risk probability that can effectively demonstrate the risk performance of the system. An overview of the existing literature indicates that the criterion of the risk probability concerning POMDPs has not been researched thus far. This paper is the first attempt to solve the optimization problem of the risk probability for POMDPs.

The optimization problem intends to minimize the risk probability criterion, that is, the probability value of the system's total rewards does not exceed the the profit goal. Because the reward levels are regarded as the second component of the extended state, it is necessary to redescribe the evolution process of the system and redefine the history-dependent, Markov and stationary policies. Thus, for any given redefined policy, a new probability space must be reconstructed using the well-known Ionescu Tulcea's theorem (see e.g., Proposition 7.45 in ^[25]), which is based on any initial system state and reward goal. Second, the unobservable state's conditional probability distribution is constructed, by redefining the Bayes operator (including the reward levels) and establishing the filter equations. Then, based on the aforementioned conditional probability distribution, a new filtered risk probability MDPs model is established by expanding the state and action space and modifying the transfer kernel and reward function. Furthermore, we prove that the newly filtered MDPs model can reveal the regular relationship between partially and completely observable optimal problems. On account of risk probability optimality theory for COMDPs, by using the value iteration advanced technique, the optimality equation is established, and the existence of optimal policies is proven. Finally, a machine maintenance example is given to present our main results, which include using the iteration algorithm to calculate the value function and an optimal policy.

The rest of the manuscript is outlined as follows. Section 2 presents a minimization risk probability problem dealing for POMDPs. Section 3 presents the main results, including the existence of the optimality equation and optimal policies. An illustration is given to present the value iteration algorithm for calculating the VF and OP in Section 4.

2.   The model of POMDPs

The model of POMDPs consists of the following elements:

which have the following meanings:

(a) $E_{X}\times E_{Y}$ represents a Borel space with a Borel $\sigma$-algebra $\mathscr{B}(E_{X}\times E_{Y})$. The element $(x, y)\in E_{X}\times E_{Y}$ is the system state, where $x$ denotes the state's observable portion, and $y$ denotes the state's unobservable portion.

(b) $A$ represents the action space of a Borel space with a Borel $\sigma$-algebra $\mathscr{A}$. $A(x)\subseteq A$ represents the set of admissible actions in state $x\in E_{X}$, which is assumed to be finite. Moreover, the set of all composable pairs of state actions is denoted by $\mathbb{K}: = \{(x, a)\mid x\in E_{X}, a\in A(x)\}$.

(c) $Q(\cdot, \cdot\mid x, y, a)$ denotes the probability of the transition from $E_{X}\times E_{Y}\times A$ to $E_{X}\times E_{Y}$, which is used to describe the transition mechanism in the controlled state process. For simplicity, we introduce $Q^{X}$ to represent the marginal transition probability $Q^{X}(B|x, y, a): = Q^{X}(B\times E_{Y}|x, y, a)$.

(e) $r(x, y, a)$ denotes a nonnegative real-valued measurable reward function from $\mathbb{K}\times E_{Y}$ to $R^{+}: = [0, +\infty)$.

(d) $Q_{0}$ denotes the initial probability distribution of the unobservable state.

The evolution of the risk probability POMDP is characterized as follows: At $s_{0} = 0$, based on the observed state $x_{0}$ and the reward goal (reward level) $\tilde{\lambda}_{0}: = \lambda_{0}$, the decision maker could pick an action $a_{0}$ from the set of allowed actions $A(x_{0})$. Then, the observed state of the system stays until time $s_{1} = 1$, at which point, the system state transfers to the state $x_{1}\in B_{1}\subseteq E_{X}$ based on the probability of the transition $\int_{B_{1}}\int_{E_{Y}}Q(x_{1}, y_{1}\mid x_{0}, y_{0}, a_{0})Q_{0}({\mathrm d}y_{0})$. Meanwhile, the unobservable state $y_{0}$ also transfers to the next state $y_{1}$ with a certain probability, which is constructed by the Bayes operator in the undermentioned (3.3). Moreover, during this period, the control system will generate the rewards $r(x_{0}, y_{0}, a_{0})$. Then, the goal of the corresponding reward would become $\tilde{\lambda}_{1} = \lambda_{0}-r(x_{0}, y_{0}, a_{0})$. At the new decision time $s_{1} = 1$, based on the observable information of the system $h_{1} = (x_{0}, \lambda_{0}, a_{0}, x_{1}, \tilde{\lambda}_{1})$, the decision maker picks a new action $a_{1}\in A(x_{1})$. Afterward, the system evolves similarly and produces a so-called observable history up to time $s_{k} = k$:

where $x_{k}, y_{k}$ denote the system state's observable and unobservable section at the $k$-th moment of decision, respectively, $a_{k}$ represents the action chosen by the decision maker at time $s_{k} = k$, $\tilde{\lambda}_{k}$ denotes the reward goal, which means that the decision maker will try his/her best to regulate the total rewards not exceeding the goal of the reward, and it conforms to the following relation:

for $k = 0, 1, \ldots.$

The sets of all the histories of observable $h_{k}$ are represented by $H_{0}: = E_{X}\times R$, $H_{k}: = H_{k-1}\times A\times E_{X}\times R$ for $k\geq1$. Based on all observable histories, some policies are introduced.

Definition 2.1. (a) A sequence $\pi = \{\pi_{k}, k\geq0\}$ is said to be a randomized history-dependent policy if a stochastic kernel $\pi_{k}: H_{k}\rightarrow A$ satisfying the following:

(b) A randomized history-dependent policy is said to be deterministic if there exists a sequence $\{g_{k}\}$ of measurable functions $g_{k}$ from $H_{k}$ to $A$ with $g_{k}(h_{k})\in A(x_{k})$, and $\pi_{k}(\cdot|h_{k})$ is the Dirac measure at $g_{k}(h_{k})$ for all $h_{k}\in H_{k}, k\geq0$. The sets of all the randomized, deterministic policies are represented by $\Pi, \Pi_{DH}$, respectively.

The risk probability POMDP needs to consider the system state and the reward goal, which is different from the conventional expectation MDP that only considers the system state. Thus, the results of the available classical expectation MDP cannot be applied to the proposed model. First, we need to reconstruct the measurable space $(\Omega, \mathscr{F})$ as follows: $\Omega: = \{(x_{0}, y_{0}, \lambda_{0}, a_{0}, x_{1}, y_{1}, \lambda_{1}, a_{1}, \ldots, x_{k}, y_{k}, \lambda_{k} \; a_{k}, \ldots, )| x_0\in E_{X}, y_{0}\in E_{Y}, \lambda_{0}\in R, a_{0}\in A, x_{l}\in E, y_{l}\in E_{Y}, \lambda_{l}\in R, a_{l}\in A$ with $1\leq l\leq k, k\geq1 \}$ denotes the sample space, which is endowed with the Borel $\sigma$-algebra $\mathscr{F}$. For any $\omega: = (x_{0}, y_{0}, \lambda_{0}, a_{0}, x_{1}, y_{1}, \lambda_{1}, a_{1}, \ldots, x_{k}, y_{k}, \lambda_{k} \; a_{k}, \ldots, )\in \Omega$, some random variables are defined as follows:

The $\omega$ will be omitted for convenience.

For any policy $\pi\in\Pi$, $(x, \lambda)\in E_{X}\times R$, $Q_{0}$ of $Y_{0}$ on $E_{Y}$, based on the Ionescu Tulcea's (e.g., Proposition 7.45 in ^[25]), the unique probability measure $P_{(x, \lambda)}^{\pi} = \int_{E_{Y}}P_{(x, \lambda, y)}^{\pi}(\cdot)Q_{0}({\mathrm d}y)$ on $(\Omega, \mathscr{F})$ is constructed as follows: for all $B\in \mathscr{B}(E_{X}), C\in \mathscr{B}(E_{Y}), G\in \mathscr{B}(R), D\in \mathscr{B}(A), h_{k}\in H_{k}, k = 0, 1, \ldots$

The expectation operator corresponding to the probability measure $P_{(x, \lambda)}^\pi$ can be expressed as $\mathbb{E}_{(x, \lambda)}^\pi$.

For any $(x, \lambda)\in E_{X}\times R$ and $\pi\in\Pi$, the risk probability criterion of POMDPs is defined as follows:

Then, $F^{*}_{N}(x, \lambda): = \inf_{\pi\in\Pi}F^{\pi}_{N}(x, \lambda)$ is known as the risk probability value function.

Definition 2.2. A policy $\pi^{*}\in\Pi$ is called the optimal of the risk probability if

3.   Main result

The general objective of the manuscript is to optimize the criterion of the risk probability and establish both the optimality equation's solution and the existence of the strategy's conditions.

Notation: Let $\mathbb{P}(E_{Y})$ be the space of all the probability measures on $E_{Y}$.

Assumption 3.1. Assume that the transition probability has a probability density function $q$ that satisfies $Q(d(x^{'}, y^{'})|x, y, a) = q(x^{'}, y^{'}|x, y, a)\rho({\mathrm d}x^{'})\nu({\mathrm d}y^{'})$ for some $\sigma$-finite measures $\rho$ and $\nu$.

The Bayes operator $\Phi:E_{X}\times R \times A\times E_{X}\times R\times\mathbb{P}(E_{Y})\rightarrow \mathbb{P}(E_{Y})$ is first defined as follows:

where $C\in\mathscr{B}(E_{Y})$, $\mu$ denotes the distribution of the unobservable state. Furthermore, by using the iterative method, for any $h_{k}\in H_{k}, k = 0, 1, \ldots$, the conditional probability distribution $\mu_{n}$ of the unobservable variable $Y_{n}$ is presented by the following:

which are called filter equations.

Lemma 3.1. Under Assumption 3.1, for any $\pi\in{\Pi}, B\in \mathscr{B}(E_{Y})$, the following statement holds:

Proof. For each $\pi\in\Pi$ and $x \in E_{X}, \lambda \in R$, the following result is proven by induction:

for the bounded and measurable function $V:H_{k}\times E_{Y} \rightarrow R$ and $\hat{V}(h_{n}) = \int V(h_{n}, y_{n})\mu_{n}({\mathrm d} y_{n}|h_{n})$. Since $\hat{V}(h_{0}) = \int V(h_{0}, y)Q_{0}({\mathrm d}y)$, Fact (3.5) is true when $n = 0$. For any given $h_{n-1}\in H_{n-1}, n\geq1$, suppose that the Fact (3.5) holds for $k = n-1$. Using (2.6), the Bayes operator's definition and Fubini's theorem, we obtain the following:

On the other hand, by induction, we have the following:

which, together with Eq (3.6), implies the fact that (3.5) is satisfied. Specially, $V(X_{0}, \Lambda_{0}, A_{0}, \ldots, X_{n}, \; \Lambda_{n}, Y_{n}) = I_{B\times C} (Y_{n}, (X_{0}, \Lambda_{0}, A_{0}, \ldots, X_{n}, \Lambda_{n}))$, we obtain the following:

which implies that the Lemma holds.

The partially observable risk probability MDPs can be transformed into the filtered risk probability MDPs by enlarging the state space, modifying the transfer kernel and reward function.

Definition 3.1. The filtered model of POMDPs consists of the following elements $\{E, A, \hat{Q}, \hat{r}\}$, which have the following meanings:

$\bullet \; E: = E_{X}\times \mathcal{P}(E_{Y})$ denotes the state space, and its element is marked as $(x, \mu)\in E$, where $x$ denotes the observable state and $\mu$ denotes the unobservable state's conditional probability distribution.

$\bullet \; A$ denotes the action space. $A(x, \mu): = A(x)\subseteq A$ denotes the class of selectable actions in the state $(x, \mu)\in E$.

$\bullet \; \hat{r}$ denotes a nonnegative real-valued measurable reward function on $\mathbb{K}$ and satisfies the following:

$\bullet \; \hat{Q}$ denotes the transition law from $E\times R \times A$ to $E\times R$, which is specifically expressed as follows:

where $Q^{X}(B| x, \mu, a) = \int_{B}Q^{X}(B| x, y, a)\mu({\mathrm d}y)$ for all $(x, \mu)\in E, \lambda\in R, a\in A(x), B\subset E_{X}, D\subset\mathcal{P}(E_{Y}), C\subset R$.

To strictly assure the optimal problems normalization, some notations and the definition of some policies are given in the filtered MDPs. ${\Phi}$ stands for the class of stochastic kernels ${\varphi}$ on $A$ provided $E\times R$ with the property ${\varphi}(A(x)|x, \lambda, \mu) = 1$ for all $(x, \lambda, \mu)\in E\times R$. ${F}$ stands for the class of all the measurable mappings ${f}$ from $E\times R$ to $A$ with ${f}(x, \lambda, \mu)\in A(x)$ for all $(x, \lambda, \mu) \in E\times R$.

Definition 3.2. A randomized Markov policy is a sequence ${\pi}_{M} = \{{\hat{\varphi}}_{k}, k\geq0\}$ of stochastic kernels $\hat{\varphi}_{k}\in {\Phi}$ satisfying $\hat{\varphi}_{k}(A(x_{k})|x_{k}, \lambda_{k}, \mu_{k}) = 1$ for each $\mu_{k}\in \mathcal{P}(E_{Y}), k\geq 0$. This randomized Markov policy is represented as ${\pi}_{M} = \{\hat{\varphi}_{k}\}$.

A randomized Markov policy ${\pi_{M}} = \{\hat{\varphi}_{k}\}$ is called a deterministic Markov if a function sequence $\{{f_{k}}, k\geq0\}$ exists such that $\hat{\varphi}_{k}(\cdot|x_{k}, \lambda_{k}, \mu_{k})$ is concentrated at ${f_{k}}(x_{k}, \lambda_{k}, \mu_{k})$ for any $f_{k}\in {F}$.

The class of all randomized and deterministic Markov policies are recorded as ${\Pi}_{RM}, {\Pi}_{DM}$, respectively. In fact, from the above definition, these randomized Markov policies rely on historical information $h_{k}, k\geq0$. Then, for any ${\pi}_{M} = \{\hat{\varphi}_{0}, \hat{\varphi}_{1}, \ldots\}\in {\Pi}_{M}$, we can find a policy ${\pi} = \{{\pi}_{0}, {\pi}_{1}, \ldots\}\in {\Pi}$ that satisfies the following:

for $k\geq0, {h}_{k}\in {H}_{k}$. Thus, ${\Pi}_{DM}\subseteq{\Pi}_{RM}\subseteq {\Pi}$.

Based on the probability of the transition $\hat{Q}$ and initial distribution $\mu_{0}$, for any $(x, \lambda, \mu) \in E\times R$ and ${\pi}\in {\Pi}$, according to the Ionescu Tulcea theorem (e.g., Proposition 7.45 in ^[25]), the probability measure $\hat{P}_{(x, \lambda, \mu)}^{{\pi}}$ can be constructed on $(\Omega, \mathcal{F})$ as follows:

for all $h_{k}\in H_{k}, a\in A(x_{k}), B\subset E_{X}, C\in E_{Y}, D\subset R, G \in \mathscr{B}(A)$. The expected operator corresponds to the probability measure $\hat{P}_{(x, \lambda, \mu)}^{{\pi}}$ and is expressed as $\hat{\mathbb{E}}_{(x, \lambda, \mu)}^{{\pi}}$.

For any $(x, \lambda, \mu) \in E\times R$ and ${\pi}\in{\Pi}$, the value function of the filtered MDP is given by the following:

Notation: For any policy $\pi\in\Pi$ and $(x, \lambda, \mu) \in E\times R$, the risk probability of the total rewards $U_{n}^{\pi}$ is defined as follows:

with $n = 0, 1, \ldots, N$.

Moreover, the minimal risk probability of the filtered MDPs model is defined by the following:

Let $\mathcal{U}$ be the class of mappings $U: E\times R \rightarrow [0, 1]$. For any $(x, \lambda, \mu)\in E\times R, U\in\mathcal{U}$, $\varphi\in\phi$, and $a\in A(x)$, the operators $T^{\varphi}U$ and $TU$ are defined as follows:

To strictly show the unobservable state's conditional distribution, some characteristics of the filter equation are given.

Lemma 3.2. Under Assumption 3.1, for each $\pi\in\Pi$, $x \in E_{X}, \lambda \in R$. Then, $F^{\pi}_{N}(x, \lambda) = U^{{\pi}}_{N}(x, \lambda, Q_{0})$, $F^{*}_{N}(x, \lambda) = U^{*}_{N}(x, \lambda, Q_{0})$.

Proof. For each $\pi\in\Pi$ and $x \in E_{X}$, the following result is first proven by induction:

with $n = -1, 0, 1, 2, \ldots$, for the reward goal function $\Lambda:K\times E_{Y} \rightarrow R^{+}$ and $\hat{\Lambda} = \int \Lambda(x, y, a)\mu({\mathrm d} y)$.

Based on $F^{\pi}_{-1} = U^{{\pi}}_{-1} = I_{[0, +\infty)}(\lambda)$, for any $\pi\in\Pi$ and $x \in E_{X}, \lambda \in R$, Eq (3.17) is valid when $n = -1$. Suppose that Fact (3.17) holds for $k = n$; by (2.6),

is obtained, where $^{1}\pi: = \{\pi_{1}, \pi_{2}, ...\}$ represents the 1-shift policy of $\pi$.

On the other side, by Eq (3.4), we have the following:

which, together with Eq (3.18) and the inductive hypothesis, can prove Eq (3.17) for $n = 0, 1, \ldots, N$, i.e, $F^{\pi}_{n}(x, \lambda) = U^{{\pi}}_{n}(x, \lambda, \mu)$.

For $n = N$, $F^{\pi}_{N}(x, \lambda) = U^{{\pi}}_{N}(x, \lambda, \mu)$, which yields $F^{*}_{N}(x, \lambda) = U^{*}_{N}(x, \lambda, Q_{0})$ for the arbitrary policy $\pi$.

The establishment of the optimality equation requires the following theorem.

Theorem 3.1. Suppose that Assumption 3.1 is satisfied. Then, for any $(x, \lambda, \mu) \in E\times R, \pi = \{\pi_{0}, \pi_{1}, \ldots\}\in\Pi, n\geq0$, the following statement holds: $U^{\pi}_{n+1}(x, \lambda, \mu) = T^{\pi_{0}}U_{n}^{^{1}\pi}(x, \lambda, \mu)$, where $U^{\pi}_{0}(x, \lambda, \mu) = I_{[0, +\infty)}(\lambda), ^{1}\pi: = \{\pi_{1}, \pi_{2}, ...\}$ represents the 1-shift policy of $\pi$.

Proof. For any $(x, \lambda, \mu) \in E\times R, \pi = \{\pi_{0}, \pi_{1}, \ldots\}\in\Pi, n = 0, 1, \ldots, N-1$, by (3.8) and the properties of conditional expectation, we can obtain the following:

The proof of this conclusion has been completed.

Theorem 3.2. Suppose that Assumption 3.1 holds. For each $(x, \lambda, \mu)\in E\times R$, then:

(a) $\{ U_{n}^{*}, n = 0, 1, \ldots, N-1\}$ satisfies the corresponding optimality equation:

(b) There exists a policy ${g}_{n}\in\Pi_{DM}$ such that $U_{n+1}^* = T^{{g}_{n}}U_{n}^*$ for $n = 0, 1, \ldots, N-1$. Then, the policy $\pi^{*}: = \{{f^{*}_0}, {f^{*}_1}, \ldots, f_{N-1}\}\in\Pi_{DH}$ is optimal, where ${f^{*}_n}(h_n): = g_{N-1-n}(x_n, {\lambda}_n, \mu_{n})$ for each $h_n\in H_n$, $n = 0, 1, \ldots, N-1$.

Proof. (a) According to Theorem 3.1 and (3.16), for each $(x, \lambda, \mu)\in E\times R, \pi = \{\pi_{0}, \pi_{1}, \ldots\}\in\Pi$, we have the following:

Since $\pi$ is arbitrary, this implies $U_{n+1}^{*}(x, \lambda, \mu)\geq TU_{n}^{*}(x, \lambda, \mu)$.

To prove the reverse condition, the following fact is needed to be proven: for any $(x, \lambda, \mu)\in E\times R$ and $n\geq-1$, there is a policy $\eta\in\Pi_{DM}$ which satisfies $U_{n}^{*}(x, \lambda, \mu) = U_{n}^{\eta}(x, \lambda, \mu)$. Since $U_{-1}^{*}(x, \lambda, \mu) = U_{-1}^{\pi}(x, \lambda, \mu) = I_{[0, +\infty)}(\lambda)$ for any $\pi\in\Pi_{M}$, this fact trivially holds for $n = -1$. Assume that there exists a policy $\zeta\in\Pi_{DM}$ that satisfies $U_{k}^{*}(x, \lambda, \mu) = U_{k}^{\zeta}(x, \lambda, \mu)$ for $n = k\geq-1$. On the other hand, since the set of actions is finite, there exists a policy $f\in \Pi_{s}$ that satisfies $T^{f}U^{*}_{k}(x, \lambda, \mu) = TU^{*}_{k}(x, \lambda, \mu)$. Then, let $\theta = \{f, \zeta\}\in\Pi_{DM}$, we know that

where the first equality is obtained by Lemma 3.1, and the second equality follows from the induction hypothesis. Combining $(3.19)$ and $(3.20)$, we have $TU_{k}^{*} = U_{k+1}^{*}$, which implies the induction hypothesis is satisfied and the result is proven.

(b) For each $(x, \lambda, \mu)\in E\times R$, the existence of a policy $g_{n}^*\in\Pi_{DM}$ satisfying $U_{n+1}^* = T^{g_{n}^*}U_{n}^*$, is determined by the finiteness of the action set for $n = 0, 1, \ldots, N-1$. Letting $\pi = \pi(n): = \{g_{n}, g_{n-1}, \ldots, g_{0}\}\in\Pi$, when $n = 0$, by Lemma 3.1 (b), $U_{1}^{\pi} = T^{g_{0}}U_{0}^{^{1}{\pi}} = T^{g_{0}}U_{0}^{*} = TU_{0}^{*} = U_{1}^{{*}}.$ Assuming that $U_{k}^{{*}} = U_{k}^{^{1}\pi^{*}}$ for $n = k$, by Lemma 3.1 (b) and part (a),

Thus, the induction hypothesis is established and $U_{N}^{\pi^{*}} = U_{N}^{{*}}$ for $\pi^{*}: = \{{f^{*}_0}, {f^{*}_1}, \ldots, f_{N-1}\}$ with ${f^{*}_n}(h_n): = g_{N-1-n}(x_n, {\lambda}_n, \mu_{n}(\cdot|h_{n})), n = 0, 1, \ldots, N-1$. Then, the policy $\pi^{*}$ is optimal.

Based on Theorem 3.2, the value iteration algorithm is established as follows:

The value iteration algorithm

Step 1. Let $U_{0}^{*}(x, \lambda, \mu): = I_{[0, +\infty)}(\lambda)$, for $(x, \lambda, \mu)\in E\times R$.

Step 2. The computation of the value function $U_{n}^{*}$ is as follows for $n = 0, 1, \ldots, N-1:$

Step 3. Find a policy ${g}_{N-1}$ that satisfies $U_{N}^* = T^{{g}_{N-1}}U_{N-1}^*$. Then, by Theorem 3.2, the policy $\pi^{*}$ is optimal.

4.   Illustration

An illustration is provided to show how both the VF and OP are calculated and illustrates the effectiveness and feasibility of the value iteration algorithm.

Example 4.1. Consider a machine production process with two types of observable product quality states (i.e., nonconforming product 0 and qualified products 1), and two types of unobservable machine operation states (i.e. poor state 1 and good state 2). According to the product quality situation $x = 1$ and the reward goal $\lambda$, at the initial time $n = 0$, when the production process is in the state $y\in\{1, 2\}$, the decision-maker can either select an ordinary maintenance action $a_{11}$ or an advanced maintenance action $a_{12}$ with a reward $r(x, y, a)$. If the product quality situation is $x = 0$, the decision maker must select an action of the advanced maintenance $a_{01}$. When the action of the maintenance $a$ is applied, the system transits to the state $(x', y')$ with probability $Q(\cdot, \cdot|x, y, a)$ at the next moment. The general objective of the decision maker is to select the optimal action to ensure that the minimum probability value of the total rewards does not exceed the target $\lambda$ from 0 to $N = 15$.

This evolution process can be formulated as a discrete-time POMDP with the state space $E_{X}\times E_{Y} = \{0, 1\}\times \{1, 2\}$; the admissible class of actions $A(0) = \{a_{01}\}, A(1) = \{a_{11}, a_{12}\}$. Assume that the probabilities of the transition are given by $Q(\cdot, \cdot| x, y, a) = Q^{X}(\cdot| x, y, a)p(\cdot|y)$, in which the probabilities of the transition $Q^{X}(\cdot| x, y, a)$ are given by the following:

The transition probabilities of the unobserved state are given by $p(2\mid2) = 1-p(1\mid2) = 0.7, p(1\mid1) = 1.$ The reward rates are given as follows:

Our main goal is to use the value iteration algorithm to compute the value function and the optimal policies.

{First, according to (3.13), since $r(0, 1, a_{01}) = r(0, 2, a_{01}) = 0$, it is known that $U^{*}(0, \lambda, \mu) = I_{[0, +\infty)}(\lambda)$. Based on the value iteration algorithm (Algorithm 1) and Matlab software, the curves of functions $T^{a_{11}}U^*(1, \lambda, \mu)$, $T^{a_{12}}U^*(1, \lambda, \mu)$ and the approximated value function $U^*(1, \lambda, \mu)$ are plotted (see Figures 1 and 2). By observing the figures, the following conclusions are attained:

$(a)$ As seen in Figure 1, when $x = 1$, if $\lambda\in (0, 4)$, the value $T^{a_{12}}U^{*}_{N}(1, \lambda, \mu)$ is less than $T^{a_{11}}U^{*}_{N}(1, \lambda, \mu)$. Otherwise, if $\lambda\in [4, +\infty)$, the value $T^{a_{11}}U^{*}_{N}(1, \lambda, \mu)$ is less than $T^{a_{12}}U^{*}_{N}(1, \lambda, \mu)$. As shown above, the observable state is $x = 1$, $\lambda\in (0, 4)$, the decision maker should choose the low risk action $a_{12}$. Conversely, if $\lambda\in [4, +\infty)$, the decision maker should choose the low risk action $a_{11}$ instead of the action $a_{12}$.

$(b)$ Based on Figure 1, the risk probability optimal policy for POMDPs at time $n = 0, 1, \ldots, N$ is given by the following:

5.   Conclusions

In this paper, we studied the problem of minimizing the risk probability criterion for finite horizon partially observable discrete-time Markov decision processes (POMDPs). Different from the classical expectation criterion, which are regarded as a component of an extended state according to the reward levels, we redefined a history-dependent policy, and reconstructed a new probability measure. Based on the Bayes operator and the filter equations we constructed, the optimization problem of risk probability can be equivalently reformulated as filtered Markov decision processes. We proposed a value iteration algorithm to establish the existence of a solution to the optimality equation, and a risk probability optimal policy.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

This work was supported by Guangxi science and technology base and talent project (Grant No. AD21159005); Foundation of Guangxi Educational Committee (Grant No. KY2022KY0342); National Natural Science Foundation of China (Grant No. 11961005, 12361091); Guangxi Natural Science Foundation Program (Grant No. 2020GXNSFAA297196); The Doctoral Foundation of Guangxi University of Science and Technology (Grant No. 18Z06).

Conflict of interest

The authors declare that there is no conflict of interests regarding the publication of this paper.