Optimal reinsurance for both an insurer and a reinsurer under general premium principles

1.   Introduction

Reinsurance is an effective risk management tool for an insurer to mitigate the underwriting risk by transferring part of the risk exposure to a reinsurer. Starting from ^[1,2], the study of optimal reinsurance has remained a fascinating topic in actuarial science. Most existing literatures on optimal reinsurance are from an insurer's point of view. For example, by maximizing the expected concave utility function of an insurer's wealth, Arrow ^[3] showed that optimal reinsurance for an insurer is a stop-loss reinsurance. The result has been extended to different settings (see, e.g., ^[4,5] and references therein). It is well known that the optimal reinsurance for an insurer, which minimizes the variance of the insurer's loss, is also a stop-loss reinsurance (see ^[6]). However, Vajda ^[7] showed that the optimal reinsurance for a reinsurer, which minimizes the variance of the reinsurer's loss with a fixed net reinsurance premium, is a quota-share reinsurance among a class of ceded loss functions that include stop-loss reinsurance. Kaluszka and Okolewski ^[8] showed that if an insurer wants to maximize his expected utility with the maximal possible claim premium principle, the optimal form of reinsurance for the insurer is a limited stop-loss reinsurance. In recent years, Cai et al. ^[9,10] introduced two classes of optimal reinsurance models by minimizing the value-at-risk (VaR) and the conditional tail expectation (CTE) of the insurer's total risk exposure. Cai et al. ^[10] proved that depending on the risk measure level of confidence, the optimal reinsurance for an insurer, which minimizes the VaR and CTE of the total risk of the insurer, can be in the form of a stop-loss reinsurance or a quota-share reinsurance or a change-loss reinsurance under the expected value principle and among the increasing convex ceded loss functions. Recent references on VaR-minimization and CTE-minimization reinsurance models can be found in ^{[11,12,13,14,15,16,17]} and references therein.

However, a reinsurance contract involves two parties, an insurer and a reinsurer. The two parties have conflicting interests. An optimal reinsurance contract for an insurer may not be optimal for a reinsurer and it might be unacceptable for a reinsurer as pointed out by Borch ^[18]. Therefore, an interesting question about optimal reinsurance is to design a reinsurance contract so that it considers the interests of both an insurer and a reinsurer. Borch ^[1] first discussed the optimal quota-share retention and stop-loss retention that maximize the product of the expected utility functions of the two parties' wealth. Cai et al. derived the optimal reinsurance contracts that maximize the joint survival probability and joint profitable probability of the two parties, and gave the sufficient conditions for optimal reinsurance contracts within a wide class of reinsurance policies and under a general reinsurance premium principle, see ^[19,20]. Cai et al. ^[21] studied the optimal reinsurance strategy, which based on the minimum convex combination of the VaR of the insurer and the reinsurer under two types of constraints. Lo ^[22] discussed the generalized problems of ^[21] by using the Neyman-Pearson approach. Based on the optimal reinsurance strategy of ^[21], Jiang et al. ^[23] proved that the optimal reinsurance strategy is a Pareto-optimal reinsurance policy and gave optimal reinsurance strategies using the geometric method. Cai et al. ^[24] studied Pareto optimality of reinsurance arrangements under general model settings and obtained the explicit forms of the Pareto-optimal reinsurance contracts under TVaR risk measure and the expected value premium principle. By geometric approach, Fang et al. ^[25] studied Pareto-optimal reinsurance policies under general premium principles and gave the explicit parameters of the optimal ceded loss functions under Dutch premium principle and Wang's premium principle. Lo and Tang ^[26] characterized the set of Pareto-optimal reinsurance policies analytically and visualized the insurer-reinsurer trade-off structure geometrically. Huang and Yin ^[27] studied two classes of optimal reinsurance models from perspectives of both insurers and reinsurers by minimizing their convex combination where the risk is measured by a distortion risk measure and the premium is given by a distortion premium principle.

In this paper, we study the optimal reinsurance models by minimizing the insurer and the reinsurer's total costs under the criteria of loss function assuming that the reinsurance premium principles satisfy risk loading and stop-loss ordering preserving. The loss function is defined by the joint VaR based on the binary lower-orthant value-at-risk and the binary upper-orthant value-at-risk, which are proposed by Embrechts and Puccetti ^[28]. Methodologically, we determine the optimal reinsurance forms using the geometric approach of ^[11] over three ceded loss function sets, the class of increasing convex ceded loss functions, the class of ceded loss functions which satisfy both ceded and retained loss functions are increasing and the class of increasing concave ceded loss functions$^{1}$^*.

$^{*1}$Throughout this paper, the terms "increasing function" and "decreasing function" mean "non-decreasing function" and "non-increasing function", respectively.

The rest of the paper is organized as follows. In Section 2, we give definitions and propose an optimal reinsurance problem that takes into consideration the interests of both an insurer and a reinsurer. In Section 3, we derive optimal reinsurance forms over three ceded loss function sets by the geometric approach of ^[11], assuming that the reinsurance premium principles satisfy risk loading and stop-loss ordering preserving. In Section 4 and Section 5, we determine the corresponding optimal parameters under expectation premium principle and Dutch premium principle respectively. In Section 6, we provide four numerical examples. Conclusions are given in Section 7.

2.   Problem formulation

Let $X$ be the loss or claim initially assumed by an insurer in a fixed time period. We assumed that $X$ is a nonnegative random variable with distribution function $F(x) = \mathbb{P}\{X\leq x\}$, survival function $S(x) = \mathbb{P}\{X\geq x\}$ and mean $\mu = \mathbb{E}(X) \ (0 < \mu < \infty)$. Under a reinsurance contract, a reinsurer will cover the part of the loss, say $f(X)$ with $0\leq f(X)\leq X$, and the insurer will retain the rest of the loss, which is denoted by $I_{f}(X) = X-f(X)$. The losses $I_{f}(X)$ and $f(X)$ are called retained loss and ceded loss, respectively. Since the reinsurer shares the risk $X$, the insurer will pay an additional cost in the form of reinsurance premium to the reinsurer. We denote the reinsurance premium by $\Pi_{f} (X)$ which corresponds to a ceded loss function $f(X)$. The total cost $T_{I}^{f}$ of the insurer is composed of two components: the retained loss $I_{f}(X)$ and the reinsurance premium $\Pi_{f} (X)$, that is

and the total cost of the reinsurer is

For individual company, an important issue is to determine their maximum agammaegate loss which can occur with some given probability, value-at-risk (VaR) serves this purpose.

Definition 2.1. For $0 < \alpha < 1$, the VaR of a non-negative random variable $X$ with distribution function $F(x) = \mathbb{P}\{X\leq x\}$ at confidence level $\alpha$ is defined as

where, $F^{-1}$ is the generalized inverse function of the distribution function $F(x)$.

The VaR defined by $(2.3)$ is the maximum loss which is not exceeded at a given probability $\alpha$. We list several properties of the VaR or the generalized inverse function $F^{-1}$.

Proposition 2.1. For any $\alpha \in (0, 1)$ and any nonnegative random variable $X$ with distribution function $F(x)$, the following properties hold:

(1) $F(F^{-1}(\alpha))\geq \alpha$.

(2) $F^{-1}(F(x))\leq x$ for $x\geq 0$.

(3) If $h$ is an increasing and left-continuous function, then ${\rm VaR}_{h(X)}(\alpha) = h({\rm VaR}_{X}(\alpha))$.

Proof. Properties (1) and (2) follow immediately from Lemma 2.13 of ^[29] and the definition of the generalized inverse function, while for property (3), see the proof of Theorem 1 in ^[30].

In this paper, we assume that the initial loss $X$ has a continuous and strictly increasing distribution function on $(0, \infty)$ with a possible mass at 0 and $\alpha\in(F(0), 1)$ to avoid trivial cases, then

For the insurer or the reinsurer, they can use Definition $2.1$ to determine their maximum agammaegate cost which can occur with some given probability $\alpha$. However, if the insurer and the reinsurer are considered as partners, then the total cost $\textbf{T}^{f}$ is a two-dimensional random vector $(T_I^{f}, T_R^{f})$. For this case, Definition $2.1$ does not make sense since, even for a one to one continuous distribution function, there are possibly infinitely many vectors $(x, y)\in [0, \infty)\times [0, \infty)$ at which $G^{f}(x, y) = \alpha$, where

is the distribution function of $(T_I^{f}, T_R^{f})$. Hence we use the definition of multivariate Value-at-Risk which is proposed by Embrechts and Puccetti (see ^[28]).

Definition 2.2. For $\alpha \in (0, 1)$, the binary lower-orthant value-at-risk at confidence level $\alpha$ for the distribution function $G^{f}(x, y)$ is the boundary of its $\alpha$-level set, defined as

Analogously, the binary upper-orthant value-at-risk at confidence level $\alpha$ for the tail function $\overline{G}^{f}(x, y)$ is defined as

where

We now provide further analysis on the binary lower-orthant value-at-risk at confidence level $\alpha$ for the distribution function $G^{f}(x, y)$ and the binary upper-orthant value-at-risk at confidence level $\alpha$ for the tail function $\overline{G}^{f}(x, y)$ over the following three admissible sets of ceded loss functions:

In the set $\mathcal{F}^2$, the increasing condition on both ceded and retained loss functions is interesting and important. Both the insurer and the reinsurer are obligated to pay more for larger loss $X$, hence it potentially reduces moral hazard. In addition, in a reinsurance contract, sometimes in order to better protect the insurer, they let the loss proportion paid by the reinsurer increases in the loss (see ^[7]). Mathematically, $f(x)/x$ is assumed to be an increasing function. If we assume that $f(x)$ is increasing and convex, then $f(x)/x$ is an increasing function. On the other hand, under the reinsurance policies with no upper limit on the indemnity, the reinsurance may be under a heavy financial burden, especially when the insurer suffers a large unexpected loss. Therefore, reinsurance contracts sometimes involve an upper limit on the indemnity in practice. In such a situation, ceded loss functions must not be convex functions but concave functions sometimes. Motivated by these observations, we consider ceded loss functions in the sets $\mathcal{F}^1$ and $\mathcal{F}^3$.

Note that $\mathcal{F}^1\subset \mathcal{F}^2$ (see ^[12]) and $\mathcal{F}^3\subset \mathcal{F}^2$. In addition, if $f \in \mathcal{F}^i, i = 1, 2, 3$, then $I_{f}$ and $f$ are increasing and continuous. Thus, from Proposition 1, we have

Based on the above analysis, we obtain the following theorem.

Theorem 2.1. For $\alpha \in (0, 1)$, the binary lower-orthant value-at-risk at confidence level $\alpha$ for the distribution function $G^{f}(x, y)$ is

and the binary upper-orthant value-at-risk at confidence level $\alpha$ for the tail function $\overline{G}^{f}(x, y)$ is

Proof. Let $S_{1} = \Big\{(x, y)\in R^{2}_{+}: G^{f}(x, y)\geq \alpha \Big\}$ and $S_{2} = \Big\{(x, y)\in R^{2}_{+}: x\geq {\rm VaR}_{T_{I}^{f}}(\alpha)\ {\rm and} \ y\geq {\rm VaR}_{T_{R}^{f}}(\alpha) \Big\}$. First, it is easy to see that $S_{1}\subseteq S_{2}$. Second, note that

then for any $(x, y)\in S_{2}$, we have $G^{f}(x, y)\geq G^{f}({\rm VaR}_{T_{I}^{f}}(\alpha), {\rm VaR}_{T_{R}^{f}}(\alpha))\geq \alpha$, thus we get $S_{2}\subseteq S_{1}$.

Similarly, let $D_{1} = \Big\{(x, y)\in R^{2}_{+}: \overline{G}^{f}(x, y)\leq 1-\alpha \Big\}$ and $D_{2} = \Big\{(x, y)\in R^{2}_{+}: x\geq {\rm VaR}_{T_{I}^{f}}(\alpha)\ {\rm or} \ y\geq {\rm VaR}_{T_{R}^{f}}(\alpha) \Big\}$. For any $(x, y)\in D_{2}$, if $x\geq {\rm VaR}_{T_{I}^{f}}(\alpha)$, then

By the same arguments, we know that if $y\geq {\rm VaR}_{T_{R}^{f}}(\alpha)$, then $\overline{G}^{f}(x, y)\leq 1-\alpha$ holds as well. Hence, $D_{2}\subseteq D_{1}$.

On the other hand, for any $(x, y)\in \overline{D}_{2}$, we have $x < {\rm VaR}_{T_{I}^{f}}(\alpha)$ and $y < {\rm VaR}_{T_{R}^{f}}(\alpha)$. Since $T_{I}^{f}$ and $T_{R}^{f}$ are co-monotonic, we have

Notice that for any random variable $Y$, if $y < {\rm VaR}_{Y}(\alpha)$, we get $\mathbb{P}\{Y\leq y\} < \alpha$. (Otherwise, suppose $\mathbb{P}\{Y\leq y\}\geq \alpha$, then from the definition of VaR, we get $y\geq{\rm VaR}_{Y}(\alpha)$.) Then, we have

which implies $\overline{G}^{f}(x, y) > 1-\alpha$. Therefore, we have $\overline{D}_{2}\subseteq \overline{D}_{1}$, and hence $D_{2} = D_{1}$.

The binary lower-orthant value-at-risk ${\rm \underline{VaR}}^{f}(\alpha)$ and the binary upper-orthant value-at-risk ${\rm \overline{VaR}}^{f}(\alpha)$ are illustrated in Figures 1 and 2.

Note that the joint VaR $\big({\rm VaR}_{T_{I}^{f}}(\alpha), {\rm VaR}_{T_{R}^{f}}(\alpha)\big)$ determines ${\rm \underline{VaR}}^{f}_{\alpha}$ and ${\rm \overline{VaR}}^{f}_{\alpha}$. From both the insurer and the reinsurer's point of view, maximum agammaegate cost $\textbf{T}^{f}$ which can occur with some given probability is the smaller the better, that is to say, ${\rm \underline{VaR}}^{f}_{\alpha}$ and ${\rm \overline{VaR}}^{f}_{\alpha}$ are closer to the origin the better. This motivates us to consider the loss function

and the optimization criteria for seeking the optimal reinsurance contract:

In the rest of the paper, we will derive the optimal solutions corresponding to the reinsurance model (2.14) under the admissible ceded loss function sets $\mathcal{F}^i, i = 1, 2, 3.$

3.   Optimal reinsurance under loss function

In this section, we consider the general reinsurance premium principles which satisfy the following two properties:

1. Risk loading: $\Pi(X)\geq E[X]$;

2. Stop-loss ordering preserving: $\Pi(Y)\leq \Pi(X)$ if $Y$ is smaller than $X$ in the stop-loss order ($Y\leq_{sl}X$)$^{2}$^†.

^†$^{2}$ A random variable $Y$ is said to be smaller than a random variable $X$ in the stop-loss order sense, notation $Y\leq_{sl}X$, if and only if $Y$ has lower stop-loss premiums than $X$: $E(Y-d)_{+}\leq E(X-d)_{+}, \ -\infty < d < +\infty$.

We emphasize that there are many premium principles which satisfy these two properties, such as expectation principle, $p-$mean value principle, Dutch principle, Wang's principle and exponential principle.

3.1.   Optimal reinsurance form with $\mathcal{F}^1$ constraint

In this subsection, we derive the optimal reinsurance policies under the condition that the ceded loss function $f \in \mathcal{F}^1$. First, we define a ceded loss function set $\mathcal{H}^{1}$, which consists of all ceded loss functions $h(x) = b(x-d)_{+}$ with $0\leq b\leq 1$ and $d\geq 0$. Note that $\mathcal{H}^{1}$ is a subclass of $\mathcal{F}^1$. Second, we show that the optimal ceded loss functions which minimize the loss function in the subclass $\mathcal{H}^1$ also optimally minimize the loss function in $\mathcal{F}^1$. We give the following proposition using the geometric method proposed by ^[11].

Proposition 3.1. For any $f\in \mathcal{F}^{1}$, there always exists a function $h\in \mathcal{H}^{1}$ such that $L(h)\leq L(f)$.

Proof. If $f\in \mathcal{F}^{1}$ is identically zero on $[0, {\rm VaR}_{X}(\alpha)]$, we consider $h: = 0 \in \mathcal{H}^{1}$. It is easy to see that $h(X)\leq f(X)$ in the usual stochastic order. It further leads to $h(X)\leq_{sl} f(X)$ according to the theory of stochastic orders in ^[31]. Then we have $\Pi_{h} (X)\leq \Pi_{f} (X)$. Consequently, from formulas $(2.8)$ and $(2.9)$, we obtain

and

Hence we have $L(h)\leq L(f)$.

If $f\in \mathcal{F}^1$ is not identically zero on $[0, {\rm VaR}_{X}(\alpha)]$, let $f'_{-}({\rm VaR}_{X}(\alpha))$ and $f'_{+}({\rm VaR}_{X}(\alpha))$ be the left-hand derivative and right-hand derivative of $f$ at ${\rm VaR}_{X}(\alpha)$. Let $b$ be an any number in $\big[f'_{-}({\rm VaR}_{X}(\alpha)), f'_{+}({\rm VaR}_{X}(\alpha))\big]$, then we have $0 < b\leq 1$. Let $d = {\rm VaR}_{X}(\alpha)-\frac{f({\rm VaR}_{X}(\alpha))}{b}$, define $h(x) = b(x-d)_{+}, x\geq 0$. Then $h\in \mathcal{H}^{1}$, $f({\rm VaR}_{X}(\alpha)) = h({\rm VaR}_{X}(\alpha))$ and $f\geq h$ for any $x\geq 0$ since $f$ is convex. Hence we have

and

Therefore, $L(h)\leq L(f)$ holds. The geometric interpretation of this proof can be seen from Figure 3.

Based on the Proposition 3.1, we know that change-loss reinsurance of the form $f(x) = b(x-d)_{+}$ with $0\leq b\leq 1$ and $d\geq 0$ is optimal among $\mathcal{F}^1$ in the sense that it minimizes the loss function $L(f)$. The optimal parameters $b^*$ and $d^*$ will be given under some specific reinsurance premium principles in the rest of sections.

3.2.   Optimal reinsurance form with $\mathcal{F}^2$ constraint

In this subsection, we focus on the loss function minimization model for any ceded loss function $f\in \mathcal{F}^2$. As shown in ^[12], the ceded loss function $f\in \mathcal{F}^2$ is Lipschitz continuous, i.e.,

Let $\mathcal{H}^{2}$ denote the class of ceded loss function with the following representation $h(x) = (x-a)_+-(x-{\rm VaR}_{X}(\alpha))_+$, $a\leqslant VaR_\alpha(X)$. It is easy to see that $\mathcal{H}^{2}$ is a subclass of $\mathcal{F}^2$. Exactly, $h(x)$ is a layer reinsurance with deductible $a$ and upper limit ${\rm VaR}_{X}(\alpha)$. We will prove that the optimal functions which minimize the loss function in the subclass $\mathcal{H}^{2}$ also optimally minimize the loss function in $\mathcal{F}^2$.

Proposition 3.2. Let $f \in \mathcal{F}^2$ be a ceded function. There always exists a function $h \in \mathcal{H}^{2}$ such that $L(h)\leqslant L(f)$.

Proof. For any $f\in \mathcal{F}^{2}$, define $a = {\rm VaR}_{X}(\alpha)-f({\rm VaR}_{X}(\alpha))\geqslant 0$ and $h(x) = (x-a)_+-(x-{\rm VaR}_{X}(\alpha))_+ = min\{(x-({\rm VaR}_{X}(\alpha)-f({\rm VaR}_{X}(\alpha))))_+, f({\rm VaR}_{X}(\alpha)\}$, $x\geqslant 0$. Then we have $h\in \mathcal{H}^{2}$ and $f({\rm VaR}_{X}(\alpha)) = h({\rm VaR}_{X}(\alpha))$.

Furthermore, recall that the ceded loss function $f \in \mathcal{F}^2$ is non-negative and Lipschitz continuous, hence inequality $f(x)\geqslant (x+f({\rm VaR}_{X}(\alpha))-{\rm VaR}_{X}(\alpha))_+$ holds for $x \in [0, {\rm VaR}_{X}(\alpha)]$. On the other hand, the increasing property of $f(x)$ leads to $h(x) = f({\rm VaR}_{X}(\alpha))\leqslant f(x)$ for all $x > {\rm VaR}_{X}(\alpha)$. Thus, inequality $h(x)\leqslant f(x)$ holds for all $x\geqslant 0$. Since the reinsurance premium preserves stop-loss order, we have

and

Thus, $L(h)\leq L(f)$ holds. The geometric interpretation of this proof can be seen from Figure 4.

By Proposition 3.2, we know that layer reinsurance with deductible $a$ and upper limit ${\rm VaR}_{X}(\alpha)$ is optimal among $\mathcal{F}^2$ in the sense that it minimizes the loss function $L(f)$.

3.3.   Optimal reinsurance form with $\mathcal{F}^3$ constraint

In this subsection, we derive the optimal solution to problem of (2.14) over $\mathcal{F}^3$. Let $\mathcal{H}^{3}$ be the class of non-negative function $h(x)$ defined on $[0, \infty]$ with

where $0\leqslant c\leqslant 1$. Note that $\mathcal{H}^3\subset \mathcal{F}^3$ and $\mathcal{H}^3$ contains the null function $h(x) = 0$. The following result shows that the optimal ceded loss functions in $\mathcal{F}^3$ which minimize the $L(f)$, must take the form of (3.1).

Proposition 3.3. For any $f\in \mathcal{F}^3$, there always exists a function $h \in \mathcal{H}^{3}$, such that $L(h)\leqslant L(f)$.

Proof. For any $f\in \mathcal{F}^3$, let $c = \frac {f({\rm VaR}_{X}(\alpha))}{{\rm VaR}_{X}(\alpha)}$, then $c\in [0, 1]$. Define $h(x) = c(x-(x-{\rm VaR}_{X}(\alpha))_+)$, obviously $h\in \mathcal{H}^{3}$ and $h({\rm VaR}_{X}(\alpha)) = f({\rm VaR}_{X}(\alpha))$.

In addition, recall that the ceded loss function $f \in \mathcal{F}^3$ is increasing concave, hence $f(x)\geqslant \frac {f({\rm VaR}_{X}(\alpha))}{{\rm VaR}_{X}(\alpha)}x = h(x)$ for $x\in [0, {\rm VaR}_{X}(\alpha)]$. On the other hand, the increasing property of $f(x)$ leads to $h(x) = f({\rm VaR}_{X}(\alpha))\leqslant f(x)$ for $x > {\rm VaR}_{X}(\alpha)$. Since the reinsurance premium preserves stop-loss order, we have

and

Thus, we have $L(h)\leqslant L(f)$. The geometric interpretation of this proof can be seen from Figure 5.

From Proposition 3.3, we know that the quota-share reinsurance with a policy limit is always optimal among $\mathcal{F}^3$ in the sense that it minimizes the loss function $L(f)$.

4.   Optimal reinsurance policies under expectation premium principle

In this section, we consider the expectation reinsurance premium principle, i.e.,

where, $\theta > 0$ is the safety loading.

4.1.   Optimal reinsurance policies among $\mathcal{F}^1$

As a result of Proposition $3.1$, we can deduce optimal ceded loss functions by confining attention to $\mathcal{H}^{1}$. For a change-loss reinsurance with $b\in [0, 1]$ and $d\in [0, \infty)$, the total costs of the insurer and the reinsurer are

where, $\Pi_{E}(b, d) = (1+\theta)E[b(X-d)_{+}] = (1+\theta)b\int_{d}^{\infty}S(x){\rm d}x$ is the reinsurance premium. Then the VaR of $T_{I}^{b, d}$ and $T_{R}^{b, d}$ at confidence level $\alpha$ are

Hence, the loss function is

Lemma 4.1. Optimal ceded functions which minimize the loss function $L_{E}(b, d)$ in the class $\mathcal{H}^1$ exist.

Proof. Note that the function $\Pi_{E}(b, d)$ is an increasing function with respect to $b$. Then the loss function $L_{E}(b, d)$ attains its minimum value over $[0, 1]\times({\rm VaR}_{X}(\alpha), \infty)$ at $b = 0$ (the ceded function is $h(x)\equiv 0$) and the minimum value is ${\rm VaR}_{X}(\alpha)$. Hence, the study of optimal ceded functions which minimize the loss function $L_{E}(b, d)$ in the class $\mathcal{H}^1$ is simplified to solving the two-parameter minimization problem over closed subset $[0, 1]\times [0, {\rm VaR}_{X}(\alpha)]$. Since $L_{E}(b, d)$ is continuous, then the minimum of $L_{E}(b, d)$ over $[0, 1]\times [0, {\rm VaR}_{X}(\alpha)]$ must attain at some stationary point or lie on the boundary.

First, we define $\mathcal{A}\triangleq [0, 1]\times[0, {\rm VaR}_{X}(\alpha)]$. In this subsection, we will identify the minimum points of $L_{E}(b, d)$ over $\mathcal{A}$ and discuss the optimal ceded function $f_{1}^{*}(x)$. We split $\mathcal{A}$ into five disjoint subsets, i.e. $\mathcal{A} = \bigcup_{i = 1}^{5}A_{i}$, where, $A_{1} = \{(0, d): 0\leq d\leq {\rm VaR}_{X}(\alpha)\}$, $A_{2} = \{(b, d): 0 < b < 1, 0 < d < {\rm VaR}_{X}(\alpha)\}$, $A_{3} = \{(1, d): 0 < d < {\rm VaR}_{X}(\alpha)\}$, $A_{4} = \{(b, 0): 0 < b\leqslant 1\}$ and $A_{5} = \{(b, {\rm VaR}_{X}(\alpha)): 0 < b\leq 1\}$.

If $(b, d)\in \mathcal{A}$, the loss function is

Let $H_{E}(b, d) = L_{E}^{2}(b, d) = \big[(1-b){\rm VaR}_{X}(\alpha)+bd+\Pi_{E}(b, d)\big]^{2}+\big[b({\rm VaR}_{X}(\alpha)-d)\big]^{2}$, then $H_{E}(b, d)$ and $L_{E}(b, d)$ have the same minimum points. Thus, we will study the minimization problem of $H_{E}(b, d)$ on $\mathcal{A}$ in the rest of this subsection. Note that $H_{E}(b, d)$ is differentiable with partial derivatives

where, $g(d) = d+(1+\theta)\int_{d}^{\infty}S(x){\rm d}x$.

Next, we divide the following analysis into five cases.

● First, we demonstrate that $H_{E}(b, d)$ has no minimum points on $A_{5}$. For any $(b, d)\in A_{5}$, $H_{E}(b, d) > [{\rm VaR}_{X}(\alpha)]^{2} = H_{E}(0, d) = min_{A_{1}} H_{E}(b, d)$, then the minimum value of $H_{E}(b, d)$ over $\mathcal{A}$ is not attainable in $A_{5}$.

● The minimum points of $H_{E}(b, d)$ are located in $A_{1}$ if and only if

In fact, if inequality (4.7) holds, then it follows from the expression of $\frac{\partial H_{E}(b, d)}{\partial b}$ in (4.6) that $\frac{\partial H_{E}(b, d)}{\partial b} > 0$. Thus, $H_{E}(b, d)$ is strictly increasing with respect to $b$. Furthermore, for any $d \in [0, {\rm VaR}_{X}(\alpha)]$, $H_{E}(0, d)\equiv [{\rm VaR}_{X}(\alpha)]^{2}$. As a result, the minimum value of $H_{E}(b, d)$ over $\mathcal{A}$ is attained at any point $(0, d)$ in $A_{1}$.

Conversely, if $min_{[0, {\rm VaR}_{X}(\alpha)]}g(d) < {\rm VaR}_{X}(\alpha)$, then there exists a $\tilde{d} \in [0, {\rm VaR}_{X}(\alpha)]$ such that $\frac{\partial H_{E}(b, \tilde{d})}{\partial b} < 0$ holds in a right neighborhood of $b = 0$. That is to say, $(0, \tilde{d})$ is not a minimum point of $H_{E}(b, d)$. Since $H_{E}(0, d) = H_{E}(0, \tilde{d})\equiv [{\rm VaR}_{X}(\alpha)]^{2}$ for any $(0, d)\in A_{1}$, then no minimum points of $H_{E}(b, d)$ are located in $A_{1}$.

● If $(b^{*}, d^{*})\in A_{2}$ is a minimum point of $H_{E}(b, d)$, then $(b^{*}, d^{*})$ is a stationary point of $H_{E}(b, d)$. Therefore, we have

By straightforward algebra, we know that $d^{*}$ is a root of equation $q(d) = 0$, where

Substituting $d^{*}$ in the second equation of $(4.8)$ yields

Furthermore, $b^{*}$ must lie in $(0, 1)$, which is equivalent to

where, the function $p(d)$ is given by $p(d) = {\rm VaR}_{X}(\alpha)-d-g(d)g'(d)$.

● If $(1, \bar{d})\in A_{3}$ is a minimum point of $H_{E}(b, d)$, then Fermat's theorem implies

which is equivalent to

● If $(\bar{b}, 0)\in A_{4}$ is a minimum point of $H_{E}(b, d)$, then $\bar{b}$ must satisfy the following conditions

From (4.14), we yield

and

Based on the above arguments, we analyze the conditions for the minimum points of $H_{E}(b, d)$ are located in sets $A_{i}, \ i = 1, 2, 3, 4$. The results are summarized in the following theorem.

Theorem 4.1. The optimal solutions to reinsurance problem (2.14) are given as follows.

(1) If one of the three conditions (C1)-(C3) holds, then the optimal ceded loss function is given by $f_{1}^{*}(x) = 0$, where, $d_{0} = S^{-1}(\frac{1}{1+\theta})$ and

(2) If condition (C4) or (C5) holds, then the optimal ceded loss function is given by $f_{1}^{*}(x) = b^{*}(x-d^{*})_{+}$, where $d^{*}$ is the unique solution of equation $q(d) = 0$, $b^{*}$ is given by (4.10) and

(3) If condition (C6) or (C7) holds, then the optimal ceded loss function is given by $f_{1}^{*}(x) = (x-\bar{d})_{+}$, where $\bar{d}$ is the unique solution of equation $p(d) = 0$ and

(4)If condition (C8) holds, then the optimal ceded loss function is given by $f_{1}^{*}(x) = \bar{b}x$, where $\bar{b}$ is given by (4.15) and

Proof. (1) If one of the three conditions (C1)–(C3) holds, it is easy to show that

Then the minimum points of $H_{E}(b, d)$ are located in $A_{1}$. That is to say, the optimal ceded loss function is $f_{1}^{*}(x) = 0$.

(2) If condition (C4) holds, then $g'(d) < 0$ for any $d\in[0, d_{0})$. From the expression of $\frac{\partial H_{E}(b, d)}{\partial d}$ in (4.6), we have $\frac{\partial H_{E}(b, d)}{\partial d} < 0$ for any $(b, d)\in (0, 1]\times [0, d_{0}]$. Thus, the minimum points are not located in $[0, 1]\times [0, d_{0}]$. Furthermore, let $d_{1} > d_{0}$ such that $g(d_{1}) = {\rm VaR}_{X}(\alpha)$, from the expression of $\frac{\partial H_{E}(b, d)}{\partial b}$ in (4.6), we have $\frac{\partial H_{E}(b, d)}{\partial b} > 0$ for any $(b, d)\in (0, 1]\times [d_{1}, {\rm VaR}_{X}(\alpha)]$. Thus, the minimum points are also not located in $[0, 1]\times [d_{1}, {\rm VaR}_{X}(\alpha)]$. As a result, the minimum points of $H_{E}(b, d)$ over $\mathcal{A}$ are located in $(0, 1]\times (d_{0}, d_{1})$ and the minimum must be attainable at some stationary point $(b^{*}, d^{*})$ or must lie on the right boundary at some point $(1, \bar{d})$. Note that $q'(d) = S'(d)\big({\rm VaR}_{X}(\alpha)-d\big) < 0$, $q(d_{0}) = S(d_{0})({\rm VaR}_{X}(\alpha)-d_{0})-\int_{d_{0}}^{\infty}S(x)dx = \frac{1}{1+\theta}({\rm VaR}_{X}(\alpha)-g(d_{0})) > 0$ and $q(d_{1}) = S(d_{1})({\rm VaR}_{X}(\alpha)-d_{1})-\int_{d_{1}}^{\infty}S(x)dx = [(1+\theta)S(d_{1})-1]\int_{d_{1}}^{\infty}S(x)dx < 0$. Thus, the equation $q(d) = 0$ has a unique solution $d^{*}$ in $(d_{0}, d_{1})$. Substituting $d^{*}$ in the second equation of $(4.8)$ yields

It is easy to show $0 < b^{*} < 1$ since $p(d^{*}) > 0$. Thus $H_{E}(b, d)$ has a unique stationary point $(b^{*}, d^{*})$. In the following, we show that $H_{E}(b, d)$ attains the minimum at the stationary point $(b^{*}, d^{*})$.

Conversely, we suppose that the minimum of $H_{E}(b, d)$ is attainable at $(1, \bar{d})$ if condition (C4) holds. Then we have $p(\bar{d}) = 0$ and $g(\bar{d})[g(\bar{d})-{\rm VaR}_{X}(\alpha)]+[{\rm VaR}_{X}(\alpha)-\bar{d}]^{2}\leq 0$. Since $g'(\bar{d}) > 0$, we yield $[g(\bar{d})[g(\bar{d})-{\rm VaR}_{X}(\alpha)]+[{\rm VaR}_{X}(\alpha)-\bar{d}]^{2}]g'(\bar{d})\leq 0$. Straightforward algebra leads to $q(\bar{d})\geq 0$. Note that $q'(d) < 0$ and $q(d^{*}) = 0$, then we have $\bar{d}\leq d^{*}$. However, since $p'(d) < 0$, $p(d^{*}) > 0$ and $p(\bar{d}) = 0$, then we have $d^{*} < \bar{d}$. This leads to contradictions. Thus, if condition (C4) holds, the function $H_{E}(b, d)$ attains the minimum at the stationary point $(b^{*}, d^{*})$, that is to say, the optimal ceded loss function is $f_{1}^{*}(x) = b^{*}(x-d^{*})_{+}$.

If condition (C5) holds, then $\frac{\partial H_{E}(b, d)}{\partial b} > 0$ for any $(b, d)\in (0, 1]\times [d_{1}, {\rm VaR}_{X}(\alpha)]$. Thus, the minimum points are not located in $[0, 1]\times [d_{1}, {\rm VaR}_{X}(\alpha)]$. As a result, the minimum points of $H_{E}(b, d)$ over $\mathcal{A}$ are located in $(0, 1]\times [0, d_{1})$ and the minimum must be attainable at some stationary point $(b^{*}, d^{*})$ or must lie on the right boundary at some point $(1, \bar{d})$ or must lie on the lower boundary at some point $(\bar{b}, 0)$. In the following, we consider equation $q(d) = 0$. Note that $q'(d) < 0$, $q(0) = S(0){\rm VaR}_{X}(\alpha)-\mu > 0$ and

Thus, the equation $q(d) = 0$ has a unique solution $d^{*}$ in $(0, d_{1})$. Further, we know that $H_{E}(b, d)$ has a unique stationary point $(b^{*}, d^{*})$ if condition (C5) holds. By the same argument as above, we show that the minimum of $H_{E}(b, d)$ is not attainable at $(1, \bar{d})$ if $p(d^{*}) > 0$ holds. Meanwhile, we demonstrate that the minimum of $H_{E}(b, d)$ is not attainable at $(\bar{b}, 0)$ if condition (C5) holds. Conversely, we suppose that the minimum value of $H_{E}(b, d)$ is attainable at $(\bar{b}, 0)$ if condition (C5) holds. Then we have conditions (4.15) and (4.16) hold. Substituting (4.15) into (4.16), we get $\mu-S(0){\rm VaR}_{X}(\alpha)\geq 0$, that is contradicted to the second inequality of condition (C5). Thus, if condition (C5) holds, the function $H_{E}(b, d)$ attains the minimum at the stationary point $(b^{*}, d^{*})$.

In summary, if condition (C4) or (C5) holds, the optimal ceded loss function is given by $f_{1}^{*}(x) = b^{*}(x-d^{*})_{+}$.

(3) If condition (C6) or (C7) holds, from the above arguments in (2), we know that $H_{E}(b, d)$ has no stationary points because $p(d^{*})\leq 0$. Furthermore, if the second inequality of (C7) holds, the minimum value of $H_{E}(b, d)$ is not attainable at $(\bar{b}, 0)$. Thus, the function $H_{E}(b, d)$ attains the minimum at the boundary point $(1, \bar{d})$ if condition (C6) or (C7) holds, that is to say, the optimal ceded loss function is given by $f_{1}^{*}(x) = (x-\bar{d})_{+}$.

(4) If condition(C8) holds, then $\frac{\partial H_{E}(b, d)}{\partial b} > 0$ for any $(b, d)\in (0, 1]\times [d_{1}, {\rm VaR}_{X}(\alpha)]$. Thus, the minimum points are not located in $[0, 1]\times [d_{1}, {\rm VaR}_{X}(\alpha)]$. As a result, the minimum points of $H_{E}(b, d)$ over $\mathcal{A}$ are located in $(0, 1]\times [0, d_{1})$ and the minimum must be attainable at some stationary point $(b^{*}, d^{*})$ or must lie on the right boundary at some point $(1, \bar{d})$ or must lie on the lower boundary at some point $(\bar{b}, 0)$. In the following, we consider equation $q(d) = 0$. Note that $q(0) = S(0){\rm VaR}_{X}(\alpha)-\mu \leq 0$ and $q'(d) < 0$, then the equation $q(d) = 0$ has no solutions in $(0, d_{1})$, namely, the function $H_{E}(b, d)$ has no stationary points. Thus, the minimum point of $H_{E}(b, d)$ over $\mathcal{A}$ must lie on the right boundary at some point $(1, \bar{d})$ or must lie on the lower boundary at some point $(\bar{b}, 0)$. If the minimum of $H_{E}(b, d)$ is attainable at $(1, \bar{d})$, the we have conditions (4.13) hold. Since $g'(\bar{d}) > 0$, we yield $[g(\bar{d})[g(\bar{d})-{\rm VaR}_{X}(\alpha)]+[{\rm VaR}_{X}(\alpha)-\bar{d}]^{2}]g'(\bar{d})\leq 0$. Straightforward algebra leads to $q(\bar{d})\geq 0$. This is contradicted to $q(0) \leq 0$ and $q'(d) < 0$. Thus, minimum point of $H_{E}(b, d)$ over $\mathcal{A}$ must lie on the lower boundary at point $(\bar{b}, 0)$, namely, the optimal ceded loss function is given by $f_{1}^{*}(x) = \bar{b}x$.

4.2.   Optimal reinsurance policies among $\mathcal{F}^2$

As a result of Proposition $3.2$, we can deduce optimal ceded loss functions by confining attention to $\mathcal{H}^{2}$. For a layer reinsurance policy $h(x) = (x-a)_+-(x-{\rm VaR}_{X}(\alpha))_+$ with $a \in [0, {\rm VaR}_{X}(\alpha)]$, the total costs of the insurer and the reinsurer under the VaR risk measure are

Hence, the loss function is

Theorem 4.2. The optimal ceded loss function that solves (2.14) with $\mathcal{F}^2$ constraint is given by

where $a_{1}^*$ is the unique solution of equation $(4.19)$

Proof. Let $H_{E}(a) = [a+(1+\theta)\int^{{\rm VaR}_{X}(\alpha)}_aS(x)dx]^2+[{\rm VaR}_{X}(\alpha)-a]^2$, then

If $\alpha \leqslant \frac{\theta}{1+\theta}$ holds, it is easy to show that $H_{E}'({\rm VaR}_{X}(\alpha))\leqslant 0$. According to (4.20) and (4.21), then $H_{E}(a)$ and $L_{E}(a)$ attain their minimum at $a = {\rm VaR}_{X}(\alpha)$. In this case, $f_{2}^{*}(x)\equiv 0$.

If $\alpha > \frac{\theta}{1+\theta}$ holds, then from (4.19) and (4.21), we have

Recall that $0\leqslant a \leqslant {\rm VaR}_{X}(\alpha)$ and $H_{E}'(0) = 2(1+\theta)\int^{{\rm VaR}_{X}(\alpha)}_0S(x)dx(1-(1+\theta)S(0))- 2{\rm VaR}_{X}(\alpha)$. If $1-(1+\theta)S(0)\leqslant 0$, then $H_{E}'(0) < 0$ and if $1-(1+\theta)S(0) > 0$, then $H_{E}'(0)\leqslant 2(1+\theta)\int^{{\rm VaR}_{X}(\alpha)}_0S(0)dx- 2{\rm VaR}_{X}(\alpha) < 0$. So $H_{E}'(0) < 0$ and $H_{E}'({\rm VaR}_{X}(\alpha)) > 0$ imply that $a_{1}^*$ exists and is the only minimum point of $H_{E}(a)$ and $L_{E}(a)$.

4.3.   Optimal reinsurance policies among $\mathcal{F}^3$

As a result of Proposition $3.3$, we can deduce optimal ceded loss functions by confining attention to $\mathcal{H}^{3}$. For a quota-share reinsurance with a policy limit $h(x) = c(x-(x-{\rm VaR}_{X}(\alpha))_+)$, the total costs of the insurer and the reinsurer under the VaR risk measure are

Hence, the loss function is

Theorem 4.3. The optimal ceded loss function that solves (2.14) with $\mathcal{F}^3$ constraint is given by

where $\phi({\rm VaR}_{X}(\alpha)) = (1+\theta)\int^{{\rm VaR}_{X}(\alpha)}_0S(x)dx-{\rm VaR}_{X}(\alpha)$ and $c_{1}^* = \frac {-\phi ({\rm VaR}_{X}(\alpha)){\rm VaR}_{X}(\alpha)}{({\rm VaR}_{X}(\alpha))^2+(\phi ({\rm VaR}_{X}(\alpha))^2}$.

Proof. Let $H_{E}(c) = [(1-c){\rm VaR}_{X}(\alpha)+(1+\theta)c\int^{{\rm VaR}_{X}(\alpha)}_0S(x)dx]^2+[c{\rm VaR}_{X}(\alpha)]^2$, then

If $\phi({\rm VaR}_{X}(\alpha))\geqslant 0$, according to (4.23), we have $\frac {\partial H_{E}(c)}{\partial c}\geqslant 0$. Thus, $L_{E}(c)$ attains its minimum at $c = 0$. Therefore, the optimal ceded loss function is given by $f_{3}^{*}(x) = 0$.

If $\phi({\rm VaR}_{X}(\alpha)) < 0$, according to (4.23) and (4.24), $L_{E}(c)$ attains its minimum at $c = c_{1}^*$. Thus, the optimal ceded loss function is given by $f_{3}^{*}(x) = c_{1}^*(x-(x-{\rm VaR}_{X}(\alpha))_+)$.

5.   Optimal reinsurance policies under Dutch premium principle

In this section, we determine the optimal reinsurance policies among the ceded loss function sets $\mathcal{F}^i (i = 1, 2, 3)$ under the Dutch premium principle. The Dutch premium principle is given by

where, $0 < \beta \leqslant 1$.

5.1.   Optimal reinsurance policies among $\mathcal{F}^1$

From Proposition $2$, we know that the optimal ceded loss function in $\mathcal{F}^1$ can be determined by confining attention to $\mathcal{H}^{1}$. For a change-loss reinsurance with $b\in [0, 1]$ and $d\in [0, \infty)$, the total costs of the insurer and the reinsurer under the Dutch premium principle are

where $\Pi_{D}(b, d) = b\int^{\infty}_dS(x)+\beta b\int^{\infty}_{d+\int^{\infty}_dS(x)dx}S(x)dx$ is the reinsurance premium. Then the VaR of $T_{I}^{b, d}$ and $T_{R}^{b, d}$ at confidence level $\alpha$ are

Hence, the loss function is

Let $H_{D}(b, d) = [(1-b){\rm VaR}_{X}(\alpha)+bd+\Pi_{D}(b, d)\big]^{2}+\big[b({\rm VaR}_{X}(\alpha)-d)]^{2}$, then

where, $k(d) = d+\int^{\infty}_d S(x)dx+\beta\int^{\infty}_{d+\int^{\infty}_dS(x)dx}S(x)dx$.

Theorem 5.1. The optimal ceded loss function to reinsurance problem (2.14) is given as follows.

(1) If condition (M1) holds, then the optimal ceded loss function is given by $f_{4}^{*}(x) = 0$, where

(2) If condition (M2) holds, then the optimal ceded loss function is given by $f_{4}^{*}(x) = b^{*}(x-d^{*})_{+}$, where,

$b^{*} = \frac{{\rm VaR}_{X}(\alpha)k'(d^{*})}{{\rm VaR}_{X}(\alpha)-d^{*}+[{\rm VaR}_{X}(\alpha)-k(d^{*})]k'(d^{*})}$, $d^{*}$ is the unique solution of equation $u(d) = 0$ and $u(d) = {\rm VaR}_{X}(\alpha)-k(d)-k'(d)({\rm VaR}_{X}(\alpha)-d)$, $v(d) = {\rm VaR}_{X}(\alpha)-d-k(d)k'(d)$.

(3) If condition (M3)holds, then the optimal ceded loss function is given by $f_{4}^{*}(x) = (x-\bar{d})_{+}$, where $\bar{d}$ is the unique solution of equation $v(d) = 0$ and

(4)If condition (M4)holds, then the optimal ceded loss function is given by $f_{4}^{*}(x) = \bar{b}x$, where $\bar{b} = \frac{{\rm VaR}_{X}(\alpha)[{\rm VaR}_{X}(\alpha)-k(0)]}{[{\rm VaR}_{X}(\alpha)-k(0)]^{2}+[{\rm VaR}_{X}(\alpha)]^{2}}$ and

Proof. Similarly to the proof of Lemma 4.1, the function $\Pi_{D}(b, d)$ is an increasing function with respect to $b$. Then the study of optimal ceded functions which minimize the loss function $L_{D}(b, d)$ in the class $\mathcal{H}^{1}$ is simplified to solving the two-parameter minimization problem over closed subset $[0, 1]\times [0, {\rm VaR}_{X}(\alpha)]$. Since $L_{D}(b, d)$ is continuous, then the minimum of $L_{D}(b, d)$ over $[0, 1]\times [0, {\rm VaR}_{X}(\alpha)]$ must attain at some stationary point or lie on the boundary.

(1) Note that the function $k(d)$ is an increasing function. If condition (M1) holds, it is easy to show that

Then from the expression of $\frac{\partial H_{D}(b, d)}{\partial b}$ in (5.4), we know that $H_{D}(b, d)$ is an increasing function with respect to $b$. Thus the minimum points of $H_{D}(b, d)$ are located in $A_{1}$.

Conversely, if condition (M1) does not hold, then there exists a $\tilde{d} \in [0, {\rm VaR}_{X}(\alpha)]$ such that $\frac{\partial H_{D}(b, \tilde{d})}{\partial b} < 0$ holds in a right neighborhood of $b = 0$. That is to say, $(0, \tilde{d})$ is not a minimum point of $H_{D}(b, d)$. Since $H_{D}(0, d) = H_{D}(0, \tilde{d})\equiv [{\rm VaR}_{X}(\alpha)]^{2}$ for any $(0, d)\in A_{1}$, then no minimum points of $H_{D}(b, d)$ are located in $A_{1}$.

That is to say, the minimum points of $H_{D}(b, d)$ are located in $A_{1}$ if and only if condition (M1) holds. In this case the optimal ceded loss function is $f_{4}^{*}(x) = 0$.

(2) We first consider the stationary points of $H_{D}(b, d)$. Let

By straightforward algebra, we obtain

If condition (M2) holds, then $u(0) > 0$ and $u({\rm VaR}_{X}(\alpha)) < 0$ hold. Since $u'(d)\leq 0$ for any $d\in[0, {\rm VaR}_{X}(\alpha)]$, then the equation $u(d) = 0$ has a unique root $d^{*}$ in $(0, {\rm VaR}_{X}(\alpha))$. Substituting $d^{*}$ in the second equation of $(5.6)$ yields

Since $v(d^{*}) > 0$, then we have $0 < b^{*} < 1$. Thus $H_{D}(b, d)$ has a unique stationary point $(b^{*}, d^{*})$. In the following, we show that $H_{D}(b, d)$ attains the minimum at the stationary point $(b^{*}, d^{*})$.

Conversely, if the minimum value of $H_{D}(b, d)$ is attainable at some point $(1, \bar{d})$ on the right boundary, then Fermat's theorem implies

which is equivalent to

Since $k'(\bar{d}) > 0$, we yield $[k(\bar{d})[k(\bar{d})-{\rm VaR}_{X}(\alpha)]+[{\rm VaR}_{X}(\alpha)-\bar{d}]^{2}]k'(\bar{d})\leq 0$. Straightforward algebra leads to $u(\bar{d})\geq 0$. Note that $u'(d) < 0$ and $u(d^{*}) = 0$, then we have $\bar{d}\leq d^{*}$. However, since $v'(d) < 0$, $v(d^{*}) > 0$ and $v(\bar{d}) = 0$, then we have $d^{*} < \bar{d}$. This leads to contradictions. Thus, if condition (M2) holds, the function $H_{D}(b, d)$ does not attain the minimum at the right boundary.

If the minimum value of $H_{D}(b, d)$ is attainable at some point $(\bar{b}, 0)$ on the lower boundary, then $\bar{b}$ must satisfy the following conditions

From (5.9), we yield

which means $u(0)\leq 0$, this is contradicted to the condition (M2).

In summary, if condition (M2) holds, the minimum of the function $H_{D}(b, d)$ must be attained at the unique stationary point $(b^{*}, d^{*})$, i.e., the optimal ceded loss function is given by $f_{4}^{*}(x) = b^{*}(x-d^{*})_{+}$.

(3) If condition (M3) holds, from the above arguments in (2), we know that $H_{D}(b, d)$ has no stationary points because $v(d^{*})\leq 0$ and $H_{E}(b, d)$ does not attain the minimum at $(\bar{b}, 0)$ because $u(0) > 0$. Thus, the function $H_{D}(b, d)$ attains the minimum at the boundary point $(1, \bar{d})$ if condition (M3) holds, that is to say, the optimal ceded loss function is given by $f_{4}^{*}(x) = (x-\bar{d})_{+}$.

(4) If condition (M4) holds, then the equation $u(d) = 0$ has no solutions in $(0, {\rm VaR}_{X}(\alpha))$, namely, the function $H_{D}(b, d)$ has no stationary points. Thus, the minimum point of $H_{D}(b, d)$ over $\mathcal{A}$ must lie on the right boundary at some point $(1, \bar{d})$ or must lie on the lower boundary at some point $(\bar{b}, 0)$. If the minimum of $H_{D}(b, d)$ is attainable at $(1, \bar{d})$, then the conditions in (5.7) hold. Since $k'(\bar{d}) > 0$, we yield $[k(\bar{d})[k(\bar{d})-{\rm VaR}_{X}(\alpha)]+[{\rm VaR}_{X}(\alpha)-\bar{d}]^{2}]k'(\bar{d})\leq 0$. Straightforward algebra leads to $u(\bar{d})\geq 0$. This is contradicted to $u(0) \leq 0$ and $u'(d) < 0$. Thus, the minimum point of $H_{E}(b, d)$ over $\mathcal{A}$ must lie on the lower boundary at point $(\bar{b}, 0)$, namely, the optimal ceded loss function is given by $f_{4}^{*}(x) = \bar{b}x$.

5.2.   Optimal reinsurance policies among $\mathcal{F}^2$

For a layer reinsurance with $a \in [0, {\rm VaR}_{X}(\alpha)]$, the total costs of the insurer and the reinsurer under the VaR risk measure are

where, $t(a) = a+\int^{{\rm VaR}_{X}(\alpha)}_aS(x)dx$. Hence, the loss function is

Theorem 5.2. The optimal ceded loss function that solves (2.14) with $\mathcal{F}^2$ constraint is given by

where $a_{2}^*$ is the unique solution of equation

Proof. Let $H_{D}(a) = L_{D}^2(a) = [t(a)+\beta \int^{{\rm VaR}_{X}(\alpha)}_{t(a)}S(x)dx]^2+[{\rm VaR}_{X}(\alpha)-a]^2$, then

From Eq (5.13), we know that

and

Hence, from (5.12) and (5.14), we have

Therefore, $a^*$ is the unique minimum point of $H_{D}(a)$. Since $L_{D}(a)$ and $H_{D}(a)$ have the same minimum points, then the optimal ceded loss function that solves (2.14) with $\mathcal{F}^2$ constraint is given by (5.11) and (5.12).

5.3.   Optimal reinsurance policies among $\mathcal{F}^3$

From Proposition $4$, we can deduce optimal ceded loss functions by confining attention to $\mathcal{H}^{3}$. For a quota-share reinsurance with a policy limit $h(x) = c(x-(x-{\rm VaR}_{X}(\alpha))_+)$, the total costs of the insurer and the reinsurer under the VaR risk measure are

Hence, the loss function is

Theorem 5.3. The optimal ceded loss function that solves (2.14) with $\mathcal{F}^3$ constraint is given by

where

and $\varphi({\rm VaR}_{X}(\alpha)) = \int^{{\rm VaR}_{X}(\alpha)}_0S(x)dx+\beta \int^{{\rm VaR}_{X}(\alpha)}_{t(0)}S(x)dx-{\rm VaR}_{X}(\alpha)$.

Proof. Let $H_{D}(c) = L_{D}^2(c) = [(1-c){\rm VaR}_{X}(\alpha)+ct(0)+\beta c\int^{{\rm VaR}_{X}(\alpha)}_{t(0)}S(x)dx]^2+[c{\rm VaR}_{X}(\alpha)]^2$, then $L_{D}(c)$ and $H_{D}(c)$ have the same minimum points. Taking the derivative of $H_{D}(c)$, we obtain

Note that

Then, according to (5.17), (5.18) and (5.19), $H_{D}(c)$ and $L_{D}(c)$ attain there minimum at $c = c_{2}^*$, where $c_{2}^* = \frac {-\varphi ({\rm VaR}_{X}(\alpha)){\rm VaR}_{X}(\alpha)}{({\rm VaR}_{X}(\alpha))^2+(\varphi ({\rm VaR}_{X}(\alpha))^2}\leq \frac{1}{2}.$

6.   Numerical examples

In this section, we construct four numerical examples to illustrate the optimal reinsurance policies that we derived in the previous sections. Let the confidence level $\alpha = 0.95$, safety loading parameters $\theta = 0.2$ and $\beta = 0.5$.

Example 6.1. Assume that the reinsurance premium is expectation premium principle and the loss variable $X$ has an exponential distribution with survival function $S(x) = e^{0.001x}$, then $F(0) = 0 < \frac{\theta}{1+\theta} = 0.1667 < \alpha = 0.95$, ${\rm VaR}_{X}(\alpha) = 2995.73 > 1182.32 = g(d_{0})$, $d^{*} = 1995.73$, $p(d^{*}) = -806.73$. By Theorem (4.1), Theorem (4.2) and Theorem (4.3), we know that the optimal ceded loss function among $\mathcal{F}^1$ is $f_{1}^{*} = (x-1599.90)_{+}$, the optimal ceded loss function among $\mathcal{F}^2$ is $f_{2}^{*} = (x-1622.55)_{+}-(x-2995.73)_{+}$ and the optimal ceded loss function among $\mathcal{F}^3$ is $f_{3}^{*} = 0.4477(x-(x-2995.73)_{+})$.

Example 6.2. Assume that the reinsurance premium is expectation premium principle and the loss variable $X$ has a Pareto distribution with survival function $S(x) = (\frac{2000}{x+2000})^{3}$, then $F(0) = 0 < \frac{\theta}{1+\theta} = 0.1667 < \alpha = 0.95$, ${\rm VaR}_{X}(\alpha) = 3428.84 > 1187.98 = g(d_{0})$, $d^{*} = 1619.22$, $p(d^{*}) = 226.05$. By Theorem (4.1), Theorem (4.2) and Theorem (4.3), we know that the optimal ceded loss function among $\mathcal{F}^1$ is $f_{1}^{*} = 0.9236(x-1619.22)_{+}$, the optimal ceded loss function among $\mathcal{F}^2$ is $f_{2}^{*} = (x-1801.98)_{+}-(x-3428.84)_{+}$ and the optimal ceded loss function among $\mathcal{F}^3$ is $f_{3}^{*} = 0.4692(x-(x-3428.84)_{+})$.

Example 6.3. Assume that the reinsurance premium is Dutch premium principle and the loss variable $X$ has an exponential distribution with survival function $S(x) = e^{0.001x}$, then ${\rm VaR}_{X}(\alpha) = 2995.73$, $\frac{{\rm VaR}_{X}(\alpha)-\mu}{\int^{\infty}_\mu S(x)dx} = 5.4250 > 0.5 = \beta$, $u(0) = 1811.79$, $d^{*} = 1950.79$, $v(d^{*}) = -689.40$. By Theorem (5.1), Theorem (5.2) and Theorem (5.3), we know that the optimal ceded loss function among $\mathcal{F}^1$ is $f_{4}^{*} = (x-1607.99)_{+}$, the optimal ceded loss function among $\mathcal{F}^2$ is $f_{5}^{*} = (x-2994.81)_{+}-(x-2995.73)_{+}$ and the optimal ceded loss function among $\mathcal{F}^3$ is $f_{6}^{*} = 0.4500(x-(x-2995.73)_{+})$.

Example 6.4. Assume that the reinsurance premium is Dutch premium principle and the loss variable $X$ as a Pareto distribution with survival function $S(x) = (\frac{2000}{x+2000})^{3}$, then ${\rm VaR}_{X}(\alpha) = 3428.84$, $\frac{{\rm VaR}_{X}(\alpha)-\mu}{\int^{\infty}_\mu S(x)dx} = 5.4649 > 0.5 = \beta$, $u(0) = 2206.61$, $d^{*} = 1525.01$, $v(d^{*}) = 397.65$. By Theorem (5.1), Theorem (5.2) and Theorem (5.3), we know that the optimal ceded loss function among $\mathcal{F}^1$ is $f_{4}^{*} = 0.8676(x-1525.01)_{+}$, the optimal ceded loss function among $\mathcal{F}^2$ is $f_{5}^{*} = (x-3427.91)_{+}-(x-3428.84)_{+}$ and the optimal ceded loss function among $\mathcal{F}^3$ is $f_{6}^{*} = 0.4690(x-(x-3428.84)_{+})$.

Remark 6.1. Note that the risks $X$ have the same mean and the parameters are same in the above four examples. For the exponential case, the optimal reinsurance policy is a stop-loss reinsurance when $f\in \mathcal{F}^1$, while for the Pareto case, the optimal reinsurance policy is a change-loss reinsurance when $f\in \mathcal{F}^1$. Therefore, the form of the optimal reinsurance policy depends on the distribution of loss variable $X$.

7.   Conclusions

The optimal reinsurance policies from the perspective of both the insurer and reinsurer have remained a fascinating topic in actuarial science. Many interesting optimal reinsurance models have been proposed. In contrast to the existing literatures, we provide two new findings to the optimal reinsurance models from both the insurer and reinsurer in this paper. First, we propose an optimization criterion to minimize their total costs under the criteria of loss function which is defined by the joint value-at-risk. Second, we extend the premium principle to a much wide class of premium principles satisfying two axioms: risk loading and stop-loss ordering preserving. Under these conditions, we derive the optimal reinsurance policies over three ceded loss function sets, (ⅰ) the change-loss reinsurance is optimal among the class of increasing convex ceded loss functions; (ⅱ) when the constraints on both ceded and retained loss functions are relaxed to increasing functions, the layer reinsurance is shown to be optimal; (ⅲ) the quota-share reinsurance with a limit is always optimal when the ceded loss functions are in the class of increasing concave functions. We further use the expectation premium principle and Dutch premium principle to illustrate the application of our results by deriving the optimal parameters.

We also wish to point out that further research on this topic is needed. First, for reinsurance, the challenges of classical insurance are amplified, particularly when it comes to dealing with extreme situations like large claims and rare events. We have to rethink classical models in order to cope successfully with the respective challenges. One of the better ways is to focus on modelling and statistics, related literature can be referred to ^[32,33]. Second, in most of optimal reinsurance problems, it is assumed that the distributions of the insurer's risks are known. However, in practice, only incomplete information on the distributions is available. How to obtain optimal reinsurance contracts with incomplete information is also an interesting topic. An attempt to such a problem is to use the statistical methods, see ^[34,35]. Third, although some papers have been devoted to deriving optimal reinsurance under model uncertainty, the optimal reinsurance with uncertainty still lacks of available analysis tools, maybe we can draw support from sub-linear expectation, for details, see ^[36,37]. We hope that these important open problems can be addressed in future research. We also believe that this article can and will foster further research in this direction.

Acknowledgments

The research was supported by Project of Shandong Province Higher Educational Science and Technology Program (J18KA249) and Social Science Planning Project of Shandong Province (20CTJJ02).

Conflict of interest

The authors declare that there is no conflict of interest.