Solving ANOVA problem with restricted Type Ⅰ and Type Ⅱ error rates

1.   Introduction

The problem of comparing the means of several normally distributed random variables has attracted researches' attention from the middle of the last century. Such problems arise in many practical applications such as when comparing treatment effects in clinical trials, or experimental studies to compare the effects of different experiments, or when comparing the failure rate of a machine with its age. In the framework of ANOVA, the problem can be formulated as follows ^[1]. Let us consider $k$ normal populations $N({\mu _i}, \sigma _i^2)$, $i = 1, ..., k$. Let ${X_{i1}}, ..., {X_{i{n_i}}}$ be a random sample from $N({\mu _i}, \sigma _i^2)$, independent from each other. The problem is to test hypotheses

In some studies, researchers may believe that the true means are simply ordered (${\mu _1} \leqslant ... \leqslant {\mu _k}$) before observing the data. For example, the treatment means may be a sequence of increasing dose levels of a drug. In this case, ANOVA takes the form

with at least one strict inequality. This statement has received considerable attention in the statistical literature (see, for example, ^[1,2,3,4,5]).

The simplest case $k = 2$, i.e., the case with only two normal populations, called the Behrens–Fisher problem, is well studied. For its solution, at statement (1), a simple and accurate method is introduced in ^[6,7], which gives good results in both cases whether variances are equal or not. For the arbitrary case; moreover, when $k > 2$, a compact but quite good review of the methods used to solve problem (1) is given in ^[8], on the basis that there is a procedure that gives satisfactory results by the Type Ⅰ error rate for all sample sizes, where $k$ and parameter configurations do not exist ^[8,9]. To improve the situation, a parametric bootstrap (PB) approach is offered in ^[8,9] and it is compared with the most promoted tests: The Welch test, the James (^[10]) second-order test, and the generalized F (GF) test ^[11,12]. The comparison was realized by Type Ⅰ error rate and powers using Monte Carlo simulation, showing that the best, based on the sample sizes, values of the error variances, and the number of means being compared is a PB test. A little worse is the James second-order test. The Welch test and the GF test have fairly poor Type Ⅰ errors when the value of $k$ is medium or large and/or the sample sizes are small. In computed examples of ^[8], the minimum value of Monte Carlo estimates of Type Ⅰ error rates is equal to 0.04 even when $k = 2$, meaning it is quite high. For other $k$, as a rule, it is greater.

For statement (2), a one-sided studentized range test is offered in Hayter ^[3]. It provides simultaneous one-sided lower confidence bounds for the ordered pairwise comparisons ${\mu _i} - {\mu _j}$, $1 \leqslant j < i \leqslant k$. Similar inference procedures are discussed in ^{[13,14,15,16,17]}. In some, the strong order of arranging the mathematical expectations is considered, and in others, the assumption that ${\mu _i}$'s follow a simple ordering is not made. The familywise error rate (FWER) is used as an optimality criterion in these cases, the computation of which is complicated for large $k$. To overcome this problem, the empirical Bayesian approach of multiple testing is offered in ^[1]. Its essence consists of the consideration of the families of hypotheses

from a Bayesian view point. To test (3), there is a new step-down multiple testing procedure, which is a step-down version of the single-step procedure ^[18]. It controls the familywise error rate better than a single-step procedure, which provides a simple comparison of each $p$-value with a single critical value. For this reason, the step-down version is more powerful than the single-step procedure.

The heart of the single-step procedure consists of the rejection of the ${H_{i0}}$ hypothesis if ${p_i} \leqslant \alpha$, where ${p_i}$ is ${H_{i0}}$ hypothesis corresponding $p$-value, and $\alpha \in (0, 1)$ is the significant level. In the step-down test, the differences ${T_i} = {\mu _{i + 1}} - {\mu _i}$ and the appropriate hypotheses are in ascending order. Testing hypothesis (3) begins with the largest $T$-value, i.e., with ${T_{(k - 1)}}$. If for some ${T_{(k - i)}}$ hypothesis ${H_{(k - i)0}}$ is accepted, then testing is stopped, and all the remaining hypotheses whose order number is less than $(k - i)$ are accepted; otherwise, ${H_{(k - i)0}}$ is rejected, and the $(i + 1)$ th step is transited. In ^[1], the critical constants for each stage chosen so that FWER is controlled at a pre-specified level $\alpha$. Because the empirical Bayesian approach is used in this method, which is based on the resampling techniques, it asymptotically controls the FWER.

An alternative method for testing hypothesis (2), similar to the parametric bootstrap method of ^[8] to improve the Type-Ⅰ error rate, is offered in ^[4]. In particular, this method is an extended Hayter's test ^[3] for heterogeneous sample sizes and variances, and uses the idea of the PB offered in ^[8], based on the simulation of the sample from the null distribution with unbiased estimators of the parameters. The non-significant improvement of Type Ⅰ error compared to existing methods of the Welch test (WT) ^[19]), the ${\bar E^2}$ test (which is a counterpart of the $F$ test in the ordered case ^[5]) and Chen's one-stage method (^[20]) for large sample sizes is shown through the modeling for the introduced novelty.

For justice, it should be said that all the considered tests have quite high Type Ⅰ and Type Ⅱ error rates, the values that depend on the parameters of the problem under consideration (the values of $k$, variances and sample sizes) and their control is impossible at a predetermined level. Therefore, to solve problems (1) and (2), we aim to develop a method that will enable us to make a decision by restricting Type Ⅰ and Type Ⅱ error rates to a predetermined level. To achieve this goal, the CBM is used below ^[21,22].

The structure of the rest of the paper is as follows. The application of CBM to solve the stated problem is given in Section 2, where formulas and criteria for testing individual and multiple Hypotheses defined by (6) are given. Algorithms for computation for known and unknown variances are developed in Section 3. Computation results for practical examples at known and unknown variances when the maximum likelihood ratio and Stein's Methods are used for overcoming the problem caused by complexity of hypotheses, are presented in Section 4. A short discussion of the obtained results is offered in Section 5, and a conclusion is presented in Section 6.

2.   Method of testing multiple directional hypotheses for solving the stated problem

Without loss of generality, we can suppose that all ${\mu _i}$ ($i = 1, ..., k$) are positive, because if this condition is not fulfilled, we can always achieve it by moving the origin.

Let us rewrite hypothesis (1) as follows

where ${\theta _i} = {\mu _{i + 1}} - {\mu _i}$, $i = 1, ..., k - 1$. In this case, hypothesis (2) takes the form

Using the concept of directional hypotheses ^[22,23,24], similarly to (3), hypothesis (4) can be considered as a set of directional hypotheses

Statement (6) enables us to test hypotheses (1) and (2) (which are the same as hypotheses (4) and (5)) simultaneously, using techniques developed in ^[22,23]. In particular: After testing hypothesis (6), if for even one value of $i$, the left sided hypothesis is accepted among accepted alternatives, accept alternative hypothesis in (4); and if all the accepted alternative hypotheses are right sided, accept alternative hypothesis in (5); otherwise accept the null hypothesis.

Let us recall that for testing hypothesis (6), we have the following information: ${X_{i1}}, ..., {X_{i{n_i}}}$ - a random sample from $N({\mu _i}, \sigma _i^2)$, $i = 1, ..., k$, on the basis of which we compute

where ${\bar X_i}$ and $S_i^2$ are the $i$ th sample mean and sample variance.

To test hypotheses concerning parameter ${\theta _i}$, sufficient statistics is ${\bar Y_i} = {\bar X_{i + 1}} - {\bar X_i}$, where the distribution is $N({\bar y_i}|{\theta _i}, \sigma _i^2/{n_i} + \sigma _{i + 1}^2/{n_{i + 1}})$, $i = 1, ..., k - 1$, because of independence of ${\bar X_{i + 1}}$ and ${\bar X_i}$. Parameter ${\theta _i} = 0$ at basic hypothesis $H_0^i$, and ${\theta _i} < 0$ or ${\theta _i} > 0$, at alternative hypotheses $H_ - ^i$ or $H_ + ^i$, accordingly.

2.1.   Testing a subset of individual hypotheses of (6)

For ease of presentation and understanding, let us first consider one subset of individual hypotheses of (6), and for simplicity, omit index $i$. Let us use one of the possible formulations of CBM, in particular, restrictions on posterior probabilities of rejected true hypotheses (Task 7) considered in ^[22,23]. In this formulation, the Type Ⅱ error rate is minimized (that means the power is maximized), and the Type Ⅰ error rate is restricted. This means, that the statement of the problem is the following ^[21,23]:

subject to

Here, for each subset of individual hypotheses of (6), the following loss functions are used

where ${K_1}$ is the loss of incorrectly accepted ${H_j}$ when ${H_i}$ is true and ${K_0}$ the loss of incorrectly rejected ${H_j}$ at testing it versus ${H_i}$, $i, j \in ( - , 0, + )$.

Using the undetermined Lagrange multiplier method and the concepts of posterior probabilities, to solve problems (8) and (9), we have ^[23]

where Lagrange multipliers $\lambda _7^ -$, $\lambda _7^0$, and $\lambda _7^ +$ are determined so that in (9), the equalities hold.

The most common criteria of estimation of the quality of testing directional hypotheses are the mixed directional false discovery rate ($mdFDR$) and the false acceptance rate (FAR), which have the following forms ^[21,22]

and

respectively.

According to the theorems proved in ^[23], if the condition $r_7^ - + r_7^ + = q \cdot {K_1} \cdot {P_{\min }}$ is satisfied, where $0 < q < 1$, ${P_{\min }} = \min \left\{ {p({H_ - }, {H_0}, {H_ + })} \right\}$, then the following condition $mdFDR \leqslant q$ is fulfilled, and if the condition $r_7^ - + r_7^0 + r_7^ + = q \cdot {K_1} \cdot {P_{\min }}$ is satisfied, the following condition $mdFDR + FAR \leqslant q$ is fulfilled. Also, when $\lambda _7^0$ is chosen so that in the second condition of (9) equality takes place, the following is fulfilled:

where ${P'_{\min }} = \min \left\{ {p({H_ - }), p({H_ + })} \right\}$.

For the case of directional hypotheses, the Type Ⅰ error rate ($TIER$) can be determined as follows

Comparing (12) and (15), it is clear that $mdFDR \geqslant TIER$.

Let us define Type Ⅱ error rates ($TIIER$) as follows

Comparing (13) and (16), we are convincing that $FAR = TIIER$. Thus, to fulfill the condition $\frac{1}{{{{P'}_{\min }}}} \cdot \frac{{r_7^0}}{{{K_1}}} = q$, the following takes place $FAR = TIIER \leqslant q$, where $0 < q < 1$.

Theorem 1. CBM 7 with restriction levels (9), to satisfy a condition $(r_7^ - + r_7^ + )/({K_1} \cdot p({H_0})) = q$, where $0 < q < 1$, ensures a decision rule with Type I error rate less or equal to $q$, i.e., with the condition $TIER \leqslant q$.

Proof. From the first and the third conditions of the restrictions of (9), we write the following

and

From here, and considering (15), we are convincing in the correctness of theorem 1.

Because of the specific nature of the acceptance regions of CBM ^[21,22,23], it can so happen that making a simple decision on the basis of existed information becomes impossible, e.g., when the test statistic belongs to the intersection areas of the acceptance regions or does not belong to any of these regions. In such a situation, it becomes impossible to make a simple decision with a specified confidence level on the basis of the information, and more information is required to achieve this. If acquiring more information is impossible, then the restriction levels in (9) must be changed until a simple decision can be made. When acquiring more information is possible, we appeal to the sequential experiment, i.e., to increase a sample size and apply Procedure A.

Procedure A

The statistics for testing a subset of individual hypotheses of (6) is ${\bar y_i} = {\bar x_{i + 1}} - {\bar x_i}$, which depends on two initial samples ${x_{i1}}, ..., {x_{in}}, ...$ and ${x_{i + 1, 1}}, ..., {x_{i + 1, n}}, ...$. Let us denote the test statistics on the basis of $n$ observations by ${\bar y_{i, n}}$. Then the sequential procedure is as follows:

Step 1

- if ${\bar y_{i, n}}$ belongs to only region $\Gamma _ - ^i$, accept hypothesis $H_ - ^i$,

- if ${\bar y_{i, n}}$ belongs to only region $\Gamma _0^i$, accept hypothesis $H_0^i$,

- if ${\bar y_{i, n}}$ belongs to only region $\Gamma _ + ^i$, accept hypothesis $H_ + ^i$,

- otherwise continue sampling; collect next observations and compute new test statistics ${\bar y_{i, n + 1}}$;

Step 2

- if ${\bar y_{i, n + 1}}$ belongs to only region $\Gamma _ - ^i$, accept hypothesis $H_ - ^i$,

- if ${\bar y_{i, n + 1}}$ belongs to only region $\Gamma _0^i$, accept hypothesis $H_0^i$,

- if ${\bar y_{i, n + 1}}$ belongs to only region $\Gamma _ + ^i$, accept hypothesis $H_ + ^i$,

- otherwise, continue sampling; collect observations and compute test statistics etc.

The sampling continues until the test statistics belong to only one hypothesis acceptance region.

Note 1: Obviously, the sample size at the beginning of the sequential test is equal to any particular value, i.e., $n = {n^*}$ (in particular, it can be ${n^*} = 1$), and if the desired level of reliability of making a decision is achievable for this amount of information, this corresponds to the parallel experiment on which the testing process finishes. In case of continuing sampling, the parallel experiment generalizes to the sequential experiment naturally.

2.2.   Testing of multiple hypotheses (6)

For testing multiple directional hypotheses (6), the concept of the total mixed directional false discovery rate (tmdFDR) is introduced in ^[23,25]

To guarantee level $q$, in testing hypothesis (6), we have to consider $k - 1$ subsets of directional hypotheses. Then, for each of them, we use procedure A described above to provide a level of ${q_i}$ for $i$ th subset of hypotheses, so that $\sum\nolimits_{i = 1}^{k - 1} {{q_i}} = q$ is achieved.

We act similarly to provide a level $q$ for the total Type Ⅰ error rate ($TTIER$) and for the total Type Ⅱ error rate ($TTIIER$). Namely, we provide ${q_i}$, the level of the appropriate criteria for the $i$ th subset of the individual directional hypotheses. As a result, we have

where $TIE{R_i}$ and $TIIE{R_i}$ are the Type Ⅰ error rate and the Type Ⅱ error rate, respectively, of the $i$ th subset of directional hypotheses ^[23,26].

The values of ${q_i}$ in all three cases (for $tmdFDR$, for $TTIER$, and for $TTIIER$) can be chosen to be equal, i.e., ${q_i} = q/(k - 1)$ or different, e.g., inversely proportional to the informational distances between the tested hypotheses in the subsets of directional hypotheses ^[23].

To test multiple hypotheses, two family-wise error rates, type Ⅰ ($FWE{R_I}$) and type Ⅱ ($FWE{R_{II}}$), are used (see, for example, ^[26,27,28]), which are the same as $TTIER$ and $TTIIER$ in the considered case, i.e., we have $TTIER = FWE{R_I}$ and $TTIIER = FWE{R_{II}}$.

In all three considered cases, with the restriction of tmdFDR, $TTIER$, and $TTIIER$ on the desired levels, we use the above described sequential Procedure A, where the sampling continues until a simple decision is made for all the subsets of multiple hypotheses (6). The stopping rules remain the same as in ^[23]. Theorems 3.2 and 3.3 of the work ^[23], proving the appropriateness of the stopping rule, are in force for the considered directional hypotheses as well.

As mentioned in ^[24], "Currently, in many real-life applications, we indeed encounter situations where the number of individual hypotheses in the set of multiple hypotheses (6) is very big, i.e., when data is big. In such a situation, determination of Lagrange multipliers for each subset of an individual hypothesis requires a long time for computation. Though the computation of Lagrange multipliers is completed in the preparatory stage before making a decision, still the reduction of computation time is important for many practical applications from the operational and cost considerations." For this purpose, Theorem 4 is proved in ^[24] in accordance with the testing of all of subsets of individual hypotheses of (6) and can be used once with the same Lagrange multipliers, determined for a subset of individual hypothesis with lowest divergence among directional hypotheses at the level ${q_i} = q/(k - 1)$, satisfying condition $\sum\nolimits_{i = 1}^{k - 1} {{q_i}} = q$ provides the total mixed directional false discovery rate (17), as well as the total Type Ⅰ error rate (18) and the total Type Ⅱ error rate (19), restricting all of them with level $q$.

3.   Algorithms for computation

3.1.   Variances are known

Let us suppose that the variances of ${X_i}$ that is $\sigma _i^2$, $i = 1, ..., k$, are known. To realize the above developed method, the concretization of distribution laws of the statistics ${\bar Y_i}$ at hypotheses $H_i^{(0)}$, $H_i^{( - )}$ and $H_i^{( + )}$ is necessary, i.e., it is necessary to determine densities $p({\bar y_i}|H_i^{(0)})$, $p({\bar y_i}|H_i^{( - )})$, and $p({\bar y_i}|H_i^{( + )})$.

As mentioned in the above ${\bar Y_i}$ is normally distributed. Therefore, taking into account the intervals of definition of the mathematical expectation at different hypotheses, we can suppose that $p({\bar y_i}|H_i^{(0)}) = N({\bar y_i}|0, \bar \sigma _i^2)$ at $H_i^{(0)}$ and $p({\bar y_i}|H_i^{( - )})$ and $p({\bar y_i}|H_i^{( + )})$ are the truncated normal densities $N(0, \omega _0^{ - 1}\bar \sigma _i^2)$ (${\omega _0}$ known) over $( - \infty , 0)$ and $(0, + \infty )$, respectively ^[23,29]. Here, $\bar \sigma _i^2 = (\sigma _i^2 + \sigma _{i + 1}^2)/n = \vartheta _i^2/n$, were $\vartheta _i^2 = (\sigma _i^2 + \sigma _{i + 1}^2)$. Thus, we have ^[24]

For these distributions, hypotheses acceptance regions (11) for observations ${\bar y_i}$ are the following ^[23]

Here, $n$ is the quantity of ${\bar y_i}$ on the basis of which hypothesis (6) is tested using sequential Procedure A, ${u_i} = n \cdot {\bar y_i}/({\vartheta _i}\sqrt {n + {\omega _0}} )$, and $\Phi ( \cdot )$ is the standard normal distribution function.

The Lagrange multipliers are determined so that in condition (9), the equalities hold. For the solution of the relevant equations, the suitable probability integrals are computed by the Monte-Carlo method (see, for example, ^[21,23]).

3.2.   Variances are not known

Let us suppose that the variances of ${X_i}$ that is $\sigma _i^2$, $i = 1, ..., k$, are not known. In this case, we can consider two methods based on the maximum likelihood ratio and Stein's method ^{[30,31,32,33,34]}. The essence of these methods consist of the following. In the maximum likelihood ratio approach, the density with unknown parameters' values is replaced with the maximum of the density over all possible values of these parameters at the truth of testing hypotheses, computed by existing samples. Stein's method integrates the density over unknown parameters using special measures to obtain the density of the maximum invariant statistic, which can then be used to analyze the problems; for example, to find the uniformly most powerful invariant test ^[32].

The method based on the maximum likelihood ratio can be applied when the numbers of observations ${n_i}$, $i = 1, ..., k$, are large, which enable us to divide every subset of the observations into two parts ${x_{i, 1}}, ..., {x_{i, {m_i}}}$ and ${x_{i, {m_i} + 1}}, ..., {x_{i, {n_i}}}$, where ${m_i} < {n_i}$. Using ${m_i}$ observations, $\bar S_i^2 = S_i^2 + S_{i + 1}^2$, $i = 1, ..., k - 1$, estimators of the variance $\bar \sigma _i^2$ of the statistics ${\bar Y_i} = {\bar X_{i + 1}} - {\bar X_i}$ are computed and are used in formulae (20) and (21) instead of $\bar \sigma _i^2$. Statistics ${u_i}$ are computed using observations ${x_{i, {m_i} + 1}}, ..., {x_{i, {n_i}}}$, $i = 1, ..., k$, to make decisions based on hypotheses acceptance regions (21) when testing hypothesis (6).

When applying Stein's method, to average the influence of $\bar \sigma _i^2$, it is necessary to use a priori distribution of $\bar S_i^2$, which is difficult to determine functionally with known parameters at the considering case. Therefore, we conduct the following: Construct a confidence interval of $\bar S_i^2 = S_i^2 + S_{i + 1}^2$ with the given confidence probability $1 - \alpha$ ($0 < \alpha < 1$) and assume that $\bar S_i^2$ takes values from this interval according to a uniform distribution.

Finding confidence intervals of $\bar S_i^2$, $i = 1, ..., k - 1$ is not difficult. It looks like:

where

where $\chi _{{m_i} - 1, \alpha }^2$ is a quantile of the $\alpha$-th order of the chi-square distribution with degree of freedom ${m_i} - 1$.

To overcome the uncertainty in the densities (20) caused by not knowing the value of the variance, we are averaging the effect of the variance on these densities by integrating them over the confidence interval of the variance with uniform density. In this case, conditional distribution densities (20) take the form

where ${k_i} = \frac{{\sqrt {{n_i}} }}{{\sqrt {2\pi } (\bar S_{i.u}^2 - \bar S_{i.L}^2)}}$, ${d_i} = \frac{{\sqrt {{n_i} \cdot {\omega _0}} }}{{\pi (\bar S_{i.u}^2 - \bar S_{i.L}^2)}}$, ${a_i} = \frac{{{n_i}\bar y_i^2}}{2}$, ${b_i} = {n_i}{\bar y_i}$, ${c_i} = \frac{{{n_i} + {\omega _0}}}{2}$.

Let us denote:

Hypotheses acceptance regions (11) take the following form using these denotations:

where the Lagrange multipliers are determined, so that in conditions (9), the equalities hold.

4.   Computation results

To compare the offered method with the known ones, we present computation outcomes of the examples considered in (^[4], p. 26, Tables 2 and 6) in Tables below where the results are obtained by the method offered above (see Item 3.1) and the Hayter PB method. As a priori probabilities, we take $p({H_ - }) = p({H_0}) = p({H_ + }) = 1/3$, coefficient ${\omega _0} = 4$ for the results given in Tables 1–5, and ${\omega _0} = 13$ for Stein's method for true alternative hypotheses the results of which are given in Table 6. To keep $mdFDR$, $TIER$, and $TIIER$ at $q = 0.05$, to test individual hypotheses of (6), we have to choose the following restriction levels in (9) for each subset of the hypotheses: $r_7^ - = r_7^0 = r_7^ + = 0.0083(3)$. To test multiple hypotheses with the $k - 1$ subset of individual hypotheses for ensuring the same restriction levels of the criteria $tmdFDR$, $TTIER$, and $TTIIER$, we have to choose the following restriction levels: $r_7^ - = r_7^0 = r_7^ + = 0.0083(3)/(k - 1)$. To test of all subsets of individual hypotheses, we use one and the same Lagrange multipliers, which are determined for a subset of individual hypotheses with the lowest divergence among directional hypotheses ^[24]. Because of $k - 1 = 3$, the restriction levels in (9) are equal to $0.0027(7)$. For the solution of equations (9), the necessary probability integrals are computed by the Monte Carlo method with a sample size equal to 5,000 (see, for example, ^[21,35]). The obtained values of the Lagrange multipliers are: $\lambda _7^ - = {\text{5}}{\text{.625}}$, $\lambda _7^0 = {\text{0}}{\text{.965194702148438}}$, and $\lambda _7^ + = {\text{2}}{\text{.34375}}$. The values of $TTIER$, $TTIIER$, and $tmdFDR$ are computed by the Monte Carlo simulation of random sequences with 5,000 observations. The codes for all necessary computations are written in MATLAB R2021b.

When using CBM, principle complexity does not arise when the problem dimension increases [21,22]. The computation time required to make a decision in CBM is equal to the time required for the classical Bayes method. The time required to determine the Lagrange multipliers certainly increases, but it is realized at the preparatory stage and does not directly affect the decision-making time.

4.1.   Variances are known

Computational results, when variances are known, are given in Table 1 (null hypotheses are true) and Table 2 (alternative hypotheses are true) for different ${n_i}$ and ${\sigma _i}$, $i = 1, 2, 3, 4$. From here, it is seen that for all computed combinations, $TTIER = tmdFDR$ and they all are significantly less than the type Ⅰ error rate of the Hayter PB method given in [4]. The powers for all computed combinations are equal to 1 and significantly surpass the appropriate values of the Hayter PB method. The offered method perfectly distinguishes one- and two-sided alternatives whose possibility is not discussed in ([4] p. 26, Table 2).

4.2.   Variances are not known

4.2.1.   Computational results obtained by the maximum likelihood ratio

Computational results obtained by the method involving the maximum likelihood estimator of the unknown variances, are given in Table 3 (null hypotheses are true) and Table 4 (alternative hypotheses are true). The computations give us the same results as we had in the case of the known variances, that is, the reliabilities of the decisions in CBM are significantly higher than the results obtained by the Hayter PB method ^[4]. However, the number of observations required to make a decision is greater in CBM for this case than for the case when variances are known. One circumstance is noteworthy: Calculating the variance estimator with less than 5–6 observations is not recommended due to the low accuracy of the obtained results in statistics. Nevertheless, in the calculated examples, even when the number of observations is 2–3, the reliability of the decisions is quite high. In our opinion, the reason for this, in addition to the high reliability of the CBM, lies in the relatively large number of observations used to make a decision.

4.2.2.   Computational results obtained by Stein's method

Computational results obtained by the Stein's method when the uniform distributions are used as a priori distributions of the unknown variances, are given in Table 5 (null hypotheses are true) and Table 6 (alternative hypotheses are true). The obtained results are very reliable. The probabilities of incorrect decisions are equal to zero, which are provided by the big number of observations necessary for making decisions when alternative hypotheses are true.

5.   Discussion

The offered method of solving the ANOVA problem, for known and unknown variances of the results, with restricted Type Ⅰ and Type Ⅱ error rates based on CBM gives very reliable results that surpass the methods of today. It is a sequential method that requires a set of observations to make a decision, the number of which is especially large in Stein's method in the case of true alternative hypotheses. Nevertheless, due to the large practical application of the ANOVA problem and the fact that, in many cases, the number of data is quite large, and the requirements for the accuracy of the decisions are high, we consider the results to be important and to deserve high attention.

It is worth noting that the CBM method using the maximum likelihood method requires more observations than the same CBM method using Stein's method to test the null hypothesis; this situation is reversed when testing alternative hypotheses. In this case, using Stein's method requires more observations than using the maximum likelihood method. The reason lies in the use of different values of ${\omega _0}$. It should be noted that the increase of ${\omega _0}$ causes the increase in the quantity of observations necessary to make a decision and, accordingly, the increase of the accuracy of made decisions ^[24,33].

The computing results obtained by CBM and the Hayter PB methods are given in Tables 1–6. In order to compare the results obtained by these methods, $q = 0.05$ is taken in CBM, since in Hayter PB methods, the calculations have been realized for a Type Ⅰ error rate of α = 0.05 (see Tables 2–4, 6 in ^[4]). Since each scenario has four normal population correspondences, i.e., $k = 4$, we have to test three hypotheses each time (see (6)). In each case under consideration, the true hypotheses are determined by the values of ${\theta _i} = {\mu _{i + 1}} - {\mu _i}$, $i = 1, ..., k - 1$. Therefore, in each scenario, three hypotheses (from the set of hypotheses ${H_0}$, ${H_ - }$ and ${H_ + }$) are accepted, and depending on ${\theta _i}$, $i = 1, ..., k - 1$, the values of the criteria $TTIER$, $TTIIER$, $tmdFDR$, and $Power = 1 - TTIIER$ are computed. The values criteria $TTIER$ and $tmdFDR$ are given when null hypotheses are true in all three cases (see Tables 1, 3, and 5), and the values all four considered criteria $TTIER$, $TTIIER$, $tmdFDR$, and $Power = 1 - TTIIER$ are given when the alternative hypothesis is true even for one of the three cases. In the columns $n$, $\sigma$, and $\mu$, the values of sample sizes for considered normal populations, standard deviations, and mathematical expectations, respectively, are given for each considered case. The second to last the final column contains the results obtained by Hayter PB methods from ^[4], and the last column contains each scenario in the CBM accepted hypotheses defined by (5) or (6). The column AN (average number) contains the average number of observations necessary to make a decision in CBM (averaged over the results of 5,000 experiments).

From the results of these Tables, the superiority of CBM over Hayter PB methods is evident. Moreover, unlike the Hayter PB method, CBM enables us to distinguish (4) and (5), which are the same as (1) and (2).

6.   Conclusions

The problem of testing hypotheses concerning the means of a set of normally distributed random variables at the restriction of both Type Ⅰ and Type Ⅱ error rates are considered in this paper. To solve the stated problem, one of the possible statements of CBM is used. In order to overcome the difficulties related to the complexity of testing hypotheses caused by not knowing the values of the variances of random variables, the maximum likelihood and Stein's methods are used. The ability to limit Type Ⅰ and Type Ⅱ error rates and other relevant criteria to the required level is shown for the developed algorithms. The latter is confirmed by the calculation results of practical examples by simulating different scenarios.

Author contributions

Kartlos J. Kachiashvili: Conceptualization, Formal analysis, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing–original draft, Writing–review & editing; Ashis SenGupta: Conceptualization, Formal analysis, Investigation, Resources, Validation, Writing–original draft, Writing–review & editing; Joseph K. Kachiashvili: Data curation, Formal analysis, Investigation, Resources, Software, Visualization, Writing–original draft, Writing–review & editing.

Use of Generative-AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Conflict of interest

All Authors declare that they have no financial interests.