On criticality coupled sub-Laplacian systems with Hardy type potentials on Stratified Lie groups

1.   Introduction and main result

In this paper, we are concerned with the system of sub-Laplacian equations with singular Hardy potentials and coupled with terms up to critical power on the Carnot group $\mathbb{G}$ given below

where $-\Delta_{\mathbb{G}}$ stands for the sub-Laplacian on Carnot group $\mathbb{G}$, $\Omega$ is a bounded domain in $\mathbb{G}$ with smooth boundary $\partial\Omega$ and $0, z_{0}\in \Omega$, $d$ is the natural gauge on $\mathbb{G}$ associated with the fundamental solution of $-\Delta_{\mathbb{G}}$, $\psi$ is the weight function defined as $\psi: = |\nabla_{\mathbb{G}}d|$ and $\nabla_{\mathbb{G}}$ is the horizontal gradient associated with $\Delta_{\mathbb{G}}$. Further $2^*(\cdot): = \frac{2(Q-\cdot)}{Q-2}$ $(\, \cdot = \alpha, \beta, \gamma)$ is the critical Hardy-Sobolev exponent, $Q$ being the homogeneous dimension of the space $\mathbb{G}$ with respect to the dilation. The parameters

and $h$ is a function defined on $\Omega$ satisfying

A fundamental role in the functional analysis on the singular sub-Laplacian problem on Carnot group is played by the following Hardy-type inequality

where $\mu_{\mathbb{G}} = (\frac{Q-2}{2})^{2}$ is the optimal constant, which is not attained, and $\psi$ is $\delta_{\gamma}$-homogeneous of degree $0$, $\psi^{2}$ is a smooth function out of the origin. The preceding inequality was firstly proved by Garofalo and Lanconelli in ^[1] for the Heisenberg group (see also ^[2]). Then, it has been extended to all Carnot groups, see ^[3].

We look for weak solutions of (1.1) in the product space $\mathcal{H}: = S^{1}_{0}(\Omega)\times S^{1}_{0}(\Omega)$, endowed with the norm

where the Folland-Stein space $S^{1}_{0}(\Omega) = \{u\in L^{2^*}(\Omega): \int_{\Omega}|\nabla_{\mathbb{G}}u|^{2}dz < +\infty\}$ is defined as the completion of $C^{\infty}_{0}(\Omega)$ with respect to the norm

Set $S^{1, 2}(\mathbb{G}) = \{u\in L^{2^*}(\mathbb{G}): |\nabla_{\mathbb{G}}u|\in L^{2}(\mathbb{G})\}$. For all $\alpha\in [0, 2)$, we define the subelliptic Hardy-Sobolev constant

From ^[4], $S_{\alpha}$ is independent of any $\Omega\subset\mathbb{G}$ in the sense that if

then, $S_{\alpha}(\Omega) = S_{\alpha}(\mathbb{G}) = S_{\alpha}$. Note that the Euler-Lagrange equation corresponding to the minimization problem for $S_{\alpha}$ is, up to a constant factor, the following:

In the case $\alpha = 0$, the existence of Sobolev extremals in the general Carnot case has been obtained by Garofalo and Vassilev ^[5] by means of a suitable adaptation of Lions' concentration-compactness principles. In the singular case, i.e., when $0 < \alpha < 2$, the existence of Hardy-Sobolev extremals has been proved by Han and Niu in ^[4], in the general quasilinear case, for the subclass of the Heisenberg groups. In ^[6], Loiudice extends this result for general Carnot groups, and states some qualitative properties of such extremals, namely, the extremal function $u\in S^{1, 2}(\mathbb{G})$ for $S_{\alpha}$, up to a change of sign, is positive and $u\in L^{p}(\mathbb{G})$, $\forall p\in (\frac{2^*}{2}, +\infty]$, and has the following decay at infinity:

Moreover, for any $\varepsilon > 0$, the family of rescaled functions

are solutions, up to multiplicative constants, of the equation (1.4) and satisfy

For $p_{1}, p_{2} > 1$ and $p_{1}+p_{2} = 2^{*}(\alpha)$, by the Young and Hardy-Sobolev inequalities, the following best constant is well-defined on the space $\mathcal{H}\backslash\{(0, 0)\}$:

From [7, Lemma 2.5], we known that

In recent years, much attention has been paid to singular problems involving both the Hardy type potential and the critical Sobolev term on Carnot group. We refer the reader to ^{[2,3,4,5,8,9,10,11,12]} and the references therein. Singular problems with Hardy type potential and critical Hardy-Sobolev term have also been extensively studied, see ^{[6,7,13,14,15,16,17,18]} and the references therein. Further, in ^{[19,20,21,22,23]}, Pucci and her collaborators have dealt with some subelliptic problems in the Heisenberg setting, while ^[24] has treated, in the Euclidean setting, a $p$-Laplacian problem with double critical Hardy type nonlinearities. On the other hand, some authors also studied the critical sub-elliptic systems on stratified Lie group. For example, Zhang ^[7] dealt with the problem

where $0\in \Omega$, $\lambda$, $\mu > 0$, $1 < q < 2$, $0\leq \alpha < 2$, $0\leq \beta < 2$, $p_{1}$, $p_{2} > 1$ satisfying $2 < p_{1}+p_{2}\leq 2^{*}(\alpha)$. By using the variational methods and Nehari manifold, the author proved that the sub-elliptic system (1.7) admits at least two positive solutions when parameters pair $(\lambda, \mu)$ belongs to a certain subset of $\mathbb R^2_{+}$. In a recent paper, Zhu and Zhang ^[18] considered the following critical systems

By using the second concentration-compactness principle and concentration-compactness principle at infinity to prove that the $(PS)_{c}$-condition holds locally, the authors prove, thanks also to Theorem 1, a new symmetric version of the mountain pass theorem due to Kajikiya in ^[25], existence of infinitely many solutions of (1.8) under suitable conditions on $\lambda_{1}$, $\lambda_{2}$ and $\beta$.

The study of problem (1.1) is motivated by two reasons. First, as far as we know, little has been done for critical singular sub-elliptic systems on Carnot group. Second, there are few results on sub-elliptic systems with multiple critical nonlinearities. In addition, we point out that the methods used in these above papers cannot be applied to sub-elliptic problem (1.1). To the best of our knowledge, problem (1.1) has not been considered before. Due to the lack of compactness of embedding, the associated functional of (1.1) fails to satisfy the Palais-Smale condition in general. Thus, the standard variational argument cannot be applied directly. However, by using the concentration-compactness principle ^[26,27], we can find a proper range of $c$ where the $(PS)_c$-condition holds for the associated functional. Then we establish the existence of a positive local minimum for the associated functional by the Ekeland variational principle ^[28] and use the mountain pass theorem ^[29] to find a second positive solution. Moreover, another difficulty relies on the fact that every nontrivial solution of (1.1) is singular at $\{z = 0\}$. So different techniques are needed to deal with the singular case. In order to obtain our results, we need more delicate estimates.

Our main result is the following.

Theorem 1.1. Assume that (1.2)-(1.3) hold. Then there exists $\Lambda > 0$ such that for $\lambda\in (0, \Lambda)$, problem (1.1) has at least two positive solutions and among them one has negative energy, the other has positive energy.

The paper is divided into three sections. Section 2 contains the main functional setting and definitions, as well as an analysis of the PS condition in critical dimension. Finally, Section 3 is devoted to prove the main result about the existence of negative and positive energy solutions of system (1.1).

2.   Preliminaries and functional setting

In this section we recall some basic facts on the Carnot groups. For a compete treatment, we refer to the monograph ^[30,31] and the classical papers ^[32,33]. We also quote for an overview on general homogeneous Lie group.

A Carnot group (or Stratified group) $(\mathbb{G}, \circ)$ is a connected, simply connected nilpotent Lie group, whose Lie algebra $\mathfrak {g}$ admits a stratification, namely a decomposition $\mathfrak{g} = \oplus_{i = 1}^{k}V_i$ such that $[V_1, V_i] = V_{i+1}$ for $i = 1, \cdots k-1$ and $[V_{1}, V_k] = \{0\}$. The number $k$ is called the step of the group $\mathbb{G}$. In this context the symbol $[V_1, V_i]$ denotes the subalgebra of $\mathfrak {g}$ generated by the commutators $[X, Y]$, where $X\in V_{1}$, $Y\in V_{i}$ and where the last bracket denotes the Lie bracket of vector fields, that is $[X, Y] = XY-YX$.

By means of the natural identification of $\mathbb{G}$ with its Lie algebra via the exponential map (which we shall assume throughout), it is not restrictive to suppose that $\mathbb{G}$ is a homogeneous Lie group on $\mathbb{R}^{N} = \mathbb{R}^{N_1}\times \mathbb{R}^{N_2}\times\cdots\times \mathbb{R}^{N_k}$, with $N_{i} = \dim(V_{i})$, equipped with a family of group-automorphisms $\delta _\gamma :\mathbb{G}\to \mathbb{G}$ of the form

where $x^{(i)}\in \mathbb{R}^{N_i}$ for $i = 1, 2, \cdots, k$. Here, $N = \sum_{i = 1}^{k}N_{i}$ is called the topological dimension of $\mathbb{G}$ and $\delta_{\gamma}$ is called the dilations of $\mathbb{G}$. Under this automorphisms $\{\delta_{\gamma}\}_{\gamma > 0}$, the homogeneous dimension of $\mathbb{G}$ is given by $Q = \sum_{i = 1}^{k}i\cdot \text{dim}V_{i}.$ From now on, we shall assume throughout that $Q\geq 3$. We remark that, if $Q\leq 3$, then $\mathbb{G}$ is necessarily the ordinary Euclidean space $\mathbb{G} = (\mathbb{R}^{Q}, +)$.

Now, if $\{X_1, \cdots, X_{N_1}\}$ ($N_{1} = \dim(V_1)$) is any basis of $V_1$, the second order differential operator

is called a sub-Laplacian on $\mathbb{G}$. We shall denote by $\nabla_\mathbb{G}: = (X_1, \cdots, X_{N_1})$ the related horizontal gradient. For $z\in \mathbb{G}$, the left translation on $\mathbb{G}$ are defined by

Then, it is easy to check that $\nabla_\mathbb{G}$ and $\Delta_{\mathbb{G}}$ are left-translation invariant with respect to the group action $\tau_{z}$ and $\delta_{\gamma}$-homogeneous, respectively, of degree one and two, that is, $\nabla_\mathbb{G}(u\circ \tau_{z}) = \nabla_\mathbb{G}u\circ \tau_{z}$, $\nabla_\mathbb{G}(u\circ \delta_{\gamma}) = \gamma \nabla_\mathbb{G}u\circ \delta_{\gamma}$, $\Delta_\mathbb{G}(u\circ \tau_{z}) = \Delta_\mathbb{G}u\circ \tau_{z}$ and $\Delta_\mathbb{G}(u\circ \delta_{\gamma}) = \gamma^{2} \Delta_\mathbb{G}u\circ \delta_{\gamma}$.

A homogeneous norm $\mathbb{G}$, adapted to the fixed homogeneous structure is continuous function $d:\mathbb{G}\to [0, +\infty)$, smooth away from the origin, such that $d(\delta_{\gamma}(z)) = \gamma d(z)$ for every $\gamma > 0$, $d(z^{-1}) = d(z)$ and $d(z) = 0$ iff $z = 0$. For the above gauge, when $Q\geq 3$, the function

is a fundamental solution of $-\Delta_{\mathbb{G}}$ with pole at $0$, for a suitable constant $C > 0$.

The variational functional $I_{\lambda}:\mathcal{H}\to \mathbb{R}$ associated to (1.1) is defined as

defined on the product space $\mathcal{H}$. Without putting great efforts, it can be shown that $I_{\lambda}$ is well defined and $C^{1}$. Now we give the definition of a weak solution of the problem (1.1).

Definition 2.1. A function $(u, v)\in \mathcal{H}$ is said to be a weak solution of equation (1.1) if $(u, v)$ satisfies

for all $(\psi_{1}, \psi_{2})\in \mathcal{H}$.

It is clear that the nozero critical points of $I_{\lambda}$ in $\mathcal{H}$ are equivalent to the nontrivial solutions of (1.1).

Now we state the following inequality which will be used in the subsequence lemmas.

Lemma 2.2. ^[6] Let $2\leq p\leq 2^*(\alpha)$, $0\leq \alpha < 2$, then there exists $C_{p} > 0$ such that for all $u\in S^{1}_{0}(\Omega)$,

Moreover, for $p = 2^*(\alpha)$, the best constant in (2.1) will be denoted by $S_{\alpha}(\Omega)$, that is,

and it is indeed achieved in the case $\Omega = \mathbb{G}$. Moreover, the extremal function for $S_{\alpha}: = S_{\alpha}(\mathbb{G})$ has the following decay behavior at infinity:

Taking $\rho > 0$ small enough such that $B_{d}(0, \rho)\subset \Omega$. Choose the cut-off function $\eta\in C_0^{\infty}(B_{d}(0, \rho))$ such that $0\leq \eta\leq 1$ and $\eta\equiv1$ in $B_{d}(0, \frac{\rho}{2})$, where $B_{d}(z, r)$ denotes the ball with center at $z$ and radius $r$ with respect to the gauge $d$. Define the function

where $u_{\varepsilon}$ is given in (1.5). Then, we have the following estimates.

Lemma 2.3. [6, Lemma 6.1] Let the homogeneous dimension $Q\geq 4$, $0\leq \alpha < 2$. Then the following estimates hold when $\varepsilon\to 0$:

and

Taking into account the exact asymptotic behavior of Hardy-Sobolev extremals, we get the following results:

Lemma 2.4. Assume that $0\leq s < 2$, $Q\geq 4$, $1\leq q < 2^*(s)$. Then, as $\varepsilon\to 0$, we have the following estimates:

Proof. For all $1\leq q < 2^*(s)$, as $\varepsilon\to 0$, it is easily seen that

where the constant $0 < \rho_{0} \ll \rho$ small enough.

$({\rm{i}})$ If $(Q-2)q+s-Q = 0$, straightforward computations yield

So, (2.6) and (2.7) yield that

$({\rm{ii}})$ If $(Q-2) q+s-Q < 0$, it follows that $(Q-2) q+s-Q+1 < 1$ and

Then, inserting (2.9) into (2.6), we obtain

$({\rm{iii}})$ If $(Q-2) q+s-Q > 0$, we have $(Q-2) q+s-Q+1 > 1$, then there exists $C > 0$ such that

Therefore, by (2.6) and (2.11),

Thus, (2.8), (2.10) and (2.12) imply that (2.5) holds.

Lemma 2.5. Let $(u, v)\in\mathcal{H}\backslash\{(0, 0)\}$ be a weak solution of problem (1.1). Then there exists a positive constant $C_{*}$ depending on $Q$, $\sigma$, $\alpha$, $q$, $|\Omega|$ and $\|h\|_{L^{q_{*}}(\Omega, \frac{\psi^{\sigma}}{d(z)^{\sigma}}dz)}$ such that

Proof. Without loss of generality, we may assume that $\alpha\geq \beta\geq \gamma$. Then, $2^*(\alpha)\leq 2^*(\beta)\leq 2^*(\gamma)$. First, by Hölder and Hardy-Sobolev inequalities, for all $u\in S_{0}^{1}(\Omega)$, we get

Then,

Therefore, it follows from $\langle I'_{\lambda}(u, v), (u, v)\rangle = 0$ and (2.14) that

Here $C_{*}$ is a positive constant depending on $Q$, $\sigma$, $\alpha$, $q$, $|\Omega|$ and $\|h\|_{L^{q_{*}}(\Omega, \frac{\psi^{\sigma}}{d(z)^{\sigma}}dz)}$.

In the following result, we show that the functional $I_{\lambda}$ satisfies $(PS)_{c}$-conditions.

Definition 2.6. Let $c\in \mathbb{R}$, $\mathcal{H}$ be a Banach space and $I_{\lambda}\in C^{1}(\mathcal{H}, \mathbb{R})$. Then $\{(u_{n}, v_{n})\}\subset \mathcal{H}$ is a Palais-Smale sequence at level $c$ $((PS)_c)$ in $\mathcal{H}$ for $I_{\lambda}$ if $I_{\lambda}(u_n, v_{n}) = c +o_{n}(1)$ and $I'_{\lambda}(u_n, v_{n}) = o_{n}(1)$ strongly in in $\mathcal{H}^{-1}$ as $n\to \infty$. We say $I_{\lambda}$ satisfies $(PS)_{c}$-condition if for any Palais-Smale sequence $\{(u_n, v_{n})\}$ in $\mathcal{H}$ for $I_{\lambda}$ has a convergent subsequence.

Lemma 2.7. Suppose that $1 < q < 2$ and $\alpha, \beta, \gamma, \sigma\in [0, 2)$. Let $\{(u_{n}, v_{n})\}\subset \mathcal{H}$ is a $(PS)_{c}$-sequence for $I_{\lambda}$. Then, $\{(u_{n}, v_{n})\}$ is bounded in $\mathcal{H}$.

Proof. Let $\{(u_{n}, v_{n})\}\subset \mathcal{H}$ be a $(PS)_{c}$-sequence of $I_{\lambda}$, then $I_{\lambda}(u_{n}, v_{n})\to c$ and $I'_{\lambda}(u_{n}, v_{n})\to 0$ as $n\to \infty$. From (2.14), we have

which implies that $\{(u_{n}, v_{n})\}$ is bounded in $\mathcal{H}$ since $q < 2 < 2^*(\alpha)$ and $\lambda > 0$.

Proposition 2.8. Under the assumptions of Theorem 1.1, the functional $I_{\lambda}$ satisfies $(PS)_{c}$-condition for all $c < c_{\infty}$, here

and $C_{*}$ is given in Lemma 2.5.

Proof. From Lemma 2.7, we know that the $(PS)_{c}$-sequence $\{(u_{n}, v_{n})\}$ is bounded in $\mathcal{H}$. Due to the critical Hardy-Sobolev inequality (2.1), there exists a subsequence, still denote by $\{(u_n, v_{n})\}$, such that $u_{n}\rightharpoonup u$, $v_{n} \rightharpoonup v$ weakly in $S^{1}_{0}(\Omega)$; $u_{n} \rightharpoonup u$, $v_{n}\rightharpoonup v$ weakly in $L^{2^*(\alpha)}(\Omega, \frac{\psi^{\alpha}}{d(z)^{\alpha}}dz)$, $L^{2^*(\beta)}(\Omega, \frac{\psi^{\beta}}{d(z)^{\beta}}dz)$ and $L^{2^*(\gamma)}(\Omega, \frac{\psi^{\gamma}}{d(z)^{\gamma}}dz)$; $u_{n}\to u$, $v_{n}\to v$ strongly in $L^{t}(\Omega, \frac{\psi^{\sigma}}{d(z)^{\sigma}}dz)$ for all $t\in [1, 2^*(\sigma))$; and $u_{n}(z)\to u(z)$, $v_{n}(z) \to v(z)$ a. e. in $\Omega$. Moreover, for the above subsequence we assume that

weakly in the sense of measures. Using the concentration-compactness principle (see ^[26,27]), there exist an at most countable set $J$, a set of points $\{z_{j}\}_{j\in J}\in \Omega\backslash\{0\}$, real numbers $\hat{\mu}_{z_{j}}$, $\hat{\nu}_{z_{j}}$, $\bar{\mu}_{z_{j}}$, $\bar{\nu}_{z_{j}}$, $\tilde{\rho}_{z_{j}}$, $j\in J$, and $\hat{\mu}_{0}$, $\hat{\nu}_{0}$, $\bar{\mu}_{0}$, $\bar{\nu}_{0}$, $\tilde{\rho}_{0}$ such that

where $\delta_{z}$ is the Dirac-mass of mass 1 concentrated at $z$.

First we consider the possibility of the concentration at $\{z_{j}\}_{j\in J}\in \Omega\backslash\{0\}$. For any $\varepsilon > 0$ small, take $\phi_{z_{j}, \varepsilon}(z) = \phi(\delta_{\frac{1}{\varepsilon}}(z_{j}^{-1}\circ z))$, where $\phi(z) \in C^{\infty}_{0}(\Omega)$ is a smooth cut-off function such that $0\leq \phi\leq 1$, $\phi = 1$ in $B_{d}(0, 1)$, and $\phi = 0$ in $\Omega\backslash B_{d}(0, 2)$. Then, $|\nabla \phi_{z_{j}, \varepsilon}|\leq \frac{C}{\varepsilon}$ and $\{(\phi^{2}_{z_{j}, \varepsilon}u_{n}, \phi^{2}_{z_{j}, \varepsilon}v_{n})\}$ is bounded in $\mathcal{H}$. Testing $I'_{\lambda}(u_{n}, v_{n})$ with $(\phi^{2}_{z_{j}, \varepsilon}u_{n}, \phi^{2}_{z_{j}, \varepsilon}v_{n})$, we obtain $\lim_{n\to \infty}\langle I'_{\lambda}(u_{n}, v_{n}), (\phi^{2}_{z_{j}, \varepsilon}u_{n}, \phi^{2}_{z_{j}, \varepsilon}v_{n})\rangle = 0$, that is,

From (2.16)-(2.20), we get

and

Thus, (2.24)–(2.27) and (2.21) imply that

Moreover, by using the Hölder inequality and boundedness of $\{u_{n}\}$, $\{v_{n}\}$ in $S^{1}_{0}(\Omega)$, we have

Similarly,

Combining with (2.29), (2.30) and (2.28), there holds

On the other hand, the definition of $S_{p_{1}, p_{2}, \gamma}$ implies that

Note that

together with (2.29) and (2.33), we get

Similarly, (2.30) and (2.33) yield that

So, (2.34), (2.35) and (2.32) imply that

Combining (2.36) and (2.31), we have that

which implies that

Now, we consider the possibility of the concentration at $0$. Similarly, we define a cut-off function $\phi\in C^{1}(\mathbb{G}\, , \, [0, 1])$ such that $\phi(z) = 0$ on $B_{d}(0, 1)$, and $\phi(z) = 1$ on $\mathbb{G}\backslash B_{d}(0, 2)$, and set $\phi_{\varepsilon}(z) = \phi(\delta_{\frac{1}{\varepsilon}}(z))$. Then, $\{\phi^{2}_{\varepsilon}u_{n}\}$ is bounded in $S^{1}_{0}(\mathbb{G})$, and $\lim\limits_{\varepsilon\to 0} \lim\limits_{n\to \infty}\langle I'_{\lambda}(u_{n}, v_{n})\, , \, (\phi^{2}_{\varepsilon}u_{n}, 0)\rangle = 0,$ that is,

From (2.18)-(2.20), one can get

and

Thus, (2.38)–(2.41) yield that

Note that

together with (2.42) and (2.43), there holds

On the other hand, by the definition of $S_{\alpha}$ we have

Thus,

Note that

together with (2.45), we have

Therefore, from (2.44) and (2.46), we have

which implies that

Similarly,

Now we claim that (2) and (4), (4)' cannot occur. For this, recall that $(u_{n}, v_{n})\rightharpoonup (u, v)$ weakly in $\mathcal{H}$, by the Brezis-Lieb Lemma we have

and

Then,

On the other hand, from $I'_{\lambda}(u_{n}, v_{n})\to 0$ as $n\to \infty$, we obtain that $I'_{\lambda}(u, v) = 0$. Thus $\langle I'_{\lambda}(u, v), (u, v)\rangle = 0$. Together with $\langle I'_{\lambda}(u_{n}, v_{n}), (u_{n}, v_{n})\rangle\to 0$, there holds

From (2.49) and (2.50) and Lemma 2.5, we have

Passing to the limit in (2.51) as $n\to \infty$, we have

By the assumption $c < c_{\infty}$ and in view of (2.37), (2.47) and (2.48), there holds $\bar{\mu}_{0} = \bar{\nu}_{0} = 0$, $\tilde{\rho}_{z_j} = 0$, $j\in J$. Up to a subsequence, $(u_{n}, v_{n})\to (u, v)$ strongly in $\mathcal{H}$ as $n\to \infty$.

3.   Proof of the main results

This section is devoted to the proof of the main results of this paper.

Theorem 3.1. Under the assumptions of Theorem 1.1, there exists $\Lambda_{2} > 0$ such that problem (1.1) has at least one positive solution for $\lambda\in (0, \Lambda_{2})$ with negative energy.

Proof. By the Hölder inequality, we have

where $t$, $f(t)$ and $g(t)$ are defined by

Note that $2 < 2^*(\alpha), 2^*(\beta), 2^*(\gamma)$, it is easy to see that there exists $t_{0} > 0$ such that $f(t)$ has a maximum at $t_{0}$ and $f(t_{0}) > 0$. Hence, there exists a positive constant $\Lambda_{1}$ such that for all $\lambda\in (0, \Lambda_{1})$,

On the other hand, set $\mathcal{S} = \{(u, v)\in \mathcal{H}:\|(u, v)\|_{\mathcal{H}}\leq t_{0}\}$. For some $(u_{0}, v_{0})\in\mathcal{H}\backslash\{(0, 0)\}$ with $\|(u_{0}, v_{0})\|_{\mathcal{H}} = 1$, we can choose $t > 0$ small enough such that

Consequently, we get

Now we can apply the Ekeland variational principle and obtain a minimizing sequence $\{(u_{n}, v_{n})\}\subset \mathcal{S}$ such that

and

Define $\mathcal{J}_{\lambda}(u, v) = I_{\lambda}(u, v)+\frac{1}{n}\|(u_{n}-u, v_{n}-v)\|_{\mathcal{H}}$. So, (3.4) implies that $\mathcal{J}_{\lambda}(u_{n}, v_{n})\leq \mathcal{J}_{\lambda}(u, v)$, which yields that $\{(u_{n}, v_{n})\}\subset \mathcal{S}$ is the minimizer of $\mathcal{J}_{\lambda}$. In view of (3.1), (3.2) and (3.3), there exists $\varepsilon > 0$ and $N_{0}\in \mathbb{Z}^{+}$ such that for all $n\geq N_{0}$, $\|(u_{n}, v_{n})\|_{\mathcal{H}}\leq t_{0}-\varepsilon$. So, for any $(\phi_{1}, \phi_{2})\in \mathcal{H}$ and $n\geq N_{0}$, there is a $t > 0$ small enough such that

That is,

Passing to the limit in (3.5) as $t\to 0$, we obtain that

which implies that

Combining (3.3) and (3.6), there holds

So, there exists $\Lambda_{2}\in (0, \Lambda_{1})$ such that $\inf_{(u, v)\in S}I_{\lambda}(u, v) < 0 < c_{\infty}$ for all $\lambda\in (0, \Lambda_{2})$. Here $c_{\infty}$ is given in (2.15). Thus, in view of Proposition 2.8, $(u_{n}, v_{n})\to (u_{1}, v_{1})$ strongly in $\mathcal{H}$ for all $\lambda\in (0, \Lambda_{2})$. Hence, $(u_{1}, v_{1})$ is a nontrivial solution of (1.1) satisfying that $I_{\lambda}(u_{1}, v_{1}) = \inf_{(u, v)\in S}I_{\lambda}(u, v) < 0$.

Note that $I_{\lambda}(u_{1}, v_{1}) = I_{\lambda}(|u_{1}|, |v_{1}|)$ and $(|u_{1}|, |v_{1}|)\in \{(u, v)\in \mathcal{H}:\|(u_{n}, v_{n})\|_{\mathcal{H}}\leq t_{0}-\varepsilon \}$, we have $I_{\lambda}(|u_{1}|, |v_{1}|) = \inf_{(u, v)\in S}I_{\lambda}(u, v) < 0$ and $I'_{\lambda}(|u_{1}|, |v_{1}|) = 0$. Then, problem (1.1) has a nontrivial nonnegative solution $(u_{1}, v_{1})\in \mathcal{H}$ with negative energy. According to Bony's maximum principle ^[34], we get that the system (1.1) has a positive solution in $\mathcal{H}$ and completes this proof.

Lemma 3.2. Under the assumptions of Theorem 1.1, there exist a function $(u, v)\in \mathcal{H}\setminus\{(0, 0)\}$ and $\Lambda_{3} > 0$ such that

for all $\lambda$ with $\lambda\in (0, \Lambda_{3})$, where $C_{*}$ is the positive constant given in Lemma 2.5.

Proof. For any $(u, v)\in \mathcal{H}$, write

First, we consider the functional $J: \mathcal{H} \to \mathbb{R}$ as

Let $u: = \sqrt{p_{1}} u_{\varepsilon}$, $v: = \sqrt{p_{2}}u_{\varepsilon}\in S^{1}_{0}(\Omega)$, where $u_{\varepsilon}$ given by (1.5), and define

Then, we know that $\lim\limits_{t\to \infty }\mathcal{J}(t) = -\infty$, and $\mathcal{J}(t) > 0$ as $t\to 0^{+}$. Hence $\sup\limits_{t\geq 0}\mathcal{J}(t)$ is attained at some finite point $t_{0} > 0$ satisfies $\mathcal{J}'(t_{0}) = 0$, that is, $\mathcal{J}$ attains its maximum at

Combining (2.2), (2.3) and (1.6), there holds

Observe that there exists a positive constant $\Lambda_{4}$ such that for all $\lambda\in (0, \Lambda_{4})$, there holds

Then for $\lambda\in (0, \Lambda_{4})$, there exists $t_0 \in (0, 1)$ such that

On the other hand, it follows from $h(z)\geq c_{0}$ and $p_{1}$, $p_{2} > 1$, we obtain

Then, combining (3.9) and (3.12) and (2.5), we get

where $C$ is a positive constant.

Now, we need to distinguish two cases:

${\rm{Case}} \; ({\rm{i}})$ $1\leq q < \frac{Q-\sigma}{Q-2}$. It follows from $q < 2$ that $Q-2 > \frac{q(Q-2)}{2}$. Then, choosing $\varepsilon$ small enough, we can deduce that there exists a $\Lambda_{5} > 0$ such that

for all $\lambda\in (0, \Lambda_5)$. Set $\Lambda_{6} = \min\{\Lambda_{4}, \Lambda_{5}\}$, then (3.13), (3.14) and (3.11) show that

${\text{Case (ii)}} \; \frac{Q-\sigma}{Q-2}\leq q < 2$. It follows from $\frac{Q-\sigma}{Q-2}\leq q$ that $Q-2 > q\frac{Q-2}{2}\geq Q-\sigma-\frac{q(Q-2)}{2}$. Then, for $\varepsilon$ small enough, there exists a $\Lambda_{7} > 0$ such that

Therefore, taking $\Lambda_{8} = \min\{\Lambda_{4}, \Lambda_{7}\}$, we get that for all $\lambda\in (0, \Lambda_{8})$,

Set $\Lambda_{3} = \min\{\Lambda_{6}, \Lambda_{8}\}$, from cases (i) and (ii), (3.8) holds by taking $(u, u) = (\sqrt{p_{1}}\, u_{\varepsilon}, \sqrt{p_{2}}\, u_{\varepsilon})$ and for all $\lambda\in (0, \Lambda_{3})$. The proof is thus complete.

Similarly the proof of Lemma 3.2, we can easy to get the following results.

Lemma 3.3. Under the assumptions of Theorem 1.1, there exist a function $(u, v)\in \mathcal{H}\setminus\{(0, 0)\}$ and $\widehat{\Lambda}_{3} > 0$ such that

Lemma 3.4. Under the assumptions of Theorem 1.1, there exist a function $(u, v)\in \mathcal{H}\setminus\{(0, 0)\}$ and $\widetilde{\Lambda}_{3} > 0$ such that

Theorem 3.5. Under the assumptions of Theorem 1.1, there exists $\widehat{\Lambda}_{1} > 0$ such that problem (1.1) has at least one positive solution for $\lambda\in (0, \widehat{\Lambda}_{1})$ with positive energy.

Proof. We show that the functional $I_{\lambda}$ satisfies the hypotheses of the mountain pass lemma. To this end, obviously $I_{\lambda}(0, 0) = 0$. (3.1) shows that the exist $\rho$, $R_{0} > 0$ such that

for all $\lambda$ with $\lambda\in (0, \Lambda_{1})$.

On the other hand, for $(u, v)\in \mathcal{H}\backslash\{(0, 0)\}$ we obtain that $\lim\limits_{t\to\infty}I_{\lambda}(tu, tv) = -\infty$. Then there exists $l_{0} > 0$ such that $\|(l_{0}u, l_{0}v)\|_{\mathcal{H}} > R_{0}$ and $I_{\lambda}(l_{0}u, l_{0}v) < 0$. Let

where $\Gamma: = \{\gamma\in C([0, 1], \mathcal{H}):\gamma(0) = (0, 0), \gamma(1) = (l_{0}u, l_{0}v)\}$. Thus, it follows from the mountain pass lemma that there exists a sequence $\{(u_{n}, v_{n})\}\subset \mathcal{H}$ such that

Let $\widehat{\Lambda}_{1}: = \min\{\Lambda_{1}, \Lambda_{3}, \widehat{\Lambda}_{3}, \widetilde{\Lambda}_{3}\}$. So Lemmas 3.2, 3.3, 3.4 imply that there exists $(u_{0}, v_{0})\in \mathcal{H}\backslash\{(0, 0)\}$ such that

From Proposition 2.8, $(u_{n}, v_{n})\to (u_{2}, v_{2})$ strongly in $\mathcal{H}$ as $n\to \infty$, which implies that $I_{\lambda}'(u_{2}, v_{2}) = 0$ and $I_{\lambda}(u_{2}, v_{2}) = c$. Then, $(u_{2}, v_{2})$ is a nontrivial solution of (1.1) with positive energy. Set $u^{+}: = \max\{u, 0\}$, $v^{+}: = \max\{v, 0\}$. Replacing $\int_{\Omega}\frac{\psi^{\alpha}|u|^{2^*(\alpha)}}{d(z)^{\alpha}}dz$, $\int_{\Omega}\frac{\psi^{\beta}|v|^{2^*(\beta)}}{d(z)^{\beta}}dz$, $\int_{\Omega}\frac{\psi^{\gamma}|u|^{p_{1}}|v|^{p_{2}}}{d(z, z_{0})^{\gamma}}dz$, $\int_{\Omega}h(z)\frac{\psi^{\sigma}(|u|^{q}+|v|^{q})}{d(z)^{\sigma}}dz$ by $\int_{\Omega}\frac{\psi^{\alpha}(u^{+})^{2^*(\alpha)}}{d(z)^{\alpha}}dz$, $\int_{\Omega}\frac{\psi^{\beta}(v^{+})^{2^*(\beta)}}{d(z)^{\beta}}dz$, $\int_{\Omega}\frac{\psi^{\gamma}(u^{+})^{p_{1}}(v^{+})^{p_{2}}}{d(z, z_{0})^{\gamma}}dz$, $\int_{\Omega}h(z)\frac{\psi^{\sigma}[(u^{+})^{q}+(v^{+})^{q}]}{d(z)^{\sigma}}dz$ in $I_{\lambda}$ respectively, we have that $(u_{2}, v_{2})\in \mathcal{H}$ is a nonnegative solution of (1.1). So by the argument of the proof of theorem 3.1, one gets that $u_{2} > 0$, $v_{2} > 0$. Therefore, we have the desired conclusion.

The ends of this section is devoted to the proofs of the main results of this paper.

Proof of theorem 1.1. Let $\Lambda: = \min\{\Lambda_{2}, \widehat{\Lambda}_{1}\}$. By Theorems 3.1 and 3.5, we known that for all $\lambda\in (0, \Lambda)$, problem (1.1) has at least two positive solution $(u_{1}, v_{1})$ and $(u_{}, v_{2})\in \mathcal{H}$ satisfying

Hence, we get the required result.

Conflict of interest

All authors hereby declare that there are no conflicts of interest in this paper.