Doubly critical problems involving Sub-Laplace operator on Carnot group

1.   Introduction

Recently, Filippucci et al. ^[1] analyzed the following quasilinear elliptic problem with multiple critical terms on the entire $\mathbb{R}^{N}$:

where $\Delta_{p}: = \text{div}(\vert \nabla u\vert^{p-2}\nabla u)$ is the $p$-Laplacian operator, $N\geq 3$, $p\in (1, N)$, $s\in (0, p)$, $\mu\in [0, (\frac{N-p}{p})^{2})$, the value $p^* = \frac{Np}{N-p}$ denotes the critical Sobolev exponents, and $p^*(s) = \frac{p(N-s)}{N-p}$ denotes the critical Hardy-Sobolev exponents. The Eq (1.1) with double critical terms induces more difficulties, and analyzing the structure of the Palais-Smale sequence approaching zero weakly and constructing a new Palais-Smale sequence at a critical value to weakly converge (PS) sequences to a nontrivial function; the authors prove that the Eq (1.1) has at least one positive solution in $W^{1, p}(\mathbb{R}^{N})$. For a similar bi-harmonic problem involving two critical nonlinearities, refer to ^[2]. The author achieved the same result as in ^[1].

Later on, Ghoussoub and Shakerian ^[3] investigated the existence of nontrivial solutions for a fractional Laplacian problem involving critical exponents, namely,

where $s\in (0, 1)$, $0\leq \mu < \mu_{H}$, $\alpha\in (0, 2s)$, $2_{s}^* = \frac{2N}{N-2s}$, and $2_{s}^*(\alpha) = \frac{2(N-\alpha)}{N-2s}$ are the critical exponents and $\mu_{H} = 2^{2s}\frac{\Gamma^{2}(\frac{N+2s}{4})}{\Gamma^{2}(\frac{N-2s}{4})}$ is the best Hardy constant. Due to the nonlocal nature of the fractional Laplace operator, this problem poses more difficulties and, as a result, the authors chose not to study the problem (1.2) directly. Instead, the authors utilized Caffarelli and Silvestre's $s$-harmonic extension method ^[4] to convert (1.2) into a local problem. Again, the fundamental approach utilized by Chen ^[5] to demonstrate the existence of a positive solution to the following fractional Laplacian problem with both critical nonlinearities having the same singularities at origin in enter space $\mathbb{R}^{N}$:

where $s\in (0, 1)$, $0\leq \mu < \mu_H$, $\alpha$, $\beta\in (0, 2s)$, and $2_{s}^*(\cdot) = \frac{2(N-\cdot)}{N-2s}$ denotes the fractional critical Sobolev-Hardy exponent.

Subsequently, Assuncão et al. ^[6] extended the Eq (1.3) to the following fractional $p$-Laplacian problem involving critical Hardy-Sobolev terms in $\mathbb{R}^{N}$:

where $s\in (0, 1)$, $p\in (1, +\infty)$, $sp < N$, $\alpha, \beta\in (0, sp)$, $\mu \in [0, \mu_{H, p})$, and $p^*_s(\alpha) = \frac{p(N-\alpha)}{N-ps}$, $p^*_s(\beta) = \frac{p(N-\beta)}{N-ps}$ denote the critical Hardy-Sobolev exponents. Using a refined version of the concentration-compactness principle and the mountain pass theorem, the authors demonstrate that the problem (1.4) has a nontrivial weak solution in $W^{s, p}(\mathbb{R}^N)$.

We recall that the Hardy inequality on the Stratified Lie group was first introduced in the pioneering work of D'Ambrosio ^[7,8], Han et al. ^[10], and Niu et al. ^[9]. With these inequalities, the subelliptic problem on the Stratified Lie group has received special attention in the past several years. For example, Lioudice ^{[11,12,13,14]} studied the version of Sobolev and Hardy-Sobolev inequalities on the Stratified Lie group and showed the existence result for the Brezis-Nirenberg type equation. Zhang ^[15,16,17] investigated the multiplicity of nontrivial solutions of subelliptic equations with critical Hardy-Sobolev exponents. In ^[18,19,20], the authors studied existence and asymptotic behavior of nontrivial solutions of a series of problems in general open subsets $\Omega$ of the Heisenberg group $\mathbb{H}^{n}$, possibly unbounded or even $\mathbb{H}^{n}$. For the results of the subellipse problem on more general homogeneous groups, we refer to ^[21,22,23] and references therein. Finally, we suggest ^[24] to the reader which is interested on the fractional Laplacian on the Heisenberg group.

Motivated by the results mentioned above, in this article we are interested in finding solutions to the following sub-Laplacian problem with Hardy-type potentials and critical terms on Carnot group $\mathbb{G}$:

where $-\Delta_{\mathbb{G}}$ is the sub-Laplace operator on the Carnot group, $\text{d}(\xi)$ is the natural gauge on $\mathbb{G}$, the weight function $\psi$ is defined as $\psi(\xi) = \vert\nabla_{\mathbb{G}}\text{d}(\xi)\vert$, the parameters $p\in (1, 2^*]$, $\alpha\in (0, 2)$, $\mu\in [0, \mu_{\mathbb{G}})$, and $2^* = \frac{2Q}{Q-2}$ is the critical Sobolev exponent, $2^*(\alpha) = \frac{2(Q-\alpha)}{Q-2}$ is the critical Hardy-Sobolev exponent, $\mu_{\mathbb{G}} = (\frac{Q-2}{2})^{2}$ is the best Hardy constant and $Q$ denotes the homogeneous dimension of the space $\mathbb{G}$ with respect to the dilation $\delta_{\gamma}$; see Section 2. The space $S^{1, 2}(\mathbb{G})$ denotes the completion of $C_{0}^{\infty}(\mathbb{G})$ with respect to norm

Problem (1.5) is related to the following Hardy-type inequality (see ^[8,25]):

where $\mu_{\mathbb{G}} = (\frac{Q-2}{2})^{2}$ is the best constant in this context. By using (1.6), it can be shown that the operator $L: = -\Delta_{\mathbb{G}} \cdot -\mu \frac{ \psi^{2}\cdot}{\text{d}(\xi)^{2}}$ is positive for all $\mu < \mu_{\mathbb{G} }$ and, therefore, we can define the following equivalent norm of $S^{1, 2}(\mathbb{G})$:

Additionally, according to Folland and Stein ^[26], the following Sobolev-type inequality holds:

where the best constant in (1.7) is achieved; refer to ^[27,28]. However, only the explicit form of the minimizers is known for the Iwasawa-type group class. For $\alpha\in [0, 2)$, from (1.6) and (1.7), the following Sobolev-Hardy inequality holds: There exists a positive constant $C(Q, \alpha)$, depending on $Q$ and $\alpha$, such that

The energy functional related to (1.5) takes the following form:

Using the previously mentioned inequalities (1.6) and (1.8), it is straghtforward to show that the functional $\Phi$ is well-defined in $S^{1, 2}(\mathbb{G})$ and $I\in C^1 (S^{1, 2}(\mathbb{G}), \mathbb{R})$. A function $u\in S^{1, 2}(\mathbb{G})$ is said to be a nontrivial solution of (1.5) if $u\neq 0$, and $\langle \Phi'(u), \phi \rangle = 0$ for all $\phi\in S^{1, 2}(\mathbb{G})$, where $\Phi'(u)$ denotes the Fréchet derivative of functional $\Phi$ at $u$.

Now, we can state our result.

Theorem 1.1. Let $\alpha\in (0, 2)$ and $\mu\in (-\infty, \mu_{\mathbb{G}})$. If $u\in S^{1, 2}(\mathbb{G})$ is a weak solution of (1.5) where $1 < p < 2^*$, then $u\equiv 0$.

The result of Theorem 1.1 tells us that we need to discuss the existence of solutions to the Eq (1.5) at $p = 2^*$. The conclusion is as follows:

Theorem 1.2. Let $Q\geq 3$, $\alpha\in (0, 2)$, $\mu \in [0, \mu_{\mathbb{G}})$, and $p = 2^*$. Then, there exists a weak nontrivial solution $u \in S^{1, 2}(\mathbb{G})$ to problem (1.5).

Furthermore, continuing in the same spirit as problem (1.5) with $p = 2^*$, we consider the following subelliptic system with critical homogeneous terms

where $\lambda > 0$, $\eta > 0$, $H_u$, $H_v$, $Q_u$, and $Q_v$ are the partial derivatives of the 2-variable $C^1$-functions $H(u, v)$ and $Q(u, v)$, respectively.

Before stating our result, we need the following assumptions.

$(H_{1})$ $H_{u}(u, 0) = H_{u}(0, v) = H_{v}(u, 0) = H_{v}(0, v) = Q_{u}(u, 0) = Q_{u}(0, v) = Q_{v}(u, 0) = Q_{v}(0, v) = 0$, where $u$, $v\in \mathbb{R}^{+}$.

$(H_2)$ $H\in C^1(\mathbb{R}^{+}\times \mathbb{R}^{+}, \mathbb{R}^{+})$ and $Q\in C^1(\mathbb{R}^{+}\times \mathbb{R}^{+}, \mathbb{R}^{+})$ are positively homogeneous of degrees $2^*$ and $2^*(\alpha)$, respectively, i.e., $H(tu, tv) = t^{2^*}H(u, v)$ and $Q(tu, tv) = t^{2^*(\alpha)}Q(u, v)$ hold for all $t\geq 0$ and $u$, $v\in \mathbb{R}^{+}$.

Now, we work on the product space $W = S^{1, 2}(\mathbb{G})\times S^{1, 2}(\mathbb{G})$ with respect to the norm $\Vert(u, v)\Vert = (\Vert u\Vert^{2}_{\mu}+ \Vert v\Vert^{2}_{\mu})^{\frac{1}{2}}$, and get the following existence result for system (1.10).

Theorem 1.3. Suppose that $\mu \in [0, \mu_{\mathbb{G}})$, $\alpha\in (0, 2)$, $\lambda > 0$, $\eta > 0$, and $(H)_{1}$, $(H_2)$ hold. Then, the system (1.10) has a nontrivial weak solution in $W$.

Remark 1.1. By Theorem 1.3, the existence of solutions to (1.10) is obvious in either of the following cases: (i) $\lambda = 0$, $\eta > 0$, $\alpha\geq 0$; (ii) $\lambda > 0, \ \eta = 0$, $\alpha\geq 0$; (iii) $\alpha = 0$, $\lambda > 0$, $\eta > 0$.

The proof of Theorems 1.2 and 1.3 follow several ideas that have appeared in ^[1,3,6]. However, since we consider the subelliptic problem on Carnot group $\mathbb{G}$ and since problem (1.5) or (1.10) contains critical nonlinearities in the sense of the Hardy-Sobolev embeddings, it follows that the Hardy-Sobolev embedding $S^{1, 2}(\mathbb{G})\hookrightarrow L^{2^*(\alpha)}(\mathbb{G}, \frac{\psi^{\alpha}}{d(z)^{\alpha}}dz)$ ($0\leq \alpha < 2$) is non-compact. This poses several difficulties to prove that bounded Palais-Smale in Banach space $S^{1, 2}(\mathbb{G})$ have at least a subsequence that converges strongly to a nontrivial function in this space. Clear enough, the presence of multiple Sobolev critical nonlinearities also contributes to the difficulties in the proof of the theorem. Based on some estimates proved by Zhang ^[15,16], we managed to overcome these difficulties and prove a refined version of the concentration-compactness principle.

The article is organized as follows. In Sections 2 and 3, some preliminary results together with our main results are verified. Meanwhile, for existence of nontrivial weak solutions, Theorems 1.2 and 1.3 will be proved in Sections 4 and 5, respectively.

2.   Preliminary results

First, we will provide a brief overview of Carnot groups. For a more comprehensive treatment of this topic, please reference the monographs ^[29,30] and the papers ^[26,31]. A Carnot group $(\mathbb{G}, \circ)$, also known as a stratified Lie group, is defined as a connected, simply connected nilpotent Lie group, whose Lie algebra $\mathfrak {g}$ is stratified. Specifically, this means that $\mathfrak{g}$ can be decomposed as $\mathfrak{g} = \oplus_{i = 1}^{k}V_i$, where $[V_1, V_i] = V_{i+1}$ for $i = 1, \cdots k-1$ and $[V_{1}, V_k] = \{0\}$. The number $k$ is called the step of the group $\mathbb{G}$. In this context, the symbol $[V_1, V_i]$ represents the subalgebra of $\mathfrak {g}$ generated by the commutators $[X, Y]$, where $X\in V_{1}$, $Y\in V_{i}$ and the last bracket denotes the Lie bracket of vector fields, i.e., $[X, Y] = XY-YX$.

By means of the natural identification of $\mathbb{G}$ with its Lie algebra via the exponential map (which we shall assume throughout), it is reasonable to assume that $\mathbb{G}$ is a homogeneous Lie group on $\mathbb{R}^{N} = \mathbb{R}^{N_1}\times \mathbb{R}^{N_2}\times\cdots\times \mathbb{R}^{N_k}$, where $N_{i}$ denotes the dimensionality of $V_{i}$, and is equipped with a set of group-automorphisms called $\delta _\gamma :\mathbb{G}\to \mathbb{G}$. These automorphisms take the form of

where $\xi^{(i)}\in \mathbb{R}^{N_i}$ for $i = 1, 2, \cdots, k$. Here, $N = \sum_{i = 1}^{k}N_{i}$ is called the topological dimension of $\mathbb{G}$ and $\delta_{\gamma}$ is called the dilations of $\mathbb{G}$. Under these automorphisms $\{\delta_{\gamma}\}_{\gamma > 0}$, the homogeneous dimension of $\mathbb{G}$ is expressed as $Q = \sum_{i = 1}^{k}i\cdot \text{dim}V_{i}.$ From now on, we will assume that $Q\geq 3$ throughout this paper. It is noteworthy that if $Q\leq 3$, $\mathbb{G}$ must be the ordinary Euclidean space $\mathbb{G} = (\mathbb{R}^{Q}, +)$.

Let $\{X_1, \cdots, X_{N_1}\}$ be a basis of $V_1$, then the second-order differential operator

is referred to as a sub-Laplacian on $\mathbb{G}$. We now use the notation of $\nabla_\mathbb{G}: = (X_1, \cdots, X_{N_1})$ to denote the horizontal gradient, and the divergence with respect to the vector fields $X_{j}$ is defined by

The homogeneous norm on $\mathbb{G}$, which conforms to a fixed homogeneous structure, is a continuous function represented by $\text{d}:\mathbb{G}\to [0, +\infty)$. This function is smooth away from the origin and satisfies $\text{d}(\delta_{\gamma}(\xi)) = \gamma \text{d}(\xi)$ for $\gamma > 0$, $\text{d}(\xi^{-1}) = \text{d}(\xi)$, $\text{d}(\xi) = 0$ iff $\xi = 0$. When $Q\geq 3$, the function

is a fundamental solution of the sub-Laplacian on Carnot group $\mathbb{G}$ with the pole at $0$, where $C > 0$ is a suitable constant. In addition, the left translation on $\mathbb{G}$ is defined by

and we can verify that $\nabla_\mathbb{G}$ and $\Delta_{\mathbb{G}}$ satisfy the following results:

The $k\, (k\geq 2)$-step Carnot group $\mathbb{G}$ and the Euclidean space $\mathbb{R}^{N}$ differ in numerous essential ways. For instance, the basis level vector field on $\mathbb{G}$ is noncommutative, meaning that there exist $1\leq i, \, j\leq m$ such that the Poisson bracket $[V_{i}, V_{j}]\neq 0$. In constrast, $\mathbb{R}^{N}$ is an exchange group with a step number 1, which means that for any $i, \, j = 1, 2, \cdots, \dim(V_{1})$, whose Poisson brackets satisfy $[\frac{\partial}{\partial x_{i}}, \frac{\partial}{\partial x_{j}}] = 0$. Therefore, there are several significant differences between the operator on Carnot group and on the Euclidean space. For example, the Laplace operator on $\mathbb{G}$ is $\Delta_{\mathbb{G}} = \sum_{i = 1}^{\dim(V_{1})}X_{i}^{2}$, which is a point-by-point degenerate elliptic operator. In contrast, the Laplace operator on $\mathbb{R}^{N}$, $\Delta = \sum_{i = 1}^{N}\frac{\partial^{2}}{\partial x_{i}^{2}}$ is a uniformly elliptic operator. Therefore, the study of partial differential equations on the Carnot group is of theoretical importance.

By (1.6) and (1.8), the following best Hardy-Sobolev constant is well-defined:

For $\mu\in [0, \mu_{\mathbb{G}})$, it can be inferred from ^[15] that $S_{\mu, \alpha}$ is achieved by the extremal functions

where $U_{\mu, \alpha}$ is a ground state solution of

Furthermore, for all $\varepsilon > 0$, the function $U_{\varepsilon, \mu, \alpha}(\xi)$ solves the Eq (2.3) and satisfies

We note that the explicit form of the Hardy-Sobolev extremals is unknown in any Carnot group, except for the trivial Euclidean case. However, the pure Sobolev extremals (when $\mu = \alpha = 0$) are known to be expressed solely in the Iwasawa-type group, as seen in ^[27,32].

For $\mu\in (-\infty, \mu_{\mathbb{G}})$ and $\alpha\in (0, 2)$, $(H_{2})$ shows that the following best Hardy-Sobolev constants are well-defined:

These constants are crucial for the study of (1.10); we then have the following result.

Theorem 2.1. Assume that $\alpha\in(0, 2)$, $\mu \in (-\infty, \mu_{\mathbb{G}})$, and $(H_2)$ holds. Then,

(i) $S_H(\mu, 0) = M_H^{-1}S_{\mu, 0}$ and $S_Q(\mu, \alpha) = M_Q^{-1}S_{\mu, \alpha}$, where $M_{H}$, $M_{Q}$ are defined by

(ii) For $\mu \in [0, \mu_{\mathbb{G} })$, $S_H(\mu, 0)$ has the minimizers $(s_1 U_{\varepsilon, \mu, 0}(\xi), t_1 U_{\varepsilon, \mu, 0}(\xi))$, $S_Q (\mu, \alpha)$ has the minimizers $(s_2 U_{\varepsilon, \mu, \alpha}(\xi), t_2 U_{\varepsilon, \mu, \alpha}(\xi))$, where $U_{\varepsilon, \mu, \alpha}(\xi)$ are defined as in (2.2) and $(s_{1}, t_{1})$, $(s_{2}, t_{2})$ are constants given in (2.8), (2.9), respectively.

Now, we study $S_H(\mu, 0)$, $S_Q(\mu, \alpha)$ and verify Theorem 2.1. First, we give some preliminary results.

Proposition 2.1. (^[33]) Let $H\in C^{1}(\mathbb{R}\times \mathbb{R}, \mathbb{R}^{+})$ and $Q\in C^{1}(\mathbb{R}\times \mathbb{R}, \mathbb{R}^{+})$ be positively homogeneous of degrees $2^*$ and $2^*(\alpha)$, respectively. Then, there exist $M_{H}$, $M_{Q} > 0$ such that

where $M_{F}$ and $M_{Q}$ are given in (2.6) and (2.7), respectively. Moreover, there exist $(s_{i}, t_{i})\in \mathbb{R}^{+}\times\mathbb{R}^{+}$ ($i = 1, 2$), such that $M_{F}$ and $M_{Q}$ are achieved respectively, that is,

Proof of Theorem 2.1. We only show the proof for $S_Q(\mu, \alpha)$.

(ⅰ) Let $\{U_{n}\}\subset S^{1, 2}(\mathbb{G}) \setminus \{0\}$ be a minimizing sequence for $S_{\mu, \alpha}$ and $(s_2, t_2)$ be defined as in (2.9). Choosing $(u_{n}, v_{n}) = (s_2U_{n}, t_2U_{n})$ in (2.5), we have

Taking $n\to \infty$ in (2.10), by (2.9) we have

On the other hand, let $\{(u_{n}, v_{n})\}\subset W\backslash \{(0, 0)\}$ be a minimizing sequence for $S_{Q}(\mu, \alpha)$. From $Q(tu, tv) = t^{2^*(\alpha)}Q(u, v)$ and Proposition 4 of ^[33], we have that

Set

Then,

From (2.1), (2.12), (2.13), and (2.9), it follows that

Passing to the limit in the above inequality (2.14), we have

which together with (2.11) implies that

(ⅱ) From (ⅰ), (2.4), and (2.5), the desired result follows. □

3.   Nonexistence result when $p < 2^*$

In order to prove Theorem 1.1, we first establish it under an additional assumption.

Proposition 3.1. Let $\alpha\in (0, 2)$ and $\mu\in (-\infty, \mu_{\mathbb{G}})$. If $u\in L^p(\mathbb{G})$ is a weak solution of (1.5) with $p\in (1, 2^*)$, then $u\equiv 0$.

Proof. Let $\phi$, $\zeta\in C^{1}(\mathbb{G}, [0, 1])$ and satisfy $\phi(t) = 1$ for $t\geq 2$, $\zeta(t) = 1$ for $t\leq 1$. Let $\eta_{\varepsilon, R}(\xi) = \phi_{\varepsilon}(\xi)\zeta_{R}(\xi)$ for $\varepsilon > 0$ and $R > 0$, where

Let $u\in S^{1, 2}(\mathbb{G})$ be a weak solution of (1.5) with $1 < p < 2^*$. Then, $u$ is smooth away from the origin and $Z u\eta_{\varepsilon, R}\in C^1_{0}(\mathbb{G})$. By multiplying the Eq (1.5) with $Z u\eta_{\varepsilon, R}$ and integrating by parts, we get

Proceeding similarly as proved in [14, Theorem 4.1], we can show that

and

Therefore, substituting back (3.3) and (3.2) in (3.1), we obtain

On the other hand, since $u\in L^{p}(\mathbb{G})$ is a solution of (1.5), we have

which together with (3.4) implies that

As $p < 2^*$, i.e., $\frac{Q-2}{2}-\frac{Q}{p} < 0$, (3.5) implies $u\equiv 0$. This completes the proof. □

Proof of Theorem 1.1. According to Proposition 3.1, once we prove $u\in L^{p}(\mathbb{G})$, the proof of Theorem 1.1 follows.

Now, let $\eta_{\varepsilon, R}\in C^{\infty}_0(\mathbb{G}\setminus \{0\})$ be a cutoff function as in the proof of Proposition 3.1. Choosing $\eta_{\varepsilon, R}u$ as the test function, we get

Hence,

Since $u\in S^{1, 2}(\mathbb{G})$, there exists a constant $C > 0$ such that $\int_{\mathbb{G}}\vert\nabla_{\mathbb{G}} u\vert^2 d\xi\leq C$. Then, based on the Hardy inequality and the Sobolev-Hardy inequality, we can conclude that $\int_{\mathbb{G}}\frac{\psi^{2}(\xi)\vert u\vert^2}{\text{d}(\xi)^2}d\xi\leq C_{1}$ and $\int_{\mathbb{G}}\frac{\psi^{\alpha}(\xi)\vert u\vert^{2^{*}(\alpha)}}{\text{d}(\xi)^{\alpha}}d\xi\leq C_{2}$, where $C_{1} > 0$ and $C_{2} > 0$ are constants. In order to prove $u\in L^{p}(\mathbb{G})$, our aim is to show that $\int_{\mathbb{G}}\vert u\vert \vert\nabla_{\mathbb{G}} u\vert \vert \nabla_{\mathbb{G}}\eta_{\varepsilon, R}\vert d\xi$ are uniformly bounded by a constant independent of $\varepsilon$ and $R$. To see this,

Here, we use the fact that $\frac{1}{\varepsilon}\leq\frac{2}{\text{d}(\xi)}$ in the first integral and $\frac{1}{R}\leq\frac{2}{\text{d}(\xi)}$ in the second integral. By the Hölder inequality and the Hardy-Sobolev inequality, for $u\in S^{1}(\mathbb{G})$, there exist $C_{1}$, $C_{2} > 0$ such that

and

So, from (3.9), (3.10), (3.8), and (3.7), we get $\int_{\mathbb{G}}\vert u\vert^{p}\eta_{\varepsilon, R} d\xi\leq C$, where $C$ is a positive constant independent of $\varepsilon$ and $R$. Therefore, letting $\varepsilon\to 0$ and $R\to\infty$, we obtain that $u\in L^{p}(\mathbb{G})$. Hence, the Theorem 1.1 follows. □

4.   Proof of Theorem 1.2

This section is devoted to proving the Theorem 1.2. To begin with, we use the following mountain pass lemma of Ambrosetti and Rabinowitz ^[34] to prove Theorem 1.2.

Lemma 4.1. Let $(E, \Vert\cdot \Vert_{E})$ be a Banach space and $I \in C^1(E, \mathbb{R})$, satisfying the following conditions:

(ⅰ) $I (0) = 0$.

(ⅱ) There exist $a > 0$, $R > 0$ such that $I(u)\geq a$ for all $u\in E$ with $\Vert u\Vert_E = R$.

(ⅲ) There exists $u_0\in E\backslash\{0\}$ such that $\lim \sup _{ t\to \infty }I(t u_0) < 0$.

Let $t_0 > 0$ be a real number such that $\Vert t_0 u_0\Vert_E > R$ and $I (t_0 u_0) < 0$. Define

where

Then, $c\geq a > 0$ and there exists a $(PS)$-sequence $\{u_{n}\}\subset E$ for $I$ at the level $c$, i.e.,

Proposition 4.1. Let $\mu\in [0, \mu_{\mathbb{G}})$, $\alpha\in (0, 2)$, and $p = 2^*$. Then, there is a $(PS)$-sequence $\{u_{n}\}\subset S^{1, 2}(\mathbb{G})$ for $\Phi$ at some $c\in (0, c^*)$, i.e.,

where

The proof of Proposition 4.1 follows from the next results.

Lemma 4.2. The energy functional $\Phi$ verifies the hypotheses of Lemma 4.1 for any $u\in S^{1, 2}(\mathbb{G})\backslash\{0\}$.

Proof. Clearly, $\Phi\in C^1(S^{1, 2}(\mathbb{G}), \mathbb{R})$ and $\Phi(0) = 0$. By (2.1), we have

Since $2^* > 2$ and $2^*(\alpha) > 2$ for all $\alpha\in (0, 2)$, there exist $R > 0$ and $a > 0$ such that $\Phi(u)\geq a$ for all $u\in S^{1, 2}(\mathbb{G})$ with $\Vert u\Vert_{\mu} = R$ small enough.

Let $u_{0}\in S^{1, 2}(\mathbb{G}) \backslash\{0\}$. For $t > 0$, we have

which implies that $\Phi(tu_{0})\to -\infty$ as $t\to +\infty$. So, there exists $t_{u_{0}} > 0$ such that $\Vert t_{u_{0}}u_{0}\Vert_{\mu} > R$ and $\Phi(tu_{0}) < 0$ for all $t > t_{u_{0}}$.

Now, we can define

where $\Gamma_{u_{0}} : = \{ \gamma \in C([0, 1], S^{1, 2}(\mathbb{G})):\gamma (0) = 0 \text{ and } \gamma (1) = t_{u_{0}}u_{0}\}$. Consequently, $\Phi$ possesses the hypotheses of Lemma 4.1. □

From Lemmas 4.1 and 4.2, for $u\in S^{1, 2}(\mathbb{G}) \setminus \{0\}$, we define

where

Then, $c_{u}\geq a > 0$ for $u\in S^{1, 2}(\mathbb{G}) \setminus \{0\}$, and there is a $(PS)$-sequence $\{u_n\}\subset S^{1, 2}(\mathbb{G})\backslash\{0\}$ for $\Phi$ at level $c_{u}$, that is,

Lemma 4.3. Let $\mu \in [0, \mu_{\mathbb{G}})$, $\alpha \in (0, 2)$, and $p = 2^*$. Then, there exists a $u \in S^{1, 2}(\mathbb{G}) \setminus \{0\}$ such that $0 < c_u < c^*$, where $c^*$ is defined in (4.1).

Proof. Let $u(\xi) = U_{\varepsilon, \mu, 0}(\xi)$ be the extremal function of $S_{\mu, 0}$ as in (2.2). By the definition of $c_{u}$, we get

where $f:\mathbb{R}^{+}\to \mathbb{R}$ is defined by

Note that

this and (4.3) imply that

Now, we will show that the equality does not hold in (4.3). Otherwise, we would have that $\sup _{t\ge 0} \Phi(tu) \leq \sup _{t\ge 0}f(t)$. Let $t_1$, $t_2 > 0$ where $\sup_{t\geq 0} \Phi(tu)$ and $\sup_{t\geq 0}f(t)$ are attained, respectively. We get

which implies that $f(t_2) < f(t_1)$ since $u\neq 0$ and $t_{1} > 0$. This contradicts the fact that $t_2$ is the unique maximum point of $f$. Thus,

Similarly,

This completes the proof of Lemma 4.3. □

Proof of Proposition 4.1. From Lemmas 4.1, 4.2, and 4.3, it follows the conclusions of Proposition 4.1 for a suitable $u\in S^{1, 2}(\mathbb{G})\backslash\{0\}$. □

Proposition 4.2. Let $\mu \in [0, \mu_{\mathbb{G}})$, $\alpha \in (0, 2)$, and $p = 2^*$, and let $\{u_n\}\subset S^{1, 2}(\mathbb{G})$ be a $(PS)_{c}$-sequence at some $c\in (0, c^*)$. If $u_n\rightharpoonup 0$ weakly in $S^{1, 2}(\mathbb{G})$ as $n\to \infty$, then there exists $\varepsilon_0 > 0$ such that for $r > 0$, one of the following limits is valid:

where $B_{\mathit{\text{d}}}(0, r)$ denotes the ball with center at $0$ and radius $r$ with respect to the gauge $\mathit{\text{d}}$.

Lemma 4.4. Let $\mu \in [0, \mu_{\mathbb{G}})$, $\alpha \in (0, 2)$, and $p = 2^*$, and let $\{u_n\}$ be a $(PS)_{c}$-sequence for $\Phi$ with $c\in (0, c^*)$. If $u_n \rightharpoonup 0$ in $S^{1, 2}(\mathbb{G})$ as $n\to \infty$, then for every compact subset $\Omega \subset \subset \mathbb{G} \setminus \{0\}$, up to a subsequence, we have

and

Proof. Let $\Omega \subset \subset \mathbb{G} \setminus \{0\}$ be a fixed compact subset. Since the embedding $S^{1, 2}(\mathbb{G}) \hookrightarrow L^p(\Omega)$ is compact for any $p\in [1, 2^*)$, and $\frac{\psi^{2}(\xi)}{\text{d}(\xi)^{2}}$, $\frac{\psi^{\alpha}(\xi)}{\text{d}(\xi)^{\alpha}}$ are bounded on $\Omega$, (4.4) follows at once being $2^*(\alpha) < 2^*$ and $u_n\rightharpoonup 0$ in $S^{1, 2}(\mathbb{G})$.

Now, we verify (4.5). Arguing as the proof of Proposition 2 in ^[1], let $\phi \in C_0^\infty (\mathbb{G}\setminus \{0\})$ be a cutoff function satisfying $\text{supp}\phi\subset \subset \mathbb{G}\setminus \{0\}$, $0\leq \phi \leq 1$, and $\phi = 1$ for all $z\in \Omega$. Then, from (4.4) we have

where $o_{n}(1)$. From now on, it is such that $o_{n}(1)\to 0$ as $n\to \infty$. By the Hölder inequatity and $u_n\rightharpoonup 0$ in $S^{1, 2}(\mathbb{G})$, we have

Combining with (4.6) and (4.7), there holds

that is,

On the other hand,

which yields that

Consequently, this together with (4.8) implies that

If $\lim\limits_{n\to \infty} \int_{\mathbb{G}}\vert\nabla_{\mathbb{G}} (\phi u_n)\vert^2 d\xi\neq0$, it follows from (4.11) that

Then, we have $\lim\limits_{n\to \infty }\int_{\mathbb{G}}\vert\nabla_{\mathbb{G}} (\phi u_n)\vert^2 d\xi = 0$, which this and $\phi\vert_{\Omega} = 1$ imply that

Therefore, the above equality and Sobolev embedding yield $\lim\limits_{n\to \infty } \int _{\Omega} \vert u_{n}\vert^{2^*}d\xi = 0$, and Lemma 4.4 is proved. □

Remark 4.1. From (4.9), we will get that

Let $r > 0$ be fixed. From Lemma 4.4, the following quantities are well-defined:

Lemma 4.5. Let $\{u_n\}\subset S^{1, 2}(\mathbb{G})$ be a $(PS)_{c}$-sequence for $\Phi$ with $c \in (0, c^*)$. If $u_n\rightharpoonup 0$ in $S^{1, 2}(\mathbb{G})$ as $n\to \infty$, then

Proof. Let $\phi \in C_0^\infty (\mathbb{G})$ be a cutoff function such that $0\leq \phi \leq 1$ and $\phi \vert_{B_{\text{d}}(0, r)} \equiv 1$. Then,

As $n\to \infty$, Lemma 4.4 implies that

Consequently, $S_{\mu, 0}\cdot \gamma^\frac{2}{2^*} \leq \beta$. The second inequality in (4.13) can be verified similarly.

Notice that $\phi u_n\in S^{1, 2}(\mathbb{G})$ and $\lim\limits_{n\to \infty } \langle \Phi'(u_n), \phi u_n\rangle = 0$. Via a similar argument as in (4.6), we get that

and the definitions of $\phi$ and (4.12) deduce that $\beta\leq\gamma+\nu$. Lemma 4.5 is verified. □

Proof of Proposition 4.2. From (4.13), it follows that $S_{\mu, 0}\cdot \gamma^\frac{2}{2^*}\leq \beta \leq\gamma+ \nu$, which implies that $S_{\mu, 0}\cdot \gamma^\frac{2}{2^*}-\gamma\leq \nu$, that is,

On the other hand, from (4.10) and $c < c^{*}$, we have that

So, $S_{\mu, 0}-\gamma^{1-\frac{2}{2^*}} > 0$, namely, there is a constant $C_1 = C_1(\mu, c, Q) > 0$ such that

Similarly,

for some constant $C_2 = C_2(\mu, c, \alpha, Q) > 0$. Then, combining with (4.15) and (4.16), there holds

where $\varepsilon_0 = \varepsilon_0(Q, \mu, \alpha)$ is a positive constant. This completes the proof of Proposition 4.2. □

Remark 4.2. The Proposition 4.2 states that every $(PS)_{c}$-sequence $\{u_n \}\subset S^{1, 2}(\mathbb{G})$ for $\Phi$ with $c\in (0, c^*)$ such that $u_n \rightharpoonup 0$ weakly in $S^{1, 2}(\mathbb{G})$ as $n\to \infty$ verifies one of the following limits:

with arbitrary $r > 0$ and a constant $\varepsilon_0$ independent on $r$.

Proof of Theorem 1.2. Let $\{u_n\}$ be a $(PS)_{c}$-sequence for $\Phi$ with $c\in (0, c^{*})$ such that $u_n \rightharpoonup 0$ in $S^{1, 2}(\mathbb{G})$ as $n\to \infty$. Then, we have that

and

Now, we claim that $\limsup\limits_{n\to \infty } \int_{\mathbb{G}} \vert u_{n}\vert^{2^*} d\xi > 0$. Arguing by contradiction, we assume that $\int_{\mathbb{G}}\vert u_{n}\vert^{2^*} d\xi = o_{n}(1)$. Then, (4.17) and (4.18) imply that

From (4.19) and the definition of $S_{\mu, \alpha}$, we get that

that is,

On the other hand, (4.20) and $c < c^*$ yield that

which together with (4.21) implies that $\int_{\mathbb{G}}\psi^{\alpha}(\xi) \frac{\vert u_{n}\vert^{2^*(\alpha)}}{\text{d}(\xi)^{\alpha}}d\xi = o_{n}(1)$, a contradiction with (4.18) and $c > 0$.

Set $\varepsilon_1 = \min \{ \frac{\varepsilon _0}{2}, \limsup\limits_{n\to \infty } \int_{\mathbb{G}} \vert u_{n}\vert^{2^*} d\xi\}$, where $\varepsilon_0$ is given in Proposition 4.2. Let $\varepsilon \in (0, \varepsilon_1)$. From Proposition 4.2 up to a subsequence still denoted by $\{u_n\}$, for $n\in \mathbb {N}$, there exists $r_n > 0$ such that

Let $\widehat{u}_{n}(\xi) = r_n^{\frac{Q-2}{2}} u_n(\delta_{r_n}(\xi))$. Then, $\widehat{u}_{n}\in S^{1, 2}(\mathbb{G})$ satisfies

Moreover, it is easy to see that $\{\widehat{u}_{n}\}$ is again a $(PS)$-sequence of the type given in Proposition 4.2. So, we have that

which implies that $\{\widehat{u}_n\}$ is bounded in $S^{1, 2}(\mathbb{G})$. Then, up to a subsequence, there exists $\widehat{u}\in S^{1, 2}(\mathbb{G})$ such that $\widehat{u}_k \rightharpoonup \widehat{u}$ weakly in $S^{1, 2}(\mathbb{G})$, $L^{2^*(\alpha)}(\mathbb{G}, \frac{\psi^{\alpha}(\xi)}{\text{d}(\xi)^{\alpha}}d\xi)$, and $L^{2^*}(\mathbb{G})$ as $n\rightarrow +\infty$. So, for any $\phi \in S^{1, 2}(\mathbb{G})$, we have

which concludes that $\widehat{u} \in S^{1, 2}(\mathbb{G})$ is a solution of problem (1.5). In addition, if $\widehat{u}\equiv 0$, Proposition 4.2 implies that either

which contradicts (4.24) as $0 < \varepsilon < \frac{\varepsilon_0}{2}$. Then, $\widehat{u}\not = 0$ and the proof of Theorem 1.2 is complete. □

5.   Proof of Theorem 1.3

In this section, we show that system (1.10) has a nontrivial weak solution. Observe that the corresponding functional of (1.10) can be written as

By the standard arguments, we can verify $I\in C^1 (W, \mathbb{R})$. A critical point of functional $I$ in $W$ is a weak solution to (1.10). We say that a pair of functions $(u, v)\in W$ is a nontrivial solution of (1.10) if $(u, v)\neq (0, 0)$ and $\langle I'(u, v), (\phi_{1}, \phi_{2}) \rangle = 0$ for all $(\phi_{1}, \phi_{2}) \in W$.

We point out that the proof of Lemma 4.2 provides us with a tool to show that the functional $I$ has a mountain pass geometrical, that is,

(ⅰ) $I (0, 0) = 0$

(ⅱ) There exist $R$, $\rho > 0$ such that $I (u, v)\geq \rho > 0$ for $(u, v)\in W\backslash\{(0, 0)\}$ with $\Vert (u, v)\Vert_{W} = R$.

(ⅲ) There exists $(u_{0}, v_0)\in W\backslash\{(0, 0)\}$ such that $\lim\limits_{ t\to \infty }I(t(u_{0}, v_0)) < 0$.

Define

where $\Gamma : = \{\gamma \in C([0, 1], W): \gamma (0) = 0 \, \, \text {and} \, I(\gamma (1)) < 0 \}$. Then, there exists a sequence $\{(u_{n}, v_{n})\}\subset W$ such that

where $c\in (0, c^{**})$ and

Proposition 5.1. Let $\{(u_n, v_n) \}\subset W$ be a $(PS)_{c}$-sequence for $I$ with $c\in (0, c^{**})$. If $(u_n, v_{n}) \rightharpoonup (0, 0)$ weakly in $W$ as $n\to \infty$, then there exists $\tilde{\varepsilon}_0 > 0$ such that for all $r > 0$, either

Proof. The argument used is similar to that of Section 4, and for completeness we give the following argument. We first show the following results held for any compact subset $\Omega\subset \mathbb{G} \setminus \{0\}$:

Arguing as Lemma 4.4, for $\Omega \subset \subset \mathbb{G} \setminus \{0\}$, (5.1) and (5.2) follow from the properties of the homogeneous function in Proposition 2.1, the compact embedding $S^{1, 2}(\mathbb{G}) \hookrightarrow L^p(\Omega)$ for $p\in [1, 2^*)$, and the fact that $\frac{\psi^{2}(\xi)}{\text{d}(\xi)^{2}}$, $\frac{\psi^{\alpha}(\xi)}{\text{d}(\xi)^{\alpha}}$ are bounded on $\Omega\backslash\{0\}$. Thus, it remains to show that (5.3) and (5.4) hold.

Let $\phi \in C_0^\infty (\mathbb{G}\setminus \{0\})$ be a cutoff function such that $\text{supp}\phi\subset \subset \mathbb{G}\setminus \{0\}$, $0\leq \phi \leq 1$, and $\phi\vert_{\Omega} = 1$. Note that the weak convergence of $\{ u_n\}$ and $\{v_n\}$ in $S^{1, 2}(\mathbb{G})$ implies the boundedness. Then,

and

From the latest inequalities and (5.1), (5.2), we get that

which implies that

and, therefore,

In addition, since $c+o_{n}(1) = I(u_n, v_n) - \frac{1}{2} \langle I'(u_n, v_n), (u_n, v_n) \rangle \geq \frac{\lambda}{Q}\int _{\mathbb{G}} H(u_{n}, v_{n}) d\xi$ and the upper bounded on $c$ yields

Consequently, (5.6) and (2.4) imply that

which together with (5.5) implies that

Then, the definition of $\phi$ implies that (5.3) and (5.4) hold.

Now, let us define

where $r > 0$ is fixed. From Lemma 4.5, we can deduce that the above quantities are well-defined and independent of $r$. If $(u_n, v_{n}) \rightharpoonup (0, 0)$ weakly in $\mathcal{H}$ as $n\to \infty$, we have the following results:

From (5.10), it follows that

which implies that

On the other hand, since $\frac{\lambda}{Q}\int _{\mathbb{G}} H(u_{n}, v_{n}) d\xi\leq c+o_{n}(1)$, we get that $\lambda\tau \leq cQ < c^{**} Q < \lambda^\frac{2-Q}{2} S_H(\mu, 0)^\frac{2^*}{2^*-2}$, and (5.11) yields that there exists a constant $C_1 = C_1(\mu, c, \lambda, \eta) > 0$ such that

Similarly, there exists $C_2 = C_2(\mu, c, \alpha, \lambda, \eta) > 0$ such that

Based on inequalities (5.12) and (5.13), we can find a constant $\tilde{\varepsilon}_0 = \varepsilon_0(Q, \mu, c, \alpha) > 0$ such that either $\tau = \omega = 0$ or $\min \{\tau, \omega \}\geq \tilde{\varepsilon}_0$. This proves Proposition 5.1. □

Proof of Theorem 1.3. Choosing the sequence $\{(u_n, v_n) \}\subset W$ defined as in Proposition 5.1, proceeding as in proof of Theorem 1.2, we have $\limsup\limits_{n\to \infty } \int_{\mathbb{G}} H(u_{n}, v_{n})d\xi > 0$. Then, there exists $\tilde{\varepsilon}_1 = \min\{\Lambda, \frac{\tilde{\varepsilon}_0}{2}\}$, such that for $\varepsilon \in (0, \tilde{\varepsilon}_1)$, there exists a positive real sequence $\{r_n\}$ such that

is again a $(PS)_{c}$-sequence of the type given in Proposition 5.1 and satisfies

Moreover, for the $(PS)_{c}$ sequence $\{ (\widetilde{u}_n, \widetilde{v}_n)\}$, we get

which implies that $\{ (\tilde{u}_n, \tilde{v}_n)\}$ is bounded in $W$. Up to a subsequence, there exist $\tilde{u}, \tilde{v} \in S^{1, 2}(\mathbb{G})$ such that $\widetilde{u}_n \rightharpoonup \tilde{u}$, $\widetilde{v}_n \rightharpoonup \tilde{v}$ weakly in $S^{1, 2}(\mathbb{G})$ as $n\to \infty$. Similar to the proof of Theorem 1.2, we can show that $(\tilde{u}, \tilde{v})\not \equiv (0, 0)$. Thus, there exists a nontrivial weak solution to system (1.10).

□

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Conflict of interest

The authors declare there is no conflicts of interest.