Existence and multiplicity results for a singular fourth-order elliptic system involving critical homogeneous nonlinearities

1.   Introduction

The purpose of this paper is to deal with the following fourth-order elliptic system

where $\Delta^{2} = \Delta(\Delta)$, $N > 4$, $0\leq\mu < \overline{\mu}$ with $\overline{\mu}\triangleq\frac{1}{16}N^{2}(N-4)^{2}$, $0\in\Omega$ is a smooth bounded domain of $\mathbb{R}^{N}$, $\Omega$ and the weight $K\in \mathscr {C}(\overline{\Omega})\cap L^{\infty}(\overline{\Omega})$ verify some invariant conditions with respect to a closed subgroup $T$ of $O(N)$, and $O(N)$ is the group of orthogonal linear transformations in $\mathbb{R}^{N}$, which will be described later. $\frac{\partial}{\partial n}$ is the outer normal derivative, $\sigma\geq 0$, $0\leq\beta < 4$, $2 < q < 2^{\ast\ast}(\beta)$ with $2^{\ast\ast}(\beta)\triangleq\frac{2(N-\beta)}{N-4}$, and $2^{\ast\ast}(0) = 2^{\ast\ast}\triangleq\frac{2N}{N-4}$ is the critical Sobolev exponent. $(H_{u}, H_{v}) = \nabla H$ and $(Q_{u}, Q_{v}) = \nabla Q$, and $H$, $Q\in \mathscr {C}^{1}(\mathbb{R}^{2}, [0, +\infty))$ are homogeneous functions of degrees $2^{\ast\ast}$ and $q$, respectively.

The single fourth-order elliptic equations in bounded domains arise in the study of traveling waves in suspension bridges and in the study of the static deflection of an elastic plate in a fluid. For unbounded regions, the nonlinear Schrödinger equation containing an extra term with higher-order derivatives is exactly related the self-focusing of whistler waves in plasmas in the final stage. In particular, the fourth-order nonlinear Schrödinger equations have been introduced by considering the role of small fourth-order dispersion terms in the propagation of intense laser beams in a bulk medium with Kerr nonlinearity (see ^[1,2,3] and the reference therein).

In recent years, considerable attention has been paid to the biharmonic elliptic systems like $(\mathscr {P}_{\sigma}^{K})$, both from concrete applications and for pure mathematical point of view. The study of this model in $(\mathscr {P}_{\sigma}^{K})$ is motivated by its various applications, such as in thin film theory, nonlinear surface diffusion on solids, interface dynamics, micro electro-mechanical system, and phase field models of multi-phase systems (see ^[4] for example). Recently, the nonlinear elliptic problems of fourth-order in unbounded domains have been extensively investigated, see ^[5,6,7,8] and references therein. For the bounded domains, the results of nontrivial solutions have been obtained by several authors, see for instance ^[9,10,11,12].

It is worthwhile to point out that many researchers have focused on the elliptic systems involving critical exponents and homogeneous nonlinearities, and hence a variety of remarkable results have been established in the last decades. In ^[13], Morais Filho and Souto generalized the pioneer work Brezis and Nirenberg ^[14] to a system of $p$-Laplacian equations in the gradient form. More exactly, the authors in ^[13] considered the following critical elliptic problem

where $\Delta_{p} = {\text{div}}(|\nabla\cdot|^{p-2}\nabla\cdot)$ is the $p$-Laplacian, $1 < p < N$, $\Omega\subset\mathbb{R}^{N}$ is a smooth bounded domain, and $H$ and $Q$ are homogeneous functions of degrees $p^{\ast} = \frac{Np}{N-p}$ and $p$, respectively. They proved the existence of positive solution to (1) by creating a variant of concentration-compactness principle due to Lions ^[15,16]. Since then, considerable attention has been paid to homogeneous systems like (1) for elliptic problems involving critical Sobolev growth, we refer the readers to ^{[17,18,19,20]} and the references therein. Very recently, Bandeira and Figueiredo ^[21] surveyed the following elliptic system with fast increasing weights

where $N\geq 3$, $2^{\ast} = \frac{2N}{N-2}$, $q\in(2, 2^{\ast})$, $K(x) = {\text{exp}}(|x|^{2}/4)$, and $Q$ and $H$ are homogeneous functions of degrees $q$ and $2^{\ast}$, respectively. The authors proved the existence of a ground state solution of (2) in critical case. Moreover, using the truncation argument, they attained an existence result of positive solution for a supercritical case of (2). For the systems of fourth–order elliptic equations, various problems related to critical exponents have also been studied by several authors, see ^[22,23,24] for instance.

This paper is inspired by some works which have been devoted to the study of $T$-invariant solutions for second-order elliptic equations and fourth-order elliptic problems. To the best of our knowledge, one of the pioneering works concerning $T$-invariant solutions of critical elliptic equations is the paper ^[25] by Bianchi, Chabrowski and Sulkin. The authors examined the following second-order elliptic problem with critical growth

where $N > 2$, $2^{\ast} = \frac{2N}{N-2}$, $K\in \mathscr {C}(\mathbb{R}^{N})\cap L^{\infty}(\mathbb{R}^{N})$, $T\subset O(N)$ and $\mathscr{D}_{T}^{1, 2}(\mathbb{R}^{N})$ is a suitable Sobolev space of $T$-invariant functions, and derived several elegant results of $T$-invariant solutions. Also, they obtained the existence and multiplicity of $T$-invariant solutions to (3) in a bounded $T$-invariant domain with Dirichlet boundary condition. Later, there have been many interesting results on $T$-invariant solutions for second-order elliptic problems such as ^[26,27,28] and the reference therein. Very recently, Baldelli, Brizi and Filippucci ^[29] by applying the classical idea of concentration compactness principle and fountain theorem showed the existence and multiplicity of $T$-invariant solutions of the following $(p, q)$-Laplacian equation

where $\Delta_{m}u = {\text{div}}(|\nabla u|^{m-2}\nabla u)$, $1 < q\leq p < N$, $\lambda > 0$, $1 < r < p^{\ast} = \frac{Np}{N-p}$, $T\subset O(N)$ and $X_{T}$ is an appropriate Sobolev space of $T$-invariant functions, the weights $V$ and $K$ are nonnegative and fulfill certain suitable conditions.

Motivated by the above works, especially by ^[25,29], in the present paper we are devoted to investigating the existence and multiplicity of $T$-invariant solutions for problem $(\mathscr {P}_{\sigma}^{K})$. As far as we are concerned, there is no article in literature so far to deal with the singular systems like $(\mathscr {P}_{\sigma}^{K})$ in bounded domains. It is worth mentioning that the related problem like $(\mathscr {P}_{\sigma}^{K})$ involving nonsingular case $\mu = 0$ and critical Sobolev exponent $2^{\ast\ast} = 2N/(N-4)$ has been discussed in ^[11].

Suppose that $\widetilde{K} > 0$ is a constant, $N > 4$, $0\leq\mu < \overline{\mu}$ with $\overline{\mu} = \frac{1}{16}N^{2}(N-4)^{2}$, $\sigma\geq 0$, $0\leq\varsigma < 4$, $H(u, v) = |u|^{2^{\ast\ast}}+|v|^{2^{\ast\ast}}+\eta|u|^{\iota}|v|^{\theta}$, and $Q(u, v) = |u|^{q_{1}}|v|^{q_{2}}$ in $(\mathscr {P}_{\sigma}^{K})$, where $\eta\geq 0$, $\iota$, $\theta > 1$ with $\iota+\theta = 2^{\ast\ast}$, and $q_{1}$, $q_{2} > 1$ with $q_{1}+q_{2} = q > 2$. Precisely, we consider the following singular biharmonic system which is a special case of the problem $(\mathscr {P}_{\sigma}^{K})$:

It is trivial to check that the assumptions (q.1), (q.2) and (h.1)–(h.3) in Section 2 are verified. Then Theorems 2.1-2.3 and Corollaries 1 and 2 of this paper hold for the problem $(\mathcal {P}_{\mu, \sigma}^{K})$. Besides, we recall that the existence and multiplicity results of $(\mathcal {P}_{0, 0}^{K})$ and the existence results of $(\mathcal {P}_{0, \sigma}^{\widetilde{K}})$ have been established in ^[11] (see Theorems 2.1–2.3 and Corollaries 1 and 2 in ^[11] for details). For the case $0 < \mu < \overline{\mu}$, even in the subcase of the prototype $(\mathcal {P}_{\mu, \sigma}^{K})$, the results of this paper are new.

To illustrate group-invariant (or group-symmetric) solutions of the fourth-order elliptic problems like $(\mathscr {P}_{\sigma}^{K})$ and $(\mathcal {P}_{\mu, \sigma}^{K})$, we take into account the following concrete examples which are valuable and helpful for the readers.

Example 1.1. Let $\mu = 0$, $\sigma = 0$, $\Omega = B_{R}(0)$ with $R > 0$, and $K(x)$ be a radial function in $(\mathcal {P}_{\mu, \sigma}^{K})$. Obviously, we deduce that the corresponding group $T = O(N)$, $|T| = \infty$, and $\Omega$ and $K$ are radial symmetric. If $K_{+}(0) = 0$, then according to [11,Theorem 2.2], the above problem $(\mathcal {P}_{0, 0}^{K})$ has infinitely many solutions, which are radially symmetric.

Example 1.2. Let $\mu = 0$, $\sigma > 0$, $\Omega = B_{R}(0)$ with $R > 0$, and $K(x)\equiv\widetilde{K} > 0$ be a constant in $(\mathcal {P}_{\mu, \sigma}^{K})$. Similarly, we know that the corresponding group $T = O(N)$, $|T| = \infty$, and $\Omega$ and $K$ are radial symmetric. By means of [11,Theorem 2.3], if $q_{1}$, $q_{2} > 1$ satisfy

where $0\leq\beta < 4$ and $2^{\ast\ast}(\beta) = 2(N-\beta)/(N-4)$, then problem $(\mathcal {P}_{0, \sigma}^{\widetilde{K}})$ possesses at least one nontrivial radial solution.

Example 1.3. Let $N\geq 5$ and $0 < \mu < \overline{\mu}$ with $\overline{\mu} = \frac{1}{16}N^{2}(N-4)^{2}$. We consider the following scalar equation in $\mathbb{R}^{N}$

Let us set

where $\mathscr{D}^{2, 2}(\mathbb{R}^{N})$ is a Hilbert space which will be defined in Section 2. It is clear to see that $T = O(N)$ and $|T| = \infty$. In view of [10,Theorem 2], we find that problem $(\mathbb {P}_{\mu})$ admits a positive radial decreasing solution which is a minimizer of the best constant $\mathscr{A}_{\mu}$.

In the following, we always assume that $\widetilde{K} > 0$ is a constant. The purpose of this work is to study both the cases of $\sigma = 0$, $K(x)$ non constant, and $\sigma > 0$, $K(x)\equiv \widetilde{K}$. The arguments of our results are mainly based on the variational methods and critical point theory. However, the difficulties of the present research are twofold. The first difficulty is caused by the usual lack of compactness since the system $(\mathscr {P}_{\sigma}^{K})$ involves critical Sobolev exponent and we have to verify that the mountain pass level is actually below the compactness threshold. The second difficulty lies in proving several estimates which are more delicate than in the second-order elliptic problems, and we should estimate some integrals involving the extremal function $y_{\epsilon}(x)$ (see (7) below) and its derivatives.

Throughout this paper, we always denote various positive constants as $C_{i}$ ($i\in\mathbb{N}$) or $C$. We denote by $B_{\varrho}(x)$ a ball centered at $x$ with radius $\varrho > 0$ and $o_{n}(1)$ is a datum that tends to $0$ as $n\rightarrow \infty$. For all $\epsilon > 0$ small enough, $O(\epsilon^{t})$ denotes the quantity satisfying $|O(\epsilon^{t})|/\epsilon^{t}\leq C$; and for $r = |x| > 0$, $O_{1}(r^{t})$ implies that there exist positive constants $C_{1}$ and $C_{2}$ such that $C_{1}r^{t}\leq |O_{1}(r^{t})|\leq C_{2}r^{t}$. We always denote by"$\rightarrow$" and "$\rightharpoonup$" strong and weak convergence in a Banach space $\mathbb{E}$, respectively. A functional $\mathscr {E}\in\mathscr{C}^{1}(\mathbb{E}, \mathbb{R})$ is called to verify the $(PS)_{c}$ condition if each sequence $\{w_n\}$ in $\mathbb{E}$ fulfilling $\mathscr {E}(w_{n})\rightarrow c$, $\mathscr {E}^{\prime}(w_{n})\rightarrow 0$ in $\mathbb{E}^{\ast}$ has a subsequence, which strongly converges to some element in $\mathbb{E}$.

The structure of this paper is as follows: Section 2 contains the variational framework and main results of this work. In Section 3, we provide the proofs of several existence and multiplicity results of $T$-invariant solutions for the problem $(\mathscr {P}_{0}^{K})$. The proof of existence result for the system $(\mathscr {P}_{\sigma}^{\widetilde{K}})$ with $\sigma > 0$ will be presented in Section 4.

2.   Preliminaries and main results

Let $O(N)$ be the group of orthogonal linear transformations in $\mathbb{R}^{N}$ and let $T$ be a closed subgroup of $O(N)$. The number of points contained in the orbit $T_{x} = \{\iota x; \iota\in T\}$ will be denoted as $|T_{x}|$. In particular, $|T_{0}| = |T_{\infty}| = 1$. If the number is infinite, then we write $|T_{x}| = \infty$. Denote $|T| = \inf_{0\neq x\in \mathbb{R}^{N}}|T_{x}|$. A function $f: \Omega\mapsto \mathbb{R}$ is called $T$-invariant (or $T$-symmetric) if $f(\iota x) = f(x)$ for every $\iota\in T$ and $x\in \Omega$, with $\Omega$ an open $T$-invariant subset of $\mathbb{R}^{N}$, namely if $x\in\Omega$, then $\iota x\in \Omega$ for all $\iota\in T$. Note that the radial functions are $T$-invariant functions with $T = O(N)$, and $|T| = \infty$.

For a bounded and $T$-invariant domain $\Omega\subset\mathbb{R}^{N}$, we denote by $H_{0, T}^{2}(\Omega)$ the subspace of $H_{0}^{2}(\Omega)$ consisting of all $T$-invariant functions, where $H_{0}^{2}(\Omega)$ is the closure of $\mathscr {C}_{0}^{\infty}(\Omega)$ with respect to the norm $(\int_{\Omega}|\Delta \cdot|^{2}dx)^{1/2}$. Similarly, we define by $\mathscr{D}^{2, 2}(\mathbb{R}^{N})$ the closure of $\mathscr {C}_{0}^{\infty}(\mathbb{R}^{N})$ with respect to the norm $(\int_{\mathbb{R}^{N}}|\Delta \cdot|^{2}dx)^{1/2}$.

The starting point of the variational method to problems like $(\mathscr {P}_{\sigma}^{K})$, namely when singular weights are involved, is the following Hardy-Rellich inequality ^[9]. Let $0\leq\beta\leq 4$ and $2\leq q\leq 2^{\ast\ast}(\beta) = 2(N-\beta)/(N-4)$. Then, there exists a constant $C = C(N, q, \beta) > 0$ such that

As $\beta = 4$ and $q = 2$, by (4) we have the well-known Rellich inequality ^[30]

where $\overline{\mu} = \frac{1}{16}N^{2}(N-4)^{2}$ is the best constant. Thanks to (5), we derive for $\mu\in [0, \overline{\mu})$ a norm, equivalent to the usual norm $(\int_{\Omega}|\Delta u|^{2}dx)^{1/2}$, defined by

Furthermore, for $\mu\in [0, \overline{\mu})$, the natural functional space to investigate the problem $(\mathscr {P}_{\sigma}^{K})$ is the Hilbert space $(H_{0, T}^{2}(\Omega))^{2}$ which is the subspace of $(H_{0}^{2}(\Omega))^{2}$ consisting of $T$-invariant functions, where the product space $(H_{0}^{2}(\Omega))^{2}$ is equipped with the following norm

Hereafter, we always presume that $0\in\Omega\subset\mathbb{R}^{N}$ is bounded and $T$-invariant. The dual space of $(H_{0, T}^{2}(\Omega))^{2}$ ($(H_{0}^{2}(\Omega))^{2}$ resp.) is denoted by $(H_{T}^{-2}(\Omega))^{2}$ ($(H^{-2}(\Omega))^{2}$, resp.). Meanwhile, we denote by $L^{q}(\Omega, |x|^{-\beta})$ the weighted $L^{q}(\Omega)$ space endowed the norm $(\int_{\Omega}|x|^{-\beta} |u|^{q}dx)^{1/q}$.

Let $\mathcal{A}_{\mu}$ be the best constant for the embedding of $H_{0}^{2}(\Omega)$ in $L^{2^{\ast\ast}}(\Omega)$ defined by

Then all the minimizers of $\mathcal{A}_{\mu}$ in (6) are obtained by

which satisfies

where $C = C(N, \mu) > 0$ is dependent only on $N$ and $\mu$. The function $U_{\mu}(x)$ in (7) is positive and radial, and solves the equation $\Delta^{2}u = \mu\frac{u}{|x|^{4}}+ |u|^{2^{\ast\ast}-2}u$ in $\mathbb{R}^{N}$. By setting $\Lambda_{0}\triangleq\frac{N-4}{2}$ and $r = |x|$, it follows from [10,Theorem 2] (see also [31,Lemma 2.1]) that

where the function $\eta_{\mu}: [0, \overline{\mu}]\mapsto [0, 1]$ in (9) verifies $\eta_{0} = 0$ and $\eta_{\overline{\mu}} = 1$, and its accuracy value is given by

What's more, there exist two positive constants $C_{3} = C_{3}(N, \mu)$ and $C_{4} = C_{4}(N, \mu)$ such that

Note that $0\in\Omega$; therefore, we may choose $\varrho > 0$ so small that $B_{2\varrho}(0)\subset\Omega$ and define a suitable cutoff function $\phi\in\mathscr {C}_{0}^{\infty}(\Omega)$ such that $\phi(x) = 1$ on $B_{\varrho}(0)$, $\phi(x) = 0$ on $\Omega\backslash B_{2\varrho}(0)$. Moreover, by setting $V_{\epsilon} = \phi y_{\epsilon}/\|\phi y_{\epsilon}\|_{\mu}$, a straightforward computation shows that (see also (30))

The assumptions on the potential $K(x)$ and the nonlinearities $H$ and $Q$ are presented as follows.

(k.1) $K(x)$ is $T$-invariant, where $T$ is a closed subgroup of $O(N)$.

(k.2) $K(x)\in \mathscr {C}(\overline{\Omega})\cap L^{\infty}(\overline{\Omega})$, and $K_{+}(x)\not\equiv 0$, where $K_{+}(x) = \max\{0, K(x)\}$.

(h.1) $H\in \mathscr {C}^{1}(\mathbb{R}^{2}, [0, +\infty))$ is $2^{\ast\ast}$-homogeneous, namely,

(h.2) $H(-\varsigma, -\tau) = H(\varsigma, \tau), \; \forall(\varsigma, \tau)\in\mathbb{R}^{2}$.

(h.3) $0 < H_{\min}\triangleq\min\{H(\varsigma, \tau); \; |\varsigma|^{2^{\ast\ast}}+|\tau|^{2^{\ast\ast}} = 1, \; (\varsigma, \tau)\in\mathbb{R}^{2}\}$.

(q.1) $q\in (2, 2^{\ast\ast}(\beta))$, and $Q\in \mathscr {C}^{1}(\mathbb{R}^{2}, [0, +\infty))$ is $q$-homogeneous, namely,

(q.2) $Q(\varsigma, \tau) > 0$ for each $\varsigma > 0$, $\tau > 0$, and $Q(\varsigma, 0) = Q(0, \tau) = 0, \forall (\varsigma, \tau)\in\mathbb{R}^{2}$.

Let $\widetilde{K} > 0$ be a constant. Note that here we treat both the cases of $\sigma = 0$, $K(x)$ non constant, and $\sigma > 0$, $K(x)\equiv \widetilde{K}$. Precisely, we have the following results.

Theorem 2.1. Let (k.1), (k.2) and (h.1)–(h.3) be satisfied. If for some $\epsilon > 0$, there holds

then problem $(\mathscr {P}_{0}^{K})$ admits at least one nontrivial solution in $(H_{0, T}^{2}(\Omega))^{2}$.

Corollary 2.1. Let (k.1), (k.2) and (h.1)–(h.3) be verified. If for some $\gamma_{0} > 0$, $\vartheta\in (0, (N-4)(1-\eta_{\mu}))$ and $|x|$ small, there holds $K(x)\geq K(0)+\gamma_{0} |x|^{\vartheta}$ and

then problem $(\mathscr {P}_{0}^{K})$ possesses at least one nontrivial solution in $(H_{0, T}^{2}(\Omega))^{2}$.

Theorem 2.2. Let (k.1), (k.2) and (h.1)–(h.3) be satisfied. If $K_{+}(0) = 0$ and $|T| = \infty$, then problem $(\mathscr{P}^{K}_{0})$ has infinitely many $T$-invariant solutions.

Corollary 2.2. Let (h.1)–(h.3) be verified. If $K$ is radial and $K_{+}(0) = 0$, then problem $(\mathscr{P}^{K}_{0})$ has infinitely many radial solutions.

Theorem 2.3. Let (q.1) and (q.2) be fulfilled. If $\sigma > 0$, $K(x)\equiv \widetilde{K} > 0$ and

then problem $(\mathscr {P}_{\sigma}^{\widetilde{K}})$ possesses at least one nontrivial solution in $(H_{0, T}^{2}(\Omega))^{2}$.

Remark 2.1. Even in the scalar case $u = v$ and $0 < \mu < \overline{\mu}$, our main results generalize, improve and complement the previous works in the literature ^[11,25,29].

3.   Existence and multiplicity results for system $(\mathscr {P}_{0}^{K})$

We define the energy functional corresponding to the problem $(\mathscr {P}_{0}^{K})$ as

Then $\mathscr{E}\in\mathscr{C}^{1}((H_{0, T}^{ 2}(\Omega))^{2}, \mathbb{R})$, by the properties of homogeneous functions [13,Remark 5]. Moreover, $(u, v)$ is a weak solution of system $(\mathscr{P}^{K}_{0})$ if and only if $(u, v)$ is a critical point of the functional $\mathscr{E}(u, v)$. According to the following principle of symmetric criticality (see Lemma 3.1), $(u, v)\in (H_{0, T}^{ 2}(\Omega))^{2}$ is said to be a weak solution of problem $(\mathscr{P}^{K}_{0})$, if for all $(\varphi_{1}, \varphi_{2})\in (H_{0}^{2}(\Omega))^{2}$,

Lemma 3.1. If $\mathscr{E}^{\prime}(u, v) = 0$ in $(H_{T}^{-2}(\Omega))^{2}$, then $\mathscr{E}^{\prime}(u, v) = 0$ in $(H^{-2}(\Omega))^{2}$.

Proof. This is a special case of Theorem 1.28 in ^[32].

Taking into account the homogeneity of $H(\varsigma, \tau)^{\frac{2}{2^{\ast\ast}}}$ and [13,Remark 5 (i)] and (h.1), we can find that there exists $\widetilde{H}_{\max} > 0$ such that

Then, in view of (h.1)–(h.3) and (17), we obtain the Euler identity

and the following inequality

Furthermore, the maximum $\widetilde{H}_{\max}$ is achieved for some $\varsigma_{0} > 0$ and $\tau_{0} > 0$, from which it yields

Let $\mathcal{A}_{\mu, H}$ be the best constant defined by

Then we derive the following result.

Lemma 3.2. Let $\mu\in [0, \overline{\mu})$ and $y_{\epsilon}(x)$ be the extremal function defined in (7) for any $\epsilon > 0$. If (h.1)–(h.3) is verified, then the following statements hold.

(i) $\mathcal{A}_{\mu, H} = \widetilde{H}_{\max}^{-1}\mathcal{A}_{\mu}$; and

(ii) $\mathcal{A}_{\mu, H}$ has the minimizers $(\varsigma_{0}y_{\epsilon}(x), \tau_{0}y_{\epsilon}(x))$.

Proof. The conclusion follows by modifying the proof of [26,Lemma 3.2].

Lemma 3.3. Let $\{(u_{n}, v_{n})\}$ be a weakly convergent sequence to $(u, v)$ in $(H_{0, G}^{2}(\Omega))^{2}$ such that $|\Delta u_{n}|^{2}\rightharpoonup\xi^{(1)}$, $|\Delta v_{n}|^{2}\rightharpoonup\xi^{(2)}$, $H(u_{n}, v_{n})\rightharpoonup \nu$, and $|x|^{-4}|u_{n}|^{2}\rightharpoonup \gamma^{(1)}$, $|x|^{-4}|v_{n}|^{2}\rightharpoonup \gamma^{(2)}$ in the sense of measures. Then, there exists some at most countable set $\mathscr{I}$, $\{\xi_{i}^{(1)}\geq 0\}_{i\in\mathscr{I}\cup\{0\}}$, $\{\xi_{i}^{(2)}\geq 0\}_{i\in\mathscr{I}\cup\{0\}}$, $\{\nu_{i}\geq 0\}_{i\in\mathscr{I}\cup\{0\}}$, $\gamma_{0}^{(1)}\geq 0$, $\gamma_{0}^{(2)}\geq 0$, $\{x_{i}\}_{i\in\mathscr{I}}\subset\overline{\Omega}\backslash\{0\}$ such that

(a) $\xi^{(1)}\geq |\Delta u|^{2}+ \sum\limits_{i\in\mathscr {I}}\xi_{i}^{(1)}\delta_{x_{i}}+\xi_{0}^{(1)}\delta_{0}$, \; $\xi^{(2)}\geq |\Delta v|^{2}+ \sum\limits_{i\in\mathscr {I}}\xi_{i}^{(2)}\delta_{x_{i}}+\xi_{0}^{(2)}\delta_{0}$;

(b) $\nu = H(u, v)+ \sum\limits_{i\in\mathscr {I}}\nu_{i}\delta_{x_{i}}+\nu_{0}\delta_{0}$;

(c) $\gamma^{(1)} = |x|^{-4}|u|^{2}+\gamma_{0}^{(1)}\delta_{0}$, \; $\gamma^{(2)} = |x|^{-4}|v|^{2}+\gamma_{0}^{(2)}\delta_{0}$;

(d) $\mathcal{A}_{0, H}\nu_{i}^{2/2^{\ast\ast}}\leq\xi_{i}^{(1)}+\xi_{i}^{(2)}$; and

(e) $\mathcal{A}_{\mu, H}\nu_{0}^{2/2^{\ast\ast}}\leq\xi_{0}^{(1)}+\xi_{0}^{(2)} -\mu\Big(\gamma_{0}^{(1)}+\gamma_{0}^{(2)}\Big)$,

where $\delta_{x_{i}}$, $i\in \mathscr {I}\cup\{0\}$, is the Dirac mass of 1 concentrated at $x_{i}\in \overline{\Omega}$.

Proof. The desired result follows by [13,Lemma 6] or ^[15,16].

It should be pointed that the functional $\mathscr{E}$ does not satisfy $(PS)$ condition due to the lack of compactness of the embeddings $H_{0}^{2}(\Omega)\hookrightarrow L^{2^{\ast\ast}}(\Omega)$ and $H_{0}^{2}(\Omega)\hookrightarrow L^{2}(\Omega, |x|^{-4})$. The standard variational argument is not applicable directly. Let $c_{0}^{\ast}$ be defined in (22) below. We are ready to analyze carefully the effect of the coefficient $K$ and show that the functional $\mathscr{E}$ verifies $(PS)_{c}$ condition for every $c\in (-\infty, c_{0}^{\ast})$, and then $\mathscr{E}$ has a suitable $(PS)_{c}$ sequence. This means that $c_{0}^{\ast}$ is actually the compactness threshold of the mountain pass levels for $\mathscr{E}$. We are now in position of proving the local $(PS)_{c}$ condition which is crucial for the proof of Theorem 2.1.

Lemma 3.4. Let (k.1), (k.2) and (h.1)–(h.3) be fulfilled. Then, the $(PS)_{c}$ condition in $(H_{0, T}^{2}(\Omega))^{2}$ is verified for

Proof. Analogous to the strategy used in [11,Lemma 3.4], we will sketch a short proof for completeness. Let $\{(u_{n}, v_{n})\}\subset (H_{0, T}^{2}(\Omega))^{2}$ be a $(PS)_{c}$ sequence for $\mathscr{E}$ with $c < c_{0}^{\ast}$. Then it is easy to verify that sequence $\{(u_{n}, v_{n})\}$ is bounded in $(H_{0, T}^{2}(\Omega))^{2}$ and we may presume that $(u_{n}, v_{n})\rightharpoonup (u, v)$ in $(H_{0, T}^{2}(\Omega))^{2}$. Thanks to Lemma 3.3, there exist nonnegative measures $\xi^{(1)}$, $\xi^{(2)}$, $\nu$, $\gamma^{(1)}$, and $\gamma^{(2)}$ such that relations (a)–(e) of this lemma hold. Let $x_{i}\neq 0$ be a singular point of measures $\xi^{(1)}$, $\xi^{(2)}$, and $\nu$. Suppose that $\phi(x)\in \mathscr {C}_{0}^{\infty}(\Omega)$ is a smooth cut-off function centered at $x_{i}$, $0\leq\phi(x)\leq1$ such that $\phi(x) = 1$ in $B_{\epsilon}(x_{i})$, $\phi(x) = 0$ on $\Omega\backslash B_{2\epsilon}(x_{i})$, $|\nabla\phi|\leq 2/\epsilon$, and $|\Delta\phi|\leq 2/\epsilon^{2}$. With the help of the Euler identity (18) and Lemma 3.1, we naturally obtain

Again by (a)–(c) of Lemma 3.3, (4) and the fact that $0\not\in{\text{supp}}\phi$, we derive

This yields

Then, passing to the limit as $\epsilon\rightarrow 0$, we deduce from Lemma 3.3 and (23) that $K(x_{i})\nu_{i}\geq \xi_{i}^{(1)}+\xi_{i}^{(2)}$. Using this together with (d) of Lemma 3.3, we find that either (i) $\nu_{i} = 0$ or (ⅱ) $\nu_{i}\geq (\mathcal{A}_{0, H}/\|K_{+}\|_{\infty})^{\frac{N}{4}}$. For the point $x = 0$, similarly it follows that

By applying (24) and (e) of Lemma 3.3, we conclude that either (ⅲ) $\nu_{0} = 0$ or (ⅳ) $\nu_{0}\geq (\mathcal{A}_{\mu, H}/K_{+}(0))^{\frac{N}{4}}$. Now, we claim that the cases (ⅱ) and (ⅳ) are impossible. For any continuous function $\psi$ such that $0\leq\psi(x)\leq 1$ on $\Omega$, we deduce from (15), (16) and the Euler identity (18) that

If we presume the existence of $i\in\mathscr{I}$ with $x_{i}\neq 0$ such that (ⅱ) holds, then we choose $\psi$ with compact support so that $\psi(\iota x_{i}) = 1$ for any $\iota\in T$ and we get

which contradicts (22). Similarly, if (ⅳ) holds at $x = 0$, we take $\psi$ with compact support, so that $\psi(0) = 1$ and we have

a contradiction with (22). Hence, it concludes proof of the claim. As a result, we derive $\nu_{i} = 0$ for all $i\in\mathscr {I}\cup\{0\}$. This yields $\lim\limits_{n\rightarrow \infty}\int_{\Omega} H(u_{n}, v_{n})dx = \int_{\Omega}H(u, v)dx$. Consequently, up to a subsequence, we obtain $(u_{n}, v_{n})\rightarrow (u, v)$ in $(H_{0}^{2}(\Omega))^{2}$.

As a direct corollary of Lemma 3.4, we obtain the following result.

Corollary 3.1. If $|T| = \infty$ and $K_{+}(0) = 0$, then the functional $\mathscr{E}$ satisfies the $(PS)_{c}$ condition for every $c\in\mathbb{R}$.

Proof of Theorem 2.1. Due to (k.2) and (15), it follows that there exist constants $\alpha_{0} > 0$ and $\rho > 0$ such that $\mathscr{E}(u, v)\geq\alpha_{0}$ for all $\|(u, v)\|_{\mu} = \rho$. Note that $V_{\epsilon} = \phi y_{\epsilon}/\|\phi y_{\epsilon}\|_{\mu}$ satisfies (11) and it is exactly suitable for (27)–(30). If we define for $t\geq 0$, $\Psi(t) = \mathscr{E}(t\varsigma_{0}V_{\epsilon}, t\tau_{0}V_{\epsilon})$, then direct calculation shows that there exists $\overline{t} > 0$ such that

We now choose $t_{0} > 0$ such that $\mathscr{E}(t_{0}\varsigma_{0}V_{\epsilon}, t_{0}\tau_{0}V_{\epsilon}) < 0$ and $\|(t_{0}\varsigma_{0}V_{\epsilon}, t_{0}\tau_{0}V_{\epsilon})\|_{\mu} > \rho$ and set

where $\Gamma = \{\gamma\in\mathscr {C}([0, 1], (H_{0, T}^{2}(\Omega))^{2}); \gamma(0) = (0, 0), \mathscr{E}(\gamma(1)) < 0\}$. It follows from (10), (12), (25), (26) and Lemma 3.2 that

If $c_{0} < c_{0}^{\ast}$, then by Lemma 3.4, the $(PS)_{c}$ condition holds and the assertion follows from the mountain pass theorem in ^[33] (see also ^[14]). If $c_{0} = c_{0}^{\ast}$, then $\gamma(t) = (tt_{0}\varsigma_{0}V_{\epsilon}, tt_{0}\tau_{0}V_{\epsilon})$ is a path in $\Gamma$ such that $\max_{t\in [0, 1]}\mathscr{E}(\gamma(t)) = c_{0}$, where $t\in [0, 1]$. Accordingly, either $\Psi^{\prime}(\overline{t}) = 0$ and we are done, or $\gamma$ can be deformed to a path $\widetilde{\gamma}\in\Gamma$ with $\max_{t\in [0, 1]}\mathscr{E}(\widetilde{\gamma}(t)) < c_{0}$, which is impossible. Then we conclude that problem $(\mathscr {P}_{0}^{K})$ possesses a nontrivial solution $(u_{0}, v_{0})\in (H_{0, T}^{2}(\Omega)\backslash\{0\})^{2}$. This, together with Lemma 3.1, implies that $(u_{0}, v_{0})$ is a nontrivial $T$-invariant solution of $(\mathscr {P}_{0}^{K})$.

Proof of Corollary 2.1. Let $\phi\in\mathscr{C}_{0}^{\infty}(\Omega)$ satisfy $0\leq\phi(x)\leq 1$, $\phi(x) = 1$ on $B_{\varrho}(0)$ and $\phi(x) = 0$ on $\Omega\backslash B_{2\varrho}(0)$, with $\varrho > 0$ to be determined. Following the analytic techniques in ^[14], we deduce from (7)–(10) that

Taking $V_{\epsilon} = \phi y_{\epsilon}/\|\phi y_{\epsilon}\|_{\mu}$, it is easy to verify from (27) and (28) that

Next, we choose $\varrho > 0$ so that $K(x)\geq K(0)+\gamma_{0}|x|^{\vartheta}$ for $|x|\leq\varrho$. Then by (30) we find

Obviously, it suffices to prove that

for sufficiently small $\epsilon > 0$. Note that here we have

For $\epsilon > 0$ sufficiently small, we deduce from (7)–(10), (27), and the fact that $N-1+\vartheta-N\eta_{\mu} > -1$, and $N-1+\vartheta-N\eta_{\mu}-2N(1-\eta_{\mu}) < -1$ that

and

where $\overline{C}_{1} > 0$ and $\overline{C}_{2} > 0$ are constants independent of $\epsilon$. Taking into account $0 < \vartheta < (N-4)(1-\eta_{\mu}) < N(1-\eta_{\mu})$, we conclude that (31) holds as $\epsilon > 0$ small enough. By (13) and Theorem 2.1, we obtain the desirable result.

To prove Theorem 2.2, we are ready to use the following symmetric mountain pass theorem as in ^[25]. It is worthwhile to point out that for the case of $\sigma = 0$ and $K(x)$ non constant, one may find that the system $(\mathscr {P}_{0}^{K})$ has infinitely many solutions by employing the Lusternik-Schnirelmann theory. Moreover, for the case of $\sigma > 0$, $|T| = \infty$, $K(0) = 0$ and $K(x)$ non constant, one can prove that the problem $(\mathscr {P}_{\sigma}^{K})$ possesses infinitely many solutions by an application of fountain theorem as in ^[28,29]. However, the application of symmetric mountain pass theorem is more direct and convenient than that of the genus. Besides, in Section 4, for simplicity, we are devoted to seeking group-invariant solutions of $(\mathscr {P}_{\sigma}^{\widetilde{K}})$ for the case of $\sigma > 0$, and $K(x)\equiv\widetilde{K} > 0$, which is different from the case of $\sigma > 0$, and $K(x)$ non constant. Accordingly, our arguments are mainly based upon the application of (symmetric) mountain pass theorem and not of fountain theorem.

Lemma 3.5. (see [34,Theorem 9.12]) Let $\mathbb{E}$ be an infinite dimensional Banach space, and let $\mathscr{E}\in \mathscr {C}^{1}(\mathbb{E}, \mathbb{R})$ be an even functional verifying $(PS)_{c}$ condition for every $c\in\mathbb{R}$ and $\mathscr{E}(0) = 0$. Furthermore,

(1) there are constants $\widetilde{\alpha} > 0$ and $\rho > 0$ such that $\mathscr{E}(w)\geq \widetilde{\alpha}$ for all $\|w\| = \rho$;

(2) there is an increasing sequence of subspaces $\{\mathbb{E}_{m}\}$ of $\mathbb{E}$, with $\dim \mathbb{E}_{m} = m$, such that for every $m$, one may derive $R_{m} > 0$ such that $\mathscr{E}(w)\leq 0$ for all $w\in \mathbb{E}_{m}$ with $\|w\|\geq R_{m}$.

Then $\mathscr{E}$ has a sequence of critical values $\{c_{m}\}$ tending to $\infty$ as $m\rightarrow \infty$.

Proof of Theorem 2.2. We shall present a direct application of Lemma 3.5 with $w = (u, v)\in \mathbb{E} = (H_{0, T}^{2}(\Omega))^{2}$. Now, just observe that

from which it follows that there are $\widetilde{\alpha} > 0$ and $\rho > 0$ such that $\mathscr{E}(u, v)\geq\widetilde{\alpha}$ for all $(u, v)\in \mathbb{E}$ with $\|(u, v)\|_{\mu} = \rho$. We now denote $\Omega_{K}^{+} = \{x\in\Omega; K(x) > 0\}$. Since $K$ is $T$-invariant, it is clear that $\Omega_{K}^{+}$ is $T$-invariant. Following the idea of [25,Theorem 3], we define $(H_{0, T}^{2}(\Omega_{K}^{+}))^{2}$ and presume that $(H_{0, T}^{2}(\Omega_{K}^{+}))^{2}\subset \mathbb{E}$. Let $\{\mathbb{E}_{m}\}$ be an increasing sequence of subspaces of $(H_{0, T}^{2}(\Omega_{K}^{+}))^{2}$ with $\dim \mathbb{E}_{m} = m$ for each $1\leq m\in\mathbb{N}$. Thanks to (19), we conclude that there exists a constant $\varsigma(m) > 0$ such that

for all $(\tilde{u}, \tilde{v})\in \mathbb{E}_{m}$ with $\|(\tilde{u}, \tilde{v})\|_{\mu} = 1$. Accordingly, if $(u, v)\in \mathbb{E}_{m}\backslash\{(0, 0)\}$, then we write $(u, v) = t(\tilde{u}, \tilde{v})$, with $t = \|(u, v)\|_{\mu}$ and $\|(\tilde{u}, \tilde{v})\|_{\mu} = 1$. Thus we arrive at

for $t$ large enough. By Corollary 3.1 and Lemma 3.5, the assertion follows.

Proof of Corollary 2.2. Observe that $T = O(N)$ and $|T| = \infty$. Then, Theorem 2.2 and Corollary 3.1 imply the desired result.

4.   Existence result for system $(\mathscr {P}_{\sigma}^{\widetilde{K}})$

In this section, we shall look for a weak solution of $(\mathscr {P}_{\sigma}^{\widetilde{K}})$ as a critical point of the associated functional $\mathscr {F}_{\sigma}: (H_{0, G}^{2}(\Omega))^{2}\rightarrow \mathbb{R}$ defined by

We first prove that the functional $\mathscr{F}_{\sigma}$ satisfies the local $(PS)_{c}$ condition for small energy levels.

Lemma 4.1. Let $\sigma > 0$ and (q.1) and (q.2) be verified. Then the $(PS)_{c}$ condition in $(H_{0, T}^{2}(\Omega))^{2}$ holds for $\mathscr{F}_{\sigma}$ if

Proof. Let $\{(u_{n}, v_{n})\}\subset (H_{0, T}^{2}(\Omega)\backslash\{0\})^{2}$ be a $(PS)_{c}$ sequence verifying (33). By (q.1) and the homogeneity of $Q$, we arrive at the following Euler identity

Using (18), (32) and the fact that $2 < q < 2^{\ast\ast}(\beta)\leq 2^{\ast\ast}$, we derive

which yields the boundedness of $\{(u_{n}, v_{n})\}$ in $(H_{0, T}^{2}(\Omega))^{2}$. Consequently, just as in Lemma 3.4, we may assume that $u_{n}\rightharpoonup u$, $v_{n}\rightharpoonup v$ in $H_{0, T}^{2}(\Omega)$ and in $L^{2^{\ast\ast}}(\Omega)$; moreover, $u_{n}\rightarrow u$, $v_{n}\rightarrow v$ in $L^{q}(\Omega, |x|^{-\beta})$ for any $0\leq\beta < 4$, $2 < q < 2^{\ast\ast}(\beta)$ (see [9,Lemma 2.1]) and a. e. on $\Omega$. Accordingly, we have

Applying a standard argument, we find that $(u, v)$ is a critical point of $\mathscr{F}_{\sigma}$, thus

Next, we set $\widetilde{u}_{n} = u_{n}-u$ and $\widetilde{v}_{n} = v_{n}-v$. By using the Brezis-Lieb lemma ^[35] and arguments as in [13,Lemma 8], we obtain

Recalling $\mathscr{F}_{\sigma}(u_{n}, v_{n}) = c+o_{n}(1)$ and $\mathscr{F}_{\sigma}^{\prime}(u_{n}, v_{n}) = o_{n}(1)$, we infer from (32), (34)–(37) that

and

Consequently, for a subsequence $\{(\widetilde{u}_{n}, \widetilde{v}_{n})\}$, we have

This, combined with (21), implies $\mathcal{A}_{\mu, H}(\widetilde{l}/\widetilde{K})^{\frac{2}{2^{\ast\ast}}}\leq \widetilde{l}$. Hence, we derive either $\widetilde{l} = 0$ or $\widetilde{l}\geq \widetilde{K}^{\frac{4-N}{4}}\mathcal{A}_{\mu, H}^{\frac{N}{4}}$. If $\widetilde{l}\geq \widetilde{K}^{\frac{4-N}{4}}\mathcal{A}_{\mu, H}^{\frac{N}{4}}$, then we see from (35), (36), (38) and (39) that

which contradicts (33). Therefore, we obtain $\|(\widetilde{u}_{n}, \widetilde{v}_{n})\|_{\mu}^{2}\rightarrow 0$ as $n\rightarrow \infty$, and hence, $(u_{n}, v_{n})\rightarrow (u, v)$ in $(H_{0, T}^{2}(\Omega))^{2}$. This completes the proof.

Lemma 4.2. Let $\sigma > 0$ and (q.1) and (q.2) be satisfied. Then there exists a pair of functions $(\overline{u}, \overline{v})\in (H_{0, T}^{2}(\Omega)\backslash\{0\})^{2}$ such that

Proof. We only need to show that $(\varsigma_{0}V_{\epsilon}, \tau_{0}V_{\epsilon})$ verifies (40) for $\epsilon > 0$ sufficiently small, where $V_{\epsilon} = \phi y_{\epsilon}/\|\phi y_{\epsilon}\|_{\mu}$ and $\varsigma_{0} > 0$ and $\tau_{0} > 0$ satisfy (20). To this end, we define two functions

and

From (42) it follows that $\sup_{t\geq 0}\Phi(t)$ can be attained at some $t_{\epsilon} > 0$ for which we derive

By (27) and (29), one finds that

where $O_{1}(\epsilon^{t})$ means that there exist constants $C_{1} > 0$ and $C_{2} > 0$ such that $C_{1}\epsilon^{t}\leq |O_{1}(\epsilon^{t})|\leq C_{2}\epsilon^{t}$. Thus for $\epsilon > 0$ small enough, we conclude from (30), (43) and (44) that

where $\overline{C}_{3}$ and $\overline{C}_{4}$ are positive constants independent of $\epsilon$. Besides, the function $\widetilde{\Phi}(t)$ defined by (42) achieves its maximum at $\widetilde{t}_{\epsilon}$ and is increasing in the interval $[0, \widetilde{t}_{\epsilon}]$, together with Lemma 3.2, (30) and (41)–(45), we obtain

According to (14), it is not difficult to check that

Choosing $\epsilon > 0$ small enough, we derive from (44), (46) and (47) that

Therefore, we find that $(\varsigma_{0}V_{\epsilon}, \tau_{0}V_{\epsilon})$ fulfills (40) and the assertion follows.

Proof of Theorem 2.3. Notice that $Q\in \mathscr {C}^{1}(\mathbb{R}^{2}, [0, +\infty))$ is $q$-homogeneous. Then, there exists $\widetilde{Q}_{\max} > 0$ such that

where $\widetilde{Q}_{\max} = \max\{Q(\varsigma, \tau); |\varsigma|^{q}+|\tau|^{q} = 1, (\varsigma, \tau)\in\mathbb{R}^{2}\}$. Consequently, for any $(u, v)\in (H_{0, T}^{2}(\Omega)\backslash\{0\})^{2}$, we deduce from (4), (21), (32) and (48) that

Due to $2 < q < 2^{\ast\ast}$, there exist constants $\widetilde{\alpha} > 0$ and $\rho > 0$ such that $\mathscr{F}_{\sigma}(u, v)\geq\widetilde{\alpha}$ for all $\|(u, v)\|_{\mu} = \rho$. Accordingly, we find from $\lim_{t\rightarrow \infty}\mathscr{F}_{\sigma}(tu, tv) = -\infty$ that there exists $t_{0}^{\ast} > 0$ such that $\mathscr{F}_{\sigma}(t_{0}^{\ast}u, t_{0}^{\ast}v) < 0$ and $\|(t_{0}^{\ast}u, t_{0}^{\ast}v)\|_{\mu} > \rho$. We now set

where $\Gamma = \{\gamma\in\mathscr {C}([0, 1], (H_{0, T}^{2}(\Omega))^{2}); \gamma(0) = (0, 0), \mathscr{F}_{\sigma}(\gamma(1)) < 0\}$. By virtue of the mountain pass theorem, we conclude that there exists a sequence $\{(u_{n}, v_{n})\}\subset (H_{0, T}^{2}(\Omega))^{2}$ such that $\mathscr{F}_{\sigma}(u_{n}, v_{n})\rightarrow c_{1}\geq\widetilde{\alpha}$, $\mathscr{F}_{\sigma}^{\prime}(u_{n}, v_{n})\rightarrow 0$ as $n\rightarrow \infty$. Let $(\overline{u}, \overline{v})$ be the function attained in Lemma 4.2. Then we derive

With the help of the above inequality and Lemma 4.1, we find a critical point $(u_{1}, v_{1})$ of $\mathscr{F}_{\sigma}$ satisfying $(\mathscr {P}_{\sigma}^{\widetilde{K}})$. Again, using the symmetric criticality principle, we coclude that $(u_{1}, v_{1})$ is a nontrivial $T$-invariant solution of $(\mathscr {P}_{\sigma}^{\widetilde{K}})$.

5.   Conclusions

In this paper, we combine the critical point theory and classical variational techniques to study the group-invariant solutions of the fourth-order elliptic systems with singular potentials and critical homogeneous nonlinearities. Using the Hardy-Rellich inequality and the symmetric criticality principle of Palais, we establish several existence and multiplicity results of $T$-invariant solutions to the considered problem. Furthermore, we provide a concrete model and some specific examples to explain the main results of this article.

Acknowledgment

This work is supported by Natural Science Foundation of China (No. 11971339) and Chongqing Natural Science Foundation in China (No. cstc2021jcyj-msxmX0412).

Conflict of interest

The authors declare that there is no conflict of interest.