Existence and asymptotic behavior for ground state sign-changing solutions of fractional Schrödinger-Poisson system with steep potential well

1.   Introduction and main results

In this work, we consider the existence of ground state sign-changing solutions for the following fractional Schrödinger-Poisson system

where $\mu > 0, s\in(\frac{3}{4}, 1), t\in(0, 1)$ and $V_\lambda(x) = \lambda V(x)+1$ with $\lambda > 0$. $f$ and $V$ satisfy the following assumptions:

$(f_1)$ $f\in C^1(\mathbb{R}, \mathbb{R})$, $\lim_{t\rightarrow0}\frac{f(t)}{t} = 0$ and $f(t)t > 0$ for all $t\in \mathbb{R}\setminus\{0\}$;

$(f_2)$ for some $4 < p < 2_{s}^{*} = \frac{6}{3-2s}$, there exists $C > 0$ such that $|f'(t)|\leq C(1+|t|^{p-2})$;

$(f_3)$ $\frac{f(t)}{|t|^3}$ is an increasing function of $t\in\mathbb{R}\setminus\{0\}$;

$(f_4)$ $\lim\limits_{t\rightarrow \infty}\frac{F(t)}{t^4} = +\infty$, where $F(t): = \int_0^tf(s)ds\geq0$;

$(V_1)$ $V\in C(\mathbb{R}^3, \mathbb{R})$ and $V(x)\geq0$ in $\mathbb{R}^3$;

$(V_2)$ there is $b > 0$ such that the set $\{x\in\mathbb{R}^3: V(x)\leq b\}$ is nonempty and has finite measure;

$(V_3)$ $\Omega: = \mathrm{int}\; V^{-1}(0)$ is nonempty and has a smooth boundary with $\bar{\Omega} = V^{-1}(0)$.

The above conditions imply that $V_\lambda$ represents a potential well whose depth is controlled by $\lambda$. If $\lambda$ large enough, the potential $\lambda V(x)$ is called a steep potential well which was first proposed by Bartsch and Wang ^[1].

As we all know, fractional differential equations have become increasingly important over the past few decades due to their different applications in science and engineering. Hence, nonlinear fractional Laplace equations have attracted much attention from many scholars. On the one hand, fractional operators appear in mathematical and physical problems, such as: conformal geometry and minimal surfaces ^[2], financial modeling ^[3], fractional quantum mechanics ^[4,5], anomalous diffusion ^[6], obstacle problems ^[7], etc. On the other hand, compared to the classical Laplacian operator $-\Delta$, the fractional Laplacian $(-\Delta)^s(s\in(0, 1))$ is a non-local, and previous methods may not be directly applicable. Therefore, problems related to fractional equations or systems have attracted a large number of scholars (^{[8,9,10,11,12,13,14,15,16,17,18]}).

In fact, there are many articles about the Schrödinger-Poisson system (see e.g.^{[8,9,10,11,12,13,14,15,16,17,18,19]}). Among them are studies of the existence of ground state sign-changing solutions or nontrivial solutions under different potentials, such as the vanishing potential (^[10,11]), forced potential (^[12,13,14]), constant potential (^[15,16,17]) and weighted potential ^[19]. In particular, Wang et al. ^[10] considered the following nonlinear fractional Schrödinger-Poisson system with the potential vanishing at infinity

where $s\in(\frac{3}{4}, 1), t\in(0, 1)$ and $V, K:\mathbb{R}^3\rightarrow \mathbb{R}$ are continuous functions and vanish at infinity; $f$ satisfies some growth conditions. They obtained that system (1.2) has a ground state sign-changing solution by using a Nehari manifold and constrained variational methods. Guo ^[12] considered the existence and asymptotic behavior of ground state sign-changing solutions to the following fractional Schrödinger-Poisson system

where $s\in(\frac{3}{4}, 1), t\in(0, 1)$, $\lambda > 0$ is a parameter and $V$ satisfies the following conditions:

$(V_4)$ $V\in C(\mathbb{R}^3, \mathbb{R}^+)$ satisfies that $\inf\limits_{x\in\mathbb{R}^3}V(x)\geq V_0 > 0$, where $V_0 > 0$ is a constant;

$(V_5)$ there is $r > 0$ such that $\lim\limits_{|y|\rightarrow \infty}$meas$(\{x\in B_r(y)|V(x)\leq M\}) = 0$ for any $M > 0$.

$f$ satisfies $(f_3)$ and

$(f_5)$ $f(u) = o(|u|^3)$ as $u\rightarrow0$;

$(f_6)$ for some $q\in(4, 2_s^*)$, $\lim\limits_{|u|\rightarrow \infty}\frac{f(u)}{|u|^{q-1}} = 0$;

$(f_7)$ $\lim\limits_{|u|\rightarrow \infty}\frac{f(u)}{|u|^{3}} = +\infty$.

By using the constrained variational method, the author showed that system (1.3) has a ground state sign-changing solution $u_\lambda$ and proved that the energy of the sign-changing solution is strictly larger than twice that of the ground state energy. Furthermore, they also studied the asymptotic behavior of the sign-changing solution $u_\lambda$ as $\lambda\rightarrow0$. Then, Ji ^[13] considered the existence of the least energy sign-changing solutions for the following system

where $\lambda > 0$, $s, t\in(0, 1), 4s+2t > 3$, $V$ satisfies $(V_4)$ and $(V_5)$, and $f$ satisfies the following assumptions:

$(f_8)$ $f:\mathbb{R}^3\times \mathbb{R}\rightarrow \mathbb{R}$ is a Carathéodory function and $f(x, u) = o(|u|)$ as $u\rightarrow0$ for $x\in\mathbb{R}^3$ uniformly;

$(f_{9})$ for some $1 < p < 2_s^*-1$, there exists $C > 0$ such that $|f(x, u)|\leq C(1+|u|^p)$;

$(f_{10})$ $\lim\limits_{|u|\rightarrow \infty}\frac{F(x, u)}{u^4} = +\infty$, where $F(x, u) = \int_0^uf(x, s)ds$;

$(f_{11})$ $\frac{f(x, t)}{|t|^3}$ is an increasing function of $t$ on $\mathbb{R}\backslash\{0\}$ for a.e. $x\in\mathbb{R}^3$.

The author proved that system (1.4) has a least energy sign-changing solution by using the constraint variational method and quantitative deformation lemma. In addition, they also proved that the energy of the least energy sign-changing solutions is strictly more than twice that of the energy of the ground state solution and they studied the convergence of the least energy sign-changing solutions as $\lambda\rightarrow0$. Besides, Chen et al. demonstrated that $f$ exhibits asymptotically cubic or super-cubic growth in ^[18]. Without assuming the usual Nehari-type monotonic condition on $\frac{f(t)}{t^3}$, they established the existence of one radial ground state sign-changing solution $u_\lambda$ with precisely two nodal domains. Moreover, they also proved that the energy of any radial sign-changing solution is strictly larger than two times the least energy, and they gave a convergence property of $u_\lambda$ as $\lambda\rightarrow0$. Moreover, there are many articles about the Schrödinger-Poisson system with steep potential wells (see e.g. ^{[20,21,22,23,24,25,26,27]}).

Inspired by the above references, we will study the existence of the ground-state sign-changing solution of system (1.1) and the relationship between the ground-state sign-changing solution and the energy of the ground-state solution. At the same time, we will also study the asymptotic behavior of the ground-state sign-changing solution as $\lambda\rightarrow \infty$ and $\mu\rightarrow0$.

Throughout this paper, we define the fractional Sobolev space given by

Let us define the Hilbert space

endowed with the inner product and induced norm

And $L^q(\mathbb{R}^{3})$ is a Lebesgue space endowed with the norm $|u|_q = (\int_{\mathbb{R}^{3}}|u|^qdx)^{\frac{1}{q}}$ for $q\in[1, +\infty).$ For any $\lambda > 0$, we introduce the following working space

with a scalar product and norm respectively given by

From $(V_1)$, we can get that $\|u\|\leq\|u\|_\lambda$ for all $u\in E_\lambda$. Then for any $2\leq q\leq 2_{s}^{*},$ the embedding $E_{\lambda}\hookrightarrow L^{q}(\mathbb{R}^{3})$ is continuous and $S_{q} > 0$ exists such that $|u|_{q}\leq S_{q}\|u\|\leq S_{q}\|u\|_{\lambda}$ for all $u\in E_{\lambda}$. Suppose that $s\in(\frac{3}{4}, 1)$ and $t\in(0, 1)$, we have

Then, by ^[28], we know that the embedding $H^s(\mathbb{R}^3)\hookrightarrow L^{\frac{12}{3+2t}}(\mathbb{R}^3)$ is continuous. Considering that $u\in H^s(\mathbb{R}^3)$ and $v\in D^{t, 2}(\mathbb{R}^3)$, by the Hölder inequality, we have

Thus, thanks to the Lax-Milgram theorem, there exists a unique $\phi^t_u\in D^{t, 2}(\mathbb{R}^{3})$ such that

That is, $\phi_u^t$ satisfies that $(-\Delta)^t\phi^t_u = u^2$ for any $u\in H^s(\mathbb{R}^{3})$. Furthermore,

which is called the t-Riesz potential, where

In subsequent work, we often omit the constant $c_t$. Hence, system (1.1) can be reduced to a single equation with a non-local term

We can see that the solutions of system (1.1) are precisely the critical points of the energy functional $J^\mu_\lambda:E_\lambda\rightarrow \mathbb{R}$ which is defined by

where $F(s) = \int_0^sf(t)dt$. It is easy to see that $J^\mu_\lambda$ is well defined and $J^\mu_\lambda\in C^1(E_\lambda, \mathbb{R})$. Moreover, for any $u, \varphi\in E_\lambda$,

Now our main results in this paper can be stated as follows.

Theorem 1.1. Let $(V_1)-(V_3)$ and $(f_1)-(f_4)$ be satisfied, $\lambda > 0$ be sufficiently large and $\mu > 0$; system (1.1) has at least one ground state sign-changing solution which has precisely two nodal domains. Moreover, the energy of the ground state sign-changing solution is strictly larger than twice that of the energy of the ground state solution.

Theorem 1.2. Under the assumptions of Theorem 1.1, for any sequence $\lambda_n\rightarrow +\infty$ as $n\rightarrow \infty$, the sequence of sign-changing solutions $\{u_{\lambda_n}\}$ for system (1.1) strongly converges to $u_*$ in $H^s(\mathbb{R}^3)$ up to a subsequence, where $u_*$ is a ground state sign-changing solution of the following system

where $\frac{1}{|x|}*u^2 = \int_{\Omega}\frac{u^2(y)}{|x-y|^{3-2t}}dy$ and there are only two nodal domains.

Theorem 1.3. Under the assumptions of Theorem 1.1, for any $\mu\in(0, 1]$, suppose that $u_\mu$ is a ground-state sign-changing solution of system (1.1) that has been obtained according to Theorem 1.1. Then there exists $u_0\in E_\lambda$ such that $u_\mu\rightarrow u_0$ in $E_\lambda$ as $\mu\rightarrow0$, where $u_0$ is a ground-state sign-changing solution to the following equation

Moreover, $u_0$ has two nodal domains.

Remark 1.4. Our results are up to date. On the one hand, similar to ^[10,12], we study the fractional Schrödinger-Poisson system with a steep potential well. On the other hand, we generalize the results of ^[20] to the fractional Laplace operator.

Remark 1.5. It is worth noting that, in ^[12,13,18], they assume that the potential is radially symmetric or forced, which ensures that the Sobolev embedding $H^s(\mathbb{R}^3)$ into $L^p(\mathbb{R}^3)$ with $p\in(2, 2_s^*)$ is compact. However, in our work, our potential is a steep potential well, which makes the Sobolev embedding $H^s(\mathbb{R}^3)$ into $L^p(\mathbb{R}^3)$ with $p\in(2, 2_s^*)$ lack compactness. In order to overcome this difficulty, we use the ideas presented in ^[20,29] to find a $(PS)$ sequence of the energy functional of system (1.1) in $E_\lambda$, and prove that the local $(PS)$ condition is valid.

We have organized this paper as follows. In Sect. 2, we present some preliminary lemmas which are essential for the proof of the theorems. In Sect. 3, we give the proof of the main results.

We conclude this section by giving some notations, which will be applied later in the work.

● $E_\lambda^\ast$ is the dual space of the Banach space of $E_\lambda$.

● $B_R(0): = \{x\in\mathbb{R}^3:|x|\leq R\}$ for any $R\in[0, +\infty)$ and $\Omega^c = \mathbb{R}^3\setminus\Omega$.

● $u^+(x): = \max\{u, 0\}, u^-(x): = -\min\{u, 0\}.$

● $C, C_i$ denote positive constants that may vary under different conditions.

2.   Some preliminary lemmas

On the one hand, we need to prove the existence of the sign-changing solutions of system (1.1); inspired by ^[30,31], the following minimization problem is given by

where

Clearly, $\mathcal{M}^\mu_{\lambda}$ contains all of the sign-changing solutions for system (1.1). On the other hand, we need to prove the relationship between the energy of the ground state sign-changing solution and that of the ground state solution. Therefore the following Nehari manifold $\mathcal{N}^\mu_{\lambda}$ is introduced as follows:

Similarly, the following minimization problem is defined by

By simple calculation, we can also get

and

where

for $u^\pm\neq0$. Hence,

In order to prove our results, we give the following propositions and some preliminary lemmas.

Proposition 2.1. (See ^[32]) For the function $\phi^t_u$ defined in (1.5), one has

$(i)$ $\phi^t_u\geq0$ and $\phi^t_{ku} = k^2\phi^t_u$ for all $t\in\mathbb{R}$ and $u\in H^s(\mathbb{R}^3)$;

$(ii)$ there is $C > 0$ such that $\int_{\mathbb{R}^3}\phi^t_uu^2dx\leq C\|u\|_{\frac{12}{3+2s}}^4$.

Proposition 2.2. (See ^[33], fractional Gagliardo-Nirendo inequality) For any $p\in[2, 2_s^*)$, there exists $C(p) > 0$ such that $|u|^p_p\leq C(p)|(-\Delta)^\frac{s}{2}u|_2^{\frac{3p-2_s^*}{2}}|u|_2^{\frac{2_s^*-p}{2}}$ for any $u\in H^s(\mathbb{R}^3)$.

Lemma 2.1. Assume that $(f_{1})-(f_{4})$ and $(V_{1})$ hold; for any $\lambda > 0$ and $u\in E_{\lambda}$ with $u^{\pm}\neq 0$, there exists a unique pair of $(s_{u}, t_{u})$ such that $s_{u}u^{+} + t_{u}u^{-} \in \mathcal{M}^\mu_{\lambda}$ and

Proof. We first establish the existence of $s_{u}$ and $t_{u}$. Let

By $(f_1), \; (f_2)$ and $(f_4)$, it is not hard to see that $g_{1}(s, s) > 0$, $g_{2}(s, s) > 0$ for small $s > 0$, and $g_{1}(t, t) < 0$, $g_{2}(t, t) < 0$ for large $t > 0$. Thus, there exists $0 < r < R$ such that

Thus we can deduce from (2.6)-(2.8) that

By way of Miranda's theorem ^[34], there exists some point $(s_{u}, t_{u})$ with $r < s_{u}, t_{u} < R$ such that $g_{1}(s_{u}, t_{u}) = g_{2}(s_{u}, t_{u}) = 0$. So, $s_{u}u^{+} + t_{u}u^{-} \in \mathcal{M}^\mu_{\lambda}$. Next, we prove that $(s_{u}, t_{u})$ is unique by the following two cases.

Case 1. $u\in\mathcal{M}^\mu_{\lambda}$.

For any $u\in\mathcal{M}^\mu_{\lambda}$, it means that

By (2.10), we have that $(s_{u}, t_{u}) = (1, 1)$. Then, we prove that $(s_u, t_u)$ is the unique. Assume that $(s_{0}, t_{0})$ is another pair of numbers such that $s_{0}u^{+} + t_{0}u^{-} \in \mathcal{M}_{\lambda}^\mu$.

It seems that $0 < s_{0}\leq t_{0}$; from (2.12), we have

From (2.10) and (2.13), we obtain

By $(f_3)$, if $t_{0} > 1$, the left-hand side of the inequality is negative and the right-hand side is positive, which leads to a contradiction. Therefore, we obtain that $0 < s_{0}\leq t_{0}\leq 1$. Similarly, by (2.10) and (2.11), we get

In view of $(f_{3})$, we have that $s_{0} \geq 1$. Hence, $s_{0} = t_{0} = 1$.

Case 2. $u\not\in\mathcal{M}^\mu_{\lambda}$

If $u\not\in\mathcal{M}^\mu_{\lambda}$, there exists a pair of positive numbers $(s_{u}, t_{u})\in\mathcal{M}^\mu_{\lambda}$. Suppose that there exists another pair of positive numbers $(\widetilde{s_{u}}, \widetilde{t_{u}})$ such that $\widetilde{s_{u}}u^{+}+\widetilde{t_{u}}u^{-}\in\mathcal{M}^\mu_{\lambda}$. Set $\overline{u}_1: = s_{u}u^{+}+t_{u}u^{-}\in\mathcal{M}^{\mu}_{\lambda}$ and $\overline{u}_{2}: = \widetilde{s_{u}}u^{+}+\widetilde{t_{u}}u^{-}\in\mathcal{M}^{\mu}_{\lambda}$; one has

Since $\overline{u}_{1}\in\mathcal{M}^{\mu}_{\lambda}$, by Case 1, we get that $\frac{\widetilde{s_{u}}}{s_{u}} = \frac{\widetilde{t_{u}}}{t_{u}} = 1$, which implies that $\widetilde{s_{u}} = s_{u}$ and $\widetilde{t_{u}} = t_{u}$ and $(s_{u}, t_{u})$ is the unique pair of numbers such that $s_{u}u^{+} + t_{u}u^{-} \in \mathcal{M}^{\mu}_{\lambda}$.

Finally, we define $\psi(s, t): = J^\mu_{\lambda}(su^{+}+tu^{-})$; it can be seen that $J^\mu_{\lambda}(su^{+}+tu^{-}) > 0$ as $|(s, t)|\rightarrow 0$ and $J^\mu_{\lambda}(su^{+}+tu^{-}) < 0$ as $|(s, t)|\rightarrow \infty.$ Then the maximum $\max_{s, t\geq0}J^\mu_{\lambda}(su^{+} + tu^{-})$ is well defined. Now, it is sufficient to check that the maximum point cannot be reached on the boundary of $[0, +\infty)\times[0, +\infty)$. Assume that $(0, t_{0})$ is a maximum point of $\psi$ with $t_{0}\geq 0$. Then, since

if $s$ is small enough, $(\psi')_s(s, t_{0}) > 0$; thus $\psi$ is an increasing function of $s$ and the pair $(0, t_{0})$ is not a maximum point of $\psi$. Similarly, $\psi$ can not achieve its global maximum on $(s_0, 0)$ with $s_0 > 0$. Since $(s_{u}, t_{u})$ is a unique pair of such that $s_{u}u^{+} + t_{u}u^{-} \in \mathcal{M}^\mu_{\lambda}$, it follows that $J^\mu_{\lambda}(s_{u}u^{+} + t_{u}u^{-}) = \max_{s, t\geq0}J^\mu_{\lambda}(su^{+} + tu^{-}).$ The proof is now finished. □

Lemma 2.2. $m_\lambda^\mu = \inf_{u\in\mathcal{M}^\mu_{\lambda}}J_\lambda^\mu(u) > 0$ for any $\lambda, \mu > 0$.

Proof. For every $u\in\mathcal{M}^\mu_\lambda$, we have that $\langle (J^\mu_{\lambda})'(u), u\rangle = 0$. By $(f_1)$ and $(f_2)$, for any $\varepsilon > 0$, there is $C_\varepsilon > 0$ such that

Then, by the Sobolev inequality, we get

Taking $\varepsilon = \frac{1}{2S^2_2}$, so there is a constant $\gamma > 0$ such that $\|u\|_\lambda^2\geq\gamma$. By $(f_3)$, one has

consequently,

which implies that $m_\lambda^\mu\geq\frac{1}{4}\gamma > 0$. Then the proof is completed. □

Next, we will prove the existence of sign-changing solutions for system (1.1). Given the lack of compactness of the Sobolev embedding $H^s(\mathbb{R}^3)$ into $L^p(\mathbb{R}^3)$, $p\in(2, 2_s^*)$, we need to construct a sign-changing $(PS)_{m^\mu_{\lambda}}$-sequence. Inspired by ^[29], we give some definitions. Let $P$ denote the cone of nonnegative functions in $E_{\lambda}$, $Q = [0, 1]\times[0, 1]$ and $\Sigma$ be the set of continuous maps $\sigma$ such that

For each $u\in E_{\lambda}$ with $u^{\pm}\neq0$, let $\sigma(s, t) = kt(1-s)u^{+}+kstu^{-}$, where $k > 0$ and $s, t\in[0, 1]$. It is easy to know that $\sigma(s, t)\in\Sigma$ for $k > 0$ sufficiently large, which means that $\Sigma\neq{\emptyset}$. Define

Apparently, $u\in\mathcal{M}_\lambda^\mu$ if and only if $l(u^{+}, u^{-}) = l(u^{-}, u^{+}) = 1$. Define

Lemma 2.3. There exists a sequence $\{u_{n}\}\subset U_{\lambda}$ satisfying that $J^\mu_{\lambda}(u_{n})\rightarrow m^\mu_{\lambda}$ and $(J^\mu_{\lambda})'(u_{n})\rightarrow 0$ in $E_\lambda^*$ as $n\rightarrow \infty.$

Proof. We divide three steps to complete the proof. First, we prove the following

For each $u\in\mathcal{M}^\mu_{\lambda}$, there is $\sigma(s, t) = kt(1-s)u^{+}+kstu^{-}\in\Sigma$ for $k > 0$ sufficiently large; by Lemma 2.1, we get

which implies that

At the same time, we assume that for each $\sigma\in\Sigma$, there exists $u_\sigma\in\sigma(Q)\cap\mathcal{M}^\mu_{\lambda}$, such that

As a matter of fact, on the one hand, for any $\sigma\in\Sigma$ and $t\in[0, 1]$, one has

On the other hand, from the definition of $\Sigma$, for any $\sigma\in\Sigma$ and $s\in[0, 1]$, by the elementary inequality $\frac{b}{a}+\frac{d}{c}\geq\frac{b+d}{a+c}$ for all $a, b, c, d > 0$, we get

Therefore,

According to Miranda's Theorem and (2.20)–(2.23), there exists $(s_{\sigma}, t_{\sigma})\in Q$ such that

then

which implies that for any $\sigma\in\Sigma$, there exists $u_{\sigma} = \sigma(s_{\sigma}, t_{\sigma})\in\sigma(Q)\cap\mathcal{M}^\mu_{\lambda}.$ Moreover,

Therefore,

So, by (2.19) and (2.24), one obtains

Secondly, we look for the $(PS)_{m^\mu_{\lambda}}$-sequence $\{u_{n}\}\subset E_{\lambda}$ for $J^\mu_{\lambda}$. Considering a minimizing sequence $\{w_{n}\}\subset\mathcal{M}^\mu_{\lambda}$ and $\sigma_{n}(s, t) = kt(1-s)w_{n}^{+}+ktsw_{n}^{-}\in\Sigma$ with $(s, t)\in Q$. Then, thanks to Lemma 2.1, we have

Using a variant form of the classical deformation lemma, we can deduce that there exists $\{u_{n}\}\subset \mathcal{M}^\mu_{\lambda}$ such that

Assume that this is a contradiction. Then it is possible to find a $\delta > 0$ such that $\sigma_{n}(Q)\cap D_{\delta} = \emptyset$ for $n$ sufficiently large, where

By ^[35], for some $\epsilon\in(0, \frac{m^\mu_{\lambda}}{2})$ and all $t\in[0, 1]$, there exists a continuous map $\eta:[0, 1]\times E_{\lambda}\rightarrow E_{\lambda}$ satisfying

(ⅰ) $\eta(0, u) = u, \; \eta(t, -u) = -\eta(t, u)$;

(ⅱ) $\eta(t, u) = u$, $\forall u\in J_{\lambda}^{m^\mu_{\lambda}-\epsilon}\cup(E_{\lambda}\setminus J_{\lambda}^{m^\mu_{\lambda}+\epsilon})$;

(ⅲ) $\eta(1, J_{\lambda}^{m^\mu_{\lambda}+\frac{\epsilon}{2}}\setminus D_{\delta})\subset J_{\lambda}^{m^\mu_{\lambda}-\frac{\epsilon}{2}}$;

(ⅳ) $\eta(1, (J_{\lambda}^{m^\mu_{\lambda}+\frac{\epsilon}{2}}\cap P)\setminus D_{\delta})\subset J_{\lambda}^{m^\mu_{\lambda}-\frac{\epsilon}{2}}\cap P, \; \mathrm{where}\; J_{\lambda}^{d} = \{u\in E_{\lambda}:J^\mu_{\lambda}(u)\leq d\}$.

By (2.25), we can choose $n$ such that

Let us define $\widetilde{\sigma}_{n}(s, t): = \eta(1, \sigma_{n}(s, t))$ for all $(s, t)\in Q$. We need to prove that $\widetilde{\sigma}_{n}(Q)\in\Sigma$, and thus that $\widetilde{\sigma}_{n}(Q)\subset J_{\lambda}^{m_{\lambda}-\frac{\epsilon}{2}}$ in view of (2.27) and property (iii) of $\eta$. This is a contradiction of the inequality below

By property (ⅱ) of $\eta$ and $\sigma_{n}\in\Sigma$, we derive that

And it is from $\sigma_{n}(0, t)\in P$, (2.27) and property (iv) of $\eta$ that $\widetilde{\sigma}_{n}(0, t)\in P$. Because of $\sigma_{n}(1, t)\in-P$ and (2.27), we obtain that $-\sigma_{n}(1, t)\in (J_{\lambda}^{m_{\lambda}+\frac{\epsilon}{2}}\cap P)\setminus D_{\delta},$ which implies that

Furthermore, by the definition of $\Sigma$, we get $J_\lambda^\mu(\sigma_n(l, 1))\leq0$. By property (ii) of $\eta$, we can infer that

which implies that

and

From the above, we can conclude that $\widetilde{\sigma}_{n}\in\Sigma$ from the continuity of $\eta$ and $\sigma_{n}$.

Finally, we claim that $\{u_{n}\}\subset U_{\lambda}$ for $n$ sufficiently large. Because $(J^\mu_{\lambda})'(u_{n})\rightarrow 0$, we can see that $\langle (J^\mu_{\lambda})'(u_{n}), u_{n}^{\pm}\rangle = o(1)$. Then we only need to prove that $u_{n}^{\pm}\neq0$ because it implies that $l(u_{n}^{+}, u_{n}^{-})\rightarrow1, \; l(u_{n}^{-}, u_{n}^{+})\rightarrow1$, and thus $\{u_n\}\subset U_\lambda$ for $n$ sufficiently large. From (2.26), there exists a sequence $\{v_n\}$ satisfying

In order to prove that $u_n^\pm\neq0$, we just need to prove that $s_nw_n^+\neq0$ and $t_nw_n^-\neq0$ for $n$ sufficiently large. Since $\{w_n\}\subset\mathcal{M}^\mu_{\lambda}$, similar to (2.15) and (2.17), we obtain that $C_1\leq\|w_n^\pm\|_\lambda\leq C_2$. Hence, we only need to prove that $\lim\limits_{n\rightarrow \infty}s_n\neq0$ and $\lim\limits_{n\rightarrow \infty}t_n\neq0$. If $\lim\limits_{n\rightarrow \infty}s_n = 0$, by the continuity of $J^\mu_\lambda$ and (2.28), we infer that

However, let $\varepsilon = \frac{1}{S_2^2}$; for $s > 0$ small enough, by (2.14) and (2.16), one gets

which is a contradiction. Therefore, $\{u_n\}\subset U_\lambda$ for $n$ sufficiently large. □

Inspired by ^[36], with the help of the Nehari manifold, the following results hold. Since the proof is similar, we omit it here.

Lemma 2.4. Assume that $(V_1)$ and $(f_1)-(f_4)$ hold, then, (ⅰ) for any $u\in E_\lambda$, there exists a unique $\widetilde{s_u} > 0$ such that $\widetilde{s_u}u\in\mathcal{N}^\mu_{\lambda}$, and

(ⅱ) system (1.1) has a positive ground state solution $\widetilde{u}\in\mathcal{N}^\mu_{\lambda}$ and $J_\lambda(\widetilde{u}) = c^\mu_\lambda$.

3.   The proof of main results

Proof of Theorem 1.1. From Lemma 2.3, there exists a sequence $\{u_n\}\subset U_\lambda$ satisfying that $J_\lambda^\mu(u_n)\rightarrow m_\lambda^\mu$ and $(J_\lambda^\mu)'(u_n)\rightarrow0$. Then, we need to prove that $\{u_n\}$ is bounded in $E_\lambda$ according to Lemma 2.3. From (2.16), one has

that is $\limsup\limits_{n\rightarrow \infty}\|u_n\|_\lambda\leq4m_\lambda^\mu$. Thus, $\{u_n\}$ is bounded in $E_\lambda$. Up to a subsequence, still denoted by $\{u_n\}$, there is $u_{\lambda, \mu}\in E_\lambda$ such that, as $n\rightarrow \infty$ the following holds:

By Lemma 2.3, we have that $(J^\mu_\lambda)'(u_n)\rightarrow0$ in $E_\lambda^*$ as $n\rightarrow \infty$, which implies that $(J^\mu_\lambda)'(u_{\lambda, \mu})\rightarrow0$ in $E_\lambda^*$. So, $u_{\lambda, \mu}$ is a solution of system (1.1).

Next, we claim that $u_{\lambda, \mu}$ is a ground state solution for system (1.1), that is, $J_\lambda^\mu(u_{\lambda, \mu}) = m_\lambda^\mu$. Since $u_{\lambda, \mu}\in\mathcal{M}_\lambda^\mu$, one obtains that $J_\lambda^\mu(u_{\lambda, \mu})\geq m_\lambda^\mu$. Then, combining Fatou's Lemma with (2.16), we get

Hence, $J_\lambda^\mu(u_{\lambda, \mu}) = m_\lambda^\mu$. So, $u_{\lambda, \mu}$ is a ground state solution of system (1.1).

Finally, we need to prove that$u^\pm_{\lambda, \mu}\neq0$, that is, $u_{\lambda, \mu}$ is a sign-changing solution of system (1.1). By Lemma 2.3, $\{u_n\}\subset U_\lambda$. It follows from (2.15) with $\varepsilon = \frac{1}{2S_2^2}$ that

which means that $\|u_n^\pm\|_\lambda\geq\left(\frac{1}{2S_p^pC_{\frac{1}{2S^2_2}}}\right)^\frac{1}{p-2}$ and

Set

Then, we have

Moreover, we have that $|D_R|\rightarrow0$ as $R\rightarrow \infty$ by $(V_2)$. Hence, from the Hölder inequality, as $R\rightarrow \infty$,

where $s\in(2, 2_s^*)$. Moreover, thanks to (3.3), (3.4) and Proposition 2.2, taking $R > 0$ large enough, we get

Let $R_1 > 0$ such that $o_R(1) < \frac{\epsilon}{4}$ for all $R > R_1$. Then, let

we can deduce that

So, for any $\lambda\geq\Lambda(\mu)$ and $R\geq R_1$, we have

Then,

By (3.2) and (3.7), one gets that $\limsup_{n\rightarrow \infty}\int_{B_R(0)}|u_n^\pm|^pdx\geq\frac{\epsilon}{2} > 0$, that is, $\int_{B_R(0)}|u_{\lambda, \mu}^\pm|^pdx > 0$. Hence, $u_{\lambda, \mu}^\pm\neq0$. In short, $u_{\lambda, \mu}$ is a ground state sign-changing solution of system (1.1).

Next, we are going to prove that $m^\mu_\lambda > 2c^\mu_\lambda$. From Lemma 2.4 (ⅰ), there exists $\widetilde{s}, \widetilde{t} > 0$ such that $\widetilde{s}u_{\lambda, \mu}^+, \widetilde{t}u_{\lambda, \mu}^-\in\mathcal{N}^\mu_{\lambda}$. Then, it follows from Lemma 2.1 that

Lastly, we prove that $u_{\lambda, \mu}$ changes sign only once, that is, $u_{\lambda, \mu}$ has two nodal domains. By contradiction, we assume that $u_{\lambda, \mu} = u_1+u_2+u_3$ with

Then, let $v = u_1+u_2, \; v^+ = u_1$ and $v^- = u_2$; by Lemma 2.1, there exists a unique pair of $(s_v, t_v)\in(0, 1]\times(0, 1]$ such that

By $\langle (J^\mu_{\lambda})'(u_{\lambda, \mu}), u_i\rangle = 0\; (i = 1, 2, 3)$, it follows that $\langle (J^\mu_{\lambda})'(v), v^\pm\rangle < 0$ since

From (2.16), we have

which is impossible, so $u_{\lambda, \mu}$ has exactly two nodal domains. □

In what follows, we will give the asymptotic behavior of the ground state sign-changing solution. We define $J^\mu_\infty$ as the energy functional of system (1.8):

It is not difficult to obtain that $J^\mu_\infty\in C^1$. Define

It is easy to get that $\mathcal{M}^\mu_{\infty}\subset\mathcal{M}^\mu_{\lambda}$ and $J^\mu_\lambda(u) = J^\mu_\infty(u)$ for $\lambda > 0$. Thus, we have that $m^\mu_\lambda\leq m^\mu_\infty$.

Proof of Theorem 1.2. For any sequence $\lambda_n\rightarrow \infty$ as $n\rightarrow \infty$, $\{u_{\lambda_n}\}$ is a sequence of sign-changing solutions for system (1.1) with $J^\mu_{\lambda_n}(u_{\lambda_n}) = m^\mu_{\lambda_n}\leq m_\infty^\mu$ and $(J^\mu_{\lambda_n})'(u_{\lambda_n}) = 0$. By (2.16), we conclude that

Hence, $\{u_{\lambda_n}\}$ is bounded in $H^s(\mathbb{R}^3)$. Passing to a subsequence, there is $u_*\in H^s(\mathbb{R}^3)$ such that

Step 1: We will prove that $u_*$ is a solution of system (1.8). By $(V_1)$ and Fatou's lemma, one gets

By $(V_3)$, we can deduce that $u_*|_{\Omega^c} = 0$. Hence, it follows that $u_*\in H^s_0(\Omega)$ from the boundary of $\Omega$ which is smooth. Because $(J^\mu_{\lambda_n})'(u_{\lambda_n}) = 0$, we can deduce that $\langle (J^\mu_{\infty})'(u_*), \upsilon\rangle = 0$ for any $\upsilon\in H^s_0(\Omega)$, which means that $u_*$ is a solution of system (1.8).

Step 2: We need to prove that $u_{\lambda_n}\rightarrow u_*$ in $H^s(\mathbb{R}^3)$. Then, similar to (3.3) and (3.4), we have that

Hence, $\lim_{n\rightarrow \infty}\int_{\mathbb{R}^3}|u_{\lambda_n}-u_*|^qdx = 0$ with $q\in[2, 2_s^*)$. That is, $u_{\lambda_n}\rightarrow u_*$ in $L^q(\mathbb{R}^3)$ with $q\in[2, 2_s^*)$. Then,

Obviously, we can draw the conclusion that $\langle (J^\mu_{\lambda_n})'(u_{\lambda_n}-u_*), u_{\lambda_n}-u_*\rangle = 0$. Applying an argument similar to that in Lemma 2.1 in ^[37], we can get

as $n\rightarrow \infty$. By the Hölder inequality and (2.14), we have

Since $u_{\lambda_n}\rightarrow u_*$ in $L^q(\mathbb{R}^3)$ for $q\in(2, 2_s^*)$, we get that $\int_{\mathbb{R}^3}[f(u_{\lambda_n})-f(u_*)](u_{\lambda_n}-u_*)dx\rightarrow0$ as $n\rightarrow \infty$. Hence, $\|u_{\lambda_n}-u_*\|_\lambda^2 = 0$, that is, $u_{\lambda_n}\rightarrow u_*$ in $H^s(\mathbb{R}^3)$ as $n\rightarrow \infty$.

Step 3: We claim that $u_*$ is a ground state sign-changing solution of system (1.8), that is, $J^\mu_\infty(u_*) = m^\mu_\infty$ and $u^\pm_{\lambda_n}\neq0$. On the one hand, for $m^\mu_{\lambda_n}\leq m^\mu_\infty$ and $m^\mu_{\lambda_n}\rightarrow J^\mu_\infty(u_*)$, we get that $J^\mu_\infty(u_*)\leq m^\mu_\infty$. On the other hand, since $u_*\in\mathcal{M}^\mu_{\infty}$, by (2.16), we have

Thus, $J^\mu_\infty(u_*) = m^\mu_\infty$, that is, $u_*$ is a ground state sign-changing solution of system (1.8) and $u_{\lambda_n}\rightarrow u_*$ in $H^s(\mathbb{R}^3)$ up to a subsequence. Then, analogous to the proof of Theorem 1.1, we can get that $u_*$ has two nodal domains. Hence, we have completed the proof of Theorem 1.2. □

Next, we will prove the asymptotic properties of sign-changing solutions given in Theorem 1.1 as $\mu\rightarrow0$. For convenience, we let $u_\mu: = u_{\lambda, \mu}$, $J_\mu: = J_\lambda^\mu$ and $m_\mu: = m_\lambda^\mu$. In addition, we set the energy functional and constraint set of (1.9) as $J_0(u) = J_\lambda^0(u)$ and $\mathcal{M}_0 = \mathcal{M}_\lambda^0$; similarly, $m_0 = \inf\limits_{u\in\mathcal{M}_0}J_0(u)$.

Proof of Theorem 1.3. For any $\{\mu_n\}\subset(0, 1)$ with $\mu_n\rightarrow0$ as $n\rightarrow \infty$, $u_{\mu_n}$ is a ground state solution of system (1.1) with $\mu = \mu_n$ which has been obtained in Theorem 1.1. In other words, $J_{\mu_n}(u_{\mu_n}) = m_{\mu_n}$ and $J_{\mu_n}'(u_{\mu_n}) = 0$. Similar to Theorem 1.1, we have that $\{u_{\mu_n}\}$ is bounded in $E_\lambda$. Up to a subsequence, we can assume the following:

Step 1: We need to prove that $u_0$ is a weak solution of (1.9).

For any $\varphi\in E_\lambda$, thanks to Proposition 2.1 (ⅱ), we have

and

Then, let $n\rightarrow \infty$; we get

Hence, $u_0$ is a weak solution of (1.9).

Step 2: We will prove that $u_{\mu_n}\rightarrow u_0$ in $E_\lambda$ as $n\rightarrow \infty$.

First, we need to prove that $u_{\mu_n}\rightarrow u_0$ in $L^q(\mathbb{R}^3)$ with $q\in(2, 2_s^*)$ as $n\rightarrow \infty$. Thus, for $r > 0$, let $\xi_r\in C^\infty(\mathbb{R}^3)$ such that

with $\int_{\mathbb{R}^3}(-\Delta)^\frac{s}{2}\xi_rdx\leq\frac{8}{r}$. Let $u\in E_\lambda$ such that $\|u_{\mu_n}\|_\infty\leq L$, for some $L > 0$. Then, for any $\eta\in C^1(\mathbb{R}^3)$ with $\eta\geq0$, we obtain

Taking $\eta = \xi_r$ and $\varepsilon = \frac{1}{2}$, by (2.14), it follows that

that is

Besides, for $R > 0$, we set

In fact, by $(V_2)$, we have that $|\widetilde{A}_R|\leq\varepsilon$ as $R\rightarrow \infty$; then, $\lambda V(x) > M$ in $\widetilde{D}_R$ from $\lambda > \frac{M}{b}$, where $M = C_{\frac{1}{2}}L^{p-2}$. Let $r = R$; by (3.12), one has

where $T = 8 \; \mathrm{sup}\|u_{\mu_n}\|_\lambda$. Since

thanks to (3.13) and (3.14), one gets

We have that $H^1(B_R(0))\hookrightarrow L^q(B_R(0))$ is compact for $2 < q < 2_s^*$, that is, $u_n\rightarrow u$ in $L^q(B_R(0))$ with $2 < q < 2_s^*$. For any $R$ large enough, according to (3.15), Proposition 2.2 and the boundedness of $\{u_n\}$, we have

Thus, $u_{\mu_n}\rightarrow u_0$ in $L^q(\mathbb{R}^3)$ with $q\in(2, 2_s^*)$ as $n\rightarrow \infty$. Then, by Lebesgue's dominated convergence theorem, we get

Let $\varphi = u_{\mu_n}$ apply capitalization (3.7) and $\varphi = u_0$ in (3.8), we have that $u_{\mu_n}\rightarrow u_0$ in $E_\lambda$ as $n\rightarrow \infty$.

Step 3: we claim that $u_0$ is a ground state sign-changing solution. That is, $u_0^\pm\neq0$ and $J_0(u_0) = m_0$.

Similar to (2.15), from $\langle J_{\mu_n}'(u_{\mu_n}), u_{\mu_n}^\pm\rangle = 0$, we can deduce that $\|u_0^\pm\|_\lambda^2 > 0$. So $u_0^\pm\neq0$, that is, $u_0$ is a sign-changing solution for (1.9).

Next, we will prove that $u_0$ is also a ground state solution for (1.9). Similar to the discussion of Theorem 1.1, we can obtain that (1.9) has a ground state sign-changing solution when $\mu = 0$. That is to say, we have that $v_0\in\mathcal{M}_0$ such that $J_0'(v_0) = 0$ and $J_0(v_0) = m_0$. Thanks to Lemma 2.1, there exists only a pair of positive numbers $(s_{\mu_n}, t_{\mu_n})$ such that $s_{\mu_n}v_0^++t_{\mu_n}v_0^-\in\mathcal{M}_{\mu_n}$. Then, we need to prove that $\{s_{\mu_n}\}$ and $\{t_{\mu_n}\}$ are bounded. Indeed, we assume that $\lim\limits_{n\rightarrow \infty}s_{\mu_n} = \infty$. According to $(f_1)$ and $(f_4)$, for any $a > 0$, there is $b > 0$ such that

Then, let $a > 0$ sufficiently enough, thanks to (3.17), Lemma 2.2 and the Young inequality, we have

This is a contradiction. Hence, $\{s_{\mu_n}\}$ is bounded in $\mathbb{R}$. Analogously, $\{t_{\mu_n}\}$ is bounded in $\mathbb{R}$. Then, by $(f_3)$, we obtain

Consequently, thanks to (3.18), we get

Hence, by $\langle J_0'(v_0), v_0^\pm\rangle = 0$, we infer that

which implies that $\limsup\limits_{n\rightarrow \infty}m_{\mu_n}\leq m_0$. Then, thanks to (2.16) and the Fatou Lemma, one has

Hence, $J_0(u_0) = m_0$. In conclusion, $u_0$ is a ground state sign-changing solution of equation (1.9). By the same proof method as in Theorem 1.1, we can obtain that $u_0$ has two nodal domains. Hence, we complete the proof of Theorem 1.3. □

Use of AI tools declaration

The authors declare that they have not used artificial intelligence tools in the creation of this article.

Acknowledgements

The authors express their gratitude to the reviewers for their careful reading and helpful suggestions which led to an improvement of the original manuscript. This research was supported by the Natural Science Foundation of Sichuan [2022NSFSC1847].

Huang Xiao-Qing wrote the main manuscript, and Liao Jia-Feng wrote and revised the manuscript.

Conflict of interest

The authors declare no conflict of interest.