Existence and asymptotic behaviour of positive ground state solution for critical Schrödinger-Bopp-Podolsky system

1.   Introduction and main results

In this paper, we deal with the following system:

where $\lambda\geqslant0$, $\varepsilon > 0$, $f$ is a continuous, superlinear and subcritical nonlinearity. $V:\mathbb{R}^{3} \rightarrow \mathbb{R}$ is a continuous function satisfying the following conditions:

$(V_{1}) \; 0 < V(x) < V_{\infty}: = \liminf\limits_{|x|\rightarrow +\infty}V(x) < +\infty$.

$(V_{2})$ There exists a constant $\alpha > 0$ such that

Furthermore, for the potential function $K$, we assume:

$(K) \; K\in C(\mathbb{R}^{3}, \mathbb{R})$ and $K_{\infty}: = \limsup\limits_{|x|\rightarrow +\infty} K(x) \in(0, +\infty)$ and $K(x)\geqslant K_{\infty}$ for $x\in\mathbb{R}^{3}$.

The system $\left(P_{\lambda, \varepsilon}\right)$ is a version of the so called Schrödinger-Bopp-Podolsky system, which is a Schrödinger equation coupled with a Bopp-Podolsky equation. Podolsky's theory has been proposed by Bopp ^[1] and independently by Podolsky-Schwed ^[2] as a second order gauge theory for the electromagnetic field. It appears when one look for standing waves solutions $\psi(x, t) = u(x)e^{i\omega t}$ of the Schrödinger equation coupled with the Bopp-Podolsky Lagrangian of the electromagnetic field, in the purely electrostatic situation. In the physical point of view, $\varepsilon$ is the parameter of the Bopp-Podolsky term, $u$ and $\phi$ represent the modulus of the wave function and the electrostatic potential, respectively. As for more details and physical applications of the Bopp-Podolsky equation, we refer to ^[3,4,5] and the references therein.

From a mathematical point of view, the study of system $\left(P_{\lambda, \varepsilon}\right)$ can be divided into two cases: (1) $\varepsilon = 0$; (2) $\varepsilon\neq0$.

If $\varepsilon = 0$, then system $\left(P_{\lambda, \varepsilon}\right)$ gives back the classical Schrödinger-Poisson system as follows:

which has been introduced by Benci-Fortunato ^[6] in quantum mechanics as a model describing the interaction of a charged particle with the electrostatic field. In such system, the potential function $V$ is regarded as an external potential, $u$ and $\phi$ represent the wave functions associated with the particle and electric potential, respectively. For more details on the physical aspects of this system, we refer the readers to ^[7,8,9] and the references therein.

In last decades, system (1.1) has been widely studied under variant assumptions on $V$ and $f$, by variational methods, and existence, nonexistence and multiplicity results are obtained in many papers. For further details, we refer the readers to previous studies ^{[10,11,12,13,14,15]} and the references therein.

In particular, Azzollini-Pomponio ^[16] proved the existence of ground state solutions to system (1.1) with $f(x, u) = |u|^{p-1}$ and $3 < p < 5$. Ambrosetti-Ruiz ^[17] obtained multiple solutions to system (1.1) by the monotonicity skills combined with minimax methods. Ruiz ^[9] dealt with the following Schrödinger-Poisson system:

where $2 < p < 6$ and $\lambda > 0$. Via a constraint variational method combining the Nehari-Pohožaev manifold, the existence and nonexistence results were obtained.

If $\varepsilon\neq0$, then system $\left(P_{\lambda, \varepsilon}\right)$ is a Schrödinger-Bopp-Podolsky system. D'Avenia-Siciliano ^[18] first studied the following system from a mathematical point of view:

where $\omega > 0$, $\varepsilon\geqslant0$ and $q\neq0$. Based on the variational methods, D'Avenia-Siciliano ^[18] proved the existence and nonexistence results to system (1.3) depending on the parameters $p$ and $q$.

Later, for $p\in(2, 3]$, Siciliano-Silva ^[19] obtained the existence and nonexistence of solutions to system (1.3) by means of the fibering map approach and the implicit function theorem.

Motivated by all results mentioned above, a series of interesting questions naturally arises such as:

${\bf(I)}$ As we can see, the authors in ^[18] and ^[19] merely considered system (1.3) with subcritical growth, so we would much like to know whether similar results hold for system $\left(P_{\lambda, \varepsilon}\right)$ if its nonlinearity is at critical growth.

${\bf(II)}$ Note that in ^[18] and ^[19], the authors studied the existence and nonexistence results to system (1.3), but it has not been considered the asymptotic behavior of solutions. Therefore, it is natural to ask a question. Can we obtain the asymptotic behavior of solutions to system $\left(P_{\lambda, \varepsilon}\right)$?

Compared to ^[18] and ^[19], the main purpose of this paper is to fill the gaps. More specifically, we will study the existence, nonexistence and asymptotic behavior of ground state solutions to system $\left(P_{\lambda, \varepsilon}\right)$ involving a critical nonlinearity.

Now we state our conditions on $f$. Let $f:\mathbb{R}\rightarrow \mathbb{R}$ be a continuous function such that

$(F_{1}) \; f(t) = o(t^{3})$ as $t\rightarrow0$ and $f(t) = 0$ for all $t\leqslant0$.

$(F_{2}) \; \frac{f(t)}{t^{3}}$ is strictly increasing on interval $(0, +\infty)$.

$(F_{3}) \; |f(t)|\leqslant C(1+|t|^{p-1})$ and $f(t)\geqslant \gamma t^{m-1}$ for some $C > 0$ and $\gamma > 0$, where $4 < p, m < 6$.

1.1.   Main Results

We divide the study of system $\left(P_{\lambda, \varepsilon}\right)$ into three parts: (I) $V(x)\equiv V_{\infty}$ and $K(x)\equiv K_{\infty}$; (II) $V(x) < V_{\infty}$ and $K(x)\geqslant K_{\infty}$; (III) $V(x)\geqslant V_{\infty}$ and $K(x)\leqslant K_{\infty}$, where one of the strictly inequality holds on a positive measure subset.

(I) For $V(x)\equiv V_{\infty}$ and $K(x)\equiv K_{\infty}$, system $\left(P_{\lambda, \varepsilon}\right)$ goes back to its limit system:

Our first result is as follows:

Theorem 1.1. Suppose that $\lambda > 0$ and conditions $(F_{1})$-$(F_{3})$ hold, then system $\left(P_{\infty}\right)$ possesses a positive ground state solution $(u_{\infty}, \phi_{\infty})\in H^{1}_{V}(\mathbb{R}^{3})\times\mathcal{D}$, where spaces $H^{1}_{V}(\mathbb{R}^{3})$ and $\mathcal{D}$ are given in section 2 below.

(II) System $\left(P_{\lambda, \varepsilon}\right)$ with $V(x) < V_{\infty}$ and $K(x)\geqslant K_{\infty}$. Our second result is as follows:

Theorem 1.2. Suppose that $\lambda > 0$, conditions $(V_{1})$-$(V_{2})$, $(K)$ and $(F_{1})$-$(F_{3})$ hold. Then the following statements are true.

$(i)$ System $\left(P_{\lambda, \varepsilon}\right)$ possesses a positive ground state solution $(u_{\lambda, \varepsilon}, \phi_{u}^{\varepsilon})\in H^{1}_{V}(\mathbb{R}^{3})\times\mathcal{D}$.

$(ii)$ For every fixed $\varepsilon > 0$, we have

$(iii)$ There exist $\lambda^{*} > 0$ and $\widetilde{\lambda} > \lambda^{*}$ be fixed. Let $(u_{\widetilde{\lambda}, \varepsilon}, \phi_{u}^{\varepsilon})$ be a solution of system $\left(P_{\lambda, \varepsilon}\right)$ in correspondence of $\widetilde{\lambda}$.Then we have

where $(u_{\widetilde{\lambda}, 0}, \phi_{u}^{0})\in H^{1}_{V}(\mathbb{R}^{3})\times D^{1, 2}(\mathbb{R}^{3})$ is a positive ground state solutionof

By virtue of the symmetric mountain pass theorem, we also obtain a supplementary result ofthe infinity many positive solutions for system $\left(P_{\lambda, \varepsilon}\right)$.Our third result is as follows:

Theorem 1.3. Suppose that conditions $(V_{1})$-$(V_{2})$, $(K)$ and $(F_{1})$-$(F_{3})$ hold, and suppose that $f(u)$ is odd.Then system $\left(P_{\lambda, \varepsilon}\right)$ possesses infinitely many positive solutions.

(III) System $\left(P_{\lambda, \varepsilon}\right)$ with $V(x)\geqslant V_{\infty}$ and $K(x)\leqslant K_{\infty}$, which one of the strictly inequality holds on a positive measure subset. Our last result is as follows:

Theorem 1.4. Suppose that conditions $(F_{1})$-$(F_{3})$ hold, then for any $\lambda > 0$, $\varepsilon > 0$, system $\left(P_{\lambda, \varepsilon}\right)$ has no ground state solution.

Remark 1.1. To our best knowledge, there is still no results concerning the existence and asymptotic behavior of solutions for Schrödinger-Bopp-Podolsky system with critical exponent. Hence our results are new. By comparing with ^[18] and ^[19], we have to face three major difficulties. First, the existence of critical term and noncompact potential function $V(x)$ set an obstacle that the bounded $(PS)$ sequences may not converge. Second, the presence of the potential functions $V(x)$ and $K(x)$ cause the splitting lemma for recovering the compactness developed in ^[18] cannot be applied to system $\left(P_{\lambda, \varepsilon}\right)$. Third, the Podolsky's term in system $\left(P_{\lambda, \varepsilon}\right)$ makes the corresponding Brézis-Lieb type convergence lemma invalid. As we will see later, these difficulties prevent us from using the way as in ^[18] and ^[19]. So we need some new tricks to deal with these essential problems.

Remark 1.2. The proof of Theorems 1.1 and 1.2 is mainly based on the methods of the Nehari manifold and the concentration compactness principle ^[20]. However, since the nonlinearity $f$ is only continuous, we cannot use standard arguments on the Nehari manifold. To overcome the non-differentiability of the Nehari manifold, we shall use some variants of critical point theorems from Szulkin-Weth ^[21]. At the same time, because of the presence of the potential functions $V(x)$ and $K(x)$, it is difficult to study the minimization problem of system $\left(P_{\lambda, \varepsilon}\right)$ directly. Therefore we first study its limit system $\left(P_{\infty}\right)$, which is given in section 3. Then by comparing the ground state energy between system $\left(P_{\lambda, \varepsilon}\right)$ and $\left(P_{\infty}\right)$, the existence results is obtained.

In Theorems 1.1 and 1.2, we just consider the following two cases: (i) $V(x)\equiv V_{\infty}$ and $K(x)\equiv K_{\infty}$; (ii) $V(x) < V_{\infty}$ and $K(x)\geqslant K_{\infty}$. This motivates an interesting open problem: Does the existence of ground state solutions for system $(P_{\lambda, \varepsilon})$ hold for $V(x) < V_{\infty}$ or $K(x)\geqslant K_{\infty}$?

The remainder of this paper is as follows. In section 2, variational setting and some preliminaries are presented. In sections 3 to 6, the proof of Theorems 1.1 to 1.4 is given, respectively.

2.   Preliminaries and variational settings

Throughout this paper, the letters $C$, $C_{i}\; (i = 1, 2...)$ will denote possibly different positive constants which may change from line to line.

Let

endowed with the inner product

and the related norm

Under conditions $(V_{1})$-$(V_{2})$, it is easy to see that the norms $\|u\|_{H_{V}^{1}(\mathbb{R}^{3})}$ and $\|u\|_{H^{1}(\mathbb{R}^{3})}$ are equivalent and the embedding $H_{V}^{1}(\mathbb{R}^{3}) \hookrightarrow L^{s}(\mathbb{R}^{3})$ is continuous for each $s\in[2, 6]$.

Next we outline the variational framework for system $\left(P_{\lambda, \varepsilon}\right)$ and give some preliminary lemmas. In particular, we give some fundamental properties on the operator $-\Delta+\varepsilon^{2}\Delta^{2}$.

2.1.   The variational settings

We define $\mathcal{D}$ be the completion of $C^{\infty}_{0}(\mathbb{R}^{3})$ with respect to the norm $\|\cdot\|_{\mathcal{D}}$ induced by the scalar product

Then $\mathcal{D}$ is a Hilbert space, which is continuously embedded into $D^{1, 2}(\mathbb{R}^{3})$ and consequently in $L^{6}(\mathbb{R}^{3})$.

Lemma 2.1. (^[18]) The space $\mathcal{D}$ is continuously embeddedinto $L^{\infty}(\mathbb{R}^{3})$.

We recall that by the Lax-Milgram theorem, for every fixed $u\in H^{1}_{V}(\mathbb{R}^{3})$, there exists a unique solution $\phi_{u}^{\varepsilon}\in\mathcal{D}$ of the second equation in system $\left(P_{\lambda, \varepsilon}\right)$. To write explicitly such a solution (see ^[5]), we consider

For $\mathcal{K}$, we have the following fundamental properties.

Lemma 2.2. (^[18]) For all $y\in\mathbb{R}^{3}$, $\mathcal{K}(\cdot-y)$ solves in the sense of distributions

Moreover,

$(i)$ if $f\in L^{1}_{loc}(\mathbb{R}^{3})$ and for a.e.$x\in\mathbb{R}^{3}$, the map $y\in\mathbb{R}^{3}\rightarrow \frac{f(y)}{|x-y|}$ is summable, then $\mathcal{K}*f\in L^{1}_{loc}(\mathbb{R}^{3})$;

$(ii)$ if $f\in L^{p}(\mathbb{R}^{3})$ with $1\leqslant p < \frac{3}{2}$, then $\mathcal{K}*f\in L^{q}(\mathbb{R}^{3})$ for $q\in(\frac{3p}{3-2p}, +\infty]$.

In both cases $\mathcal{K}*f$ solves

Then if we fix $u\in H^{1}_{V}(\mathbb{R}^{3})$, the unique solution in $\mathcal{D}$ of the second equation in system $\left(P_{\lambda, \varepsilon}\right)$ can be expressed by

Now, let us summarize some properties of $\phi_{u}^{\varepsilon}$.

Lemma 2.3. (^[18]) For every $u, v\in H^{1}_{V}(\mathbb{R}^{3})$, the following statements are true.

$(i) \; \phi_{u}^{\varepsilon}\geqslant0$.

$(ii)$ For each $t > 0$, $\phi_{tu}^{\varepsilon} = t^{2}\phi_{u}^{\varepsilon}$.

$(iii)$ If $u_{n}\rightharpoonup u$ in $H^{1}_{V}(\mathbb{R}^{3})$, then $\phi_{u_{n}}^{\varepsilon}\rightharpoonup\phi_{u}^{\varepsilon}$ in $\mathcal{D}$.

$(iv) \|\phi_{u}^{\varepsilon}\|_{\mathcal{D}} \leqslant C\|u\|_{L^{\frac{12}{5}}(\mathbb{R}^{3})}^{2} \leqslant C\|u\|_{H_{V}^{1}(\mathbb{R}^{3})}^{2}$ and $\int_{\mathbb{R}^{3}}\phi_{u}^{\varepsilon}|u|^{2}\text{d}x \leqslant C\|u\|_{L^{\frac{12}{5}}(\mathbb{R}^{3})}^{4} \leqslant C\|u\|^{4}_{H_{V}^{1}(\mathbb{R}^{3})}$.

Lemma 2.4. (^[18]) Consider $f\in L^{\frac{6}{5}}(\mathbb{R}^{3})$, $\{f_{\varepsilon}\}_{\varepsilon\in(0, 1)}\subset L^{\frac{6}{5}}(\mathbb{R}^{3})$ and let

and

As $\varepsilon\rightarrow0$, we have:

$(i)$ If $f_{\varepsilon}\rightharpoonup f$ in $L^{\frac{6}{5}}(\mathbb{R}^{3})$, then $\phi_{u_{f}}^{\varepsilon}\rightharpoonup\phi_{u_{f}}^{0}$ in $D^{1, 2}(\mathbb{R}^{3})$.

$(ii)$ If $f_{\varepsilon}\rightarrow f$ in $L^{\frac{6}{5}}(\mathbb{R}^{3})$, then $\phi_{u_{f}}^{\varepsilon}\rightarrow \phi_{u_{f}}^{0}$ in $D^{1, 2}(\mathbb{R}^{3})$ and $\varepsilon\Delta\phi_{u_{f}}^{\varepsilon}\rightarrow0$ in $L^{2}(\mathbb{R}^{3})$.

By using the classical reduction argument, system $\left(P_{\lambda, \varepsilon}\right)$ can be reduced to a single equation:

Then from now on we speak of solutions of system $\left(P_{\lambda, \varepsilon}\right)$ is equal to the solutions of equation (2.1). It is easy to see that the solutions of equation (2.1) can be regarded as critical points of the energy functional $I_{\lambda, \varepsilon}$: $H^{1}_{V}(\mathbb{R}^{3})\rightarrow \mathbb{R}$ defined by

From $(F_{1})$ and $(F_{3})$, it is easy to check that $I_{\lambda, \varepsilon}$ is a well defined $C^{1}$ functional in $H^{1}_{V}(\mathbb{R}^{3})$. Moreover, $\forall\varphi\in H^{1}_{V}(\mathbb{R}^{3})$,

2.2.   The Key Lemmas

The following lemma is the Young convolution inequality, which is a fundamental tool in our analysis.

Lemma 2.5. (^[27]) If $G\in L^{q}(\mathbb{R}^{3})$ and $H\in L^{r}(\mathbb{R}^{3})$ with $1 < \frac{1}{q}+\frac{1}{r}\leqslant2$, then $G*H\in L^{s}(\mathbb{R}^{3})$ with $\frac{1}{s} = \frac{1}{q}+\frac{1}{r}-1$ and

We will apply the concentration compactness principle ^[20] and vanishing lemma ^[22] to prove the compactness of $(PS)$ sequence of $I_{\lambda, \varepsilon}$. Now, we recall them as follows.

Proposition 2.1. (^[20]) Let $\rho_{n}(x)\in L^{1}(\mathbb{R}^{3})$ be a nonnegative sequence satisfying

Then there exists a subsequence, still denoted by $\{\rho_{n}(x)\}$, such that one of the following cases occurs.

(i) Compactness: There exists $\{y_{n}\}\in\mathbb{R}^{3}$, such that for each $\epsilon > 0$, there exists $R > 0$ such that

(ii) Vanishing: For every fixed $R > 0$, there holds

(iii) Dichotomy: There exist $\beta > 0$ with $0 < \beta < l$, sequence $\{R_{n}\}$ with $R_{n}\rightarrow +\infty$ and two functions $\rho_{n}^{1}(x), \rho_{n}^{2}(x)\in L^{1}(\mathbb{R}^{3})$, $\{y_{n}\}\subset\mathbb{R}^{3}$ such that for each $\epsilon > 0$, there exists $n_{0}\in\mathbb{N}^{*}$, for $n\geqslant n_{0}$, there holds

and

Proposition 2.2. (^[22]) Suppose that $\{u_{n}\}$ is bounded in $H^{1}(\mathbb{R}^{3})$ and it satisfies

where $R > 0$.Then $u_{n}\rightarrow0$ in $L^{s}(\mathbb{R}^{3})$ for $s\in(2, 6)$.

3.   The Proof of Theorem 1.1

In this section, we shall prove the existence of positive ground state solutions to system $\left(P_{\infty}\right)$.

Set

endowed with the inner product

and the related norm

By the Lax-Milgram theorem and Lemma 2.2, we can define the energy functional corresponding to system $\left(P_{\infty}\right)$ by

3.1.   Mountain Pass Geometry and Nehari Manifold

The Nehari manifold corresponding to $I_{\infty}$ is defined by

We can conclude $\mathcal{N}_{\infty}$ has the following elementary properties.

Lemma 3.1. (See Appendix) Suppose that $\varepsilon > 0$ be fixed and conditions $(F_{1})$-$(F_{3})$ hold. Then the following statements are true.

(i) The functional $I_{\infty}$ possesses the mountain pass geometry.

(ii) For each $u\in H^{1}_{V_{\infty}}(\mathbb{R}^{3})\backslash\{0\}$, there exists a unique $t_{u} > 0$ such that $I_{\infty}(t_{u}u) = \max\limits_{t\geqslant0}I_{\infty}(tu)$.Moreover, $tu\in\mathcal{N}_{\infty}$ if and only if $t = t_{u}$ and

(iii) $c_{\infty} = \bar{c}_{\infty} = \bar{\bar{c}}_{\infty} > 0$, where

and $\Gamma = \big\{\gamma\in C([0, 1], H_{V_{\infty}}^{1}(\mathbb{R}^{3}))\big|\; \gamma(0) = 0, I_{\infty}(\gamma(1)) < 0\big\}$.

According to Lemma 3.1 $(i)$, it follows that for any $u\in H_{V_{\infty}}^{1}(\mathbb{R}^{3}) \backslash \{0\}$, there exists a unique $t_{u} > 0$ such that $t_{u}u\in\mathcal{N}_{\infty}$. We define a mapping $\widehat{m}_{\infty}: H_{V_{\infty}}^{1}(\mathbb{R}^{3}) \backslash\{0\} \rightarrow \mathcal{N}_{\infty}$ by

Moreover, the inverse of $m_{\infty}$ can be given by

Considering the functionals $\widehat{\Upsilon}_{\infty}: H_{V_{\infty}}^{1}(\mathbb{R}^{3}) \backslash \{0\} \rightarrow \mathbb{R}$ and $\Upsilon_{\infty}: S_{\infty}\rightarrow \mathbb{R}$ given by

Then we have the following lemma.

Lemma 3.2. (^[21]) Suppose that all conditions described in Lemma 3.1 hold. Then the following statements are true.

$(i) \; \Upsilon_{\infty}\in C^{1}(S_{\infty}, \mathbb{R})$ and

for all $z\in T_{\omega}(S_{\infty}): = \big\{v\in H_{V_{\infty}}^{1}(\mathbb{R}^{3})\big|\big\langle\omega, v\big\rangle = 0\big\}$.

$(ii) \; \{\omega_{n}\}$ is a $(PS)$ sequence for $\Upsilon_{\infty}$, if and only if $\{m_{\infty}(\omega_{n})\}$ is a $(PS)$ sequence for $I_{\infty}$. If $\{u_{n}\}\subset\mathcal{N}_{\infty}$ is a bounded $(PS)$ sequence for $I_{\infty}$, then $\{m_{\infty}^{-1}(u_{n})\}$ is a $(PS)$ sequence for $\Upsilon_{\infty}$.

$(iii) \; \omega\in S_{\infty}$ is a critical point of $\Upsilon_{\infty}$, if and only if $m_{\infty}(\omega)$ is a critical point of $I_{\infty}$.Moreover, the corresponding values of $I_{\infty}$ and $\Upsilon_{\infty}$ coincide and

3.2.   Estimates of $c_{_{\infty}}$

The main feature of the functional $I_{\infty}$ is that it satisfies the local compactness condition, as we can see in the following result.

Lemma 3.3. For all $\lambda, \varepsilon > 0$, there exists some $v\in H^{1}_{V_{\infty}}(\mathbb{R}^{3})\backslash\{0\}$ such that

where $\mathcal{S} = \inf\limits_{u\in D^{1, 2}(\mathbb{R}^{3})\backslash\{0\}}\frac{\|u\|_{D^{1, 2}(\mathbb{R}^{3})}^{2}}{\|u\|_{L^{6}(\mathbb{R}^{3})}^{2}}$.

Proof. For each $\epsilon > 0$, consider the function

where $C$ is a normalized constant. We recall that $U_{\epsilon}$ satisfies

and

Let $\eta\in C^{\infty}_{0}(\mathbb{R}^{3}, [0, 1])$ be such that $0\leqslant\eta\leqslant1$, if $|x| < 1$ and $\eta = 0$ if $|x|\geqslant2$. Now, consider $v_{\epsilon}(x) = \eta U_{\epsilon}/\|\eta U_{\epsilon}\|_{L^{6}(\mathbb{R}^{3})}$ then we have the following estimates, if $\epsilon > 0$ small enough:

By $(F_{3})$, we obtain

Note that $\lim\limits_{t\rightarrow +\infty} J_{\infty}(t) = -\infty$ and $J_{\infty}(t) > 0$ as $t > 0$ small enough. So $\sup\limits_{t\geqslant0}J_{\infty}(t)$ is attained at some $t_{\epsilon} > 0$.

From

we have

which implies that $t_{\epsilon}$ is bounded from above by some $t^{*} > 0$. In view of (3.3), we get

Choosing $\epsilon > 0$ small enough, by (3.1), we obtain

Thus, we have $t_{\epsilon}$ is bounded from above and below for $\epsilon > 0$ small enough.

Next, we estimate $J_{\infty}(t)$. Set

Then $g(t)$ attains its maximum at $\overline{t} = \big(\int_{\mathbb{R}^{3}}|\nabla v_{\epsilon}|^{2}\text{d}x\big)^{\frac{1}{4}}$. Consequently, by (3.2) and Lemma 2.3, there holds

for $\epsilon > 0$ small enough. Thus, $\max\limits_{t\geqslant0} I_{\infty}(tv_{\epsilon}) < \frac{1}{3}\mathcal{S}^{\frac{3}{2}}$ is obtained. The proof is completed.

Lemma 3.4. The following statement holds:

Proof. We need to prove that for every $\epsilon > 0$, there exists $\overline{\lambda} > 0$ such that

Let $v\in C_{0}^{\infty}(\mathbb{R}^{3})$, with $\|v\|_{H^{1}_{V_{\infty}}(\mathbb{R}^{3})} = 1$. In view of Lemma 3.1, we know that there exists $t_{v} > 0$ such that $I_{\infty}(t_{v}v) = \max\limits_{t\geqslant0}I_{\infty}(tv)$ and $\lim\limits_{\lambda\rightarrow +\infty} \sup\limits_{\varepsilon > 0}t_{v} = 0$. By virtue of Lemmas 2.3 and 3.1 for $\lambda > \overline{\lambda}$, we have the following estimates:

The proof is completed.

To prove the compactness of the minimizing sequence for $I_{\infty}$, we need the following result.

Lemma 3.5. Let $\{u_{n}\}\subset\mathcal{N}_{\infty}$ be a minimizing sequence for $I_{\infty}$.Then $\{u_{n}\}$ is bounded.Moreover, there exist $r, \delta > 0$ and a sequence $\{y_{n}\}\subset \mathbb{R}^{3}$ such that

where $B_{r}(y_{n}) = \big\{y\in\mathbb{R}^{3}\big||y-y_{n}|\leqslant r\big\}$.

Proof. For any $\epsilon > 0$, it follows from $(F_{1})$, $(F_{3})$ that there exists $C_{\epsilon} > 0$ such that

In view of $(F_{2})$, one can see that

By $\{u_{n}\}\subset\mathcal{N}_{\infty}$, we have

Hence, $I_{\infty}$ is coercive on $\mathcal{N}_{\infty}$, i.e., $I_{\infty}(u)\rightarrow +\infty$ as $\|u\|_{H^{1}_{V_{\infty}}(\mathbb{R}^{3})} \rightarrow +\infty$, for $u\in\mathcal{N}_{\infty}$. Thus, we can easily get the boundedness of $\{u_{n}\}$.

Next we prove the latter conclusion of this lemma. Arguing by contradiction, we assume

then by Proposition 2.2, there holds $u_{n}\rightarrow0$ in $L^{s}(\mathbb{R}^{3})$ for $s\in(2, 6)$. Taking into account (3.5) and Lemma 2.3, we can deduce

So, combined $\langle I'_{\infty} (u_{n}), u_{n} \rangle = 0$ with (3.7), we have

We assume $\|u_{n}\|^{2}_{H^{1}_{V_{\infty}}(\mathbb{R}^{3})} \rightarrow l \geqslant0$. If $l > 0$, by $\{u_{n}\}$ is a minimizing sequence of $I_{\infty}$ and (3.7), we get

Thus, we obtain $c_{\infty} = \frac{1}{3}l$. On the other hand, by the definition of $\mathcal{S}$, we know that $l\geqslant\mathcal{S}l^{\frac{1}{3}}$. Namely, $l\geqslant\mathcal{S}^{\frac{3}{2}}$. So $c_{\infty} = \frac{1}{3}l \geqslant \frac{1}{3} \mathcal{S}^{\frac{3}{2}}$. This contradicts with Lemma 3.3. Hence $l = 0$. However, this contradicts with Lemma 3.1. The proof is completed.

Now we are in a position to give the proof of Theorem 1.1.

Proof of Theorem 1.1. Let $\{\omega_{n}\}\subset S_{V_{\infty}}$ be a minimizing sequence of $\Upsilon_{\infty}$. By the Ekeland variational principle ^[23], we assume

Set $u_{n} = m_{\infty} (\omega_{n}) \in \mathcal{N}_{\infty}$ for all $n\in\mathbb{N}^{*}$. Then

By Lemma 3.5, we know that $\{u_{n}\}$ is bounded and there exist $r, \delta > 0$ and a sequence $\{y_{n}\}\subset\mathbb{R}^{3}$ such that

So we can choose $r_{1} > r > 0$ and a sequence $\{y_{n}^{1}\} \subset \mathbb{R}^{3}$ such that

Since $I_{\infty}$ and $\mathcal{N}_{\infty}$ are invariant under translations in our case, so we can assume $\{y_{n}\} \subset \mathbb{Z}^{3}$ is bounded. Moreover we assume, up to a subsequence, there exists $u_{\infty} \in H_{V_{\infty}}^{1}(\mathbb{R}^{3})$ such that $u_{n}\rightharpoonup u_{\infty}$ and $u_{n}\rightarrow u_{\infty}$ a.e. in $\mathbb{R}^{3}$. Then the weak convergence of $\{u_{n}\}$ implies $I'_{\infty} (u_{\infty}) = 0$.

According to the Fatou lemma, we can obtain

which implies $I_{\infty} (u_{\infty}) = c_{\infty}$. Next, we need to show the ground state solution $u_{\infty}$ is positive. In fact, for $|u_{\infty}|\in H^{1}_{V_{\infty}}(\mathbb{R}^{3})$, there exists $t_{\infty} > 0$ such that $t_{\infty}|u_{\infty}|\in \mathcal{N}_{\infty}$. From $(F_{1})$ and the form of $I_{\infty}$, we can infer $I_{\infty} (t_{\infty}|u_{\infty}|) \leqslant I_{\infty} (t_{\infty}u_{\infty})$. Furthermore, it follows from $u_{\infty} \in \mathcal{N}_{\infty}$ that $I_{\infty} (t_{\infty}u_{\infty}) \leqslant I_{\infty} (u_{\infty})$. So, we obtain $I_{\infty} (t_{\infty}|u_{\infty}|) \leqslant I_{\infty} (u_{\infty})$, which implies $t_{\infty}|u_{\infty}|$ is a nonnegative ground state solution. It follows from the Harnack inequality ^[24] that $t_{\infty}|u_{\infty}| > 0$, for all $x\in\mathbb{R}^{3}$. The proof is completed.

4.   Proof of Theorem 1.2

In this section, we investigate the existence of positive ground state solutions to system $\left(P_{\lambda, \varepsilon}\right)$.

4.1.   Mountain pass geometry and Nehari Manifold

Define the Nehari manifold of system $\left(P_{\lambda, \varepsilon}\right)$ as follows:

We can conclude $\mathcal{N}_{\lambda, \varepsilon}$ has the following elementary properties without proof.

Lemma 4.1. Suppose that all conditions described in Theorem 1.2 hold. Then the following statements are true.

(i) The functional $I_{\lambda, \varepsilon}$ possesses the mountain pass geometry.

(ii) For every $u\in H^{1}_{V}(\mathbb{R}^{3})\backslash\{0\}$ and a fixed $\varepsilon > 0$, there exists a unique $t_{u} > 0$ such that $I_{\lambda, \varepsilon}(t_{u}u) = \max\limits_{t\geqslant0}I_{\lambda, \varepsilon}(tu)$.Moreover, $tu\in\mathcal{N}_{\lambda, \varepsilon}$ if and only if $t = t_{u}$ and

(iii) $c_{\lambda, \varepsilon} = \bar{c}_{\lambda, \varepsilon} = \bar{\bar{c}}_{\lambda, \varepsilon} > 0$, where

and $\Gamma = \big\{\gamma\in C([0, 1], H_{V}^{1}(\mathbb{R}^{3}))\big|\; \gamma(0) = 0, I_{\lambda, \varepsilon}(\gamma(1)) < 0\big\}$.

Proof. The proof is similar to Lemma 3.1, so we omit it for details.

Similar to section 2, we define the mappings $\widehat{m}_{\lambda, \varepsilon}: H_{V}^{1}(\mathbb{R}^{3}) \backslash\{0\} \rightarrow \mathcal{N}_{\lambda, \varepsilon}$ by

Moreover, the inverse of $m_{\lambda, \varepsilon}$ can be given by

Considering the functionals $\widehat{\Upsilon}_{\lambda, \varepsilon}: H^{1}_{V}(\mathbb{R}^{3}) \backslash \{0\} \rightarrow \mathbb{R}$ and $\Upsilon_{\lambda, \varepsilon}: S\rightarrow \mathbb{R}$ given by

Then we have the following lemma.

Lemma 4.2. (^[21]) Suppose that all conditions described in Lemma 4.1 hold. Then the following statements are true.

$(i) \; \Upsilon_{\lambda, \varepsilon}\in C^{1}(S, \mathbb{R})$ and

for all $z \in T_{\omega}(S): = \big\{v\in H^{1}_{V}(\mathbb{R}^{3})\big|\big\langle\omega, v\big\rangle = 0\big\}$.

$(ii) \; \{\omega_{n}\}$ is a $(PS)$ sequence for $\Upsilon_{\lambda, \varepsilon}$, if and only if $\{m_{\lambda, \varepsilon}(\omega_{n})\}$ is a $(PS)$ sequence for $I_{\lambda, \varepsilon}$. If $\{u_{n}\}\subset\mathcal{N}_{\lambda, \varepsilon}$ is a bounded $(PS)$ sequence for $I_{\lambda, \varepsilon}$, then $\{m_{\lambda, \varepsilon}^{-1}(u_{n})\}$ is a $(PS)$ sequence for $\Upsilon_{\lambda, \varepsilon}$.

$(iii) \; \omega\in S$ is a critical point of $\Upsilon_{\lambda, \varepsilon}$, if and only if $m_{\lambda, \varepsilon}(\omega)$ is a critical point of $I_{\lambda, \varepsilon}$.Moreover, the corresponding values of $I_{\lambda, \varepsilon}$ and $\Upsilon_{\lambda, \varepsilon}$ coincide and

In order to prove that the minimizer of $I_{\lambda, \varepsilon}$ constrained on $\mathcal{N}_{\lambda, \varepsilon}$ is a critical point of $I_{\lambda, \varepsilon}$, we need the following lemmas.

4.2.   The behaviors of $(PS)_{c}$ sequence

In this subsection, we study the behaviors of $(PS)_{c}$ sequence, which play key roles in the proof of Theorem 1.2.

Lemma 4.3. If $u_{n}\rightharpoonup u$ in $H^{1}_{V}(\mathbb{R}^{3})$ and $u_{n}\rightarrow u$ a.e. in $\mathbb{R}^{3}$, then

Proof. Since $\mathcal{K}\in L^{\tau}(\mathbb{R}^{3})$ for $\tau\in(3, +\infty]$. As a result of $\{u_{n}\}$ is bounded in $H^{1}_{V}(\mathbb{R}^{3})$ and converges almost everywhere to $u$, the sequence $\{|u_{n}-u|^{2}\}$ converges weakly to 0 in $L^{\frac{8}{7}}(\mathbb{R}^{3})$ and by the Brézis-Lieb lemma ^[25], the sequence $\{|u_{n}|^{2}-|u_{n}-u|^{2}\}$ converges strongly to the function $|u|^{2}$ in $L^{\frac{8}{7}}(\mathbb{R}^{3})$. Putting together Lemma 2.5 with the definition of $\phi_{u}^{\varepsilon}$ and letting $n\rightarrow +\infty$, we get

Therefore, we can deduce

The proof is completed.

Lemma 4.4. If $u_{n}\rightharpoonup u$ in $H^{1}_{V}(\mathbb{R}^{3})$ and $u_{n}\rightarrow u$ a.e. in $\mathbb{R}^{3}$, then

Proof. The proof is similar to [26,Lemma 3.2], so we omit it here.

Lemma 4.5. Let $\{u_{n}\}\subset H^{1}_{V}(\mathbb{R}^{3})$ be a $(PS)_{c}$ sequence of $I_{\lambda, \varepsilon}$ with $0 < c\leqslant c_{\infty}$. If $u_{n}\rightharpoonup0$ in $H^{1}_{V}(\mathbb{R}^{3})$, then one of the following statements is true.

$(i)\; u_{n}\rightarrow 0$ in $H^{1}_{V}(\mathbb{R}^{3})$.

$(ii)$ There exist a sequence $\{y_{n}\}\subset\mathbb{R}^{3}$ and constants $r, \delta > 0$ such that

Proof. Suppose that $(ii)$ does not occur, then there exists $r > 0$ such that

In view of Proposition 2.2, we get $u_{n}\rightarrow0$ in $L^{s}(\mathbb{R}^{3})$ for $s\in(2, 6)$. So from (3.7) and $\big\langle I'_{\lambda, \varepsilon}(u_{n}), u_{n} \big\rangle = 0$, it follows that

Assume that $\|u_{n}\|_{H^{1}_{V}(\mathbb{R}^{3})}^{2} \rightarrow l \geqslant 0$. So, we get $c = \frac{1}{3}l$. Moreover, we have

Taking the limit as $n\rightarrow +\infty$ in the above inequality, we obtain

which contradicts with our assumption. Thus, $l = 0$. The proof is completed.

Lemma 4.6. Suppose the all conditions described in Theorem 1.2 hold. Let $\{u_{n}\}\subset H^{1}_{V}(\mathbb{R}^{3})$ be a $(PS)_{c}$ sequence of $I_{\lambda, \varepsilon}$ with $0 < c\leqslant c_{\lambda, \varepsilon} < c_{\infty}$.If $u_{n}\rightharpoonup0$ in $H^{1}_{V}(\mathbb{R}^{3})$, then $u_{n}\rightarrow0$ in $H^{1}_{V}(\mathbb{R}^{3})$.

Proof. It is easy to see that $\{u_{n}\}$ is bounded in $H^{1}_{V}(\mathbb{R}^{3})$. Therefore, up to a subsequence, we have

Next, we use Proposition 2.1 to prove $u_{n}\rightarrow0$ in $H^{1}_{V}(\mathbb{R}^{3})$. For this purpose, we set

Clearly, one has $\{\rho_{n}\} \subset L^{1}(\mathbb{R}^{3})$. Thus, passing to a subsequence, we assume that $\Phi(u_{n}) : = \|\rho_{n}\|_{L^{1}(\mathbb{R}^{3})}\rightarrow l$ as $n\rightarrow +\infty$. Using the fact that $\Phi(u_{n}) = I_{\lambda, \varepsilon}(u_{n}) - \frac{1}{4} \big\langle I'_{\lambda, \varepsilon}(u_{n}), u_{n} \big\rangle = l$, we get $l = c > 0$. We next claim that neither vanishing nor dichotomy occurs.

Claim 1. Vanishing does not occur.

If $\{\rho_{n}\}$ vanishing, then $\{u_{n}^{2}\}$ also vanishing, i.e., there exists $R > 0$ such that

As in the proof of Lemma 4.5, we can prove vanishing does not happen.

Claim 2. Dichotomy does not occur.

Otherwise, there exist $\beta\in(0, l)$ and $\{y_{n}\}\subset\mathbb{R}^{3}$ such that for every $\epsilon_{n} > 0$, we can choose $\{R_{n}\}\subset\mathbb{R}^{+} (R_{n} > \widetilde{R}+R_{0}/\epsilon$, for any fixed $\epsilon > 0$, $\widetilde{R}, R_{0}$ are positive constants defined later) with $R_{n}\rightarrow +\infty$ satisfying

Consider a smooth cut-off function $\psi:[0, +\infty)\rightarrow \mathbb{R}^{+}$ such that

Set

Then, one can infer

and

Let $\Omega_{n} = B_{2R_{n}}(y_{n}) \backslash B_{R_{n}}(y_{n})$. Taking the limit as $n\rightarrow +\infty$, then we have

By (4.4), we can deduce

According to Lemma 2.3, we get

Putting (3.5), (4.5), (4.6) together with the definition of $\theta_{n}$, $\omega_{n}$, we can easily get

Taking into account (4.7)–(4.11), we get

Combining (4.2) and (4.3), we have

Therefore, we obtain

Moreover, from (4.7) to (4.11), we get

In order to finish our proof, it suffices to show (4.13) is not true. We separate the following discussion into three possibilities and show each leads to a contradiction.

Case 1. After passing to a subsequence, we assume $\big\langle I'_{\lambda, \varepsilon}(\theta_{n}), \theta_{n} \big\rangle \leqslant0$, then

By Lemma 4.1, we know that there exists $t_{\theta_{n}} > 0$ such that $t_{\theta_{n}}\theta_{n}\in\mathcal{N}_{\lambda, \varepsilon}$. Then

Combined (4.14) with (4.15), one has

which implies $t_{\theta_{n}}\leqslant1$. From $t_{\theta_{n}}\theta_{n}\in\mathcal{N}_{\lambda, \varepsilon}$ and (4.12), we deduce

which leads to a contradiction.

Case 2. After passing to a subsequence, we assume $\big\langle I'_{\lambda, \varepsilon}(\omega_{n}), \omega_{n} \big\rangle \leqslant0$. This case will lead to a contradiction again as in Case 1.

Case 3. After passing to a subsequence, we assume $\big\langle I'_{\lambda, \varepsilon}(\theta_{n}), \theta_{n} \big\rangle > 0$ and $\big\langle I'_{\lambda, \varepsilon}(\omega_{n}), \omega_{n} \big\rangle > 0$. In view of (4.13), we get $\big\langle I'_{\lambda, \varepsilon}(\theta_{n}), \theta_{n} \big\rangle = o_{n}(1)$ and $\big\langle I'_{\lambda, \varepsilon}(\omega_{n}), \omega_{n} \big\rangle = o_{n}(1)$. Moreover, from (4.7) to (4.11), one has

If the sequence $\{y_{n}\}\subset\mathbb{R}^{3}$ is bounded, then by conditions $(V_{1})$ and $(K)$, we have for every $\epsilon > 0$, there exists $R_{0} > 0$, such that

By the boundedness of $\{y_{n}\}\subset\mathbb{R}^{3}$, there exists $\widetilde{R} > 0$ such that $|y_{n}|\leqslant\widetilde{R}$. Therefore, we have $\mathbb{R}^{3}\backslash B_{R_{n}}(y_{n}) \subset \mathbb{R}^{3}\backslash B_{R_{n}-\widetilde{R}}(0) \subset \mathbb{R}^{3}\backslash B_{R_{0}/\epsilon}(0)$ for $n > 0$ large enough. According to (4.17), it follows that

which implies

Similarly, it is easy to check

Combined (4.18) with (4.19), there holds

By the latter conclusion of (4.20), one has $\big\langle I'_{\infty}(\omega_{n}), \omega_{n} \big\rangle \leqslant0$, as $n\rightarrow +\infty$. Similar to the proof in Case 1, there exists $t_{\omega_{n}}\leqslant1$ such that $t_{\omega_{n}}\omega_{n}\in\mathcal{N}_{\infty}$. Then, we can derive from (4.19) and (4.20) that

which leads to a contradiction.

If $\{y_{n}\}\subset\mathbb{R}^{3}$ is unbounded, we choose a subsequence, stilled denoted by $\{y_{n}\}$, such that $|y_{n}|\geqslant3R_{n}$. Then $B_{2R_{n}}(y_{n}) \subset \mathbb{R}^{3} \backslash B_{R_{n}}(0) \subset \mathbb{R}^{3} \backslash B_{R_{0}/\epsilon}(0)$. Using the fact of (4.17) and a similar proof of (4.18) and (4.19), one has

and

Similar to the case $\{y_{n}\}$ is bounded, we can obtain a contradiction by comparing $I_{\lambda, \varepsilon}(\theta_{n})$ and $c_{\infty}$. Therefore, dichotomy does not occur.

According to the above arguments, by Proposition 2.1, we know that $\{\rho_{n}\}$ must be compactness; i.e, there exists $\{y_{n}\}\subset\mathbb{R}^{3}$ such that for every $\epsilon > 0$, there exists $\widehat{R} > 0$ such that

From the Hölder inequality, we obtain

where $m\in[2, 6]$, $\alpha\in[0, 1]$ and satisfies $\frac{1}{m} = \frac{\alpha}{2}+\frac{1-\alpha}{6}$. By (4.21), we conclude $\{u_{n}^{m}\}$ is also compactness with $m\in[2, 6]$.

Next we prove the sequence $\{y_{n}\}$ is bounded. Otherwise, up to a subsequence, we can choose $\{R_{n}\}\subset\mathbb{R}^{+}$ with $R_{n}\rightarrow +\infty$ satisfying $|y_{n}|\geqslant R_{n}\geqslant\widehat{R}+R_{0}/\epsilon$. Then we have $B_{\widehat{R}}(y_{n}) \subset \mathbb{R}^{3} \backslash B_{R_{n}-\widehat{R}}(0) \subset \mathbb{R}^{3} \backslash B_{R_{0}/\epsilon}(0)$. In view of (4.21), there holds

Similarly, we get

It follows from (4.22) and (4.23) that

By the latter conclusion of (4.24), one can see $\big\langle I'_{\infty}(u_{n}), u_{n} \big\rangle \leqslant0$, as $n\rightarrow +\infty$. Similar to the proof of Case 1, there exists $t_{u_{n}}\leqslant1$ such that $t_{u_{n}}u_{n}\in\mathcal{N}_{\infty}$. It follows from (4.23) and (4.24) that

which leads to a contradiction. Hence, $\{y_{n}\}$ is bounded in $\mathbb{R}^{3}$.

In view of the boundedness of $\{y_{n}\}$ and $u_{n}\rightarrow u$ in $L^{s}_{loc}(\mathbb{R}^{3})$ for $2\leqslant s < 6$, by (4.21) it is easy to check $u_{n}\rightarrow0$ in $L^{s}(\mathbb{R}^{3})$ for $s\in[2, 6)$. Thus, we can derive from Lemma 4.5 that $u_{n}\rightarrow0$ in $H^{1}_{V}(\mathbb{R}^{3})$. The proof is completed.

Now, we state the proof of Theorem 1.2.

Proof of Theorem 1.2. We divide this proof into five steps.

Step 1. Making use of the Ekeland variational principle ^[23], there exists a sequence $\{\omega_{n}\}\subset S$ such that

Set $v_{n} = m_{\lambda, \varepsilon} (\omega_{n})$, we have $v_{n} \in \mathcal{N_{\lambda, \varepsilon}}$ for all $n\in\mathbb{N}^{*}$. By Lemma 4.2, we can deduce

By $\{v_{n}\}$ is bounded in $H^{1}_{V}(\mathbb{R}^{3})$, there exists $v\in H^{1}_{V}(\mathbb{R}^{3})$ such that $v_{n}\rightharpoonup v$ in $H^{1}_{V}(\mathbb{R}^{3})$. From Lemma 2.3, by a standard argument, we know that $v$ is a critical point of $I_{\lambda, \varepsilon}$ and $I'_{\lambda, \varepsilon}(v_{n})\rightarrow I'_{\lambda, \varepsilon}(v) = 0$. Set $u_{n} = v_{n}-v$, then $u_{n}\rightharpoonup0$ in $H^{1}_{V}(\mathbb{R}^{3})$. Making use of Lemmas 4.3-4.4 and the Brézis-Lieb lemma ^[25], it is easy to check

It follows from $I'_{\lambda, \varepsilon}(v) = 0$ and (3.6) that

Thus, we have

where $d: = I_{\lambda, \varepsilon}(v)\geqslant0$.

For any $\varphi\in H^{1}_{V}(\mathbb{R}^{3})$, according to $u_{n}\rightharpoonup0$ in $H^{1}_{V}(\mathbb{R}^{3})$, one has

Hence, we know that $\{u_{n}\}$ is a $(PS)_{c_{\lambda, \varepsilon}-d}$ sequence of $I_{\lambda, \varepsilon}$. In view of $I'_{\lambda, \varepsilon}(u_{n}) = 0$, it is easy to obtain $d\in[0, c_{\lambda, \varepsilon}]$.

Step 2. In this step, we show $c_{\lambda, \varepsilon} < c_{\infty}$. Denote by $u_{\infty}$ be a positive ground state solution of system $\left(P_{\infty}\right)$. Then, we have $I_{\infty}(u_{\infty}) = c_{\infty}$. Moreover, by Lemma 4.1, we know that there exists $t_{\infty} > 0$ such that $t_{\infty}u_{\infty}\in\mathcal{N}_{\lambda, \varepsilon}$. We next claim $t_{\infty} < 1$.

Since $u_{\infty}\in\mathcal{N}_{\infty}$, then we have

Furthermore, it follows from $t_{\infty}u_{\infty}\in\mathcal{N}_{\lambda, \varepsilon}$ that

Comparing (4.25) and (4.26), it is easy to get $t_{\infty} < 1$. Moreover, we have

Taking into account $(V_{1})$, $(K)$ and $(F_{1})$, there holds

So in general, we get

Step 3. According to $d\in[0, c_{\lambda, \varepsilon}]$ and $c_{\lambda, \varepsilon} < c_{\infty}$, we have

By Lemma 4.6, we derive $v$ is a ground state solution of system $\left(P_{\lambda, \varepsilon}\right)$. Similar to the arguments in the proof of Theorem 1.1, one can easily prove $v$ is positive. Denote it by $(u_{\lambda, \varepsilon}, \phi_{u}^{\varepsilon})$. So conclusion $(i)$ follows.

Step 4. Similar to the proof of Lemma 3.4, for a fixed $\varepsilon > 0$, it follows that

which implies $\lim\limits _{\lambda\rightarrow +\infty} \|u_{\lambda, \varepsilon}\| _{H^{1}_{V}(\mathbb{R}^{3})} = 0$. By Lemma 2.3, we get also $\lim\limits_{\lambda\rightarrow +\infty} \|\phi_{u}^{\varepsilon}\|_{\mathcal{D}} = 0$. At last, using the fact of Lemma 2.1, one can deduce $\lim\limits_{\lambda\rightarrow +\infty} \|\phi_{u}^{\varepsilon}\| _{L^{\infty}(\mathbb{R}^{3})} = 0$. So conclusion $(ii)$ follows.

Step 5. For fixed $\lambda = \widetilde{\lambda} > 0$, it is easy to get $\{u_{\widetilde{\lambda}, \varepsilon}\} _{\varepsilon\geqslant0}$ is bounded. Therefore, up to a subsequence, there exists $u_{\widetilde{\lambda}, 0}\in H^{1}_{V}(\mathbb{R}^{3})$ such that

Set $\eta_{\varepsilon} = u_{\widetilde{\lambda}, \varepsilon} - u_{\widetilde{\lambda}, 0}$. Then $\eta_{\varepsilon}\rightharpoonup0$ in $H^{1}_{V}(\mathbb{R}^{3})$. Similar to the proof of Lemma 3.4, we can deduce there exists $\lambda^{*} > 0$ such that

Hence we get $c_{\widetilde{\lambda}, \varepsilon} < c_{\infty}$, for all $\widetilde{\lambda} > \lambda^{*}$, $\varepsilon > 0$. Note that all the conditions of Lemma 4.6 are satisfied, so by Lemma 4.6 we obtain the strong convergence, more precisely it satisfies

In particular, we have $(u_{\widetilde{\lambda}, \varepsilon})^{2} \rightarrow (u_{\widetilde{\lambda}, 0})^{2}$ in $L^{\frac{6}{5}}(\mathbb{R}^{3})$.

Let $\varphi\in H_{V}^{1}(\mathbb{R}^{3})$. Then we have

Pass the limit as $\varepsilon\rightarrow0$ to the above equality. Now we see each term in (4.27), then we have

and as follows by standard arguments we can deduce

and

Making use of Lemma 2.4 and taking into account $u_{\widetilde{\lambda}, \varepsilon} \rightarrow u_{\widetilde{\lambda}, 0}$ in $L^{\frac{12}{5}}(\mathbb{R}^{3})$, $\varphi \in L^{\frac{12}{5}}(\mathbb{R}^{3})$ and the Hölder inequality, we get

It follows from (4.28)–(4.31) that

which shows $(u_{\widetilde{\lambda}, 0}, \phi_{u}^{0})$ solves system $\left(P_{\widetilde{\lambda}, 0}\right)$. Using the same method in proving Theorem 1.1, we can prove $(u_{\widetilde{\lambda}, 0}, \phi_{u}^{0})$ is a positive ground state solution of system $\left(P_{\widetilde{\lambda}, 0}\right)$. So conclusion $(iii)$ follows. The proof is completed.

5.   Proof of Theorem 1.3

In this section, we study the existence of infinitely many solutions to system $\left(P_{\lambda, \varepsilon}\right)$. To complete this proof, we need the following result.

Lemma 5.1. (^[28]) Let $X$ be an infinite dimensional Banach space and let $I\in C^{1}(X, \mathbb{R})$ be even, satisfy $(PS)$ condition, and $I(0) = 0$, If $X = Y\bigoplus Z$, where $Y$ is finite dimensional and $I$ satisfies the following conditions.

$(i)$ There exist constants $\rho, \alpha > 0$ such that $I|_{\{u|\|u\| = \rho\}\cap Z}\geqslant\alpha$;

$(ii)$ For any finite dimensional subspace $\widetilde{X}\subset X$, there is $R = R(\widetilde{X}) > 0$ such that $I(u)\leqslant0$ on $\widetilde{X}\backslash B_{R}$.

Then $I$ possesses an unbounded sequence of critical values.

Now we give the proof of Theorem 1.3.

Proof of Theorem 1.3. To prove Theorem 1.3, it suffices to give the verification of $(i)$ and $(ii)$.

Verification of $(i)$: In view of (3.5) and the Sobolev inequality, we have

For $\rho > 0$ small enough, let $\delta = \frac{1}{2} \rho^{2} - \big(\lambda\varepsilon C_{3} + \lambda C_{4}C_{\varepsilon} + C_{5} \big) \rho^{4}$, then $I_{\lambda, \varepsilon}(u)|_ {\partial B_{\rho} \cap Z} \geqslant \delta > 0$.

Verification of $(ii)$: For any finite dimensional subspace $\widetilde{X} \subset H^{1}_{V}(\mathbb{R}^{3})$, by the equivalence of norms in the finite dimensional space, there exists constant $C > 0$ such that

Putting this together with (3.5) and Lemma 2.3, one can infer

Since $4 < p < 6$, there exists $R > 0$ large enough such that $I_{\lambda, \varepsilon}(u) < 0$ on $\widetilde{X}\backslash B_{R}$. Based on the above facts, all conditions described in Lemma 5.1 are satisfied. Similar to the proof of Theorem 1.1, we can show that the infinitely many solutions are positive. The proof is completed.

6.   Proof of Theorem 1.4

In this section, our goal is to show the nonexistence of ground state solution to system $\left(P_{\lambda, \varepsilon}\right)$.

Lemma 6.1. Suppose that all conditions described in Theorem 1.4 hold. Then for any $\lambda, \varepsilon > 0$, $c_{\lambda, \varepsilon} = c_{\infty}$.

Proof. By the assumptions of $V(x)$ and $K(x)$, one can easily get $I_{\infty}(u) < I_{\lambda, \varepsilon}(u)$, for all $u\in H^{1}_{V}(\mathbb{R}^{3})$. In view of Lemma 4.1, we have for each $u\in\mathcal{N}_{\infty}$, there exists $t_{u} > 0$ such that $t_{u}u\in\mathcal{N}_{\lambda, \varepsilon}$. So, for each $u\in\mathcal{N}_{\infty}$, there holds

Moreover, according to Lemma 4.1,

Hence, it remains to show $c_{\lambda, \varepsilon}\leqslant c_{\infty}$.

By Theorem 1.1, we know that system $\left(P_{\infty}\right)$ has a positive ground state solution $u_{\infty}\in\mathcal{N}_{\infty}$. Denote by $\omega_{n}(x) = u_{\infty}(x-y_{n})$, where $\{y_{n}\}\subset\mathbb{R}^{3}$ and $|y_{n}|\rightarrow +\infty$ as $n\rightarrow +\infty$. Then, there exists a $t_{\omega_{n}} > 0$ such that $t_{\omega_{n}}\omega_{n}\in\mathcal{N}_{\lambda, \varepsilon}$, that is,

It is easy to see that $\{t_{\omega_{n}}\}$ cannot converge to zero and infinity. We assume $t_{\omega_{n}}\rightarrow t_{0}$, as $n\rightarrow +\infty$. Passing the limit as $n\rightarrow +\infty$ in (6.1), we get

By $u_{\infty}\in\mathcal{N}_{\infty}$, we can conclude $\lim\limits_{n\rightarrow +\infty}t_{\omega_{n}} = 1$. Since

Furthermore, by the assumption of $V(x)$, we can infer for any $\epsilon > 0$, there exists $R > 0$ such that

By $|y_{n}|\rightarrow +\infty$ and the Lebesgue dominated convergence theorem, we have

Thus, we get

Similarly, we can arrive at

Hence, using $t_{\omega_{n}}\rightarrow1$ and letting $n\rightarrow +\infty$ in (6.2), we have $c_{\lambda, \varepsilon} \leqslant c_{\infty}$. The proof is completed.

We give the proof of Theorem 1.4.

Proof of Theorem 1.4. By way of contradiction, we assume that there exist $\lambda_{0} > 0$ and $u_{0}\in\mathcal{N}_{\lambda_{0}, \varepsilon}$ such that $I_{\lambda_{0}, \varepsilon}(u_{0}) = c_{\lambda_{0}, \varepsilon}$. In view of Lemma 6.1, one has $c_{\lambda_{0}, \varepsilon} = c_{\infty}$. According to Lemma 3.1, we know that there exists $t_{0} > 0$ such that $t_{0}u_{0}\in\mathcal{N}_{\infty}$. Thus, we have

which yields a contradiction. Moreover, the proof of $\varepsilon$ is similar to $\lambda$, so we omit it here. The proof is completed.

Appendix

Proof of Lemma 3.1. $(i)$ It is standard to show that $I_{\infty}$ satisfies the mountain pass geometry. By the mountain pass theorem, we can obtain a $(PS)_{c_{\infty}}$ sequence of $I_{\infty}$.

$(ii)$ For $t > 0$, let

For $t > 0$ small enough, it follows from (3.5) and Sobolev inequality that

Hence, we get $h(t) > 0$ for $t > 0$ small enough. Moreover, it is easy to see $I_{\infty}(tu) \rightarrow -\infty$ as $t\rightarrow +\infty$. Therefore, $h(t)$ has a maximum at $t = t_{u} > 0$. So that $h'(t_{u}) = 0$ and $t_{u}u\in\mathcal{N}_{\infty}$. Next, we show that $t_{u}$ is unique. By the way of contradiction, we assume that there exist $0 < t_{u} < \tilde{t}_{u}$ such that $\tilde{t}_{u}u$, $t_{u}u\in\mathcal{N}_{\infty}$. Then, we have

which is impossible by $0 < t_{u} < \tilde{t}_{u}$. We now show

By $I'_{\infty}(t_{u}u) = 0$, then $t_{u}$ satisfies

If $\lim\limits_{\lambda\rightarrow +\infty}t_{u} = +\infty$, then in view of $(F_{1})$, it is easy to lead a contradiction. Thus, $\lim\limits_{\lambda\rightarrow +\infty} t_{u} = \eta\geqslant0$. If $\eta > 0$, then combined (A.1) with Lemma 2.3, as $\lambda\rightarrow +\infty$, we can infer

which yields a contradiction. Hence we conclude $\eta = 0$.

$(iii)$ By $(ii)$ one has $\bar{c}_{\infty} = \bar{\bar{c}}_{\infty}$. Choosing $t_{1} > 0$ large enough such that

Define a path $\gamma:[0, 1] \rightarrow H_{V_{\infty}}^{1}(\mathbb{R}^{3})$ by $\gamma(t) = t_{1}tu$, then we have $\gamma\in\Gamma$. Thus, we obtain $c_{\infty} \leqslant \bar{\bar{c}}_{\infty}$. On the other hand, let $k(t): = \langle I_{\infty}'\big(\gamma(t)\big), \gamma(t) \rangle$, where $\gamma\in\Gamma$. Then, $k(t) > 0$ for $t > 0$ small enough. Set $\gamma(1) = e$, one has

from which we obtain

Then there exists $t_{2}\in(0, 1)$ such that $\langle I_{\infty}'\big(\gamma(t_{2})\big), \gamma(t_{2}) \rangle = 0$, which implies $\gamma(t_{2})\in\mathcal{N}_{\infty}$. Therefore, we get $\bar{c}_{\infty} \leqslant c_{\infty}$. The proof is completed.

Acknowledgments

S. Liu is supported by the Fundamental Research Funds for the Central Universities of Central South University 2019zzts210. H. Chen is supported by National Natural Science Foundation of China 12071486.

Conflict of interest

All authors declare no conflicts of interest in this paper.