The existence results for a class of generalized quasilinear Schrödinger equation with nonlocal term

1.   Introduction

The purpose of this paper is to explore the quasilinear Schrödinger equation with nonlocal term:

where $N\geq 3$, $\mu\in(0, N)$, $V$ is nonnegative, $f$ is continuous and $g\in \mathbb{C}^{1}(\mathbb{R}, \mathbb{R}^{+})$. To obtain solutions of equation $(\rm P)$, we make the following assumptions about $g$, $V$ and $f$:

$\rm({g})$  $g\in \mathbb{C}^{1}(\mathbb{R}, \mathbb{R}^{+})$ is even with $g^{'}(t)\geq 0$ for all $t\geq 0$;

$({\rm{V}}_{1})$  $V\in \mathbb{C}(\mathbb{R}^{N}, [0, \infty))$ and $V(x)\leq V_{\infty }: = \lim_{\left | x \right |\rightarrow \infty }V(x),$ for all $x\in \mathbb{R}^{N};$

$\rm({F}_{1})$  $f\in \mathbb{C}(\mathbb{R}, \mathbb{R})$;

$\rm({F}_{2})$  $\lim_{|t|\rightarrow 0}\frac{f(t)}{g(t)|G(t)|^{\frac{N-\mu}{N}}} = 0$, $\lim_{|t|\rightarrow \infty}\frac{f(t)}{g(t)|G(t)|^{\frac{N+2-\mu}{N-2}}} = 0$.

Such a problem is often referred to as being nonlocal due to the appearance of the term $\left(|x|^{-\mu}\ast F(u)\right)f(u)$ which implies that $\rm(P)$ is no longer a pointwise identity. In particular, when $\mu\rightarrow 0$ in $\rm(P)$, then it be reduced to the following generalized quasilinear Schrödinger equation with $f: = Ff$:

Equation (1.1) has received wide attention and solutions of (1.1) are related to the standing wave solutions of the quasilinear Schrödinger equation:

where $z:\; \mathbb{R}\times \mathbb{R}^{N}\rightarrow \mathbb{C}; \; W:\; \mathbb{R}^{N}\rightarrow \mathbb{R}$ is a given potential; $h:\; \mathbb{R}^{N}\times \mathbb{R}\rightarrow \mathbb{R}$ and $\; l:\mathbb{R}\rightarrow \mathbb{R}$ are suitable functions. Different expressions of $l$ represent different physical backgrounds. For example, when $l(s) = s$, ^[1] applied (1.2) to superfluid film equation in plasma physics and fluid mechanics; when $l(s) = s^{\alpha}$ and $\alpha > 1$, we can see ^[2]. Let $z(t, x) = exp(-iEt)u(x)$, where $u(x)$ is a real function and $E\in \mathbb{R}$. Then equation (1.2) can be converted into (see ^[3]):

where $f(x, t) = h(x, |t|)t$ and $V(x) = W(x)-E$.

About Eq (1.1), there are a lot of papers studying the existence of solutions by using variational methods. Especially, In ^[4], Liu et al. firstly attained the positive solution through using variational method and the idea of change of variables. Moreover, in ^[5], Deng et al. obtained the existence of positive solutions with critical exponents by using a change of variable and variational argument. In ^[6], Li et al. proved the existence of a positive ground state solution which possesses a unique local maximum and decays exponentially by variational methods. For more about the results of (1.1), we can see ^[7,8,9] and the references therein.

When $g(t) = 1$ and $\mu\nrightarrow 0$, $(\rm P)$ is reduced to the classical elliptic equation

When $N = 3$, $\mu = 1$, $V\equiv1$ and $f(t) = t$, the equation of (1.4) become

which arises in the description of the quantum theory of a polaron at rest by Pekar in 1954 ^[10] and the modeling of an electron trapped in its own hole in 1976 in the work of Choquard, see ^[11].

To recall the literature in mathematics, Lieb ^[11] proved the existence and uniqueness, up to translations, of the ground state solution for (1.5) and Lions ^[12] showed the existence of a sequence of radially symmetric solutions via variational methods. In the last decades, a great deal of efforts have been devoted to the study of existence, multiplicity and properties of the solutions of (1.4). For example, in ^[13], Gao et al. proved the existence and multiplicity of semiclassical states by critical point theory; in ^[14], Yang established some existence and concentration results of the semiclassical solutions of (1.4) in the whole plane by suppose that the nonlinearity $f$ is critical exponential growth in $\mathbb{R}^{2}$.

It is worth emphasizing that $\rm(P)$ is more general than (1.1) and (1.4). So it is meaningful to study $\rm(P)$. Usually, people study the existence of the solution of problem $\rm(P)$ by studying problem $\rm(\bar{P})$. A typical way to deal with $\rm(\bar{P})$ is using the mountain-pass theorem. For this purpose, one usually assumes that $V\equiv 1$, is periodic, $V(x) = V(|x|)$, or is coercive while $f$ satisfies one of the following conditions:

(i) Super quadratic condition

$\rm(SF)$ $\lim_{|t|\rightarrow \infty} \frac{F(x, t)}{t^{2}} = \infty$ uniformly in $x\in \mathbb{R}^{N}$, where $F(x, t) = \int^{t}_{0}f(x, s) ds$;

(ii) Ambrosetti-Rabinowitz type condition

$\rm(AR)$ there exists $\alpha > 2$ such that $f(x, t)\geq \alpha F(x, t)\geq 0\; \; \; {\rm{for\; all}}\; t\in \mathbb{R}$;

(iii) Monotonicity condition

$\rm(SI)$ $\frac{f(x, t)}{t^{2}}$ is increasing for $t\in \mathbb{R}\setminus\{0\}$.

Under these conditions, it is easy to get a bounded (PS) sequence and verify the Mountain Pass geometry about the corresponding energy functional of $\rm(\bar{P})$.

To the authors' knowledge, in recent paper ^[15], Yang et al. obtained the existence, multiplicity and concentration behavior of positive solutions by variational method and the assumption of $\rm(SI)$; in ^[16], Li et al. proved that the equation admits a solution by using a constrained minimization argument and the assumptions of $\rm(SF)$; in ^[17], Yang et al. got the concentration behavior of ground states via dual approach and the assumptions of $\rm(SF)$ and $\rm(AR)$. For other related results of $\rm(P)$, we refer the readers to ^{[18,19,20,21,22]} and the references therein.

Different from the existing literature, in the present paper, we shall establish the existence of ground state solutions of $\rm(\bar{P})$ and get the existence of solutions of $\rm(P)$ under $\rm({F}_{1})$, $\rm({F}_{2})$ and

$\rm({F}_{3})$ there exists $s_{0} > 0$ such that $F(s_{0})\neq0$, where $F(s) = \int^{s}_{0}f(t)dt$.

We know that $\rm({F}_{3})$ is the Berestycki-Lions type assumption which is satisfied by a very wide class of nonlinearities. These types of nonlinearities were first introduced by Berestycki and Lions in ^[23] to get an existence result of the Schrödinger equation

It is easy to see that $\rm({F}_{3})$ is much weaker than $\rm(SF)$, $\rm(AR)$, $\rm(SI)$ and the others in the related literature. Such kind of conditions are almost necessary for the existence of nontrivial solutions to autonomous problem or to the scalar field equation. Compared with autonomous problem, the nonautonomous problem $\rm(P)$ is much more difficult to study. Motivated by the analysis above, in this paper, our goal is to study the ground state solution of $\rm(\bar{P})$ and then get nontrivial solutions of $\rm(P)$.

In view of $\rm(F_1)$, $\rm(F_2)$ and Hardy-Littlewood Sobolev inequality, for $p\in(2, 2^{\ast})$, any $\varepsilon > 0$ and $u\in H^{1}(\mathbb{R}^{N})$, one have

It is standard to check that, under (1.6), $\rm(V_1)$, $\rm(F_1)$ and $\rm(F_2)$, the Euler-Lagrange functional associated with problem $(\rm P)$ in $H^{1}(\mathbb{R}^{N})$ is given by

Since the term $\int_{\mathbb{R}^{N}} g^{2}(u)|\nabla u|^{2} dx$ may not be well-posed in $u\in H^{1}(\mathbb{R}^{N})$, to overcome this obstacle, Shen and Wang ^[24] made a substitution of variable as $v = G(u) = \int_{0}^{u} g(t) dt$. So for all $v\in H^{1}(\mathbb{R}^{N})$, we have

Therefore, by this change of variable, (1.7) becomes

Furthermore, we can find that if $v\in \mathbb{C}^{2}(\mathbb{R}^{N})$ is a critical point of (1.8), then $u = G^{-1}(v)\in \mathbb{C}^{2}(\mathbb{R}^{N})$ is a corresponding one of $(\rm P)$. Hence, to obtain nontrivial weak solutions of $(\rm P)$, one just need to look for nontrivial weak solutions of the equation

The energy functional of $(\rm \bar{P})$ is

It is evident that $v\in H^{1}(\mathbb{R}^{N})$ is a weak solution of $(\rm \bar{P})$, if it satisfies for all $\varphi \in \mathbb{C}^{\infty}_{0}(\mathbb{R}^{N})$

From $\rm(g)$, $\rm(V1)$, $\rm(V2)$, $\rm(F1)$, $\rm(F2)$ and the Appendix B of ^[25], we have the Pohožaev type functional $\mathcal{P}$ of $(\rm \bar{P})$ in $H^{1}(\mathbb{R}^{N})$:

Define the Pohožaev manifold of $\mathcal{I}$ by

Then every nontrivial solution of $(\rm \bar{P})$ is contained in $\mathcal{M}$. To state our first result, we need to introduce the following monotonicity condition on $V$:

$\rm({V}_{2})$ $V\in \mathbb{C} ^{1}(\mathbb{R}^{N}, \mathbb{R})$ and there exists $\theta\in [0, 1)$ such that $t\rightarrow \frac{NV(tx)+\nabla V(tx)\cdot (tx)}{t^{N-\mu}}+\frac{g^{2}(0)(N-2)^{3}\theta}{4t^{N+2-\mu}|x|^{2}}$ is nonincreasing in $(0, \infty)$ for every $x\in \mathbb{R}^{N}\backslash\{0\}$.

Theorem 1.1. Assume that $\rm(g)$, $\rm(V_{1})$, $\rm(V_{2})$ and $\rm(F_{1})-\rm(F_{3})$ hold. Then problem $(\rm \bar{P})$ has a ground state solution $\bar{v}$ such that

and $\bar{u} = G^{-1}(\bar{v})$ is a nontrivial solution of $(\rm P)$, where

Applying Theorem 1.1 to the following "limiting problem" of $(\rm P)$:

Similarly, using the same variable $v = G(u) = \int_{0}^{u} g(t) dt$. Then $\rm(P^{\infty})$ become the following problem

One has the following Corollary:

Corollary 1.2. Assume that $\rm(g)$, $\rm({F}_{1})-\rm({F}_{3})$ hold. Then problem $(\rm \bar{P}^{\infty})$ has a ground state solution $v^{\infty}$ such that

and $u^{\infty} = G^{-1}(v^{\infty})$ is a nontrivial solution of $(\rm P^{\infty})$, where

and

To prove the above conclusions, we shall divide our arguments into three steps: (i). Choosing a minimizing sequence $\{v_{n}\}$ of $\mathcal{I}$ on $\mathcal{M}$, which satisfies

Then showing that $\{v_{n}\}$ is bounded in $H^{1}(\mathbb{R}^{N})$ and $v_{n}\rightarrow \bar{v}$ in $H^{1}(\mathbb{R}^{N})\setminus\{0\}$ up to translations and extractions of a subsequence. (ii). Showing that $\bar{v}\in \mathcal{M }$ and $\mathcal{I}(\bar{v}) = \inf_{\mathcal{M}}\mathcal{I}$. The difficulties of step (i) are the lake of global compactness and adequate information on $\mathcal{I}'(v) = 0$. To overcome these difficulties, for any $t > 0$ and $v\in H^{1}(\mathbb{R}^{N})$, we establish a crucial inequality which related to $\mathcal{I}(v)$, $\mathcal{I}(v_{t})$ and $\mathcal{P}(v)$:

With the help of the inequality, we complete step (i) by Lions' concentration compactness principle, the least energy sequence approach and some subtle analysis. (iii). Similar to the proof of Lemma 2.14 in ^[26], we showing that $\bar{v}$ is a critical point of $\mathcal{I}$.

Remark 1.3. By the Pohožaev type identity related to $(\rm \bar{P}^{\infty})$, it is easy to see that $\rm(F_{3})$ is necessary and $\rm(F_{1})-\rm(F_{3})$ are almost necessary for the existence of nontrivial solutions of $(\rm P)$.

To admit the other classes of ground state solutions of $(\rm \bar{P})$, we need to introduce the following decay assumption on $\nabla V$:

$({\rm{V}}_{3})$ $V\in \mathbb{C} ^{1}(\mathbb{R}^{N}, \mathbb{R})$, and there exists $\bar{R} > 1$ such that

Remark 1.4. There are indeed many functions which satisfy $\rm(V_{1})$ and $\rm(V_{2})$. For example

(i). $V(x) = \alpha-\beta e^{-|x|^{(N-\mu)}}$, where $\alpha > \beta > 0$;

(ii). $V(x) = \alpha-\frac{\beta}{|x|^{(N-\mu)} +1 }$, where $\alpha > \beta > 0$, $N\alpha\geq (3N-\mu)\beta$.

In particular, when $\alpha > \beta > 0$, $\beta(N-\mu)\geq \min\left\{ \frac{(n-\mu)(2\alpha-\beta)}{4}, \frac{(g(0))^{2}(N-2)^{2} }{2}\right\}$ in (ii), the function of (ii) also satisfies $\rm(V_{1})$ and $\rm(V_{3})$.

Theorem 1.5. Assume that $\rm(g)$, $\rm(V_{1})$, $\rm(V_{3})$ and $\rm(F_{1})-\rm(F_{3})$ hold. Then problem $(\rm \bar{P})$ has a ground state solution $v$ and $u = G^{-1}(v)$ is a nontrivial solution of $(\rm P)$.

To prove Theorem 1.5, we will use the idea from Jeanjean and Tanaka ^[27], that is an approximation procedure to obtain a bounded (PS)-sequence of $\mathcal{I}$. Firstly, for $\lambda\in [\frac{1}{2}, 1]$ we consider a family of functionals $\mathcal{I}_{\lambda}: H^{1}(\mathbb{R}^{N})\rightarrow \mathbb{R}$ defined by

These functionals have a Mountain Pass geometry. In what follows, we use $c_{\lambda}$ to express the corresponding Mountain Pass levels of $\mathcal{I}_{ \lambda}$. Let

Unfortunately, $B(v)$ is not sign definite under $\rm(F_{1})-\rm(F_{3})$, which prevents us from employing Jeanjean's monotonicity trick used in ^[28]. Thanks to the idea of ^[27], $\mathcal{I}_{\lambda}$ still has a bounded (PS)-sequence $\{v_{n}\}\subset H^{1}(\mathbb{R}^{N})$ at level $c_{\lambda}$ for almost every $\lambda\in [\frac{1}{2}, 1]$. Secondly, we use the global compactness lemma to show that the bounded sequence $\{v_{n}\}$ converges weakly to a nontrivial point of $\mathcal{I}_{\lambda}$. Finally, we choose two sequences $\{\lambda_{n}\}\subset (\lambda^{\ast}, 1]$ and $\{v_{\lambda_n}\}\subset H^{1}(\mathbb{R}^{N})\setminus\{0\}$ such that $\lambda_{n}\rightarrow 1$ and $\mathcal{I}'_{\lambda_{n}}(v_{\lambda_{n}}) = 0$, where $\lambda^{\ast}$ is defined in Lemma 3.5. By Lemmas 3.5–3.9, we get a nontrivial critical point $\bar{v}$ of $\mathcal{I}$.

Throughout the paper we make use of the following notations:

$\clubsuit\; H^{1}(\mathbb{R}^{N})$ denotes the usual Sobolev space equipped with the inner product and norm

$\clubsuit\; L^{s}(\mathbb{R}^{N})\; (1\leq s < \infty)$ denotes the Lebesgue space with the norm $\left \| u \right \|_{s} = \left (\int_{\mathbb{R}^{N}}\left | u \right |^{s} dx\right)^{\frac{1}{s}}$;

$\clubsuit$ for any $u\in H^{1}(\mathbb{R}^{N}), \; \; u_{t}(x): = u(t^{-1}x)$ for $t > 0$;

$\clubsuit$ for any $x\in \mathbb{R}^{N}$ and $r > 0, \; B_{r}(x): = \{y\in \mathbb{R}^{N}:|y-x| < r\}$;

$\clubsuit\; \mathcal{C}, \; \mathcal{C}_{1}, \mathcal{C}_{2}\ldots$ denote positive constants which are possibly different in different places.

$\clubsuit$ $S$ is the best constant for the embedding of $D^{1, 2}(\mathbb{R}^{N})\hookrightarrow L^{2^{\ast}}(\mathbb{R}^{N})$, where $D^{1, 2}(\mathbb{R}^{N}) = \left\{u\in L^{2}(\mathbb{R}^{N}); \nabla u \in L^{2}(\mathbb{R}^{N})\right\}$ and $2^{\ast} = \frac{2N}{N-2}$.

The paper is organized as follows: In $\S$ 2, we study the existence of ground state solutions of $(\rm \bar{P})$ by using the Pohožaev manifold and give the proof of Theorem 1.1. In $\S$ 3, based on an approximation procedure developed by Jeanjean and Tanaka ^[27], we show the existence of ground state solutions of $(\rm \bar{P})$ and complete the proof of Theorem 1.5.

2.   Preliminaries

In this section, we present some fundamental lemmas and corollaries, study the existence of ground state solutions of $(\rm \bar{P})$ by using the Pohožaev manifold, and give the proof of Theorem 1.1.

Lemma 2.1 (see ^[5]) Assume that $\rm(g)$ holds. Then the functions $G(.)$ and $G^{-1}(.)$ have the following properties:

(1) the functions $G(.)$ and $G^{-1}(.)$ are odd and strictly increasing;

(2) for all $t\in \mathbb{R}$, $|G^{-1}(t)|\leq\frac{1}{g(0)}|t|$ and $\frac{G^{-1}(t)t}{g(G^{-1}(t))}\leq |G^{-1}(t)|^{2}$;

(3) $\frac{G^{-1}(t)}{t}$ is increasing on $(-\infty, 0)$ but decreasing on $(0, +\infty)$ and

(4) $\lim_{|t|\rightarrow 0}\frac{f(G^{-1}(t))}{g(G^{-1}(t))t^{\frac{N-\mu}{N}}} = 0$ and $\lim_{|t|\rightarrow 0}\frac{F(G^{-1}(t))}{t^{\frac{2N-\mu}{N} }} = 0$;

(5) $\lim_{|t|\rightarrow \infty}\frac{|f(G^{-1}(t))|}{g(G^{-1}(t))|t|^{\frac{N+2-\mu}{N-2}}} = 0$ and $\lim_{|t|\rightarrow \infty}\frac{F(G^{-1}(t))}{|t|^{\frac{2N-\mu}{N-2}}} = 0$.

Lemma 2.2 Assume that $\rm(g)$, $\rm(V_1)$, $\rm(V_2)$, $\rm(F_1)$ and $\rm(F_2)$ hold. Then, for any $t > 0$ and $v\in H^{1}(\mathbb{R}^{N})$, we have

where $h(t) = (2N-\mu)(1-t^{N-2})-(N-2)(1-t^{2N-\mu})$.

Proof. Note that

By simple calculation, we have $h(t) > 0$ with $t\in [0, 1)\cup (1, \infty)$. Through $\rm(V_{2})$ and a simple calculation, we can verify that

According to Hardy inequality, we have

Using (1.9), (1.11), (2.2)–(2.4) and (2) of Lemma 2.1, it is easy to check that, for any $t > 0$, we have

This shows that (2.1) holds.

Corollary 2.3 Assume that $\rm(g)$, $\rm(F_1)$ and $\rm(F_2)$ hold. Then, for any $t > 0$ and $v\in H^{1}(\mathbb{R}^{N})$, we have

where $k(t) = (2N-\mu)(1-t^{N-2})-N(1-t^{2N-\mu}) > 0, \; \; \forall\; t\in [0, 1)\cup (1, \infty)$.

Corollary 2.4 Assume that $\rm(g)$, $\rm(V_1)$, $\rm(V_2)$, $\rm(F_1)$ and $\rm(F_2)$ hold. Then

Lemma 2.5 Assume that $\rm(g)$, $\rm(V_1)$, $\rm(V_2)$ hold. Then there exist tow constants $\gamma_{1}, \; \gamma_{2} > 0$ such that for all $v\in H^{1}(\mathbb{R}^{N})$

Proof. Let $t\rightarrow0$, $t\rightarrow \infty$ in (2.3) respectively, we have

and

From (2.7), (2.8) and $\rm(V_{1})$, there exists a constant $M_{0}$ such that

By (2.3), for $\; \forall \; t > 0, \; x\in \mathbb{R}^{N}\backslash\{0\}$, one has

According to $\rm(V_{1})$, there exists $t_{0} > 1$ and $R_{0} > 0$ such that $V(x)\geq \frac{V_{\infty}}{2}$ for all $|x|\geq t_{0}R_{0} > R_{0}$ and

From (2.10) and (2.11), we have

Making use of the Hölder inequality and Sobolev inequality, we get

where $\omega_{N}$ denotes the volume of the unit ball of $\mathbb{R}^{N}$. Then, it follows from $\rm(V_{1})$, (2.4), (2.12), (2.13), (2) of Lemma 2.1 and Sobolev inequality that

So we completes the proof of the lemma.

Lemma 2.6 Assume that $\rm(V_1)$, $\rm(V_{2})$ hold. Then

Proof. Arguing by contradiction, we assume that there exist $\{x_{n}\}\subset \mathbb{R}^{N}$ and $\varepsilon > 0$ such that

Now, we distinguish two case.

Case i) ${\rm{as}}\; |x_{n}|\rightarrow \infty, {\rm{we\; have\; }} \nabla V(x_{n})\cdot x_{n}\geq\varepsilon, \; \forall\; n\in \mathbb{N}$. In this case, by (2.3), one has

Since

there exists $0 < t_{1} < 1$ such that

Then it follows from $\rm (V_{1})$, (2.20) and (2.16) that

which is a contradiction.

Case ii) ${\rm{as}}\; |x_{n}|\rightarrow \infty, {\rm{we\; have\; }} \nabla V(x_{n})\cdot x_{n}\leq-\varepsilon, \; \forall\; n\in \mathbb{N}$. In this case, by (2.3), one has

From (2.15), there exists $t_{2} > 1$ such that

Then it follows from $\rm (V_{1})$, (2.18) and (2.19) that

which is a contradiction.

Lemma 2.7 Assume that $\rm(V_1)$, $\rm(V_{2})$ and $\rm(F_1)-\rm(F_3)$ hold. Then $\Lambda \neq \emptyset$ and

Proof. It follows from the proof of Theorem 2 in ^[23], the properties of $g$ and condition $\rm (F_{3})$ that $\Lambda \neq \emptyset$. Next, we have two cases to distinguish:

(1) $v\in H^{1}(\mathbb{R}^{N})\setminus\{0\}$ and $\mathcal{P}^{\infty}(v)\leq 0$, then (1.17) implies $v\in \Lambda$.

(2) $v\in H^{1}(\mathbb{R}^{N})\setminus\{0\}$ and $\mathcal{P}(v) \leq 0$. By (1.11), (2.3) and (2.8), one has

which implies $v\in \Lambda$.

From the above two cases, we complete the proof of this lemma.

Lemma 2.8 Assume that $\rm(g)$, $\rm(V_1)$, $\rm(V_2)$ and $\rm(F_1)-\rm(F_3)$ hold. Then for any $v\in \Lambda$, there exists a unique $t_{v} > 0$ such that $v_{t_{v}}\in \mathcal{M}$.

Proof. Let $v\in \Lambda \setminus \{0\}$ be fixed. Define a function $\aleph (t): = \mathcal{I} (v_{t})$ on $(0, \infty)$. Clearly, by (1.9) and (2.2) we have

Using (2.4), (2.8) and (2) of Lemma 2.1, we have $\aleph (t) > 0$ for $t$ small and $\aleph (t) < 0$ for $t$ is large enough. Therefore, $\max_{t\in [0.\infty)}\aleph (t)$ is achieved at some $t_{v} > 0$ such that $\aleph'(t_{v}) = 0$ and $v_{_{t_{v}}}\in \mathcal{M}$. Next, we claim that $t_{v}$ is unique. In fact, if $t_{1}\; ,t_{2} > 0$ such that $v_{_{t_{1}}}, \; v_{_{t_{2}}} \in \mathcal{M}$, then $\mathcal{P}(v_{t_{1}}) = \mathcal{P}(v_{t_{2}}) = 0$. From (2.1), we have

which implies $t_{1} = t_{2}$. So, we complete the proof.

Combining Corollary 2.4 with Lemma 2.8, we have the following corollary:

Corollary 2.9 Assume that $\rm(g)$ and $\rm(F_1)-\rm(F_3)$ hold. Then for any $v\in \Lambda$, there exists a unique $t_{v} > 0$ such that $v_{t_{v}}\in \mathcal{M}^{\infty}$.

Lemma 2.10 Assume that $\rm(g)$, $\rm(V_1)$, $\rm(V_2)$ and $\rm(F_1)-\rm(F_3)$ hold. Then

From Corollaries 2.3 and 2.9, we have the following corollary:

Corollary 2.11 Assume that $\rm(g)$ and $\rm(F_1)-\rm(F_3)$ hold. Then

The following version of Brezis-Lieb lemma for the nonlocal term is useful for our analysis. We refer to ^[29] for a proof.

Lemma 2.12 Assume that $\rm(g)$, $\rm(F_1)$ and $\rm(F_2)$ hold. If $u_{n}\rightharpoonup u$ in $H^{1}(\mathbb{R}^{N})$, then

From the above Lemma 2.12 and Lemma 1.32 of ^[25], we have the following lemma

Lemma 2.13 Assume that $\rm(g)$, $\rm(V_1)$, $\rm(V_2)$ and $\rm (F_{1})-\rm (F_{3})$ hold. If $v_{n}\rightharpoonup v$ in $H^{1}(\mathbb{R}^{N})$, then

Lemma 2.14 Assume that $\rm(g)$, $\rm(V_1)$, $\rm(V_2)$ and $\rm(F_1)-\rm(F_3)$ hold. Then there exists some $\delta > 0$ such that

$\rm(i)$ $\inf_{v\in\mathcal{ M}}\int_{\mathbb{R}^{N}}[|\nabla v|^{2}+| G^{-1}(v)|^{2}] dx\geq\delta$ for any $v\in \mathcal{M}$; $\rm(ii)$ $m = \inf_{v\in \mathcal{M} }\mathcal{I} (v) > 0$.

Proof. (i) Since $\mathcal{P}(v) = 0$ for any $v\in \mathcal{M}$, it follows from $\rm(F_1)$, $\rm(F_2)$, (1.6), (2.6), Sobolev embedding inequality and Lemma 2.1 that

which implies

(ii). Let $\{v_{n}\}\subset \mathcal{M}$ be such that $\mathcal{I}(v_{n})\rightarrow m$ as $n\rightarrow \infty$. There are two possible case:

Case i) $\inf_{n\in N}\|\nabla v_{n}\|^{2}_{2}\geq\varrho > 0$. Let $t\rightarrow 0$ in (2.1), we find

Case ii) $\inf_{n\in N}\|\nabla v_{n}\|^{2}_{2} = 0$. From (2.21), passsing to a subsequence, we have

Together with (1.6), (4), (5) of Lemma 2.1 and the Sobolev inequality, for $v\in H^{1}(\mathbb{R}^{N})$

From $\rm(V_{1})$, there exists $R > 0$ such that $V(x)\geq \frac{V_{\infty}}{2}$ for $|x|\geq R$, and we have

Making use of the Sobolev inequality and Hölder inequality, for all $t > 0\; {\rm{and}}\; v\in H^{1}(\mathbb{R}^{N})$, we have

Let

Then (2.22) shows $\{t_{n}\}$ is bounded. Finally combine $\rm(V_{1})$, (2.1), (2.23)–(2.26) and Corollary 2.4, to discover

From the above analysis we know that $m = \inf_{u\in \mathcal{M} }\mathcal{I} (u) > 0$.

Note that since $V(x)\equiv V_{\infty}$ satisfies $\rm(V_{1})$ and $\rm(V_{2})$, all above conclusions on $\mathcal{I}$ are still true for $\mathcal{I}^{\infty}$.

Lemma 2.15 Assume that $\rm(g)$, $\rm(V_1)$, $\rm(V_2)$ and $\rm(F_1)-\rm(F_3)$ hold. Then

Proof. In view of Corollary 2.9, we have $\mathcal{M}^{\infty}\neq \emptyset$. Arguing indirectly, we assume that $m^{\infty} < m$. Let $\varepsilon = m-m^{\infty}$. Then there exists $v^{\infty}_{\varepsilon}$ such that

In view of Lemma 2.8, there exists $t_{\varepsilon} > 0$ such that $(v^{\infty}_{\varepsilon})_{t_\varepsilon}\in \mathcal{M}$. Thus, it follows from $\rm(V_{1})$, (1.9), (1.12), (2.1) and (2.27) that

This contradiction shows that $m^{\infty}\geq m$.

Lemma 2.16 Assume that $\rm(g)$, $\rm(V_{1})$, $\rm(V_{2})$ and $\rm(F_1)-\rm(F_3)$ hold. Then $m$ is achieved.

Proof. From Lemmas 2.8, and 2.14, we know that $\mathcal{M}\neq \emptyset$ and $m > 0$. Let $\{v_{n}\}\subset\mathcal{M}$ be a sequence verifying $\mathcal{I} (v_{n})\rightarrow m$. From $\mathcal{P} (v_{n}) = 0$, (1.9) and (1.11), we have

This shows that $\{\|\nabla v_{n}\|_{2}\}$ is bounded. Next, we need to prove $\{v_{n}\}$ is also bounded in $L^{2}(\mathbb{R}^{N})$. Firstly, we claim that $\{G^{-1}(v_{n})\}$ is bounded in $L^{2}(\mathbb{R}^{N})$. Arguing by contradiction, suppose that $\|G^{-1}(v_{n})\|_{2}\rightarrow \infty$. Combine (1.6), Lemma 2.1 and the Sobolev inequality, we get

for $\forall\; v\in H^{1}(\mathbb{R}^{N})$, where $\delta_{0}$ is given by (2.26). Let $\hat{t}_{n} = \left(\frac{16 m}{\delta_{0}} \right)^{1/(N-\mu)}\|G^{-1}(v_{n})\|_{2}^{-2/N}$, then $\hat{t}_{n}\rightarrow 0$ as $n\rightarrow \infty$. Thus, from (2.2), (2.29), $\rm(V_{1})$ and Corollary 2.4, we have

This contradiction shows that $\{\|G^{-1}(v_{n}))\|_{2}\}$ is bounded. Secondly, we show that $\{\|v_{n}\|_{2}\}$ is also bounded. Note that (3) of Lemma 2.1 implies that

So, we have

Combine with (2.28) and (2.32), we know that $\{v_{n}\}$ is bounded in $H^{1}(\mathbb{R}^{N})$. Passing to a subsequence, there exists $\bar{v}\in H^{1}(\mathbb{R}^{N})\backslash \{0\}$ such that $v_{n}\rightharpoonup \bar{v}, \; {\rm{in}}\; H^{1}(\mathbb{R}^{N});$ $v_{n}\rightarrow \bar{v}, \; {\rm{on}}\; L^{s}_{loc}(\mathbb{R}^{N}), \; \forall s\in(2, 2^{\ast})$ and $v_{n}\rightarrow \bar{v}, {\rm{a.e. on }}\mathbb{R}^{N}.$ There are tow cases: (i) $\bar{v} = 0$ and (ii) $\bar{v}\neq0$.

Case i). $\bar{v} = 0$, i.e., $v_{n}\rightharpoonup 0$ in $H^{1}(\mathbb{R}^{N})$. By $\rm(V_{1})$ and Lemma 2.6, it is easy to show that

It follows from (1.9) and (2.33) that

Since $\mathcal{P}(u_{n}) = 0$, from (1.6), (2.6) and Sobolev embedding inequality, one has

Together with (2.35) and Lions' concentration compactness principle ^[25], one can easily verify that there exist $\delta_{1} > 0$ and $\{y_{n}\}\subset \mathbb{R}^{N}$ such that $\int_{B(y_{n}, 1)}|v_{n}|^{2} dx > \frac{\delta_{1}}{2}$. Let $\hat{v}_{n}(x) = v_{n}(x+y_{n})$, we have

and there exists $\hat{v}\in H^{1}(\mathbb{R}^{N})\backslash \{0\}$ such that $\hat{v}_{n}\rightharpoonup \hat{v}, \; {\rm{in}}\; H^{1}(\mathbb{R}^{N});$ $\hat{v}_{n}\rightarrow \hat{v}, \; {\rm{on}}\; L^{s}_{loc}(\mathbb{R}^{N}), \; \forall s\in(2, 2^{\ast})$ and $\hat{v}_{n}\rightarrow \hat{v}, \; {\rm{a.e. on }}\mathbb{R}^{N}.$ By (2.34) and (2.36), one has

Let $\omega_{n} = \hat{v}_{n}-\hat{v}$. From Lemma 2.13, we deserve

For any $v\in H^{1}(\mathbb{R}^{N})$, set

From (2.37)–(2.39), it is easy to check that

If there is some subsequence $\{\omega_{n_{i}}\}$ of $\{\omega_{n}\}$ such that $\omega_{n_{i}} = 0$, then for this subsequence, there holds

Next, we show that $\omega_{n}\neq0$. We assert that $\mathcal{P}_{\infty}(\hat{v})\leq0$. On the contrary, if $\mathcal{P}^{\infty}(\hat{v}) > 0$, then (2.40) indicates that for sufficiently large $n$, $\mathcal{P}^{\infty}(\omega_{n}) < 0$. Because of Lemma 2.7 and Corollary 2.9, there exists $t_{n} > 0$ such that $(\omega_{n})_{t_{n}}\in \mathcal{M}^{\infty}$. (1.12), (1.13), (2.37), (2.39) and(2.40) tell us that

which implies $\mathcal{P}^{\infty}(\hat{v})\leq0$ due to $m\leq m^{\infty}$. Hence, as $\hat{v}\neq0$, in view of Corollary 2.9, there exists $t_{\infty} > 0$ such that $\hat{v}_{t_{\infty}}\in \mathcal{M}^{\infty}$. According to (1.12), (1.13), (2.37), (2.39), (2.40), Corollary 2.3 and Fatou's lemma, we find

which implies (2.41). In view of Lemma 2.8, there exists $\hat{t} > 0$ such that $\hat{v}_{\hat{t}}\in \mathcal{M}$. By (1.9), (1.12), (2.41), $(V_{1})$, Corollaries 2.4, we obtain

This shows that $m$ is achieved at $\hat{v}_{\hat{t}}\in \mathcal{M}$.

Case ii). $\bar{v}\neq0$. Let $u_{n} = v_{n}-\bar{v}$. Then Lemma 2.13 yields

Through (1.9), (1.11), (2.4) and (2.8), we obtain

Since

it follows from (2.42)–(2.44) that

If there is some subsequence $\{u_{n_{i}}\}$ of $\{u_{n}\}$ such that $u_{n_{i}} = 0$, then for this subsequence, there holds

Next, we show that $u_{n}\neq0$. We assert that $\mathcal{P}(\bar{v})\leq0$. On the contrary, if $\mathcal{P} (\bar{v}) > 0$, then (2.45) indicates that for sufficiently large $n$, $\mathcal{P} (u_{n}) < 0$. Because of Lemma 2.8, there exists $t_{n } > 0$ such that $(u_{n })_{t_{n }}\in \mathcal{M}$. From (1.9), (1.11), (2.43) and (2.45), we have

which implies $\mathcal{P}(\bar{v})\leq0$ due to $\Psi(\bar{v}) > 0$. Hence, as $\bar{v}\neq0$, in view of Lemma 2.8, there exists a $\bar{t} > 0$ such that $\bar{v}_{\bar{t}}\in \mathcal{M}$. From (1.9), (1.11), (2.11), (2.43), (2.45) and Fatou's lemma, one has

which implies (2.46). This implies that the desired conclusion holds.

Lemma 2.17 Assume that $\rm(g)$, $\rm(V_1)$, $\rm(V_2)$ and $\rm(F_{1})-\rm(F_{3})$ hold. If $\bar{v}\in \mathcal{M}$ and $\mathcal{I}(\bar{v}) = m$, then $\bar{v}$ is a critical point of $\mathcal{I}$.

Proof. From $\rm(g)$, $\rm(V_1)$, $\rm(V_2)$ and $\rm(F_{1})-\rm(F_{3})$ (1.11), there exist $T_{1}\in(0, 1)$ and $T_{2}\in(1, \infty)$ such that

Similar to the proof Lemma 2.13 in ^[26], we can prove this lemma only by using

for any $t > 0$ and

respectively, instead of (2.40) and $\varepsilon$ in ^[26].

Proof of Theorem 1.1. In view of Lemmas 2.10, 2.16 and 2.17, there exists $\bar{v}\in \mathcal{M}$ such that

This shows that $\bar{v}$ is a ground state solution of $(\rm \bar{P})$ such that $\mathcal{I} (\bar{v}) = \inf_{v\in \mathcal{M} }\mathcal{I} (v)$ and $\bar{u} = G^{-1}(\bar{v})$ is a nontrivial solution of $(\rm P)$.

3.   Proof of Theorem 1.5

In this section, we assume that $V(x) \not\equiv V_{\infty}$ and give the proof of Theorems 1.5. In order to find a bounded (PS)-sequence of $\mathcal{I}$, we use the idea employed by Jeanjean and Tanaka ^[27] which is an approximation procedure.

Proposition 3.1 ^[27] Let $X$ be a Banach space and $\Omega\subset \mathbb{R}^{+}$ be an interval, and

be a family of $\mathbb{C}^{1}$-functional on $X$ such that

i) either $A(v)\rightarrow \infty$ or $B(v)\rightarrow \infty$, as $\|v\|\rightarrow \infty$;

ii) $B$ maps every bounded set of $X$ into a set of $\mathbb{R}$ bounded below;

iii) there are two points $v_{1}, \; v_{2}$ in $X$ such that

where $\Gamma = \left\{ \gamma \in \mathbb{C} \left([ 0, 1], \; X\right):\gamma(0) = v_{1}, \gamma(1) = v_{2} \right\}$. Then, for almost every $\lambda\in \Omega$, there is a bounded (PS)-sequence for $c_{\lambda}$, that is, there exists a sequence such that

(i) $\{v_{n}(\lambda)\}$ is bounded in $X$; (ii) $\mathcal{J}_{\lambda} (v_{n}(\lambda))\rightarrow c_{\lambda}$;

(iii) $\mathcal{J}'_{\lambda} (v_{n}(\lambda))\rightarrow 0$ in $X^{\ast}$, where $X^{\ast}$ is the dual of $X$.

Lemma 3.2 (see ^[25], Appendix B). Assume that $\rm(g)$, $\rm(V_1)$, $\rm(V_3)$, $\rm(F_{1})$ and $\rm(F_{2})$ hold. Let $v$ be a critical point of $\mathcal{I}_{ \lambda}$ in $H^{1}(\mathbb{R}^{N})$, then for $\lambda\in [\frac{1}{2}, 1]$, we have the following Pohožaev type identity:

Let

We also let

Similarly, the Pohožaev type identity of $\mathcal{I}^{\infty}_{ \lambda}$ is

and

By Corollary 2.3, we have the following lemma:

Lemma 3.3 Assume that $\rm(g)$, $\rm(F_1)$ and $\rm(F_3)$ hold. Then, for any $t > 0$ and $v\in H^{1}(\mathbb{R}^{N})$, we have

where $k(t) = (2N-\mu)(1-t^{N-2})-N(1-t^{2N-\mu}) > 0, \; \; \forall\; t\in [0, 1)\cup(1, \infty)$.

In view of Corollary 1.2, $\mathcal{I}^{\infty}_{1} = \mathcal{I}^{\infty}$ has a minimizer $v^{\infty}_{1}\neq0$ on $\mathcal{M}^{\infty}_{1} = \mathcal{M}^{\infty}$, i.e.,

where $m^{\infty}_{1}$ is defined by (3.5). From $\rm(V_{1})$ and $V(x)\leq V_{\infty}$ but $V(x) \not\equiv V_{\infty}$, there exist $\bar{x}\in \mathbb{R}^{N}$ and $\bar{r} > 0$ such that

Lemma 3.4 Assume that $\rm(g)$, $\rm(V_1)$ and $\rm(F_1)-\rm(F_3)$ hold. Then

(i) there exists $T > 0$ such that $\mathcal{I}_{\lambda}((v^{\infty}_{1})_{T }) < 0$, for all $\lambda\in [\frac{1}{2}, 1]$;

(ii) there exists a positive constant $\tilde {k}$, independent of $\lambda$, such that for all $\lambda\in [\frac{1}{2}, 1]$, we have

where $\Gamma = \left\{ \gamma \in \mathbb{ C} ([ 0, 1], H^{1}(\mathbb{R}^{N})):\gamma(0) = 0, \gamma(1) = (v^{\infty}_{1})_{T } \right\};$

(iii) $c_{\lambda}$ is bounded for $\lambda\in [\frac{1}{2}, 1]$;

(iv) $m^{\infty}_{\lambda}$ is non-increasing on $\lambda\in [\frac{1}{2}, 1]$;

(v) $\limsup_{\lambda \rightarrow \lambda_{0}} c_{\lambda}\leq c_{ \lambda_{0}}$, for all $\lambda_{0}\in [\frac{1}{2}, 1]$.

Proof. Since $m^{\infty}_{\lambda} = \mathcal{I}^{\infty}_{ \lambda}(v^{\infty}_{\lambda})$ and $\int_{\mathbb{R}^{N}}\left(|x|^{-\mu}\ast F(G^{-1}(v^{\infty}_{\lambda}))\right)F(G^{-1}(v^{\infty}_{\lambda})) dx > 0$, (i)-(iv) of Lemma 3.4 are standard and (v) can be proved similar to Lemma 2.3 of ^[28], so we omit it.

Lemma 3.5 Assume that $\rm(g)$, $\rm(V_1)$ and $\rm(F_1)-\rm(F_3)$ hold. Then there exists $\lambda^{\ast}\in [\frac{1}{2}, 1)$ such that $c_{ \lambda} < m^{\infty}_{\lambda}$ for $\lambda\in (\lambda^{\ast}, 1]$.

Proof. It is easy to see that $\mathcal{I}_{ \lambda}(v^{\infty}_{t})$ is continuous on $t\in [0, +\infty)$. Hence for any $\lambda\in [\frac{1}{2}, 1]$, we can choose $t_{\lambda}\in (0, T)$ such that $\mathcal{I}_{ \lambda}(v^{\infty}_{t_{ \lambda}}) = \max_{t\in [0, T]} \mathcal{I}_{\lambda}(v^{\infty}_{t})$. Setting $\gamma_{0}(t) = (v^{\infty}_{1})_{tT}$ for $t > 0$ and $\gamma_{0}(t) = 0$ for $t = 0$. Then $\gamma_{0}\in \Gamma$ defined by (ii) of Lemma 3.4. Moreover, one has

Let

Then it follows from (3.8) and (3.10) that

Since $\mathcal{P}_{\lambda}^{\infty}(v_{1}^{\infty}) = 0$ and $v_{1}^{\infty}\neq0$, then $\int_{\mathbb{R}^{N}}\left(|x|^{-\mu}\ast F(G^{-1}(v^{\infty}_{1}))\right)F(G^{-1}(v^{\infty}_{1})) dx > 0$. Let

where $\beta(t) = \frac{h(t)}{2(2N-\mu)}$. Then (3.8), (3.11) and (3.12) imply that $\lambda^{\ast}\in [\frac{1}{2}, 1)$. We have two cases to distinguish:

Case (i). $t_{\lambda}\in [1-\zeta_{0}, 1+\zeta_{0}]$. From (1.14), (3.3)–(3.8), (3.11), (3.12) and (iv) of Lemma 3.4, we have

Case (ii). $t_{\lambda}\in (0, 1-\zeta_{0})\cup (1+\zeta_{0}, T)$. Since $V_{\infty}\geq V(x)$ for all $x\in \mathbb{R}^{N}$, it follows from (1.14), (3.3)–(3.8), (3.11), (3.12) and (iv) of Lemma 3.4 that

In both cases, we obtain that $c_{\lambda} < m^{\infty}_{\lambda}$ for $\lambda\in (\lambda^{\ast}, 1]$.

In order to prove that the functional $\mathcal{I}_{\lambda}(v)$ satisfies $(PS)_{c_{\lambda}}$ condition for a.e. $\lambda \in[\frac{1}{2}, 1]$, we need the following new version of global compactness lemma, which is suitable for quasilinear Schrödinger equation with nonlocal term.

Lemma 3.6 Assume that $\rm(g)$, $\rm(V_1)$, $\rm(V_3)$ and $\rm(F_1)-\rm(F_3)$ hold. For any $c_{\lambda} > 0$, $\lambda\in [\frac{1}{2}, 1]$, if $\{v_{n}(\lambda)\}\subset H^{1}(\mathbb{R}^{N})$ is a bounded $(PS)_{c_{ \lambda}}$ sequence of $\mathcal{I}_{ \lambda}$, then there exist a subsequence of $\{v_{n}(\lambda)\}$, still denoted by $\{v_{n}(\lambda)\}$, $v_{ \lambda} \in H^{1}(\mathbb{R}^{N})$ such that

(i) $v_{n}(\lambda)\rightarrow v_{ \lambda}$ in $H^{1}(\mathbb{R}^{N})$;

(ii) there exist $l\in \mathbb{N}\cup \{0\}$, $\{y^{k}_{n}\}\subset \mathbb{R}^{N}$ with $|y^{k}_{n}|\rightarrow \infty$ and nonzero $\omega_{\lambda}^{k}$ for each $1\leq k\leq l$ satisfy $(\mathcal{I}_{\lambda}^{\infty})'(\omega_{\lambda}^{k}) = 0$;

(iii) $\| v_{n}(\lambda)-v_{ \lambda} -\sum^{l}_{k = 1} \omega^{k}_{ \lambda} (\cdot-y^{k}_{n}) \|\rightarrow 0;$

(iv) $\mathcal{I}_{ \lambda}(v_{n})\rightarrow \mathcal{I}_{ \lambda}(v)+\sum^{l}_{k = 1} \mathcal{I}^{\infty}_{ \lambda}(\omega^{k}).$

Proof. With the aid of Brézis-Lieb lemma in ^[30], P. L. Lions vanishing lemma in ^[31], and using the idea of Lemma 4.2 in ^[32], we can verify this lemma.

Lemma 3.7 Assume that $\rm(V_1)$ and $\rm(V_3)$ hold. Then for any $v\in H^{1}(\mathbb{R}^{N})$, there exists $\gamma_{3} > 0$ such that

Proof. From (2.4) and $\rm(V_1)$ and $\rm(V_3)$, we have

where $\gamma_{3}$ due to $\mu$ and $\rm(V_1)$.

Lemma 3.8 Assume that $\rm(V_1)$, $\rm(V_3)$ and $\rm(F_1)-\rm(F_3)$ hold. Then for every $\lambda\in (\lambda^{\ast}, 1]$, there exists $v_{ \lambda}\in H^{1}(\mathbb{R}^{N})\setminus \{0\}$ such that

Proof. According to Proposition 3.1, Lemmas 3.4 and 3.6. For almost every $\lambda\in (\lambda^{\ast}, 1]$, there exist a subsequence of $\{v^{1}_{n} \}$ (for simplicity, we still denoted by $\{v_{n} \}$) of $\{v_{n}(\lambda)\} \subset H^{1}(\mathbb{R}^{N})$ and $v_{ \lambda}\in H^{1}(\mathbb{R}^{N})$ satisfying

and $v_{n} \rightharpoonup v_{ \lambda}$ in $H^{1}(\mathbb{R}^{N})$, $\mathcal{I}'_{ \lambda}(v_{ \lambda}) = 0$, an integer $l\in \mathbb{N}\cup \{0\}$ and $\omega_{ \lambda}^{1}, ..., \omega_{ \lambda}^{l}\in H^{1}(\mathbb{R}^{N})\setminus \{0\}$ such that

Since $(\mathcal{I}_{ \lambda})'(v_{ \lambda}) = 0$, we have the Pohožaev identity of the functional $\mathcal{I}_{ \lambda}$

Since $\|v_{n}\|\nrightarrow 0$, we deduce from (3.16) that if $v_{\lambda} = 0$ then $l\geq 1$ and

which conditions with Lemma 3.5. Thus $v_{\lambda}\neq0$, . It follows from (3.13) and (3.17), we have

For $\lambda\in (\lambda^{\ast}, 1]$, from (3.19) and (3.18), we have

which contradicts with Lemma 3.5. One gets $l = 0$, $\mathcal{I}_{ \lambda}(v_{ \lambda}) = c_{ \lambda}$ and $\mathcal{I}'_{ \lambda}(v_{ \lambda}) = 0$. Obviously, $v_{\lambda}\neq0$ and we complete the proof. Set

Lemma 3.9 Assume that $\rm(V_1)$, $\rm(V_3)$, $\rm(F_1)$-$\rm(F_3)$ hold. Then there exists $\hat{v}\in H^{1}(\mathbb{R}^{N})\backslash\{0\}$ such that

Proof. In view of Lemmas 3.4, 3.5 and 3.8, there exist a sequence $\{\lambda_{n}\}\subset (\lambda^{\ast}, 1]$ and $\{v_{\lambda_{n}}\}\subset H^{1} (\mathbb{R}^{N})\backslash \{0\}$ (for the sake of convenience, we denote the latter by $\{v_{n}\}$) such that

It follows from $\rm(V_3)$, (1.9), (3.1), (3.16) and Lemmas 3.4, 3.5 and 3.8 that

which combine with (2.32) yields that $\{v_{n}\}$ is bounded in $H^{1} (\mathbb{R}^{N})$. From (iv) of Lemma 3.4, we have $\lim_{n\rightarrow \infty}c_{\lambda_{n}} = c^{\ast}\leq c_{1}$. Then, it follows from (1.9) and (3.20) that $\mathcal{I}(v_{n})\rightarrow c^{\ast}\leq c_{1}, \; \mathcal{I}'(v_{n})\rightarrow 0$. Similar to the proof of (3.14), we get that there exists $\hat{v}\in H^{1} (\mathbb{R}^{N})\setminus \{0\}$ such that (3.19) holds.

Proof of Theorem 1.5. From Lemma 3.9, we know that $\mathcal{K} \neq\emptyset$ and $m^{\ast}\leq c_{1}$. For any $v\in \mathcal{K}$, Lemma 3.2 implies $\mathcal{P} (v) = \mathcal{P}_{1} (v) = 0$. Hence, as the proof of (3.19), we have $\mathcal{I}(v) = \mathcal{I}_{1}(v) > 0$ for any $v\in \mathcal{K}$, and so $m^{\ast}\geq0$. Let $\{v_{n}\}\subset \mathcal{K}$ such that $\mathcal{I} '(v_{n}) = 0\; {\rm{for\; all }}\; n\in \mathbb{N}$ and $\mathcal{I} (v_{n})\rightarrow m^{\ast} \; {\rm{as }}\; n\rightarrow \infty$. In view of Lemmas 3.4 and 3.9, $m^{\ast}\leq c_{1} < m^{\infty}$. By a similar argument as in the proof of Lemma 3.8, we can prove that there exists $\bar{ v} \in H^{1} (\mathbb{R}^{N})\setminus \{0\}$ such that $\mathcal{I} '(\bar{v}) = 0\; {\rm{and }} \; \mathcal{I} (\bar{v}) = m^{\ast}$. So, $\bar{v}$ is a least energy solution of $\rm(\bar{P})$ and $\bar{u} = G^{-1}(\bar{v})$ is a nontrivial solution of $\rm(P)$.

Acknowledgments

The authors are grateful to the anonymous referees for their carefully reading this paper and giving valuable comments and advices. This work is supported by the NSFC (11871475) and the National Natural Science Foundation of China (No:11971485)

Conflict of interest

The authors declare there is no conflicts of interest.