Existence of positive ground state solutions to a nonlinear fractional Schrödinger system with linear couplings

Abstract

In this paper, we investigate a nonlinear fractional Schrödinger system with linear couplings as follows:

$$ \textstyle\begin{cases} (-\Delta )^{\alpha }u+(1+a(x))u=F_{u}(u,v)+\lambda v,& \text{in } \mathbb{R}^{3}, \\ (-\Delta )^{\alpha }v+(1+b(x))v=F_{v}(u,v)+\lambda u,& \text{in } \mathbb{R}^{3}, \\ u,v\in H^{\alpha }(\mathbb{R}^{3}), \end{cases} $$

where $(-\Delta )^{\alpha }, \alpha \in (0,1)$, denotes the fractional Laplacian and $\lambda >0$ is the coupling parameter. Under some assumptions, we prove the existence of positive ground state solutions to the above system with the help of the method of Nehari manifold and concentration compactness lemma.

1 Introduction

This paper is devoted to the existence of positive ground state solutions of the following nonlinear fractional Schrödinger system with linear couplings:

$$ \textstyle\begin{cases} (-\Delta )^{\alpha }u+(1+a(x))u=F_{u}(u,v)+\lambda v,& \text{in } \mathbb{R}^{3}, \\ (-\Delta )^{\alpha }v+(1+b(x))v=F_{v}(u,v)+\lambda u,& \text{in } \mathbb{R}^{3}, \\ u,v\in H^{\alpha }(\mathbb{R}^{3}), \end{cases} $$

(1.1)

where $(-\Delta )^{\alpha }$, $\alpha \in (0,1)$, denotes the fractional Laplacian, $\lambda >0$ is the coupling parameter, and $a(x), b(x) \in L^{\infty }(\mathbb{R}^{3})$ satisfy the following conditions:

• (V1) $\inf_{\mathbb{R}^{3}} a(x) >\lambda -1$, $\inf_{\mathbb{R}^{3}} b(x) >\lambda -1$;

• (V2) $\lim_{|x|\rightarrow \infty }a(x)=\lim_{|x|\rightarrow \infty }b(x)=0$, $a(x)+b(x)\leq 0$.

When $a(x)=b(x)=0$, system (1.1) is reduced to the following system:

$$ \textstyle\begin{cases} (-\Delta )^{\alpha }u+u=F_{u}(u,v)+\lambda v,& \text{in } \mathbb{R}^{3}, \\ (-\Delta )^{\alpha }v+v=F_{v}(u,v)+\lambda u,& \text{in } \mathbb{R}^{3}, \\ u,v\in H^{\alpha }(\mathbb{R}^{3}). \end{cases} $$

(1.2)

Systems like (1.1) and (1.2) are often referred to as nonlocal problems because of the appearance of the terms $(-\Delta )^{ \alpha }u$ and $(-\Delta )^{\alpha }v$, which implies that problems (1.1) and (1.2) are no longer point-wise identities. In other words, the quantities $(-\Delta )^{\alpha }u$ and $(-\Delta )^{ \alpha }v$ depend on not only the value of u and v in a neighborhood of x (as in the case for the Laplacian), but also on the values of u and v at any point $y\in \mathbb{R}^{3}$. This phenomenon provokes some mathematical difficulties, which makes the study of such problems particularly interesting. In recent years, great interest has been devoted to the study of elliptic equations involving the fractional Laplacian operator $(-\Delta )^{\alpha }$, $\alpha \in (0,1)$. This operator appears in a quite natural way in many different applications, such as fractional quantum mechanics, population dynamics, soft thin films, and so on. For more details and applications about fractional Laplacian operators, we refer the reader to [1–3].

In the local case, that is, when $\alpha =1$, if the space $\mathbb{R}^{3}$ is replaced by $\mathbb{R}^{n}$, then problem (1.1) becomes

$$ \textstyle\begin{cases} -\Delta u+(1+a(x))u=F_{u}(u,v)+\lambda v,& \text{in } \mathbb{R} ^{n}, \\ -\Delta v+(1+b(x))v=F_{v}(u,v)+\lambda u, & \text{in } \mathbb{R} ^{n}, \\ (u,v)\in W^{1,2}(\mathbb{R}^{n})*W^{1,2}(\mathbb{R}^{n}). \end{cases} $$

(1.3)

In [4], Zhang, Xu, and Zhang studied the existence of positive ground state solutions to system (1.3) by virtue of the methods of Nehari manifold and concentration compactness lemma. In [5], Luo and Zhang studied the existence of radially symmetric solutions in $H_{\mathrm{rad}}^{1}(\mathbb{R}^{N})\times H_{ \mathrm{rad}}^{1}(\mathbb{R}^{N})$ of the Schrödinger system

$$ \textstyle\begin{cases} -\Delta u+\lambda _{1}u=\mu _{1} u^{3}+\beta uv^{2}-\epsilon v, &\text{in }\mathbb{R}^{N},\\ -\Delta v+\lambda _{2} v=\mu _{2}v^{3}+ \beta u^{2}v-\epsilon u, &\text{in }\mathbb{R}^{N},\\ u,v \in H^{1}(\mathbb{R}^{N}), \end{cases} $$

where $N=2, 3$, $\lambda _{1}=\lambda _{2}=\lambda$, $\mu _{1}, \mu _{2}>0$, $0<\beta <\min \{ \mu _{1},\mu _{2} \}$, and $\epsilon \in \mathbb{R}$ is a small linear coupling constant. By using the perturbation method, they proved that if ϵ is a small linear coupling constant, then the above system possesses at least one nontrivial bound state solution. Furthermore, by using the method of energy estimate, they proved that each component of the bound state solution is nonnegative. If $a(x)=b(x)=0$ and $F_{u}(u,v)$ and $F_{v}(u,v)$ are replaced by $f(u)$ and $g(v)$, respectively, then system (1.3) reduces to the following semilinear Schrödinger system:

$$ \textstyle\begin{cases} -\Delta u+u=f(u)+\lambda v ,& \text{in } \mathbb{R}^{n}, \\ -\Delta v+v=g(v)+\lambda u ,& \text{in } \mathbb{R}^{n}, \\ (u,v)\in H^{1}(\mathbb{R}^{n})*H^{1}(\mathbb{R}^{n}). \end{cases} $$

(1.4)

Due to its broad applications in several physical fields such as in nonlinear optics, plasma physics, and so on, problem (1.4) has been widely investigated by many authors in the past several years. For more details about the semilinear Schrödinger system, we refer the reader to [6–9] and the references therein.

In the nonlocal case, that is, when $\alpha \in (0,1)$, in [10], Lv and Peng studied the problem

$$ \textstyle\begin{cases} (-\Delta )^{\alpha } u+u=f(u)+\lambda v, & \text{in } \mathbb{R} ^{n}, \\ (-\Delta )^{\alpha } v+v=g(v)+\lambda u, & \text{in } \mathbb{R} ^{n}, \\ (u,v)\in H^{\alpha }(\mathbb{R}^{n})*H^{\alpha }(\mathbb{R}^{n}). \end{cases} $$

(1.5)

Under some assumptions on the nonlinear terms $f(u)$ and $g(v)$, they obtained the existence of positive vector solutions and vector ground state solutions for problem (1.5). Moreover, the asymptotic behavior of these solutions as $\lambda \rightarrow 0$ was described. In [11], by using the Nehari manifold and fibering map, Shen studied the multiplicity and concentration behavior of nontrivial solutions for the following coupled system:

$$ \textstyle\begin{cases} (-\Delta )^{s}u+\lambda V(x)u=f(x) \vert u \vert ^{q-2}u+\frac{\alpha }{\alpha + \beta } \vert u \vert ^{\alpha -2}u \vert v \vert ^{\beta }, &\text{in }\mathbb{R}^{N},\\ (-\Delta )^{s}v+\lambda W(x)v=g(x) \vert v \vert ^{q-2}v+\frac{ \alpha }{\alpha +\beta } \vert u \vert ^{\alpha } \vert v \vert ^{\beta -2}v, &\text{in }\mathbb{R}^{N}, \end{cases} $$

where $(-\Delta )^{s}$ is the fractional Laplacian operator with $s\in (0,1)$, the parameter $\lambda >0$, $1< q<2$, $\alpha , \beta >1$ with $\alpha +\beta <2_{s}^{*}$, and $N\geq 3$. When linear and nonlinear coupling terms both exist, in [12], Du and Mao showed that the problem

$$ \textstyle\begin{cases} (-\Delta )^{\alpha } u+(\lambda _{1}+V(x))u+kv=\mu _{1}u^{3}+\beta uv ^{2}, & \text{in } \mathbb{R}^{3}, \\ (-\Delta )^{\alpha } v+(\lambda _{2}+V(x))v+ku=\mu _{2}v^{3}+\beta u ^{2}v, & \text{in } \mathbb{R}^{3}, \\ (u,v)\in H^{\alpha }(\mathbb{R}^{3})*H^{\alpha }(\mathbb{R}^{3}), \end{cases} $$

(1.6)

possesses ground state solutions by using variational methods. For other work about the fractional Laplacian system, we would like to mention [13–18] and the references therein.

Motivated by the above works, especially by [12], it is very natural for us to pose the question: If the nonlinear terms $\mu _{1}u^{3}+\beta uv^{2}$ and $\mu _{2}v^{3}+\beta u^{2}v$ in system (1.6) are replaced by more general nonlinear terms $F_{u}(u,v)$ and $F_{v}(u,v)$, respectively, what will happen? Can we obtain the existence of positive ground state solutions to system (1.1)? In this paper, we will focus our attention on this question and give a definite answer.

To state our results, we impose the following conditions on the nonlinear terms $F_{u}$ and $F_{v}$ (where $\nabla F(u,v):=(F_{u},F _{v})$):

• (F1) $F\in C^{1}(\mathbb{R}^{2},\mathbb{R})$ and $F(0,0)=0$;

• (F2) $|\nabla F(u,v)|\leq C (1+|(u,v)|^{p-1})$, $2< p<2_{\alpha }^{*}$;

• (F3) $|\nabla F(u,v)|=o(|(u,v)|)$, as $|(u,v)|\rightarrow 0$;

• (F4) $\forall (u,v)\neq (0,0)$, $s>0$, $s\rightarrow \frac{ \nabla F(su,sv)(u,v)}{s}$ is strictly increasing;

• (F5) $\frac{F(u,v)}{u^{2}+v^{2}}\rightarrow \infty$ as $|(u,v)|\rightarrow \infty$;

• (F6) $F_{u}(u,v)\geq 0$, $F_{v}(u,v)\geq 0$, $F_{u}(0,v)=F_{v}(u,0)=0$, $u\geq 0$, $v\geq 0$;

• (F7) $F_{u}(u,v)=F_{v}(v,u)$, $u>0$, $v>0$.

In order to prove the existence of a positive ground states solution, the function $F(u,v)$ will be assumed to satisfy:

(F8):

$F(u,v)\leq F(|u|,|v|)$, $u,v\in \mathbb{R}$.

We say that $(u,v)\neq (0,0)$ is a ground state solution to system (1.1) if its energy is minimal among all nontrivial solutions. Moreover, a ground state solution $(u,v)$ with $u>0$ and $v>0$ ($u \geq 0$ and $v\geq 0$) is called a positive (non-negative) ground state solution.

The main results of this paper are the following.

Theorem 1.1

Suppose that (F1)–(F6) and (F8) are satisfied, $\lambda \in (0,1)$, then problem (1.2) possesses a positive ground state solution.

Theorem 1.2

Suppose that (V1), (V2) and (F1)–(F8) are satisfied, then problem (1.1) possesses a positive ground state solution.

Remark 1

Compared with the previous results, Theorem 1.2 can be regarded as an extension of Theorem 1.1 in [4] from the local to nonlocal case. A typical example of the assumption (F8) with $F(u,v)$ is the condition that $f(s,t)>0$ for $s,t>0$ and $f(s,t)=0$ for $s\leq 0$ or $t\leq 0$, which was considered in [19].

To prove Theorem 1.2, we will employ the method of Nehari manifold, which is based on the ideas developed in [20] and the concentration compactness lemma. We will reduce the variational problem to a constrained minimization problem on a Nehari manifold, and then apply the concentration compactness lemma to get our main result.

The remainder of this paper is organized as follows. In Sect. 2, some notations and preliminaries are presented. In Sect. 3, we will study Eq. (1.2) and prove Theorem 1.1. In Sect. 4, we will give the proof of our main result—Theorem 1.2.

2 Preliminary

In this section, we will establish the variational setting, and give some notations and preliminary lemmas which will be useful in the paper.

Throughout this paper, $C_{i}$ ($i=1,2,3,\dots$) will denote various positive constants, the strong convergence is denoted by →, and the weak convergence denoted as ⇀; $2_{\alpha }^{*}=\frac{2N}{N-2\alpha }$ is the fractional Sobolev critical exponent. For $1< p<+\infty$, and $f\in L^{p}(\mathbb{R}^{3})$, let $|f|_{L^{p}(\mathbb{R}^{3})}$ denote the usual $L^{p}$ norm of f. Let $L^{p}(\mathbb{R}^{3})*L^{p}(\mathbb{R}^{3})$ be the Cartesian product of two $L^{p}(\mathbb{R}^{3})$ spaces, and for $(f,g)\in L^{p}( \mathbb{R}^{3})*L^{p}(\mathbb{R}^{3})$, $|(f,g)|_{L^{p}*L^{p}}=|(f,g)|_{L ^{p}(\mathbb{R}^{3})*L^{p}(\mathbb{R}^{3})}:=(|f|_{L^{p}(\mathbb{R} ^{3})}^{p}+|g|_{L^{p}(\mathbb{R}^{3})}^{p})^{\frac{1}{p}}$. For any $\rho >0$ and $z\in \mathbb{R}^{3}$, $B_{\rho }(z):=\{x\in \mathbb{R} ^{3}:|x-z|\leq \rho \}$.

For any $\alpha \in (0,1)$, the fractional Sobolev space $H^{\alpha }( \mathbb{R}^{3})$ is defined by

$$ H^{\alpha }\bigl(\mathbb{R}^{3}\bigr)= \biggl\{ u\in L^{2}\bigl(\mathbb{R}^{3}\bigr):\frac{ \vert u(x)-u(y) \vert }{ \vert x-y \vert ^{\frac{3+2 \alpha }{2}}}\in L^{2}\bigl(\mathbb{R}^{3}*\mathbb{R}^{3} \bigr) \biggr\} . $$

It is a Hilbert space, when endowed with the scalar product given by

$$ (u,v)= \int _{\mathbb{R}^{3}} uv\, dx+ \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}} \frac{(u(x)-u(y))(v(x)-v(y))}{ \vert x-y \vert ^{3+2\alpha }}\, dx\,dy. $$

The corresponding norm is

$$ \Vert u \Vert _{H^{\alpha }}= \biggl( \int _{\mathbb{R}^{3}} \vert u \vert ^{2}\, dx+ \int _{\mathbb{R}^{3}} \int _{\mathbb{R}^{3}}\frac{ \vert u(x)-u(y) \vert ^{2}}{ \vert x-y \vert ^{3+2 \alpha }}\, dx\,dy \biggr)^{\frac{1}{2}}. $$

It is well-known that the fractional Laplacian $(-\triangle )^{\alpha }$ of a function u: $\mathbb{R}^{3}\rightarrow \mathbb{R}$ is defined by

$$ (-\triangle )^{\alpha }u(x)=\mathcal{F}^{-1}\bigl( \vert \xi \vert ^{2\alpha }( \mathcal{F}u)\bigr) (x), \quad \forall \xi \in \mathbb{R}^{3}, $$

where $\mathcal{F}$ is the Fourier transform, i.e.,

$$ \mathcal{F}u(\xi )=\frac{1}{(2\pi )^{\frac{3}{2}}} \int _{\mathbb{R} ^{3}} \exp (-2\pi i\xi \cdot x)u(x)\, dx, $$

where i is the imaginary unit. If u is smooth enough, it can be computed by the following singular integral:

$$ (-\triangle )^{\alpha }u(x)=c_{\alpha }\mathrm{P.V.} \int _{\mathbb{R}^{3}} \frac{u(x)-u(y)}{ \vert x-y \vert ^{3+2 \alpha }}\, dy,\quad \forall x\in \mathbb{R}^{3}, $$

where $c_{\alpha }$ is a normalization constant and $P.V$. stands for the principle value. Now we can get an alternative definition of the fractional Sobolev space $H^{\alpha }(\mathbb{R}^{3})$ via the Fourier transform as follows:

$$ H^{\alpha }\bigl(\mathbb{R}^{3}\bigr)= \biggl\{ u\in L^{2}\bigl(\mathbb{R}^{3}\bigr): \int _{\mathbb{R}^{3}} \vert \xi \vert ^{2\alpha } \vert \hat{u} \vert ^{2}+ \bigl\vert \hat{u}(\xi ) \bigr\vert ^{2} \, d\xi < \infty \biggr\} , $$

endowed with the norm

$$ \Vert u \Vert _{\alpha }= \biggl( \int _{\mathbb{R}^{3}}\bigl(1+ \vert \xi \vert ^{2\alpha } \bigr) \vert \hat{u} \vert ^{2}\, d\xi \biggr)^{\frac{1}{2}}, $$

where $\hat{u}=\mathcal{F}(u)$ denotes the Fourier transform of u. From Plancherel’s theorem, we have $\|u\|_{2}=\|\hat{u}\|_{2}$ and $\||\xi |^{\alpha }\hat{u}\|_{2}=\|(-\triangle )^{\frac{\alpha }{2}}u \|_{2}$. Hence

$$ \Vert u \Vert _{H^{\alpha }}= \biggl( \int _{\mathbb{R}^{3}}\bigl( \bigl\vert (-\triangle )^{\frac{ \alpha }{2}}u \bigr\vert ^{2}+ \bigl\vert u(x) \bigr\vert ^{2} \bigr)\, dx \biggr)^{\frac{1}{2}},\quad \forall u \in H^{\alpha }\bigl( \mathbb{R}^{3}\bigr). $$

It is easy to see that $\|\cdot \|_{H^{\alpha }}$ is equivalent to $\|\cdot \|_{\alpha }$.

The homogeneous Sobolev space $D^{\alpha ,2}(\mathbb{R}^{3})$ is defined by

$$ D^{\alpha }\bigl(\mathbb{R}^{3}\bigr)= \bigl\{ u\in L^{2_{\alpha }^{*}}\bigl( \mathbb{R}^{3}\bigr): \vert \xi \vert ^{\alpha }\hat{u}\in L^{2}\bigl(\mathbb{R}^{3} \bigr) \bigr\} , $$

which is the completion of $C_{0}^{\infty }(\mathbb{R}^{3})$ with respect to the norm

$$ \Vert u \Vert _{D^{\alpha ,2}}= \biggl( \int _{\mathbb{R}^{3}} \bigl\vert (-\triangle )^{\frac{ \alpha }{2}}u \bigr\vert ^{2}\, dx \biggr)^{\frac{1}{2}}= \biggl( \int _{\mathbb{R} ^{3}} \vert \xi \vert ^{2\alpha } \vert \hat{u} \vert ^{2}\, d\xi \biggr)^{\frac{1}{2}}. $$

Letting $H:=H^{\alpha }(\mathbb{R}^{3})*H^{\alpha }(\mathbb{R}^{3})$, we define an inner product on H as follows:

$$ \begin{aligned} \bigl((u_{1},v_{1}),(u_{2},v_{2}) \bigr)={} & \int _{\mathbb{R}^{3}}(-\Delta )^{\frac{ \alpha }{2}}u_{1}(- \Delta )^{\frac{\alpha }{2}}u_{2}\, dx+ \int _{\mathbb{R}^{3}}(-\Delta )^{\frac{\alpha }{2}}v_{1}(- \Delta )^{\frac{ \alpha }{2}}v_{2}\, dx \\ &{}+ \int _{\mathbb{R}^{3}}u_{1}(x)u_{2}(x)\, dx+ \int _{\mathbb{R}^{3}}v_{1}(x)v_{2}(x)\, dx, \end{aligned} $$

for $(u_{1},v_{1}),(u_{2},v_{2})\in H$; $\|(u,v)\|=((u,v),(u,v))^{ \frac{1}{2}}$ is the corresponding norm.

For $(u,v)\in H$, the energy functional associated with (1.1) is

$$ \begin{aligned} \varPhi (u,v)={} &\frac{1}{2} \int _{\mathbb{R}^{3}}\bigl( \bigl\vert (-\triangle )^{\frac{ \alpha }{2}}u \bigr\vert ^{2}+u^{2}\bigr)\, dx+ \frac{1}{2} \int _{\mathbb{R}^{3}}\bigl( \bigl\vert (- \triangle )^{\frac{\alpha }{2}}v \bigr\vert ^{2}+v^{2}\bigr)\, dx+ \frac{1}{2} \int _{\mathbb{R}^{3}}a(x)u^{2}\, dx \\ &{}+\frac{1}{2} \int _{\mathbb{R} ^{3}}b(x)v^{2}\, dx-\lambda \int _{\mathbb{R}^{3}} uv\, dx- \int _{\mathbb{R}^{3}}F(u,v)\, dx \\ ={}&\frac{1}{2} \bigl\Vert (u,v) \bigr\Vert ^{2}+ \frac{1}{2} \int _{\mathbb{R}^{3}}a(x)u^{2}\, dx+\frac{1}{2} \int _{\mathbb{R}^{3}}b(x)v^{2}\, dx-\lambda \int _{\mathbb{R}^{3}} uv \, dx- \int _{\mathbb{R}^{3}}F(u,v)\, dx. \end{aligned} $$

The Nehari manifold corresponding to Φ is defined by

$$ N = \bigl\{ (u,v)\in H\backslash \bigl\{ (0,0)\bigr\} :\bigl\langle \varPhi ^{\prime }(u,v),(u,v) \bigr\rangle =0 \bigr\} , $$

and thus for $(u,v)\in N$, one sees that

$$ \Vert u \Vert ^{2}+ \Vert v \Vert ^{2}+ \int _{\mathbb{R}^{3}}a(x)u^{2}\, dx+ \int _{\mathbb{R}^{3}}b(x)v^{2}\, dx-2\lambda \int _{\mathbb{R}^{3}} uv \, dx= \int _{\mathbb{R}^{3}}{\nabla F(u,v) (u,v)}\, dx. $$

In [21], since $F_{u}$ and $F_{v}$ are $C^{1}$ functions, it follows that $\varPhi \in C^{2}$ and $N\in C^{1}$. From these properties of Φ and N, one can easily deduce that critical points of Φ on N are critical points of Φ on H. Furthermore, one can use the standard Ljusternik–Schnirelmann category theory on N directly [22, 23]. However, in this paper, we cannot obtain these properties, since $F_{u}$ and $F_{v}$ are just continuous functionals, and N is only a continuous submanifold of H. To overcome these difficulties, we should carefully study the elementary properties for N as in [20]. By doing this, we can reduce a variational problem for an indefinite functional to a minimax problem on a manifold and find a positive solution to system (1.1). The main idea of this method is to find the minimizing sequence by using the relationship between N and the unit sphere of H. For more about the Nehari manifold, we refer the reader to [24–26] and the references therein.

Now we will introduce the following lemmas.

Lemma 2.1

([27])

For any $\alpha \in (0,1)$, $H^{\alpha }(\mathbb{R}^{3})$is continuously embedded into $L^{p}(\mathbb{R}^{3})$for $p\in [2,2_{\alpha }^{*}]$and compactly embedded into $L_{\mathrm{loc}}^{p}(\mathbb{R}^{3})$for $p\in [1,2_{ \alpha }^{*})$.

Lemma 2.2

([28])

If $\{u_{n}\}$is bounded in $H^{\alpha }(\mathbb{R}^{3})$with $\alpha \in (0,1)$and it satisfies

$$ \lim_{n\rightarrow \infty }\sup_{y\in \mathbb{R}^{3}} \int _{B_{R}(y)} \bigl\vert u_{n}(x) \bigr\vert ^{2}\, dx=0, $$

where $R>0$, then $u_{n}\rightarrow 0$in $L^{r}(\mathbb{R}^{N})$for every $2< r<2_{\alpha }^{*}$.

Remark 2

Similarly, in the case when the sequence $\{|u_{n}|^{2_{\alpha }^{*}}\}$ is vanishing, we can prove that $u_{n}\rightarrow 0$ in $L^{r}(\mathbb{R}^{3})$ for every $2< r\leq 2_{ \alpha }^{*}$.

Lemma 2.3

• (1) Suppose that (F2) and (F3) hold, then for any $\varepsilon >0$, there exists $C_{\epsilon }>0$such that

  $$ \bigl\vert \nabla F(u,v) \bigr\vert \leq \varepsilon \bigl\vert (u,v) \bigr\vert +C_{\epsilon } \bigl\vert (u,v) \bigr\vert ^{p-1}, \quad \forall u,v\in \mathbb{R}. $$

  (2.1)

• (2) Suppose that (F3) and (F4) hold, then the following inequality holds:

  $$ 0< 2F(u,v)< \nabla F(u,v) (u,v),\quad \forall (u,v)\neq (0,0). $$

  (2.2)

Proof

The proof of (1) is standard, we omit it here.

(2) By (F3) and (F4),

$$ \begin{aligned} 0< F(u,v) &= \int _{0}^{1}\frac{dF(su,sv)}{ds}\,ds \\ &= \int _{0}^{1}\frac{F _{u}(su,sv)u+F_{v}(su,sv)v}{s}s\,ds \\ &< \int _{0}^{1}\bigl(F_{u}(u,v)u+F_{v}(u,v)v \bigr)s\,ds \\ &=\frac{1}{2}\nabla F(u,v) (u,v), \end{aligned} $$

where $(u,v)\in H$, $(u,v)\neq (0,0)$. Therefore, the proof of Lemma 2.3 is completed. □

Remark 3

From Lemma 2.3, it is easy to see that

$$ \varPhi |_{N}(u,v)= \int _{\mathbb{R}^{3}} \biggl[\frac{1}{2}\nabla F(u,v) (u,v)-F(u,v) \biggr] \, dx>0. $$

Let us define the least energy on N as $c:=\inf_{N}\varPhi$; obviously, $c\geq 0$.

Lemma 2.4

• (1) Suppose that (V1) holds. Then there exist $\zeta ,\eta >0$such that for every $(u,v)\in H$,

  $$ \zeta \bigl\Vert (u,v) \bigr\Vert ^{2}\leq \Vert u \Vert ^{2}+ \Vert v \Vert ^{2}+ \int _{\mathbb{R}^{3}}a(x)u ^{2}\, dx+ \int _{\mathbb{R}^{3}}b(x)v^{2}\, dx-2\lambda \int _{\mathbb{R}^{3}} uv\, dx\leq \eta \bigl\Vert (u,v) \bigr\Vert ^{2}. $$

• (2) Suppose that (V1) and (F3)–(F5) hold. If $(u_{n},v_{n})\rightharpoonup (u,v)$and $(u,v)\neq (0,0)$inH, then for every positive sequence $\{t_{n}\}$satisfying $t_{n}\rightarrow \infty$,

  $$ \int _{\mathbb{R}^{3}}\frac{F(t_{n}u_{n},t_{n}v_{n})}{t_{n}^{2}}\, dx \rightarrow \infty . $$

  In particular, $\varPhi (t_{n}u_{n},t_{n}v_{n})\rightarrow -\infty$, as $n\rightarrow \infty$.

Proof

(1) Let

$$ \zeta =\min \Bigl\{ 1,~\inf_{\mathbb{R}^{3}} \bigl\{ a(x)- \lambda +1 \bigr\} , \inf_{\mathbb{R}^{3}} \bigl\{ b(x)- \lambda +1 \bigr\} \Bigr\} , $$

and set

$$ \eta =\max \bigl\{ 1+\lambda ,~ \bigl\vert a(x) \bigr\vert _{L^{\infty }(\mathbb{R}^{3})},~ \bigl\vert b(x) \bigr\vert _{L ^{\infty }(\mathbb{R}^{3})} \bigr\} . $$

A direct calculation shows that

$$ \begin{aligned} & \Vert u \Vert ^{2}+ \Vert v \Vert ^{2}+ \int _{\mathbb{R}^{3}}a(x)u^{2}\, dx+ \int _{\mathbb{R}^{3}}b(x)v^{2}\, dx-2\lambda \int _{\mathbb{R}^{3}} uv \, dx \\ &\quad \geq \Vert u \Vert ^{2}+ \Vert v \Vert ^{2}+ \int _{\mathbb{R}^{3}}a(x)u^{2}\, dx+ \int _{\mathbb{R}^{3}}b(x)v^{2}\, dx-\lambda \int _{\mathbb{R}^{3}} u ^{2}\, dx-\lambda \int _{\mathbb{R}^{3}} v^{2}\, dx \\ &\quad = \int _{\mathbb{R}^{3}} \bigl\vert (-\Delta )^{\frac{\alpha }{2}}u \bigr\vert ^{2}\, dx+ \int _{\mathbb{R}^{3}}\bigl(1+a(x)-\lambda \bigr)u^{2}\, dx \\ & \qquad {}+ \int _{\mathbb{R}^{3}} \bigl\vert (-\Delta )^{\frac{\alpha }{2}}v \bigr\vert ^{2}\, dx+ \int _{\mathbb{R}^{3}}\bigl(1+b(x)-\lambda \bigr)v^{2}\, dx \\ &\quad \geq \zeta \bigl\Vert (u,v) \bigr\Vert ^{2}, \end{aligned} $$

and

$$ \begin{aligned} & \Vert u \Vert ^{2}+ \Vert v \Vert ^{2}+ \int _{\mathbb{R}^{3}}a(x)u^{2}\, dx+ \int _{\mathbb{R}^{3}}b(x)v^{2}\, dx-2\lambda \int _{\mathbb{R}^{3}} uv \, dx \\ &\quad \leq \int _{\mathbb{R}^{3}} \bigl( \bigl\vert (-\Delta )^{\frac{\alpha }{2}}u \bigr\vert ^{2}+u _{2}\bigr)\, dx+ \int _{\mathbb{R}^{3}} \bigl( \bigl\vert (-\Delta )^{\frac{\alpha }{2}}v \bigr\vert ^{2}+v _{2}\bigr)\, dx+ \int _{\mathbb{R}^{3}}a(x)u^{2}\, dx \\ & \qquad {} + \int _{\mathbb{R}^{3}}b(x)v^{2}\, dx+2\lambda \int _{\mathbb{R}^{3}} uv \, dx \\ &\quad \leq \int _{\mathbb{R}^{3}} \bigl\vert (-\Delta )^{\frac{\alpha }{2}}u \bigr\vert ^{2} \, dx+ \int _{\mathbb{R}^{3}} \bigl\vert (-\Delta )^{\frac{\alpha }{2}}v \bigr\vert ^{2}\, dx+ \int _{\mathbb{R}^{3}}a(x)u^{2}\, dx+ \int _{\mathbb{R}^{3}}b(x)v^{2}\, dx \\ & \qquad {}+(1+\lambda ) \int _{\mathbb{R}^{3}} u^{2}\, dx+(1+\lambda ) \int _{\mathbb{R}^{3}} v^{2}\, dx \\ &\quad \leq \eta \bigl\Vert (u,v) \bigr\Vert ^{2}. \end{aligned} $$

(2) Since $(u_{n},v_{n})\rightharpoonup (u,v)$ in H, Lemma 2.2 implies that

$$\begin{aligned}& (u_{n},v_{n})\rightarrow (u,v) \quad \text{in } L_{\mathrm{loc}}^{2}\bigl(\mathbb{R}^{3} \bigr)*L_{\mathrm{loc}}^{2}\bigl(\mathbb{R} ^{3}\bigr), \\& u_{n}\rightarrow u, \qquad v_{n}\rightarrow v \quad \text{a.e. } x\in \mathbb{R}^{3}. \end{aligned}$$

In view of $(u,v)\neq (0,0)$, there exists a subset Ω of $\mathbb{R}^{3}$ with positive measure such that $(u(x),v(x))\neq (0,0)$, a.e. in Ω, thus $t_{n}|(u_{n},v_{n})| \rightarrow \infty$, $x\in \varOmega$. By (F5), we have

$$ \int _{\varOmega }\liminf_{n\rightarrow \infty } \frac{F(t_{n}u_{n},t _{n}v_{n})}{t_{n}^{2}(u_{n}^{2}+v_{n}^{2})}\bigl(u_{n}^{2}+v_{n}^{2} \bigr)\, dx= \infty , $$

and then in view of Fatou’s lemma, we can get

$$ \liminf_{n\rightarrow \infty } \int _{\mathbb{R}^{3}}\frac{F(t _{n}u_{n},t_{n}v_{n})}{t_{n}^{2}}\, dx=\infty . $$

Thus from conclusion (1) of Lemma 2.4 and the fact that $(u_{n},v_{n})$ is bounded in H, we obtain

$$ \varPhi (t_{n}u_{n},t_{n}v_{n}) \leq \frac{t_{n}^{2}}{2} \biggl[\eta \bigl( \Vert u \Vert ^{2}+ \Vert v \Vert ^{2}\bigr)- \int _{\mathbb{R}^{3}}\frac{2F(t_{n}u_{n},t_{n}v_{n})}{t _{n}^{2}}\, dx \biggr]\rightarrow - \infty . $$

The proof Lemma 2.4 is completed. □

In the following, we shall prove some elementary properties of N.

Lemma 2.5

Suppose that (V1) and (F1)–(F5) are satisfied. The the following statements hold:

• (1) For all $(u,v)\in H\backslash \{(0,0)\}$, there exists a unique $t_{(u,v)}>0$such that $t_{(u,v)}(u,v)\in N$and $\varPhi (t_{(u,v)}(u,v))= \max_{t>0}\varPhi (t(u,v))$.

• (2) The setNis bounded away from 0.

• (3) There exists $\gamma >0$such that $t_{(w,z)}\geq \gamma$for all $(w,z)\in S:=\{(u,v)\in H: \Vert (u,v)\|^{2}=1 \}$and for each compact subset $W \subset S$, there exists a constant $C_{w}$, such that $t_{(u,v)}\leq C_{w}$for all $(u,v)\in W$.

• (4) Nis a regular manifold, diffeomorphic to the unit sphere ofH.

• (5) $\varPhi ^{\prime }$is weakly sequentially continuous, namely, if $(u_{n},v_{n})\rightharpoonup (u,v)$inH, then $\varPhi ^{\prime }(u _{n},v_{n})\rightharpoonup \varPhi ^{\prime }(u,v)$inH.

Proof

(1) Letting $(u,v)\in H\backslash \{(0,0)\}$ be fixed and for $t\geq 0$, we consider the map $g: t\mapsto \varPhi (tu,tv)$ defined by

$$ \begin{aligned} g_{(u,v)}(t)={} &\frac{t^{2}}{2} \int _{\mathbb{R}^{3}}\bigl( \bigl\vert (-\triangle )^{\frac{ \alpha }{2}}u \bigr\vert ^{2}+u^{2}\bigr)\, dx+ \frac{t^{2}}{2} \int _{\mathbb{R}^{3}}\bigl( \bigl\vert (- \triangle )^{\frac{\alpha }{2}}v \bigr\vert ^{2}+v^{2}\bigr)\, dx+ \frac{t^{2}}{2} \int _{\mathbb{R}^{3}}a(x)u^{2}\, dx \\ & +{}\frac{t^{2}}{2} \int _{\mathbb{R}^{3}}b(x)v^{2}\, dx-t^{2} \int _{\mathbb{R}^{3}} \lambda uv\, dx- \int _{\mathbb{R}^{3}}F(tu,tv)\, dx \\ ={}& \frac{t^{2}}{2} \bigl\Vert (u,v) \bigr\Vert ^{2}+ \frac{t^{2}}{2} \int _{\mathbb{R}^{3}}\bigl(a(x)u ^{2}+b(x)v^{2} \bigr)\, dx-t^{2} \int _{\mathbb{R}^{3}} \lambda uv\, dx- \int _{\mathbb{R}^{3}}F(tu,tv)\, dx. \end{aligned} $$

It is easy to verify that there exists a unique $t_{(u,v)}>0$ such that $g_{(u,v)}^{\prime }(t)>0$ for $0< t< t_{(u,v)}$ and $g_{(u,v)}^{\prime }(t)<0$ for $t>t_{(u,v)}$. Indeed, noting that

$$ \begin{aligned} g_{(u,v)}^{\prime }(t)={}&t \biggl( \Vert u \Vert ^{2}+ \Vert v \Vert ^{2}+ \int _{\mathbb{R} ^{3}}\bigl(a(x)u^{2}+b(x)v^{2} \bigr)\,dx-2\lambda \int _{\mathbb{R}^{3}}uv\,dx \\ &{}- \int _{\mathbb{R}^{3}}\frac{\nabla F(tu,tv)(u,v)}{t}\,dx \biggr), \end{aligned} $$

from (F3) and Lemma 2.4 (1), we can see that $g_{(u,v)}^{\prime }(t)>0$ as long as $t>0$ is small enough.

On the other hand, from Lemma 2.4 (2) and (2.2), we can deduce that

$$ \int _{\mathbb{R}^{3}}\frac{\nabla F(tu,tv)(u,v)}{t}\, dx\geq \int _{\mathbb{R}^{3}}\frac{2F(tu,tv)}{t^{2}}\, dx\rightarrow \infty , \quad t\rightarrow \infty , $$

which implies that $g_{(u,v)}^{\prime }(t)<0$ when t is large enough. Hence, there exists a $t_{(u,v)}>0$ such that $g_{(u,v)}^{\prime }(t _{(u,v)}(u,v))=0$ and $\max_{t>0}g_{(u,v)}(t)=g(t_{(u,v)}(u,v))$. Namely, $\max_{t>0}g_{(u,v)}(t)$ is achieved at $t_{(u,v)}$ and $t_{(u,v)}(u,v)\in N$.

It remains to prove the uniqueness of $t_{(u,v)}$. In fact, if for any given $(u,v)\in H\backslash \{(0,0)\}$, if there exist two positive constants $t^{\prime }_{(u,v)}$, $t_{(u,v)}$ satisfying $t^{\prime } _{(u,v)}>t_{(u,v)}>0$ such that $t^{\prime }_{(u,v)}(u,v),~t_{(u,v)}(u,v) \in N$, then

$$\begin{aligned}& \Vert u \Vert ^{2}+ \Vert v \Vert ^{2}+ \int _{\mathbb{R}^{3}}a(x)u^{2}\, dx+ \int _{\mathbb{R}^{3}}b(x)v^{2}\, dx-2\lambda \int _{\mathbb{R}^{3}} uv \, dx= \int _{\mathbb{R}^{3}}\frac{\nabla F(tu,tv)(u,v)}{t}\, dx, \\& \Vert u \Vert ^{2}+ \Vert v \Vert ^{2}+ \int _{\mathbb{R}^{3}}a(x)u^{2}\, dx+ \int _{\mathbb{R}^{3}}b(x)v^{2}\, dx-2\lambda \int _{\mathbb{R}^{3}} uv \, dx= \int _{\mathbb{R}^{3}}\frac{\nabla F(t^{\prime }u,t^{\prime }v)(u,v)}{t ^{\prime }}\, dx. \end{aligned}$$

Thus

$$ 0= \int _{\mathbb{R}^{3}} \biggl(\frac{\nabla F(t^{\prime }u,t^{\prime }v)(u,v)}{t ^{\prime }}-\frac{\nabla F(tu,tv)(u,v)}{t} \biggr)\, dx, $$

which makes no sense in view of (F4) and $t^{\prime }_{(u,v)}>t_{(u,v)}>0$. Hence conclusion (1) holds.

(2) Let $(u,v)\in N$, then

$$ \Vert u \Vert ^{2}+ \Vert v \Vert ^{2}+ \int _{\mathbb{R}^{3}}a(x)u^{2}\, dx+ \int _{\mathbb{R}^{3}}b(x)v^{2}\, dx-2\lambda \int _{\mathbb{R}^{3}} uv \, dx= \int _{\mathbb{R}^{3}}{\nabla F(u,v) (u,v)}\, dx. $$

By (2.1) and Lemma 2.4 (1), we deduce that for every $\varepsilon >0$,

$$ \zeta \bigl( \Vert u \Vert ^{2}+ \Vert v \Vert ^{2}\bigr)\leq \varepsilon \bigl( \vert u \vert _{2}^{2}+ \vert v \vert _{2}^{2} \bigr)+C _{\varepsilon }\bigl( \vert u \vert _{p}^{p}+ \vert v \vert _{p}^{p}\bigr)\leq \varepsilon \bigl( \Vert u \Vert ^{2}+ \Vert v \Vert ^{2} \bigr)+CC_{\varepsilon }\bigl( \Vert u \Vert ^{p}+ \Vert v \Vert ^{p}\bigr). $$

Noting that $p>2$, there exists a $C_{1}>0$ such that

$$ \Vert u \Vert ^{p-2}+ \Vert v \Vert ^{p-2}\geq C_{1}. $$

Therefore, there exists a $\rho >0$ such that

$$ \Vert u \Vert ^{2}+ \Vert v \Vert ^{2}\geq \rho ^{2},\quad \forall (u,v) \in N. $$

(2.3)

So the conclusion (2) is proved.

(3) From (1), we know that for every $\{(u_{n},v_{n})\}\subset H \backslash \{(0,0)\}$ there exists a $t_{(u_{n},v_{n})}$ such that $t_{(u_{n},v_{n})}(u_{n},v_{n})\in N$. By conclusion (2), $\|t_{(u _{n},v_{n})}(u_{n},v_{n})\|=t_{(u_{n},v_{n})}\|(u_{n},v_{n})\|\geq \rho >0$. Thus $t_{(u_{n},v_{n})}\rightarrow 0$ is impossible. Consequently, there exists a $\gamma >0$ such that, for every $(w,z)\in S$, one has $t_{(w,z)}\geq \gamma$.

In order to prove $t_{(u,v)}\leq C_{w}$ for every $(u,v)\in W\subset S$, we argue indirectly and let $\{(u_{n},v_{n})\}\subset W\subset S$ be a sequence with $t_{(u_{n},v_{n})}\rightarrow \infty$. Since W is compact, there exists a $(u,v)\in W$ such that $(u_{n},v_{n})\rightarrow (u,v)$ in H. By the proof of Lemma 2.4 (2), $\varPhi (t_{(u_{n},v_{n})}(u _{n},v_{n}))\rightarrow -\infty$. On the other hand, $t_{(u_{n},v _{n})}(u_{n},v_{n})\in N$ shows $\varPhi (t_{(u_{n},v_{n})}(u_{n},v_{n})) \geq 0$, which is a contradiction to $\varPhi (t_{(u_{n},v_{n})}(u_{n},v _{n}))\rightarrow -\infty$.

(4) Define the mappings $\hat{m}:H\backslash \{(0,0)\}\rightarrow N$ and $m:S\rightarrow N$ by setting

$$ \hat{m}(u,v):=t_{(u,v)}(u,v), \quad m:= \hat{m}|_{S}. $$

(2.4)

By conclusions (1)–(3) of Lemma 2.5, we know that the conditions of Proposition 3.1 in [20] are satisfied. So the mapping m is a homeomorphism between S and N. Moreover, the inverse of m is given by

$$ \check{m}(u,v)=m^{-1}(u,v)=\frac{(u,v)}{ \Vert (u,v) \Vert }. $$

Thus, N is a regular manifold, diffeomorphic to the unit sphere of H.

(5) Assume that $(u_{n},v_{n})\rightharpoonup (u,v)$ in H, then we may assume that $(u_{n},v_{n})\rightarrow (u,v)$ in $L_{\mathrm{loc}}^{p}( \mathbb{R}^{3})*L_{\mathrm{loc}}^{p}(\mathbb{R}^{3})$ after passing to a subsequence. By (F2), we have

$$ F_{u}(u_{n},v_{n})\rightarrow F_{u}(u,v), \qquad F_{v}(u_{n},v_{n}) \rightarrow F_{v}(u,v), $$

in $L_{\mathrm{loc}}^{\frac{p}{p-1}}(\mathbb{R}^{3})*L_{\mathrm{loc}} ^{\frac{p}{p-1}}(\mathbb{R}^{3})$. Therefore, for every $(\phi , \varphi )\in C_{0}^{\infty }(\mathbb{R}^{3})*C_{0}^{\infty }( \mathbb{R}^{3})$, with the help of Hölder inequality, we have

$$\begin{aligned}& \int _{\mathbb{R}^{3}}F_{u}(u_{n},v_{n}) \phi \, dx\rightarrow \int _{\mathbb{R}^{3}}F_{u}(u,v)\phi \, dx, \\& \int _{\mathbb{R}^{3}}F_{v}(u_{n},v_{n}) \varphi \, dx\rightarrow \int _{\mathbb{R}^{3}}F_{v}(u,v)\varphi \, dx, \end{aligned}$$

which implies that

$$ \bigl\langle \varPhi ^{\prime }(u_{n},v_{n}),( \phi ,\varphi )\bigr\rangle \rightarrow \bigl\langle \varPhi ^{\prime }(u,v),( \phi ,\varphi )\bigr\rangle . $$

(2.5)

We claim that $\{\varPhi ^{\prime }(u_{n},v_{n})\}$ is bounded in H. Indeed, for every $(h,k)\in H$, by (2.1), Sobolev embedding theorem, and Hölder inequality, we get, for any $\varepsilon >0$,

$$ \begin{aligned} \biggl\vert \int _{\mathbb{R}^{3}}F_{u}(u_{n},v_{n})h \, dx \biggr\vert &\leq \varepsilon \int _{\mathbb{R}^{3}}\bigl( \vert u_{n} \vert + \vert v_{n} \vert \bigr) \vert h \vert \, dx+C_{\varepsilon } \int _{\mathbb{R}^{3}}\bigl( \vert u_{n} \vert ^{p-1}+ \vert v_{n} \vert ^{p-1}\bigr) \vert h \vert \, dx \\ &\leq \varepsilon \bigl( \vert u_{n} \vert _{2}+ \vert v_{n} \vert _{2}\bigr) \vert h \vert _{2}+C_{\varepsilon }\bigl( \vert u_{n} \vert _{p} ^{p-1}+ \vert v_{n} \vert _{p}^{p-1}\bigr) \vert h \vert _{p} \\ &\leq \varepsilon \bigl( \Vert u_{n} \Vert + \Vert v _{n} \Vert \bigr) \Vert h \Vert +CC_{\varepsilon }\bigl( \Vert u_{n} \Vert ^{p-1}+ \Vert v_{n} \Vert ^{p-1}\bigr) \Vert h \Vert \\ &\leq C' \Vert h \Vert . \end{aligned} $$

Similarly, $|\int _{\mathbb{R}^{3}}F_{v}(u_{n},v_{n})k\, dx| \leq C\|k \|$. Hence

$$ \bigl\vert \bigl\langle \varPhi ^{\prime }(u_{n},v_{n}),(h,k) \bigr\rangle \bigr\vert \leq C \bigl\Vert (h,k) \bigr\Vert . $$

Since $\{\varPhi ^{\prime }(u_{n},v_{n})\}$ is bounded in H and $C_{0}^{\infty }(\mathbb{R}^{3})*C_{0}^{\infty }(\mathbb{R}^{3})$ is dense in H, we conclude that (2.5) holds for every $(\phi , \varphi )\in H$ and so $\varPhi ^{\prime }(u_{n},v_{n})\rightharpoonup \varPhi ^{\prime }(u,v)$ in H.

Now we shall consider the functional $\varPsi :S\rightarrow R$, defined by

$$ \varPsi (u,v):=\varPhi \bigl(m(u,v)\bigr), $$

where $m(u,v)$ is given in (2.4). As in [20], we have the following lemma. □

Lemma 2.6

Let assumptions (V1) and (F1)–(F5) be satisfied. Then the following statements hold:

• (1) If $\{(w_{n},z_{n})\}$is a $(P.S.)$sequence forΨ, then $\{m(w_{n},z_{n})\}$is a $(P.S.)$sequence forΦ,

• (2) $(w,z)$is a critical point ofΨif and only if $m(w,z)$is a nontrivial critical point ofΦ. Moreover, $\inf_{S}\varPsi =\inf_{N}\varPhi$,

• (3) A minimizer ofΦonNis a ground state solution to the system (1.1).

Proof

Let us first prove conclusions (1) and (2). Since H is a Hilbert space, it follows that H satisfies $(A_{1})$ of Corollary 3.3 in [20]. From Lemma 2.5 (3), we know that the functional Φ satisfies $(A_{2})$ and $(A_{3})$ of Corollary 3.3 in [20]. Consequently, by Corollary 3.3 of [20], conclusions (1) and (2) hold.

Now we prove conclusion (3). Indeed, let $(u,v)\in N$ be such that $\varPhi (u,v)=c=\inf_{N}\varPhi$, then $\varPsi (w,z)=c$, where $(w,z)=m^{-1}(u,v)\in S$. From (2), one has $\varPsi (w,z)=\inf_{S}\varPsi$. Once again, (2) implies $\varPsi ^{\prime }(w,z)=0$. Thus, $\varPhi ^{\prime }(u,v)=0$.

For any $(u^{\prime },v^{\prime })\neq (0,0)$ satisfying $\varPhi ^{ \prime }(u^{\prime },v^{\prime })=0$, we have $(u^{\prime },v^{\prime })\in N$, and from the definition of c, we obtain $\varPhi (u^{\prime },v^{\prime })\geq c=\varPhi (u,v)$. Thus, $(u,v)$ is a ground state solution to (1.1). □

Remark 4

From Lemma 2.6 (3), we know that the problem of seeking for a ground state solution to system (1.1) can be transformed into that of finding a minimizer of $\varPhi |_{N}$. In the process of finding the minimizer, what we mainly need to do is to recover the compactness of a $(P.S.)$ sequence for Φ. Using the concentration compactness lemma, the relationship between Eq. (1.1) and its corresponding limit system, we can overcome this obstacle.

3 The limit system

In the process of looking for ground state solutions to system (1.1), the corresponding limit system is very important. In this section, we shall prove some properties of a ground state solution to the limit system and give the proof of Theorem 1.1. Precisely, we will be concerned with the following equation:

$$ \textstyle\begin{cases} (-\Delta )^{\alpha }u+u=F_{u}(u,v)+\lambda v,& \text{in } \mathbb{R}^{3}, \\ (-\Delta )^{\alpha }v+v=F_{v}(u,v)+\lambda u,& \text{in } \mathbb{R}^{3}, \\ u,v\in H^{\alpha }(\mathbb{R}^{3}), \end{cases} $$

(3.1)

and the energy functional corresponding to system (3.1) is

$$ \begin{aligned} \varPhi _{\infty }(u,v)={} & \frac{1}{2} \int _{\mathbb{R}^{3}}\bigl( \bigl\vert (-\triangle )^{\frac{ \alpha }{2}}u \bigr\vert ^{2}+u^{2}\bigr)\, dx+ \frac{1}{2} \int _{\mathbb{R}^{3}}\bigl( \bigl\vert (- \triangle )^{\frac{\alpha }{2}}v \bigr\vert ^{2}+v^{2}\bigr)\, dx \\ &{} -\lambda \int _{\mathbb{R}^{3}} uv\, dx- \int _{\mathbb{R}^{3}}F(u,v)\, dx \\ ={}&\frac{1}{2}\bigl( \Vert u \Vert ^{2}+ \Vert u \Vert ^{2}\bigr)-\lambda \int _{\mathbb{R}^{3}} uv \, dx- \int _{\mathbb{R}^{3}}F(u,v)\, dx. \end{aligned} $$

As in Sect. 2, in order to find a critical point of the functional $\varPhi _{\infty }$, we also use the Nehari manifold methods. The Nehari manifold corresponding to $\varPhi _{\infty }$ is defined by

$$ N_{\infty } =\bigl\{ (u,v)\in H\backslash \bigl\{ (0,0)\bigr\} :\bigl\langle \varPhi _{\infty } ^{\prime }(u,v),(u,v)\bigr\rangle =0 \bigr\} , $$

and the least energy on $N_{\infty }$ is defined by $c_{\infty }:= \inf_{N_{\infty }}\varPhi$. One checks immediately that $c_{\infty }>0$. Replacing Φ, N, m, Ψ by $\varPhi _{\infty }$, $N_{\infty }$, $m_{\infty }$, $\varPsi _{\infty }$ respectively, Lemmas 2.5 and 2.6 still hold. For the reader’s convenience, we give the conclusions.

Lemma 3.1

Suppose that (F1)–(F4) hold and $\lambda \in (0,1)$, then

• (1) $N_{\infty }$is a regular manifold, diffeomorphic to the unit sphere ofH;

• (2) If $\{(w_{n},z_{n})\}$is a $(P.S.)$sequence for $\varPsi _{\infty }$, then $\{m_{\infty }(w_{n},z_{n})\}$is a $(P.S.)$sequence for $\varPhi _{\infty }$;

• (3) $(w,z)$is a critical point of $\varPsi _{\infty }$if and only if $m(w,z)$is a nontrivial critical point of $\varPhi _{\infty }$. Moreover, $\inf_{S}\varPsi _{\infty }=\inf_{N_{\infty }} \varPhi _{\infty }$;

• (4) A minimizer of $\varPhi _{\infty }$on $N_{\infty }$is a ground state solution to system (3.1);

• (5) $\varPhi _{\infty }^{\prime }$is weakly sequentially continuous.

With the above preparations at hand, now we give the proof of Theorem 1.1.

Proof

We divide the proof into six steps:

Step 1. The existence of (P.S.) sequence for $\varPhi _{\infty }$.

Assume that $\{(\bar{w}_{n},\bar{z}_{n})\}\subset S$ is a minimizing sequence such that $\varPsi _{\infty }(\bar{w}_{n},\bar{z}_{n})\rightarrow \inf_{S}\varPsi _{\infty }$. By Ekeland’s variational principle in [22], without loss of generality, we may suppose $\varPsi _{\infty }^{\prime }(\bar{w}_{n},\bar{z}_{n})\rightarrow 0$. It follows from Lemma 3.1 (2) that $\varPhi _{\infty }^{\prime }(u_{n},v_{n}) \rightarrow 0$, where $(u_{n},v_{n})=m_{\infty }(\bar{w}_{n},\bar{z} _{n})\in N_{\infty }$. Moveover, by Lemma 3.1 (3), we have $\varPhi _{ \infty }(u_{n},v_{n})=\varPsi _{\infty }(\bar{w}_{n},\bar{z}_{n})\rightarrow c_{\infty }$.

Step 2. $\{(u_{n},v_{n})\}$ is bounded in H.

Suppose on the contrary that $s_{n}:=\|(u_{n},v_{n})\|\rightarrow \infty$. Define $(w_{n},z_{n})=\frac{(u_{n},v_{n})}{\|(u_{n},v_{n}) \|}$. Clearly, $\{(w_{n},z_{n})\}$ is bounded in H and there exists $(w,z)\in H$ such that, passing if necessary to a subsequence,

$$\begin{aligned}& (w_{n},z_{n})\rightharpoonup (w,z)\in H, \\& (w_{n},z_{n})\rightarrow (w,z)\in L_{\mathrm{loc}}^{2} \bigl(\mathbb{R}^{3}\bigr)*L _{\mathrm{loc}}^{2}\bigl( \mathbb{R}^{3}\bigr), \\& w_{n}\rightarrow w,\qquad z_{n}\rightarrow z\quad \mbox{a.e. in } \mathbb{R}^{3}. \end{aligned}$$

Obviously, the sequence $(w_{n},z_{n})$ is bounded in $L^{p}( \mathbb{R}^{3})*L^{p}( \mathbb{R}^{3})$ by Sobolev embedding theorem, i.e., $\{|w_{n}|_{p}+|z_{n}|_{p}\}$ is a bounded sequence in $\mathbb{R}$ and therefore we can assume $|w_{n}|_{p}+|z_{n}|_{p} \rightarrow A\in [0,\infty )$. In the following, we shall prove $|w_{n}|_{p}+|z_{n}|_{p}\rightarrow A\in [0,\infty )$ cannot happen, which implies that $\{(u_{n},v_{n})\}$ is bounded in H. To do this, we will consider the following two cases.

Case A. $|w_{n}|_{p}+|z_{n}|_{p}\rightarrow A=0$.

From (2.1), for any $\varepsilon >0$, $s>0$, we have

$$ \biggl\vert \int _{\mathbb{R}^{3}}F(sw_{n},sz_{n})\, dx \biggr\vert \leq \varepsilon s^{2}\bigl( \vert w _{n} \vert _{2}^{2}+ \vert z_{n} \vert _{2}^{2}\bigr)+s^{p}CC_{\varepsilon } \bigl( \vert w_{n} \vert _{p}^{p}+ \vert z _{n} \vert _{p}^{p}\bigr). $$

Since $A=0$ and $\{(w_{n},z_{n})\}$ is bounded in $L^{2}(\mathbb{R} ^{3})*L^{2}(\mathbb{R}^{3})$, one has

$$ \biggl\vert \int _{\mathbb{R}^{3}}F(sw_{n},sz_{n})\, dx \biggr\vert \leq C\varepsilon . $$

Thus $\int _{\mathbb{R}^{3}}F(sw_{n},sz_{n})\, dx\rightarrow 0$ as $n\rightarrow \infty$. So we infer from Lemma 2.4 (1), that for each $s>0$,

$$ \begin{aligned} c_{\infty }+o(1) &=\varPhi _{\infty }(u_{n},v_{n}) \\ &\geq \varPhi _{\infty }(sw _{n},sz_{n}) \\ &\geq \frac{s^{2}\zeta }{2}\bigl( \Vert w_{n} \Vert ^{2}+ \Vert z_{n} \Vert ^{2}\bigr)- \int _{\mathbb{R}^{3}}F(sw_{n},sz_{n})\, dx \\ & \rightarrow \frac{s^{2} \zeta }{2}, \end{aligned} $$

as $n\rightarrow \infty$. This is a contradiction if $s>\sqrt{\frac{2c _{\infty }}{\zeta }}$.

Case B. $|w_{n}|_{p}+|z_{n}|_{p}\rightarrow A\in (0,\infty )$.

Without loss of generality, we may assume that $w_{n}\nrightarrow 0$ in $L^{p}(\mathbb{R}^{3})$. From Lemma 2.2, there exist a $\delta >0$ and a sequence $\{x_{n}\}\subset \mathbb{R}^{3}$ such that

$$ \int _{B(x_{n},1)} \vert w_{n} \vert ^{2}\, dx>\delta . $$

(3.2)

We first prove that $\{x_{n}\}$ is bounded in $\mathbb{R}^{3}$. Indeed, suppose on the contrary that $|x_{n}|\rightarrow \infty$. Let $\tilde{w}_{n}(x)=w_{n}(x+x_{n})$, $\tilde{z}_{n}(x)=z_{n}(x+x_{n})$. It is obvious that $\|(\tilde{w}_{n},\tilde{z}_{n})\|=\|(w_{n},z_{n}) \|=1$, thus, $\{(\tilde{w}_{n},\tilde{z}_{n})\}$ is bounded. Up to a subsequence, there exists a $(\tilde{w},\tilde{z})\in H$ such that $(\tilde{w}_{n},\tilde{z}_{n})\rightharpoonup (\tilde{w},\tilde{z})$ in H, $(\tilde{w}_{n},\tilde{z}_{n})\rightarrow (\tilde{w},\tilde{z})$ in $L_{\mathrm{loc}}^{2}(\mathbb{R}^{3})*L_{\mathrm{loc}}^{2}( \mathbb{R}^{3})$, $\tilde{w}_{n}\rightarrow \tilde{w}$, $\tilde{z} _{n}\rightarrow \tilde{z}$ for a.e. $x\in \mathbb{R}^{3}$. It follows from (3.2) that

$$ \int _{B(0,1)} \vert \tilde{w}_{n} \vert ^{2}\, dx>\delta . $$

Since $\tilde{w}_{n}\rightarrow \tilde{w}$ in $L_{\mathrm{loc}}^{2}( \mathbb{R}^{3})$, the latter implies $\tilde{w}\neq 0$. Therefore $(\tilde{w},\tilde{z})\neq (0,0)$. And consequently from Lemma 2.4, we get

$$\begin{aligned}& \int _{\mathbb{R}^{3}}\frac{F(s_{n}\tilde{w}_{n},s_{n}\tilde{z}_{n})}{s _{n}^{2}}\, dx=\infty , \\& \begin{aligned} 0 &\leq \frac{\varPhi _{\infty }(u_{n},v_{n})}{ \Vert (u_{n},v_{n}) \Vert ^{2}} \\ &= \frac{1}{2} \biggl[\frac{ \Vert u_{n} \Vert ^{2}+ \Vert v_{n} \Vert ^{2}-2\lambda \int _{\mathbb{R}^{3}} u_{n}v_{n}\, dx}{ \Vert (u_{n},v_{n}) \Vert ^{2}} \biggr]- \int _{\mathbb{R}^{3}}\frac{F(s_{n}w_{n},s_{n}z_{n})}{s_{n}^{2}}\, dx \\ &\leq \frac{c_{1}}{2}- \int _{\mathbb{R}^{3}}\frac{F(s_{n}\tilde{w} _{n},s_{n}\tilde{z}_{n})}{s_{n}^{2}}\, dx \\ &\rightarrow - \infty , \end{aligned} \end{aligned}$$

where $c_{1}\geq \lambda +1$ is a positive constant, which yields to a contradiction.

It follows from $w_{n}\rightarrow w$ in $L_{\mathrm{loc}}^{2}( \mathbb{R}^{3})$ that $w\neq 0$. Thus by Lemma 2.4 (2) we obtain $\varPhi _{\infty }(s_{n}w_{n},s_{n}z_{n})\rightarrow -\infty$, which is a contradiction to the fact that $\varPhi _{\infty }(s_{n}w_{n},s_{n}z _{n})=\varPhi _{\infty }(u_{n},v_{n})\geq 0$.

Step 3. $\varPhi _{\infty }^{\prime }(\check{u},\check{v})=0$.

From Step 2, $\{(u_{n},v_{n})\}$ is bounded in H. Thus, up to a subsequence, there exists a $(\check{u},\check{v})\in H$ such that $(u_{n},v_{n})\rightharpoonup (\check{u},\check{v})$ in H, $(u_{n},v_{n})\rightarrow (\check{u},\check{v})$ in $L_{\mathrm{loc}} ^{2}(\mathbb{R}^{3})*L_{\mathrm{loc}}^{2}(\mathbb{R}^{3})$, $u_{n}\rightarrow \check{u}$, $v_{n}\rightarrow \check{v}$ for a.e. $x\in \mathbb{R}^{3}$. By Lemma 3.1 (5), we see that $\varPhi _{\infty } ^{\prime }(\check{u},\check{v})=0$.

Step 4. $(\check{u},\check{v})\neq (0,0)$.

By Sobolev embedding theorem, it is easy to see that the sequence $\{(u_{n},v_{n})\}$ is bounded in $L^{p}( \mathbb{R}^{3})*L^{p}( \mathbb{R}^{3})$, i.e., $\{|u_{n}|_{p}+|v_{n}|_{p}\}$ is a bounded sequence in $\mathbb{R}$. Selecting a subsequence if necessary, we can assume that $|u_{n}|_{p}+|v_{n}|_{p}\rightarrow B\in [0,\infty )$. If $B=0$, then $\int _{\mathbb{R}^{3}}\nabla F(u_{n},v_{n})(u_{n},v_{n}) \, dx\rightarrow 0$. It follows from Lemma 2.4 (1) that

$$ \begin{aligned} o(1) &=\bigl\langle \varPhi _{\infty }^{\prime }(u_{n},v_{n}),(u_{n},v_{n}) \bigr\rangle \\ &\geq \zeta \bigl( \Vert u_{n} \Vert ^{2}+ \Vert v_{n} \Vert ^{2}\bigr)- \int _{\mathbb{R} ^{3}}\nabla F(u_{n},v_{n}) (u_{n},v_{n})\, dx \\ &=\zeta \bigl( \Vert u_{n} \Vert ^{2}+ \Vert v_{n} \Vert ^{2}\bigr)+o(1). \end{aligned} $$

This yields immediately that $(u_{n},v_{n})\rightarrow (0,0)$ in H. Since $(u_{n},v_{n})\in N_{\infty }$, we obtain, for some $\rho >0$, $\|(u_{n},v_{n})\|>\rho$, which shows that one cannot have $B=0$. Without loss of generality, we may assume that $u_{n}\nrightarrow 0$ in $L^{p}(\mathbb{R}^{3})$. From Lemma 2.2, there exist $y_{n} \in \mathbb{R}^{3}$ and $\delta >0$ such that

$$ \int _{B(y_{n},1)} \vert u_{n} \vert ^{2}\, dx>\delta . $$

(3.3)

By using the same arguments as in Step 2, we can prove that $\{y_{n}\}$ is bounded in $\mathbb{R}^{3}$. Observe that $u_{n}\rightarrow \check{u}$ in $L_{\mathrm{loc}}^{2}(\mathbb{R}^{3})$ and (3.3) imply $\check{u}\neq 0$. So $(\check{u},\check{v})\neq (0,0)$.

Step 5. $\varPhi _{\infty }(\check{u},\check{v}) = c_{\infty }$.

From $\varPhi _{\infty }^{\prime }(\check{u},\check{v})=0$ and $( \check{u},\check{v})\neq (0,0)$, we have $(\check{u},\check{v})\in N _{\infty }$. Thus $\varPhi _{\infty }(\check{u},\check{v}) \geq c_{\infty }$. By using Fatou’s lemma, it is easy to check that

$$ \begin{aligned} c_{\infty }+o(1) &=\varPhi _{\infty }(u_{n},v_{n}) \\ &= \int _{\mathbb{R}^{3}} \biggl[\frac{1}{2}\nabla F(u_{n},v_{n}) (u_{n},v_{n})-F(u_{n},v_{n}) \biggr] \, dx \\ &\geq \int _{\mathbb{R}^{3}} \biggl[\frac{1}{2}\nabla F(\check{u}, \check{v}) (\check{u},\check{v})-F(\check{u},\check{v}) \biggr]\, dx+o(1) \\ &=\varPhi _{\infty }(\check{u},\check{v})+o(1). \end{aligned} $$

Therefore $\varPhi _{\infty }(\check{u},\check{v}) = c_{\infty }$, which means $(\check{u},\check{v})$ is a ground state solution to system (3.1).

Step 6. $\check{u}>0$, $\check{v}>0$.

In the following, we first prove $\check{u}\geq 0$, $\check{v}\geq 0$. Namely, $(\check{u},\check{v})$ is a non-negative ground state solution. Indeed, since $(\check{u},\check{v})\in H$,

$$ \bigl\vert (-\triangle )^{\frac{\alpha }{2}} \check{u} \bigr\vert _{2}^{2} \geq \bigl\vert (-\triangle )^{\frac{\alpha }{2}} \vert \check{u} \vert \bigr\vert _{2}^{2}, \qquad \bigl\vert (-\triangle )^{\frac{ \alpha }{2}} \check{v} \bigr\vert _{2}^{2} \geq \bigl\vert (-\triangle )^{ \frac{\alpha }{2}} \vert \check{v} \vert \bigr\vert _{2}^{2}, $$

(3.4)

so $(|\check{u}|,|\check{v}|)\in H$. By Lemma 2.5 (1), there exists a $\tau >0$ such that $\tau (|\check{u}|,|\check{v}|)\in N_{\infty }$, which implies that $\varPhi _{\infty }(\tau (|\check{u}|,|\check{v}|) ) \geq c_{\infty }$. By (3.4) and (F8), we have $\varPhi _{\infty }( \tau |\check{u}|,\tau |\check{v}|)\leq \varPhi _{\infty }(\tau \check{u}, \tau \check{v})$. Since $\varPhi _{\infty }(\tau \check{u},\tau \check{v}) \leq \varPhi _{\infty }(\check{u},\check{v})$, we obtain $\varPhi _{\infty }( \tau |\check{u}|,\tau |\check{v}|)\leq c_{\infty }$, so that $\varPhi _{\infty }(\tau |\check{u}|,\tau |\check{v}|)= c_{\infty }$, which means $(\tau |\check{u}|,\tau |\check{v}|)$ is also a solution to system (3.1), that is, system (3.1) has a non-negative ground state solution.

Next we prove $\check{u}>0$, $\check{v}>0$. If $\check{u}\equiv 0$, then it follows from the first equation of (3.1), (F5) and $\lambda >0$ that $\check{v}\equiv 0$. Therefore $(\check{u}, \check{v})=(0,0)$, a contradiction. Reasoning as above, we obtain $\check{v}\not \equiv 0$. By using (F5) and the strong maximum principle to each single equation in (3.1), we know that $\check{u}>0$, $\check{v}>0$. Hence, $(\check{u},\check{v})$ is a positive ground state solution to system (3.1).

The proof of Theorem 1.1 is completed. □

Remark 5

By the above arguments, we proved that limit system (3.1) has a positive ground state solution. Namely, if $a(x)=b(x)=0$, then system (1.1) has a positive ground state solution. In the following section, we shall consider the case $a^{2}(x)+b^{2}(x)\not \equiv 0$.

4 Proof of Theorem 1.2

The following lemma establishes an inequality between c and $c_{\infty }$ that will be essential in the proof of Theorem 1.2. Loosely speaking, with the aid of this inequality, we can recover the compactness of a $(P.S.)$ sequence for Φ.

Lemma 4.1

Suppose that (V1), (V2) and (F1)–(F8) hold. Then $c< c_{\infty }$.

Proof

Let $(u_{0},v_{0})\in N_{\infty }$ be a positive ground state solution to system (3.1). Since (F7), we see that $\varPhi _{\infty }(v_{0},u_{0})=c_{\infty }$, that is, $(v_{0},u_{0})$ is a minimizer of $\varPhi _{\infty }|_{N_{\infty }}$. Choose $t,\tau >0$ such that $t(u_{0},v_{0}),\tau (v_{0},u_{0})\in N$. By (V2),

$$ \int _{\mathbb{R}^{3}}\bigl(a(x)+b(x)\bigr) \bigl(u_{0}^{2}+v_{0}^{2} \bigr)\, dx\leq 0. $$

Therefore, $\int _{\mathbb{R}^{3}}a(x)u_{0}^{2}\, dx+\int _{\mathbb{R} ^{3}}b(x)v_{0}^{2}\, dx\leq 0$ or $\int _{\mathbb{R}^{3}}a(x)v_{0}^{2} \, dx+\int _{\mathbb{R}^{3}}b(x)u_{0}^{2}\, dx\leq 0$. Without loss of generality, we may assume that $\int _{\mathbb{R}^{3}}a(x)u_{0}^{2}\, dx+ \int _{\mathbb{R}^{3}}b(x)v_{0}^{2}\, dx\leq 0$. This implies that $\varPhi (tu_{0},tv_{0})\leq \varPhi _{\infty }(tu_{0},tv_{0})$. Since $(u_{0},v_{0})\in N_{\infty }$, we have $\varPhi _{\infty }(tu_{0},tv_{0}) \leq \varPhi _{\infty }(u_{0},v_{0})$. Hence, by the definition of c and $c_{\infty }$, we can guarantee that

$$ c\leq \varPhi (tu_{0},tv_{0})\leq \varPhi _{\infty }(tu_{0},tv_{0})\leq \varPhi _{\infty }(u_{0},v_{0})=c_{\infty }. $$

(4.1)

In the sequel, we shall prove that $c=c_{\infty }$ cannot happen, which implies $c< c_{\infty }$. Indeed, if $c=c_{\infty }$, then $t=1$, which means $(u_{0},v_{0})\in N$ and $\varPhi (u_{0},v_{0})=c$, i.e., $(u_{0},v_{0})$ is a solution to system (1.1). Still from $t=1$, we also get $(u_{0},v_{0})\in N_{\infty }$ and $\varPhi _{\infty }(u_{0},v _{0})=c_{\infty }$, i.e., $(u_{0},v_{0})$ is a solution to system (3.1). Comparing the first equation of (1.1) to that of (3.1), we have $a(x)u_{0}(x)\equiv 0$. This and the fact that $u_{0}(x)>0$ then imply $a(x)\equiv 0$. Repeating the above argument, we deduce that $b(x)\equiv 0$. This is impossible, and the proof is complete. □

Proof of Theorem 1.2

Using the same arguments carried out in the preceding section, we divide the proof into the following five steps:

Step 1. The existence of a (P.S.) sequence for Φ.

In fact, from Ekeland’s variational principle and Lemma 2.6, there exists a sequence $\{(u_{n},v_{n})\}\in N$ such that $\varPhi ^{\prime }(u _{n},v_{n})\rightarrow 0$, $\varPhi (u_{n},v_{n})\rightarrow c$.

Step 2. $\{(u_{n},v_{n})\}$ is bounded in H.

Suppose on the contrary that $s_{n}:=\|(u_{n},v_{n})\|\rightarrow \infty$. Define $(w_{n},z_{n})=\frac{(u_{n},v_{n})}{\|(u_{n},v_{n}) \|}$. Obviously, $\{(w_{n},z_{n})\}$ is bounded in H and there exists a $(w,z)\in H$ such that, passing if necessary to a subsequence,

$$\begin{aligned}& (w_{n},z_{n})\rightharpoonup (w,z)\quad \text{in } H, \\& (w_{n},z_{n})\rightarrow (w,z) \quad \text{in } L_{\mathrm{loc}}^{2}\bigl( \mathbb{R}^{3} \bigr)*L_{\mathrm{loc}}^{2}\bigl(\mathbb{R}^{3}\bigr), \\& w_{n}\rightarrow w, \qquad z_{n}\rightarrow z \quad \text{a.e. in } \mathbb{R}^{3}. \end{aligned}$$

Clearly, the sequence $(w_{n},z_{n})$ is bounded in $L^{p}( \mathbb{R}^{3})*L^{p}( \mathbb{R}^{3})$ by Sobolev embedding theorem, i.e., $\{|w_{n}|_{p}+|z_{n}|_{p}\}$ is a bounded sequence in $\mathbb{R,}$ and therefore we can assume $|w_{n}|_{p}+|z_{n}|_{p} \rightarrow \tilde{A}\in [0,\infty )$. Arguing as in the proof of Theorem 1.1 shows that $\tilde{A}=0$ cannot happen. Hence, we assume straight away that $\tilde{A}\neq 0$. Without loss of generality, we may assume that $w_{n}\nrightarrow 0$ in $L^{p}(\mathbb{R}^{3})$. From Lemma 2.2, there exist a $\delta >0$ and a sequence $\{x_{n}\}\subset \mathbb{R}^{3}$ such that

$$ \int _{B(x_{n},1)} \vert w_{n} \vert ^{2}\, dx>\delta . $$

(4.2)

In the sequel, we distinguish the following two cases:

Case A. $\{x_{n}\}$ is bounded in $\mathbb{R}^{3}$.

Since $w_{n}\rightarrow w$ in $L_{\mathrm{loc}}^{2}(\mathbb{R}^{3})$, this implies $w\neq 0$. Thus $(w,z)\neq (0,0)$. And consequently from Lemma 2.4 (2), we get $\varPhi _{\infty }(s_{n}w_{n},s_{n}z_{n})\rightarrow -\infty$, which is a contradiction to the fact that $\varPhi _{\infty }(s _{n}w_{n},s_{n}z_{n})=\varPhi _{\infty }(u_{n},v_{n})\geq 0$.

Case B. $\{x_{n}\}$ is unbounded in $\mathbb{R}^{3}$.

If $\{x_{n}\}$ is unbounded, then, without loss of generality, we may assume $|x_{n}|\rightarrow \infty$. Let $\tilde{w}_{n}(x)=w_{n}(x+x _{n})$, $\tilde{z}_{n}(x)=z_{n}(x+x_{n})$. Up to a subsequence, there exists a $(\tilde{w},\tilde{z})\in H$ such that $(\tilde{w}_{n}, \tilde{z}_{n})\rightharpoonup (\tilde{w},\tilde{z})$ in H, $(\tilde{w}_{n},\tilde{z}_{n})\rightarrow (\tilde{w},\tilde{z})$ in $L_{\mathrm{loc}}^{2}(\mathbb{R}^{3})*L_{\mathrm{loc}}^{2}(\mathbb{R} ^{3})$, $\tilde{w}_{n}\rightarrow \tilde{w}$, $\tilde{z}_{n}\rightarrow \tilde{z}$ for a.e. $x\in \mathbb{R}^{3}$. It follows from (4.2) that

$$ \int _{B(0,1)} \vert \tilde{w}_{n} \vert ^{2}\, dx>\delta . $$

Since $\tilde{w}_{n}\rightarrow \tilde{w}$ in $L_{\mathrm{loc}}^{2}( \mathbb{R}^{3})$, this implies $\tilde{w}\neq 0$. Therefore $(\tilde{w},\tilde{z})\neq (0,0)$. And consequently from Lemma 2.4 (2), we obtain

$$\begin{aligned}& \int _{\mathbb{R}^{3}}\frac{F(s_{n}\tilde{w}_{n},s_{n}\tilde{z}_{n})}{s _{n}^{2}}\, dx=\infty , \\& \begin{aligned} 0 \leq {}&\frac{\varPhi (u_{n},v_{n})}{ \Vert (u_{n},v_{n}) \Vert ^{2}} \\ ={}& \frac{1}{2} \biggl[\frac{ \Vert u_{n} \Vert ^{2}+ \Vert v_{n} \Vert ^{2}+\int _{\mathbb{R} ^{3}}a(x)u_{n}^{2}\, dx+\int _{\mathbb{R}^{3}}b(x)v_{n}^{2}\, dx-2 \lambda \int _{\mathbb{R}^{3}} u_{n}v_{n}\, dx}{ \Vert (u_{n},v_{n}) \Vert ^{2}} \biggr] \\ & {} - \int _{\mathbb{R}^{3}}\frac{F(s_{n}w_{n},s_{n}z_{n})}{s_{n}^{2}}\, dx \\ \leq {}&\frac{\eta }{2}- \int _{\mathbb{R}^{3}}\frac{F(s_{n}\tilde{w} _{n},s_{n}\tilde{z}_{n})}{s_{n}^{2}}\, dx \\ \rightarrow{}& {-} \infty , \end{aligned} \end{aligned}$$

which is a contradiction. So $\{(u_{n},v_{n})\}$ is bounded in H.

Step 3. $\varPhi ^{\prime }(\hat{u},\hat{v})=0$.

From Step 2, $\{(u_{n},v_{n})\}$ is bounded in H. Hence, up to a subsequence, there exists a $(\hat{u},\hat{v})\in H$ such that $(u_{n},v_{n})\rightharpoonup (\hat{u},\hat{v})$ in H, $(u_{n},v _{n})\rightarrow (\hat{u},\hat{v})$ in $L_{\mathrm{loc}}^{2}( \mathbb{R}^{3})*L_{\mathrm{loc}}^{2}(\mathbb{R}^{3})$, $u_{n}\rightarrow \hat{u}$, $v_{n}\rightarrow \hat{v}$ for a.e. $x\in \mathbb{R}^{3}$. From Lemma 2.5 (5), we see that $\varPhi ^{\prime }(\hat{u},\hat{v})=0$.

Step 4. $(\hat{u},\hat{v})\neq (0,0)$.

By Sobolev embedding theorem, it is clear that the sequence $\{(u_{n},v_{n})\}$ is bounded in $L^{p}( \mathbb{R}^{3})*L^{p}( \mathbb{R}^{3})$, i.e., $\{|u_{n}|_{p}+|v_{n}|_{p}\}$ is a bounded sequence in $\mathbb{R}$. Selecting a subsequence if necessary, we can assume $|u_{n}|_{p}+|v_{n}|_{p}\rightarrow \tilde{B}\in [0,\infty )$. Arguing as in the proof of Theorem 1.1 shows that $\tilde{B}=0$ cannot happen. Hence, we assume straight away that $\tilde{B}\neq 0$, Without loss of generality, we may assume that $u_{n}\nrightarrow 0$ in $L^{p}(\mathbb{R}^{3})$. From Lemma 2.2, there exist $y_{n}\in \mathbb{R}^{3}$ and $\delta >0$ such that

$$ \int _{B(y_{n},1)} \vert u_{n} \vert ^{2}\, dx>\delta . $$

(4.3)

We first prove that $\{y_{n}\}$ is bounded in $\mathbb{R}^{3}$. Indeed, supposing on the contrary, we would have $|y_{n}|\rightarrow \infty$. Let $\bar{u}_{n}(x)=u_{n}(x+x_{n})$, $\bar{v}_{n}(x)=v_{n}(x+x_{n})$. Up to a subsequence, there exists a $(\bar{u},\bar{v})\in H$ such that $(\bar{u}_{n},\bar{v}_{n})\rightharpoonup (\bar{u},\bar{v})$ in H, $(\bar{u}_{n},\bar{v}_{n})\rightarrow (\bar{u},\bar{v})$ in $L_{\mathrm{loc}}^{2}(\mathbb{R}^{3})*L_{\mathrm{loc}}^{2}(\mathbb{R} ^{3})$, $\bar{u}_{n}\rightarrow \bar{u}$, $\bar{v}_{n}\rightarrow \bar{v}$ for a.e. $x\in \mathbb{R}^{3}$. It follows from (4.3) that

$$ \int _{B(0,1)} \vert \bar{u}_{n} \vert ^{2}\, dx>\delta . $$

We pass to the limit in the above inequality and get $\bar{u}\neq 0$. Note that $\varPhi ^{\prime }(u_{n},v_{n})\rightarrow 0$ and $|y_{n}| \rightarrow \infty$, so we obtain that $\varPhi _{\infty }^{\prime }( \bar{u}_{n},\bar{v}_{n})\rightharpoonup 0$. Whence, it follows from Lemma 3.1 (5) that $\varPhi _{\infty }^{\prime }(\bar{u},\bar{v})= 0$, that is, $(\bar{u},\bar{v})\in N_{\infty }$. From Remark 3 and Fatou’s lemma, one has

$$ \begin{aligned} c+o(1)=\varPhi (u_{n},v_{n}) &= \int _{\mathbb{R}^{3}} \biggl[\frac{1}{2} \nabla F(u_{n},v_{n}) (u_{n},v_{n})-F(u_{n},v_{n}) \biggr]\,dx \\ &= \int _{\mathbb{R}^{3}} \biggl[\frac{1}{2}\nabla F( \bar{u}_{n},\bar{v} _{n}) (\bar{u}_{n}, \bar{v}_{n})-F(\bar{u}_{n},\bar{v}_{n}) \biggr]\,dx \\ & \geq \int _{\mathbb{R}^{3}} \biggl[\frac{1}{2}\nabla F(\bar{u}, \bar{v}) ( \bar{u},\bar{v})-F(\bar{u},\bar{v}) \biggr]\,dx+o(1) \\ &=\varPhi _{\infty }( \bar{u},\bar{v})+o(1) \\ &\geq c_{\infty }+o(1). \end{aligned} $$

Thus, we have $c\geq c_{\infty }$, which contradicts the fact that $c< c_{\infty }$. So $\{y_{n}\}$ is bounded in $\mathbb{R}^{3}$.

From the boundedness of $\{y_{n}\}$ and (3.2), there exists a positive constant $M>0$ such that

$$ \begin{aligned} \int _{B(0,M+1)} \vert u_{n} \vert ^{2}\, dx>\delta . \end{aligned} $$

This, together with $u_{n}\rightarrow \hat{u}$ in $L_{\mathrm{loc}} ^{2}(\mathbb{R}^{3})$, implies $\hat{u}\neq 0$, which means that $(\hat{u},\hat{v})\neq (0,0)$.

Step 5. $\varPhi (\hat{u},\hat{v})= c$.

From $\varPhi ^{\prime }(\hat{u},\hat{v})=0$ and $(\hat{u},\hat{v})\neq (0,0)$, we have $(\hat{u},\hat{v})\in N$. Thus $\varPhi (\hat{u},\hat{v})\geq c$. By using Fatou’s lemma, it is easy to check that

$$ \begin{aligned} c+o(1)=\varPhi (u_{n},v_{n}) &= \int _{\mathbb{R}^{3}} \biggl[\frac{1}{2} \nabla F(u_{n},v_{n}) (u_{n},v_{n})-F(u_{n},v_{n}) \biggr]\, dx \\ & \geq \int _{\mathbb{R}^{3}} \biggl[\frac{1}{2}\nabla F(\hat{u}, \hat{v}) ( \hat{u},\hat{v})-F(\hat{u},\hat{v}) \biggr]\, dx+o(1) \\ &=\varPhi (\hat{u}, \hat{v})+o(1). \end{aligned} $$

Hence $\varPhi (\hat{u},\hat{v})= c$, which means $(\hat{u},\hat{v})$ is a ground state solution to system (1.1).

By using the same method as in Theorem 1.1, we can prove that system (1.1) admits a positive ground state solution. The proof of Theorem 1.2 is completed. □