On symmetric solutions of a critical semilinear elliptic system involving the Caffarelli-Kohn-Nirenberg inequality in $\mathbb{R}^{N}$

Abstract

This paper deals with the existence and multiplicity of symmetric solutions for a weighted semilinear elliptic system with multiple critical Hardy-Sobolev exponents and singular potentials in $\mathbb{R}^{N}$. Applying the symmetric criticality principle and Caffarelli-Kohn-Nirenberg inequality, we establish several existence and multiplicity results of G-symmetric solutions under certain appropriate hypotheses on the parameters and the weighted functions.

1 Introduction

The present paper is devoted to the study of the following weighted semilinear elliptic system:

$$ \textstyle\begin{cases} L_{a, \mu} u=\frac{K(x)}{\vert x\vert ^{bp_{b}^{\ast}}} (\vert u\vert ^{p_{b}^{\ast}-2}u+\sum_{i=1}^{l} \frac{\varsigma_{i}\alpha_{i}}{p_{b}^{\ast}} \vert u\vert ^{\alpha_{i}-2}u\vert v\vert ^{\beta_{i}} )+\sigma h(x) \frac{\vert u\vert ^{q-2}u}{\vert x\vert ^{dp_{d}^{\ast}}}, &\text{in } \mathbb{R}^{N},\\ L_{a, \mu} v=\frac{K(x)}{\vert x\vert ^{bp_{b}^{\ast}}} (\vert v\vert ^{p_{b}^{\ast}-2}v+\sum_{i=1}^{l} \frac{\varsigma_{i}\beta_{i}}{p_{b}^{\ast}} \vert u\vert ^{\alpha_{i}}\vert v\vert ^{\beta_{i}-2}v )+\sigma h(x) \frac{\vert v\vert ^{q-2}v}{\vert x\vert ^{dp_{d}^{\ast}}}, &\text{in } \mathbb{R}^{N},\\ u(x), v(x)\rightarrow0, &\text{as } \vert x\vert \rightarrow +\infty, \end{cases} $$

(1.1)

where $L_{a, \mu}\triangleq-\operatorname{div}(\vert x\vert ^{-2a}\nabla\cdot) -\mu\frac{\cdot}{ \vert x\vert ^{2(1+a)}}$ is a singular elliptic operator, $N\geq3$, $0\leq a<\frac{N-2}{2}$, $a\leq b\leq d<1+a$, $\sigma\geq0$, $0\leq\mu<\overline{\mu}$ with $\overline{\mu}\triangleq(\frac{N-2-2a}{2})^{2}$, $0< \varsigma_{i}<+\infty$, and $\alpha_{i}$, $\beta_{i}>1$ satisfy $\alpha_{i}+\beta_{i}=p_{b}^{\ast}$ ($i=1, \ldots, l$; $1\leq l\in \mathbb{N}$), $1< q<2$, $p_{b}^{\ast}\triangleq\frac{2N}{N-2(1+a-b)}$, and $p_{d}^{\ast}\triangleq\frac{2N}{N-2(1+a-d)}$ are the critical Hardy-Sobolev exponents, and $p_{a}^{\ast}=2^{\ast}\triangleq\frac{2N}{N-2}$ is the critical Sobolev exponent; K and h are G-symmetric functions (see Section 2 for details) satisfying some appropriate conditions which will be specified later.

The critical growth in elliptic problems has been extensively studied in the last decades, starting with the seminal paper [1]. Limiting ourselves to problems involving the singular potentials and critical exponents, we would like to mention the works [2–6] and the references therein contained. These equations involving singular nonlinearities, as well as the singular elliptic systems, describe naturally several physical phenomena and applied economical models (see [7] for example). Recently, Deng and Jin [8] investigated the existence of nontrivial solutions of the following critical singular problem:

$$ -\Delta u-\mu\frac{u}{\vert x\vert ^{2}}=K(x)\vert x\vert ^{-s}u^{2^{\ast}(s)-1}, \quad\text{and} \quad u>0 \text{ in } \mathbb{R}^{N}, $$

(1.2)

where $0\leq s<2$ and $0<\mu<(\frac{N-2}{2})^{2}$ are parameters, $N> 2$, $2^{\ast}(s)=\frac{2(N-s)}{N-2}$, and K satisfies some symmetry conditions with respect to a subgroup G of $O(\mathbb{N})$. By means of the variational approach, the authors proved the existence and multiplicity of G-symmetric solutions to (1.2) under certain hypotheses on K. Very recently, Deng and Huang [9] extended the results in [8] to quasilinear singular elliptic problems in a bounded G-symmetric domain. Moreover, we also mention that when $\mu=s=0$ and the right-hand side term $\vert x\vert ^{-s}u^{2^{\ast}(s)-1}$ is replaced by $u^{q-1}$ ($1< q<\frac{2N}{N-2}$ or $q=\frac{2N}{N-2}$) in (1.2), many elegant results of G-symmetric solutions of (1.2) were established in [10–12].

On the other hand, in recent years, more and more attention have been paid to the existence and multiplicity of nontrivial solutions for singular elliptic systems. In a recent paper, Huang and Kang [13] considered the following critical semilinear elliptic systems:

$$ \textstyle\begin{cases}-\Delta u-\mu_{1}\frac{u}{\vert x-a_{1}\vert ^{2}}=\vert u\vert ^{2^{\ast}-2}u +\frac{\varsigma\alpha}{\alpha+\beta} \vert u\vert ^{\alpha-2}u\vert v\vert ^{\beta} +\lambda_{1}\vert u\vert ^{q_{1}-2}u, &\text{in } \Omega,\\ -\Delta v-\mu_{2}\frac{v}{\vert x-a_{2}\vert ^{2}}=\vert v\vert ^{2^{\ast}-2}v +\frac{\varsigma\beta}{\alpha+\beta} \vert u\vert ^{\alpha} \vert v\vert ^{\beta-2}v +\lambda_{2}\vert v\vert ^{q_{2}-2}v, &\text{in } \Omega,\\ u=v=0, &\text{on } \partial\Omega, \end{cases} $$

(1.3)

where $\Omega\subset\mathbb{R}^{N}$ ($N\geq3$) is a smooth bounded domain containing the origin, $\varsigma>0$, $a_{i}\in\Omega$, $\lambda_{i}>0$, $\mu_{i}<(\frac{N-2}{2})^{2}$, $2\leq q_{i}<2^{\ast}$ ($i=1, 2$), and $\alpha, \beta>1$ fulfill $\alpha+\beta=2^{\ast}$. Note that $\vert u\vert ^{\alpha-2}u\vert v\vert ^{\beta}$ and $\vert u\vert ^{\alpha} \vert v\vert ^{\beta-2}v$ in (1.3) are called strongly coupled terms, and $\vert u\vert ^{2^{\ast}-2}u$, $\vert v\vert ^{2^{\ast}-2}v$ are weakly coupled terms. By variational methods and the Moser iteration techniques, the authors proved the existence of positive solutions and some properties of the nontrivial solutions to (1.3). Subsequently, by employing variational methods and the analytic techniques of Nehari manifold, Kang [14], Nyamoradi [15], Nyamoradi and Hsu [16] extended and generalized the results of [13] to the critical singular quasilinear systems. These results give us a good insight into the corresponding problems. Other further results relating to the nonlinear elliptic systems can be found in [17–20] and the references therein.

However, concerning the existence and multiplicity of G-symmetric solutions for elliptic systems, we only find some results for singular elliptic systems in [21–23] and when $G=O(\mathbb{N})$, several radial and nonradial results for nonsingular elliptic systems in [24]. Motivated by the works [8, 10, 13], in the present paper, we study the existence and multiplicity of G-symmetric solutions for singular elliptic system (1.1) in $\mathbb{R}^{N}$. The main difficulties lie in the fact that there are not only the singular perturbations $h(x)\vert x\vert ^{-dp_{d}^{\ast}} \vert u\vert ^{q-2}u$ and $h(x)\vert x\vert ^{-dp_{d}^{\ast}} \vert v\vert ^{q-2}v$, but also the nonlinear strong coupled terms $\sum_{i=1}^{l} \frac{\varsigma_{i}\alpha_{i}}{p_{b}^{\ast}} \vert u\vert ^{\alpha_{i}-2}u\vert v\vert ^{\beta_{i}}$, $\sum_{i=1}^{l} \frac{\varsigma_{i}\beta_{i}}{p_{b}^{\ast}} \vert u\vert ^{\alpha_{i}}\vert v\vert ^{\beta_{i}-2}v$ and weak coupled terms $\vert u\vert ^{p_{b}^{\ast}-2}u$, $\vert v\vert ^{p_{b}^{\ast}-2}v$. Compared with problems (1.2) and (1.3), the singular elliptic system (1.1) becomes more complicated to deal with and therefore we have to overcome more difficulties. So far as we know, it seems that there are few results for (1.1) even in the particular cases $\mu=a=b=0$, $\sigma=0$, and $\varsigma_{i}>0$ ($i=1,\ldots, l$). Consequently, it make sense for us to investigate system (1.1) thoroughly. Let $K_{0}>0$ be a constant. Note that here we try to consider both the cases of $\sigma=0$, $K(x)\not\equiv K_{0}$, and $\sigma>0$, $K(x)\equiv K_{0}$.

The remainder of this paper is organized as follows. Some preliminaries and the variational setting and the main results of this paper are presented in Section 2. The proofs of several existence and multiplicity results for the cases $\sigma=0$ and $K(x)\not\equiv K_{0}$ in (1.1) are given in Section 3, while the proofs of multiplicity results for the cases $\sigma>0$ and $K(x)\equiv K_{0}$ are detailed in Section 4. The methods of this paper are mainly based upon the symmetric criticality principle of Palais (see [25]) and variational arguments.

2 Preliminaries and main results

Let $\mathscr{D}_{a}^{1,2}(\mathbb{R}^{N})$ denote the closure of $\mathscr{C}_{0}^{\infty}(\mathbb{R}^{N})$ functions with respect to the norm $(\int_{\mathbb{R}^{N}}\vert x\vert ^{-2a}\vert \nabla u\vert ^{2}\,dx)^{1/2}$. We recall that the well-known Caffarelli-Kohn-Nirenberg inequality (see [26]) asserts that, for all $u\in\mathscr{D}_{a}^{1,2}(\mathbb{R}^{N})$, there is a constant $C=C(N, a, e)>0$ such that

$$ \biggl( \int_{\mathbb{R}^{N}}\vert x\vert ^{-ep_{e}^{\ast}} \vert u\vert ^{p_{e}^{\ast}}\,dx \biggr) ^{\frac{2}{p_{e}^{\ast}}}\leq C \int_{\mathbb{R}^{N}}\vert x\vert ^{-2a}\vert \nabla u \vert ^{2}\,dx, $$

(2.1)

where $-\infty< a<\frac{N-2}{2}$, $a\leq e\leq1+a$, and $p_{e}^{\ast}=\frac{2N}{N-2(1+a-e)}$. If $e=1+a$, then $p_{e}^{\ast}=2$ and we have the following weighted Hardy inequality (see [2]):

$$ \int_{\mathbb{R}^{N}}\vert x\vert ^{-2(1+a)}\vert u\vert ^{2}\,dx \leq \frac{1}{\overline{\mu}} \int_{\mathbb{R}^{N}}\vert x\vert ^{-2a}\vert \nabla u\vert ^{2}\,dx,\quad \forall u\in\mathscr {D}_{a}^{1,2} \bigl(\mathbb{R}^{N}\bigr), $$

(2.2)

where $\overline{\mu}=(\frac{N-2-2a}{2})^{2}$. Now we employ the following norm in $\mathscr{D}_{a}^{1, 2}(\mathbb{R}^{N})$:

$$\Vert u\Vert _{\mu}\triangleq \biggl[ \int_{\mathbb{R}^{N}} \bigl(\vert x\vert ^{-2a}\vert \nabla u \vert ^{2}-\mu\vert x\vert ^{-2(1+a)}u^{2} \bigr)\,dx \biggr]^{\frac{1}{2}},\quad 0\leq\mu< \overline{\mu}. $$

By the inequality (2.2), we easily see that the above norm is equivalent to the usual norm $(\int_{\mathbb{R}^{N}}\vert x\vert ^{-2a}\vert \nabla u\vert ^{2}\,dx)^{1/2}$. Moreover, we define the product space $(\mathscr{D}_{a}^{1, 2}(\mathbb{R}^{N}))^{2}$ endowed with the norm

$$\bigl\Vert (u, v)\bigr\Vert _{\mu}= \bigl(\Vert u\Vert _{\mu}^{2}+\Vert v\Vert _{\mu}^{2} \bigr)^{1/2},\quad \forall(u, v)\in \bigl(\mathscr{D}_{a}^{1, 2} \bigl(\mathbb{R}^{N}\bigr) \bigr)^{2}. $$

Let $O(\mathbb{N})$ be the group of orthogonal linear transformations in $\mathbb{R}^{N}$ with natural action and let $G\subset O(\mathbb{N})$ be a closed subgroup. For $x\neq0$ we denote the cardinality of $G_{x}=\{gx; g\in G\}$ by $\vert G_{x}\vert$ and set $\vert G\vert =\inf_{0\neq x\in\mathbb{R}^{N}}\vert G_{x}\vert$. Note that here $\vert G\vert$ may be +∞. For any function $f: \mathbb{R}^{N}\rightarrow\mathbb{R}$, we call $f(x)$ a G-symmetric function if for all $g\in G$ and $x\in \mathbb{R}^{N}$, $f(gx)=f(x)$ holds. In particular, if f is radially symmetric, then the corresponding group G is $O(\mathbb{N})$ and $\vert G\vert =+\infty$. Other further examples of G-symmetric functions can be found in [8].

The natural functional space to frame the analysis of (1.1) by variational techniques is the Hilbert space $(\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$, which is the subspace of $(\mathscr{D}_{a}^{1, 2}(\mathbb{R}^{N}))^{2}$ consisting of all G-symmetric functions. In the present paper, we are concerned with the following problems:

$$\bigl(\mathscr{P}^{K}_{\sigma}\bigr) \quad\textstyle\begin{cases} L_{a, \mu} u=\frac{K(x)}{\vert x\vert ^{bp_{b}^{\ast}}} (\vert u\vert ^{p_{b}^{\ast}-2}u+\sum_{i=1}^{l} \frac{\varsigma_{i}\alpha_{i}}{p_{b}^{\ast}} \vert u\vert ^{\alpha_{i}-2}u\vert v\vert ^{\beta_{i}} )+\sigma h(x) \frac{\vert u\vert ^{q-2}u}{\vert x\vert ^{dp_{d}^{\ast}}}, &\text{in } \mathbb{R}^{N},\\ L_{a, \mu} v=\frac{K(x)}{\vert x\vert ^{bp_{b}^{\ast}}} (\vert v\vert ^{p_{b}^{\ast}-2}v+\sum_{i=1}^{l} \frac{\varsigma_{i}\beta_{i}}{p_{b}^{\ast}} \vert u\vert ^{\alpha_{i}}\vert v\vert ^{\beta_{i}-2}v )+\sigma h(x) \frac{\vert v\vert ^{q-2}v}{\vert x\vert ^{dp_{d}^{\ast}}}, &\text{in } \mathbb{R}^{N},\\ (u, v)\in (\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}) )^{2}, \quad\text{and}\quad u\geq0, v\geq0, (u, v)\neq(0, 0), &\text{in } \mathbb{R}^{N}. \end{cases} $$

Let $(u, v)\in (\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}) )^{2}$ be a weak solution of $(\mathscr{P}^{K}_{\sigma})$ (see Sections 3 and 4). We need the following basic definition.

Definition 2.1

• (i) A weak solution $(u, v)\in (\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}) )^{2}$ is called a semi-positive solution to $(\mathscr{P}^{K}_{\sigma})$ if $u>0$, $v=0$ or $u=0$, $v>0$ on $\mathbb{R}^{N}$.

• (ii) A weak solution $(u, v)\in (\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}) )^{2}$ is called a strictly positive solution to $(\mathscr{P}^{K}_{\sigma})$ if $u>0$ and $v>0$ on $\mathbb{R}^{N}$.

• (iii) A weak solution $(u, v)\in (\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}) )^{2}$ is called a positive solution to $(\mathscr{P}^{K}_{\sigma})$ if $u\geq0$, $v\geq0$ and $(u, v)\neq(0, 0)$ on $\mathbb{R}^{N}$.

The purpose of this paper is to investigate the existence and multiplicity of the positive solutions (including semi-positive solutions and strictly positive solutions) to the problem $(\mathscr{P}^{K}_{\sigma})$.

Before stating our main results, we present the following two notations: $S_{\mu}$ and $y_{\epsilon}(x)$, which are, respectively, defined by

$$ S_{\mu}\triangleq\inf_{u\in\mathscr{D}_{a}^{1, 2}(\mathbb{R}^{N})\backslash\{0\}} \frac{\int_{\mathbb{R}^{N}} (\vert x\vert ^{-2a}\vert \nabla u\vert ^{2}-\mu\vert x\vert ^{-2(1+a)}u^{2} )\,dx}{ (\int_{\mathbb{R}^{N}}\vert x\vert ^{-bp_{b}^{\ast}} \vert u\vert ^{p_{b}^{\ast}}\,dx ) ^{\frac{2}{p_{b}^{\ast}}}} $$

(2.3)

and

$$ y_{\epsilon}(x)\triangleq \frac{C\epsilon^{\frac{1}{p_{b}^{\ast}-2}}}{ \vert x\vert ^{\sqrt{\overline{\mu}}-\sqrt{\overline{\mu}-\mu}} (\epsilon +\vert x\vert ^{(p_{b}^{\ast}-2)\sqrt{\overline{\mu}-\mu}} )^{\frac {2}{p_{b}^{\ast}-2}}}, $$

(2.4)

where $\epsilon>0$, and the constant $C=C(N, a, b, \mu)>0$, depending only on N, a, b, and μ. According to [2], we find that $y_{\epsilon}(x)$ satisfies the equations

$$ \int_{\mathbb{R}^{N}} \bigl(\vert x\vert ^{-2a}\vert \nabla y_{\epsilon}\vert ^{2}-\mu\vert x\vert ^{-2(1+a)}y_{\epsilon}^{2} \bigr)\,dx=1 $$

(2.5)

and

$$\int_{\mathbb{R}^{N}}\vert x\vert ^{-bp_{b}^{\ast}}y_{\epsilon}^{p_{b}^{\ast }-1} \varphi \,dx=S_{\mu}^{-\frac{p_{b}^{\ast}}{2}} \int_{\mathbb{R}^{N}} \bigl(\vert x\vert ^{-2a}\nabla y_{\epsilon}\nabla\varphi-\mu \vert x\vert ^{-2(1+a)}y_{\epsilon} \varphi \bigr)\,dx $$

for all $\varphi\in\mathscr{D}_{a}^{1, 2}(\mathbb{R}^{N})$. In particular, we have (let $\varphi=y_{\epsilon}$)

$$ \int_{\mathbb{R}^{N}}\vert x\vert ^{-bp_{b}^{\ast}}y_{\epsilon}^{p_{b}^{\ast}} \,dx=S_{\mu}^{-\frac{p_{b}^{\ast}}{2}}=S_{\mu}^{\frac{-N}{N-2(1+a-b)}}. $$

(2.6)

We suppose that the functions $K(x)$ and $h(x)$ verify the following hypotheses:

• (k.1) $K(x)$ is G-symmetric.

• (k.2) $K(x)\in\mathscr{C}(\mathbb{R}^{N})\cap L^{\infty}(\mathbb{R}^{N})$, and $K_{+}(x)\not\equiv0$, where $K_{+}(x)=\max\{0, K(x)\}$.

• (h.1) $h(x)$ is G-symmetric.

• (h.2) $h(x)$ is a nonnegative function in $\mathbb{R}^{N}$ such that

  $$0< \Vert h\Vert _{\theta}\triangleq \biggl( \int_{\mathbb{R}^{N}}\vert x\vert ^{-dp_{d}^{\ast}} h^{\theta}(x) \,dx \biggr)^{\frac{1}{\theta}}< +\infty\quad \text{with } \theta\triangleq \frac{p_{d}^{\ast}}{p_{d}^{\ast}-q}. $$

The main results of this paper are the following.

Theorem 2.1

Suppose that (k.1) and (k.2) hold. If

$$ \int_{\mathbb{R}^{N}}K(x)\frac{y_{\epsilon}^{p_{b}^{\ast }}}{\vert x\vert ^{bp_{b}^{\ast}}}\,dx\geq \max \biggl\{ \frac{\Vert K_{+}\Vert _{\infty}}{ \vert G\vert ^{\frac{2(1+a-b)}{N-2(1+a-b)}} S_{0}^{\frac{N}{N-2(1+a-b)}}}, \frac{K_{+}(0)}{S_{\mu}^{\frac{N}{N-2(1+a-b)}}}, \frac{K_{+}(\infty)}{S_{\mu}^{\frac{N}{N-2(1+a-b)}}} \biggr\} >0 $$

(2.7)

for some $\epsilon>0$, where $K_{+}(\infty)=\limsup_{\vert x\vert \rightarrow\infty}K_{+}(x)$, then problem $(\mathscr{P}^{K}_{0})$ has at least one positive solution in $(\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$.

Remark 2.1

Under the condition (k.2), we only assume that $K(x)$ is bounded and continuous on $\mathbb{R}^{N}$, and $K_{+}(x)\not\equiv0$, where $K_{+}(x)=\max\{0, K(x)\}$. In particular, we do not require any continuity of $K(x)$ at infinity. Moreover, we also do not require $K_{+}(0)=K(0)$, where $K_{+}(0)=\max\{0, K(0)\}$. For example, setting $K(x)=\sin (\vert x\vert ^{2}-1)$, $x\in\mathbb{R}^{N}$, we easily check that $K(x)\in\mathscr{C}(\mathbb{R}^{N})\cap L^{\infty}(\mathbb{R}^{N})$ and $K_{+}(x)=\max\{0, \sin (\vert x\vert ^{2}-1)\}\not\equiv0$ on $\mathbb{R}^{N}$, but $K(0)=-\sin 1<0$ and $K_{+}(0)=0$.

Corollary 2.1

Suppose that (k.1) and (k.2) hold. Then we have the following statements.

• (1) Problem $(\mathscr{P}^{K}_{0})$ has a positive solution if

  $$ K(0)>0,\qquad K(0)\geq\max \biggl\{ \frac{\Vert K_{+}\Vert _{\infty}}{ \vert G\vert ^{\frac{2(1+a-b)}{N-2(1+a-b)}}} \biggl( \frac{S_{\mu}}{S_{0}} \biggr)^{\frac{N}{N-2(1+a-b)}}, K_{+}(\infty) \biggr\} , $$

  (2.8)

  and either (i) $K(x)\geq K(0)+\Lambda_{0} \vert x\vert ^{p_{b}^{\ast}\sqrt{\overline{\mu}-\mu}}$ for some $\Lambda_{0}>0$ and $\vert x\vert$ small or (ii) $\vert K(x)-K(0)\vert \leq \Lambda_{1}\vert x\vert ^{\kappa}$ for some constant $\Lambda_{1}>0$, $\kappa>p_{b}^{\ast}\sqrt{\overline{\mu}-\mu}$ and $\vert x\vert$ small and

  $$ \int_{\mathbb{R}^{N}} \bigl(K(x)-K(0) \bigr) \vert x\vert ^{-(b+\sqrt{\overline{\mu}}+\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}}\,dx>0. $$

  (2.9)

• (2) Problem $(\mathscr{P}^{K}_{0})$ admits at least one positive solution if $\lim_{\vert x\vert \rightarrow\infty}K(x)=K(\infty)$ exists and is positive,

  $$ K(\infty)\geq\max \biggl\{ \frac{\Vert K_{+}\Vert _{\infty}}{ \vert G\vert ^{\frac{2(1+a-b)}{N-2(1+a-b)}}} \biggl( \frac{S_{\mu}}{S_{0}} \biggr)^{\frac{N}{N-2(1+a-b)}}, K_{+}(0) \biggr\} , $$

  (2.10)

  and either (i) $K(x)\geq K(\infty)+\Lambda_{2} \vert x\vert ^{-p_{b}^{\ast}\sqrt{\overline{\mu}-\mu}}$ for some $\Lambda_{2}>0$ and large $\vert x\vert$ or (ii) $\vert K(x)-K(\infty)\vert \leq\Lambda_{3}\vert x\vert ^{-\iota}$ for some constants $\Lambda_{3}>0$, $\iota>p_{b}^{\ast}\sqrt{\overline{\mu}-\mu}$, and large $\vert x\vert$ and

  $$ \int_{\mathbb{R}^{N}} \bigl(K(x)-K(\infty) \bigr) \vert x\vert ^{-(b+\sqrt{\overline{\mu}}-\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}}\,dx>0. $$

  (2.11)

• (3) If $K(x)\geq K(\infty)=K(0)>0$ on $\mathbb{R}^{N}$ and

  $$K(\infty)=K(0)\geq \Vert K_{+}\Vert _{\infty} \vert G \vert ^{\frac{-2(1+a-b)}{N-2(1+a-b)}} (S_{\mu}/S_{0} )^{\frac{N}{N-2(1+a-b)}}, $$

  then problem $(\mathscr{P}^{K}_{0})$ has at least one positive solution.

Theorem 2.2

Suppose that $K_{+}(0)=K_{+}(\infty)=0$ and $\vert G\vert =+\infty$. Then problem $(\mathscr{P}^{K}_{0})$ has infinitely many G-symmetric solutions.

Corollary 2.2

If $K(x)$ is a radially symmetric function such that $K_{+}(0)=K_{+}(\infty)=0$, then problem $(\mathscr{P}^{K}_{0})$ has infinitely many solutions which are radially symmetric.

Theorem 2.3

Let $K_{0}>0$ be a constant. Suppose that $K(x)\equiv K_{0}$ and (h.1), (h.2) hold. Then there exists $\sigma^{\ast}>0$ such that, for any $\sigma\in(0, \sigma^{\ast})$, problem $(\mathscr{P}^{K_{0}}_{\sigma})$ possesses at least two positive solutions in $(\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$.

Remark 2.2

The main results of this paper extend and complement some results of [8–10, 21]. Even in the particular cases $\mu=a=b=0$, $\sigma=0$, and $\varsigma_{i}>0$ ($i=1,\ldots, l$), the above results to problem $(\mathscr{P}^{K}_{0})$ are new in the whole space $\mathbb{R}^{N}$.

Throughout this paper, the ball of center x and radius r is denoted by $B_{r}(x)$. We denote by $(\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$ the subspace of $(\mathscr{D}_{a}^{1, 2}(\mathbb{R}^{N}))^{2}$ consisting of all G-symmetric functions. The dual space of $(\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$ ($(\mathscr{D}_{a}^{1, 2}(\mathbb{R}^{N}))^{2}$, resp.) is denoted by $(\mathscr{D}_{a, G}^{-1, 2}(\mathbb{R}^{N}))^{2}$ ($(\mathscr{D}_{a}^{-1, 2}(\mathbb{R}^{N}))^{2}$, resp.). $O(\epsilon^{t})$ denotes the quantity satisfying $\vert O(\epsilon^{t})\vert /\epsilon^{t}\leq C$, and $o_{n}(1)$ a datum which tends to 0 as $n\rightarrow\infty$. We employ C, $C_{1}$, $C_{2}$, … to denote (possibly different) positive constants, and denote by ‘→’ convergence in norm in a given Banach space X and by ‘⇀’ weak convergence. A functional $\mathscr{F}\in\mathscr{C}^{1}(X, \mathbb{R})$ is said to satisfy the $(\mathit{PS})_{c}$ condition if each sequence $\{w_{n}\}$ in X satisfying $\mathscr{F}(w_{n})\rightarrow c$, $\mathscr{F}^{\prime}(w_{n})\rightarrow0$ in $X^{\ast}$ has a subsequence which strongly converges to some element in X. Hereafter, $L^{q}(\mathbb{R}^{N}, \vert x\vert ^{-\varsigma})$ denotes the weighted $L^{q}(\mathbb{R}^{N})$ space with the norm $(\int_{\mathbb{R}^{N}}\vert x\vert ^{-\varsigma} \vert u\vert ^{q}\,dx)^{1/q}$.

3 Existence and multiplicity results for problem $(\mathscr {P}^{K}_{0})$

The energy functional corresponding to problem $(\mathscr{P}^{K}_{0})$ is defined on $(\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$ by

$$ \mathscr{F}(u, v)=\frac{1}{2}\bigl\Vert (u, v)\bigr\Vert _{\mu}^{2} -\frac{1}{p_{b}^{\ast}} \int_{\mathbb{R}^{N}} \frac{K(x)}{\vert x\vert ^{bp_{b}^{\ast}}} \Biggl(\vert u\vert ^{p_{b}^{\ast}}+\sum_{i=1}^{l} \varsigma_{i}\vert u\vert ^{\alpha_{i}}\vert v\vert ^{\beta_{i}}+\vert v\vert ^{p_{b}^{\ast}} \Biggr)\,dx. $$

(3.1)

Then $\mathscr{F}\in\mathscr{C}^{1}((\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}, \mathbb{R})$, and it is well known that the critical points of the functional $\mathscr{F}$ on $(\mathscr {D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$ correspond to the weak solutions of problem $(\mathscr{P}^{K}_{0})$. More precisely, by the symmetric criticality principle due to Palais (see Lemma 3.1), we say that $(u, v)\in(\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$ is a weak solution of $(\mathscr{P}^{K}_{0})$, if for any $(\varphi_{1}, \varphi_{2})\in (\mathscr{D}_{a}^{1, 2}(\mathbb{R}^{N}))^{2}$, we have

$$ \begin{aligned}[b] &\int_{\mathbb{R}^{N}} \biggl(\vert x\vert ^{-2a}\nabla u \nabla\varphi_{1}+\vert x\vert ^{-2a}\nabla v\nabla \varphi_{2}-\mu\frac{u\varphi_{1} +v\varphi_{2}}{\vert x\vert ^{2(1+a)}} \biggr)\,dx- \int_{\mathbb{R}^{N}}\frac {K(x)}{\vert x\vert ^{bp_{b}^{\ast}}} \Biggl\{ \vert u\vert ^{p_{b}^{\ast}-2}u\varphi_{1} \\ &\quad{}+\sum_{i=1}^{l} \frac{\varsigma_{i}}{p_{b}^{\ast}} \bigl( \alpha_{i}\vert u\vert ^{\alpha_{i}-2}u\vert v\vert ^{\beta_{i}}\varphi_{1} +\beta_{i}\vert u\vert ^{\alpha_{i}}\vert v\vert ^{\beta_{i}-2}v\varphi_{2} \bigr) + \vert v\vert ^{p_{b}^{\ast}-2}v\varphi_{2} \Biggr\} \,dx=0. \end{aligned} $$

(3.2)

Lemma 3.1

Let $K(x)$ be a G-symmetric function; $\mathscr{F}^{\prime}(u, v)=0$ in $(\mathscr{D}_{a, G}^{-1, 2}(\mathbb{R}^{N}))^{2}$ implies $\mathscr{F}^{\prime}(u, v)=0$ in $(\mathscr{D}_{a}^{-1, 2}(\mathbb{R}^{N}))^{2}$.

Proof

The proof is a repeat of that in [10], Lemma 1 (see also [24], Proposition 2.8) and therefore omitted here. □

For all $\mu\in[0, \overline{\mu})$, $0<\varsigma_{i}<+\infty$, $\alpha_{i}$, $\beta_{i}>1$, and $\alpha_{i}+\beta_{i}=p_{b}^{\ast}$ ($i=1, \ldots, l$), we define

$$\begin{aligned}& S_{\mu, l}\triangleq\inf_{(u, v)\in(\mathscr{D}_{a}^{1, 2}(\mathbb{R}^{N})\backslash\{0\})^{2}} \frac{\int_{\mathbb{R}^{N}} (\vert x\vert ^{-2a}\vert \nabla u\vert ^{2}+\vert x\vert ^{-2a}\vert \nabla v\vert ^{2}-\mu \frac{\vert u\vert ^{2}+\vert v\vert ^{2}}{\vert x\vert ^{2(1+a)}} )\,dx}{ [\int_{\mathbb{R}^{N}}\vert x\vert ^{-bp_{b}^{\ast}} (\vert u\vert ^{p_{b}^{\ast}} +\sum_{i=1}^{l}\varsigma_{i}\vert u\vert ^{\alpha_{i}} \vert v\vert ^{\beta_{i}}+\vert v\vert ^{p_{b}^{\ast}} )\,dx ]^{\frac{2}{p_{b}^{\ast}}}}, \end{aligned}$$

(3.3)

$$\begin{aligned}& \mathscr{B}(\tau)\triangleq\frac{1+\tau^{2}}{ (1 +\sum_{i=1}^{l}\varsigma_{i}\tau^{\beta_{i}} +\tau^{p_{b}^{\ast}} )^{\frac{2}{p_{b}^{\ast}}}},\quad \tau\geq0, \end{aligned}$$

(3.4)

$$\begin{aligned}& \mathscr{B}(\tau_{\min})\triangleq\min _{\tau\geq 0}\mathscr{B}(\tau)>0, \end{aligned}$$

(3.5)

where $\tau_{\min}>0$ is a minimal point of $\mathscr{B}(\tau)$ and hence a root of the equation

$$ \tau^{p_{b}^{\ast}-2}+\sum_{i=1}^{l} \frac{\varsigma_{i}}{ p_{b}^{\ast}} \bigl(\beta_{i}\tau^{\beta_{i}-2} - \alpha_{i}\tau^{\beta_{i}} \bigr)-1=0,\quad \tau\geq0. $$

(3.6)

Lemma 3.2

Let $y_{\epsilon}(x)$ be the minimizer of $S_{\mu}$ defined in (2.3) and (2.4), $0\leq\mu<\overline{\mu}$, $0<\varsigma_{i}<+\infty$, $\alpha_{i}$, $\beta_{i}>1$, and $\alpha_{i}+\beta_{i}=p_{b}^{\ast}$ ($i=1, \ldots, l$). Then we have the following statements.

• (i) $S_{\mu, l}=\mathscr{B}(\tau_{\min})S_{\mu}$.

• (ii) $S_{\mu, l}$ has the minimizer $(y_{\epsilon}(x), \tau_{\min}y_{\epsilon}(x))$ for all $\epsilon>0$.

Proof

Similar to the proof in Nyamoradi [15], Theorem 2. □

Lemma 3.3

Let $\{(u_{n}, v_{n})\}$ be a weakly convergent sequence to $(u, v)$ in $(\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$ such that $\vert x\vert ^{-2a}\vert \nabla u_{n}\vert ^{2}\rightharpoonup\eta^{(1)}$, $\vert x\vert ^{-2a}\vert \nabla v_{n}\vert ^{2}\rightharpoonup\eta^{(2)}$, $\vert \vert x\vert ^{-b}u_{n}\vert ^{p_{b}^{\ast}}\rightharpoonup\rho^{(1)}$, $\vert \vert x\vert ^{-b}v_{n}\vert ^{p_{b}^{\ast}}\rightharpoonup\rho^{(2)}$, $\vert x\vert ^{-bp_{b}^{\ast}}\vert u_{n}\vert ^{\alpha_{i}}\vert v_{n}\vert ^{\beta_{i}} \rightharpoonup\nu^{(i)}$ ($i=1, \ldots, l$), $\vert x\vert ^{-2(1+a)}\vert u_{n}\vert ^{2}\rightharpoonup\gamma^{(1)}$, $\vert x\vert ^{-2(1+a)}\vert v_{n}\vert ^{2}\rightharpoonup\gamma^{(2)}$ in the sense of measures. Then there exists some at most countable set $\mathscr{J}$, $\{\eta_{j}^{(1)}\geq 0\}_{j\in\mathscr{J}\cup\{0\}}$, $\{\eta_{j}^{(2)}\geq 0\}_{j\in\mathscr{J}\cup\{0\}}$, $\{\rho_{j}^{(1)}\geq 0\}_{j\in\mathscr{J}\cup\{0\}}$, $\{\rho_{j}^{(2)}\geq 0\}_{j\in\mathscr{J}\cup\{0\}}$, $\{\nu_{j}^{(i)}\geq 0\}_{j\in\mathscr{J}\cup\{0\}}$, $\gamma_{0}^{(1)}\geq0$, $\gamma_{0}^{(2)}\geq0$, $\{x_{j}\}_{j\in\mathscr{J}}\subset\mathbb{R}^{N}\backslash\{0\}$ such that

• (a) $\eta^{(1)}\geq\frac{\vert \nabla u\vert ^{2}}{\vert x\vert ^{2a}}+ \sum_{j\in\mathscr {J}}\eta_{j}^{(1)}\delta_{x_{j}}+\eta_{0}^{(1)}\delta_{0}$, $\eta^{(2)}\geq\frac{\vert \nabla v\vert ^{2}}{\vert x\vert ^{2a}}+ \sum_{j\in\mathscr {J}}\eta_{j}^{(2)}\delta_{x_{j}}+\eta_{0}^{(2)}\delta_{0}$,

• (b) $\rho^{(1)}= \frac{\vert u\vert ^{p_{b}^{\ast}}}{\vert x\vert ^{bp_{b}^{\ast}}}+ \sum_{j\in\mathscr {J}}\rho_{j}^{(1)}\delta_{x_{j}}+\rho_{0}^{(1)}\delta_{0}$, $\rho^{(2)}= \frac{\vert v\vert ^{p_{b}^{\ast}}}{\vert x\vert ^{bp_{b}^{\ast}}}+ \sum_{j\in\mathscr {J}}\rho_{j}^{(2)}\delta_{x_{j}}+\rho_{0}^{(2)}\delta_{0}$, $\nu^{(i)}=\vert x\vert ^{-bp_{b}^{\ast}}\vert u\vert ^{\alpha_{i}}\vert v\vert ^{\beta_{i}}+ \sum_{j\in\mathscr {J}}\nu_{j}^{(i)}\delta_{x_{j}}+\nu_{0}^{(i)}\delta_{0}$, $i=1, \ldots, l$,

• (c) $\gamma^{(1)}= \vert x\vert ^{-2(1+a)}\vert u\vert ^{2}+\gamma_{0}^{(1)}\delta_{0}$, $\gamma^{(2)}=\vert x\vert ^{-2(1+a)}\vert v\vert ^{2}+\gamma_{0}^{(2)}\delta_{0}$,

• (d) $S_{0, l} [\rho_{j}^{(1)}+\rho_{j}^{(2)}+ \sum_{i=1}^{l}\varsigma_{i}\nu_{j}^{(i)} ]^{\frac{2}{p_{b}^{\ast}}}\leq\eta_{j}^{(1)}+\eta_{j}^{(2)}$, $S_{0}(\rho_{j}^{(1)})^{\frac{2}{p_{b}^{\ast}}}\leq\eta_{j}^{(1)}$, $S_{0}(\rho_{j}^{(2)})^{\frac{2}{p_{b}^{\ast}}}\leq\eta_{j}^{(2)}$,

• (e) $S_{\mu, l} [\rho_{0}^{(1)}+\rho_{0}^{(2)}+ \sum_{i=1}^{l}\varsigma_{i}\nu_{0}^{(i)} ]^{\frac{2}{p_{b}^{\ast}}}\leq\eta_{0}^{(1)}+\eta_{0}^{(2)} -\mu(\gamma_{0}^{(1)}+\gamma_{0}^{(2)})$, $S_{\mu}(\rho_{0}^{(1)})^{\frac{2}{p_{b}^{\ast}}} \leq\eta_{0}^{(1)}-\mu\gamma_{0}^{(1)}$, $S_{\mu}(\rho_{0}^{(2)})^{\frac{2}{p_{b}^{\ast}}} \leq\eta_{0}^{(2)}-\mu\gamma_{0}^{(2)}$,

where $\delta_{x_{j}}$, $j\in\mathscr{J}\cup \{0 \}$, is the Dirac mass of 1 concentrated at $x_{j}\in\mathbb{R}^{N}$.

Proof

Similar to the proof of the concentration compactness principle in [27] (see also [18], Lemma 2.2). □

To prove the existence results of problem $(\mathscr{P}^{K}_{0})$, we need the following local $(\mathit{PS})_{c}$ condition.

Lemma 3.4

Suppose that (k.1) and (k.2) hold. Then the $(\mathit{PS})_{c}$ condition in $(\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$ holds for $\mathscr{F}$ if

$$ c< c_{0}^{\ast}\triangleq\frac{1+a-b}{N}\min \biggl\{ \frac{\vert G\vert S_{0, l}^{\frac{N}{2(1+a-b)}}}{\Vert K_{+}\Vert _{\infty}^{\frac{N-2(1+a-b)}{2(1+a-b)}}}, \frac{S_{\mu, l}^{\frac{N}{2(1+a-b)}}}{K_{+}(0)^{\frac{N-2(1+a-b)}{2(1+a-b)}}}, \frac{S_{\mu, l}^{\frac{N}{2(1+a-b)}}}{K_{+}(\infty)^{\frac{N-2(1+a-b)}{2(1+a-b)}}} \biggr\} . $$

(3.7)

Proof

The proof is analogous to that of [10], Proposition 2, but we exhibit it here for completeness. Let $\{(u_{n}, v_{n})\}\subset(\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$ be a $(\mathit{PS})_{c}$ sequence for $\mathscr{F}$ with $c< c_{0}^{\ast}$. Then we see from (k.2), (2.1), and (3.7) that $\{(u_{n}, v_{n})\}$ is bounded in $(\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$. Therefore, up to a subsequence, we may assume that $(u_{n}, v_{n})\rightharpoonup(u, v)$ in $(\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$. According to Lemma 3.3, there exist measures $\eta^{(1)}$, $\eta^{(2)}$, $\rho^{(1)}$, $\rho^{(2)}$, $\nu^{(i)}$ ($i=1, \ldots, l$), $\gamma^{(1)}$, and $\gamma^{(2)}$ such that relations (a)-(e) of this lemma hold. Let $x_{j}\neq0$ be a singular point of measures $\eta^{(1)}$, $\eta^{(2)}$, $\rho^{(1)}$, $\rho^{(2)}$, and $\nu^{(i)}$ ($i=1, \ldots, l$). We define two functions $\phi_{1}, \phi_{2}\in\mathscr{C}^{1}(\mathbb{R}^{N})$ such that $0\leq\phi_{1}$, $\phi_{2}\leq1$, $\phi_{1}=\phi_{2}=1$ for $\vert x-x_{j}\vert \leq\epsilon/2$, $\phi_{1}=\phi_{2}=0$ for $\vert x-x_{j}\vert \geq\epsilon$ and $\vert \nabla\phi_{1}\vert \leq4/\epsilon$, $\vert \nabla\phi_{2}\vert \leq4/\epsilon$. By Lemma 3.1, $\lim_{n\rightarrow\infty}\langle\mathscr{F}^{\prime}(u_{n}, v_{n}), (u_{n}\phi_{1}, v_{n}\phi_{2})\rangle=0$; hence, using (2.1) and the Hölder inequality and the fact that $p_{a}^{\ast}=2^{\ast}$, we get

$$ \begin{aligned}[b]& \int_{\mathbb{R}^{N}} \Biggl\{ \bigl(\phi_{1}\,d \eta^{(1)}+\phi_{2}\,d\eta^{(2)} \bigr) -K(x) \Biggl[ \phi_{1}\,d\rho^{(1)}+\sum_{i=1}^{l} \frac{\varsigma_{i}}{p_{b}^{\ast}} (\alpha_{i}\phi_{1} + \beta_{i}\phi_{2} )\,d\nu+\phi_{2}\,d \rho^{(2)} \Biggr] \Biggr\} \\ &\qquad{}- \int_{\mathbb{R}^{N}}\mu \bigl(\phi_{1}\,d\gamma^{(1)}+ \phi_{2}\,d\gamma ^{(2)} \bigr)\\ &\quad \leq\limsup _{n\rightarrow\infty} \int_{\mathbb{R}^{N}}\vert x\vert ^{-2a} \bigl[\vert u_{n}\nabla u_{n}\nabla\phi_{1}\vert +\vert v_{n}\nabla v_{n}\nabla\phi_{2}\vert \bigr]\,dx \\ &\quad\leq\sup_{n\geq1} \biggl( \int_{\mathbb{R}^{N}}\vert x\vert ^{-2a}\vert \nabla u_{n}\vert ^{2}\,dx \biggr)^{\frac{1}{2}}\limsup _{n\rightarrow\infty} \biggl( \int_{\mathbb{R}^{N}}\vert x\vert ^{-2a}\vert u_{n}\vert ^{2}\vert \nabla\phi_{1}\vert ^{2}\,dx \biggr)^{\frac{1}{2}} \\ &\qquad{} +\sup_{n\geq1} \biggl( \int_{\mathbb{R}^{N}}\vert x\vert ^{-2a}\vert \nabla v_{n}\vert ^{2}\,dx \biggr)^{\frac{1}{2}}\limsup _{n\rightarrow\infty} \biggl( \int_{\mathbb{R}^{N}}\vert x\vert ^{-2a}\vert v_{n}\vert ^{2}\vert \nabla\phi_{2}\vert ^{2}\,dx \biggr)^{\frac{1}{2}} \\ &\quad\leq C \biggl\{ \biggl( \int_{\mathbb{R}^{N}} \vert x\vert ^{-2a}\vert u\vert ^{2}\vert \nabla\phi_{1}\vert ^{2}\,dx \biggr)^{\frac{1}{2}}+ \biggl( \int _{\mathbb{R}^{N}} \vert x\vert ^{-2a}\vert v\vert ^{2}\vert \nabla\phi_{2}\vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggr\} \\ &\quad\leq C \biggl\{ \biggl( \int_{B_{\epsilon}(x_{j})} \frac{\vert u\vert ^{2^{\ast}}}{ \vert x\vert ^{2^{\ast}a}} \biggr)^{\frac{1}{2^{\ast}}} \biggl( \int_{\mathbb{R}^{N}} \vert \nabla\phi_{1}\vert ^{N} \biggr)^{\frac{1}{N}}+ \biggl( \int_{B_{\epsilon}(x_{j})} \frac{\vert v\vert ^{2^{\ast}}}{ \vert x\vert ^{2^{\ast}a}} \biggr)^{\frac{1}{2^{\ast}}} \biggl( \int_{\mathbb{R}^{N}} \vert \nabla\phi_{2}\vert ^{N} \biggr)^{\frac{1}{N}} \biggr\} \\ &\quad\leq C \biggl\{ \biggl( \int_{B_{\epsilon}(x_{j})}\vert x\vert ^{-2a}\vert \nabla u\vert ^{2}\,dx \biggr)^{\frac{1}{2}}+ \biggl( \int_{B_{\epsilon}(x_{j})} \vert x\vert ^{-2a}\vert \nabla v \vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggr\} . \end{aligned} $$

(3.8)

Taking the limits as $\epsilon\rightarrow0$, we deduce from (3.8) and Lemma 3.3 that

$$ K(x_{j}) \Biggl(\rho_{j}^{(1)}+ \sum_{i=1}^{l} \varsigma_{i} \nu_{j}^{(i)}+\rho_{j}^{(2)} \Biggr)\geq \eta_{j}^{(1)}+\eta_{j}^{(2)}. $$

(3.9)

This means that the concentration of the measures $\rho^{(1)}$, $\rho^{(2)}$ and $\nu^{(i)}$ cannot occur at points where $K(x_{j})\leq0$, that is, if $K(x_{j})\leq0$ then $\eta_{j}^{(1)}=\eta_{j}^{(2)}=\rho_{j}^{(1)}=\sum_{i=1}^{l} \varsigma_{i}\nu_{j}^{(i)}=\rho_{j}^{(2)}=0$. Combining (3.9) and (d) of Lemma 3.3 we conclude that either

• (i) $\rho_{j}^{(1)} =\sum_{i=1}^{l}\varsigma_{i}\nu_{j}^{(i)}=\rho_{j}^{(2)}=0$ or

• (ii) $\rho_{j}^{(1)} +\sum_{i=1}^{l}\varsigma_{i}\nu_{j}^{(i)}+\rho_{j}^{(2)}\geq (S_{0, l}/K(x_{j}) )^{\frac{N}{2(1+a-b)}}\geq (S_{0, l}/\Vert K_{+}\Vert _{\infty} )^{\frac{N}{2(1+a-b)}}$.

For the point $x=0$, similarly to the case $x_{j}\neq0$, we obtain

$$\eta_{0}^{(1)}+\eta_{0}^{(2)} -\mu \bigl(\gamma_{0}^{(1)}+\gamma_{0}^{(2)} \bigr) -K(0) \Biggl(\rho_{0}^{(1)} +\sum _{i=1}^{l}\varsigma_{i} \nu_{0}^{(i)}+\rho_{0}^{(2)} \Biggr) \leq0. $$

This, combined with (e) of Lemma 3.3, implies that either

• (iii) $\rho_{0}^{(1)} =\sum_{i=1}^{l}\varsigma_{i}\nu_{0}^{(i)}=\rho_{0}^{(2)}=0$ or

• (iv) $\rho_{0}^{(1)} +\sum_{i=1}^{l}\varsigma_{i}\nu_{0}^{(i)}+\rho_{0}^{(2)}\geq (S_{\mu, l}/K_{+}(0) )^{\frac{N}{2(1+a-b)}}$.

To study the concentration at infinity of the sequence we need to define the following quantities:

• (1) $\eta_{\infty}^{(1)}=\lim_{R\rightarrow\infty}\limsup_{n\rightarrow\infty}\int_{\vert x\vert >R}\frac{\vert \nabla u_{n}\vert ^{2}}{\vert x\vert ^{2a}}\,dx$, $\eta_{\infty}^{(2)}=\lim_{R\rightarrow\infty}\limsup_{n\rightarrow\infty}\int_{\vert x\vert >R}\frac{\vert \nabla v_{n}\vert ^{2}}{\vert x\vert ^{2a}}\,dx$,

• (2) $\rho_{\infty}^{(1)}= \lim_{R\rightarrow\infty}\limsup_{n\rightarrow\infty}\int_{\vert x\vert >R}\frac{\vert u_{n}\vert ^{p_{b}^{\ast}}}{ \vert x\vert ^{bp_{b}^{\ast}}}\,dx$, $\rho_{\infty}^{(2)}= \lim_{R\rightarrow\infty}\limsup_{n\rightarrow\infty}\int_{\vert x\vert >R}\frac{\vert v_{n}\vert ^{p_{b}^{\ast}}}{ \vert x\vert ^{bp_{b}^{\ast}}}\,dx$, $\nu_{\infty}^{(i)}= \lim_{R\rightarrow\infty}\limsup_{n\rightarrow\infty}\int_{\vert x\vert >R} \frac{\vert u_{n}\vert ^{\alpha_{i}}\vert v_{n}\vert ^{\beta_{i}}}{\vert x\vert ^{bp_{b}^{\ast}}}\,dx$, $i=1, \ldots, l$,

• (3) $\gamma_{\infty}^{(1)}=\lim_{R\rightarrow\infty}\limsup_{n\rightarrow\infty}\int_{\vert x\vert >R}\frac{\vert u_{n}\vert ^{2}}{\vert x\vert ^{2(1+a)}}\,dx$, $\gamma_{\infty}^{(2)}=\lim_{R\rightarrow\infty}\limsup_{n\rightarrow\infty}\int_{\vert x\vert >R}\frac{\vert v_{n}\vert ^{2}}{\vert x\vert ^{2(1+a)}}\,dx$.

Obviously, $\eta_{\infty}^{(1)}$, $\eta_{\infty}^{(2)}$, $\rho_{\infty}^{(1)}$, $\rho_{\infty}^{(2)}$, $\nu_{\infty}^{(i)}$ ($i=1, \ldots, l$), $\gamma_{\infty}^{(1)}$, and $\gamma_{\infty}^{(2)}$ exist and are finite. For $R>1$, let $\psi_{R}^{(1)}$ and $\psi_{R}^{(2)}$ be two regular functions such that $0\leq\psi_{R}^{(1)}$, $\psi_{R}^{(2)}\leq1$, $\psi_{R}^{(1)}=\psi_{R}^{(2)}=1$ for $\vert x\vert >R+1$, $\psi_{R}^{(1)}=\psi_{R}^{(2)}=0$ for $\vert x\vert < R$, and $\vert \nabla\psi_{R}^{(1)}\vert \leq4/R$, $\vert \nabla\psi_{R}^{(2)}\vert \leq 4/R$. Since the sequence $\{(u_{n}\psi_{R}^{(1)}, v_{n}\psi_{R}^{(2)})\}$ is bounded in $(\mathscr{D}_{a}^{1, 2}(\mathbb{R}^{N}))^{2}$, we see from (3.1) that

$$ \begin{aligned}[b] 0={}&\lim_{n\rightarrow\infty}\bigl\langle \mathscr{F}^{\prime}(u_{n}, v_{n}), \bigl(u_{n}\psi_{R}^{(1)}, v_{n} \psi_{R}^{(2)}\bigr)\bigr\rangle \\ ={}&\lim_{n\rightarrow\infty} \Biggl\{ \int_{\mathbb{R}^{N}} \biggl(\vert x\vert ^{-2a}\vert \nabla u_{n}\vert ^{2}\psi_{R}^{(1)}+ \vert x\vert ^{-2a}\vert \nabla v_{n}\vert ^{2}\psi_{R}^{(2)}-\mu\frac{\vert u_{n}\vert ^{2}\psi_{R}^{(1)} +\vert v_{n}\vert ^{2}\psi_{R}^{(2)}}{\vert x\vert ^{2(1+a)}} \biggr) \,dx \\ &{}+ \int_{\mathbb{R}^{N}} \bigl(\vert x\vert ^{-2a}u_{n} \nabla u_{n}\nabla\psi_{R}^{(1)}+\vert x\vert ^{-2a}v_{n}\nabla v_{n}\nabla \psi_{R}^{(2)} \bigr)\,dx\\ &{}- \int_{\mathbb{R}^{N}} \frac{K(x)}{\vert x\vert ^{bp_{b}^{\ast}}} \Biggl[\vert u_{n} \vert ^{p_{b}^{\ast}}\psi_{R}^{(1)}\\ &{} +\sum _{i=1}^{l}\frac{\varsigma_{i}}{p_{b}^{\ast}} \vert u_{n}\vert ^{\alpha_{i}}\vert v_{n}\vert ^{\beta_{i}} \bigl(\alpha_{i}\psi_{R}^{(1)}+ \beta_{i}\psi_{R}^{(2)} \bigr) +\vert v_{n}\vert ^{p_{b}^{\ast}}\psi_{R}^{(2)} \Biggr]\,dx \Biggr\} . \end{aligned} $$

(3.10)

Moreover, using the inequality (2.1) and the Hölder inequality, we obtain

$$\begin{aligned} &\lim_{R\rightarrow\infty} \limsup_{n\rightarrow\infty} \int_{\mathbb{R}^{N}} \vert x\vert ^{-2a} \bigl\vert u_{n}\nabla u_{n}\nabla\psi_{R}^{(1)} \bigr\vert \, dx \\ &\quad\leq\lim_{R\rightarrow\infty} \limsup_{n\rightarrow\infty} \biggl( \int_{R< \vert x\vert < R+1} \vert x\vert ^{-2a}\vert u_{n}\vert ^{2}\bigl\vert \nabla\psi_{R}^{(1)} \bigr\vert ^{2}\,dx \biggr)^{\frac{1}{2}} \biggl( \int_{\mathbb{R}^{N}} \vert x\vert ^{-2a}\vert \nabla u_{n}\vert ^{2}\,dx \biggr)^{\frac{1}{2}} \\ &\quad\leq C\lim_{R\rightarrow\infty} \biggl( \int_{R< \vert x\vert < R+1} \vert x\vert ^{-2a}\vert u\vert ^{2}\bigl\vert \nabla\psi_{R}^{(1)}\bigr\vert ^{2}\,dx \biggr)^{\frac{1}{2}} \\ &\quad\leq C\lim_{R\rightarrow\infty} \biggl( \int_{R< \vert x\vert < R+1} \vert x\vert ^{-2^{\ast}a} \vert u\vert ^{2^{\ast}}\,dx \biggr)^{\frac{1}{2^{\ast}}} \biggl( \int _{\mathbb{R}^{N}} \bigl\vert \nabla\psi_{R}^{(1)} \bigr\vert ^{N}\,dx \biggr)^{\frac{1}{N}} \\ &\quad\leq C\lim_{R\rightarrow\infty} \biggl( \int_{R< \vert x\vert < R+1} \vert x\vert ^{-2a}\vert \nabla u \vert ^{2}\,dx \biggr)^{\frac{1}{2}}=0. \end{aligned}$$

Similarly, we have $\lim_{R\rightarrow \infty}\limsup_{n\rightarrow\infty} \int_{\mathbb{R}^{N}} \vert x\vert ^{-2a}\vert v_{n}\nabla v_{n}\nabla\psi_{R}^{(2)}\vert \, dx=0$. Therefore, we deduce from (3.10) and the definitions (1)-(3) that

$$ K(\infty) \Biggl(\rho_{\infty}^{(1)} +\sum _{i=1}^{l}\varsigma_{i} \nu_{\infty}^{(i)}+\rho_{\infty}^{(2)} \Biggr) \geq \eta_{\infty}^{(1)}+\eta_{\infty}^{(2)} -\mu \bigl( \gamma_{\infty}^{(1)}+\gamma_{\infty}^{(2)} \bigr). $$

(3.11)

On the other hand, according to the inequality (2.2) and the definition (3.3) of $S_{\mu, l}$ we find that $\overline{\mu}\gamma_{\infty}^{(1)}\leq\eta_{\infty}^{(1)}$, $\overline{\mu}\gamma_{\infty}^{(2)}\leq\eta_{\infty}^{(2)}$, and

$$S_{\mu, l} \Biggl(\rho_{\infty}^{(1)} +\sum _{i=1}^{l}\varsigma_{i} \nu_{\infty}^{(i)} +\rho_{\infty}^{(2)} \Biggr)^{\frac{2}{p_{b}^{\ast}}} \leq\eta_{\infty}^{(1)}+ \eta_{\infty}^{(2)} -\mu \bigl(\gamma_{\infty}^{(1)}+ \gamma_{\infty}^{(2)} \bigr). $$

This, combined with (3.11), implies that either

• (v) $\rho_{\infty}^{(1)}= \sum_{i=1}^{l}\varsigma_{i}\nu_{\infty}^{(i)}=\rho_{\infty}^{(2)}=0$ or

• (vi) $\rho_{\infty}^{(1)} +\sum_{i=1}^{l}\varsigma_{i}\nu_{\infty}^{(i)} +\rho_{\infty}^{(2)}\geq (S_{\mu, l}/K_{+}(\infty) )^{\frac{N}{2(1+a-b)}}$.

In the following, we rule out the cases (ii), (iv), and (vi). For every continuous nonnegative function ψ such that $0\leq\psi(x)\leq1$ on $\mathbb{R}^{N}$, we get from (3.1) and (3.2)

$$\begin{aligned} c&=\lim_{n\rightarrow\infty} \biggl(\mathscr{F}(u_{n}, v_{n}) -\frac{1}{p_{b}^{\ast}}\bigl\langle \mathscr{F}^{\prime}(u_{n}, v_{n}), (u_{n}, v_{n})\bigr\rangle \biggr) \\ &=\frac{1+a-b}{N} \lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}} \biggl(\vert x\vert ^{-2a}\vert \nabla u_{n}\vert ^{2}+\vert x\vert ^{-2a} \vert \nabla v_{n}\vert ^{2}-\mu\frac{\vert u_{n}\vert ^{2}+\vert v_{n}\vert ^{2}}{\vert x\vert ^{2(1+a)}} \biggr)\,dx \\ &\geq\frac{1+a-b}{N}\limsup_{n\rightarrow \infty} \int_{\mathbb{R}^{N}} \biggl(\vert x\vert ^{-2a}\vert \nabla u_{n}\vert ^{2}+\vert x\vert ^{-2a}\vert \nabla v_{n}\vert ^{2}-\mu\frac{\vert u_{n}\vert ^{2}+\vert v_{n}\vert ^{2}}{\vert x\vert ^{2(1+a)}} \biggr)\psi(x)\,dx. \end{aligned} $$

If (ii) occurs, then the set $\mathscr{J}$ must be finite because the measures $\rho^{(1)}$, $\rho^{(2)}$, and $\nu^{(i)}$ ($i=1, \ldots, l$) are bounded. Since functions $(u_{n}, v_{n})$ are G-symmetric, the measures $\rho^{(1)}$, $\rho^{(2)}$, and $\nu^{(i)}$ ($i=1, \ldots, l$) must be G-invariant. This means that if $x_{j}\neq0$ is a singular point of $\rho^{(1)}$, $\rho^{(2)}$ and $\nu^{(i)}$ ($i=1, \ldots, l$), so is $gx_{j}$ for each $g\in G$, and the mass of $\rho^{(1)}$, $\rho^{(2)}$, and $\nu^{(i)}$ ($i=1, \ldots, l$) concentrated at $gx_{j}$ is the same for each $g\in G$. Assuming that (ii) holds for some $j\in\mathscr{J}$ with $x_{j}\neq0$, we choose ψ with compact support so that $\psi(gx_{j})=1$ for each $g\in G$ and we have

$$\begin{aligned} c&\geq\frac{\vert G\vert (\eta_{j}^{(1)}+\eta_{j}^{(2)})}{N(1+a-b)^{-1}}\geq \frac{1+a-b}{N}\vert G\vert S_{0, l} \Biggl(\rho_{j}^{(1)} +\sum _{i=1}^{l}\varsigma_{i} \nu_{j}^{(i)}+\rho_{j}^{(2)} \Biggr) ^{\frac{2}{p_{b}^{\ast}}} \\ &\geq\frac{1+a-b}{N}\vert G\vert S_{0, l} \bigl(S_{0, l}/ \Vert K_{+}\Vert _{\infty} \bigr)^{\frac{N-2(1+a-b)}{2(1+a-b)}} = \frac{(1+a-b)\vert G\vert S_{0, l}^{\frac{N}{2(1+a-b)}}}{N\Vert K_{+}\Vert _{\infty}^{\frac{N-2(1+a-b)}{2(1+a-b)}}}, \end{aligned} $$

which contradicts (3.7). Similarly, if (iv) holds for $x=0$, we choose ψ with compact support, so that $\psi(0)=1$ and we get

$$\begin{aligned} c&\geq\frac{\eta_{0}^{(1)}+\eta_{0}^{(2)} -\mu\gamma_{0}^{(1)}-\mu\gamma_{0}^{(2)}}{N(1+a-b)^{-1}}\geq \frac{1+a-b}{N}S_{\mu, l} \Biggl(\rho_{0}^{(1)} +\sum_{i=1}^{l} \varsigma_{i}\nu_{0}^{(i)}+\rho_{0}^{(2)} \Biggr) ^{\frac{2}{p_{b}^{\ast}}} \\ &\geq\frac{1+a-b}{N}S_{\mu, l} \bigl(S_{\mu, l}/K_{+}(0) \bigr)^{\frac{N-2(1+a-b)}{2(1+a-b)}} =\frac{(1+a-b)S_{\mu, l}^{\frac{N}{2(1+a-b)}}}{NK_{+}(0)^{\frac{N-2(1+a-b)}{2(1+a-b)}}}, \end{aligned} $$

a contradiction with (3.7). Finally, if (vi) occurs at ∞, we take $\psi=\psi_{R}^{(1)}=\psi_{R}^{(2)}$ to get

$$\begin{aligned} c&\geq\frac{\eta_{\infty}^{(1)}+\eta_{\infty}^{(2)} -\mu\gamma_{\infty}^{(1)}-\mu\gamma_{\infty}^{(2)}}{N(1+a-b)^{-1}}\geq \frac{1+a-b}{N}S_{\mu, l} \Biggl(\rho_{\infty}^{(1)} +\sum_{i=1}^{l} \varsigma_{i}\nu_{\infty}^{(i)}+\rho_{\infty}^{(2)} \Biggr) ^{\frac{2}{p_{b}^{\ast}}} \\ &\geq\frac{1+a-b}{N}S_{\mu, l} \bigl(S_{\mu, l}/K_{+}( \infty) \bigr)^{\frac{N-2(1+a-b)}{2(1+a-b)}} =\frac{(1+a-b)S_{\mu, l}^{\frac{N}{2(1+a-b)}}}{NK_{+}(\infty)^{\frac{N-2(1+a-b)}{2(1+a-b)}}}, \end{aligned} $$

which is impossible. Therefore, $\rho_{j}^{(1)}=\rho_{j}^{(2)}=\nu_{j}^{(i)}=0$ ($i=1, \ldots, l$) for all $j\in\mathscr{J}\cup\{0, \infty\}$, and this implies that

$$\begin{aligned} &\lim_{n\rightarrow\infty} \int_{\mathbb{R}^{N}} \frac{\vert u_{n}\vert ^{p_{b}^{\ast}} +\sum_{i=1}^{l}\varsigma_{i}\vert u_{n}\vert ^{\alpha_{i}}\vert v_{n}\vert ^{\beta_{i}} +\vert v_{n}\vert ^{p_{b}^{\ast}}}{ \vert x\vert ^{bp_{b}^{\ast}}}\,dx\\ &\quad = \int_{\mathbb{R}^{N}}\frac{\vert u\vert ^{p_{b}^{\ast}} +\sum_{i=1}^{l}\varsigma_{i}\vert u\vert ^{\alpha_{i}}\vert v\vert ^{\beta_{i}} +\vert v\vert ^{p_{b}^{\ast}}}{ \vert x\vert ^{bp_{b}^{\ast}}}\,dx. \end{aligned}$$

Finally, since $\lim_{n\rightarrow\infty}\langle \mathscr{F}^{\prime}(u_{n}, v_{n})-\mathscr{F}^{\prime}(u, v), (u_{n}-u, v_{n}-v)\rangle=0$, we naturally conclude that $(u_{n}, v_{n})\rightarrow(u, v)$ in $(\mathscr{D}_{a}^{1, 2}(\mathbb{R}^{N}))^{2}$. □

From Lemma 3.4 we immediately obtain the following corollary.

Corollary 3.1

If $K_{+}(0)=K_{+}(\infty)=0$ and $\vert G\vert =+\infty$, then the functional $\mathscr{F}$ satisfies the $(\mathit{PS})_{c}$ condition for every $c\in\mathbb{R}$.

Proof of Theorem 2.1

First of all, we choose $\epsilon>0$ such that the condition (2.7) holds, where $y_{\epsilon}$ is the extremal function satisfying (2.4), (2.5), and (2.6). By (k.2), (3.1), and (3.3), we get

$$\begin{aligned} \mathscr{F}(u, v)&=\frac{1}{2}\bigl\Vert (u, v)\bigr\Vert _{\mu}^{2} -\frac{1}{p_{b}^{\ast}} \int_{\mathbb{R}^{N}} \frac{K(x)}{\vert x\vert ^{bp_{b}^{\ast}}} \Biggl(\vert u\vert ^{p_{b}^{\ast}} +\sum_{i=1}^{l} \varsigma_{i}\vert u\vert ^{\alpha_{i}}\vert v\vert ^{\beta_{i}} +\vert v\vert ^{p_{b}^{\ast}} \Biggr)\,dx \\ &\geq\frac{1}{2}\bigl\Vert (u, v)\bigr\Vert _{\mu}^{2}- \frac{1}{p_{b}^{\ast}} \Vert K\Vert _{\infty}S_{\mu, l}^{\frac{-N}{N-2(1+a-b)}} \bigl\Vert (u, v)\bigr\Vert _{\mu}^{p_{b}^{\ast}}. \end{aligned} $$

In view of $p_{b}^{\ast}>2$, we conclude that there exist constants $\alpha_{0}>0$ and $\rho>0$ such that $\mathscr{F}(u, v)\geq\alpha_{0}$ for all $\Vert (u, v)\Vert _{\mu}=\rho$. Moreover, if we set $u=y_{\epsilon}$, $v=\tau_{\min}y_{\epsilon}$, and

$$\begin{aligned} \Phi(t)={}&\mathscr{F}(ty_{\epsilon}, t\tau_{\min}y_{\epsilon})\\ ={}& \frac{t^{2}}{2} \bigl(1+\tau_{\min}^{2} \bigr) \int_{\mathbb{R}^{N}} \biggl(\vert x\vert ^{-2a}\vert \nabla y_{\epsilon}\vert ^{2}-\mu\frac{y_{\epsilon}^{2}}{\vert x\vert ^{2(1+a)}} \biggr)\,dx \\ &{}-\frac{t^{p_{b}^{\ast}}}{p_{b}^{\ast}} \Biggl(1 +\sum_{i=1}^{l} \varsigma_{i}\tau_{\min}^{\beta_{i}} +\tau_{\min}^{p_{b}^{\ast}} \Biggr) \int_{\mathbb{R}^{N}} K(x)\vert x\vert ^{-bp_{b}^{\ast}}y_{\epsilon}^{p_{b}^{\ast}} \,dx \end{aligned} $$

with $t\geq0$, then we easily check that $\Phi(t)$ has a unique maximum at some $\overline{t}>0$. Simple arithmetic gives us the value

$$\overline{t}= \biggl\{ \frac{ (1+\tau_{\min}^{2} ) \int_{\mathbb{R}^{N}} (\vert x\vert ^{-2a}\vert \nabla y_{\epsilon}\vert ^{2}-\mu\frac{y_{\epsilon}^{2}}{\vert x\vert ^{2(1+a)}} )\,dx}{ (1 +\sum_{i=1}^{l}\varsigma_{i}\tau_{\min}^{\beta_{i}} +\tau_{\min}^{p_{b}^{\ast}} ) \int_{\mathbb{R}^{N}}K(x)\vert x\vert ^{-bp_{b}^{\ast}}y_{\epsilon}^{p_{b}^{\ast}}\,dx} \biggr\} ^{\frac{1}{p_{b}^{\ast}-2}}. $$

Hence, we have

$$ \begin{aligned}[b]\max_{t\geq 0}\Phi(t)&=\mathscr{F}( \overline{t}y_{\epsilon}, \overline{t}\tau_{\min}y_{\epsilon}) \\ &=\frac{1+a-b}{N} \biggl\{ \frac{ (1+\tau_{\min}^{2} ) \int_{\mathbb{R}^{N}} (\vert x\vert ^{-2a}\vert \nabla y_{\epsilon}\vert ^{2}-\mu\frac{y_{\epsilon}^{2}}{\vert x\vert ^{2(1+a)}} )\,dx}{ [ (1 +\sum_{i=1}^{l}\varsigma_{i}\tau_{\min}^{\beta_{i}} +\tau_{\min}^{p_{b}^{\ast}} ) \int_{\mathbb{R}^{N}}K(x)\vert x\vert ^{-bp_{b}^{\ast}}y_{\epsilon}^{p_{b}^{\ast }}\,dx ] ^{\frac{2}{p_{b}^{\ast}}}} \biggr\} ^{\frac{p_{b}^{\ast}}{p_{b}^{\ast}-2}}. \end{aligned} $$

(3.12)

Furthermore, since $\mathscr{F}(ty_{\epsilon}, t\tau_{\min}y_{\epsilon})\rightarrow-\infty$ as $t\rightarrow\infty$, we can choose $t_{0}>0$ such that $\Vert (t_{0}y_{\epsilon}, t_{0}\tau_{\min}y_{\epsilon})\Vert _{\mu}>\rho$ and $\mathscr{F}(t_{0}y_{\epsilon}, t_{0}\tau_{\min}y_{\epsilon})<0$, and set

$$ c_{0}=\inf_{\gamma\in\Gamma}\max _{t\in[0, 1]}\mathscr{F}\bigl(\gamma(t)\bigr), $$

(3.13)

where $\Gamma=\{\gamma\in\mathscr{C}([0, 1], (\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}); \gamma(0)=(0, 0), \mathscr{F}(\gamma(1))<0, \Vert \gamma(1)\Vert _{\mu}>\rho\}$. From (2.5), (2.7), (3.4), (3.5), (3.7), (3.12), (3.13), and Lemma 3.2, we obtain

$$\begin{aligned} c_{0}&\leq\mathscr{F}(\overline{t}y_{\epsilon}, \overline{t}\tau_{\min}y_{\epsilon}) \\ &=\frac{1+a-b}{N} \biggl\{ \frac{ (1+\tau_{\min}^{2} ) \int_{\mathbb{R}^{N}} (\vert x\vert ^{-2a}\vert \nabla y_{\epsilon}\vert ^{2}-\mu\frac{y_{\epsilon}^{2}}{\vert x\vert ^{2(1+a)}} )\,dx}{ [ (1 +\sum_{i=1}^{l}\varsigma_{i}\tau_{\min}^{\beta_{i}} +\tau_{\min}^{p_{b}^{\ast}} ) \int_{\mathbb{R}^{N}}K(x)\vert x\vert ^{-bp_{b}^{\ast}}y_{\epsilon}^{p_{b}^{\ast }}\,dx ] ^{\frac{2}{p_{b}^{\ast}}}} \biggr\} ^{\frac{p_{b}^{\ast}}{p_{b}^{\ast }-2}} \\ &\leq\frac{1+a-b}{N} \biggl\{ \frac{\mathscr{B}(\tau_{\min})\int_{\mathbb{R}^{N}} (\vert x\vert ^{-2a}\vert \nabla y_{\epsilon}\vert ^{2}-\mu\frac{y_{\epsilon}^{2}}{\vert x\vert ^{2(1+a)}} )\,dx}{ [\max \{\frac{\Vert K_{+}\Vert _{\infty}}{ \vert G\vert ^{\frac{2(1+a-b)}{N-2(1+a-b)}} S_{0}^{\frac{N}{N-2(1+a-b)}}}, \frac{K_{+}(0)}{S_{\mu}^{\frac{N}{N-2(1+a-b)}}}, \frac{K_{+}(\infty)}{S_{\mu}^{\frac{N}{N-2(1+a-b)}}} \} ] ^{\frac{2}{p_{b}^{\ast}}}} \biggr\} ^{\frac{p_{b}^{\ast}}{p_{b}^{\ast}-2}} \\ &=\frac{1+a-b}{N}\min \biggl\{ \frac{\vert G\vert S_{0, l}^{\frac{N}{2(1+a-b)}}}{\Vert K_{+}\Vert _{\infty}^{\frac{N-2(1+a-b)}{2(1+a-b)}}}, \frac{S_{\mu, l}^{\frac{N}{2(1+a-b)}}}{K_{+}(0)^{\frac{N-2(1+a-b)}{2(1+a-b)}}}, \frac{S_{\mu, l}^{\frac{N}{2(1+a-b)}}}{K_{+}(\infty)^{\frac{N-2(1+a-b)}{2(1+a-b)}}} \biggr\} =c_{0}^{\ast}. \end{aligned} $$

If $c_{0}< c_{0}^{\ast}$, then by Lemma 3.4, the $(\mathit{PS})_{c}$ condition holds and the conclusion follows by the mountain pass theorem in [28] (see also [1]). If $c_{0}=c_{0}^{\ast}$, then $\gamma(t)=(tt_{0}y_{\epsilon}, tt_{0}\tau_{\min}y_{\epsilon})$, with $0\leq t\leq1$, is a path in Γ such that $\max_{t\in[0, 1]}\mathscr{F}(\gamma(t))=c_{0}$. Hence, either $\Phi^{\prime}(\overline{t})=\mathscr{F}^{\prime}(\overline {t}y_{\epsilon}, \overline{t}\tau_{\min}y_{\epsilon})=0$, and we are done, or γ can be deformed to a path $\widetilde{\gamma}\in\Gamma$ with $\max_{t\in[0, 1]}\mathscr{F}(\widetilde{\gamma}(t))< c_{0}$ and we get a contradiction. This part of the proof shows that a nontrivial solution $(u_{0}, v_{0})\in(\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$ of $(\mathscr{P}_{0}^{K})$ exists. In the following, we show that the solution $(u_{0}, v_{0})$ can be chosen to be positive on $\mathbb{R}^{N}$. Taking into account $\mathscr{F}(u_{0}, v_{0})=\mathscr{F}(\vert u_{0}\vert , \vert v_{0}\vert )$ and

$$\begin{aligned} 0&=\bigl\langle \mathscr{F}^{\prime}(u_{0}, v_{0}), (u_{0}, v_{0})\bigr\rangle \\ &=\bigl\Vert (u_{0}, v_{0})\bigr\Vert _{\mu}^{2} - \int_{\mathbb{R}^{N}}\frac{K(x)}{\vert x\vert ^{bp_{b}^{\ast}}} \Biggl(\vert u_{0} \vert ^{p_{b}^{\ast}}+\sum_{i=1}^{l} \varsigma_{i} \vert u_{0}\vert ^{\alpha_{i}}\vert v_{0}\vert ^{\beta_{i}}+\vert v_{0}\vert ^{p_{b}^{\ast}} \Biggr)\,dx, \end{aligned} $$

we obtain

$$\int_{\mathbb{R}^{N}}K(x)\vert x\vert ^{-bp_{b}^{\ast}} \Biggl(\vert u_{0}\vert ^{p_{b}^{\ast}}+\sum_{i=1}^{l} \varsigma_{i} \vert u_{0}\vert ^{\alpha_{i}}\vert v_{0}\vert ^{\beta_{i}} +\vert v_{0}\vert ^{p_{b}^{\ast}} \Biggr)\,dx=\bigl\Vert (u_{0}, v_{0}) \bigr\Vert _{\mu}^{2}>0, $$

which implies $c_{0}=\mathscr{F}(\vert u_{0}\vert , \vert v_{0}\vert )=\max_{t\geq 0}\mathscr{F}(t\vert u_{0}\vert , t\vert v_{0}\vert )$. Hence, either $(\vert u_{0}\vert , \vert v_{0}\vert )$ is a critical point of $\mathscr{F}$ or $\gamma(t)=(tt_{0}\vert u_{0}\vert , tt_{0}\vert v_{0}\vert )$, with $\mathscr{F}(t_{0}\vert u_{0}\vert , t_{0}\vert v_{0}\vert )<0$, can be deformed, as in the first part of the proof, to a path $\widetilde{\gamma}(t)$ with $\max_{t\in[0, 1]}\mathscr{F}(\widetilde{\gamma}(t))< c_{0}$, which is impossible. Thus, we may assume that $u_{0}\geq0$, $v_{0}\geq0$ on $\mathbb{R}^{N}$ and $(u_{0},v_{0})$ is a positive solution of problem $(\mathscr{P}_{0}^{K})$ by the strong maximum principle. □

Proof of Corollary 2.1

First of all, we find that due to the identity (2.6), inequality (2.7) is equivalent to $\int_{\mathbb{R}^{N}}(K(x)-\overline{K})\vert x\vert ^{-bp_{b}^{\ast}} y_{\epsilon}^{p_{b}^{\ast}}\,dx\geq 0$ for some $\epsilon>0$, or equivalently

$$ \int_{\mathbb{R}^{N}}\frac{K(x)-\overline{K}}{\vert x\vert ^{(b+\sqrt{\overline {\mu}} -\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}} (\epsilon+\vert x\vert ^{(p_{b}^{\ast}-2) \sqrt{\overline{\mu}-\mu}} )^{\frac{2p_{b}^{\ast}}{p_{b}^{\ast}-2}}}\,dx\geq0 $$

(3.14)

for some $\epsilon>0$, where

$$\overline{K}=\max\bigl\{ \Vert K_{+}\Vert _{\infty} \vert G\vert ^{\frac {-2(1+a-b)}{N-2(1+a-b)}}(S_{\mu}/S_{0}) ^{\frac{N}{N-2(1+a-b)}}, K_{+}(0), K_{+}(\infty)\bigr\} . $$

Part (1), case (i). According to (2.8) and (3.14), we need to show that

$$ \int_{\mathbb{R}^{N}}\frac{K(x)-K(0)}{\vert x\vert ^{(b+\sqrt{\overline{\mu}} -\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}} (\epsilon+\vert x\vert ^{(p_{b}^{\ast}-2) \sqrt{\overline{\mu}-\mu}} )^{\frac{2p_{b}^{\ast}}{p_{b}^{\ast}-2}}}\,dx\geq0 $$

(3.15)

for some $\epsilon>0$. We choose $\varrho_{0}>0$ so that $K(x)\geq K(0)+\Lambda_{0} \vert x\vert ^{p_{b}^{\ast}\sqrt{\overline{\mu}-\mu}}$ for $\vert x\vert \leq\varrho_{0}$. In view of $(b+\sqrt{\overline{\mu}} -2\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}+(p_{b}^{\ast}-2) \sqrt{\overline{\mu}-\mu}\cdot\frac{2p_{b}^{\ast}}{ p_{b}^{\ast}-2}=N$, we obtain

$$ \begin{aligned}[b] & \int_{\vert x\vert \leq\varrho_{0}}\frac{K(x)-K(0)}{\vert x\vert ^{(b+\sqrt{\overline{\mu}} -\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}} (\epsilon+\vert x\vert ^{(p_{b}^{\ast}-2) \sqrt{\overline{\mu}-\mu}} )^{\frac{2p_{b}^{\ast}}{p_{b}^{\ast}-2}}}\,dx \\ &\quad\geq\Lambda_{0} \int_{\vert x\vert \leq\varrho_{0}}\frac{1}{\vert x\vert ^{(b+\sqrt {\overline{\mu}} -2\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}} (\epsilon+\vert x\vert ^{(p_{b}^{\ast}-2) \sqrt{\overline{\mu}-\mu}} )^{\frac{2p_{b}^{\ast}}{p_{b}^{\ast}-2}}}\,dx\rightarrow+\infty \end{aligned} $$

(3.16)

as $\epsilon\rightarrow0$. On the other hand, for all $\epsilon>0$, we have

$$ \begin{aligned}[b]& \int_{\vert x\vert > \varrho_{0}}\frac{\vert K(x)-K(0)\vert }{\vert x\vert ^{(b+\sqrt{\overline{\mu}} -\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}} (\epsilon+\vert x\vert ^{(p_{b}^{\ast}-2) \sqrt{\overline{\mu}-\mu}} )^{\frac{2p_{b}^{\ast}}{p_{b}^{\ast}-2}}}\,dx \\ &\quad\leq \int_{\vert x\vert > \varrho_{0}}\bigl\vert K(x)-K(0)\bigr\vert \vert x\vert ^{-(b+\sqrt{\overline{\mu}} +\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}}\,dx \leq\overline{C}_{1} \end{aligned} $$

(3.17)

for some constant $\overline{C}_{1}>0$ independent of ϵ. Combining (3.16) and (3.17), we get (3.15) for ϵ sufficiently small.

Part (1), case (ii). We choose $\varrho_{1}>0$ so that $\vert K(x)-K(0)\vert \leq\Lambda_{1}\vert x\vert ^{\kappa}$ for $\vert x\vert \leq\varrho_{1}$. Since $\kappa>p_{b}^{\ast}\sqrt{\overline{\mu}-\mu}>0$, we find from the fact $(b+\sqrt{\overline{\mu}} +\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}=N+p_{b}^{\ast}\sqrt{\overline {\mu}-\mu}$ that

$$\begin{aligned} & \int_{\mathbb{R}^{N}}\frac{\vert K(x)-K(0)\vert }{\vert x\vert ^{(b+\sqrt{\overline{\mu}} -\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}} (\epsilon+\vert x\vert ^{(p_{b}^{\ast}-2) \sqrt{\overline{\mu}-\mu}} )^{\frac{2p_{b}^{\ast}}{p_{b}^{\ast}-2}}}\,dx\\ &\quad\leq \int_{\mathbb{R}^{N}}\frac{\vert K(x)-K(0)\vert }{ \vert x\vert ^{(b+\sqrt{\overline{\mu}} +\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}}}\,dx \\ &\quad\leq\Lambda_{1} \int_{\vert x\vert \leq\varrho_{1}} \vert x\vert ^{\kappa-(b+\sqrt{\overline{\mu}} +\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}}\,dx + \int_{\vert x\vert >\varrho_{1}}\bigl\vert K(x)-K(0)\bigr\vert \vert x\vert ^{-(b+\sqrt{\overline{\mu}} +\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}}\,dx \\ &\quad= \Lambda_{1} \int_{\vert x\vert \leq\varrho_{1}} \vert x\vert ^{-N+(\kappa-p_{b}^{\ast}\sqrt{\overline{\mu}-\mu})}\,dx + \int_{\vert x\vert >\varrho_{1}}\bigl\vert K(x)-K(0)\bigr\vert \vert x\vert ^{-N-p_{b}^{\ast}\sqrt{\overline {\mu}-\mu}}\,dx \leq C. \end{aligned} $$

So by (2.9) and the Lebesgue dominated convergence theorem we have

$$\begin{aligned} &\lim_{\epsilon\rightarrow0} \int_{\mathbb{R}^{N}}\frac{K(x)-K(0)}{\vert x\vert ^{(b+\sqrt{\overline{\mu}} -\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}} (\epsilon+\vert x\vert ^{(p_{b}^{\ast}-2) \sqrt{\overline{\mu}-\mu}} )^{\frac{2p_{b}^{\ast}}{p_{b}^{\ast}-2}}}\,dx \\ &\quad= \int_{\mathbb{R}^{N}} \bigl(K(x)-K(0) \bigr)\vert x\vert ^{-(b+\sqrt{\overline{\mu}} +\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}}\,dx>0. \end{aligned} $$

Therefore (3.15) holds for ϵ sufficiently small.

Part (2), case (i). From (2.10) and (3.14) it is sufficient to show that

$$ \int_{\mathbb{R}^{N}}\frac{ (K(x)-K(\infty) )\epsilon^{\frac {2p_{b}^{\ast}}{p_{b}^{\ast}-2}}}{\vert x\vert ^{(b+\sqrt{\overline{\mu}} -\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}} (\epsilon+\vert x\vert ^{(p_{b}^{\ast}-2) \sqrt{\overline{\mu}-\mu}} )^{\frac{2p_{b}^{\ast}}{p_{b}^{\ast}-2}}}\,dx\geq0 $$

(3.18)

for some $\epsilon>0$. We choose $R_{1}>0$ such that $K(x)\geq K(\infty)+\Lambda_{2}\vert x\vert ^{-p_{b}^{\ast}\sqrt{\overline{\mu}-\mu}}$ for all $\vert x\vert \geq R_{1}$. Then

$$\begin{aligned}& \int_{\vert x\vert \geq R_{1}}\frac{ (K(x)-K(\infty) )\epsilon^{\frac{2p_{b}^{\ast}}{p_{b}^{\ast}-2}}}{\vert x\vert ^{(b+\sqrt{\overline{\mu}} -\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}} (\epsilon+\vert x\vert ^{(p_{b}^{\ast}-2) \sqrt{\overline{\mu}-\mu}} )^{\frac{2p_{b}^{\ast}}{p_{b}^{\ast}-2}}}\,dx \\ &\quad\geq\Lambda_{2} \int_{\vert x\vert \geq R_{1}}\vert x\vert ^{-N} \biggl( \frac{\epsilon}{\epsilon+\vert x\vert ^{(p_{b}^{\ast}-2) \sqrt{\overline{\mu}-\mu}}} \biggr)^{\frac{2p_{b}^{\ast}}{p_{b}^{\ast}-2}}\,dx\rightarrow+\infty \end{aligned} $$

as $\epsilon\rightarrow+\infty$. Moreover, in view of $-(b+\sqrt{\overline{\mu}} -\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}=-N+p_{b}^{\ast}\sqrt{\overline {\mu}-\mu}>-N$, we get

$$\begin{aligned}& \int_{\vert x\vert \leq R_{1}}\frac{ (K(x)-K(\infty) )\epsilon^{\frac{2p_{b}^{\ast}}{p_{b}^{\ast}-2}}}{\vert x\vert ^{(b+\sqrt{\overline{\mu}} -\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}} (\epsilon+\vert x\vert ^{(p_{b}^{\ast}-2) \sqrt{\overline{\mu}-\mu}} )^{\frac{2p_{b}^{\ast}}{p_{b}^{\ast}-2}}}\,dx \\ &\quad\leq \int_{\vert x\vert \leq R_{1}} \bigl(K(x)-K(\infty) \bigr)\vert x\vert ^{-N+p_{b}^{\ast}\sqrt{\overline{\mu }-\mu}}\,dx\leq \overline{C}_{2} \end{aligned} $$

for some constant $\overline{C}_{2}>0$ independent of $\epsilon>0$. These two estimates combined together give (3.18) for $\epsilon>0$ large.

Part (2), case (ii). We choose $R_{2}>0$ such that $\vert K(x)-K(\infty)\vert \leq\Lambda_{3}\vert x\vert ^{-\iota}$ for all $\vert x\vert \geq R_{2}$. Since $\iota>p_{b}^{\ast}\sqrt{\overline{\mu}-\mu}>0$, we obtain

$$\begin{aligned}& \int_{\mathbb{R}^{N}}\bigl\vert K(x)-K(\infty)\bigr\vert \vert x \vert ^{-N+p_{b}^{\ast}\sqrt{\overline{\mu}-\mu}}\,dx \\ &\quad\leq\Lambda_{3} \int_{\vert x\vert \geq R_{2}}\vert x\vert ^{-N-\iota+p_{b}^{\ast}\sqrt{\overline{\mu}-\mu}}\,dx + \int_{\vert x\vert \leq R_{2}}\bigl\vert K(x)-K(\infty)\bigr\vert \vert x \vert ^{-N+p_{b}^{\ast}\sqrt{\overline{\mu}-\mu}}\,dx \\ &\quad< +\infty. \end{aligned} $$

Consequently, by (2.11) and the Lebesgue dominated convergence theorem, we obtain

$$\begin{aligned} &\lim_{\epsilon\rightarrow+\infty} \int_{\mathbb{R}^{N}}\frac{ (K(x)-K(\infty) )\epsilon^{\frac {2p_{b}^{\ast}}{p_{b}^{\ast}-2}}}{\vert x\vert ^{(b+\sqrt{\overline{\mu}} -\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}} (\epsilon+\vert x\vert ^{(p_{b}^{\ast}-2) \sqrt{\overline{\mu}-\mu}} )^{\frac{2p_{b}^{\ast}}{p_{b}^{\ast}-2}}}\,dx \\ &\quad= \int_{\mathbb{R}^{N}} \bigl(K(x)-K(\infty) \bigr) \vert x\vert ^{-(b+\sqrt{\overline{\mu}} -\sqrt{\overline{\mu}-\mu})p_{b}^{\ast}}\,dx>0 \end{aligned} $$

and (3.18) holds for $\epsilon>0$ large. Similarly to above, we know part (3) holds. □

To prove Theorem 2.2 we need the following version of the symmetric mountain pass theorem (see [29], Theorem 9.12).

Lemma 3.5

Let X be an infinite dimensional Banach space and let $\mathscr{F}\in\mathscr{C}^{1}(X, \mathbb{R})$ be an even functional satisfying $(\mathit{PS})_{c}$ condition for each c and $\mathscr{F}(0)=0$. Furthermore, one supposes that:

• (i) there exist constants $\overline{\alpha}>0$ and $\rho>0$ such that $\mathscr{F}(w)\geq\overline{\alpha}$ for all $\Vert w\Vert =\rho$;

• (ii) there exists an increasing sequence of subspaces $\{X_{m}\}$ of X, with $\dim X_{m}=m$, such that for every m one can find a constant $R_{m}>0$ such that $\mathscr {F}(w)\leq0$ for all $w\in X_{m}$ with $\Vert w\Vert \geq R_{m}$.

Then $\mathscr{F}$ possesses a sequence of critical values $\{c_{m}\}$ tending to ∞ as $m\rightarrow\infty$.

Proof of Theorem 2.2

Applying Lemma 3.5 with $X=(\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$ and $w=(u, v)\in X$, we see from (k.2), (3.1), and (3.3) that

$$\mathscr{F}(u, v)\geq\frac{1}{2}\bigl\Vert (u, v)\bigr\Vert _{\mu}^{2}-\frac{1}{p_{b}^{\ast}} \Vert K\Vert _{\infty} S_{\mu, l}^{\frac{-N}{N-2(1+a-b)}}\bigl\Vert (u, v)\bigr\Vert _{\mu}^{p_{b}^{\ast}}. $$

Since $p_{b}^{\ast}>2$, there exist constants $\overline{\alpha}>0$ and $\rho>0$ such that $\mathscr{F}(u, v)\geq\overline{\alpha}$ for all $(u, v)$ with $\Vert (u, v)\Vert _{\mu}=\rho$. To find a suitable sequence of finite dimensional subspaces of $(\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$, we set $\Omega_{K}^{+}=\{x\in\mathbb{R}^{N}; K(x)>0\}$. Obviously, the set $\Omega_{K}^{+}$ is G-symmetric and we can define $(\mathscr {D}_{a, G}^{1, 2}(\Omega_{K}^{+}))^{2}$, which is the subspace of G-symmetric functions of $(\mathscr{D}_{a}^{1, 2}(\Omega_{K}^{+}))^{2}$ (see Section 2). By extending the functions in $(\mathscr{D}_{a, G}^{1, 2}(\Omega_{K}^{+}))^{2}$ by 0 outside $\Omega_{K}^{+}$ we can assume that $(\mathscr{D}_{a, G}^{1, 2}(\Omega_{K}^{+}))^{2}\subset(\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$. Let $\{X_{m}\}$ be an increasing sequence of subspaces of $(\mathscr{D}_{a, G}^{1, 2}(\Omega_{K}^{+}))^{2}$ with $\dim X_{m}=m$ for each m. Then there exists a constant $\xi(m)>0$ such that

$$\int_{\Omega_{K}^{+}}K(x)\vert x\vert ^{-bp_{b}^{\ast}} \Biggl(\vert \tilde{u}\vert ^{p_{b}^{\ast}} +\sum_{i=1}^{l} \varsigma_{i} \vert \tilde{u}\vert ^{\alpha_{i}}\vert \tilde{v}\vert ^{\beta_{i}} +\vert \tilde{v}\vert ^{p_{b}^{\ast}} \Biggr) \,dx\geq\xi(m) $$

for all $(\tilde{u}, \tilde{v})\in X_{m}$, with $\Vert (\tilde{u}, \tilde{v})\Vert _{\mu}=1$. Consequently, if $(u, v)\in X_{m}\backslash\{(0, 0)\}$ then we write $(u, v)=t(\tilde{u}, \tilde{v})$, with $t=\Vert (u, v)\Vert _{\mu}$ and $\Vert (\tilde{u}, \tilde{v})\Vert _{\mu}=1$. Therefore we obtain

$$\mathscr{F}(u, v)=\frac{t^{2}}{2} -\frac{t^{p_{b}^{\ast}}}{p_{b}^{\ast}} \int_{\Omega_{K}^{+}} \frac{K(x)}{\vert x\vert ^{bp_{b}^{\ast}}} \Biggl(\vert \tilde{u}\vert ^{p_{b}^{\ast}} +\sum_{i=1}^{l} \varsigma_{i} \vert \tilde{u}\vert ^{\alpha_{i}}\vert \tilde{v}\vert ^{\beta_{i}} +\vert \tilde{v}\vert ^{p_{b}^{\ast}} \Biggr) \,dx\leq \frac{t^{2}}{2}-\frac{\xi(m)}{p_{b}^{\ast}} t^{p_{b}^{\ast}}\leq0 $$

for t large enough. By Lemma 3.5 and Corollary 3.1, we conclude that there exists a sequence of critical values $c_{m}\rightarrow\infty$ and the results follow. □

Proof of Corollary 2.2

Since $K(x)$ is radially symmetric, we find the corresponding group $G=O(\mathbb{N})$ and $\vert G\vert =+\infty$. According to Corollary 3.1, $\mathscr{F}$ satisfies the $(\mathit{PS})_{c}$ condition for every $c\in\mathbb{R}$. Therefore we deduce from Theorem 2.2 that the results follow. □

4 Multiplicity results for problem $(\mathscr{P}_{\sigma }^{K_{0}})$

Throughout this section we assume that $\sigma>0$ and $K(x)\equiv K_{0}>0$ is a constant. Since we are interested in positive G-symmetric solutions of problem $(\mathscr {P}_{\sigma}^{K_{0}})$, we define a functional $\mathscr{T}_{\sigma}: (\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}\rightarrow\mathbb{R}$ given by

$$ \begin{aligned}[b] \mathscr{T}_{\sigma}(u, v)={}&\frac{1}{2}\bigl\Vert (u, v)\bigr\Vert _{\mu}^{2} -\frac{K_{0}}{p_{b}^{\ast}} \int_{\mathbb{R}^{N}} \vert x\vert ^{-bp_{b}^{\ast}} \Biggl(\bigl\vert u^{+}\bigr\vert ^{p_{b}^{\ast}} +\sum_{i=1}^{l} \varsigma_{i}\bigl\vert u^{+}\bigr\vert ^{\alpha_{i}}\bigl\vert v^{+}\bigr\vert ^{\beta_{i}} +\bigl\vert v^{+}\bigr\vert ^{p_{b}^{\ast}} \Biggr)\,dx \\ &{}-\frac{\sigma}{q} \int_{\mathbb{R}^{N}}h(x)\vert x\vert ^{-dp_{d}^{\ast}} \bigl(\bigl\vert u^{+}\bigr\vert ^{q} +\bigl\vert v^{+}\bigr\vert ^{q} \bigr)\,dx, \end{aligned} $$

(4.1)

where $1< q<2$, $u^{+}=\max\{0, u\}$, and $v^{+}=\max\{0, v\}$. By (h.2), (2.1), the Hölder inequality, and the fact that $\theta=p_{d}^{\ast}/(p_{d}^{\ast}-q)$, we obtain

$$ \begin{aligned}[b] & \int_{\mathbb{R}^{N}}h(x)\vert x\vert ^{-dp_{d}^{\ast}} \bigl(\bigl\vert u^{+}\bigr\vert ^{q} +\bigl\vert v^{+}\bigr\vert ^{q} \bigr)\,dx \\ &\quad\leq \biggl( \int_{\mathbb{R}^{N}}\vert x\vert ^{-dp_{d}^{\ast}} h^{\theta}(x) \,dx \biggr)^{\frac{1}{\theta}} \biggl[ \biggl( \int_{\mathbb{R}^{N}}\vert x\vert ^{-dp_{d}^{\ast}} \bigl\vert u^{+}\bigr\vert ^{p_{d}^{\ast}}\,dx \biggr)^{\frac{q}{p_{d}^{\ast}}} + \biggl( \int_{\mathbb{R}^{N}}\vert x\vert ^{-dp_{d}^{\ast}} \bigl\vert v^{+}\bigr\vert ^{p_{d}^{\ast}}\,dx \biggr)^{\frac{q}{p_{d}^{\ast}}} \biggr] \\ &\quad\leq C\Vert h\Vert _{\theta} \bigl(\Vert u\Vert _{\mu}^{q}+\Vert v\Vert _{\mu}^{q} \bigr)\leq C\Vert h\Vert _{\theta}\bigl\Vert (u, v)\bigr\Vert _{\mu}^{q}. \end{aligned} $$

(4.2)

Thus we see from (4.2) that $\mathscr{T}_{\sigma}$ is well defined, $\mathscr{T}_{\sigma}\in\mathscr{C}^{1}((\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}, \mathbb{R})$ and there exists a one-to-one correspondence between the weak solutions of $(\mathscr {P}_{\sigma}^{K_{0}})$ and the critical points of $\mathscr{T}_{\sigma}$. Furthermore, an analogously symmetric criticality principle of Lemma 3.1 clearly holds; hence the weak solutions of problem $(\mathscr{P}_{\sigma}^{K_{0}})$ are exactly the critical points of the functional $\mathscr{T}_{\sigma}$.

Lemma 4.1

Suppose that (h.1) and (h.2) hold. Then there exists a positive constant M depending on N, a, b, q, and $\Vert h\Vert _{\theta}$, such that any bounded sequence $\{(u_{n}, v_{n})\}\subset(\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$ satisfying

$$ \begin{aligned}&\mathscr{T}_{\sigma}(u_{n}, v_{v})\rightarrow c< \frac{1+a-b}{N} K_{0}^{-\frac{N-2(1+a-b)}{2(1+a-b)}}S_{\mu, l}^{\frac{N}{2(1+a-b)}}-M \sigma^{\frac{2}{2-q}}, \\ &\mathscr{T}_{\sigma}^{\prime}(u_{n}, v_{v})\rightarrow0\quad (n\rightarrow\infty) \end{aligned} $$

(4.3)

contains a convergent subsequence.

Proof

Since $\{(u_{n}, v_{n})\}$ is bounded in $(\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$, we obtain a subsequence, still denoted by $\{(u_{n}, v_{n})\}$, satisfying $(u_{n}, v_{n})\rightharpoonup (u, v)$ in $(\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$, $(u_{n}, v_{n})\rightarrow(u, v)$ a.e. in $\mathbb{R}^{N}$ and $(u_{n}, v_{n})\rightarrow(u, v)$ in $(L_{\mathrm{loc}}^{r}(\mathbb{R}^{N}))^{2}$ for all $r\in[1, 2^{\ast})$. Moreover, using (h.2), the Hölder inequality and the Lebesgue dominated theorem, we may also assume

$$ \lim_{n\rightarrow \infty} \int_{\mathbb{R}^{N}}h(x)\vert x\vert ^{-dp_{d}^{\ast}} \bigl(\bigl\vert u_{n}^{+}\bigr\vert ^{q} +\bigl\vert v_{n}^{+}\bigr\vert ^{q} \bigr)\,dx = \int_{\mathbb{R}^{N}}h(x)\vert x\vert ^{-dp_{d}^{\ast}} \bigl(\bigl\vert u^{+}\bigr\vert ^{q} +\bigl\vert v^{+}\bigr\vert ^{q} \bigr)\,dx. $$

(4.4)

By (4.4) and the standard argument, we easily check that $(u, v)$ is a critical point of $\mathscr{T}_{\sigma}$. Therefore, we deduce from (h.2), (4.1), (4.2), the Hölder inequality, and the fact that $1< q<2<p_{d}^{\ast}\leq p_{b}^{\ast}$ that

$$ \begin{aligned}[b] \mathscr{T}_{\sigma}(u, v)&=\mathscr{T}_{\sigma}(u, v)-\frac{1}{p_{b}^{\ast}}\bigl\langle \mathscr{T}_{\sigma}^{\prime}(u, v), (u, v)\bigr\rangle \\ &=\frac{1+a-b}{N}\bigl\Vert (u, v)\bigr\Vert _{\mu}^{2}- \frac{p_{b}^{\ast}-q}{qp_{b}^{\ast}}\sigma \int_{\mathbb{R}^{N}}h(x)\vert x\vert ^{-dp_{d}^{\ast}} \bigl(\bigl\vert u^{+}\bigr\vert ^{q} +\bigl\vert v^{+}\bigr\vert ^{q} \bigr)\,dx \\ &\geq\frac{1+a-b}{N} \bigl(\Vert u\Vert _{\mu}^{2}+ \Vert v\Vert _{\mu}^{2} \bigr) -\frac{p_{b}^{\ast}-q}{qp_{b}^{\ast}}C\sigma \Vert h\Vert _{\theta} \bigl(\Vert u\Vert _{\mu}^{q}+ \Vert v\Vert _{\mu}^{q} \bigr) \\ &\geq-(2-q) \biggl[\frac{qN}{2(1+a-b)} \biggr]^{\frac{q}{2-q}} \biggl( \frac{p_{b}^{\ast}-q}{qp_{b}^{\ast}}C \Vert h\Vert _{\theta} \biggr)^{\frac{2}{2-q}} \sigma^{\frac{2}{2-q}} \\ &\triangleq-M\sigma^{\frac{2}{2-q}}, \end{aligned} $$

(4.5)

where $M=(2-q)[\frac{qN}{2(1+a-b)}]^{q/(2-q)} (\frac{p_{b}^{\ast}-q}{qp_{b}^{\ast}}C \Vert h\Vert _{\theta})^{2/(2-q)}$ is a positive constant. Now we set $\widetilde{u}_{n}=u_{n}-u$ and $\widetilde{v}_{n}=v_{n}-v$. Then, by the Brezis-Lieb lemma [30] and arguing as in [31], Lemma 2.1, we obtain

$$\begin{aligned}& \bigl\Vert (\widetilde{u}_{n}, \widetilde{v}_{n}) \bigr\Vert _{\mu}^{2} = \bigl\Vert (u_{n}, v_{n})\bigr\Vert _{\mu}^{2}-\bigl\Vert (u, v)\bigr\Vert _{\mu}^{2}+o_{n}(1), \end{aligned}$$

(4.6)

$$\begin{aligned}& \int_{\mathbb{R}^{N}}\vert x\vert ^{-bp_{b}^{\ast}}\vert \widetilde {u}_{n}^{+}\vert ^{p_{b}^{\ast}}\,dx = \int_{\mathbb{R}^{N}}\vert x\vert ^{-bp_{b}^{\ast}}\bigl\vert u_{n}^{+}\bigr\vert ^{p_{b}^{\ast}}\,dx - \int_{\mathbb{R}^{N}}\vert x\vert ^{-bp_{b}^{\ast}}\bigl\vert u^{+}\bigr\vert ^{p_{b}^{\ast}}\,dx+o_{n}(1), \end{aligned}$$

(4.7)

$$\begin{aligned}& \int_{\mathbb{R}^{N}}\vert x\vert ^{-bp_{b}^{\ast}}\vert \widetilde {v}_{n}^{+}\vert ^{p_{b}^{\ast}}\,dx = \int_{\mathbb{R}^{N}}\vert x\vert ^{-bp_{b}^{\ast}}\bigl\vert v_{n}^{+}\bigr\vert ^{p_{b}^{\ast}}\,dx - \int_{\mathbb{R}^{N}}\vert x\vert ^{-bp_{b}^{\ast}}\bigl\vert v^{+}\bigr\vert ^{p_{b}^{\ast}}\,dx+o_{n}(1), \end{aligned}$$

(4.8)

$$\begin{aligned}& \int_{\mathbb{R}^{N}}\vert x\vert ^{-bp_{b}^{\ast}}\bigl\vert \widetilde{u}_{n}^{+}\bigr\vert ^{\alpha_{i}} \bigl\vert \widetilde{v}_{n}^{+}\bigr\vert ^{\beta_{i}} \,dx = \int_{\mathbb{R}^{N}}\vert x\vert ^{-bp_{b}^{\ast}}\bigl\vert u_{n}^{+}\bigr\vert ^{\alpha_{i}} \bigl\vert v_{n}^{+}\bigr\vert ^{\beta_{i}}\,dx \\& \hphantom{\int_{\mathbb{R}^{N}}\vert x\vert ^{-bp_{b}^{\ast}}\bigl\vert \widetilde{u}_{n}^{+}\bigr\vert ^{\alpha_{i}} \bigl\vert \widetilde{v}_{n}^{+}\bigr\vert ^{\beta_{i}} \,dx =} {}- \int_{\mathbb{R}^{N}}\vert x\vert ^{-bp_{b}^{\ast}}\bigl\vert u^{+}\bigr\vert ^{\alpha_{i}} \bigl\vert v^{+}\bigr\vert ^{\beta_{i}}\,dx+o_{n}(1),\quad i=1, \ldots, l. \end{aligned}$$

(4.9)

According to $\mathscr{T}_{\sigma}(u_{n}, v_{n})=c+o_{n}(1)$ and $\mathscr{T}_{\sigma}^{\prime}(u_{n}, v_{n})=o_{n}(1)$, we find from (4.1), (4.4) and (4.6)-(4.9) that

$$ \begin{aligned}[b] c+o_{n}(1)={}&\mathscr{T}_{\sigma}(u_{n}, v_{n})\\ ={}&\mathscr{T}_{\sigma}(u, v)+ \frac{1}{2} \bigl\Vert (\widetilde{u}_{n}, \widetilde{v}_{n})\bigr\Vert _{\mu}^{2} \\ &{}-\frac{K_{0}}{p_{b}^{\ast}} \int_{\mathbb{R}^{N}} \vert x\vert ^{-bp_{b}^{\ast}} \Biggl(\bigl\vert \widetilde{u}_{n}^{+}\bigr\vert ^{p_{b}^{\ast}} +\sum _{i=1}^{l}\varsigma_{i} \bigl\vert \widetilde{u}_{n}^{+}\bigr\vert ^{\alpha_{i}} \bigl\vert \widetilde{v}_{n}^{+}\bigr\vert ^{\beta_{i}} +\bigl\vert \widetilde{v}_{n}^{+}\bigr\vert ^{p_{b}^{\ast}} \Biggr)\,dx+o_{n}(1) \end{aligned} $$

(4.10)

and

$$ \bigl\Vert (\widetilde{u}_{n}, \widetilde{v}_{n}) \bigr\Vert _{\mu}^{2} -K_{0} \int_{\mathbb{R}^{N}} \vert x\vert ^{-bp_{b}^{\ast}} \Biggl(\bigl\vert \widetilde{u}_{n}^{+}\bigr\vert ^{p_{b}^{\ast}} +\sum _{i=1}^{l}\varsigma_{i} \bigl\vert \widetilde{u}_{n}^{+}\bigr\vert ^{\alpha_{i}} \bigl\vert \widetilde{v}_{n}^{+}\bigr\vert ^{\beta_{i}} +\bigl\vert \widetilde{v}_{n}^{+}\bigr\vert ^{p_{b}^{\ast}} \Biggr)\,dx=o_{n}(1). $$

(4.11)

Hence, for a subsequence $\{(\widetilde{u}_{n}, \widetilde{v}_{n})\}$, we find

$$\bigl\Vert (\widetilde{u}_{n}, \widetilde{v}_{n})\bigr\Vert _{\mu}^{2} \rightarrow\widetilde{\xi}\geq0,\qquad K_{0} \int_{\mathbb{R}^{N}} \vert x\vert ^{-bp_{b}^{\ast}} \Biggl(\bigl\vert \widetilde{u}_{n}^{+}\bigr\vert ^{p_{b}^{\ast}} +\sum _{i=1}^{l}\varsigma_{i} \bigl\vert \widetilde{u}_{n}^{+}\bigr\vert ^{\alpha_{i}} \bigl\vert \widetilde{v}_{n}^{+}\bigr\vert ^{\beta_{i}} +\bigl\vert \widetilde{v}_{n}^{+}\bigr\vert ^{p_{b}^{\ast}} \Biggr)\,dx\rightarrow \widetilde{\xi} $$

as $n\rightarrow\infty$. It follows from (3.3) that $S_{\mu, l}(\widetilde{\xi}/K_{0})^{\frac{2}{p_{b}^{\ast}}}\leq \widetilde{\xi}$, which implies either $\widetilde{\xi}=0$ or $\widetilde{\xi}\geq K_{0}^{-\frac{N-2(1+a-b)}{2(1+a-b)}}S_{\mu, l}^{\frac{N}{2(1+a-b)}}$. If $\widetilde{\xi}\geq K_{0}^{-\frac{N-2(1+a-b)}{2(1+a-b)}}S_{\mu, l}^{\frac{N}{2(1+a-b)}}$, then we conclude from (4.5), (4.10), and (4.11) that

$$c=\mathscr{T}_{\sigma}(u, v)+ \biggl(\frac{1}{2}- \frac{1}{p_{b}^{\ast}} \biggr)\widetilde{\xi} \geq\frac{1+a-b}{N} K_{0}^{-\frac{N-2(1+a-b)}{2(1+a-b)}}S_{\mu, l}^{\frac{N}{2(1+a-b)}}-M \sigma^{\frac{2}{2-q}}, $$

which contradicts (4.3). Therefore, we have $\Vert (\widetilde{u}_{n}, \widetilde{v}_{n})\Vert _{\mu}^{2}\rightarrow 0$ as $n\rightarrow\infty$, and hence, $(u_{n}, v_{n})\rightarrow (u, v)$ in $(\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$. The lemma is proved. □

Lemma 4.2

Suppose that (h.1) and (h.2) hold. Then there exists $\sigma_{1}^{\ast}>0$ such that for any $\sigma\in(0, \sigma_{1}^{\ast})$ the following geometric conditions for $\mathscr{T}_{\sigma}(u, v)$ hold:

• (i) $\mathscr{T}_{\sigma}(0, 0)=0$; there exist constants $\widetilde{\alpha}>0$ and $\rho>0$ such that $\mathscr{T}_{\sigma}(u, v)\geq\widetilde{\alpha}$ for all $\Vert (u, v)\Vert _{\mu}=\rho$;

• (ii) there exists $(e_{u}, e_{v})\in (\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}$ such that $\Vert (e_{u}, e_{v})\Vert _{\mu}>\rho$ and $\mathscr{T}_{\sigma}(e_{u}, e_{v})< 0$.

Proof

In view of (h.1) and (h.2), for all $0<\varsigma_{0}<\frac{1}{2}$, we obtain from (2.1), (3.3), (4.1), and (4.2) and the Hölder inequality that

$$ \begin{aligned}[b] \mathscr{T}_{\sigma}(u, v)&\geq\frac{1}{2} \bigl\Vert (u, v)\bigr\Vert _{\mu}^{2}-\frac{K_{0}}{p_{b}^{\ast}}S_{\mu, l}^{-\frac{p_{b}^{\ast}}{2}} \bigl\Vert (u, v)\bigr\Vert _{\mu}^{p_{b}^{\ast}}-\frac{\sigma}{q}C \Vert h\Vert _{\theta} \bigl\Vert (u, v)\bigr\Vert _{\mu}^{q} \\ &\geq \biggl(\frac{1}{2}-\varsigma_{0} \biggr)\bigl\Vert (u, v) \bigr\Vert _{\mu}^{2}-\frac{K_{0}}{p_{b}^{\ast}}S_{\mu, l}^{-\frac{p_{b}^{\ast}}{2}} \bigl\Vert (u, v)\bigr\Vert _{\mu}^{p_{b}^{\ast}}-C(\varsigma_{0}) \sigma^{\frac{2}{2-q}}, \end{aligned} $$

(4.12)

where $C(\varsigma_{0})=(\frac{2}{q}-1)\varsigma_{0} [C\Vert h\Vert _{\theta}/(2\varsigma_{0})]^{2/(2-q)}>0$ is a constant depending on $\varsigma_{0}\in(0, \frac{1}{2})$. By the last inequality in (4.12) and the fact $p_{b}^{\ast}>2$, we conclude that, for small $\varsigma_{0}$, there exist constants $\widetilde{\alpha}>0$, $\rho>0$, and $\sigma_{1}^{\ast}>0$ such that $\mathscr{T}_{\sigma}(u, v)\geq\widetilde{\alpha}>0$ for all $\Vert (u, v)\Vert _{\mu}=\rho$ and $0<\sigma<\sigma_{1}^{\ast}$. On the other hand, since $\int_{\mathbb{R}^{N}}h(x)\vert x\vert ^{-dp_{d}^{\ast }}(\vert u^{+}\vert ^{q}+\vert v^{+}\vert ^{q})\,dx\geq 0$, we deduce from (4.1) that there exists $(u, v)\in (\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N})\backslash\{0\})^{2}$ such that $\mathscr{T}_{\sigma}(tu, tv)\rightarrow-\infty$ as $t\rightarrow+\infty$. Thus the conclusion of this lemma follows. □

Lemma 4.3

Suppose that (h.1) and (h.2) hold. Then there exists $\sigma_{2}^{\ast}>0$ such that

$$ \sup_{t\geq0}\mathscr{T}_{\sigma} (ty_{\epsilon}, t\tau_{\min}y_{\epsilon} ) < \frac{1+a-b}{N} K_{0}^{-\frac{N-2(1+a-b)}{2(1+a-b)}}S_{\mu, l}^{\frac{N}{2(1+a-b)}}-M \sigma^{\frac{2}{2-q}} $$

(4.13)

for any $\sigma\in(0, \sigma_{2}^{\ast})$ and small $\epsilon>0$, where $\tau_{\min}>0$ satisfies (3.4)-(3.6) and $M>0$ is given in Lemma  4.1.

Proof

We follow the arguments of [32], Theorem 3. First, we define the functions

$$ \begin{aligned}[b] \Psi(t)={}&\mathscr{T}_{\sigma} (ty_{\epsilon}, t \tau_{\min}y_{\epsilon} )\\ ={}&\frac{1+\tau_{\min}^{2}}{2} t^{2}- \frac{K_{0}}{p_{b}^{\ast}} \Biggl(1+\sum_{i=1}^{l} \varsigma_{i} \tau_{\min}^{\beta_{i}}+\tau_{\min}^{p_{b}^{\ast}} \Biggr)t^{p_{b}^{\ast}} \int_{\mathbb{R}^{N}}\frac{y_{\epsilon}^{p_{b}^{\ast}}}{ \vert x\vert ^{bp_{b}^{\ast}}}\,dx \\ &{}-\frac{\sigma }{q} \bigl(1+\tau_{\min}^{q} \bigr)t^{q} \int_{\mathbb{R}^{N}}h(x) \frac{y_{\epsilon}^{q}}{\vert x\vert ^{dp_{d}^{\ast}}}\,dx,\quad t\geq0, \end{aligned} $$

(4.14)

and

$$ \widetilde{\Psi}(t)=\frac{1+\tau_{\min}^{2}}{2} t^{2}- \frac{K_{0}}{p_{b}^{\ast}} \Biggl(1+\sum_{i=1}^{l} \varsigma_{i} \tau_{\min}^{\beta_{i}}+\tau_{\min}^{p_{b}^{\ast}} \Biggr)t^{p_{b}^{\ast}} \int_{\mathbb{R}^{N}}\frac{y_{\epsilon}^{p_{b}^{\ast}}}{ \vert x\vert ^{bp_{b}^{\ast}}}\,dx,\quad t\geq0. $$

(4.15)

Note that $\widetilde{\Psi}(0)=0$, $\widetilde{\Psi}(t)>0$ for $t\rightarrow0^{+}$, and $\lim_{t\rightarrow+\infty}\widetilde{\Psi}(t)=-\infty$. Thus $\sup_{t\geq0}\widetilde{\Psi}(t)$ can be achieved at some finite $\widetilde{t}_{\epsilon}>0$ at which $\widetilde{\Psi}^{\prime}(t)$ becomes zero. By simple calculation, we obtain from (2.6), (3.4), (3.5), and Lemma 3.2 that

$$ \begin{aligned}[b] \sup_{t\geq 0}\widetilde{\Psi}(t)&= \widetilde{\Psi}(\widetilde{t}_{\epsilon}) = \biggl(\frac{1}{2}- \frac{1}{p_{b}^{\ast}} \biggr) \biggl\{ \frac{1+\tau_{\min}^{2}}{ [ (1 +\sum_{i=1}^{l}\varsigma_{i}\tau_{\min}^{\beta}+\tau_{\min }^{p_{b}^{\ast}} )K_{0}\int_{\mathbb{R}^{N}} \frac{y_{\epsilon}^{p_{b}^{\ast}}}{ \vert x\vert ^{bp_{b}^{\ast}}}\,dx ]^{\frac {2}{p_{b}^{\ast}}}} \biggr\} ^{\frac{p_{b}^{\ast}}{p_{b}^{\ast}-2}} \\ &=\frac{1+a-b}{N}K_{0}^{-\frac{N-2(1+a-b)}{2(1+a-b)}}S_{\mu, l}^{\frac{N}{2(1+a-b)}}. \end{aligned} $$

(4.16)

Let $\widetilde{\sigma}>0$ be such that

$$\frac{1+a-b}{N}K_{0}^{-\frac{N-2(1+a-b)}{2(1+a-b)}}S_{\mu, l}^{\frac{N}{2(1+a-b)}}-M \sigma^{\frac{2}{2-q}}>0,\quad \forall\sigma\in(0, \widetilde{\sigma}). $$

Then we conclude from (h.2) and (4.14) that

$$\Psi(t)=\mathscr{T}_{\sigma} (ty_{\epsilon}, t\tau_{\min}y_{\epsilon} )\leq\frac{1+\tau_{\min}^{2}}{2}t^{2},\quad \forall t\geq0, \sigma>0, $$

and there exists $T_{0}\in(0, 1)$ independent of ϵ such that

$$ \begin{aligned}[b]\sup_{0\leq t\leq T_{0}}\Psi(t)&\leq\frac{1+\tau_{\min}^{2}}{2}T_{0}^{2}\\ &< \frac{1+a-b}{N}K_{0}^{-\frac{N-2(1+a-b)}{2(1+a-b)}}S_{\mu, l}^{\frac{N}{2(1+a-b)}}-M \sigma^{\frac{2}{2-q}},\quad \forall\sigma\in(0, \widetilde{\sigma}). \end{aligned} $$

(4.17)

Moreover, we deduce from (4.14), (4.15), and (4.16) that

$$ \begin{aligned}[b] \sup_{t\geq T_{0}}\Psi(t)&\leq\sup _{t\geq 0}\widetilde{\Psi}(t)-\frac{\sigma}{q} \bigl(1+ \tau_{\min}^{q} \bigr)T_{0}^{q} \int_{\mathbb{R}^{N}} h(x)\vert x\vert ^{-dp_{d}^{\ast}}y_{\epsilon}^{q} \,dx \\ &=\frac{1+a-b}{N}K_{0}^{-\frac{N-2(1+a-b)}{2(1+a-b)}}S_{\mu, l}^{\frac{N}{2(1+a-b)}}- \frac{\sigma}{q} \bigl(1+\tau_{\min}^{q} \bigr)T_{0}^{q} \int_{\mathbb{R}^{N}}h(x)\vert x\vert ^{-dp_{d}^{\ast}}y_{\epsilon}^{q} \,dx. \end{aligned} $$

(4.18)

Now, taking $\sigma>0$ such that

$$-\frac{\sigma}{q} \bigl(1+\tau_{\min}^{q} \bigr)T_{0}^{q} \int_{\mathbb{R}^{N}}h(x)\vert x\vert ^{-dp_{d}^{\ast}}y_{\epsilon}^{q} \,dx < -M\sigma^{\frac{2}{2-q}}, $$

that is,

$$0< \sigma< \biggl(\frac{1+\tau_{\min}^{q}}{qM}T_{0}^{q} \int_{\mathbb{R}^{N}}h(x)\vert x\vert ^{-dp_{d}^{\ast}} y_{\epsilon}^{q}\,dx \biggr)^{\frac{2-q}{2}}\triangleq\overline{ \sigma}, $$

we obtain from (4.18)

$$ \sup_{t\geq T_{0}}\Psi(t)< \frac{1+a-b}{N}K_{0}^{-\frac{N-2(1+a-b)}{2(1+a-b)}}S_{\mu, l}^{\frac{N}{2(1+a-b)}}-M \sigma^{\frac{2}{2-q}},\quad \forall\sigma\in(0, \overline{\sigma}). $$

(4.19)

Choosing $\sigma_{2}^{\ast}=\min\{\widetilde{\sigma}, \overline{\sigma}\}$, we conclude from (4.17) and (4.19) that

$$\sup_{t\geq0}\Psi(t) < \frac{1+a-b}{N}K_{0}^{-\frac{N-2(1+a-b)}{2(1+a-b)}}S_{\mu, l}^{\frac{N}{2(1+a-b)}}-M \sigma^{\frac{2}{2-q}},\quad \forall\sigma\in\bigl(0, \sigma_{2}^{\ast} \bigr), $$

which implies (4.13). Therefore the results of this lemma follow. □

Proof of Theorem 2.3

Taking $\rho>0$ and $\sigma^{\ast}=\min\{\sigma_{1}^{\ast}, \sigma_{2}^{\ast}\}$, for $0<\sigma<\sigma^{\ast}$, given in the proofs of Lemmas 4.2 and 4.3, we define

$$\overline{B_{\rho}(0)}=\bigl\{ (u, v)\in\bigl(\mathscr{D}_{a, G}^{1, 2} \bigl(\mathbb{R}^{N}\bigr)\bigr)^{2}; \bigl\Vert (u, v) \bigr\Vert _{\mu}\leq\rho\bigr\} \quad\text{and}\quad c_{1} \triangleq\inf_{\overline{B_{\rho}(0)}}\mathscr{T}_{\sigma}(u, v). $$

Since the metric space $\overline{B_{\rho}(0)}$ is complete, we conclude from the Ekeland variational principle [33] that there exists a sequence $\{(u_{n}, v_{n})\}\subset\overline{B_{\rho}(0)}$ such that $\mathscr {T}_{\sigma}(u_{n}, v_{n})\rightarrow c_{1}$ and $\mathscr {T}_{\sigma}^{\prime}(u_{n}, v_{n})\rightarrow0$ as $n\rightarrow\infty$.

Let $\varphi_{0}$, $\psi_{0}\in\mathscr {C}_{0}^{\infty}(\mathbb{R}^{N})$ be the G-symmetric functions such that $\varphi_{0}$, $\psi_{0}> 0$. In view of (h.1) and (h.2), we find $\int_{\mathbb{R}^{N}}h(x)\vert x\vert ^{-dp_{d}^{\ast}}( \varphi_{0}^{q}+\psi_{0}^{q})\,dx>0$. Hence we conclude from $1< q<2<p_{b}^{\ast}$ that there exists $\overline{t}_{0}=\overline{t}_{0}(\varphi_{0}, \psi_{0})>0$ sufficiently small such that

$$\begin{aligned} \mathscr{T}_{\sigma}(\overline{t}_{0} \varphi_{0}, \overline{t}_{0}\psi_{0})={}& \frac{\overline{t}_{0}^{2}}{2}\bigl\Vert (\varphi_{0}, \psi_{0})\bigr\Vert _{\mu}^{2}-\frac{K_{0}}{p_{b}^{\ast}} \overline{t}_{0}^{p_{b}^{\ast}} \int_{\mathbb{R}^{N}}\vert x\vert ^{-bp_{b}^{\ast }} \Biggl( \varphi_{0}^{p_{b}^{\ast}} +\sum_{i=1}^{l} \varsigma_{i}\varphi_{0}^{\alpha_{i}} \psi_{0}^{\beta_{i}}+\psi_{0}^{p_{b}^{\ast}} \Biggr)\,dx \\ &{}-\frac{\sigma}{q}\overline{t}_{0}^{q} \int_{\mathbb {R}^{N}}h(x)\vert x\vert ^{-dp_{d}^{\ast}} \bigl( \varphi_{0}^{q}+\psi_{0}^{q} \bigr) \,dx\\ < {}&0.\end{aligned} $$

This implies

$$c_{1}< 0< \frac{1+a-b}{N}K_{0}^{-\frac{N-2(1+a-b)}{2(1+a-b)}}S_{\mu, l}^{\frac{N}{2(1+a-b)}}-M \sigma^{\frac{2}{2-q}},\quad \forall\sigma\in\bigl(0, \sigma^{\ast}\bigr). $$

According to Lemma 4.1, $\mathscr{T}_{\sigma}$ possesses a critical point $(u_{1}, v_{1})$ with $\mathscr{T}_{\sigma}(u_{1}, v_{1})=c_{1}<0$. Taking $(u_{1}^{-}, v_{1}^{-})$ as a pair of test functions, where $u_{1}^{-}=\min\{0, u_{1}\}$ and $v_{1}^{-}=\min\{0, v_{1}\}$, we get from (4.1) that $0=\langle\mathscr{T}_{\sigma}^{\prime}(u_{1}, v_{1}), (u_{1}^{-}, v_{1}^{-})\rangle=\Vert (u_{1}^{-}, v_{1}^{-})\Vert _{\mu}^{2}$. This means $u_{1}\geq0$ and $v_{1}\geq0$ in $\mathbb{R}^{N}$. By the strong maximum principle and the symmetric criticality principle, we conclude that $(u_{1}, v_{1})$ is a positive G-symmetric solution of problem $(\mathscr{P}_{\sigma}^{K_{0}})$.

On the other hand, we define

$$c_{2}\triangleq\inf_{\gamma\in\Gamma}\max_{t\in[0, 1]} \mathscr{T}_{\sigma} \bigl(\gamma(t) \bigr), $$

where $\Gamma=\{\gamma\in\mathscr{C}([0, 1], (\mathscr{D}_{a, G}^{1, 2}(\mathbb{R}^{N}))^{2}); \gamma(0)=(0, 0), \gamma(1)=(e_{u}, e_{v})\}$. It follows from Lemmas 4.2 and 4.3 that

$$0< \widetilde{\alpha}\leq c_{2}< \frac{1+a-b}{N}K_{0}^{-\frac{N-2(1+a-b)}{2(1+a-b)}}S_{\mu, l}^{\frac{N}{2(1+a-b)}}-M \sigma^{\frac{2}{2-q}},\quad \forall\sigma\in\bigl(0, \sigma^{\ast}\bigr). $$

Therefore $c_{2}$ is a critical value of $\mathscr{T}_{\sigma}$ by the mountain pass theorem. Similar to the arguments above, problem $(\mathscr{P}_{\sigma}^{K_{0}})$ admits another positive G-symmetric solution $(u_{2}, v_{2})$ with $\mathscr{T}_{\sigma}(u_{2}, v_{2})=c_{2}>0$. □