Multiple positive solutions for Schrödinger-Poisson system with singularity on the Heisenberg group

Abstract

In this work, we study the following Schrödinger-Poisson system

$$ \textstyle\begin{cases} -\Delta _{H}u+\mu \phi u=\lambda u^{-\gamma}, &\text{in } \Omega , \\ -\Delta _{H}\phi =u^{2}, &\text{in } \Omega , \\ u>0, &\text{in } \Omega , \\ u=\phi =0, &\text{on } \partial \Omega , \end{cases} $$

where $\Delta _{H}$ is the Kohn-Laplacian on the first Heisenberg group $\mathbb{H}^{1}$, and $\Omega \subset \mathbb{H}^{1}$ is a smooth bounded domain, $\mu =\pm 1$, $0<\gamma <1$, and $\lambda >0$ are some real parameters. For the above system, we prove the existence and uniqueness of positive solution for $\mu =1$ and each $\lambda >0$. Multiple solutions of the system are also considered for $\mu =-1$ and $\lambda >0$ small enough using the critical point theory for nonsmooth functional.

1 Introduction and main results

This paper consider the following singular Schrödinger-Poisson system

$$ \textstyle\begin{cases} -\Delta _{H} u+\mu \phi u=\lambda u^{-\gamma}, &\text{in } \Omega , \\ -\Delta _{H}\phi =u^{2}, &\text{in } \Omega , \\ u>0, &\text{in } \Omega , \\ u=\phi =0, &\text{on } \partial \Omega , \end{cases} $$

(1.1)

where $\Delta _{H}$ is the Kohn-Laplacian on the first Heisenberg group $\mathbb{H}^{1}$, Ω is a smooth bounded domain of $\mathbb{H}^{1}$, $\mu =\pm 1$, $0<\gamma <1$, and $\lambda >0$ are some real parameters.

Over the years, many scholars have been widely studied the Heisenberg group due to its crucial role in several branches of mathematics, such as quantum mechanics, complex variables, and harmonic analysis, so one can refer to [9, 12, 19] and the references therein.

In 2022, Liu et al. [15] investigated the following Schrödinger-Poisson system on the Heisenberg group

$$ \textstyle\begin{cases} -(a-b\int _{\Omega} \vert \nabla _{H} u \vert ^{2}\,d\xi )\Delta _{H} u+\mu \phi u= \lambda \vert u \vert ^{q-2}u+ \vert u \vert ^{2}u, &\text{in } \Omega , \\ -\Delta _{H}\phi =u^{2}, &\text{in } \Omega , \\ u=\phi =0, &\text{on } \partial \Omega , \end{cases} $$

(1.2)

where $\Omega \subset \mathbb{H}^{1}$ is a smooth bounded domain, $a, b>0$, $1< q<2$ or $2< q<4$, $\lambda >0$, and $\mu \in \mathbb{R}$ are some real parameters. They obtained the existence and multiplicity of solutions. In particular, when $a=1$, $b=0$, An and Liu in [1] established the existence and multiplicity of solutions of problem (1.2). Using the Green representation formula, the concentration compactness, and the critical point theory, they proved that the above system has at least two positive solutions for $\mu < S\times \operatorname{meas}(\Omega )^{-\frac{1}{2}}$ and λ small enough. In addition, they also established that there is a positive ground-state solution for (1.2).

Lei and Liao [13] considered the following system

$$ \textstyle\begin{cases} -\Delta u+\lambda \phi u=\frac{\lambda}{ \vert x \vert ^{\beta}u^{\gamma}}+ \vert u \vert ^{4}u, &\text{in } \Omega , \\ -\Delta \phi =u^{2}, &\text{in } \Omega , \\ u>0, &\text{in } \Omega , \\ u=\phi =0, &\text{on } \partial \Omega , \end{cases} $$

where $0<\gamma <1$, $0\leq \beta <\frac{5+\gamma}{2}$ and $\lambda >0$ is parameter, they obtained two positive solutions using the variational method and the Nehari manifold method.

In [20], Pucci and Ye studied the logarithmic and critical nonlinearities for the Kirchhoff-Poisson system

$$ \textstyle\begin{cases} -M(\int _{\Omega} \vert \nabla _{H}u \vert ^{2}\,d\xi )\Delta _{H} u+\mu \phi u= \vert u \vert ^{2}u+ \lambda \vert u \vert ^{q-2}u \ln \vert u \vert ^{2}, &\text{in } \Omega , \\ -\Delta _{H}\phi =u^{2}, &\text{in } \Omega , \\ u=\phi =0, &\text{on } \partial \Omega , \end{cases} $$

where Ω is a smooth bounded domain of $\mathbb{H}^{1}$, $q\in (2\theta ,4)$, $\mu \in \mathbb{R}$, and $\lambda >0$ are some real parameters. Under suitable assumptions on the Kirchhoff function M, covering the degenerate case, they proved the existence of nontrivial solutions for the above system when $\lambda >0$ is sufficiently large. For more on the results of the Heisenberg group, we refer the reader to [2, 8, 10, 14, 16–18, 21] and the references therein.

Furthermore, for the system (1.1) in the Heisenberg group, there is no result that this paper answers positively. Before giving the theorem, we define the solutions of (1.1) if u satisfies

$$ \int _{\Omega}\nabla _{H}u\nabla _{H} v\, d\xi +\mu \int _{ \Omega}\phi _{u}uv\, d\xi -\lambda \int _{\Omega}\frac{v}{u^{\gamma}}\,d\xi =0,\quad \forall v\in S^{1}_{0}(\Omega ), $$

we say that u is a solution of problem (1.1).

Theorem 1.1

Assume that $0<\gamma <1$, $\mu =1$ and $\lambda >0$, then system (1.1) has a unique solution.

Theorem 1.2

Assume that $0<\gamma <1$ and $\mu =-1$, then there exists $\Lambda _{0}>0$ such that for every $\lambda \in (0,\Lambda _{0})$, system (1.1) has at least two positive solutions.

Remark 1.3

Our approach is novel, unlike the Euclidean case, since the presence of singular terms gives us great difficulties; the critical point theory for nonsmooth functional is used to overcome the difficulties, generalizing the results of literature [22].

2 Some preliminary results

In this section, we review the Heisenberg group. For more results, see [7, 11]. Let $\mathbb{H}^{1}$ be the Heisenberg group of topological dimension 3, that is, the Lie group where underlying manifold is $\mathbb{R}^{3}$, endowed with the non-Abelian law

$$ \begin{aligned} \tau : {\mathbb{H}^{1}}\rightarrow { \mathbb{H}^{1}}, \qquad \tau _{\xi}\bigl(\xi '\bigr)=\xi \circ \xi ', \end{aligned} $$

where

$$ \begin{aligned} \xi \circ \xi '=\bigl(x+x',y+y',t+t'+2 \bigl(x'y-xy'\bigr)\bigr), \end{aligned} $$

for $\forall\xi , \xi '\in \mathbb{H}^{1}$, with $\xi =(x, y, t)$ and $\xi '=(x', y', t')$, satisfy the inverse operation. Consider the family of dilations on $\mathbb{H}^{1}$ defined by

$$ \begin{aligned} \delta _{s}(\xi )=\bigl(sx, sy, s^{2}t\bigr), \quad \forall \xi \in {\mathbb{H}^{1}}, \end{aligned} $$

so $\delta _{s}(\xi \circ \xi ')=\delta _{s}(\xi )\circ \delta _{s}(\xi ')$ (see [19]). The number $Q=4$ is the homogeneous dimension of $\mathbb{H}^{1}$, definition

$$ \begin{aligned} \bigl\vert B_{H}(\xi _{0}, r) \bigr\vert =\omega _{Q}r^{Q}, \end{aligned} $$

where $B_{H}(\xi _{0}, r)$ is the Heisenberg ball of radius r centered at $\xi _{0}$, i.e.,

$$ \begin{aligned} B_{H}(\xi _{0}, r)=\bigl\{ \xi \in {\mathbb{H}^{1}}:d_{H}(\xi _{0}, \xi )< r\bigr\} , \end{aligned} $$

$d_{H}(\xi _{0}, \xi )=|\xi ^{-1}\circ \xi _{0}|_{H}$ and $\omega _{Q}=|B_{H}(0, 1)|$.

The Kohn-Laplacian $\Delta _{H}$ on $\mathbb{H}^{1}$ is defined as

$$ \begin{aligned} \Delta _{H}u=\operatorname{div}_{H}( \nabla _{H}u), \end{aligned} $$

where $\nabla _{H}u=(Xu, Yu)$. Indeed, the vector fields

$$ X=\frac{\partial}{\partial x}+2y\frac{\partial}{\partial t},\qquad Y= \frac{\partial}{\partial y}-2x \frac{\partial}{\partial t},\quad \text{and}\quad T=\frac{\partial}{\partial t}, $$

are a basis of the Lie algebra of $\mathbb{H}^{1}$ thus constituting a set of left invariant vector fields on $\mathbb{H}^{1}$. Widely known that $\Delta _{H}$ is a degenerate elliptic operator, and the Bony maximum principle is satisfied (see [4]). In the present section, the existence and multiplicity of solutions of system (1.1), when $\mu =-1$, are studied. We prove that system (1.1) has two positive solutions using the critical point theory for nonsmooth functional and the variational method for $\lambda >0$ small enough.

Let us review critical points of nonsmooth functions related concepts. Let $(X,d)$ be a complete metric space with metric d and $f:X\rightarrow \mathbb{R}$ be a continuous functional in X. Denote by $|df|(u)$ the supremum of δ in $[0,\infty )$ such that there exist $r>0$ and a continuous map $\sigma :U\times [0,r]\rightarrow X$, satisfying

$$ \textstyle\begin{cases} f(\sigma (v,t))\leq f(v)-\delta t, &(v,t)\in U\times [0,r], \\ d(\sigma (v,t),v)\leq t, &(v,t)\in U\times [0,r]. \end{cases} $$

(2.1)

The number $|df|(u)$ is called the weak slope of f at u. Thus, $u\in X$ is a critical point of f if $|df|(u)=0$, and $c\in \mathbb{R}$ is a critical value of f if there exists a critical point $u\in X$ of f with $f(u)=c$.

Since we are solving for the positive solution of system (1.1), so consider the functional $I_{\lambda}$ defined on the closed positive cone $U^{+}$ of $S^{1}_{0}(\Omega )$, which is defined as

$$ \begin{aligned} U^{+}=\bigl\{ u\in S^{1}_{0}( \Omega ), u(x)\geq 0, \text{a.e. } x\in \Omega \bigr\} . \end{aligned} $$

The Hilbert space $S^{1}_{0}(\Omega )$ is defined as the closure of $C^{\infty}_{0}(\Omega )$ under the inner product $\langle u,v\rangle =\int _{\Omega}\nabla _{H} u\nabla _{H} v\, d\xi$. Accordingly, the norm is denoted by $\|u\|=\|u\|_{S^{1}_{0}(\Omega )}= (\int _{\Omega}|\nabla _{H} u|^{2}\,d\xi )^{\frac{1}{2}}$. The norm in $L^{p}(\Omega )$ is denoted by $\|u\|_{p}=(\int _{\Omega}|u|^{p}\,d\xi )^{\frac{1}{p}}$. The embedding $S^{1}_{0}(\Omega )\hookrightarrow L^{p}(\Omega )$ is continuous for $p\in [1, Q^{*}]$, where $Q^{*}=\frac{2Q}{Q-2}=4$ is the critical exponent in $\mathbb{H}^{1}$. Let us denote by $B_{\rho}$ and $S_{\rho}$ a closed ball and a sphere, respectively, of a center of zero and radius ρ. Let S be the best Sobolev constant, namely

$$ S=\inf_{u\in S^{1}_{0}(\mathbb{H}^{1})\backslash \{0\}} \frac{\int _{\mathbb{H}^{1}} \vert \nabla _{H} u \vert ^{2}\,d\xi }{ (\int _{\mathbb{H}^{1}} \vert u \vert ^{4}\,d\xi )^{\frac{1}{2}}}. $$

(2.2)

First, using the Lax-Milgram theorem, for each $u\in S^{1}_{0}(\Omega )$, there exists a unique solution $\phi _{u} \in S^{1}_{0}(\Omega )$, which satisfies the second equation of system (1.1). Then, system (1.1) is transformed into the following problem

$$ \textstyle\begin{cases} -\Delta _{H} u+\mu \phi _{u} u=\lambda u^{-\gamma}, &\text{in } \Omega , \\ u>0, &\text{in } \Omega , \\ u=0, &\text{on } \partial \Omega . \end{cases} $$

(2.3)

For problem (2.3), we define the functional

$$ \begin{aligned} I_{\lambda}(u)= \frac{1}{2} \Vert u \Vert ^{2}+\frac{\mu}{4} \int _{\Omega}\phi _{u}u^{2}\,d\xi - \frac{\lambda}{1-\gamma} \int _{\Omega} \vert u \vert ^{1-\gamma}\,d\xi . \end{aligned} $$

(2.4)

We know that the functional $I_{\lambda}$ is well defined and $I_{\lambda}\in C^{1}(S^{1}_{0}(\Omega ), \mathbb{R})$. Besides, we say that u is a weak solution of problem (2.3) if u satisfies

$$ \begin{aligned} \bigl\langle I'_{\lambda}(u),v \bigr\rangle &= \int _{\Omega} \nabla _{H}u\nabla _{H} v\, d\xi +\mu \int _{\Omega}\phi _{u}u\, vd\xi - \lambda \int _{\Omega}\frac{v}{u^{\gamma}}\,d\xi =0,\quad \forall v\in S^{1}_{0}( \Omega ). \end{aligned} $$

(2.5)

By the Hölder inequality and (2.2), we obtain

$$ \begin{aligned} \int _{\Omega} \vert u \vert ^{1-\gamma}\,d\xi \leq S^{-\frac{1-\gamma}{2}} \vert \Omega \vert ^{ \frac{3+\gamma}{4}} \Vert u \Vert ^{1-\gamma}. \end{aligned} $$

(2.6)

Lemma 2.1

(See [1])

For all $u\in S^{1}_{0}(\Omega )$, there exists a unique solution $\phi _{u} \in S^{1}_{0}(\Omega )$ of

$$ \textstyle\begin{cases} -\Delta _{H}\phi =u^{2}, &\textit{in } \Omega , \\ \phi =0, &\textit{on } \partial \Omega , \end{cases} $$

and

• (1) $\phi _{u}\geq 0$ and $\phi _{tu}=t^{2}\phi _{u}$ for each $t>0$;

• (2) If $u_{n}\rightharpoonup u$ in $S^{1}_{0}(\Omega )$, then $\phi _{u_{n}}\rightarrow \phi _{u}$ in $S^{1}_{0}(\Omega )$ and

  $$ \begin{aligned} \lim_{n\rightarrow \infty} \int _{\Omega}\phi _{u_{n}}u_{n}v\, d\xi = \int _{\Omega}\phi _{u}uv\, d\xi , \quad \forall v\in S^{1}_{0}(\Omega ); \end{aligned} $$

• (3) For all $u\in S^{1}_{0}(\Omega )$, there holds that

  $$ \begin{aligned} \int _{\Omega} \vert \nabla _{H}\phi _{u} \vert ^{2}\,d\xi = \int _{\Omega}\phi _{u}u^{2}\,d\xi \leq S^{-1} \Vert u \Vert ^{4}_{8/3}\leq S^{-3} \vert \Omega \vert ^{\frac{1}{2}} \Vert u \Vert ^{4}; \end{aligned} $$

• (4) For $u, v\in S^{1}_{0}(\Omega )$, $\int _{\Omega}(\phi _{u}u-\phi _{v}v)(u-v)\,d\xi \geq \frac{1}{2}\| \phi _{u}-\phi _{v}\|^{2}$.

Lemma 2.2

Assume that $u\in U^{+}$ and $|dI_{\lambda}|(u)<+\infty$. Then, for all $v\in U^{+}$, one obtains

$$ \begin{aligned} \lambda \int _{\Omega}\frac{v-u}{u^{\gamma}}\,d\xi \leq \int _{\Omega} \nabla _{H}u\nabla _{H}(v-u)\,d\xi - \int _{\Omega}\phi _{u}u(v-u)\,d\xi + \vert dI_{ \lambda} \vert (u) \Vert v-u \Vert . \end{aligned} $$

(2.7)

Proof

Let $u\neq v\in U^{+}$ and $\|v-u\|>2\delta$. Define $\sigma :U\times [0,\delta ]\rightarrow U^{+}$ by

$$ \begin{aligned} \sigma (z,t)=z+t\frac{v-z}{ \Vert v-z \Vert }, \end{aligned} $$

where U is a neighborhood of u, then $\|\sigma (z,t)-z\|=t$. By (2.1), there exists $(z,t)\in U\times [0,\delta ]$ such that $I_{\lambda}(\sigma (z,t))>I_{\lambda}(z)-ct$. Hence, we assume that there exist sequences $\{u_{n}\}\subset U^{+}$ and $\{t_{n} \} \subset [0,+\infty )$, such that $u_{n}\rightarrow u$, $t_{n} \rightarrow 0^{+}$, and

$$ I_{\lambda} \biggl(u_{n}+t_{n} \frac{v-u_{n}}{ \Vert v-u_{n} \Vert } \biggr)\geq I_{\lambda} (u_{n} )-ct_{n}. $$

That is say

$$ \begin{aligned} I_{\lambda} \bigl(u_{n}+s_{n} (v-u_{n} ) \bigr)\geq I_{\lambda} (u_{n} )-cs_{n} \Vert v-u_{n} \Vert , \end{aligned} $$

(2.8)

where $s_{n}=\frac{t_{n}}{ \Vert v-u_{n} \Vert } \rightarrow 0^{+}$ as $n\rightarrow \infty$. Dividing (2.8) by $s_{n}$, we deduce that

$$ \begin{aligned} &\frac{\lambda}{1-\gamma} \int _{\Omega} \frac{ [u_{n}+s_{n} (v-u_{n} ) ]^{1-\gamma}-u_{n}^{1-\gamma}}{s_{n}}\,d\xi \\ &\quad \leq \frac{1}{2} \frac{ \Vert u_{n}+s_{n} (v-u_{n} ) \Vert ^{2}- \Vert u_{n} \Vert ^{2}}{s_{n}} \\ &\quad\quad {} -\frac{1}{4} \int _{\Omega} \frac{\phi _{u_{n}+s_{n}(v-u_{n})}(u_{n}+s_{n}(v-u_{n}))^{2}-\phi _{u_{n}}u_{n}^{2}}{s_{n}}\,d\xi +c \Vert v-u_{n} \Vert . \end{aligned} $$

(2.9)

Further, we can infer that

$$ \begin{aligned} \int _{\Omega} \frac{ [u_{n}+s_{n} (v-u_{n} ) ]^{1-\gamma}-u_{n}^{1-\gamma}}{s_{n} (1-\gamma )}\,d\xi =& \int _{\Omega} \frac{ [u_{n}+s_{n} (v-u_{n} ) ]^{1-\gamma}- [ (1-s_{n} )u_{n} ]^{1-\gamma}}{s_{n} (1-\gamma )}\,d\xi \\ &+ \int _{\Omega} \frac{ [ (1-s_{n} )u_{n} ]^{1-\gamma}-u_{n}^{1-\gamma}}{s_{n} (1-\gamma )}\,d\xi \\ =& \int _{\Omega} \frac{ [u_{n}+s_{n} (v-u_{n} ) ]^{1-\gamma}- [ (1-s_{n} )u_{n} ]^{1-\gamma}}{s_{n} (1-\gamma )}\,d\xi \\ &+ \frac{ (1-s_{n} )^{1-\gamma}-1}{s_{n} (1-\gamma )} \int _{\Omega} \vert u_{n} \vert ^{1-\gamma}\,d\xi \\ =&I_{1n}+I_{2n}. \end{aligned} $$

(2.10)

By mean value theorem, one has

$$ I_{1n}= \int _{\Omega}\frac{\zeta _{n}^{-\gamma}s_{n}v}{s_{n}}\,d\xi = \int _{\Omega}\frac{v}{\zeta _{n}^{\gamma}}\,d\xi , $$

where $\zeta _{n}\in (u_{n}-s_{n} u_{n},u_{n}+s_{n} (v-u_{n} ) )$, that is $\zeta _{n}\rightarrow u (u_{n} \rightarrow u )$ as $s_{n}\rightarrow 0^{+}$, since $I_{1n} \geq 0$ for all n. Applying Fatou’s Lemma to $I_{1n}$, one gets

$$ \liminf_{n\rightarrow \infty}I_{1n}\geq \int _{\Omega} \frac{v}{u^{\gamma}}\,d\xi , \quad \forall v\in U^{+}. $$

For $I_{2n}$, by the dominated convergence theorem, it holds that

$$ \lim_{n\rightarrow \infty}I_{2n}= - \int _{\Omega} \vert u \vert ^{1- \gamma}\,d\xi . $$

For every $v\in U^{+}$, and the above information, we have

$$ \begin{aligned} \lambda \int _{\Omega}\frac{v-u}{u^{\gamma}}\,d\xi &\leq \liminf _{n\rightarrow \infty} (I_{1n}+I_{2n} ) \\ &\leq \int _{\Omega}\nabla _{H}u\nabla _{H}(v-u)\,d\xi - \int _{\Omega} \phi _{u}u(v-u)\,d\xi +c \Vert v-u \Vert , \end{aligned} $$

where $|dI_{\lambda}|(u)< c$ is arbitrary. □

Lemma 2.3

$I_{\lambda}$ satisfies the $(P.S. )$ condition.

Proof

Let $\{u_{n}\}\subset U^{+}$ be $(P.S. )$ sequence of $I_{\lambda}$, that is

$$ \vert dI_{\lambda} \vert (u_{n} )\rightarrow 0,\qquad I_{ \lambda} (u_{n} )\rightarrow c\in \mathbb{R}\quad \text{as } n \rightarrow \infty . $$

By Lemma 2.2, $\forall v\in U^{+}$, we can infer that

$$ \begin{aligned} \lambda \int _{\Omega}\frac{v-u_{n}}{u_{n}^{\gamma}}\,d\xi \leq \int _{ \Omega}\nabla _{H}u_{n}\nabla _{H}(v-u_{n})\,d\xi - \int _{\Omega}\phi _{u_{n}}u_{n}(v-u_{n})\,d\xi +o(1) \Vert v-u_{n} \Vert , \end{aligned} $$

(2.11)

taking $v=2u_{n}\in U^{+}$ in (2.11), we have that

$$ \begin{aligned} \lambda \int _{\Omega} \vert u_{n} \vert ^{1-\gamma}\,d\xi \leq \int _{\Omega} \vert \nabla _{H}u_{n} \vert ^{2}\,d\xi - \int _{\Omega}\phi _{u_{n}}u_{n}^{2}\,d\xi +o(1) \Vert u_{n} \Vert . \end{aligned} $$

(2.12)

Since $I_{\lambda}(u_{n})\rightarrow c$,

$$ \begin{aligned} \frac{1}{2} \int _{\Omega} \vert \nabla _{H}u_{n} \vert ^{2}\,d\xi -\frac{1}{4} \int _{ \Omega}\phi _{u_{n}}u_{n}^{2}\,d\xi -\frac{\lambda}{1-\gamma} \int _{ \Omega} \vert u_{n} \vert ^{1-\gamma}\,d\xi =c+o(1). \end{aligned} $$

(2.13)

From (2.12) and (2.13), we have

$$ \begin{aligned} \frac{1}{4} \int _{\Omega} \vert \nabla _{H}u_{n} \vert ^{2}\,d\xi &\leq \lambda \biggl( \frac{1}{1-\gamma}- \frac{1}{4}\biggr) \int _{\Omega} \vert u_{n} \vert ^{1-\gamma}\,d\xi +c+o(1)+o(1) \Vert u_{n} \Vert \\ &\leq \lambda \frac{3+\gamma}{4(1-\gamma )} \Vert u_{n} \Vert ^{1-\gamma}+C+o(1) \Vert u_{n} \Vert , \end{aligned} $$

(2.14)

which implies that $\{u_{n}\}$ is bounded in $S^{1}_{0}(\Omega )$. Thus, there exists a subsequence, still denoted by itself, and a function $u \in S^{1}_{0}(\Omega )$, such that $u_{n}\rightharpoonup u$ in $S^{1}_{0}( \Omega )$, $u_{n}(x)\rightarrow u(x)$ a.e. in Ω as $n\rightarrow \infty$. Choosing $v=u_{m}$ as the test function in (2.11), we have

$$ \begin{aligned} \lambda \int _{\Omega}\frac{u_{m}-u_{n}}{u_{n}^{\gamma}}\,d\xi \leq \int _{\Omega}\nabla _{H}u_{n}\nabla _{H}(u_{m}-u_{n})\,d\xi - \int _{ \Omega}\phi _{u_{n}}u_{n}(u_{m}-u_{n})\,d\xi +o(1) \Vert u_{m}-u_{n} \Vert . \end{aligned} $$

Exchanging $u_{m}$ and $u_{n}$ gives a similar inequality, and adding two inequalities together and Lemma 2.1(4), it holds that

$$ \begin{aligned} \Vert u_{n}-u_{m} \Vert ^{2}&\leq \lambda \int _{\Omega}(u_{n}-u_{m}) \biggl( \frac{1}{u_{n}^{\gamma}}-\frac{1}{u_{m}^{\gamma}}\biggr)\,d\xi \\ &\quad{}- \int _{\Omega}(\phi _{u_{m}}u_{m}-\phi _{u_{n}}u_{n}) (u_{n}-u_{m})\,d\xi +o(1) \Vert u_{m}-u_{n} \Vert \\ &\leq - \int _{\Omega}(\phi _{u_{m}}u_{m}-\phi _{u_{n}}u_{n}) (u_{m}-u_{n})\,d\xi +o(1) \Vert u_{m}-u_{n} \Vert \\ &\leq -\frac{1}{2} \Vert \phi _{u_{m}}-\phi _{u_{n}} \Vert ^{2}+o(1) \Vert u_{m}-u_{n} \Vert \\ &\leq o(1) \Vert u_{m}-u_{n} \Vert . \end{aligned} $$

We have $\lim_{n\rightarrow \infty} \Vert u_{n}-u_{m} \Vert =0$. Therefore, $u_{n}\rightarrow u$ in $S^{1}_{0}(\Omega )$ as $n \rightarrow \infty$. □

Lemma 2.4

Suppose that $\vert dI_{\lambda} \vert (u )=0$, then u is a weak solution of the problem (2.3). Namely, $u^{-\gamma}\varphi \in L^{1} (\Omega )$ for all $\varphi \in S^{1}_{0}(\Omega )$, there holds

$$ \begin{aligned} \int _{\Omega}\nabla _{H}u\nabla _{H} \varphi \,d\xi - \int _{\Omega} \phi _{u}u\varphi \,d\xi =\lambda \int _{\Omega} \frac{\varphi}{u^{\gamma}}\,d\xi . \end{aligned} $$

(2.15)

Proof

By Lemma 2.2, we deduce that

$$ \begin{aligned} \lambda \int _{\Omega}\frac{v-u}{u^{\gamma}}\,d\xi \leq \int _{\Omega} \nabla _{H}u\nabla _{H}(v-u)\,d\xi - \int _{\Omega}\phi _{u}u(v-u)\,d\xi , \end{aligned} $$

for every $v\in U^{+}$. Letting $s\in \mathbb{R}$, $\varphi \in S^{1}_{0}( \Omega )$, taking $v=(u+s\varphi )^{+}$ and $v\in U^{+}$ as a test function in (2.7), one gets

$$ \begin{aligned} 0&\leq \int _{\Omega}\nabla _{H}u\nabla _{H} \bigl((u+s\varphi )^{+}-u\bigr)\,d\xi - \int _{\Omega}\phi _{u}u\bigl((u+s\varphi )^{+}-u\bigr)\,d\xi -\lambda \int _{ \Omega}\frac{(u+s\varphi )^{+}-u}{u^{\gamma}}\,d\xi \\ &=s \biggl( \int _{\Omega}\nabla _{H}u\nabla _{H} \varphi \,d\xi - \int _{ \Omega}\phi _{u}u\varphi \,d\xi -\lambda \int _{\Omega} \frac{\varphi}{u^{\gamma}}\,d\xi \biggr)- \int _{\{u+s\varphi < 0\}} \nabla _{H}u\nabla _{H}(u+s\varphi )\,d\xi \\ &\quad{}+ \int _{\{u+s\varphi < 0\}}\phi _{u}u(u+s\varphi )\,d\xi + \int _{ \{u+s\varphi < 0\}}\frac{u+s\varphi}{u^{\gamma}}\,d\xi \\ &\leq s \biggl( \int _{\Omega}\nabla _{H}u\nabla _{H} \varphi \,d\xi - \int _{\Omega}\phi _{u}u\varphi \,d\xi -\lambda \int _{\Omega} \frac{\varphi}{u^{\gamma}}\,d\xi \biggr) \\ &\quad {}-s \int _{\{u+s\varphi < 0\}}( \nabla _{H}u\nabla _{H} \varphi -\phi _{u}u\varphi )\,d\xi , \end{aligned} $$

since $\nabla _{H} u(x)=0$ for a.e. $x\in \Omega$ with $u(x)=0$, and $\operatorname{Meas}\{x\in \Omega :u(x)+s\varphi (x)<0,u(x)>0\}\rightarrow 0$ as $s\rightarrow 0$, one obtains

$$ \begin{aligned} \int _{\{u+s\varphi < 0\}}(\nabla _{H}u\nabla _{H} \varphi -\phi _{u}u \varphi )\,d\xi = \int _{\{u+s\varphi < 0,u>0\}}(\nabla _{H}u\nabla _{H} \varphi -\phi _{u}u\varphi )\,d\xi \rightarrow 0. \end{aligned} $$

Therefore

$$ \begin{aligned} 0\leq s \biggl( \int _{\Omega}\nabla _{H}u\nabla _{H} \varphi \,d\xi - \int _{\Omega}\phi _{u}u\varphi \,d\xi -\lambda \int _{\Omega} \frac{\varphi}{u^{\gamma}}\,d\xi \biggr)+o(s), \end{aligned} $$

as $s\rightarrow 0$, we obtain that

$$ \begin{aligned} \int _{\Omega}\nabla _{H}u\nabla _{H} \varphi \,d\xi - \int _{\Omega} \phi _{u}u\varphi \,d\xi -\lambda \int _{\Omega} \frac{\varphi}{u^{\gamma}}\,d\xi \geq 0. \end{aligned} $$

By the arbitrariness of φ, also holds for −φ

$$ \begin{aligned} \int _{\Omega}\nabla _{H}u\nabla _{H} \varphi \,d\xi - \int _{\Omega} \phi _{u}u\varphi \,d\xi -\lambda \int _{\Omega} \frac{\varphi}{u^{\gamma}}\,d\xi =0. \end{aligned} $$

Hence, we can deduce that (2.15) holds. □

Lemma 2.5

Given $0<\gamma <1$, there exist constants $r,\rho ,\Lambda _{0}>0$, such that the functional $I_{\lambda}$ satisfies the following conditions for $0<\lambda <\Lambda _{0}$:

• (i) $I_{\lambda}(u)|_{u\in S_{\rho}}\geq r> 0$, $\inf_{u\in B_{\rho}}I_{\lambda}(u)<0$;

• (ii) There exists $e\in S^{1}_{0}(\Omega )$ with $\|e\|>\rho$ such that $I_{\lambda}(e)<0$.

Proof

(i) It follows from (2.6) and Lemma 2.1(3) that

$$ \begin{aligned} I_{\lambda}(u)&=\frac{1}{2} \Vert u \Vert ^{2}-\frac{1}{4} \int _{\Omega}\phi _{u}u^{2}\,d\xi - \frac{\lambda}{1-\gamma} \int _{\Omega} \vert u \vert ^{1-\gamma}\,d\xi \\ &\geq \frac{1}{2} \Vert u \Vert ^{2}- \frac{1}{4}S^{-3} \vert \Omega \vert ^{\frac{1}{2}} \Vert u \Vert ^{4}- \frac{\lambda}{1-\gamma}S^{-\frac{1-\gamma}{2}} \vert \Omega \vert ^{ \frac{3+\gamma}{4}} \Vert u \Vert ^{1-\gamma}, \end{aligned} $$

which implies that there exist constants $r, \rho , \Lambda _{0}>0$, such that $I_{\lambda}| _{u\in S_{\rho}}\geq r>0$ for every $\lambda \in (0,\Lambda _{0} )$. Moreover, for $u\in S^{1}_{0}( \Omega )\backslash \{0 \}$, it holds that

$$ \lim_{t \rightarrow 0^{+}} \frac{I_{\lambda} (tu )}{t^{1-\gamma}}=- \frac{\lambda}{1-\gamma} \int _{\Omega} \vert u \vert ^{1-\gamma}\,d\xi < 0. $$

So, we obtain that $I_{\lambda}(tu)<0$ for all $u\neq 0$ and t small enough. Therefore, for $\|u\|$ small enough, one has

$$ m=\inf_{u\in B_{\rho}}I_{\lambda}(u)< 0. $$

(ii) For every $u^{+}\in S^{1}_{0}(\Omega )$, $u^{+}\neq 0$ and $t>0$, we get

$$ \begin{aligned} I_{\lambda}(tu)&=\frac{t^{2}}{2} \Vert u \Vert ^{2}-\frac{t^{4}}{4} \int _{ \Omega}\phi _{u}u^{2}\,d\xi - \frac{\lambda t^{1-\gamma}}{1-\gamma} \int _{ \Omega} \vert u \vert ^{1-\gamma}\,d\xi \rightarrow -\infty , \end{aligned} $$

as $t\rightarrow +\infty$. Therefore, we can find $e\in S^{1}_{0}(\Omega )$ such that $\|e\|>\rho$ and $I_{\lambda}(e)<0$. □

3 Proof of main results

In this section, we show that for each $\lambda >0$, the functional $I_{\lambda}$ attains the global minimizer in $S^{1}_{0}(\Omega )$, which is the unique solution of system (1.1) for $\mu =1$ and multiple solutions of the system for $\mu =-1$, $\lambda >0$ small enough.

Proof of Theorem 1.1

We prove Theorem 1.1 in three steps.

Step 1. For every $\lambda >0$ and $\mu =1$, the functional $I_{\lambda}$ attains the global minimizer in $S^{1}_{0}(\Omega )$, in other words, there exists $u_{*}\in S^{1}_{0}(\Omega )$ such that

$$ I_{\lambda}(u_{*})=m_{\lambda}=\inf _{S^{1}_{0}(\Omega )}I_{\lambda}< 0. $$

In fact, for all $u\in S^{1}_{0}(\Omega )$, combining with Lemma 2.1(1) and (2.6), we infer that

$$ \begin{aligned} I_{\lambda}(u)&= \frac{1}{2} \Vert u \Vert ^{2}+\frac{1}{4} \int _{\Omega}\phi _{u}u^{2}\,d\xi - \frac{\lambda}{1-\gamma} \int _{\Omega} \vert u \vert ^{1-\gamma}\,d\xi \\ &\geq \frac{1}{2} \Vert u \Vert ^{2}- \frac{\lambda}{1-\gamma}S^{- \frac{1-\gamma}{2}} \vert \Omega \vert ^{\frac{3+\gamma}{4}} \Vert u \Vert ^{1-\gamma}, \end{aligned} $$

(3.1)

this implies that $I_{\lambda}$ is coercive and bounded from below on $S^{1}_{0}(\Omega )$ for each $\lambda >0$. Thus, $m_{\lambda}=\inf_{S^{1}_{0}(\Omega )}I_{\lambda}$. For $t>0$ and given $u\in S^{1}_{0}(\Omega )\backslash \{0\}$,

$$ \begin{aligned} I_{\lambda}(tu)=\frac{t^{2}}{2} \Vert u \Vert ^{2}+\frac{t^{4}}{4} \int _{ \Omega}\phi _{u}u^{2}\,d\xi - \frac{\lambda t^{1-\gamma}}{1-\gamma} \int _{\Omega} \vert u \vert ^{1-\gamma}\,d\xi . \end{aligned} $$

We deduce from that for $t>0$ small enough, $I_{\lambda}(tu)<0$. Therefore, $m_{\lambda}=\inf_{S^{1}_{0}(\Omega )}I_{\lambda}<0$.

From the definition of $m_{\lambda}$, existence of minimizing sequence $\{u_{n}\}\subset S^{1}_{0}(\Omega )$ such that $\lim_{n\rightarrow \infty}I_{\lambda}(u_{n})=m_{\lambda}<0$. Since $I_{\lambda}(u_{n})=I_{\lambda}(|u_{n}|)$, we can assume that $u_{n}\geq 0$. By (3.1), we know that $\{u_{n}\}$ is bounded in $S^{1}_{0}(\Omega )$. Suppose there exists a subsequence, still denoted by $\{u_{n}\}$, and $u_{*}\in S^{1}_{0}(\Omega )$ such that

$$\begin{aligned} \textstyle\begin{cases} u_{n}\rightharpoonup u_{*},& \text{weakly in } S^{1}_{0}(\Omega ), \\ u_{n}\rightarrow u_{*},& \text{strongly in } L^{p}(\Omega )\ (1\leq p< 4), \\ u_{n}(x)\rightarrow u_{*}(x),& \text{a.e. in } \Omega . \end{cases}\displaystyle \end{aligned}$$

Then, combining with the weakly lower semi-continuity of the norm and Lemma 2.1 (2), one has

$$ \begin{aligned} I_{\lambda}(u_{*})&= \frac{1}{2} \Vert u_{*} \Vert ^{2}+ \frac{1}{4} \int _{ \Omega}\phi _{u_{*}}u_{*}^{2}\,d\xi -\frac{\lambda}{1-\gamma} \int _{\Omega} \vert u_{*} \vert ^{1-\gamma}\,d\xi \\ &\leq \liminf_{n\rightarrow \infty}I_{\lambda}(u_{n})=m_{\lambda}. \end{aligned} $$

Furthermore, $I_{\lambda}(u_{*})\geq m_{\lambda}$, thus $I_{\lambda}(u_{*})= m_{\lambda}<0$.

In addition, we show $u_{*}>0$ in Ω. From the information above, $u_{*}\geq 0$ and $u_{*}\neq 0$. Fix $\eta \in S^{1}_{0}(\Omega )$, $\eta >0$ and $t\geq 0$, we obtain that

$$ \begin{aligned} 0&\leq \liminf_{t\rightarrow 0} \frac{I_{\lambda}(u_{*}+t\eta )-I_{\lambda}(u_{*})}{t} \\ &= \int _{\Omega}(\nabla _{H}u_{*}\nabla _{H}\eta +\phi _{u_{*}}u_{*} \eta )\,d\xi - \frac{\lambda}{1-\gamma}\limsup_{t\rightarrow 0} \int _{ \Omega}\frac{(u_{*}+t\eta )^{1-\gamma}-u_{*}^{1-\gamma}}{t}\,d\xi , \end{aligned} $$

that is

$$ \begin{aligned} \frac{\lambda}{1-\gamma}\limsup _{t\rightarrow 0} \int _{\Omega} \frac{(u_{*}+t\eta )^{1-\gamma}-u_{*}^{1-\gamma}}{t}\,d\xi \leq \int _{ \Omega}[\nabla _{H}u_{*}\nabla _{H}\eta +\phi _{u_{*}}u_{*}\eta ]\,d\xi . \end{aligned} $$

(3.2)

Notice that

$$ \begin{aligned} \int _{\Omega}\frac{(u_{*}+t\eta )^{1-\gamma}-u_{*}^{1-\gamma}}{t}\,d\xi =(1-\gamma ) \int _{\Omega}(u_{*}+t\eta \zeta )^{-\gamma} \eta \,d\xi , \end{aligned} $$

where $\zeta (x)\in (0,1)$ and

$$ \bigl(u_{*}(x)+t\eta (x)\zeta (x)\bigr)^{-\gamma}\eta (x) \rightarrow u_{*}(x)^{- \gamma}\eta (x),\quad \text{a.e. } x\in \Omega , t\rightarrow 0. $$

Since $(u_{*}(x)+t\eta (x)\zeta (x))^{-\gamma}\eta (x)\geq 0$, using Fatou’s Lemma, from (3.2), it holds

$$ \begin{aligned} \frac{\lambda}{1-\gamma} \int _{\Omega}u_{*}^{-\gamma}\eta \,d\xi \leq \int _{\Omega}[\nabla _{H}u_{*}\nabla _{H}\eta +\phi _{u_{*}}u_{*} \eta ]\,d\xi . \end{aligned} $$

Using a similar approach, the above equation also holds for $0\leq \eta \in S^{1}_{0}(\Omega )$, that is

$$ \begin{aligned} \int _{\Omega}(\nabla _{H}u_{*}\nabla _{H}\eta +\phi _{u_{*}}u_{*} \eta )\,d\xi - \frac{\lambda}{1-\gamma} \int _{\Omega}u_{*}^{-\gamma} \eta \,d\xi \geq 0,\quad \eta \in S^{1}_{0}(\Omega ), \eta \geq 0. \end{aligned} $$

(3.3)

Thus,

$$ \begin{aligned} -\Delta _{H}u_{*}+\phi _{u_{*}}u_{*}\geq 0. \end{aligned} $$

Note that $\phi _{u_{*}}(\xi )>0$ for any $\xi \in \Omega$, $u_{*}\geq 0$ and $u_{*}\neq 0$. According to the maximum principle (see [3, 4]), $u_{*}>0$ in Ω.

Step 2. We prove that $u_{*}$ satisfies (2.5) for $\mu =1$. Let $\delta >0$ and define $h: [-\delta ,\delta ]\rightarrow \mathbb{R}$ by $h(t)=I_{\lambda}(u_{*}+tu_{*})$, then h attains its minimum at $t=0$, and it holds that

$$ \begin{aligned} h'(0)&= \Vert u_{*} \Vert ^{2}+ \int _{\Omega}\phi _{u_{*}}u_{*}^{2}\,d\xi - \lambda \int _{\Omega} \vert u_{*} \vert ^{1-\gamma}\,d\xi =0. \end{aligned} $$

(3.4)

We take $\eta \in S^{1}_{0}(\Omega )\backslash \{0\}$, $\varepsilon >0$ and define $\Phi =(u_{*}+\varepsilon \eta )^{+}$. Let

$$ \begin{aligned} \Omega _{1}=\bigl\{ x\in \Omega :u_{*}(x)+\varepsilon \eta (x)>0\bigr\} ,\qquad \Omega _{2}= \bigl\{ x\in \Omega :u_{*}(x)+\varepsilon \eta (x)\leq 0\bigr\} . \end{aligned} $$

Then, $\Phi |_{\Omega _{1}}=u_{*}+\varepsilon \eta$, $\Phi |_{\Omega _{2}}=0$. Inserting Φ into (3.3) and using (3.4), we can get

$$ \begin{aligned} 0&\leq \int _{\Omega}\bigl(\nabla _{H}u_{*} \nabla _{H}\Phi +\phi _{u_{*}}u_{*} \Phi - \lambda u_{*}^{-\gamma}\Phi \bigr)\,d\xi \\ &= \int _{\Omega _{1}}\bigl[\nabla _{H}u_{*} \nabla _{H}(u_{*}+\varepsilon \eta )+\phi _{u_{*}}u_{*}(u_{*}+\varepsilon \eta )-\lambda u_{*}^{- \gamma}(u_{*}+\varepsilon \eta )\bigr]\,d\xi \\ &= \int _{\Omega \backslash \Omega _{2}}\bigl[\nabla _{H}u_{*} \nabla _{H}(u_{*}+ \varepsilon \eta )+\phi _{u_{*}}u_{*}(u_{*}+\varepsilon \phi )- \lambda u_{*}^{-\gamma}(u_{*}+\varepsilon \eta ) \bigr]\,d\xi \\ &=\varepsilon \int _{\Omega}\bigl(\nabla _{H}u_{*} \nabla _{H}\eta +\phi _{u_{*}}u_{*} \eta - \lambda u_{*}^{-\gamma}\eta \bigr)\,d\xi - \int _{\Omega _{2}}\bigl[\nabla _{H}u_{*} \nabla _{H}(u_{*}+\varepsilon \eta ) \\ &\quad{}+\phi _{u_{*}}u_{*}(u_{*}+\varepsilon \eta )-\lambda u_{*}^{- \gamma}(u_{*}+\varepsilon \eta )\bigr]\,d\xi \\ &\leq \varepsilon \int _{\Omega}\bigl(\nabla _{H}u_{*} \nabla _{H}\eta + \phi _{u_{*}}u_{*}\eta - \lambda u_{*}^{-\gamma}\eta \bigr)\,d\xi - \varepsilon \int _{\Omega _{2}}(\nabla _{H}u_{*}\nabla _{H}\eta + \phi _{u_{*}}u_{*}\eta )\,d\xi . \end{aligned} $$

(3.5)

Due to $u_{*}>0$ and the measure of the domain $\Omega _{2}=\{x\in \Omega :u_{*}(x)+\varepsilon \eta (x)\leq 0\}$ tends to zero as $\varepsilon \rightarrow 0$, there holds

$$ \begin{aligned} \int _{\Omega _{2}}(\nabla _{H}u_{*}\nabla _{H}\eta +\phi _{u_{*}}u_{*} \eta )\,d\xi \rightarrow 0. \end{aligned} $$

Then, dividing by $\varepsilon >0$ and letting $\varepsilon \rightarrow 0$ in (3.5), we have

$$ \begin{aligned} \int _{\Omega}\bigl(\nabla _{H}u_{*} \nabla _{H}\eta +\phi _{u_{*}}u_{*} \eta - \lambda u_{*}^{-\gamma}\eta \bigr)\,d\xi \geq 0,\quad \eta \in S^{1}_{0}( \Omega ). \end{aligned} $$

The above inequality also holds for −η, and we can get

$$ \begin{aligned} \int _{\Omega}\bigl(\nabla _{H}u_{*} \nabla _{H}\eta +\phi _{u_{*}}u_{*} \eta - \lambda u_{*}^{-\gamma}\eta \bigr)\,d\xi =0,\quad \eta \in S^{1}_{0}( \Omega ). \end{aligned} $$

Then, $u_{*}\in S^{1}_{0}(\Omega )$ is a solution of system (1.1) for $\lambda >0$ and $\mu =1$.

Step 3. We prove that $u_{*}$ is the unique solution of (1.1) for $\mu =1$. We may assume that $v_{\star}\in S^{1}_{0}(\Omega )$ is also a solution of system (1.1), and from (2.5), we get

$$ \begin{aligned} \int _{\Omega}\bigl[\nabla _{H}u_{*} \nabla _{H}(u_{*}-v_{\star})+\phi _{u_{*}}u_{*}(u_{*}-v_{ \star})\bigr]\,d\xi -\lambda \int _{\Omega} u_{*}^{-\gamma}(u_{*}-v_{\star})\,d\xi =0, \end{aligned} $$

(3.6)

and

$$ \begin{aligned} \int _{\Omega}\bigl[\nabla _{H}v_{\star} \nabla _{H}(u_{*}-v_{\star})+\phi _{v_{ \star}}v_{\star}(u_{*}-v_{\star})\bigr]\,d\xi -\lambda \int _{\Omega} v_{ \star}^{-\gamma}(u_{*}-v_{\star})\,d\xi =0. \end{aligned} $$

(3.7)

Combining with (3.6) and (3.7), it holds that

$$ \begin{aligned} \Vert u_{*}-v_{\star } \Vert ^{2}+ \int _{\Omega}(\phi _{u_{*}}u_{*}-\phi _{v_{ \star}}v_{\star}) (u_{*}-v_{\star})\,d\xi =\lambda \int _{ \Omega}\bigl(u_{*}^{-\gamma}-v_{\star}^{-\gamma} \bigr) (u_{*}-v_{\star})\,d\xi . \end{aligned} $$

(3.8)

For $\gamma \in (0, 1)$, $u_{*}, v_{\star}>0$ in Ω, and

$$ \begin{aligned} \int _{\Omega}\bigl(u_{*}^{-\gamma}-v_{\star}^{-\gamma} \bigr) (u_{*}-v_{ \star})\,d\xi \leq 0. \end{aligned} $$

Hence, by (3.8) and Lemma 2.1(4), we get

$$ \begin{aligned} \Vert u_{*}-v_{\star} \Vert ^{2}\leq 0, \end{aligned} $$

that is say

$$ \begin{aligned} \Vert u_{*}-v_{\star} \Vert ^{2}=0, \end{aligned} $$

that is $u_{*}=v_{\star}$. Hence, $u_{*}\in S^{1}_{0}(\Omega )$ is the unique solution of system (1.1). □

Proof of Theorem 1.2

We prove Theorem 1.2 in two steps.

Step 1. Suppose that $0<\lambda <\Lambda _{0}$, then system (1.1) admits a positive solution $u_{*}$ such that $I_{\lambda} (u_{\lambda} )=m<0$.

In fact, we claim that there exists $u_{\lambda}\in B_{\rho}$, such that $I_{\lambda}(u_{\lambda})=m<0$. By the definition of m, we know that there exists a minimizing sequence $\{u_{n}\}\subset B_{\rho}\subset U^{+}$ such that $\lim_{n\rightarrow \infty}I_{\lambda}(u_{n})=m<0$. Since $\{u_{n}\}$ is bounded in $B_{\rho}$, we may assume that, up to a subsequence still denoted by itself, there exists $u_{\lambda}\in S^{1}_{0}(\Omega )$, such that

$$ \begin{aligned} \textstyle\begin{cases} u_{n}\rightharpoonup u_{\lambda} , &\text{weakly in } S^{1}_{0}( \Omega ), \\ u_{n} \rightarrow u_{\lambda}, & \text{strongly in } L^{p} (\Omega )\ (1\leq p< 4 ), \\ u_{n} (x )\rightarrow u_{\lambda} (x ), & \text{a.e. in } \Omega , \end{cases}\displaystyle \end{aligned} $$

as $n\rightarrow \infty$. Set $w_{n}=u_{n}-u_{\lambda}$, and using Brézis-Lieb’s Lemma (see [5]), one has

$$ \begin{aligned} \Vert u_{n} \Vert ^{2}= \Vert w_{n} \Vert ^{2}+ \Vert u_{\lambda} \Vert ^{2}+o (1 ). \end{aligned} $$

(3.9)

Hence, by Lemma 2.3, we can infer that

$$ \begin{aligned} m&=\lim_{n \rightarrow \infty}I_{\lambda}(u_{n})=I_{ \lambda}(u_{\lambda})+ \lim_{n \rightarrow \infty}\frac{1}{2} \Vert w_{n} \Vert ^{2}\geq I_{\lambda}(u_{\lambda}), \end{aligned} $$

(3.10)

which implies that $m\geq I_{\lambda}(u_{\lambda})$. Since $B_{\rho}$ is closed and convex, one has that $u_{\lambda}\in B_{\rho}$. Thus, we obtain $I_{\lambda}(u_{\lambda})=m<0$ and $u_{\lambda}\not \equiv 0$ in Ω. From the above arguments, we know that $u_{\lambda}$ is a local minimizer of $I_{\lambda}$.

Now, we prove that $u_{\lambda}$ is a critical point of $I_{\lambda}$. Note that $u_{\lambda}\geq 0$ and $u_{\lambda}\not \equiv 0$. Then, for any $\psi \in U^{+}\subset S^{1}_{0}(\Omega )$, let $t>0$ such that $u_{\lambda}+t\psi \in S^{1}_{0}(\Omega )$, and one has

$$ \begin{aligned} 0&\leq I_{\lambda} (u_{\lambda}+t\psi )-I_{\lambda}(u) \\ &=\frac{1}{2} \Vert u_{\lambda}+t\psi \Vert ^{2}-\frac{1}{4} \int _{ \Omega}\phi _{u_{\lambda}+t\psi}(u_{\lambda}+t\psi )^{2}\,d\xi - \frac{\lambda}{1-\gamma} \int _{\Omega} \vert u_{\lambda}+t\psi \vert ^{1-\gamma}\,d\xi \\ &\quad{}-\frac{1}{2} \Vert u_{\lambda} \Vert ^{2}+\frac{1}{4} \int _{ \Omega}\phi _{u_{\lambda}}u_{\lambda}^{2}\,d\xi + \frac{\lambda}{1-\gamma} \int _{\Omega} \vert u_{\lambda} \vert ^{1-\gamma}\,d\xi . \end{aligned} $$

(3.11)

Actually, from (3.11), we can see that

$$ \begin{aligned} &\frac{\lambda}{1-\gamma} \int _{\Omega} \bigl[(u_{\lambda}+t\psi )^{1- \gamma}-u_{\lambda}^{1-\gamma} \bigr]\,d\xi \\ &\quad \leq \frac{1}{2}\bigl( \Vert u_{\lambda}+t\psi \Vert ^{2}- \Vert u_{ \lambda} \Vert ^{2}\bigr)- \frac{1}{4} \int _{\Omega} \bigl[\phi _{u_{ \lambda}+t\psi}(u_{\lambda}+t \psi )^{2}-\phi _{u_{\lambda}}u_{\lambda}^{2} \bigr]\,d\xi . \end{aligned} $$

Dividing by $t>0$ and passing to the limit as $t\rightarrow 0^{+}$, it holds that

$$ \begin{aligned} \frac{\lambda}{1-\gamma}\liminf _{t\rightarrow 0^{+}} \int _{\Omega} \frac{(u_{\lambda}+t\psi )^{1-\gamma}-u_{\lambda}^{1-\gamma}}{t}\,d\xi \leq \int _{\Omega}\nabla _{H}u_{\lambda}\nabla _{H}\psi \,d\xi - \int _{ \Omega}\phi _{u_{\lambda}}u_{\lambda}\psi \,d\xi . \end{aligned} $$

(3.12)

Notice that

$$ \begin{aligned} \frac{\lambda}{1-\gamma} \int _{\Omega} \frac{(u_{\lambda}+t\psi )^{1-\gamma}-u_{\lambda}^{1-\gamma}}{t}\,d\xi = \lambda \int _{\Omega}(u_{\lambda}+\zeta t\psi )^{-\gamma} \psi \,d\xi . \end{aligned} $$

Where $\zeta \rightarrow 0^{+}$ and $(u_{\lambda}+\zeta t\psi )^{-\gamma}\psi \rightarrow u_{\lambda}^{- \gamma}\psi$ a.e. $x\in \Omega$ as $t\rightarrow 0^{+}$, since $(u_{\lambda}+\zeta t\psi )^{-\gamma}\psi \geq 0$. By Fatou’s Lemma, one has

$$ \begin{aligned} \lambda \int _{\Omega}u_{\lambda}^{-\gamma}\psi \,d\xi \leq \frac{\lambda}{1-\gamma}\liminf_{t\rightarrow 0^{+}} \int _{\Omega} \frac{(u_{\lambda}+t\psi )^{1-\gamma}-u_{\lambda}^{1-\gamma}}{t}\,d\xi . \end{aligned} $$

Therefore, we deduce from (3.12) and the above estimate that

$$ \begin{aligned} \int _{\Omega}\nabla _{H}u_{\lambda}\nabla _{H}\psi \,d\xi - \int _{ \Omega}\phi _{u_{\lambda}}u_{\lambda}\psi \,d\xi - \lambda \int _{\Omega}u_{ \lambda}^{-\gamma}\psi \,d\xi \geq 0, \quad \psi \geq 0. \end{aligned} $$

(3.13)

Since $I_{\lambda}(u_{\lambda})<0$, this, together with Lemma 2.5, implies that $u_{\lambda}\notin S_{\rho}$; therefore, we obtain $\|u_{\lambda}\|<\rho$. For $u_{\lambda}$, there is $\delta _{1}\in (0,1)$ such that $(1+t)u_{\lambda}\in B_{\rho}$ for $|t|\leq \delta _{1}$. Define $k:[-\delta _{1},\delta _{1}]$ by $k(t)=I_{\lambda}((1+t)u_{\lambda})$. Clearly, $k(t)$ achieves its minimum at $t=0$, namely

$$ \begin{aligned} k'(t)|_{t=0}= \Vert u_{\lambda} \Vert ^{2}- \int _{\Omega}\phi _{u_{\lambda}}u_{ \lambda}^{2} \,d\xi -\lambda \int _{\Omega}u_{\lambda}^{1-\gamma}\,d\xi =0. \end{aligned} $$

(3.14)

Suppose that for any $v\in S^{1}_{0}(\Omega )$, $\epsilon >0$. Define $\Psi \in U^{+}$ by

$$ \begin{aligned} \Psi =(u_{\lambda}+\epsilon v)^{+}. \end{aligned} $$

Combining with (3.13) and (3.14), we get

$$ \begin{aligned} 0&\leq \int _{\Omega} \bigl(\nabla _{H}u_{\lambda} \nabla _{H}\Psi - \phi _{u_{\lambda}}u_{\lambda}\Psi - \lambda u_{\lambda}^{-\gamma} \Psi \bigr)\,d\xi \\ &= \int _{\{u_{\lambda}+\epsilon v>0\}} \bigl[\nabla _{H}u_{\lambda} \nabla _{H}(u_{\lambda}+\epsilon v)-\phi _{u_{\lambda}}u_{\lambda}(u_{ \lambda}+ \epsilon v)-\lambda u_{\lambda}^{-\gamma}(u_{\lambda}+ \epsilon v) \bigr]\,d\xi \\ &= \biggl( \int _{\Omega}- \int _{\{u_{\lambda}+\epsilon v\leq 0\}} \biggr) \bigl[\nabla _{H}u_{\lambda} \nabla _{H}(u_{\lambda}+ \epsilon v)-\phi _{u_{\lambda}}u_{\lambda}(u_{\lambda}+ \epsilon v)- \lambda u_{\lambda}^{-\gamma}(u_{\lambda}+ \epsilon v) \bigr]\,d\xi \\ &= \Vert u_{\lambda} \Vert ^{2}- \int _{\Omega}\phi _{u_{\lambda}}u_{\lambda}^{2} \,d\xi -\lambda \int _{\Omega} \vert u_{\lambda} \vert ^{1-\gamma}\,d\xi \\ &\quad {}+\epsilon \int _{\Omega}\bigl(\nabla _{H}u_{\lambda} \nabla _{H}v-\phi _{u_{\lambda}}u_{ \lambda}v-\lambda u_{\lambda}^{-\gamma}v\bigr)\,d\xi \\ &\quad{}- \int _{\{u_{\lambda}+\epsilon v\leq 0\}}\bigl[\nabla _{H}u_{ \lambda} \nabla _{H}(u_{\lambda}+\epsilon v)-\phi _{u_{\lambda}}u_{ \lambda}(u_{\lambda}+ \epsilon v)-\lambda u_{\lambda}^{-\gamma}(u_{ \lambda}+\epsilon v)\bigr]\,d\xi \\ &\leq \epsilon \int _{\Omega} \bigl(\nabla _{H}u_{\lambda} \nabla _{H}v- \phi _{u_{\lambda}}u_{\lambda}v-\lambda u_{\lambda}^{-\gamma}v \bigr)\,d\xi \\ &\quad {}-\epsilon \int _{\{u_{\lambda}+\epsilon v\leq 0\}}(\nabla _{H}u_{ \lambda}\nabla _{H}v-\phi _{u_{\lambda}}u_{\lambda}v)\,d\xi . \end{aligned} $$

(3.15)

Since the measure of the domain of integration $\{u_{\lambda}+\epsilon v\leq 0\}\rightarrow 0$ as $\epsilon \rightarrow 0$, it follows that

$$ \begin{aligned} \lim_{\epsilon \rightarrow 0} \int _{u_{\lambda}+\epsilon v\leq 0}( \nabla _{H}u_{\lambda}\nabla _{H}v-\phi _{u_{\lambda}}u_{\lambda}v)\,d\xi =0. \end{aligned} $$

Therefore, dividing by ϵ and setting $\epsilon \rightarrow 0$ in (3.15), one gets

$$ \begin{aligned} \int _{\Omega}\nabla _{H}u_{\lambda}\nabla _{H}v \,d\xi - \int _{\Omega} \phi _{u_{\lambda}}u_{\lambda}v \,d\xi - \lambda \int _{\Omega}u_{ \lambda}^{-\gamma}v \,d\xi \geq 0. \end{aligned} $$

(3.16)

By the arbitrariness of v, the inequality also holds for −v

$$ \begin{aligned} \int _{\Omega}\nabla _{H}u_{\lambda}\nabla _{H}v \,d\xi - \int _{\Omega} \phi _{u_{\lambda}}u_{\lambda}v \,d\xi - \lambda \int _{\Omega}u_{ \lambda}^{-\gamma}v \,d\xi =0. \end{aligned} $$

(3.17)

Since $u_{\lambda}\geq 0$ and $u_{\lambda}\not \equiv 0$, from (3.17), there holds

$$ \begin{aligned} -\Delta _{H}u_{\lambda}\geq \phi _{u_{\lambda}}u_{\lambda}&\geq 0. \end{aligned} $$

Note that $u_{\lambda}\geq 0$ and $u_{\lambda}\neq 0$, then by the maximum principle (see [3, 4]), it suggests that $u_{\lambda}>0$ in Ω. From the above arguments, we obtain that $u_{\lambda}$ is a positive solution of system (1.1) with $I_{\lambda}(u_{\lambda})=m<0$.

Step 2. Assume that $0<\lambda <\Lambda _{0}$, then system (1.1) has a positive solution $v_{*}$ such that $I_{\lambda}(v_{*})>0$.

In fact, by Lemma 2.5, $I_{\lambda}$ satisfies the geometric structure of the mountain pass Lemma. By Lemma 2.3, there exists a sequence $\{v_{n}\}$ such that

$$ \vert dI_{\lambda} \vert (v_{n})\rightarrow 0, \qquad I_{\lambda}(v_{n})\rightarrow c,\quad \text{as } n \rightarrow \infty . $$

We know that $\{v_{n}\}\subset S^{1}_{0} (\Omega )$ has a convergent subsequence, still denoted by $\{v_{n}\}$, we may assume that $v_{n}\rightarrow v_{*}$ in $S^{1}_{0}(\Omega )$, and

$$ I_{\lambda}(v_{*})=\lim_{n\rightarrow \infty}I_{\lambda}(v_{n})=c, \qquad \vert dI_{ \lambda} \vert (v_{n})\rightarrow 0. $$

Applying Theorem 1.3.1 in [6], similar to step 1, $v_{*}$ satisfies problem (2.3) with $I_{\lambda}(v_{*})=c>0$. Thus, $v_{*}$ is the second positive solution of system (1.1). □