# Liouville type theorems for the system of fractional nonlinear equations in $${R^{n}_{+}}$$

## Abstract

In this paper we consider the following system of fractional nonlinear equations in the half space $${R^{n}_{+}}$$:

$$\textstyle\begin{cases} (-\Delta)^{\frac{\alpha}{2}}u_{1}(x)=x^{\gamma}_{n}u_{1}^{\alpha _{1}}(x)u_{2}^{\beta_{1}}(x), &\mbox{ x \in R^{n}_{+}},\\ (-\Delta)^{\frac{\alpha}{2}}u_{2}(x)=x^{\gamma}_{n}u_{1}^{\alpha _{2}}(x)u_{2}^{\beta_{2}}(x), &\mbox{x \in R^{n}_{+}},\\ u_{1}(x)=u_{2}(x)=0, &\mbox{x \notin R^{n}_{+}}, \end{cases}$$
(1)

where $$\gamma\geq0$$, $$0<\alpha<2$$, $$\alpha_{i}$$, $$\beta_{i} >0$$, $$i=1,2$$.

First, we use the Kelvin transform and the method of moving planes in integral forms to prove that (1) is equivalent to the following system of integral equations with $$1<\alpha_{i}+\beta_{i}\leq\frac{n+\alpha +2\gamma}{n-\alpha}$$:

$$\textstyle\begin{cases} u_{1}(x)=\int_{R^{n}_{+}}G(x,y)y^{\gamma}_{n}u_{1}^{\alpha _{1}}(y)u_{2}^{\beta_{1}}(y)\,dy, &\mbox{x \in R^{n}_{+}},\\ u_{2}(x)=\int_{R^{n}_{+}}G(x,y)y^{\gamma}_{n}u_{1}^{\alpha _{2}}(y)u_{2}^{\beta_{2}}(y)\,dy, &\mbox{x \in R^{n}_{+}}, \end{cases}$$
(2)

where $$G(x,y)$$ is the Green’s function associated with $$(-\Delta )^{\frac{\alpha}{2}}$$ in $$R^{n}_{+}$$.

Then we continue work on integral systems (2) to establish Liouville type theorems, i.e. the nonexistence of positive solutions in the subcritical case and the critical case, $$1<\alpha_{i}+\beta_{i}\leq\frac{n+\alpha+2\gamma}{n-\alpha}$$.

## Introduction

In recent years, there has been a great deal of interests in using the fractional Laplacian to model diverse physical phenomena, such as anomalous diffusion and quasi-geostrophic flows, turbulence and water waves, molecular dynamics, and relativistic quantum mechanics of stars (see [16], and the references therein). The fractional Laplacian in $$R^{n}$$ is a nonlocal operator, taking the form

$$(-\Delta)^{\frac{\alpha}{2}}u(x)=C_{n,\alpha} \textit{ P.V. } \int _{R^{n}}\frac{u(x)-u(z)}{\vert x-z\vert ^{n+\alpha}}\,dz,$$
(1.1)

where $$0<\alpha<2$$, P.V. stands for the Cauchy principal value. This operator is well defined in $$\mathcal {S}$$, the Schwartz space of rapidly decreasing $$C^{\infty}$$ functions in $$R^{n}$$. One can extend this operator to a wider space $$\mathcal {L}_{\alpha}$$ of distributions as follows.

Let

$${\mathcal {L}}_{\alpha}= \biggl\{ u:R^{n}\rightarrow R \Bigm| \int_{R^{n}} \frac {\vert u(x)\vert }{1+\vert x\vert ^{n+\alpha}}\,dx < \infty \biggr\} .$$

For $$u \in \mathcal {L}_{\alpha}$$, we define $$(-\Delta)^{\frac{\alpha }{2}}u(x)$$ as a distribution:

$$\bigl\langle (-\Delta)^{\frac{\alpha}{2}}u(x), \phi\bigr\rangle = \bigl\langle u, (- \Delta)^{\frac {\alpha}{2}}\phi\bigr\rangle ,\quad \forall \phi\in \mathcal {S}.$$

Zhang and Cheng [7] considered the positive solutions of the following single equation in $$R^{n}$$:

$$(-\Delta)^{\frac{\alpha}{2}}u(x)=x^{\gamma}_{n}u^{p}(x),\quad u(x)>0, x\in R^{n}.$$
(1.2)

They showed the following.

### Proposition 1.1

[7]

Assume $$p >\frac{n}{n-\alpha}$$ and $$\gamma\geq0$$, if $$u(x) \in L^{\frac{n(p-1)}{\alpha}}(R^{n}_{+})$$ is a non-negative solution of equation (1.2), then $$u(x) \equiv0$$.

### Proposition 1.2

[7]

Assume $$1< p\leq\frac{n+\alpha+2\gamma}{n-\alpha}$$ and $$\gamma\geq 0$$, if $$u(x)$$ is a locally bounded nonnegative solution of the equation (1.2), then $$u(x)\equiv0$$. In particular, when $$\frac{n+\alpha}{n-\alpha}\leq p\leq \frac{n+\alpha+2\gamma}{n-\alpha}$$, we only require $$u(x) \in L^{\frac{n(p-1)}{\alpha}}(R^{n}_{+})$$.

Motivated by [7], in this paper we consider the Dirichlet problem for the following pseudo differential system in $${R^{n}_{+}}$$:

$$\textstyle\begin{cases} (-\Delta)^{\frac{\alpha}{2}}u_{1}=x^{\gamma}_{n}u_{1}^{\alpha _{1}}(x)u_{2}^{\beta_{1}}(x), &\mbox{x \in R^{n}_{+}},\\ (-\Delta)^{\frac{\alpha}{2}}u_{2}=x^{\gamma}_{n}u_{1}^{\alpha _{2}}(x)u_{2}^{\beta_{2}}(x), &\mbox{x \in R^{n}_{+}},\\ u_{1}(x)=u_{2}(x)=0, &\mbox{x \notin R^{n}_{+}}, \end{cases}$$
(1.3)

where $$\gamma\geq0$$, $$0<\alpha<2$$, $$\alpha_{i}$$, $$\beta_{i} >0$$, $$i=1,2$$.

First, we use the maximum principle and the Liouville theorem in $${R^{n}_{+}}$$ in [6] to show that the positive solutions of problem (1.3) satisfy the following integral equations under some weak integrability condition:

$$\textstyle\begin{cases} u_{1}(x)=c_{1}x^{\frac{\alpha}{2}}_{n}+\int _{R^{n}_{+}}G(x,y)y^{\gamma}_{n}u_{1}^{\alpha_{1}}(y)u_{2}^{\beta _{1}}(y)\,dy, &\mbox{x \in R^{n}_{+}},\\ u_{2}(x)=c_{2}x^{\frac{\alpha}{2}}_{n}+\int _{R^{n}_{+}}G(x,y)y^{\gamma}_{n}u_{1}^{\alpha_{2}}(y)u_{2}^{\beta _{2}}(y)\,dy, &\mbox{x \in R^{n}_{+}}, \end{cases}$$
(1.4)

where

$$G(x,y)=\frac{A_{n,\alpha }}{\vert x-y\vert ^{n-\alpha}} \int_{0}^{\frac{4 x_{n} y_{n}}{\vert x-y\vert ^{2}}}\frac {b^{\frac{\alpha}{2}-1}}{(1+b)^{\frac{n}{2}}}\,db$$
(1.5)

is the Green’s function associated with $$(-\Delta)^{\frac{\alpha}{2}}$$ in $$R^{n}_{+}$$ and $$A_{n,\alpha}$$ is a constant depending on n and α.

Then we use Kelvin transform and the method of moving planes in integral forms to prove that $$c_{1}$$ and $$c_{2}$$ must be 0. We derive that (1.3) is equivalent to the following integral equations under some locally integrable conditions:

$$\textstyle\begin{cases} u_{1}(x)=\int_{R^{n}_{+}}G(x,y)y^{\gamma}_{n}u_{1}^{\alpha _{1}}(y)u_{2}^{\beta_{1}}(y)\,dy, &\mbox{x \in R^{n}_{+}},\\ u_{2}(x)=\int_{R^{n}_{+}}G(x,y)y^{\gamma}_{n}u_{1}^{\alpha _{2}}(y)u_{2}^{\beta_{2}}(y)\,dy, &\mbox{x \in R^{n}_{+}}. \end{cases}$$
(1.6)

In the subcritical case and critical case: $$1< \alpha_{i}+\beta _{i}\leq\frac{n+\alpha+2\gamma}{n-\alpha}$$, we continue work on the integral systems (1.6) to show the nonexistence of positive solutions. That is, we have the following result.

### Theorem 1.1

Assume that $$u(x)=(u_{1}(x),u_{2}(x))$$ is a positive solution of equations (1.3). If $$\vert u\vert \in L^{\frac{n(\alpha_{i}+\beta_{i}-1)}{\alpha }}_{\mathrm{loc}}(R^{n}_{+})\cap L^{\infty}_{\mathrm{loc}}(R^{n}_{+})$$, then in the case $$1<\alpha_{i}+\beta_{i}\leq\frac{n+\alpha+2\gamma }{n-\alpha}$$, $$u(x)$$ is also a solution of integral equations (1.6), and vice versa.

Next, we establish the Liouville theorem for the integral equations as follows.

### Theorem 1.2

Assume that $$\alpha_{i}+\beta_{i} >\frac {n}{n-\alpha}$$ and $$\gamma\geq0$$, if $$\vert u\vert \in L^{\frac{n(\alpha _{i}+\beta_{i}-1)}{\alpha}}(R^{n}_{+})$$ is a nonnegative solution of the system of the integral equations (1.6), then $$u(x)\equiv0$$.

### Theorem 1.3

Assume that $$u(x)=(u_{1}(x),u_{2}(x))$$ is a nonnegative solution of equations (1.6). If $$\vert u\vert \in L^{\frac{n(\alpha_{i}+\beta _{i}-1)}{\alpha}}_{\mathrm{loc}}(R^{n}_{+})\cap L^{\infty}_{\mathrm{loc}}(R^{n}_{+})$$ and $$1<\alpha_{i}+\beta_{i}\leq\frac{n+\alpha+2\gamma}{n-\alpha}$$, we have $$u(x)\equiv0$$.

### Remark 1

In this paper we use the new method in [6] to prove Theorem 1.1, we believe that this new approach can be applied to a variety of other situations.

## Equivalence between the two systems

The proof of Theorem 1.1 is based on the following maximum principle and the Liouville theorem.

### Proposition 2.1

[8]

Let Ω be a bounded open set, and let $$f(x)$$ be a lower-semicontinuous function in Ω̄ such that $$(-\Delta )^{\frac{\alpha}{2}}f(x)\geq0$$ in Ω and $$f(x)\geq0$$ in $${R^{n}} \setminus\Omega$$. Then $$f(x)\geq0$$ in $${R^{n}}$$.

### Proposition 2.2

[8]

If $$f (x)\in \mathcal {L}_{\alpha}$$ and $$(-\Delta)^{\frac{\alpha }{2}}f(x)\geq0$$ in an open set, then $$f(x)$$ is lower semicontinuous in Ω.

### Theorem 2.1

[9]

Let $$0 < \alpha< 2$$, $$u \in \mathcal {L}_{\alpha}$$. Assume u is a nonnegative solution of

$$\textstyle\begin{cases} (-\Delta)^{\frac{\alpha}{2}}u(x)=0, &x \in{R^{n}_{+}}, \\ u(x)\equiv0, &x \notin{R^{n}_{+}}. \end{cases}$$
(2.1)

Then we have either

$$u(x)\equiv0 x\in R^{n}\quad\textit{or}\quad u(x)=\textstyle\begin{cases} Cx_{n}^{\frac{\alpha}{2}}, &x \in{R^{n}_{+}}, \\ 0, &x \notin{R^{n}_{+}}, \end{cases}$$

for some positive constant C.

### Proof

Assume $$u(x)\in \mathcal {L}_{\alpha}$$ is a positive solution of the system of the fractional nonlinear PDEs system:

$$\textstyle\begin{cases} (-\Delta)^{\frac{\alpha}{2}}u_{i}(x)=x^{\gamma}_{n}u_{1}^{\alpha _{i}}(x)u_{2}^{\beta_{i}}(x), &\mbox{x \in R^{n}_{+}},\\ u_{i}(x)=0, &\mbox{x \notin R^{n}_{+}}, \end{cases}$$
(2.2)

where $$i=1,2$$, $$\gamma\geq0$$, $$0<\alpha<2$$, $$\alpha_{i}$$, $$\beta_{i} >0$$.

We first show that

$$\int_{R^{n}_{+}}G(x,y){y^{\gamma}_{n}} u^{\alpha_{i}}_{1}(y)u^{\beta _{i}}_{2}(y)\,dy< \infty.$$
(2.3)

Set $$P_{R}:=(0, \ldots, 0,R )\in R^{n}_{+}$$, $$B^{+}_{R}(P_{R}):= \{x\in R^{n} : \vert x-P_{R} \vert < R \}$$, the ball of radius R centered at $$P_{R}$$. Let

$$v_{i}^{R}(x)= \int_{B^{+}_{R}(P_{R})}G_{R}(x,y)y^{\gamma}_{n}u_{1}^{\alpha _{i}}(y)u_{2}^{\beta_{i}}(y) \,dy,$$
(2.4)

where $$G_{R}(x,y)$$, the Green’s function on the ball $$B^{+}_{R}(P_{R})$$, was given in [10],

$$G_{R}(x,y)=\frac{A_{n,\alpha}}{ \vert x-y\vert ^{n-\alpha}} \biggl[1-\frac{B_{n,\alpha}}{(s_{R}+t_{R})^{\frac{(n-2)}{2}}} { \int_{0}^{\frac{s_{R}}{t_{R}}}\frac{(s_{R}-t_{R}b)^{\frac {(n-2)}{2}}}{b^{\frac{\alpha}{2}}(1+b)}}\,db \biggr],\quad x, y \in B^{+}_{ R }(P_{R}),$$

here $$s_{R}=\frac{\vert x-y\vert ^{2}}{R^{2}}$$, $$t_{R}=(1-\frac {\vert x-P_{R}\vert ^{2}}{R^{2}})(1-\frac{\vert y-P_{R}\vert ^{2}}{R^{2}})$$, $$A_{n,\alpha}$$, and $$B_{n,\alpha}$$ are constants depending on n and α.

From the local bounded-ness assumption on u, one can see that, for each $$R > 0$$, $$v_{i}^{R}(x)$$ is well defined and is continuous. Moreover,

$$\textstyle\begin{cases} (-\Delta)^{\frac{\alpha}{2}}v_{i}^{R}(x)=x^{\gamma}_{n}u_{1}^{\alpha _{i}}(x)u_{2}^{\beta_{i}}(x), &\mbox{x\in B^{+}_{R}(P_{R})},\\ v_{i}^{R}(x)=0, &\mbox{x\notin B^{+}_{R}(P_{R})}. \end{cases}$$
(2.5)

Let $$w_{i}^{R}(x)=u_{i}(x)-v_{i}^{R}(x)$$, by (2.2) and (2.5), we derive

$$\textstyle\begin{cases} (-\Delta)^{\frac{\alpha}{2}}w_{i}^{R}(x)=0, &\mbox{x\in B^{+}_{R}(P_{R})},\\ w_{i}^{R}(x)\geq0, &\mbox{ x\notin B^{+}_{R}(P_{R})}. \end{cases}$$
(2.6)

Applying the maximum principle (see Proposition 2.1), we derive that

$$w_{i}^{R}(x) \geq0 ,\quad \forall x\in B^{+}_{R}(P_{R}).$$
(2.7)

It is easy to prove that

$$v_{i}^{R}(x)\rightarrow v_{i}(x)= \int_{R^{n}_{+}}G(x,y)y^{\gamma }_{n}u_{1}^{\alpha_{i}}(y)u_{2}^{\beta_{i}}(y) \,dy , \quad \mbox{as }R\rightarrow \infty.$$
(2.8)

Obviously,

$$\textstyle\begin{cases} (-\Delta)^{\frac{\alpha}{2}}v_{i}(x)=x^{\gamma}_{n}u_{1}^{\alpha _{i}}(x)u_{2}^{\beta_{i}}(x),&\mbox{x\in R^{n}_{+}},\\ v_{i}(x)\equiv0,&\mbox{x\notin R^{n}_{+}}. \end{cases}$$
(2.9)

Denote $$w_{i}(x)=u_{i}(x)-v_{i}(x)$$. Using (2.2), (2.7), (2.8), and (2.9), we have

$$\textstyle\begin{cases} (-\Delta)^{\frac{\alpha}{2}}w_{i}(x)=0,& w_{i}(x)\geq0,\mbox{ x\in {R^{n}_{+}}},\\ w_{i}(x)\equiv0, &\mbox{ x\notin{R^{n}_{+}} }. \end{cases}$$

Applying the Liouville theorem (see Theorem 2.1), we deduce that either

$$w_{i}(x)\equiv0 ,\quad x \in{R^{n}} \quad\mbox{or}\quad w_{i} (x)\equiv c_{i}x_{n}^{\frac{\alpha}{2}},\quad \forall x \in{R^{n}_{+}}, i=1,2,$$

for some positive constants $$c_{i}>0$$, then we could write $$w_{i} = c_{i}x_{n}^{\frac{\alpha}{2}}$$, $$c_{i}\geq0$$. That is, the solutions of (2.2) satisfy

$$u_{i}(x)=c_{i}x^{\frac{\alpha}{2}}_{n}+\int _{R^{n}_{+}}G(x,y)y^{\gamma}_{n} u_{1}^{\alpha_{i}}(y)u_{2}^{\beta_{i}}(y)\,dy, \quad x \in R^{n}_{+}, i=1,2,$$
(2.10)

where $$c_{i}\geq0$$, $$G(x,y)$$ is defined in (1.5).

Next we need to prove $$c_{i}$$ must be zero for $$i=1,2$$. To this end, we employ a certain type of Kelvin transform and the method of moving planes in integral forms.

For $$z^{0}=\{z^{0}_{1}, \ldots,z^{0}_{n-1} ,0 \}\in\partial {R^{n}_{+}}$$, let $$\bar{u}_{i}^{z^{0}}(x)=\bar{u}_{i}(x) =\frac{1}{\vert x-z^{0}\vert ^{n-\alpha}}u_{i}(\frac{x-z^{0}}{\vert x-z^{0}\vert ^{2}}+z^{0})$$, be the Kelvin transform of $$u_{i}(x)$$ centered at $$z^{0}$$.

Through a straightforward calculation by (2.10), we derive

$$\bar{u}_{i}(x) =\frac{c_{i}x_{n}^{\frac{\alpha}{2}}}{\vert x-z^{0}\vert ^{n-\alpha}}+\frac {1}{\vert x-z^{0}\vert ^{n-\alpha}} \int_{R^{n}_{+}}G \biggl(\frac{x-z^{0}}{\vert x-z^{0}\vert ^{2}}+z^{0},y \biggr)y^{\gamma }_{n}u_{1}^{\alpha_{i}}(y)u_{2}^{\beta_{i}}(y) \,dy.$$

Let $$y=\frac{z-z^{0}}{\vert z-z^{0}\vert ^{2}}+z^{0}$$, then $$dy=\frac {1}{\vert z-z^{0}\vert ^{2n}}\,dz$$,

\begin{aligned} \bar{u}_{i}(x) ={}&\frac{c_{i}x_{n}^{\frac{\alpha}{2}}}{\vert x-z^{0}\vert ^{n-\alpha}}+ \frac {1}{\vert x-z^{0}\vert ^{n-\alpha}} \int_{R^{n}_{+}}G(x,z)\bigl\vert x-z^{0}\bigr\vert ^{n-\alpha} \bigl\vert z-z^{0}\bigr\vert ^{n-\alpha} \\ &{}\times\frac{\vert \frac{z_{n}}{\vert z-z^{0}\vert ^{2}}\vert ^{\gamma }u^{\alpha_{i}}_{1}(\frac{z-z^{0}}{\vert z-z^{0}\vert ^{2}}+z^{0})u^{\beta _{i}}_{2}(\frac{z-z^{0}}{\vert z-z^{0}\vert ^{2}}+z^{0})}{\vert z-z^{0}\vert ^{2n}}\,dz \\ ={}&\frac{c_{i}x_{n}^{\frac{\alpha}{2}}}{\vert x-z^{0}\vert ^{n-\alpha}}+ \int _{R^{n}_{+}}G(x,z)\frac{z_{n}^{\gamma}}{ \vert z-z^{0}\vert ^{n+\alpha+2\gamma}} \frac{u^{\alpha_{i}}_{1}(\frac {z-z^{0}}{\vert z-z^{0}\vert ^{2}}+z^{0})}{\vert z-z^{0}\vert ^{(n-\alpha)\alpha_{i}}} \\ &{}\times\frac{u^{\beta_{i}}_{2}(\frac {z-z^{0}}{\vert z-z^{0}\vert ^{2}}+z^{0})}{\vert z-z^{0}\vert ^{(n-\alpha)\beta_{i}}} \bigl\vert z-z^{0}\bigr\vert ^{(n-\alpha)\alpha_{i}} \bigl\vert z-z^{0}\bigr\vert ^{(n-\alpha)\beta_{i}}\,dz \\ ={}&\frac{c_{i}x_{n}^{\frac{\alpha}{2}}}{\vert x-z^{0}\vert ^{n-\alpha}}+ \int _{R^{n}_{+}}G(x,z) \frac{z_{n}^{\gamma}\bar{u}^{\alpha_{i}}_{1}(z)\bar{u}^{\beta _{i}}_{2}(z)}{\vert z-z^{0}\vert ^{n+2\gamma+\alpha-(n-\alpha)(\alpha _{i}+\beta_{i})}}\,dz \\ ={}&\frac{c_{i}x_{n}^{\frac{\alpha}{2}}}{\vert x-z^{0}\vert ^{n-\alpha}}+ \int _{R^{n}_{+}}G(x,y)\frac{y_{n}^{\gamma}\bar{u}^{\alpha_{i}}_{1}(y)\bar{u}^{\beta_{i}}_{2}(y)}{\vert y-z^{0}\vert ^{\delta}}\,dy, \quad \forall x\in {R^{n}_{+}}\backslash B_{\epsilon}\bigl(z^{0} \bigr), \end{aligned}
(2.11)

where $$\epsilon>0$$, $$\delta=n+\alpha+2\gamma-(n-\alpha)(\alpha _{i}+\beta_{i})$$.

Then we have $$\delta=0$$ i.e. $$\alpha_{i}+\beta_{i}=\frac{n+\alpha+2\gamma}{n-\alpha}$$, it is called critical case. When $$\delta>0$$, we have $$1<\alpha_{i}+\beta_{i}<\frac{n+\alpha +2\gamma}{n-\alpha}$$ and it is called the subcritical case. In this section, we consider these two cases $$1<\alpha_{i}+\beta_{i}\leq \frac{n+\alpha+2\gamma}{n-\alpha}$$, then we have $$\delta\geq0$$.

Now we introduce some basic notations in the method of moving planes. For a given real number λ, denote $${\Sigma_{\lambda}}=\{x=(x_{1},\ldots,x_{n})\in R^{n}_{+}\mid x_{1}< \lambda\}$$, $${T_{\lambda}}=\{x=(x_{1},\ldots,x_{n})\in R^{n}_{+}\mid x_{1}= \lambda\}$$. Let $$x^{\lambda}=(2\lambda-x_{1},x_{2},\ldots,x_{n})$$ be the reflection of the point $$x=(x_{1},\ldots,x_{n})$$ about the plane $$T_{\lambda}$$, and $$\bar{u}_{i}^{\lambda}(x)=\bar{u}_{i}(x^{\lambda})$$, $$\bar{w}^{\lambda}_{i}(x)=\bar{u}^{\lambda}_{i}(x)-\bar{u}_{i}(x)$$.

For $$x,y \in{\Sigma_{\lambda}}$$, $$x\neq y$$, by [5], we have

$$G(x,y) =G \bigl(x^{\lambda},y^{\lambda} \bigr) > G \bigl(x,y^{\lambda} \bigr)=G \bigl(x^{\lambda},y \bigr).$$
(2.12)

Obviously, we have

\begin{aligned} \bar{u}_{i}(x)={}&\frac{c_{i}x_{n}^{\frac{\alpha }{2}}}{\vert x-z^{0}\vert ^{n-\alpha}}+ \int_{R^{n}_{+}}G(x,y) \frac{y_{n}^{\gamma}\bar{u}^{\alpha_{i}}_{1}(y)\bar{u}^{\beta _{i}}_{2}(y)}{\vert y-z^{0}\vert ^{\delta}}\,dy \\ ={}&\frac{c_{i}x_{n}^{\frac{\alpha}{2}}}{\vert x-z^{0}\vert ^{n-\alpha}}+ \int _{\Sigma_{\lambda}}G(x,y) \frac{y_{n}^{\gamma}\bar{u}^{\alpha_{i}}_{1}(y)\bar{u}^{\beta _{i}}_{2}(y)}{\vert y-z^{0}\vert ^{\delta}}\,dy \\ &{}+ \int_{\Sigma_{\lambda}}G \bigl(x^{\lambda},y \bigr)\frac{y_{n}^{\gamma}\bar{u}^{\alpha_{i}}_{1}(y^{\lambda}) \bar{u}^{\beta_{i}}_{2}(y^{\lambda})}{\vert y^{\lambda}-z^{0}\vert ^{\delta}} \,dy , \\ \bar{u}^{\lambda}_{i}(x) ={}& \frac{c_{i}x_{n}^{\frac{\alpha}{2}}}{\vert x^{\lambda}-z^{0}\vert ^{n-\alpha }}+ \int_{\Sigma_{\lambda}}G \bigl(x^{\lambda},y \bigr) \frac{y_{n}^{\gamma}\bar{u}^{\alpha_{i}}_{1}(y)\bar{u}^{\beta _{i}}_{2}(y)}{\vert y-z^{0}\vert ^{\delta}} \,dy \\ &{}+ \int_{\Sigma_{\lambda }}G(x,y)\frac{y_{n}^{\gamma}\bar{u}^{\alpha_{i}}_{1}(y^{\lambda })\bar{u}^{\beta_{i}}_{2}(y^{\lambda})}{\vert y^{\lambda}-z^{0} \vert ^{\delta}}\,dy . \end{aligned}

By an elementary calculation, we derive

\begin{aligned} \bar{u}_{i}(x)- \bar{u}^{\lambda}_{i}(x) ={}& \frac{c_{i}x_{n}^{\frac{\alpha}{2}}}{\vert x-z^{0}\vert ^{n-\alpha}}- \frac {c_{i}x_{n}^{\frac{\alpha}{2}}}{\vert x^{\lambda}-z^{0}\vert ^{n-\alpha}} \\ &{}+ \int_{\Sigma_{\lambda}} \bigl[G(x,y)-G \bigl(x^{\lambda},y \bigr) \bigr]y_{n}^{\gamma } \biggl[\frac{\bar{u}^{\alpha_{i}}_{1}(y)\bar{u}^{\beta _{i}}_{2}(y)}{\vert y-z^{0}\vert ^{\delta}}-\frac{\bar{u}^{\alpha _{i}}_{1}(y^{\lambda})\bar{u}^{\beta_{i}}_{2}(y^{\lambda })}{\vert y^{\lambda}-z^{0}\vert ^{\delta}} \biggr]\,dy \\ \leq{}& \int_{\Sigma_{\lambda}} \bigl[G(x,y)-G \bigl(x^{\lambda},y \bigr) \bigr]y_{n}^{\gamma } \biggl[\frac{\bar{u}^{\alpha_{i}}_{1}(y)\bar{u}^{\beta _{i}}_{2}(y)}{\vert y-z^{0}\vert ^{\delta}}-\frac{\bar{u}^{\alpha _{i}}_{1}(y^{\lambda})\bar{u}^{\beta_{i}}_{2}(y^{\lambda })}{\vert y^{\lambda}-z^{0}\vert ^{\delta}} \biggr]\,dy \\ \leq{}& \int_{\Sigma_{\lambda}} \bigl[G(x,y)-G \bigl(x^{\lambda},y \bigr) \bigr] \frac {y_{n}^{\gamma}[\bar{u}^{\alpha_{i}}_{1}(y)\bar{u}^{\beta _{i}}_{2}(y)-\bar{u}^{\alpha_{i}}_{1}(y^{\lambda})\bar{u}^{\beta _{i}}_{2}(y^{\lambda})]}{\vert y-z^{0}\vert ^{\delta}}\,dy. \end{aligned}
(2.13)

The proof consists of two steps. In step 1, we will show that, for λ sufficiently negative,

$$\bar{w}_{i}^{\lambda}(x)= \bar{u}_{i}^{\lambda}(x)- \bar{u}_{i}(x)\geq0 ,\quad \mbox{a.e. } \forall x\in \Sigma_{\lambda} .$$

In step 2, we deduce that $$T_{\lambda}$$ can be moved to the right all the way to $$z_{1}^{0}$$. Furthermore, we obtain $$\bar{w}_{z^{0}_{1}}\equiv0$$, $$\forall x \in\Sigma_{z_{1}^{0}}$$.

Step 1. (Prepare to move the plane from near $$x_{1}=-\infty$$.) In this step, we will show that, for λ sufficiently negative, $$\epsilon>0$$ sufficiently small

$$\bar{u}_{i}^{\lambda}(x)\geq\bar{u}_{i}(x) ,\quad \mbox{a.e. } \forall x\in{\Sigma_{\lambda} \backslash{B_{\epsilon } \bigl( \bigl({z^{0}} \bigr)^{\lambda} \bigr)}},$$
(2.14)

where $$({z^{0}})^{\lambda}$$ is the reflection of $$z^{0}$$ about the plane $$T_{\lambda}$$. Define $$\Gamma^{\lambda}_{i}=\{x\in\Sigma_{\lambda} \backslash B_{\epsilon}(({z^{0}})^{\lambda}) \mid\bar{u}^{\lambda}_{i}(x)<\bar{u}_{i}(x) \}$$, the sets where the inequalities (2.14) are violated. We will prove that $$\Gamma^{\lambda}_{i}$$ are empty, where $$i=1,2$$.

Without loss of generality, we consider $$\bar{u}_{1}$$. Denote $$\Sigma ^{\lambda}_{i}=\{x\in\Sigma_{\lambda} \backslash B_{\epsilon }(({z^{0}})^{\lambda}) \mid\bar{u}_{1}^{\alpha_{i}}(x^{\lambda})\times \bar{u}_{2}^{\beta_{i}}(x^{\lambda})<\bar{u}_{1}^{\alpha_{i}}(x)\bar{u}_{2}^{\beta_{i}}(x) \}$$, for $$y\in\Sigma^{\lambda}_{1}$$, we may assume that $$\bar{u}_{1}(y)>\bar{u}^{\lambda}_{1}(y)$$ and $$\bar{u}_{2}(y)\leq\bar{u}^{\lambda}_{2}(y)$$. Define

$$\bar{w}^{\lambda}_{i}(y)=\textstyle\begin{cases} 0,&\mbox{for } \bar{u}_{i}(y)< \bar{u}^{\lambda}_{i}(y) , \\ \bar{u}_{i}(y)-\bar{u}^{\lambda}_{i}(y), &\mbox{for } \bar{u}_{i}(y)>\bar{u}^{\lambda}_{i}(y), \end{cases}$$

and $$\bar{w}^{\lambda}(y)=(\bar{w}^{\lambda}_{1}(y),\bar{w}^{\lambda}_{2}(y) )$$. By the expression of $$G(x, y)$$, it is easy to see

$$G(x, y)\leq\frac{A_{n,\alpha}}{ \vert x-y\vert ^{n-\alpha}}.$$
(2.15)

Applying the mean value theorem, combining (2.13) and (2.15), we have, for $$x\in\Gamma^{\lambda}_{i}$$,

\begin{aligned} \bar{u}_{i}(x)-\bar{u}^{\lambda}_{i}(x) \leq{}& \int_{\Sigma_{\lambda}}G(x,y)\frac{y_{n}^{\gamma}[\bar{u}^{\alpha_{i}}_{1}(y)\bar{u}^{\beta_{i}}_{2}(y)-\bar{u}^{\alpha _{i}}_{1}(y^{\lambda})\bar{u}^{\beta_{i}}_{2}(y^{\lambda })]}{\vert y-z^{0}\vert ^{\delta}}\,dy \\ \leq{}& \int_{\Sigma^{\lambda}_{i}}G(x,y)\frac{y_{n}^{\gamma}[\bar{u}^{\alpha_{i}}_{1}(y)\bar{u}^{\beta_{i}}_{2}(y)-\bar{u}^{\alpha _{i}}_{1}(y^{\lambda})\bar{u}^{\beta_{i}}_{2}(y^{\lambda })]}{\vert y-z^{0}\vert ^{\delta}}\,dy \\ ={}& \int_{\Sigma^{\lambda}_{i}}G(x,y)\frac{y_{n}^{\gamma}\{[\bar{u}^{\alpha_{i}}_{1}(y)-\bar{u}^{\alpha_{i}}_{1}(y^{\lambda})]\bar{u}^{\beta_{i}}_{2}(y)+ \bar{u}^{\alpha_{i}}_{1}(y)[\bar{u}^{\beta_{i}}_{2}(y)-u^{\beta _{i}}_{2}(y^{\lambda})]\} }{\vert y-z^{0}\vert ^{\delta}} \,dy \\ \leq{}&c \int_{\Sigma^{\lambda}_{i}}G(x,y)\frac{y_{n}^{\gamma}[\bar{u}^{\alpha_{i}}_{1}(y)-\bar{u}^{\alpha_{i}}_{1}(y^{\lambda})]\bar{u}^{\beta_{i}}_{2}(y)}{\vert y-z^{0}\vert ^{\delta}} \,dy \\ \leq{}&c \int_{\Sigma^{\lambda}_{i}}G(x,y)\frac{y_{n}^{\gamma}\psi ^{\alpha_{i}-1}_{1}(y)[\bar{u}_{1}(y)-\bar{u}_{1}(y^{\lambda})]\bar{u}^{\beta_{i}}_{2}(y)}{\vert y-z^{0}\vert ^{\delta}} \,dy \\ \leq{}&c \int_{\Sigma^{\lambda}_{i}}\frac{1}{\vert x-y\vert ^{n-\alpha}}\frac {y_{n}^{\gamma}\bar{u}^{\alpha_{i}-1}_{1}(y)[\bar{u}_{1}(y)-\bar{u}_{1}(y^{\lambda})]\bar{u}^{\beta_{i}}_{2}(y)}{\vert y-z^{0}\vert ^{\delta}} \,dy. \end{aligned}
(2.16)

Noticing $$\Sigma_{i}^{\lambda} \subseteq\Gamma^{\lambda}_{j}$$ for some j, applying the Hardy-Littlewood-Sobolev inequality and the Hölder inequality we obtain for any $$q>\frac{n}{n-\alpha}$$,

$$\Vert \bar{w}_{\lambda} \Vert _{L^{q}({\Gamma_{\lambda}})}\leq c\biggl\Vert \frac{y_{n}^{\gamma} \vert \bar{u}\vert ^{\alpha_{i}+\beta_{i}-1}}{\vert y-z^{0}\vert ^{\delta}} \bar{w}_{\lambda}\biggr\Vert _{L^{\frac{nq}{n+\alpha q}}({\Gamma_{\lambda}}) } \leq c\biggl\Vert \frac{y_{n}^{\gamma} \vert \bar{u}\vert ^{\alpha_{i}+\beta_{i}-1}}{\vert y-z^{0}\vert ^{\delta}}\biggr\Vert _{L^{\frac {n}{\alpha}}({\Gamma_{\lambda}}) }\Vert \bar{w}_{\lambda} \Vert _{L^{q}({ \Gamma_{\lambda}})},$$
(2.17)

where $$\Gamma_{\lambda} =\Gamma^{\lambda}_{1}\cup\Gamma ^{\lambda}_{2}$$. Since $$\gamma\geq0$$, we can easily see that $$y^{\gamma}_{n}$$ is bounded in each bounded domain $$\Omega\subset R^{n}_{+}$$. Therefore, by our assumption $$\vert u\vert \in L^{\frac{n(\alpha_{i}+\beta _{i}-1)}{\alpha}}_{\mathrm{loc}}(R^{n}_{+})$$, i.e. $$\vert u\vert ^{\alpha _{i}+\beta_{i}-1}\in L^{\frac{n}{\alpha}}_{\mathrm{loc}}(R^{n}_{+})$$, we derive

$$y^{\gamma}_{n} \vert u\vert ^{\alpha_{i}+\beta_{i}-1} \in L^{\frac{n}{\alpha }}_{\mathrm{loc}} \bigl(R^{n}_{+} \bigr).$$
(2.18)

Hence, we obtain

$$\int_{ \hat{\Omega}} \biggl[\frac{y^{\gamma}_{n}\vert \bar{u}\vert ^{\alpha_{i}+\beta _{i}-1}(y)}{{\vert y-z^{0}\vert ^{\delta}}} \biggr]^{\frac{n}{\alpha}} \,dy= \int_{ \Omega} \bigl(y^{\gamma}_{n} \vert u \vert ^{\alpha_{i}+\beta_{i}-1}(y) \bigr)^{\frac{n}{\alpha}}\,dy < \infty$$
(2.19)

for any domain Ω that is a positive distance away from the $$z^{0}$$ and Ω̂ is the image of Ω about the Kelvin transform. By (2.19), for λ sufficiently negative and for $$\epsilon > 0$$ sufficiently small, $$c\Vert \frac{y^{\gamma}_{n}\vert \bar{u}\vert ^{\alpha_{i}+\beta_{i}-1}}{\vert y-z^{0}\vert ^{\delta}} \Vert _{L^{\frac{n}{\alpha }}({\Gamma_{\lambda}}) }$$ can be made very small. Combining this with (2.17), we arrive at $$\Vert \bar{w}_{\lambda} \Vert _{L^{q}({\Gamma_{\lambda}})}=0$$, and $$\Gamma^{\lambda}_{i}$$ must be of measure zero. Hence (2.14) holds.

Step 2. (Move the plane to the limiting position to derive symmetry.)

Inequality (2.14) provides a starting point to move the plane $$T_{\lambda}$$. Now we start to move the plane $$T_{\lambda}$$ along the $$x_{1}$$ direction as long as (2.14) holds. Define

$$\lambda_{0} =\operatorname{sup} \bigl\{ \lambda\leq z^{0}_{1} \mid \bar{u}_{i}^{\mu}(x)\geq \bar{u}_{i}(x) , \mbox{ a.e. } \forall x\in{ \Sigma_{\mu}} , \mu\leq\lambda \bigr\} .$$

We prove that $$\lambda_{0}= z^{0}_{1}$$. On the contrary, suppose that $$\lambda_{0}< z^{0}_{1}$$. We will show that $$\bar{u}(x)$$ is symmetric about the plane $$T_{\lambda _{0}}$$, that is,

$$\bar{u}(x)\equiv\bar{u}^{\lambda_{0}}(x) , \quad\mbox{a.e. } \forall x\in{ \Sigma _{\lambda_{0}}\backslash{B_{\epsilon} \bigl( \bigl(z^{0} \bigr)^{\lambda_{0}} \bigr)}}.$$
(2.20)

Suppose (2.20) is not true, then, for such $$\lambda_{0}< z^{0}_{1}$$, for all $$i=1,2$$, we have

$$\bar{u}_{i}^{\lambda_{0}}(x) >\bar{u}_{i}(x)\quad \mbox{a.e. } x \in{\Sigma_{\lambda _{0}}\backslash{B_{\epsilon} \bigl( \bigl(z^{0} \bigr)^{\lambda_{0}} \bigr)}}.$$
(2.21)

In fact, by (2.14), we have two cases for all $$i=1,2$$; one is

$$\bar{u}_{i}^{\lambda_{0}}(y)>\bar{u}_{i}(y) \quad\mbox{on a set of positive measure, i=1 and 2.}$$
(2.22)

For the other case, without loss of generality, we may assume that $$\bar{u}_{1}^{\lambda_{0}}(z_{1})>\bar{u}_{1}(z_{1})$$ and

$$\bar{u}_{2}^{\lambda_{0}}(z_{2})=\bar{u}_{2}(z_{2}).$$
(2.23)

For the first case, (2.21) is proved. For the other case, we have

$$\bar{u}_{1}^{\alpha_{i}}(y)\bar{u}_{2}^{\beta_{i}}(y)- \bar{u}_{1}^{\alpha_{i}} \bigl(y^{\lambda_{0}} \bigr)\bar{u}_{2}^{\beta_{i}} \bigl(y^{\lambda _{0}} \bigr)< 0.$$
(2.24)

Combining (2.23) with (2.24), we obtain

$$0=\bar{u}_{i}(z_{i})-\bar{u}_{i}^{\lambda_{0}}(z_{i}) < \int_{\Sigma_{\lambda_{0}} } \bigl[G(z_{i},y)-G \bigl({z_{i}}^{\lambda},y \bigr) \bigr]\frac {y^{\gamma}_{n} [\bar{u}_{1}^{\alpha_{i}}(y)\bar{u}_{2}^{\beta_{i}}(y)-\bar{u}_{1}^{\alpha_{i}}(y^{\lambda_{0}})\bar{u}_{2}^{\beta_{i}}(y^{\lambda _{0}})]}{\vert y-z^{0}\vert ^{\delta}}\,dy< 0.$$

This is impossible. Hence (2.21) holds. Next based on (2.21), we will verify that the plane can be moved further to the right. More precisely, there exists a $$\zeta> 0$$ such that, for all $$\lambda\in[\lambda_{0}, \lambda_{0} + \zeta)$$ $$\bar{u}_{i}^{\lambda}(x)\geq\bar{u}_{i} (x)$$, a.e. $$\forall x\in\Sigma _{\lambda}\backslash B_{\epsilon}(z^{0})^{\lambda_{0}}$$. By inequality (2.17), we have

$$\Vert \bar{w}_{\lambda} \Vert _{L^{q}(\Gamma_{\lambda})}\leq \biggl\{ \int_{\Gamma_{\lambda}} \biggl(\frac{y^{\gamma}_{n}\vert \bar{u}\vert ^{\alpha_{i}+\beta_{i}-1}(y)}{\vert y-z^{0}\vert ^{\delta}} \biggr)^{\frac{n}{\alpha}}\,dy \biggr\} ^{\frac{\alpha}{n}} \Vert \bar{w}_{\lambda} \Vert _{L^{q}(\Gamma_{\lambda})}.$$
(2.25)

Equation (2.19) ensures that one can choose η sufficiently small so that, for all λ in $$[\lambda_{0}, \lambda _{0}+\eta)$$,

$$c \biggl\{ \int_{\Gamma_{\lambda}} \biggl(\frac{y^{\gamma}_{n}\vert \bar{u}\vert ^{\alpha_{i}+\beta_{i}-1}(y)}{\vert y-z^{0}\vert ^{\delta}} \biggr)^{\frac{n}{\alpha}}\,dy \biggr\} ^{\frac{\alpha}{n}} \leq\frac{1}{2}.$$
(2.26)

We postpone the proof of this inequality (2.26) for a moment.

Now combining (2.25) and (2.26), we have $$\Vert \bar{w}_{\lambda} \Vert _{L^{q}(\Gamma_{\lambda})}=0$$, and $$\Gamma_{\lambda}$$ must be of measure zero. Hence, for these values of $$\lambda> \lambda_{0}$$, we have $$\bar{w}_{\lambda}(x)\geq0$$, a.e. $$\forall x\in\Sigma_{\lambda _{0}}\backslash B_{\epsilon}((z^{0})^{\lambda})$$, $$\epsilon>0$$. This contradicts the definition of $$\lambda_{0}$$. Therefore (2.20) must hold. That is, if $$\lambda_{0}< z^{0}_{1}$$, then we must have

$$\bar{u}_{i}(x)\equiv\bar{u}_{i}^{\lambda}(x),\quad \mbox{a.e. } \forall x \in\Sigma _{\lambda_{0}}\backslash B_{\epsilon}\bigl( \bigl(z^{0} \bigr)^{\lambda_{0}} \bigr).$$
(2.27)

Recall that, by our assumption, $$c_{i_{0}}>0$$ and

$$\bar{u}_{i_{0}}(x)=\frac{c_{i_{0}}x^{\frac{\alpha }{2}}_{n}}{\vert x-z^{0}\vert ^{n-\alpha}}+ \int_{R^{n}_{+}}G(x,y)\frac{y^{\gamma}_{n}\bar{u}_{1}^{\alpha _{1}}(y)\bar{u}_{2}^{\beta_{1}}(y)}{\vert y-z^{0}\vert ^{\delta}}\,dy.$$
(2.28)

It follows that $$\bar{u}_{i_{0}}$$ is singular at $$z^{0}$$, hence by (2.27), $$\bar{u}_{i_{0}}$$ must also be singular at $$(z^{0})^{\lambda}$$. This is impossible, because $$z^{0}$$ is the only singularity of ū. Hence, we must have $$\lambda_{0}=z^{0}_{1}$$. Since ϵ is an arbitrary positive number, we have actually derived that

$$\bar{u}_{i}^{\lambda_{0}}(x)\geq\bar{u}_{i}(x), \quad\mbox{a.e. } \forall x\in\Sigma _{\lambda_{0}}, \lambda_{0}=z^{0}_{1} .$$

Entirely similarly, we can move the plane from near $$x_{1} = +\infty$$ to the left and obtain $$\bar{u}_{i}^{\lambda_{0}}(x)\leq\bar{u}_{i}(x)$$, a.e. $$\forall x\in\Sigma_{\lambda_{0}}$$, $$\lambda_{0}= z^{0}_{1}$$. Therefore we have $$\bar{w}_{\lambda_{0}} (x)\equiv0$$, a.e. $$\forall x\in\Sigma_{\lambda _{0}}$$, $$\lambda_{0}=z_{1}^{0}$$.

Now we prove inequality (2.26). For any small $$\eta>0$$, $$\forall\varepsilon>0$$, we can choose R sufficiently large so that

$$\biggl( \int_{(R^{n}_{+} \backslash B_{\varepsilon}(z^{0}))\backslash B_{R}} \biggl[\frac{y_{n}^{\gamma} \vert \bar{u}\vert ^{\alpha _{i}+\beta_{i}-1}(y)}{\vert y-z^{0}\vert ^{\delta}}\,dy \biggr]^{\frac{n}{\alpha}} \biggr)^{\frac {\alpha}{n}} \leq\eta.$$
(2.29)

For any $$\tau>0$$, define $$E_{i}^{\tau}=\{x \in({\Sigma_{\lambda_{0}}}\backslash {B_{\varepsilon}((z^{0})^{\lambda_{0}})}) \cap{B_{R}(0)} \mid\bar{u}_{i}^{\lambda_{0}}(x)- \bar{u}_{i}(x)>\tau \}$$, and $$F_{i}^{\tau}=\{ ({\Sigma_{\lambda_{0}}}\backslash {B_{\varepsilon}((z^{0})^{\lambda_{0}})}) \cap{B_{R}(0)} \} \backslash E_{i}^{\tau}$$. Obviously, $${\lim_{\tau\rightarrow0}}\mu(F_{i}^{\tau})=0$$.

For $$\lambda> {\lambda_{0}}$$, let $$D_{\lambda}=\{ ({\Sigma_{\lambda}}\backslash{B_{\varepsilon }((z^{0})^{\lambda})}) \backslash({\Sigma_{\lambda_{0}}}\backslash {B_{\varepsilon}((z^{0})^{\lambda_{0}})})\cap{B_{R}(0)} \}$$.

It is easy to see that

$$\bigl\{ {\Gamma^{\lambda}_{i}} \cap{B_{R}(0)} \bigr\} \subset \bigl({\Gamma ^{\lambda}_{i}} \cap E_{i}^{\tau} \bigr)\cup F_{i}^{\tau} \cup D_{\lambda}.$$
(2.30)

For λ sufficiently close to $$\lambda_{0}$$, $$\mu (D_{\lambda})$$ is very small. We will show that $$\mu(\Gamma^{\lambda}_{i} \cap E_{i}^{\tau} )$$ is sufficiently small as λ close to $$\lambda_{0}$$.

In fact, $$\bar{w}_{i}^{\lambda}(x)=\bar{u}_{i}^{\lambda}(x)-\bar{u}_{i}(x) = \bar{u}_{i}^{\lambda}(x)-\bar{u}_{i}^{\lambda_{0}}(x)+\bar{u}_{i}^{\lambda _{0}}(x) -\bar{u}_{i}(x) < 0$$, $$\forall x \in(\Gamma^{\lambda}_{i} \cap E_{i}^{\tau} )$$. Therefore, $$\bar{u}_{i}^{\lambda_{0}}(x)-\bar{u}_{i}^{\lambda}(x)>\bar{u}_{i}^{\lambda }(x)-\bar{u}_{i}^{\lambda}(x) >\tau$$, $$\forall x \in(\Gamma^{\lambda }_{i} \cap E_{i}^{\tau} )$$. It follows that

$$\bigl(\Gamma^{\lambda}_{i} \cap E_{i}^{\tau} \bigr) \subset H_{i}^{\tau} = \bigl\{ x \in B_{R}(0) \mid\bar{u}_{i}^{\lambda_{0}}(x)- \bar{u}_{i}^{\lambda}(x) >\tau \bigr\} .$$
(2.31)

By the well-known Chebyshev inequality, for fixed τ, as λ is close to $$\lambda_{0}$$, $$\mu (E_{i}^{\tau})$$ can be sufficiently small. By (2.30) and (2.31), we derive that $$\mu (\Gamma^{\lambda}_{i} \cap B_{R}(0) )$$ can be made as small as we wish. Combining this with (2.29), we deduce that (2.26) holds.

Since we can choose any direction that is perpendicular to the $$x_{n}$$-axis as the $$x_{1}$$ direction, we have actually shown that the Kelvin transform of the solution $$\bar{u}(x)$$ is rotationally symmetric about the line parallel to the $$x_{n}$$-axis and passing through $$z^{0}$$. Now we take any two points $$X^{1}$$ and $$X^{2}$$, with $$X^{l}=(x^{\prime l},x_{n})\in R^{n-1}\times[0,\infty)$$, $$l=1,2$$. Let $$z^{0}$$ be the projection of $$\bar{X}= \frac{X^{1}+X^{2}}{2}$$ on $$\partial R^{n}_{+}$$. Set $$Y^{l}= \frac{X^{l}-z^{0}}{\vert X^{l}-z^{0}\vert ^{2}}+z^{0}$$, $$l=1,2$$. From the above arguments, it is easy to see $$\bar{u}(Y ^{1}) = \bar{u}(Y^{2})$$, hence $$u(X^{1}) = u(X^{2})$$. This implies that $$u_{i}$$ is independent of $$x'=(x_{1},\ldots,x_{n-1})$$. That is, $$u_{i}=u_{i}(x_{n})$$, and we will show that this will contradict the finiteness of the integral $$\int_{R^{n}_{+}}G(x,y)y^{\gamma}_{n} u^{\alpha_{i}}_{1}(y)u^{\beta _{i}}_{2}(y)\,dy$$. To continue, we need the following lemma.

### Lemma 2.1

[5]

If $$\frac{t}{s}$$ is sufficiently small, then $$\forall x=(x',x_{n})$$, $$y=(y',y_{n})\in R^{n}_{+}$$, one can derive that

$$\frac{c_{n,\alpha}}{s^{\frac{n-\alpha}{2}}}\cdot\frac{t^{\frac {\alpha}{2}}}{s^{\frac{\alpha}{2}}}\leq G(x , y)\leq \frac{C_{n,\alpha}}{s^{\frac{n-\alpha}{2}}} \cdot\frac{t^{\frac {\alpha}{2}}}{s^{\frac{\alpha}{2}}}, \quad\textit{i.e. } G(x , y) \sim\frac{t^{\frac{\alpha}{2}}}{s^{\frac{n}{2}}}.$$

Here $$s=\vert x-y\vert ^{2}$$, $$t=4x_{n}y_{n}$$, $$c_{n,\alpha}$$, and $$C_{n,\alpha}$$ stand for different positive constants that only depend on n and α.

Set $$x=(x',x_{n})$$, $$y=(y',y_{n})\in R^{n-1}\times(0,+\infty)$$, $$r^{2}=\vert x'-y'\vert ^{2}$$ and $$a^{2}=\vert x_{n}-y_{n}\vert ^{2}$$. If $$u_{i}=u_{i}(x_{n})$$ is a solution of

$$u_{i}(x)= \int_{R^{n}_{+}}G(x,y)y^{\gamma}_{n} u^{\alpha_{i}}_{1}(y)u^{\beta _{i}}_{2} (y)\,dy,$$
(2.32)

then, for each fixed $$x\in R^{n}_{+}$$, letting R be large enough, by elementary calculations, we have

\begin{aligned} +\infty>u_{i}(x_{n})={}& \int^{\infty}_{0}y^{\gamma}_{n} u^{\alpha _{i}}_{1}(y_{n})u^{\beta_{i}}_{2}(y_{n}) \int_{R^{n-1}} G(x,y)\,dy'\,dy_{n} \\ \geq{}&C \int^{\infty}_{R}y^{\gamma}_{n} u^{\alpha_{i}}_{1}(y_{n})u^{\beta _{i}}_{2}(y_{n})y_{n}^{\frac{\alpha}{2}} \int_{R^{n-1}\backslash B_{R}(0)} \frac{1}{\vert x-y\vert ^{n}}\,dy' \,dy_{n} \\ \geq{}&C \int^{\infty}_{R}y^{\gamma}_{n} u^{\alpha_{i}}_{1}(y_{n})u^{\beta _{i}}_{2}(y_{n})y_{n}^{\frac{\alpha}{n-2}}{ \int^{\infty}_{R}} \frac {r^{2}}{(r^{2}+a^{2})^{\frac{n}{2}}}\,dr \,dy_{n} \\ \geq{}&C \int^{\infty}_{R}y^{\gamma}_{n} u^{\alpha_{i}}_{1}(y_{n})u^{\beta _{i}}_{2}(y_{n})y_{n}^{\frac{\alpha}{2}} \frac{1}{\vert x_{n}-y_{n}\vert }{ \int ^{\infty}_{\frac{R}{a}}} \frac{\tau^{n-2}}{(\tau^{2}+1)^{\frac {n}{2}}}\,d\tau \,dy_{n} \\ \geq{}&C \int^{\infty}_{R} u^{\alpha_{i}}_{1}(y_{n})u^{\beta _{i}}_{2}(y_{n})y_{n}^{\gamma+\frac{\alpha}{2}-1} \,dy_{n}. \end{aligned}
(2.33)

Equation (2.33) implies that there exists a sequence $$\{ y^{k}_{n}\}\rightarrow\infty$$ as $$k\rightarrow\infty$$, such that

$$u^{\alpha_{i}}_{1} \bigl(y^{k}_{n} \bigr)u^{\beta_{i}}_{2} \bigl(y^{k}_{n} \bigr) \bigl(y^{k}_{n} \bigr)^{\gamma +\frac{\alpha}{2}}\rightarrow0.$$
(2.34)

Similarly to (2.33), for any $$x=(0,x_{n})\in R^{n}_{+}$$, we derive that

$$+\infty>u_{i}(x_{n})\geq C_{0}{ \int_{0}^{\infty}} y_{n}^{\gamma}u^{\alpha _{i}}_{1}(y_{n})u^{\beta_{i}}_{2}(y_{n})y_{n}^{\frac{ \alpha}{2}}\frac{1}{\vert x_{n}-y_{n}\vert } \,dy_{n} x_{n}^{\frac{ \alpha}{2}}.$$
(2.35)

Let $$x_{n} = 2R$$ be sufficiently large. By (2.35), we deduce that

\begin{aligned} +\infty>u_{i}(x_{n})\geq{}& C_{0}{ \int_{0}^{1}} y_{n}^{\gamma}u^{\alpha _{i}}_{1}(y_{n})u^{\beta_{i}}_{2}(y_{n})y_{n}^{\frac{ \alpha}{2}}\frac{1}{\vert x_{n}-y_{n}\vert } \,dy_{n} x_{n}^{\frac{ \alpha}{2}} \\ \geq{}&\frac{C_{0}}{2R}(2R)^{\frac{\alpha}{2}} { \int_{0}^{1}} y_{n}^{\gamma }u^{\alpha_{i}}_{1}(y_{n})u^{\beta_{i}}_{2}(y_{n})y_{n}^{\frac{ \alpha }{2}}\,dy_{n} \geq C_{1}(2R)^{\frac{\alpha}{2}-1} =C_{1}x_{n}^{\frac{\alpha}{2}-1}. \end{aligned}
(2.36)

Then by (2.35) and (2.36), for $$x_{n} = 2R$$ sufficiently large, we also obtain

\begin{aligned} u_{i}(x_{n})\geq{}&C_{0}{ \int_{\frac{R}{2}^{R}}}y_{n}^{\gamma}u^{\alpha _{i}}_{1}(y_{n})u^{\beta_{i}}_{2}(y_{n})y_{n}^{\frac{ \alpha}{2}} \frac{1}{\vert x_{n}-y_{n}\vert }\,dy_{n} x_{n}^{\frac{ \alpha}{2}} \\ \geq{}&C_{0}{ \int_{\frac{R}{2}^{R}}}y_{n}^{\gamma}C_{1}^{\alpha_{i}+\beta _{i}}y_{n}^{(\alpha_{i}+\beta_{i})(\frac{\alpha}{2}-1)}y_{n}^{\frac{ \alpha}{2}} \frac{1}{\vert x_{n}-y_{n}\vert }\,dy_{n} x_{n}^{\frac{ \alpha}{2}} \\ \geq{}&C_{0} C_{1}^{\alpha_{i}+\beta_{i}}R^{(\alpha_{i}+\beta_{i})(\frac{\alpha }{2}-1)+\gamma}(2R)^{\frac{ \alpha}{2}}\frac{2}{3R} { \int_{\frac{R}{2}^{R}}}y_{n}^{\frac{ \alpha}{2}}\,dy_{n} \\ :={}&AR^{(\alpha_{i}+\beta_{i})(\frac{\alpha}{2}-1)+\gamma+\alpha} :=A_{1}x_{n}^{(\alpha_{i}+\beta_{i})(\frac{\alpha}{2}-1)+\gamma+\alpha}. \end{aligned}

Continuing this way m times, for $$x_{n} = 2R$$, we have

$$u_{i}(x_{n})\geq A(m,\alpha_{i}+ \beta_{i},\alpha,\gamma)x_{n}^{(\alpha _{i}+\beta_{i})^{m}(\frac{\alpha}{2}-1)+\frac{(\alpha_{i}+\beta _{i})^{m}-1}{\alpha_{i}+\beta_{i}-1}(\gamma+\alpha)}.$$
(2.37)

For any fixed α and γ in their respective domain, we choose m to be an integer greater than $$\frac{-\alpha^{2}-\alpha \gamma+\gamma+3}{\alpha+\gamma}$$ and 1. That is,

$$m\geq \operatorname{max} \biggl\{ \biggl\lceil \frac{-\alpha^{2}- \alpha\gamma+\gamma+3}{\alpha +\gamma} \biggr\rfloor +1,1 \biggr\} ,$$
(2.38)

where $$\lceil a\rfloor$$ is the integer part of a.

We claim that, for such a choice of m, we have

$$\tau(\alpha_{i}+\beta_{i}):= \biggl[({ \alpha_{i}+\beta_{i}})^{m} \biggl( \frac{\alpha }{2}-1 \biggr)+\frac{({\alpha_{i}+\beta_{i}})^{m}-1}{{\alpha_{i}+\beta _{i}}-1}(\alpha+\gamma) \biggr]({ \alpha_{i}+\beta_{i}})+\frac{\alpha }{2}+\gamma\geq0.$$
(2.39)

We postpone the proof of (2.39) for a moment. Now by (2.37) and (2.39), we derive that

$$u^{\alpha_{i}+\beta_{i}}_{i}(x_{n})x^{\frac{\alpha}{2}+\gamma}_{n} \geq A(m,{\alpha_{i}+\beta_{i}},\alpha, \gamma)x^{\tau({\alpha_{i}+\beta _{i}})}_{n} \geq A(m,{\alpha_{i}+ \beta_{i}},\alpha,\gamma)>0,$$

for all $$x_{n}$$ sufficiently large. This contradicts (2.34). So there is no positive solution of (2.32). This implies that $$u(x)$$ must be constant. By our positive assumption on u, we have $$u_{i}(x) = b_{i} > 0$$, $$i = 1, 2$$. Taking $$u_{i}$$ into (2.2), we have $$0=(-\Delta)^{\frac{\alpha }{2}}u_{i}(x)=x^{\gamma}_{n}u_{1}^{\alpha_{i}}(x)u_{2}^{\beta_{i}}(x)>0$$. This is impossible. Hence, in (2.10), $$c_{i}$$ must be zero, $$i = 1, 2$$.

Now it is left to verify (2.39). In fact, if we let

\begin{aligned} f({\alpha_{i}+\beta_{i}}):={}&\tau({ \alpha_{i}+ \beta_{i}}) ({\alpha_{i}+\beta _{i}}-1) \\ ={}& ({\alpha_{i}+\beta_{i}})^{m+2} \biggl( \frac{\alpha}{2}-1 \biggr)+({\alpha_{i}+\beta _{i}})^{m+1} \biggl(\frac{\alpha}{2}+\gamma+1 \biggr) -\frac{\alpha}{2}({ \alpha_{i}+\beta_{i}})-\frac{\alpha}{2}-\gamma, \end{aligned}

then

$$f'({\alpha_{i}+\beta_{i}})=({ \alpha_{i}+\beta_{i}})^{m} \biggl[(m+2) \biggl( \frac{\alpha }{2}-1 \biggr) ({\alpha_{i}+\beta_{i}})+ ({m+1}) \biggl(\frac{\alpha}{2}+\gamma+1 \biggr) \biggr]-\frac{\alpha}{2}.$$

We show that $$f'({\alpha_{i}+\beta_{i}})>0$$, for $$1<{\alpha_{i}+\beta_{i}}\leq \frac{n+\alpha+2\gamma}{n-\alpha}$$. Since $${\alpha_{i}+\beta_{i}}>1$$, it suffices to show $$(m+2)(\frac{\alpha}{2}-1)({\alpha_{i}+\beta_{i}})+(m+1)(\frac{\alpha }{2}+\gamma+1) \geq\frac{\alpha}{2}$$. Due to the fact $$\frac{\alpha}{2}-1 < 0$$, $$n \geq3$$, and $${\alpha _{i}+\beta_{i}} \leq\frac{n+\alpha+2\gamma}{n-\alpha}$$, we only need to verify that

$$(m+2) \biggl(\frac{\alpha}{2}-1 \biggr)\frac{3+\alpha+2\gamma}{3-\alpha}+(m+1) \biggl( \frac{\alpha}{2}+\gamma+1 \biggr) \geq\frac{\alpha}{2},$$

which can be derived directly from (2.38).

On the other hand, assume that $$u(x)$$ is a solution of the integral equation (1.6). Then, for any $$\phi\in C^{\infty }_{0}({R^{n}_{+}})$$, we have

\begin{aligned} \bigl\langle (-\Delta)^{\frac{\alpha}{2}}u_{i}, \phi\bigr\rangle ={}& \biggl\langle \int_{R^{n}_{+}}G_{\infty}(x,y)y^{\gamma}_{n}u_{1}^{\alpha _{i}}(y)u_{2}^{\beta_{i}}(y) \,dy, (-\Delta)^{\frac{\alpha}{2}}\phi (x)\biggr\rangle \\ ={}& \int_{R^{n}_{+}} \biggl\{ \int_{R^{n}_{+}}G_{\infty}(x,y)y^{\gamma }_{n}u_{1}^{\alpha_{i}}(y)u_{2}^{\beta_{i}}(y) \,dy \biggr\} (-\Delta)^{\frac {\alpha}{2}}\phi(x)\,dx \\ ={}& \int_{R^{n}_{+}} \biggl\{ \int_{R^{n}_{+}}G_{\infty}(x,y) (-\Delta)^{\frac {\alpha}{2}} \phi(x)\,dx \biggr\} y^{\gamma}_{n}u_{1}^{\alpha _{i}}(y)u_{2}^{\beta_{i}}(y) \,dy \\ ={}& \int_{R^{n}_{+}} \biggl\{ \int_{R^{n}_{+}}\delta(x-y)\phi(x)\,dx \biggr\} y^{\gamma}_{n}u_{1}^{\alpha_{i}}(y)u_{2}^{\beta_{i}}(y) \,dy \\ ={}& \int_{R^{n}_{+}}y^{\gamma}_{n}u_{1}^{\alpha_{i}}(y)u_{2}^{\beta _{i}}(y) \phi(y)\,dy =\bigl\langle y^{\gamma}_{n}u_{1}^{\alpha_{i}}u_{2}^{\beta_{i}}, \phi\bigr\rangle . \end{aligned}

Hence u also satisfies equations (1.3).

This completes the proof of Theorem 1.1.  □

## Liouville theorems

### The proof of Theorem 1.2

In this section, we will establish the nonexistence of the solutions to (1.6) by using the method of moving planes base up in the positive $$x_{n}$$ direction.

For a given positive real number λ, define $$\hat{\Sigma}_{\lambda}=\{x=(x_{1},\ldots,x_{n-1},x_{n})\in R^{n}_{+}\mid0< x_{n}< \lambda\}$$, $$\hat{T}_{\lambda}=\{x=(x_{1},\ldots,x_{n-1},x_{n})\in R^{n}_{+}\mid x_{n}= \lambda\}$$.

Let $$x^{\lambda}=(x_{1},x_{2},\ldots,x_{n-1},2\lambda-x_{n})$$ be the reflection of the point $$x=(x_{1},\ldots,x_{n})$$ about the plane $$\hat{T}_{\lambda}$$, set $$\Sigma^{c}_{\lambda }=R^{n}_{+}\backslash\hat{\Sigma}_{\lambda}$$ the complement of $$\hat{\Sigma}_{\lambda}$$, and write $$u_{i}^{\lambda }(x)=u_{i}(x^{\lambda})$$ and $$w_{i}^{\lambda}(x)=u_{i}^{\lambda}(x)-u_{i}(x)$$.

The following two lemmas are the key ingredient in our integral estimate.

### Lemma 3.1

[5]

(i) For any $$x, y \in\hat{\Sigma}_{\lambda}$$, $$x\neq y$$, we have

\begin{aligned} &G \bigl(x^{\lambda},y^{\lambda}\bigr)>\operatorname{max} \bigl\{ G \bigl(x^{\lambda},y \bigr),G \bigl(x,y^{\lambda}\bigr) \bigr\} , \\ & G \bigl(x^{\lambda},y^{\lambda}\bigr)-G(x,y)>\bigl\vert G \bigl(x^{\lambda},y \bigr)-G \bigl(x,y^{\lambda}\bigr)\bigr\vert . \end{aligned}

(ii) For any $$x \in\hat{\Sigma}_{\lambda}$$, $$y \in\Sigma_{\lambda}^{C}$$, we have $$G(x^{\lambda},y)>G(x,y)$$.

### Lemma 3.2

For any $$x \in\hat{\Sigma}_{\lambda}$$, $$u_{i}$$ are the positive solution of (1.6), we have

$$u_{i}(x)- u_{i}^{\lambda}(x)\leq \int_{\hat{\Sigma}_{\lambda }} \bigl[G \bigl(x^{\lambda},y^{\lambda}\bigr)-G \bigl(x,y^{\lambda}\bigr) \bigr]y_{n}^{\gamma}\bigl[u^{\alpha_{i}}_{1}(y) u^{\beta_{i}}_{2}(y)- u^{\alpha _{i}}_{1} \bigl(y^{\lambda} \bigr) u^{\beta_{i}}_{2} \bigl(y^{\lambda} \bigr) \bigr] \,dy.$$

### Proof

Let $$\tilde{\Sigma}_{\lambda}$$ be the reflection of $$\hat{\Sigma}_{\lambda}$$ about the plane $$\hat{T}_{\lambda}$$. we have

\begin{aligned} &u_{i}(x)= \int_{R^{n}_{+}}G(x,y) y_{n}^{\gamma} u^{\alpha_{i}}_{1}(y) u^{\beta_{i}}_{2}(y)\,dy \\ &\phantom{u_{i}(x)}= \int _{\hat{\Sigma}_{\lambda}}G(x,y) y_{n}^{\gamma} u^{\alpha_{i}}_{1}(y) u^{\beta_{i}}_{2}(y)\,dy \\ &\phantom{\phantom{u_{i}(x)}={}}+ \int_{\hat{\Sigma}_{\lambda}}G \bigl(x,y^{\lambda} \bigr) \bigl(y_{n}^{\lambda } \bigr)^{\gamma} u^{\alpha_{i}}_{1} \bigl(y^{\lambda} \bigr) u^{\beta _{i}}_{2} \bigl(y^{\lambda} \bigr)\,dy+{ \int_{\Sigma^{c}_{\lambda}\backslash\tilde {\Sigma_{\lambda}}}}G(x,y)y_{n}^{\gamma} u_{\lambda}^{\alpha _{i}}(y) u_{\lambda}^{\beta_{i}}(y)\,dy , \\ &u_{i} \bigl(x^{\lambda} \bigr) = \int_{\hat{\Sigma}_{\lambda}}G \bigl(x^{\lambda},y \bigr) y_{n}^{\gamma} u^{\alpha_{i}}_{1}(y) u^{\beta_{i}}_{2}(y)\,dy \\ &\phantom{u_{i} \bigl(x^{\lambda} \bigr) ={}}+ \int_{\hat{\Sigma}_{\lambda}}G \bigl(x^{\lambda},y^{\lambda } \bigr) \bigl(y_{n}^{\lambda} \bigr)^{\gamma} u^{\alpha_{i}}_{1} \bigl(y^{\lambda} \bigr) u^{\beta_{i}}_{2} \bigl(y^{\lambda} \bigr)\,dy + \int_{\Sigma^{c}_{\lambda}\backslash\tilde{\Sigma}_{\lambda }}G \bigl(x^{\lambda},y \bigr)y_{n}^{\gamma} u^{\alpha_{i}}_{1}(y) u^{\beta _{i}}_{2}(y)\,dy . \end{aligned}

By Lemma 3.1, we arrive at

\begin{aligned} u_{i}(x)- u_{i} \bigl(x^{\lambda} \bigr) \leq& \int_{\hat{\Sigma}_{\lambda}} \bigl[G(x,y)-G \bigl(x^{\lambda},y \bigr) \bigr] y_{n}^{\gamma} u^{\alpha_{i}}_{1}(y) u^{\beta_{i}}_{2}(y)\,dy \\ &{}- \int_{\hat{\Sigma}_{\lambda}} \bigl[G \bigl(x^{\lambda},y^{\lambda } \bigr)-G \bigl(x,y^{\lambda} \bigr) \bigr] \bigl(y_{n}^{\lambda} \bigr)^{\gamma} u^{\alpha _{i}}_{1} \bigl(y^{\lambda} \bigr) u^{\beta_{i}}_{2} \bigl(y^{\lambda} \bigr)\,dy \\ =& \int_{\hat{\Sigma}_{\lambda}} \bigl[G \bigl(x^{\lambda},y^{\lambda } \bigr)-G \bigl(x,y^{\lambda} \bigr) \bigr] \bigl[y_{n}^{\gamma} u^{\alpha_{i}}_{1}(y) u^{\beta _{i}}_{2}(y)- \bigl(y_{n}^{\lambda} \bigr)^{\gamma} u^{\alpha _{i}}_{1} \bigl(y^{\lambda} \bigr) u^{\beta_{i}}_{2} \bigl(y^{\lambda} \bigr) \bigr]\,dy \\ \leq& \int_{\hat{\Sigma}_{\lambda}} \bigl[G \bigl(x^{\lambda},y^{\lambda } \bigr)-G \bigl(x,y^{\lambda} \bigr) \bigr] \bigl[y_{n}^{\gamma} u^{\alpha_{i}}_{1}(y) u^{\beta _{i}}_{2}(y)- \bigl(y_{n}^{\lambda} \bigr)^{\gamma} u^{\alpha _{i}}_{1} \bigl(y^{\lambda} \bigr) u^{\beta_{i}}_{2} \bigl(y^{\lambda} \bigr) \bigr]\,dy \\ \leq& \int_{\hat{\Sigma}_{\lambda}} \bigl[G \bigl(x^{\lambda},y^{\lambda } \bigr)-G \bigl(x,y^{\lambda} \bigr) \bigr] y_{n}^{\gamma} \bigl[ u^{\alpha_{i}}_{1}(y) u^{\beta_{i}}_{2}(y)- u^{\alpha_{i}}_{1} \bigl(y^{\lambda} \bigr) u^{\beta_{i}}_{2} \bigl(y^{\lambda} \bigr) \bigr]\,dy . \end{aligned}

This completes the proof of Lemma 3.2. □

In order to prove the Liouville theorem, Theorem 1.2, we carry out the method of moving planes in integral form in the positive $$x_{n}$$ direction.

The proof consists of two steps. In the first step, we start from the very low end of our region $$R^{n}_{+}$$, i.e. near $$x_{n} = 0$$ and show that, for λ sufficiently small,

$$w_{i}^{\lambda}(x)=u_{i}^{\lambda}(x)- u_{i}(x)\geq0,\quad \mbox{a.e. } \forall x\in\hat{\Sigma}_{\lambda}.$$
(3.1)

In the second step, we will move our plane $$\hat{T}_{\lambda}$$ up in the positive $$x_{n}$$ direction as long as the inequality (3.1) holds and show that $$u(x)$$ is monotone increasing in $$x_{n}$$ and thus derive a contradiction.

Step 1. Define $$\Gamma^{\lambda}_{i}=\{x\in\hat{\Sigma}_{\lambda} \mid u^{\lambda }_{i}(x)< u_{i}(x) \}$$, $$i=1,2$$ and $$\Sigma^{\lambda}_{i}=\{x\in\hat{\Sigma}_{\lambda} \mid u_{1}^{\alpha_{i}}(x^{\lambda}) u_{2}^{\beta_{i}}(x^{\lambda})< u_{1}^{\alpha_{i}}(x) u_{2}^{\beta_{i}}(x) \}$$. We show that, for λ sufficiently small, $$\Gamma^{\lambda }_{i}$$ must be measure zero. In fact, for any $$x \in\Gamma^{\lambda}_{i}$$, by the mean value theorem similar to (2.16) and Lemma 3.2, we obtain

\begin{aligned} u_{i}(x)-u_{i}^{\lambda}(x) \leq{}& \int_{\Sigma^{\lambda }_{i}}G \bigl(x^{\lambda},y^{\lambda} \bigr) y_{n}^{\gamma} \bigl[ u^{\alpha_{i}}_{1}(y) u^{\beta_{i}}_{2}(y)- u^{\alpha_{i}}_{1} \bigl(y^{\lambda} \bigr) u^{\beta_{i}}_{2} \bigl(y^{\lambda} \bigr) \bigr]\,dy \\ = {}& \int_{\Sigma^{\lambda}_{i}}G \bigl(x^{\lambda},y^{\lambda} \bigr) y_{n}^{\gamma} \bigl[ \bigl(u^{\alpha_{i}}_{1}(y)- u^{\alpha _{i}}_{1} \bigl(y^{\lambda} \bigr) \bigr) u^{\beta_{i}}_{2}(y)+ u^{\alpha_{i}}_{1}(y) \bigl( u^{\beta_{i}}_{2}(y)-u^{\beta _{i}}_{2} \bigl(y^{\lambda} \bigr) \bigr) \bigr]\,dy \\ \leq{}& \int_{\Sigma^{\lambda}_{i}}G \bigl(x^{\lambda},y^{\lambda} \bigr) y_{n}^{\gamma} \bigl[ u^{\alpha_{i}}_{1}(y)- u^{\alpha_{i}}_{1} \bigl(y^{\lambda } \bigr) \bigr] u^{\beta_{i}}_{2}(y)\,dy \\ \leq{}& c \int_{\Sigma^{\lambda}_{i}}G \bigl(x^{\lambda},y^{\lambda} \bigr) y_{n}^{\gamma} u^{\alpha_{i}-1}_{1}(y) \bigl[ u_{1}(y)- u_{1} \bigl(y^{\lambda } \bigr) \bigr] u^{\beta_{i}}_{2}(y)\,dy. \end{aligned}
(3.2)

By the expression of $$G(x, y)$$, it is easy to see $$G(x, y)\leq\frac{A_{n,\alpha}}{ \vert x-y\vert ^{n-\alpha}}$$. From (3.2), we have

\begin{aligned} {1} u_{i}(x)-u_{i}^{\lambda}(x)\leq{}&c \int_{\Sigma^{\lambda}_{i}}\frac {1}{\vert x-y\vert ^{n-\alpha}}\bigl\vert y^{\gamma}_{n} u^{\alpha_{i}+\beta _{i}-1}(y)\bigr|\bigl| w^{\lambda}(y)\bigr\vert \end{aligned}
(3.3)
\begin{aligned} \leq{}&c \int_{\Sigma^{\lambda}_{i}}\frac{1}{\vert x-y\vert ^{n-\alpha}}\bigl\vert u^{\alpha_{i}+\beta_{i}-1}(y) \bigr|\bigl| w^{\lambda}(y)\bigr\vert . \end{aligned}
(3.4)

Notice that now γ is only a little larger than 0, so $$y_{n}$$ is bounded within $$\Sigma^{\lambda}_{i}$$, and since $$\gamma\geq0$$, we get $$\vert y^{\gamma}_{n}\vert \leq C$$, hence we derive (3.4) from (3.3).

Noticing $$\Sigma_{i}^{\lambda} \subseteq\Gamma^{\lambda}_{j}$$ for some j, applying Hardy-Littlewood-Sobolev inequality and the Hölder inequality we obtain, for any $$q>\frac{n}{n-\alpha}$$,

$$\Vert w_{\lambda} \Vert _{L^{q}({\Gamma_{\lambda}})}\leq c\bigl\Vert \vert u\vert ^{\alpha _{i}+\beta_{i}-1}w_{\lambda} \bigr\Vert _{L^{\frac{nq}{n+\alpha q}}({\Gamma _{\lambda}}) } \leq c\bigl\Vert \vert u\vert ^{\alpha_{i}+\beta_{i}-1}\bigr\Vert _{L^{\frac{n}{\alpha }}({\Gamma_{\lambda}}) }\Vert w_{\lambda} \Vert _{L^{q}({\Gamma_{\lambda }})},$$
(3.5)

where $$\Gamma_{\lambda} =\Gamma^{\lambda}_{1}\cup\Gamma ^{\lambda}_{2}$$.

Since $$\vert u\vert \in L^{\frac{n(\alpha_{i}+\beta_{i}-1)}{\alpha}}(R^{n}_{+})$$, we can choose sufficiently small positive λ such that

$$c\bigl\Vert u^{\alpha_{i}+\beta_{i}-1} \bigr\Vert _{L^{\frac{n}{\alpha}}(\Gamma _{\lambda})}= c \biggl\{ { \int_{{\Gamma_{\lambda}}}} \vert u\vert ^{\frac{n(\alpha _{i}+\beta_{i}-1)}{\alpha}} (y) \,dy \biggr\} ^{\frac{\alpha}{n}} \leq\frac{1}{2}.$$
(3.6)

By (3.5) and (3.6), we derive $$\Vert w_{\lambda} \Vert _{L^{q}({\Gamma_{\lambda}})}=0$$, and $$\Gamma^{\lambda}_{i}$$ must be of measure zero, hence (3.1) holds. This provides us a starting point for moving the plane.

Step 2. Now we start from such small λ and move the plane $$\hat{T}_{\lambda}$$ up as long as (3.1) holds. Define

$$\lambda_{0}=\operatorname{sup} \bigl\{ \lambda\mid w_{\rho}(x)\geq0, \rho\leq \lambda, \forall x\in\hat{\Sigma}_{\rho}\bigr\} .$$

We will prove

$$\lambda_{0}=+\infty.$$
(3.7)

Suppose to the contrary that $$\lambda_{0}<+\infty$$, we will show that $$u_{i}(x)$$ is symmetric about the plane $$\hat{T}_{\lambda_{0}}$$, i.e.

$$u^{\lambda_{0} }_{i}(x)\equiv u_{i}(x),\quad\mbox{a.e. } \forall x\in\hat{\Sigma}_{\lambda_{0}}.$$
(3.8)

This will contradict the strict positivity of $$u_{i}(x)$$. Suppose (3.8) does not hold. Then, for such a $$\lambda_{0}$$, we have $$u^{\lambda_{0} }_{i}(x)\geq u_{i}(x)$$, but $$u^{\lambda_{0} }_{i}(x)\not \equiv u_{i}(x)$$ a.e. on $$\hat{\Sigma}_{\lambda_{0}}$$. We show that the plane can be moved further up. More precisely, there exists an $$\epsilon> 0$$ such that for all $$\lambda\in[\lambda_{0}, \lambda _{0}+\epsilon)$$

$$u^{\lambda}_{i}(x)\geq u_{i}(x),\quad \mbox{a.e. on } \hat{\Sigma}_{\lambda}.$$
(3.9)

To verify this, we will again resort to inequality (3.5). If one can show that, for ϵ sufficiently small so that, for all λ in $$[\lambda_{0}, \lambda_{0}+\epsilon)$$, we have

$$c \biggl\{ { \int_{{\Gamma_{\lambda}}}} \vert u\vert ^{\frac{n(\alpha_{i}+\beta _{i}-1)}{\alpha}}(y) \,dy \biggr\} ^{\frac{\alpha}{n}} \leq\frac{1}{2},$$
(3.10)

then by (3.3) and (3.10), we have $$\Vert w_{\lambda_{0} }\Vert _{L^{q}(\Gamma_{\lambda})}=0$$, and therefore $$\Gamma^{\lambda}$$ must be of measure zero. Hence, for these values of $$\lambda> \lambda_{0}$$, we have (3.9). This contradicts the definition of $$\lambda_{0}$$. Therefore (3.8) must hold.

The proof of the inequality (3.10) is similar to the argument of the inequality (2.26) in Section 2 and the proof is standard.

By (3.8), we derive that $$u_{i}(x) = 0$$ on the plane $$x_{n} = 2\lambda_{0}$$, the symmetric image of the boundary $$\partial R^{n}_{+}$$ with respect to the plane $$\hat{T}_{\lambda_{0}}$$. This contradicts our assumption $$u_{i}(x) > 0$$ in $$R^{n}_{+}$$. Therefore, (3.7) must be valid. Now we have proved that the positive solution of (1.6) is monotone increasing with respect to $$x_{n}$$, and this contradicts $$u^{\alpha_{i}+\beta _{i}-1}\in L^{\frac{n}{\alpha}}(R^{n}_{+})$$. Therefore the positive solutions of (1.6) do not exist.

This completes the proof of Theorem 1.2.

### The proof of Theorem 1.3

In this section, we will use a proper Kelvin type transforms and derive the nonexistence of positive solutions for (1.6) in $$R^{n}_{+}$$ under much weaker conditions, i.e. the solution u of (1.6) is only locally integrable and locally bounded.

With no explicit global integrability assumptions on the solution u, we cannot directly carry out the method of moving planes on u. To overcome this difficulty, we employ Kelvin type transforms.

For $$z^{0} \in R^{n}_{+}$$, let

$$\bar{u}_{i}(x)=\frac{1}{\vert x-z^{0}\vert ^{n-\alpha}}u_{i} \biggl(\frac {x-z^{0}}{\vert x-z^{0}\vert ^{2}}+z^{0} \biggr)$$
(3.11)

be the Kelvin transform of $$u_{i}(x)$$ centered at $$z^{0}$$.

Through a straightforward calculation, we have $$\bar{u}_{i}(x) =\int_{R^{n}_{+}}G(x,y)\frac{y_{n}^{\gamma}\bar{u}^{\alpha_{i}}_{1}(y)\bar{u}^{\beta_{i}}_{2}(y)}{\vert y-z^{0}\vert ^{\delta }}\,dy$$, $$\forall x\in{R^{n}_{+}}\backslash{B_{\epsilon}(z^{0})}$$, where $$\delta=n+\alpha+2\gamma-(n-\alpha)(\alpha_{i}+\beta_{i})$$, $$\epsilon>0$$, $$i=1,2$$.

### Proof of Theorem 1.3 in the subcritical case

$$1<\alpha_{i}+\beta_{i}<\frac{n+\alpha+2\gamma}{n-\alpha}$$:

$$\bar{u}_{i}(x) = \int _{R^{n}_{+}}G(x,y)\frac{y_{n}^{\gamma}\bar{u}^{\alpha_{i}}_{1}(y)\bar{u}^{\beta_{i}}_{2}(y)}{\vert y-z^{0}\vert ^{\delta}}\,dy,\quad \forall x\in {R^{n}_{+}}\backslash{B_{\epsilon}\bigl(z^{0} \bigr)},$$
(3.12)

where $$\delta=n+\alpha+2\gamma-(n-\alpha)(\alpha_{i}+\beta _{i})>0$$, $$\epsilon>0$$.

This specific proof is the same as the proof of Theorem 1.1 and we omit here. □

### Remark

When we carry out the method of moving planes on equation (1.4), we derive the fact $$c_{i}=0$$ and consequently obtain the equivalence. While applying the same method on equation (1.6), surprisingly, we arrive at a Liouville type theorem for it.

### Proof of Theorem 1.3 in the critical case

$$1<\alpha_{i}+\beta_{i}=\frac{n+\alpha+2\gamma}{n-\alpha}$$:

$$u_{i}(x)= \int_{R^{n}_{+}}G(x,y) y_{n}^{\gamma} u^{\alpha_{i}}_{1}(y) u^{\beta_{i}}_{2}(y)\,dy.$$
(3.13)

By the Kelvin transform of $$u_{i}(x)$$ we derive

$$\bar{u}_{i}(x)= \int_{R^{n}_{+}}G(x,y) y_{n}^{\gamma}\bar{u}^{\alpha_{i}}_{1}(y)\bar{u}^{\beta_{i}}_{2}(y)\,dy.$$
(3.14)

If $$u(x)$$ is a solution of (3.13), then $$\bar{u} (x)$$ is also a solution of (3.14). Therefore, by our assumption $$\vert u\vert \in L^{\frac{n(\alpha_{i}+\beta_{i}-1)}{\alpha}}_{\mathrm{loc}}(R^{n}_{+})$$, we derive $$y^{\gamma}_{n} \vert u\vert ^{\alpha_{i}+\beta_{i}-1}\in L^{\frac{n}{\alpha }}_{\mathrm{loc}}(R^{n}_{+})$$. then

$$\int_{\hat{\Omega}} \bigl[y^{\gamma}_{n} \bar{u}^{\alpha_{i}+\beta_{i}-1}(y) \bigr]^{\frac{n}{\alpha}}\,dy= \int_{ \Omega} \bigl[y^{\gamma}_{n} u^{\alpha _{i}+\beta_{i}-1}(y) \bigr]^{\frac{n}{\alpha}}\,dy< \infty,$$
(3.15)

where Ω̂ is the image of Ω about the Kelvin transform. Now we consider two possibilities.

Possibility 1. If there is a $$z^{0}=(z^{0}_{1}, \ldots ,z^{0}_{n-1}, 0 )\in\partial R^{n}_{+}$$ such that $$\bar{u}_{i}(x)$$ is bounded near $$z^{0}$$, then by (3.11), we obtain

$$u_{i}(y)=\frac{1}{\vert y-z^{0}\vert ^{n-\alpha}}\bar{u}_{i} \biggl( \frac {y-z^{0}}{\vert y-z^{0}\vert ^{2}}+z^{0} \biggr) .$$

And we further deduce

$$u_{i}(y)=O \biggl(\frac{1}{\vert y\vert ^{n-\alpha}} \biggr), \quad \mbox{as } \vert y\vert \rightarrow\infty.$$
(3.16)

Since $$\alpha_{i}+\beta_{i}=\frac{n+\alpha+2\gamma}{n-\alpha }>\frac{n}{n-\alpha}$$ and $$\vert u\vert \in L^{\frac{n(\alpha_{i}+\beta _{i}-1)}{\alpha}}_{\mathrm{loc}}(R^{n}_{+})$$, together with (3.16), we have

$$\int_{ R^{n}_{+}} u^{\frac{n(\alpha_{i}+\beta_{i}-1)}{\alpha}}(y) \,dy\leq c \int_{ R^{n}_{+}}\frac{1}{\vert y\vert ^{\frac{n(\alpha_{i}+\beta _{i}-1)(n-\alpha)}{\alpha}}}\,dy< \infty.$$
(3.17)

In this situation, we still carry on the moving planes on u. Going through exactly the same arguments as in the proof of Theorem 1.2, we obtain the nonexistence of positive solutions for (3.13).

Possibility 2. For all $$z^{0}=(z^{0}_{1}, \ldots ,z^{0}_{n-1}, 0 )\in\partial R^{n}_{+}$$, $$\bar{u}_{i}(x)$$ is unbounded near $$z^{0}$$, we will carry out the method of moving planes on $$\bar{u}(x)$$ in $$R^{n-1}$$ to prove that it is rotationally symmetric about the line passing through $$z^{0}$$ and parallel to the $$x_{n}$$-axis. From this, we will deduce that u is independent of the first $$n-1$$ variables $$x_{1}, \ldots,x_{n-1}$$. That is $$u = u(x_{n})$$, and we will derive a contradiction with the finiteness of $$\int_{R^{n}_{+}}G(x,y)y^{\gamma}_{n} u^{\alpha_{i}}_{1}(y)u^{\beta _{i}}_{2}(y)\,dy$$.

For a given real number λ, the notations such as $$\Sigma_{\lambda}$$, $$T_{\lambda}$$ are the same as the ones in Section 2. By (2.12), obviously we have

\begin{aligned} &\bar{u}_{i}(x) = \int_{\Sigma_{\lambda}}G(x,y) y_{n}^{\gamma}\bar{u}^{\alpha_{i}}_{1}(y)\bar{u}^{\beta_{i}}_{2}(y)\,dy + \int_{\Sigma_{\lambda}}G \bigl(x^{\lambda},y \bigr)y_{n}^{\gamma} \bar{u}^{\alpha_{i}}_{1} \bigl(y^{\lambda} \bigr) \bar{u}^{\beta_{i}}_{2} \bigl(y^{\lambda} \bigr)\,dy, \\ &\bar{u}^{\lambda}_{i}(x) = \int_{\Sigma_{\lambda}}G \bigl(x^{\lambda},y \bigr) y_{n}^{\gamma} \bar{u}^{\alpha_{i}}_{1}(y)\bar{u}^{\beta_{i}}_{2}(y)\,dy + \int_{\Sigma_{\lambda}}G(x,y)y_{n}^{\gamma}\bar{u}^{\alpha _{i}}_{1} \bigl(y^{\lambda} \bigr)\bar{u}^{\beta_{i}}_{2} \bigl(y^{\lambda} \bigr)\,dy . \end{aligned}

By elementary calculation we derive

\begin{aligned} \bar{u}_{i}(x)- \bar{u}^{\lambda}_{i}(x) = \int_{\Sigma_{\lambda}} \bigl[G(x,y)-G \bigl(x^{\lambda},y \bigr) \bigr]y_{n}^{\gamma } \bigl[\bar{u}^{\alpha_{i}}_{1}(y) \bar{u}^{\beta_{i}}_{2}(y)-\bar{u}^{\alpha _{i}}_{1} \bigl(y^{\lambda} \bigr)\bar{u}^{\beta_{i}}_{2} \bigl(y^{\lambda} \bigr) \bigr]\,dy. \end{aligned}
(3.18)

We will move the plane $${T _{\lambda}}$$ along the direction of the $$x_{1}$$-axis to show that the solution is rotationally symmetric about the line passing through $$z^{0}$$ and parallel to the $$x_{n}$$-axis. The proof is the same as the proof of $$c_{i}=0$$ in Section 2, in fact we only need to apply arguments of the inequality (2.13) to equation (3.18). Similarly, we derive that $$u_{i}=u_{i}(x_{n})$$ and any positive solution u of (3.13) must be $$u(x)\equiv0$$. This implies that there is no positive solution of (3.13) in the critical case.

This completes the proof of Theorem 1.3. □

## References

1. Cao, L, Chen, W: Liouville type theorems for poly-harmonic Navier problems. Discrete Contin. Dyn. Syst. 33, 3937-3955 (2013)

2. Serra, E: Non radial positive solutions for the Hénon equation with critical growth. Calc. Var. 23, 301-326 (2005)

3. Chen, W, Li, C, Ou, B: Qualitative properties of solutions for an integral equation. Discrete Contin. Dyn. Syst. 12, 347-354 (2005)

4. Caffarelli, L, Vasseur, A: Drift diffusion equations with fractional diffusion and the quasi-geostrophic equation. Ann. Math. 171, 1903-1930 (2010)

5. Chen, W, Fang, Y, Yang, R: Liouville theorems involving the fractional Laplacian on a half space. Adv. Math. 274, 167-198 (2015)

6. Zhuo, R, Chen, W, Cui, X, Yuan, Z: Symmetry and non-existence of solutions for a nonlinear system involving the fractional Laplacian. Discrete Contin. Dyn. Syst. 36(2), 1125-1141 (2016)

7. Zhang, L, Cheng, T: Liouville theorems involving the fractional Laplacian on the upper half Euclidean space. Preprint

8. Silvestre, L: Regularity of the obstacle problem for a fractional power of the Laplace operator. Commun. Pure Appl. Math. 60, 67-112 (2007)

9. Zhang, L, Li, C, Chen, W, Cheng, T: A Liouville theorem for α-harmonic functions in $$R^{n}_{+}$$. Discrete Contin. Dyn. Syst. 36(3), 1721-1736 (2016)

10. Kulczycki, T: Properties of Green function of symmetry stable processes. Probab. Math. Stat. 17, 339-364 (1997)

## Acknowledgements

This work is partially supported by NSFC (No. U1304101 and No. 11671121) and HASTIT (No. 15HASTIT012).

## Author information

Authors

### Corresponding author

Correspondence to Linfen Cao.

### Competing interests

The authors declare that they have no competing interests.

### Authors’ contributions

ZD participated in the method of moving plane studies in the paper and drafted the manuscript; LC carried out the Liouville type theorem, and PW carried out the evaluation of inequalities. All authors read and approved the final manuscript.

## Rights and permissions

Reprints and Permissions

Dai, Z., Cao, L. & Wang, P. Liouville type theorems for the system of fractional nonlinear equations in $${R^{n}_{+}}$$ . J Inequal Appl 2016, 267 (2016). https://doi.org/10.1186/s13660-016-1207-9

• Accepted:

• Published:

• DOI: https://doi.org/10.1186/s13660-016-1207-9

### Keywords

• the fractional Laplacian
• Green’s function
• method of moving planes in integral forms
• Liouville theorem
• Kelvin transform