On a fractional p-q Laplacian equation with critical nonlinearity

In this paper, we consider the existence of nontrivial solutions for a fractional p-q Laplacian equation with critical nonlinearity in a bounded domain. Our approach is based on variational methods and some analytical techniques.


Introduction
In this paper, we are interested in the existence of nontrivial solutions for the following equation: (-) s p u + (-) s q u = λ|u| r-2 u + |u| p * s -2 u, x ∈ Ω, u = 0, x ∈ R N \ Ω, (1.1) where Ω is a bounded domain in R N , 0 < s < 1, 1 < q < p < r < p * s , λ is a positive constant, p * s = pN/(Nsp) is the fractional critical exponent, and (-) s p is the fractional p-Laplacian operator defined on smooth functions as (-) s p u(x) = P.V .
The definition is consistent, up to a normalization constant depending on N and s, with the usual definition of the linear fractional Laplacian operator (-) s when p = 2. When s = 1, Eq. (1.1) becomes a local problem of the form p uq u = λ|u| r-2 u + |u| p * -2 u, (1.2) which has been studied before, and some existence results have been proven under different conditions. For 1 < q < p < r < p * , there exists λ * > 0 such that for any λ > λ * , problem (1.2) has a nontrivial solution in W λ ∈ (0, λ 0 ) (see Li and Zhang [2]). Our result can be viewed as extension on [1] for fractional setting. As explained in [1], the study of Eq. (1.2) comes from a general reaction-diffusion system where H(u) = |∇u| p-2 + |∇u| q-2 . This system has a wide range of applications in physics and related sciences such as biophysics, plasma physics, and chemical reaction design. In applications the function u represents a concentration, div[H(u)∇u] corresponds to the diffusion with diffusion coefficient H(u), whereas c(x, u) is related to source and loss processes. Typically, in chemical and biological applications the reaction term c(x, u) has a polynomial form with respect to the concentration u. When p = q = r, problem (1.1) reduces to the fractional p-Laplacian problem which has been studied by Mosconi et al. [3], who obtained nontrivial solutions to this Brezis-Nirenberg problem for fractional p-Laplacian operator and extended some wellknown results of critical p-Laplacian problems to the fractional setting; see, for example, Azorero and Alonso [4] and Egnell [5]. In fact, there is a rapidly growing literature on problems involving these nonocal operators. For example, the fractional p-eigenvalue problem has been studied by Franzina and Palatucci [6] and Lindgren and Lindqvist [7]. Concerning the existence results for this kind of equations, some well-known existence results for classical Laplace operators have also been extended to the nonlocal fractional setting; see [8][9][10][11][12]. When p = q, there also are some recent results on the fractional p-Laplacian operator. In 2017, Mahwin and Bisci [13] proved a Brezis-Nirenberg-type result for the fractional p-Laplacian equation in a bounded domain with p ≥ 2, where g is a subcritical nonlinearity. By variational methods they prove the existence of a local minimizer of the associated functional to (1.4), which turns to be a weak solution of problem (1.4), provided that the constant λ is sufficiently small. It is worth mentioning that there is also some literature concerning the fractional Laplacian equation with constant γ attached to the critical term, where f satisfies some subcritical conditions; see Fiscella et al. [14]. By variational methods they obtain multiplicity and bifurcation results for (1.5), which generalized those given in [15] to the nonlocal framework of the fractional Laplacian. It is easy to see that the critical term and the subcritical term have different influences on the functional structure.
Motivated by the papers mentioned, we tend to investigate the existence of a nontrivial solution for problem (1.1). To our knowledge, not many critical results for fractional p-q Laplacian are present. We denote by X the fractional space W s,p By the results of [16] there is a continuous . For more details on fractional Sobolev spaces, we refer to Palatucci et al. [16] and references therein.
Our approach to studying problem (1.1) is variational and uses critical point theorems.
The main difficulty in dealing with this problem is the fact that in general the associated energy functional does not satisfy the Palais-Smale condition. Hence we cannot directly use the standard variational methods. To overcome this, we prove that the corresponding functional satisfies the Palais-Smale condition on a certain range. We also mention that there is a local weak lower semicontinuity result for the corresponding energy functional of problem (1.1), which leads to the existence of a critical point under certain conditions. At last, when p = 2, the spectrum result of the fractional operator ensures a suitable decomposition of the functional space, which leads to a multiplicity result. It is worth noting that our results can also be generalized by the abstract result proposed by Devillanova and Solimini [17,18]. Our main results read as follows.  The present paper is organized as follows. Section 2 is devoted to the functional structure and Palais-Smale condition of problem (1.1). In Sect. 3, we prove our results.

Preliminaries
In this section, we give some preliminary results about the functional structure of problem (1.1). The fact that u is a weak solution of the problem (1.1) is equivalent to being a critical point of the functional It is trivial that I λ (u) ∈ C 1 (X, R) and for any v ∈ X, the weak solution satisfies We denote by S the best fractional Sobolev constant: Now we define the PS sequence and condition in W s,p 0 (Ω).
We first show that I λ possesses the mountain pass geometry.
Proof (i) By the Hölder inequality and fractional Sobolev inequality we have Since 1 < q < p < r < p * , there exits two constants ρ, β > 0 such that I λ (u) > β for all u ∈ X with u p = ρ.
We denote by c λ the mountain pass level: Then we have the following result.
Then for any λ > 0, I λ satisfies the (PS) c conditions for all c ∈ (0, 1 Proof Let {u k } be a (PS) c sequence of I at the level c, that is, Thus {u k } is bounded in W s,p 0 (Ω). Taking if necessary a subsequence, we can assume that there exists u ∈ W s,p (2.6) Noting that the sequences Thus for any v ∈ W s,p 0 (Ω), we have that is, u is a critical point of I λ . Then we get The fractional form of the Brezis-Lieb lemma leads to (1) and where o(1) → 0 as k → ∞. From this and from (2.7) we have v k Without loss of generality, we assume that If a = 0, then we complete the proof. Otherwise, a ≥ S N/p . Combining this with (2.1) and 1 ≤ q < p < r < p * , as n → ∞, we have which contradicts the assumption on c. Thus we have a = 0, and I λ satisfies the (PS) c conditions when c ∈ (0, 1 N S N/p ). So we try to show that c λ ∈ (0, 1 N S N p ). We now choose a nonnegative u 0 ∈ W s,p 0 (Ω) with |u 0 | p * = 1. Since lim t→∞ I λ (tu 0 ) = -∞ and lim t→0 I λ (tu 0 ) = 0, there exists a t λ > 0 such that sup t≥0 I λ (tu 0 ) = I λ (t λ u 0 ), and thus t λ satisfies Then we get Since 1 ≤ q < p < r < p * , we get t λ → 0 as λ → ∞. Then there exists λ * > 0 such that for any λ > λ * > 0, we have that is, This completes the proof.
Next, we prove the local weak lower semicontinuity of I λ . From now on we denote the best constant of the continuous Sobolev embedding W Proof It suffices to prove that I 0 is weakly lower semicontinuous. Let {u j } be a weakly convergent sequence in ⊂ B(0, ρ), that is, there exists u ∈ B(0, ρ) satisfying u j → u a.e. in Ω. (2.13) We try to check that (2.14) Since 2 ≤ q < p, by the elementary inequality

Main theorems
To prove the first existence result, we need the following general version of the mountain pass lemma.
Lemma 3.1 Let I ∈ C 1 (X, R) be a functional on Banach space X. Assume that there exist β, ρ > 0 such that (i) I(u) > β for all u ∈ X with u p = ρ.
(ii) I(0) = 0, and I(v 0 ) < β for some v 0 ∈ X with v 0 p > ρ. Set α := inf{max t≥0 I(tu), u ∈ X \ 0}.Tthen there exists a sequence {u n } ⊂ X such that I(u n ) → α and I (u n ) → 0 in X * as n → ∞. Next, we define the auxiliary function where |Ω| denotes the Lebesgue measure of the domain Ω, S is the critical Sobolev constant given in (2.8), and C is the embedding constant satisfying u q q ≤ C u q p . By the weak lower semicontinuity result in Lemma 2.4 we can prove the existence of a critical point of the energy functional by a direct minimization approach.
At last, we give a multiplicity result for problem (1.1) when 1 ≤ q < p = 2, based on a suitable decomposition of the functional space H s 0 . We first recall that H s 0 is a Hilbert space with the inner product |x -y| n+2s dx dy (3.9) and the norm u = u, u . Denote by {λ j } j∈N the sequence of the eigenvalues of the eigenvalue problem with 0 < λ 1 < λ 2 ≤ · · · ≤ λ j ≤ λ j+1 ≤ · · · (3.11) and eigenfunctions e j corresponding to λ j . Also, we can normalize {e j } j∈N to construct an orthonormal basis of L 2 (Ω) and an orthogonal basis of H s 0 (Ω). For details on the spectrum theory of the fractional Laplacian, we refer to [6] and [7]. Then we set, for any j ∈ N , P j+1 = u ∈ H s 0 (Ω) : u, e i = 0 for i = 1, . . . , j ,