Existence of nontrivial solutions for a fractional p & q -Laplacian equation with sandwich-type and sign-changing nonlinearities

In this paper, we deal with the following fractional p & q -Laplacian problem:


Introduction
The main purpose of this paper is to study the existence of solutions for the following fractional p&q-Laplacian problem: where ⊂ R N is a bounded domain with smooth boundary, s ∈ (0, 1), p, q, r, θ ∈ (1, p * s ], p * s = Np N-sp denotes the fractional critical Sobolev exponent, λ, μ > 0, the weights a(x) and b(x) are possibly sign changing, and (-) s m (m ∈ {p, q}) is the fractional m-Laplacian operator, which (up to normalization factors) is defined for every function u ∈ C ∞ 0 (R N ) as For convenience, we refer the readers to [8,9,24] for an elementary introduction to fractional Sobolev spaces and corresponding nonlocal fractional operators.
In the local case, that is, where s = 1, (1.1) reduces to the classical p&q-Laplacian equation which has been extensively investigated by many scholars; see, for instance, [13-15, 17-19, 33] and the references therein.
Coming back to the fractional setting, we point out that in recent years, a great attention has been focused on the study of nonlocal problems driven by fractional operators, from a pure mathematical point of view and from concrete applications, since they naturally arise in many different contexts, such as the thin obstacle problem, phase transitions, optimization, anomalous diffusion, stratified materials, crystal dislocation, semipermeable membranes, soft thin films, flame propagation, conservation laws, ultra-relativistic limits of quantum mechanics, multiple scattering, quasi-geostrophic flows, minimal surfaces, material science, and water waves.For more detail, we refer to [8,24].Indeed, when p = q = 2, problem (1.1) has been studied by many researchers, and for some qualitative results on solutions such as existence, multiplicity, regularity and concentration, etc., we refer the interested readers to [1,3,6,14,28,29].
When p = q = 2, (1.1) boils down to the following fractional p-Laplacian problem: It is worth mentioning that the fractional p-Laplacian operator has a great attraction mainly because of its nonlinearity and nonlocal character.In fact, some standard tools often used to deal with the case p = 2 are not trivially adaptable in the case p = 2 due to the lack of Hilbertian structure of W s,p (R N ) for p = 2.For such a reason, there has been a large amount of attention to problems involving the fractional p-Laplacian operator.These problems include the Schrödinger-type equations, Kirchhof-type models, fractional systems, etc.; we refer to [12, 16, 20-22, 26, 27, 30-32] and the reference therein.On the other hand, only few researchers and few papers [2,4,5,7,10] deal with the fractional p&q-Laplacian problems.Inspired by the works mentioned, we study the existence of nontrivial solutions for problem (1.1) in both critical and subcritical cases.Moreover, we stress that the case q < θ < p, which will be called sandwich-type, has not been studied even under the subcritical case.In addition to q < θ < p, we carefully consider other relationships of the parameters p, q, r, and θ , and we prove that both these parameters and the signs of a(x) and b(x) play important roles on the existence of solutions.
For the critical case r = p * s , we make the following assumptions on weights a(x) and b(x): .
We get the following main result.
Theorem 1.3 Assume that (A 4 ) or (A 5 ) is satisfied.Then problem (1.1) admits at least one nontrivial solution.

Notations and preliminaries
In this section, we first provide the functional setting for problem (1.1).Let be an open set in R N , s ∈ (0, 1), and Consider the functional space X 0,s,p , the closure of C ∞ 0 ( ) in X s,p .By Theorems 6.5 and 7.1 in [24] the space X 0,s,p is a uniformly convex Banach space endowed with the norm which is equivalent to the usual one defined in (2.1).Note that since u = 0 a.e. in R N \ , integrals (2.1) and (2.2) can be extended to R 2N .The embedding X 0,s,p → L ι ( ) is continuous for ι ∈ [1, p * s ] and compact for ι ∈ [1, p * s ).Moreover, as proved in Lemma 2.2 of [34], we can see that for 1 < q < p, X 0,s,p ⊂ X 0,s,q , and this allows us to study our problem in X 0,s,p .
Using the Vitali convergence theorem, we can easily get the following compactness result.
, and let u n u in X 0,s,p .Then Now we provide the following Brźis-Lieb-type result which can be proved by adopting the same method as Lemma 2.1 in [34].

Proof of Theorem 1.1
In this section, we study the existence of nontrivial solutions for the following problem: The energy functional associated with problem (3.1) is defined as Clearly, I λ,μ (u) ∈ C 1 (X 0,s,p , R), and its critical points are in fact nontrivial weak solutions of problem (3.1).
Recall that we say a sequence {u n } ⊂ X 0,s,p is a (PS) sequence at level c ((PS) c sequence for short) if I λ,μ (u n ) → c and I λ,μ (u n ) → 0; I λ,μ is said to satisfy the (PS) c condition if any (PS) c sequence contains a convergent subsequence.
Proof Let {u n } ⊂ X 0,s,p be a (PS) c sequence for I λ,μ .First, we show that {u n } is bounded in X 0,s,p .Indeed, where S is the best constant of the embedding from X s,p 0 ( ) to L p * s ( ), that is, Now we show that J is empty, that is, J = ∅.Suppose by contradiction that there exists i ∈ J.For this x i , we set where φ ∈ C ∞ 0 (R N , [0, 1]); more precisely, φ is defined as Thus where (-) s m u n , φ i δ u n (m = p, q) is defined as as n → ∞.
Using the Hölder inequality and the fact that {u n } is bounded in X 0,s,p , we get for some C 0 > 0.
Next, we prove that It follows from the Lipschitz regularity of φ i δ that there exists M > 0 such that Then by the dominated convergence theorem we can easily deduce that Now following similar calculations as in (3.14), we have where U and V are two generic subsets of R N .
In the following, we claim that (3.17) ), then we can use Proposition 2.1 of [23] to prove the claim.

It follows by the Hölder inequality that
where Then from (3.6) and (3.20) we obtain that Letting δ → 0 in (3.23), we easily get Thus from (3.8) and (3.24) we obtain that where (1).
Then taking the limit as n → ∞ and using Lemma 2.2, we have Moreover, by the Hölder inequality we get Now we consider where By direct calculation we get that ξ (t) has minimum at Then from (3.29)-(3.30) it follows that which contradicts with (3.3).Thus J = ∅, which implies Note that for ϕ ∈ X 0,s,p , as n → ∞.
Taking ϕ = u n and using the dominated convergence theorem, we have On the other hand, for ϕ = u in (3,32), we get Then Thus from Lemma 2.3 we can easily deduce that u n → u in X 0,s,p , which completes the proof.
Now we are ready to prove Theorem 1.1.

Proof of Theorems 1.2 and 1.3
Here we consider the subcritical case r < p * s .For this case, the functional to problem (1.1) is To apply the Ekeland methods, we also need the following (PS) c conditions.
Lemma 4.1 J λ,μ (u) satisfies the (PS) c conditions for all c ∈ R.
Proof Let {u n } ⊂ X 0,s,p be a (PS) c -sequence of J λ,μ for c ∈ R. First, we prove {u n } is bounded.Note that q < θ < p < r.Then which implies that u n 0,s,p is bounded.Next, from J λ,μ (u n ), ϕ → 0 we get that Thus from (4.3)-(4.4) it follows that u n → u in X 0,s,p , and we complete the proof.
Proof of Theorem 1.2 For the subcritical case, noting that X 0,s,p → L r ( ) is compact for r < p * s , Theorem 1.2 can be similarly proved as (3.36)-(3.45) in Theorem 1.1.
Next, we give the proof of Theorem 1.3, which is based on the standard mountain pass lemma.According to Lemma 4.1, we need only to check that J λ,μ verifies the mountain pass geometry.Lemma 4.2 Assume that (A 4 ) or (A 5 ) hold.Then J λ,μ satisfies: (i) there exist constants ρ, α > 0 such that J λ,μ (u)| ∂B ρ ≥ α; (ii) there exists e ∈ X 0,s,p \B ρ such that J λ,μ (e) ≤ 0.
Proof of Theorem 1.3 Note that Lemma 4.1 holds under condition (A 4 ) or (A 5 ).Therefore the mountain pass lemma shows that J λ,μ (u) has a critical point, which is a weak solution of problem (1.1), and we complete the proof of Theorem 1.3.