Infinitely many solutions for hemivariational inequalities involving the fractional Laplacian

Abstract

In the paper, we consider the following hemivariational inequality problem involving the fractional Laplacian:

$$ \textstyle\begin{cases} (-\Delta )^{s}u+\lambda u\in \alpha (x) \partial F(x,u) & x \in \varOmega , \\ u=0 & x\in \mathbb{R} ^{N} \backslash \varOmega , \end{cases} $$

where Ω is a bounded smooth domain in $\mathbb{R} ^{N}$ with $N\geq 3$, $(-\Delta )^{s}$ is the fractional Laplacian with $s\in (0,1)$, $\lambda >0$ is a parameter, $\alpha (x): \varOmega \rightarrow \mathbb{R}$ is a measurable function, $F(x, u):\varOmega \times \mathbb{R} \rightarrow \mathbb{R}$ is a nonsmooth potential, and $\partial F(x,u)$ is the generalized gradient of $F(x, \cdot )$ at $u\in \mathbb{R}$. Under some appropriate assumptions, we obtain the existence of a nontrivial solution of this hemivariational inequality problem. Moreover, when F is autonomous, we obtain the existence of infinitely many solutions of this problem when the nonsmooth potentials F have suitable oscillating behavior in any neighborhood of the origin (respectively the infinity) and discuss the properties of the solutions.

1 Introduction

In the present paper, we are concerned with the following hemivariational inequality:

$$ (P_{\lambda }) \textstyle\begin{cases} (-\Delta )^{s}u+\lambda u\in \alpha (x) \partial F(x,u) & x \in \varOmega , \\ u=0 & x\in \mathbb {R}^{N} \backslash \varOmega , \end{cases} $$

where Ω is a bounded smooth domain in $\mathbb {R}^{N}$ with $N\geq 3$, $\lambda >0$ is a parameter, $\alpha (x): \varOmega \rightarrow \mathbb {R}$ is a measurable function, $F(x, u):\varOmega \times \mathbb {R}\rightarrow \mathbb {R}$ is a nonsmooth potential, while $\partial F(x,u)$ is the generalized gradient of $F(x, \cdot )$ at $u\in \mathbb {R}$, and $(-\Delta )^{s}$ with $s\in (0,1)$ is the fractional Laplacian which may be defined as

$$ -(-\Delta )^{s} u(x)=\frac{1}{2} \int _{\mathbb {R}^{n}} \frac{u(x+y)+u(x-y)-2u(x)}{ \vert y \vert ^{N+2s}}\,dy $$

for $x\in \mathbb {R}^{N}$.

In recent years, boundary value problems involving fractional operators and more general nonlocal operators have attracted more interest since these operators appear in concrete applications in many fields, such as anomalous diffusion [1], quantum mechanics [2], obstacle problems [3], phase transition [4], minimal surface [5], and so on. In the literature, various papers deal with the existence and multiplicity of nontrivial solutions for the fractional Laplacian equations with superlinear or subcritical, critical, asymptotically linear nonlinearities, and some elliptic boundary problems involving the nonlocal integrodifferential operator are also exploited, see for example [6,7,8,9,10] and the references therein.

We note that the existence of infinitely many solutions for elliptic boundary value problems without the symmetric functionals is an important topic in nonlinear analysis, hence there are a lot of papers focused on the existence of infinitely many solutions of elliptic boundary value problems involving the local Laplacian and the p-Laplacian, see for example [11, 12]. Also, this study for boundary value problems involving fractional Laplacian has received attention of some authors via variational methods recently. For instance, in [13], under some subcritical growth assumptions on the nonlinearity, Servadei established results on the existence of infinitely many solutions for the nonlocal fractional Laplace equations; in [14], with the help of the Ambrosetti–Rabinowitz type condition, Zhang et al. established some results on the existence of infinitely many solutions for fractional Laplace equations with subcritical growth nonlinearities and superlinear growth nonlinearities; in [15], under some local growth conditions on the nonlinearity, Li and Wei obtained the existence of infinitely many solutions for fractional Laplace equations; in [16], by using variational and topological methods, Ambrosio et al. obtained the existence of infinitely many solutions for fractional nonlocal p-Laplacian problem under some oscillating conditions near the origin or at infinity.

We point out that the above works on nonlocal boundary value problems can be formulated as “smooth” since the involving nonlinearities are continuous. So we wonder what happens if the nonlocal boundary value problems have nonsmooth nonlinearities (this kind of problems is called hemivariational inequality). In fact, the research on the existence and multiplicity of solutions for the hemivariational inequality problems involving a local Laplace or p-Laplace type operator has attracted the interest of many authors in the past thirty years, see for instance [17,18,19,20,21,22] and the references therein.

Therefore, motivated by the papers mentioned above, especially by[13,14,15,16,17,18, 22], we are interested in the existence of a nontrivial solution and infinitely solutions for the fractional hemivariational inequality problem $(P_{\lambda })$ in the present paper. By using the theory of nonsmooth critical point and the idea of constructing a special set in the working function space such that the minimum point of the energy functional on this set is actually a weak solution of problem $(P_{\lambda })$, we obtain an existence result (see Theorem 3.2 for more details). Moreover, under suitable oscillatory assumptions on the autonomous nonsmooth potential $F:\mathbb {R}\rightarrow \mathbb {R}$ at zero or at infinity, we establish the existence of infinitely many solutions for problem $(P_{\lambda })$ (see Theorems 4.1 and 4.2 for more details). It is worth noting that, if F is a primitive of a continuous function f, problem $(P_{\lambda })$ will become a boundary value problem involving a fractional operator. The method to obtain the existence of infinitely many solutions for this boundary value problem is different from the ones used in [13,14,15,16].

This paper is organized as follows. In Sect. 2 we give some notations and preliminaries. In Sect. 3, under appropriate assumptions, we present a result about the existence of a solution for problem $(P_{\lambda })$. In Sect. 4, we show the existence of infinitely many solutions whenever the autonomous nonlinearity F oscillates in any neighborhood of the origin (respectively infinity) and obtain some properties of the solutions.

2 Preliminaries

In the section, we gather some notions and results which will be useful in the proofs of our results.

Our method of proof uses the nonsmooth critical point theory, which in turn is based on the subdifferential theory for locally Lipschitz functional. In the following, firstly we briefly recall some basic definitions and results from these two theories. For details, we refer to Clarke [23] and Gasinński and Papageorgiou [24].

Let X be a Banach space and $\mathbf{X}^{*}$ be its topological dual. By $\langle \cdot ,\cdot \rangle$ we denote the duality pairing between $\mathbf{X}^{*}$ and X. Given a locally Lipschitz function $f:\mathbf{X}\rightarrow \mathbb {R}$, the generalized directional derivative of f at a point $u\in \mathbf{X}$ along the direction $h\in \mathbf{X}$ is defined by

$$ f^{\circ }(u;h)=\limsup_{\substack{v\to u \\ t\downarrow 0}}\frac{f(v+th)-f(v)}{t}, $$

and the generalized gradient of f at a point $u\in \mathbf{X}$ is defined by

$$ \partial f(u)=\bigl\{ u^{*}\in \mathbf{X}^{*}: f^{\circ }(u;h)\geq \bigl\langle u^{*},h \bigr\rangle , \forall h\in \mathbf{X}\bigr\} . $$

It is clear that, by using the Hahn–Banach theorem, $\partial f(u) \neq \emptyset$. If f is also convex, then the multifunction $\mathbf{X}\ni u\rightarrow \partial f(u)\in 2^{X^{*}}\backslash \{\emptyset \}$ coincides with the Clarke subdifferential in the sense of convex analysis, defined by

$$ \partial f(u)=\bigl\{ u^{*}\in X^{*}: f(v)-f(u)\geq \bigl\langle u^{*},v-u\bigr\rangle , \forall v\in \mathbf{X}\bigr\} . $$

Also the generalized gradient satisfies the mean value rule (so-called Lebourg’s mean value theorem). Namely, if $f:\mathbf{X}\rightarrow \mathbb {R}$ is Lipschitz on an open set containing the line segment $[u,v]$, we can find $w=ut+(1-t)v$ with $t\in (0,1)$ and $w^{*} \in \partial f(w)$ such that

$$ f(v)-f(u)=\bigl\langle w^{*},v-u\bigr\rangle . $$

Let $\varPhi :\mathbf{X}\rightarrow \mathbb {R}$ be a locally Lipschitz function and $\varPsi :\mathbf{X}\rightarrow \mathbb {R}\cup \{+\infty \}$ be a proper, convex, and low semicontinuous functional. Then $\varPhi +\varPsi$ is called a Motreanu–Panagiotopoulos-type functional (see [25]).

Definition 2.1

Let $\varPhi +\varPsi$ be a Motreanu–Panagiotopoulos-type functional, $u\in \mathbf{X}$. Then u is a critical point of $\varPhi +\varPsi$ if, for every $v\in \mathbf{X}$,

$$ \varPhi ^{\circ }(u) (v-u)+\varPsi (v)-\varPsi (u)\geq 0. $$

In the sequel, for the reader’s convenience, we briefly recall the definition of the fractional Sobolev space and give some notations and useful lemmas. For further details on the fractional Sobolev space, we refer to [8, 9] and to the references therein.

Given $0< s<1$, we denote the sets $\mathscr{X}$ and $\mathscr{X}_{0}$ by

$$ \mathscr{X}=\biggl\{ u:\mathbb {R}^{N}\rightarrow \mathbb {R}\text{ is Lebesgue measurable }:u|_{\varOmega } \in L^{2} (\varOmega ), \frac{u(x)-u(y)}{ \vert x-y \vert ^{\frac{N+2s}{2}}} \in L^{2} (Q)\biggr\} $$

and

$$ \mathscr{X}_{0}=\bigl\{ u\in \mathscr{X}:u=0 \text{ a.e. in } \mathbb {R}^{N}\backslash \varOmega \bigr\} , $$

(2.1)

where $Q=\mathbb {R}^{2N}\backslash \mathcal {O}$ and $\mathcal {O}=(\mathbb {R}^{N}\backslash \varOmega )\times (\mathbb {R}^{N}\backslash \varOmega )$. Moreover, the spaces $\mathscr{X}$ and $\mathscr{X}_{0}$ are endowed with the norms respectively defined as

$$ \Vert u \Vert _{\mathscr{X}}= \Vert u \Vert _{L^{2}(\varOmega )}+ \biggl( \int _{Q} \frac{(u(x)-u(y))^{2}}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \biggr)^{\frac{1}{2}}, \quad \forall u\in \mathscr{X}, $$

and

$$ \Vert u \Vert _{\mathscr{X}_{0}}= \biggl( \int _{Q} \frac{(u(x)-u(y))^{2}}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \biggr)^{\frac{1}{2}}, \quad \forall u\in \mathscr{X}_{0}. $$

(2.2)

By Lemma 6 of [8], the norms $\|\cdot \|_{\mathscr{X}}$ and $\|\cdot \|_{\mathscr{X}_{0}}$ are equivalent. We define an inner product $\langle \cdot ,\cdot \rangle$ on $\mathscr{X}_{0}$ as follows:

$$ \langle u,v\rangle = \int _{Q} \frac{(u(x)-u(y))(v(x)-v(y))}{ \vert x-y \vert ^{N+2s}}\,dx\,dy, \quad \forall u,v\in \mathscr{X}_{0}, $$

(2.3)

then $\mathscr{X}_{0}$ is a Hilbert space (see [8, Lemma 7]). Also note that in (2.2) and (2.3) the integral can be extended to all $\mathbb {R}^{N}\times \mathbb {R}^{N}$ since $u,v\in \mathscr{X}_{0}$. $\mathscr{X}_{0}$ is called the fractional Sobolev space (also denote $\mathscr{X}_{0}$ as $H^{s}(\varOmega )$).

Throughout this paper, we will always respectively denote $\|u\|_{p}= \|u\|_{L^{p}(\varOmega )}$ ($1\leq p\leq \infty$). As usual, we denote by “→” and “⇀” the strong and weak convergence.

Now, we give a convergence property for bounded sequences in $\mathscr{X}_{0}$ and a property for eigenvalues of $(-\Delta )^{s}$, which will be used in the following. These results are proved in [8, 9].

Lemma 2.1

([8, Lemma 8])

Let $\{v_{n}\}$be a bounded sequence in $\mathscr{X}_{0}$. Then there exists $v\in L^{p} (\mathbb {R}^{N})$such that, up to a subsequence, $v_{n}\rightarrow v$in $L^{p} (\mathbb {R}^{N})$as $n\rightarrow \infty$for any $p\in [1,2^{*})$.

Lemma 2.2

([9, Lemma 9])

For the fractional eigenvalue problem

$$ \textstyle\begin{cases} (-\Delta )^{s}u=\lambda u & x\in \varOmega , \\ u=0 & x\in \mathbb {R}^{N} \backslash \varOmega , \end{cases} $$

there exists an eigenvalues sequence $\{\lambda _{n}\}$ with

$$ 0< \lambda _{1}< \lambda _{2}\leq \cdots \leq \lambda _{n}\leq \cdots , $$

and

$$ \lambda _{n}\to \infty \quad \textit{as } n\to \infty , $$

where

$$ \lambda _{1}=\min_{u\in \mathscr{X}_{0}\backslash \{0\}} \frac{ \int _{\mathbb {R}^{2N}} \vert u(x)-u(y) \vert ^{2}\frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy}{\int _{ \varOmega } \vert u(x) \vert ^{2}\,dx}. $$

(2.4)

Let us introduce the Euler functional $J_{\lambda }:\mathscr{X}_{0} \rightarrow \mathbb {R}$ corresponding to problem $(P_{\lambda })$ as follows:

$$ \begin{aligned}[b] J_{\lambda }(u) &= \frac{1}{2} \int _{Q} \bigl\vert u(x)-u(y) \bigr\vert ^{2} \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy +\frac{\lambda }{2} \int _{\varOmega } \bigl\vert u(x) \bigr\vert ^{2}\,dx - \int _{\varOmega }\alpha (x)F(x,u)\,dx \\ &=\frac{1}{2} \Vert u \Vert _{\mathscr{X} _{0}}^{2} + \frac{\lambda }{2} \Vert u \Vert _{2}^{2} - \int _{\varOmega }\alpha (x)F(x,u)\,dx. \end{aligned} $$

(2.5)

We denote

$$\begin{aligned}& \varPsi (u)\triangleq \frac{1}{2} \int _{Q} \bigl\vert u(x)-u(y) \bigr\vert ^{2} \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy=\frac{1}{2} \Vert u \Vert _{\mathscr{X}_{0}}^{2}, \end{aligned}$$

(2.6)

$$\begin{aligned}& \varPhi _{1} (u)\triangleq \frac{1}{2} \int _{\varOmega } \bigl\vert u(x) \bigr\vert ^{2}\,dx= \frac{1}{2} \Vert u \Vert _{2}^{2} \quad \text{and} \quad \varPhi _{2} (u)\triangleq \int _{\varOmega }\alpha (x)F(x,u)\,dx, \end{aligned}$$

(2.7)

then

$$ J_{\lambda } (u)=\varPsi (u){+}\lambda \varPhi _{1} (u)-\varPhi _{2} (u). $$

(2.8)

3 Existence of a solution for problem $(P_{\lambda })$

Let $\alpha : \varOmega \to \mathbb {R}$, $F: \varOmega \times \mathbb {R}\to \mathbb {R}$. In this section, we obtain the existence of a solution on problem $(P_{\lambda })$ under the following assumptions:

$(\alpha )$:

$\alpha \in L^{2} (\varOmega )$ is nonnegative, and there exists $D\subset \varOmega$ with $\operatorname{meas}(D)>0$ such that $\alpha (x)>0$ for almost all $x\in D$;

$(f_{1})$:

$F(\cdot ,u)$ is measurable for all $u\in \mathbb {R}$, $F(x,\cdot )$ is locally Lipschitz for almost all $x\in \varOmega$, $F(x,0)=0$.

$(f_{2})$:

There exist $q\in (1,2^{*})$ and $C_{0}>0$ such that

$$ \bigl\vert u^{*} \bigr\vert \leq C_{0}\bigl(1+ \vert u \vert ^{q-1}\bigr) $$

for almost all $x\in \varOmega$, every $u\in \mathbb {R}$, and $u^{*}\in \partial F(x,u)$;

$(f_{3})$:

There are constants a, b, c, d with $d< c\leq 0< a< b$, such that $u^{*}\leq 0$ for almost all $x\in \varOmega$, every $u\in [a,b]$, and $u^{*}\in \partial F(x,u)$; $u^{*}\geq 0$ for almost all $x\in \varOmega$, every $u\in [d,c]$, and $u^{*}\in \partial F(x,u)$.

In order to prove the main result, we define the set

$$ \mathscr{U}=\bigl\{ u\in \mathscr{X}_{0}: d\leq u(x)\leq b \text{ for almost every } x\in \mathbb {R}^{N}\bigr\} , $$

(3.1)

where constants b and d are given in condition $(f_{3})$.

Lemma 3.1

Assume that $F(x,u)$satisfies $(f_{1})$and $(f_{2})$, $\alpha (x)$satisfies condition $(A)$. Then the functional $J_{\lambda }(u)$is sequentially weakly lower semicontinuous on $\mathscr{U}$, where the set $\mathscr{U}$is defined by (3.1).

Proof

Firstly, we claim that the set $\mathscr{U}$ is weakly closed. The set $\mathscr{U}$ is clearly convex. Moreover, it is closed in $\mathscr{X}_{0}$. In fact, let $\{u_{n}\}\subset \mathscr{U}$ with

$$ u_{n}\rightarrow u\in \mathscr{X}_{0} \quad \text{as } n\rightarrow \infty , $$

then $\{u_{n}\}$ is bounded in $\mathscr{X}_{0}$. By Lemma 2.1, up to a subsequence of $\{u_{n}\}$ (which is still denoted as $\{u_{n}\}$)

$$ u_{n}\rightarrow u \quad \text{in } L^{p} \bigl( \mathbb {R}^{N}\bigr) \text{ as } n\to \infty , $$

where $1\leq p<2^{*}$. Thus

$$ u_{n}(x)\rightarrow u(x) \quad \text{for almost every } x\in \mathbb {R}^{N}. $$

Since $d\leq u_{n} (x)\leq b$, $d\leq u(x)\leq b$ for almost every $x\in \mathbb {R}^{N}$. So $u\in \mathscr{U}$. Then $\mathscr{U}$ is weakly closed.

In the sequel, we prove that $J_{\lambda }(u)$ is weakly lower semicontinuous. Note that $\varPsi (u)$ and $\varPhi _{1} (u)$ are weakly lower semicontinuous, where $\varPsi (u)$ and $\varPhi _{1} (u)$ are defined by (2.6) and (2.7), respectively. From (2.8), we only need to prove that $\varPhi _{2} (u)$ is weakly continuous, where $\varPhi _{2} (u)$ is defined by (2.7). Arguing by contradiction, we assume that $\{u_{n}\}\subset \mathscr{U}$ is a sequence with $u_{n}\rightharpoonup u\in \mathscr{X}_{0}$ but $\varPhi _{2} (u_{n})\nrightarrow \varPhi _{2} (u)$ as $n\rightarrow \infty$. Then, up to a subsequence of $\{u_{n}\}$, we can choose a constant $\varepsilon _{0}$ such that

$$ 0< \varepsilon _{0}\leq \bigl\vert \varPhi _{2} (u_{n})-\varPhi _{2} (u) \bigr\vert $$

(3.2)

for large enough $n\in \mathbb{N}$. According to Lemma 2.1 and $\mathscr{U}$ is weakly closed, we see that

$$ u_{n}\rightarrow u\in \mathscr{U} \quad \text{in } L^{2} \bigl(\mathbb {R}^{N}\bigr) \text{ as } n\to \infty . $$

(3.3)

By Lebourg’s mean value theorem, for almost all $x\in \varOmega$, there exist $\theta _{n}\in (0,1)$ and $w_{n}^{*} \in \partial F(x,w_{n})$ with $w_{n}=u+\theta _{n}(u_{n}-u)\in \mathscr{U}$ such that

$$ \begin{aligned}[b] \bigl\vert F(x,u_{n})-F(x,u) \bigr\vert &= \bigl\vert w_{n}^{*} \bigr\vert \vert u_{n}-u \vert \\ &\le C_{0}\bigl(1+ \vert w_{n} \vert ^{q-1}\bigr) \vert u _{n}-u \vert \\ &\le C_{1} \vert u_{n}-u \vert , \end{aligned} $$

(3.4)

where the first inequality is due to condition $(f_{2})$ and the last inequality comes from (3.1), the definition of $\mathscr{U}$, $C_{1}$ is a positive constant. Therefore, it follows from (3.2), (3.3), (3.4), and Hölder’s inequality that

$$ \begin{aligned} 0< \varepsilon _{0} &\leq \bigl\vert \varPhi _{2} (u_{n})-\varPhi _{2} (u) \bigr\vert \\ &\leq \int _{\varOmega }\alpha (x) \bigl\vert F(x,u_{n})-F(x,u) \bigr\vert \,dx \\ &\leq C_{1} \Vert \alpha \Vert _{2} \Vert u_{n}-u \Vert _{2}\rightarrow 0, \quad \text{as } n\to \infty , \end{aligned} $$

which is impossible. The proof is completed. □

Lemma 3.2

Let $\lambda >0$. Assume that $F(x,u)$satisfies $(f_{1})$and $(f_{3})$, $\alpha (x)$satisfies condition $(A)$. If there exists $u_{0}\in \mathscr{U}$such that

$$ J_{\lambda }(u_{0})=\inf_{\substack{u\in \mathscr{U}}}J_{\lambda }(u), $$

where $J_{\lambda }$is defined by (2.5) and the set $\mathscr{U}$is defined by (3.1), then $u_{0}(x)\in [c,a]$for almost every $x\in \mathbb {R}^{N}$, where constantsaandcare given in condition $(f_{3})$.

Proof

Since $u_{0}\in \mathscr{U}$, $u_{0}\in \mathscr{X} _{0}$ and $d\leq u_{0}(x)\leq b$ for almost every $x \in \mathbb {R}^{N}$.

Denote

$$ \begin{aligned} & A=\bigl\{ x\in \mathbb {R}^{N}: u_{0}(x)\notin [c,a]\bigr\} , \\ & A_{1}=\bigl\{ x\in A: u _{0}(x)< c\bigr\} , \\ & A_{2}=\bigl\{ x\in A: u_{0}(x)>a\bigr\} . \end{aligned} $$

(3.5)

Clearly, $A_{1}\cup A_{2}=A$. Let us define

$$ v_{0} (x)=\textstyle\begin{cases} c & x\in A_{1}, \\ u_{0}(x) & x\in \mathbb {R}^{N}\backslash A, \\ a & x\in A_{2}. \end{cases} $$

(3.6)

Firstly, we will prove that

$$ \Vert v_{0} \Vert _{\mathscr{X}_{0}}\leq \Vert u_{0} \Vert _{\mathscr{X}_{0}}. $$

(3.7)

From $u_{0}\in \mathscr{U}\subset \mathscr{X}_{0}$ and (2.1), the definition of $\mathscr{X}_{0}$, we have $u_{0}(x)=0$ for almost every $x\in \mathbb {R}^{N}\backslash \varOmega$. Since $d< c\leq 0< a< b$, $u_{0}(x) \in [c,a]$ for almost every $x\in \mathbb {R}^{N}\backslash \varOmega$. From (3.5) and (3.6), we have $v_{0} (x)=0$ for almost every $x\in \mathbb {R}^{N}\backslash \varOmega$. Therefore,

$$ \begin{aligned}[b] & \int _{Q} \bigl\vert v_{0} (x)-v_{0} (y) \bigr\vert ^{2}\frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy- \int _{Q} \bigl\vert u _{0} (x)-u_{0} (y) \bigr\vert ^{2} \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ & \quad = \int _{\mathbb {R}^{2N}}\bigl( \bigl\vert v_{0} (x)-v_{0} (y) \bigr\vert ^{2}- \bigl\vert u_{0} (x)-u_{0} (y) \bigr\vert ^{2}\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy, \end{aligned} $$

(3.8)

where $Q=\mathbb {R}^{2n}\backslash \mathcal {O}$ and $\mathcal {O}=(\mathbb {R}^{n}\backslash \varOmega )\times (\mathbb {R}^{n}\backslash \varOmega )$. According to the definitions of $v_{0}$, $A_{1}$ and $A_{2}$, we get

$$ \begin{aligned} & \int _{Q} \bigl\vert v_{0} (x)-v_{0} (y) \bigr\vert ^{2}\frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy- \int _{Q} \bigl\vert u _{0} (x)-u_{0} (y) \bigr\vert ^{2} \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &\quad =\biggl( \int _{A_{1} \times A_{1}} + \int _{A_{1} \times A_{2}} + \int _{A_{1} \times (\mathbb {R}^{N}\backslash A)} + \int _{A_{2} \times A_{1}} + \int _{A_{2} \times A_{2}}+ \int _{A_{2} \times (\mathbb {R}^{N}\backslash A)} + \int _{(\mathbb {R}^{N}\backslash A)\times A_{1}} \\ &\qquad {}+ \int _{(\mathbb {R}^{N}\backslash A)\times A_{2}}+ \int _{(\mathbb {R}^{N}\backslash A)\times (\mathbb {R}^{N}\backslash A)}\biggr) \bigl( \bigl\vert v_{0} (x)-v _{0} (y) \bigr\vert ^{2}- \bigl\vert u_{0}(x)-u_{0}(y) \bigr\vert ^{2}\bigr) \\ &\qquad {}\times \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &\quad \leq 0, \end{aligned} $$

which implies that $\|v_{0}\|_{\mathscr{X}_{0}}^{2}\leq \|u_{0}\|_{ \mathscr{X}_{0}}^{2}$. That is, (3.7) holds. Clearly, $v_{0} \in \mathscr{U}$.

Secondly, we claim that $\operatorname{meas}(A)=0$. Indeed, by Lebourg’s mean value theorem, for almost all $x\in \varOmega$, there exist $\theta _{1}, \theta _{2}\in (0,1)$, $w_{1}^{*} \in \partial F(x,w_{1})$ with $w_{1}=c+\theta _{1}(u_{0}-c)\in \mathscr{U}$ and $w_{2}^{*} \in \partial F(x,w_{2})$ with $w_{2}=a+\theta _{2}(u_{0}-a)\in \mathscr{U}$, such that

$$\begin{aligned}& F(x,u_{0})-F(x,c)) = w_{1}^{*} (u_{0}-c), \end{aligned}$$

(3.9)

$$\begin{aligned}& F(x,u_{0})-F(x,a)) = w_{2}^{*} (u_{0}-a). \end{aligned}$$

(3.10)

Therefore, we have

$$ \begin{aligned}[b] & \int _{\varOmega }\alpha (x) \bigl(F(x,u_{0})-F(x,v_{0}) \bigr)\,dx \\ &\quad = \int _{A_{1}} \alpha (x) \bigl(F(x,u_{0})-F(x,c) \bigr)\,dx+ \int _{A_{2}}\alpha (x) \bigl(F(x,u_{0})-F(x,a) \bigr)\,dx \\ &\quad = \int _{A_{1}}\alpha (x)w_{1}^{*} (u_{0}-c)\,dx + \int _{A_{2}}\alpha (x)w_{2}^{*} (u_{0}-a)\,dx\leq 0, \end{aligned} $$

(3.11)

where the first equality follows from the definitions of $A_{1}$, $A _{2}$, A and $v_{0} (x)=u_{0} (x)=0$ for almost every $x\in \mathbb {R}^{N}\backslash \varOmega$, the second equality is due to (3.9) and (3.10), the last inequality comes from condition $(f_{3})$ and the definitions of $A_{1}$, $A_{2}$. On the other hand, we know that

$$ \Vert v_{0} \Vert _{2}^{2} - \Vert u_{0} \Vert _{2}^{2}= \int _{A_{1}}\bigl(c^{2}-u_{0}^{2} \bigr)\,dx+ \int _{A_{2}}\bigl(a^{2} -u_{0}^{2} \bigr)\,dx\leq 0. $$

(3.12)

By (3.7), (3.11), and (3.12), we deduce that

$$ \begin{aligned} &J_{\lambda }(v_{0})-J_{\lambda }(u_{0}) \\ &\quad =\frac{1}{2}\bigl( \Vert v_{0} \Vert _{\mathscr{X}_{0}}^{2}- \Vert u_{0} \Vert _{\mathscr{X}_{0}}^{2}\bigr)+\frac{\lambda }{2}\bigl( \Vert v_{0} \Vert _{2}^{2} - \Vert u_{0} \Vert _{2}^{2}\bigr)+ \int _{\varOmega }\alpha (x) (F(x,u _{0})-F(x,v_{0})\,dx \leq 0, \end{aligned} $$

this, together with $J_{\lambda }(u_{0})= \inf_{\substack{u\in \mathscr{U}}}J(u)$, yields that $J_{\lambda }(v _{0})-J_{\lambda }(u_{0})=0$. Then in particular

$$ \int _{A_{1}}\bigl(c^{2}-u_{0}^{2} \bigr)\,dx=0 \quad \text{and}\quad \int _{A_{2}}\bigl(a^{2} -u_{0}^{2} \bigr)\,dx=0, $$

which implies

$$ \operatorname{meas}(A)=\operatorname{meas}(A_{1})+\operatorname{meas}(A_{2})=0. $$

Hence, $c\leq u_{0} (x)\leq a$ for almost every $x\in \mathbb {R}^{N}$. The proof is complete. □

Let $\theta \in \mathbb {R}$ and $\varepsilon \in \mathbb {R}_{+}$. In the following, we define the function $h(\theta )=\min \{b,\max \{d, \theta \}\}$, where b and d are given in condition $(f_{3})$, and let $w(x)=h(u_{0}(x)+\varepsilon v(x))$ for any $v\in \mathscr{X}_{0}$, where $u_{0}$ is given in Lemma 3.2. Then, by the definition of $\mathscr{X}_{0}$ and $\mathscr{U}$, we have

$$ w(x)=h\bigl(u_{0}(x)+\varepsilon v(x) \bigr)=\textstyle\begin{cases} d & u_{0}+\varepsilon v< d, \\ u_{0}(x) +\varepsilon v(x) & d\leq u_{0}+\varepsilon v < b, \\ b & u_{0}+\varepsilon v\geq b, \end{cases} $$

(3.13)

and $w\in \mathscr{U}$. We introduce the sets

$$ \begin{aligned} & B_{1} (\varepsilon )= \bigl\{ x\in \mathbb {R}^{N}: u_{0}(x)+\varepsilon v(x)< d \bigr\} , \\ & B_{2} (\varepsilon )=\bigl\{ x\in \mathbb {R}^{N}: d\leq u_{0}(x)+\varepsilon v(x)< b \bigr\} , \\ & B_{3} (\varepsilon )=\bigl\{ x\in \mathbb {R}^{N}: u_{0}(x)+\varepsilon v(x)\geq b\bigr\} . \end{aligned} $$

(3.14)

Clearly, $B_{1} (\varepsilon )\cup B_{2} (\varepsilon )\cup B _{3} (\varepsilon )=\mathbb {R}^{N}$ and $B_{1} (\varepsilon )\subset \varOmega$, $B_{3} (\varepsilon )\subset \varOmega$. Moreover, the following lemma holds.

Lemma 3.3

$\operatorname{meas}(B_{1} (\varepsilon ))\rightarrow 0$and $\operatorname{meas}(B _{3} (\varepsilon ))\rightarrow 0$as $\varepsilon \rightarrow 0^{+}$, respectively.

Proof

Suppose the contrary, i.e., $\operatorname{meas}(B_{1} ( \varepsilon ))\nrightarrow 0$ as $\varepsilon \rightarrow 0^{+}$. Thus there exists a number $\eta _{0} >0$, $\forall n\in \mathbb {N}$, $\exists n _{0} \in \mathbb {N}$, $n_{0}>n$, such that

$$ \operatorname{meas}\biggl(B_{1} \biggl( \frac{1}{n_{0}}\biggr)\biggr)\geq \eta _{0}. $$

(3.15)

Let $v\in \mathscr{X}_{0}$. Since, for any $M>0$, we have

$$ M\operatorname{meas}\bigl\{ x\in \mathbb {R}^{N}: \vert v \vert >M\bigr\} \leq \int _{\{x\in \mathbb {R}^{N}: \vert v \vert >M\}} v(x)\,dx\leq \int _{\varOmega } \bigl\vert v(x) \bigr\vert \,dx\leq c_{1} \Vert v \Vert _{2}, $$

then

$$ \operatorname{meas}\bigl\{ x\in \mathbb {R}^{N}: \vert v \vert >M\bigr\} \leq \frac{c_{1}}{M} \Vert v \Vert _{2} \rightarrow 0, \quad \text{as } M\rightarrow \infty . $$

So there exists a positive constant $M_{0}$ such that

$$ \operatorname{meas}\bigl\{ x\in \mathbb {R}^{N}: \vert v \vert >M_{0}\bigr\} \leq \frac{\eta _{0}}{2}. $$

(3.16)

On the other hand, taking into account $u_{0}(x)\in [c,a]\subset (d,b)$. For each $|v(x)|\leq M_{0}$, there exists large enough $n_{0} \in \mathbb {N}$ which satisfies (3.15) such that

$$ u_{0}(x) +\frac{1}{n_{0}} v(x)\geq \frac{c+d}{2}>d. $$

(3.17)

It follows from (3.14) and (3.17) that

$$ B_{1} \biggl(\frac{1}{n_{0}}\biggr)\cap \bigl\{ x\in \mathbb {R}^{N}: \vert v \vert \leq M_{0}\bigr\} =\emptyset . $$

(3.18)

Hence, combining with the above (3.15), (3.16), and (3.18), we obtain

$$ \begin{aligned} \eta _{0}&\leq \operatorname{meas}\biggl(B_{1} \biggl(\frac{1}{n_{0}}\biggr)\biggr) \\ &= \int _{B_{1} (\frac{1}{n_{0}})\cap \{x\in \mathbb {R}^{N}:|v|>M_{0}\}}\,dx+ \int _{B_{1} (\frac{1}{n_{0}})\cap \{x\in \mathbb {R}^{N}:|v|\leq M_{0}\}}\,dx \\ &= \int _{B_{1} (\frac{1}{n_{0}})\cap \{x\in \mathbb {R}^{N}:|v|>M_{0} \}}\,dx\leq \frac{\eta _{0}}{2}, \end{aligned} $$

which is a contradiction. Similarly, we can prove that $\operatorname{meas}(B _{3} (\varepsilon ))\rightarrow 0$ as $\varepsilon \rightarrow 0^{+}$. The proof is completed. □

Theorem 3.1

Let $\lambda >0$. Assume that $F(x,u)$satisfies $(f_{1})$, $(f_{2})$, and $(f_{3})$, $\alpha (x)$satisfies condition $(A)$. Then there exists $u_{0}\in \mathscr{U}$such that the functional

$$ J_{\lambda }(u_{0})=\inf_{\substack{u\in \mathscr{U}}}J_{\lambda }(u), $$

where $J_{\lambda }$is defined by (2.5) and the set $\mathscr{U}$is defined by (3.1). Moreover, $u_{0}(x) \in [c,a]$for almost every $x\in \mathbb {R}^{N}$.

Proof

Let $u\in \mathscr{U}$. By Lebourg’s mean value theorem, for almost all $x\in \varOmega$, there exist $\theta \in (0,1)$ and $w^{*} \in \partial F(x,w)$ with $w=\theta u\in \mathscr{U}$ such that

$$ \begin{aligned}[b] \bigl\vert F(x,u) \bigr\vert &= \bigl\vert F(x,u)-F(x,0) \bigr\vert = \bigl\vert w^{*} \bigr\vert \vert u \vert \\ &\le C_{0}\bigl(1+ \vert u \vert ^{q-1}\bigr) \vert u \vert \le C_{2}, \end{aligned} $$

(3.19)

where the first equality is due to condition $(f_{1})$, the first inequality is due to condition $(f_{2})$, and the last inequality comes from the definition of $\mathscr{U}$ and $C_{2}$ is a positive constant. By the definition of $J_{\lambda }(u)$ and (3.19), we know that

$$ J_{\lambda }(u)\geq - \int _{\varOmega }\alpha (x)F(x,u)\,dx\geq -C_{2} \Vert \alpha \Vert _{1}, \quad \forall u\in \mathscr{U}. $$

(3.20)

Then $J_{\lambda }(u)$ is bounded from below on $\mathscr{U}$.

Let $\eta =\inf_{u\in \mathscr{U}}J_{\lambda }(u)$. There are $\{u_{n}\}\subset \mathscr{U}$ such that

$$ \eta \leq J_{\lambda }(u_{n})\leq \eta +\frac{1}{n}, \quad \forall n \in \mathbb {N}. $$

(3.21)

So by (2.5), combining the definition of $J_{\lambda }(u)$ with (3.20) and (3.21), we obtain

$$ \frac{1}{2} \Vert u_{n} \Vert _{\mathscr{X}_{0}}^{2} \leq J_{\lambda }(u_{n})+ \int _{\varOmega }\alpha (x)F(x,u)\,dx\leq \eta +1 + C_{2} \Vert \alpha \Vert _{1}, \quad n\in \mathbb {N}. $$

Hence $\{u_{n}\}\subset \mathscr{U}$ is bounded in $\mathscr{X}_{0}$. Note that $\mathscr{X}_{0}$ is a Hilbert space and $\mathscr{U}$ is weakly closed, there exists a subsequence of $\{u_{n}\}$ (which is still denoted as $\{u_{n}\}$) such that $u_{n}\rightharpoonup u_{0}$ for some $u_{0}\in \mathscr{U}$. Due to the weak lower semicontinuity of $J_{\lambda }(u)$ (Lemma 3.1), we have

$$ \eta =\liminf_{\substack{n\rightarrow \infty }}J_{\lambda }(u_{n}) \geq J_{\lambda }(u_{0})\geq \eta . $$

Hence

$$ J_{\lambda }(u_{0})=\eta =\inf_{u\in \mathscr{U}}J_{\lambda }(u). $$

By Lemma 3.2, $u_{0}(x)\in [c,a]$ for almost every $x\in \mathbb {R}^{N}$. □

Remark 3.1

Functions satisfying all the conditions in Theorem 3.1 exist. For instance, let $\varOmega \subset \mathbb{R}^{3}$, the function $F(x,u):\varOmega \times \mathbb{R}\to \mathbb{R}$ be defined by

$$ F(x,u)= \textstyle\begin{cases} (\sum_{i=1}^{3} \sin x_{i}+5)\cos \frac{\pi u}{2}, &\text{if } x=(x_{1}, x_{2},x_{3})\in \varOmega , 1< u< +\infty , \\ 0, &\text{if } x=(x_{1}, x_{2},x_{5})\in \varOmega , 0\leq u\leq 1, \\ (\sum_{i=1}^{3} \sin x_{i}+5)u^{3}, &\text{if } x=(x_{1}, x_{2},x _{3})\in \varOmega , -\infty < u< 0. \end{cases} $$

Obviously, $F(x,u)$ satisfies $(f_{1})$ and $|u^{*}|\leq 20(1+|u|^{2})$ for every $x\in \varOmega$, $u\in \mathbb {R}$ and $u^{*}\in \partial F(x,u)$, where $2^{*}=6$, $q=3<2^{*}$. That is, $F(x,u)$ satisfies $(f_{2})$. Take $a=\frac{3}{2}$, $b=2$, $c=-\frac{3}{2}$, $d=-2$, then $F(x,u)$ satisfies $(f_{3})$. Let $\alpha (x)=1$, $\forall x=(x_{1}, x_{2},x_{3})\in \varOmega$. Then $\alpha :\varOmega \to \mathbb{R}$, condition $(A)$.

Theorem 3.2

Let $\lambda >0$. Assume that $F(x,u)$satisfies $(f_{1})$, $(f_{2})$, and $(f_{3})$, $\alpha (x)$satisfies condition $(A)$. Then problem $(P_{\lambda })$has a solution.

Proof

By Theorem 3.1, there exists $u_{0}\in \mathscr{X}_{0}$ with $u_{0}\in [c,a]$ such that $J_{\lambda }(u_{0})= \inf_{\substack{u\in \mathscr{U}}}J_{\lambda }(u)$. In the following, we only need to prove that $u_{0}$ is a solution of problem $(P_{\lambda })$.

Let $\varGamma _{\mathscr{U}}$ be the indicator function of the set $\mathscr{U}$, i.e.,

$$ \varGamma _{\mathscr{U}} (u)=\textstyle\begin{cases} 0 & u\in \mathscr{U}, \\ +\infty & u\notin U. \end{cases} $$

Obviously, $\varGamma _{\mathscr{U}}$ is convex, lower semicontinuous, and proper. Define the functional $I_{\lambda }: \mathscr{X}_{0}\rightarrow \mathbb {R}\cup \{+\infty \}$ by $I_{\lambda }=J_{\lambda }+ \varGamma _{\mathscr{U}}$. Since $J_{\lambda }$ is of class $C^{1}$ on $\mathscr{X}_{0}$, $I_{\lambda }$ is the Szulkin-type functional. Note that $u_{0}$ is a local minimum point of $J_{\lambda }$ on $\mathscr{U}$, thus a local minimum point of the functional $I_{\lambda }$. Moreover, $u_{0}$ is a critical point of $I_{\lambda }$, that is,

$$ J^{\circ }_{\lambda }(u_{0}) (w-u_{0})+ \varGamma _{\mathscr{U}} (w)- \varGamma _{\mathscr{U}} (u_{0}) \geq 0, \quad \forall w\in \mathscr{X} _{0}. $$

In particular,

$$ J^{\circ }_{\lambda }(u_{0}) (w-u_{0}) \geq 0, \quad \forall w\in \mathscr{U}. $$

Note that

$$\begin{aligned}& (-\varPhi _{2})^{\circ }(u_{0},w-u_{0})= \varPhi ^{\circ }_{2} (u_{0},u_{0}-w), \\ & \varPhi _{2}^{\circ }(u_{0};u_{0}-w) \leq \int _{\varOmega }\alpha (x)F^{ \circ }(x,u_{0};u_{0}-w)\,dx, \end{aligned}$$

that is,

$$ \begin{aligned}[b] 0 \leq{}& \int _{\mathbb {R}^{2N}}\bigl(u_{0}(x)-u_{0}(y) \bigr)[\bigl(w(x)-u_{0}(x)\bigr)-\bigl(w(y)-u _{0}(y) \bigr)\frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &{}+\lambda \int _{\varOmega }u_{0}(x) \bigl(w(x)-u _{0}(x)\bigr)\,dx+ \int _{\varOmega }\alpha (x)F^{\circ }(x,u_{0};u_{0} -w)\,dx, \quad \forall w\in \mathscr{U}. \end{aligned} $$

(3.22)

For each $v\in \mathscr{X}_{0}$, we choose w defined as (3.13) and estimate every term of the right-hand side of (3.22). We shall complete the proof by the following steps.

Step 1: We estimate the second term of the right-hand side of (3.22).

Due to $u_{0} =0$ in $\mathbb {R}^{N}\backslash \varOmega$, we have

$$ \begin{aligned}[b] & \lambda \int _{\varOmega }u_{0}(x) \bigl(w(x)-u_{0}(x) \bigr)\,dx \\ &\quad =\lambda \varepsilon \int _{\varOmega }u_{0}(x)v(x)\,dx+\lambda \int _{\mathbb {R}^{N}}\bigl[u_{0}(x) \bigl(w(x)-u _{0}(x)\bigr)-\varepsilon u_{0} (x)v(x)\bigr]\,dx \\ &\quad =\lambda \varepsilon \int _{\varOmega }u_{0}(x)v(x)\,dx+\lambda \int _{B_{1} (\varepsilon )}u_{0}(x) \bigl(d-u _{0}(x)-\varepsilon v(x)\bigr)\,dx \\ &\qquad {} +\lambda \int _{B_{3} (\varepsilon )}u_{0}(x) \bigl(b-u_{0}(x)- \varepsilon v(x)\bigr)\,dx, \end{aligned} $$

(3.23)

and

$$ \begin{aligned}[b] & \int _{B_{1} (\varepsilon )}u_{0}(x) \bigl(d-u_{0}(x)- \varepsilon v(x)\bigr)\,dx \\ &\quad = \int _{B_{1} (\varepsilon )}\,d\bigl(d-u_{0}(x)-\varepsilon v(x) \bigr)\,dx- \int _{B_{1} (\varepsilon )}\bigl(u_{0}(x)-d\bigr)^{2} \\ &\qquad {}- \varepsilon \int _{B_{1} (\varepsilon )}\bigl(u_{0}(x)-d\bigr)v(x)\,dx \\ &\quad \leq \varepsilon \int _{B_{1} (\varepsilon )}\bigl(d-u_{0}(x)\bigr)v(x)\,dx. \end{aligned} $$

(3.24)

Similarly, arguing as above, we get

$$ \int _{B_{3} (\varepsilon )}u_{0}(x) \bigl(b-u_{0}(x)- \varepsilon v(x)\bigr)\,dx \leq \varepsilon \int _{B_{3} (\varepsilon )}\bigl(b-u_{0}(x)\bigr)v(x)\,dx. $$

(3.25)

By (3.23), (3.24), and (3.25), we obtain

$$ \begin{aligned}[b] & \lambda \int _{\varOmega }u_{0}(x) \bigl(w(x)-u_{0}(x) \bigr)\,dx \\ &\quad \leq \lambda \varepsilon \biggl( \int _{\varOmega }u_{0}(x)v(x)\,dx+ \int _{B_{1} (\varepsilon )}\bigl(d-u _{0}(x)\bigr)v(x)\,dx \\ &\qquad {}+ \int _{B_{3} (\varepsilon )}\bigl(b-u_{0}(x)\bigr)v(x)\,dx\biggr). \end{aligned} $$

(3.26)

Step 2: We estimate the first term of the right-hand side of (3.22). Due to (3.14), we have

$$ \begin{aligned}[b] & \int _{\mathbb {R}^{2N}}\bigl(u_{0}(x)-u_{0}(y) \bigr)\bigl[\bigl(w(x)-u_{0}(x)\bigr)-\bigl(w(y)-u_{0}(y) \bigr)\bigr] \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &\quad \leq \varepsilon \int _{\mathbb {R}^{2N}}\bigl(u_{0}(x)-u_{0}(y) \bigr) \bigl(v(x)-v(y)\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &\qquad {} -\varepsilon \int _{B_{1} (\varepsilon ) \times B_{1} (\varepsilon )} \bigl(u _{0}(x)-u_{0}(y) \bigr) \bigl(v(x)-v(y)\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &\qquad {} -\varepsilon \int _{B_{3} (\varepsilon ) \times B_{3} (\varepsilon )} \bigl(u _{0}(x)-u_{0}(y) \bigr) \bigl(v(x)-v(y)\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &\qquad {} +2 \int _{B_{1} (\varepsilon ) \times B_{2} (\varepsilon )} \bigl(u_{0}(x)-u _{0}(y) \bigr) \bigl(d-u_{0}(x)-\varepsilon v(x)\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &\qquad {} +2 \int _{B_{1} (\varepsilon ) \times B_{3} (\varepsilon )}\bigl(u_{0}(x)-u _{0}(y) \bigr) \bigl(d-u_{0}(x)-\varepsilon v(x)+u_{0}(y)+ \varepsilon v(y)-b\bigr) \\ &\qquad {}\times \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &\qquad {} +2 \int _{B_{2} (\varepsilon ) \times B_{3} (\varepsilon ) }\bigl(u_{0}(x)-u _{0}(y) \bigr) \bigl(u_{0}(y)+\varepsilon v(y)-b\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy. \end{aligned} $$

(3.27)

In the following, we estimate the fourth term of (3.27). Now take $R>0$ with $\varOmega \subset B_{R}=\{x\in \mathbb {R}^{N}:|x|\leq R\}$. Owing to $u_{0} (x)=v(x)=0$ for $x\in B_{R}^{c}$, then $B_{2} (\varepsilon ) \cap B_{R}^{c}=B_{R}^{c}$. Therefore, we obtain

$$ \begin{aligned}[b] & \int _{B_{1} (\varepsilon ) \times B_{2} (\varepsilon )} \bigl(u_{0}(x)-u _{0}(y) \bigr) \bigl(d-u_{0}(x)-\varepsilon v(x)\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &\quad \leq \varepsilon \int _{B_{1} (\varepsilon ) \times (B_{2} (\varepsilon )\cap B_{R})}\bigl(v(y)-v(x)\bigr) \bigl(d-u _{0}(x)-\varepsilon v(x)\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &\qquad {} +\varepsilon \int _{B_{1} (\varepsilon ) \times B_{R}^{c}}\bigl(v(y)-v(x)\bigr) \bigl(d-u _{0}(x)-\varepsilon v(x)\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy. \end{aligned} $$

(3.28)

Since $u_{0} (x)+\varepsilon v(x)< d$ for $x\in B_{1} (\varepsilon )$ and $c\leq u_{0} (x)\leq a$, we know $v(x)<0$ for $x\in B_{1} (\varepsilon )$. Consequently,

$$ \begin{aligned}[b] & \int _{B_{1} (\varepsilon ) \times B_{R}^{c}}\bigl(v(y)-v(x)\bigr) \bigl(d-u_{0}(x)- \varepsilon v(x)\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ & \quad \leq C_{3} \int _{B_{1} (\varepsilon )}\bigl(-v(x)\bigr) \bigl(d-u_{0}(x)- \varepsilon v(x)\bigr)\,dx \int _{R}^{+\infty }\frac{\rho ^{N-1}}{\rho ^{N+2s}}\,d\rho \\ &\quad = C_{4} \int _{B_{1} (\varepsilon )}\bigl(-v(x)\bigr) \bigl(d-u_{0}(x)- \varepsilon v(x)\bigr)\,dx, \end{aligned} $$

(3.29)

where $C_{3}$, $C_{4}$ are constants. By (3.28) and (3.29), we get

$$\begin{aligned}& \int _{B_{1} (\varepsilon ) \times B_{2} (\varepsilon )} \bigl(u_{0}(x)-u _{0}(y) \bigr) \bigl(d-u_{0}(x)-\varepsilon v(x)\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\& \quad \leq \varepsilon \int _{B_{1} (\varepsilon ) \times (B_{2} (\varepsilon )\cap B_{R})}\bigl(v(y)-v(x)\bigr) \bigl(d-u _{0}(x)-\varepsilon v(x)\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\& \qquad {} +\varepsilon C_{4} \int _{B_{1} (\varepsilon )}\bigl(-v(x)\bigr) \bigl(d-u_{0}(x)- \varepsilon v(x)\bigr)\,dx. \end{aligned}$$

(3.30)

Similarly, we can estimate the sixth term of (3.27) and obtain

$$ \begin{aligned}[b] & \int _{B_{2} (\varepsilon ) \times B_{3} (\varepsilon )} \bigl(u_{0}(x)-u _{0}(y) \bigr) \bigl(u_{0}(y)+\varepsilon v(y)-b\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &\quad \leq \varepsilon \int _{(B_{2} (\varepsilon )\cap B_{R})\times B_{3} (\varepsilon )}\bigl(v(y)-v(x)\bigr) \bigl(u _{0}(y)+ \varepsilon v(y)-b\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &\qquad {} +\varepsilon C_{5} \int _{B_{3} (\varepsilon )}v(y) \bigl(u_{0}(y)+\varepsilon v(y)-b \bigr)\,dy, \end{aligned} $$

(3.31)

where $C_{5}$ is a constant. Then we estimate the fifth term of (3.27). According to $u_{0} (x)+\varepsilon v(x)< d$ for $x\in B_{1} (\varepsilon )$ and $u_{0} (y)+\varepsilon v(y)\geq b$ for $y\in B_{3} (\varepsilon )$, a simple calculation shows that $u_{0} (x)-u_{0} (y)\leq \varepsilon (v(y)-v(x))$. Hence,

$$ \begin{aligned}[b] & \int _{B_{1} (\varepsilon ) \times B_{3} (\varepsilon )}\bigl(u_{0}(x)-u _{0}(y) \bigr) \bigl(d-u_{0}(x)-\varepsilon v(x)+u_{0}(y)+ \varepsilon v(y)-b\bigr) \\ &\qquad {}\times \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ & \quad \leq \varepsilon \int _{B_{1} (\varepsilon ) \times B_{3} (\varepsilon )}\bigl(v(y)-v(x)\bigr) \bigl(d-u _{0}(x)-\varepsilon v(x)+u_{0}(y)+\varepsilon v(y)-b \bigr) \\ &\qquad {}\times \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy. \end{aligned} $$

(3.32)

Combining with the above (3.27), (3.30), (3.31), and (3.32), we obtain the estimation of the first term of the right-hand side of (3.22), i.e.,

$$\begin{aligned} & \int _{\mathbb {R}^{2N}}\bigl(u_{0}(x)-u_{0}(y) \bigr)[\bigl(w(x)-u_{0}(x)\bigr)-\bigl(w(y)-u_{0}(y) \bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &\quad \leq \varepsilon \int _{\mathbb {R}^{2N}}\bigl(u_{0}(x)-u _{0}(y) \bigr) \bigl(v(x)-v(y)\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &\qquad {} -\varepsilon \int _{B_{1} (\varepsilon ) \times B_{1} (\varepsilon )} \bigl(u _{0}(x)-u_{0}(y) \bigr) \bigl(v(x)-v(y)\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &\qquad {} -\varepsilon \int _{B_{3} (\varepsilon ) \times B_{3} (\varepsilon )} \bigl(u _{0}(x)-u_{0}(y) \bigr) \bigl(v(x)-v(y)\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &\qquad {} +2\varepsilon \int _{B_{1} (\varepsilon ) \times (B_{2} (\varepsilon )\cap B_{R})}\bigl(v(y)-v(x)\bigr) \bigl(d-u _{0}(x)-\varepsilon v(x)\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &\qquad {} +2\varepsilon C_{4} \int _{B_{1} (\varepsilon )}\bigl(-v(x)\bigr) \bigl(d-u_{0}(x)- \varepsilon v(x)\bigr)\,dx \\ &\qquad {} +2\varepsilon C_{5} \int _{B_{3} (\varepsilon )}v(y) \bigl(u _{0}(y)+\varepsilon v(y)-b\bigr)\,dy \\ &\qquad {} +2\varepsilon \int _{(B_{2} (\varepsilon )\cap B_{R})\times B_{3} (\varepsilon )}\bigl(v(y)-v(x)\bigr) \bigl(u _{0}(y)+ \varepsilon v(y)-b\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &\qquad {} +2\varepsilon \int _{B_{1} (\varepsilon ) \times B_{3} (\varepsilon )}\bigl(v(y)-v(x)\bigr) \bigl(d-u_{0}(x)- \varepsilon v(x)+u_{0}(y)+ \varepsilon v(y)-b\bigr) \\ &\qquad {}\times \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy. \end{aligned}$$

(3.33)

Step 3: We estimate the third term of the right-hand side of (3.22).

For each $w_{0}^{*} \in \partial F(x,u_{0})$, we have $\langle w_{0} ^{*},h\rangle \le F^{\circ }(x,u_{0};h)$, $\forall h\in \mathscr{X}_{0}$. By (3.13) and (3.14), we obtain

$$ \begin{aligned}[b] & \int _{\varOmega }\alpha (x)w_{0}^{*} \bigl(u_{0}(x)-w(x)\bigr)\,dx \\ &\quad =-\varepsilon \int _{\varOmega }\alpha (x)w_{0}^{*} v(x)\,dx+ \int _{B_{1} (\varepsilon )} \alpha (x)w_{0}^{*} \bigl(u_{0}(x)+\varepsilon v(x)-d\bigr)\,dx \\ &\qquad {} + \int _{B_{3} (\varepsilon )}\alpha (x)w_{0}^{*} \bigl(u_{0}(x)+\varepsilon v(x)-b\bigr)\,dx. \end{aligned} $$

(3.34)

Furthermore, from condition $(f_{2})$ and the fact that $u_{0}\in [d, b]$, we obtain

$$\begin{aligned}& \int _{B_{1} (\varepsilon )}\alpha (x)w_{0}^{*} \bigl(u_{0}(x)+\varepsilon v(x)-d\bigr)\,dx \leq { -\varepsilon C_{6} \int _{B_{1} (\varepsilon )}\alpha (x)v(x)\,dx}, \end{aligned}$$

(3.35)

$$\begin{aligned}& \int _{B_{3} (\varepsilon )}\alpha (x)w_{0}^{*} \bigl(u_{0}(x)+\varepsilon v(x)-b\bigr)\,dx \leq { \varepsilon C_{7} \int _{B_{3} (\varepsilon )}\alpha (x)v(x)\,dx}, \end{aligned}$$

(3.36)

where $C_{6}$, $C_{7}$ are positive constants. Therefore, according to (3.34), (3.35), and (3.36), we obtain

$$ \begin{aligned}[b] & \int _{\varOmega }\alpha (x)w_{0}^{*} \bigl(u_{0}(x)-w(x)\bigr)\,dx \\ &\quad \leq {-\varepsilon \biggl( \int _{\varOmega }\alpha (x)w_{0}^{*} v(x)\,dx+C_{6} \int _{B_{1} (\varepsilon )}\alpha (x)v(x)\,dx -C_{7} \int _{B_{3} (\varepsilon )}\alpha (x)v(x)\,dx\biggr)}. \end{aligned} $$

(3.37)

Step 4: In the sequel, from the above inequalities (3.22), (3.26), (3.33), and (3.37), we deduce that

$$\begin{aligned} 0 \leq{} & \int _{\mathbb {R}^{2N}}\bigl(u_{0}(x)-u_{0}(y) \bigr) \bigl(v(x)-v(y)\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy+\lambda \int _{\varOmega }u_{0}(x)v(x)\,dx \\ &{} - \int _{\varOmega }\alpha (x){ w_{0}^{*} }v(x)\,dx - \int _{B_{1} (\varepsilon ) \times B_{1} (\varepsilon ) } \bigl(u_{0}(x)-u _{0}(y) \bigr) \bigl(v(x)-v(y)\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &{} - \int _{B_{3} (\varepsilon ) \times B_{3} (\varepsilon ) } \bigl(u_{0}(x)-u _{0}(y) \bigr) \bigl(v(x)-v(y)\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &{} +2 \int _{B_{1} (\varepsilon ) \times (B_{2} (\varepsilon ) \cap B_{R})}\bigl(v(y)-v(x)\bigr) \bigl(d-u _{0}(x)-\varepsilon v(x)\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ & {}+2 C_{4} \int _{B_{1} (\varepsilon ) }\bigl(-v(x)\bigr) \bigl(d-u_{0}(x)- \varepsilon v(x)\bigr)\,dx +2 C_{5} \int _{B_{3} (\varepsilon ) }v(y) \bigl(u_{0}(y)+\varepsilon v(y)-b \bigr)\,dy \\ &{} +2 \int _{(B_{2} (\varepsilon ) \cap B_{R})\times B_{3} (\varepsilon ) }\bigl(v(y)-v(x)\bigr) \bigl(u_{0}(y)+ \varepsilon v(y)-b\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ & {}+2 \int _{B_{1} (\varepsilon ) \times B_{3} (\varepsilon ) }\bigl(v(y)-v(x)\bigr) \bigl(d-u _{0}(x)-\varepsilon v(x)+u_{0}(y)+\varepsilon v(y)-b \bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &{} +\lambda \int _{B_{1} (\varepsilon ) }\bigl(d-u _{0}(x)\bigr)v(x)\,dx+\lambda \int _{B_{3} (\varepsilon ) }\bigl(b-u_{0}(x)\bigr)v(x)\,dx \\ & {}-C_{6} \int _{B_{1} (\varepsilon ) }\alpha (x)v(x)\,dx+C_{7} \int _{B_{3} (\varepsilon ) }\alpha (x)v(x)\,dx. \end{aligned}$$

(3.38)

It follows from Lemma 3.3 that $\operatorname{meas}(B_{1} (\varepsilon ))\rightarrow 0$ and $\operatorname{meas}(B_{3} (\varepsilon ))\rightarrow 0$ as $\varepsilon \to 0^{+}$. Therefore, take $\varepsilon \rightarrow 0^{+}$ in (3.38), we obtain

$$ \begin{aligned} 0\leq{} & \int _{\mathbb {R}^{2N}}\bigl(u_{0}(x)-u_{0}(y) \bigr) \bigl(v(x)-v(y)\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &{}+\lambda \int _{\varOmega }u_{0}(x)v(x)\,dx - \int _{\varOmega }\alpha (x)w_{0}^{*} v(x)\,dx. \end{aligned} $$

By the arbitrariness of $v\in \mathscr{X}_{0}$, we have

$$ \begin{aligned} & \int _{\mathbb {R}^{2N}}\bigl(u_{0}(x)-u_{0}(y) \bigr) \bigl(v(x)-v(y)\bigr) \frac{1}{ \vert x-y \vert ^{N+2s}}\,dx\,dy \\ &\quad {}+\lambda \int _{\varOmega }u_{0}(x)v(x)\,dx - \int _{\varOmega }\alpha (x)w_{0}^{*} v(x)\,dx=0. \end{aligned} $$

Note that $w_{0}^{*} \in \partial F(x,u_{0})$, so $u_{0}$ is a solution of problem $(P_{\lambda })$. Then the conclusion of Theorem 3.2 is proved. □

4 Existence of infinitely many solutions for problem $(P_{\lambda })$

In this section, we assume that F is autonomous, i.e., $F(x,u)=F(u)$, $F: \mathbb {R}\to \mathbb {R}$, and satisfies the following conditions:

$(\bar{f_{1}})$:

$F: \mathbb {R}\to \mathbb {R}$ is locally Lipschitz and $F(0)=0$;

$(\bar{f_{2}})$:

There exist $q\in (1,2^{*})$ and $C_{0}>0$ such that

$$ \bigl\vert u^{*} \bigr\vert \leq C_{0}\bigl(1+ \vert u \vert ^{q-1}\bigr) $$

for every $u\in \mathbb {R}$ and $u^{*}\in \partial F(u)$.

Let $\alpha : \varOmega \to \mathbb {R}$. We will obtain two results on infinitely many solutions for the problem

$$ (\bar{P}_{\lambda }) \textstyle\begin{cases} (-\Delta )^{s}u+\lambda u\in \alpha (x) \partial F(u) & x\in \varOmega , \\ u=0 & x\in \mathbb {R}^{N} \backslash \varOmega , \end{cases} $$

when the nonlinearity F satisfies the above conditions and has a suitable oscillation near the origin or at infinity (see hypotheses $(F_{1}^{0})$ and $(F_{2}^{0})$, or $(F_{1}^{\infty })$ and $(F_{2}^{\infty })$ in the following).

Lemma 4.1

Let $F(u)$satisfy $(\bar{f_{1}})$and $\hat{u}\in \mathbb {R}$. If $\hat{u}^{*} <0$for $\hat{u}^{*} \in \partial F( \hat{u})$, then there exists $\eta >0$such that ${z}^{*} \le 0$for ${z}^{*} \in \partial F( z)$, where $z\in (\hat{u}-\eta , \hat{u}+\eta )$.

Proof

We prove it by contradiction. Suppose on the contrary that, for each $k\in \mathbb {N}$, there exist $z_{k}\in (\hat{u}- \frac{1}{k}, \hat{u}+\frac{1}{k})$ and $z_{k}^{*} \in \partial F( z _{k})$ such that $z_{k}^{*} > 0$. Let $\hat{z}^{*}$ be a cluster point of $\{z_{k}^{*} \}$, then $\hat{z}^{*}\ge 0$. Note that $\lim_{k\to \infty }z_{k}=\hat{u}$. By virtue of Proposition 2.1.5 of [23](P.29), we have $\hat{z}^{*}\in \partial F( \hat{u})$, hence $\hat{z}^{*}< 0$, which contradicts $\hat{z}^{*}\ge 0$. This completes the proof. □

Theorem 4.1

Let $\lambda >0$and $\alpha (x)$satisfy condition $(A)$. Assume that $F(u)$satisfies $(\bar{f_{1}})$, $(\bar{f_{2}})$, and the following conditions:

$(F_{1}^{0})$:

$-\infty <\liminf_{s\to 0} \frac{F(s)}{s ^{2}}\leq \limsup_{s\to 0} \frac{F(s)}{s^{2}}=+\infty$;

$(F_{2}^{0})$:

There exist two sequences $\{\hat{u}_{k}\}\subset (0,+\infty )$and $\{\bar{u}_{k}\}\subset (-\infty ,0)$with

$$ \lim_{k\to \infty }\hat{u}_{k} =\lim _{k\to \infty }\bar{u}_{k} =0 $$

such that, for all $k\in \mathbb {N}$,

$$ \hat{u}_{k}^{*} < 0 \quad \textit{and}\quad \bar{u}_{k}^{*} >0 $$

for $\hat{u}_{k}^{*} \in \partial F(\hat{u}_{k})$and $\bar{u}_{k}^{*} \in \partial F(\bar{u}_{k})$, respectively.

Then there exists a sequence $\{u_{k}\}$of distinct weak solutions of problem $(\bar{P}_{\lambda })$such that

$$ \lim_{k\to \infty }J_{\lambda }(u_{k})=0 \quad \textit{and}\quad \lim_{k\to \infty } \Vert u_{k} \Vert _{X_{0}}=0. $$

Proof

Let us define a function $\mu : \mathbb {R}^{N}\rightarrow \mathbb {R}$ such that

$$ \begin{aligned} &\mbox{(i)} \quad \mu (x)=1 \quad \text{for } x\in D; \\ &\mbox{(ii)}\quad 0\leq \mu (x)\leq 1 \quad \text{for } x\in \varOmega \backslash D ; \\ &\mbox{(iii)}\quad \mu (x)=0 \quad \text{for } x\in \mathbb {R}^{N} \backslash \varOmega , \end{aligned} $$

(4.1)

where D is the set given in condition $(A)$. Then the function $\mu \in \mathscr{X}_{0}$ exists thanks to the fact that $C_{0}^{2}( \varOmega )\subseteq \mathscr{X}_{0}$ (see [9] Lemma 11).

Due to condition $(F_{2}^{0})$, without loss of generality, we can suppose that sequences $\{\hat{u}_{k}\}$ and $\{\bar{u}_{k}\}$ are respectively decreasing and increasing. By virtue of Lemma 4.1, we may choose the sequences $\{a_{k}\},\{b_{k}\}\subset (0,+\infty )$ and $\{c_{k}\},\{d_{k}\}\subset (-\infty ,0)$ such that, for all $k\in \mathbb {N}$ and almost all $x\in \varOmega$,

$$\begin{aligned}& b_{k+1}< a_{k}< \hat{u}_{k}< b_{k} \quad \text{and}\quad u^{*}\leq 0, \quad \forall u\in [a_{k},b_{k}], u^{*}\in \partial F(u); \\& d_{k}< \bar{u}_{k}< c_{k}< d_{k+1} \quad \text{and}\quad u^{*}\geq 0, \quad \forall u\in [d_{k},c_{k}], u^{*}\in \partial F(u). \end{aligned}$$

Take the set

$$ \mathscr{U}_{k}=\bigl\{ u\in \mathscr{X}_{0}: d_{k}\leq u(x)\leq b_{k} \text{ for almost every } x\in \mathbb {R}^{N}\bigr\} . $$

(4.2)

By Theorems 3.1 and 3.2, there exists $u_{k} \in \mathscr{U}_{k}$ such that the functional

$$ J_{\lambda }(u_{k})=\inf _{\substack{u\in \mathscr{U}_{k}}}J_{\lambda }(u). $$

(4.3)

Moreover, $u_{k}(x)\in [c_{k},a_{k}]$ for almost every $x\in \mathbb {R}^{N}$ and $u_{k}$ is a weak solution of problem $(\bar{P}_{\lambda })$.

Firstly, we claim that when k is large enough, $J_{\lambda }(u_{k})<0$.

Indeed, by using the first inequality in condition $(F_{1}^{0})$, there exist two numbers $l_{0}>0$ and $\rho _{0}\in (0,b_{1})$ such that

$$ F(s)\geq -l_{0} s^{2}, \quad \forall s \in (-\rho _{0},\rho _{0}). $$

(4.4)

Recall that $\lambda _{1}$ is the first eigenvalue of $(-\Delta )^{s}$, from (2.4), the definition of $\lambda _{1}$, we clearly know

$$ \Vert \mu \Vert _{2}^{2}\leq \frac{1}{\lambda _{1}} \Vert \mu \Vert _{\mathscr{X}_{0}} ^{2}, $$

(4.5)

where μ is defined by (4.1). Due to condition $(A)$ and $\|\mu \|_{\mathscr{X}_{0}}<+\infty$, we can choose $L_{0}>0$ large enough so that

$$ \biggl(\frac{1}{2}+\frac{\lambda }{2 \lambda _{1}}\biggr) \Vert \mu \Vert _{\mathscr{X}_{0}} ^{2}+l_{0} \Vert \alpha \Vert _{1}< L_{0} \int _{D} \alpha (x)\,dx. $$

(4.6)

Using the last equality in condition $(F_{1}^{0})$, for the above $L_{0}$, there exist $s_{k} \in (-\rho _{0},\rho _{0})$ with $c_{k}\leq s_{k}\leq a_{k}$, $s_{k}\neq 0$ and $\lim_{k\to \infty } s _{k}=0$ such that

$$ F(s_{k})>L_{0} s_{k}^{2} $$

(4.7)

for large enough $k\in \mathbb {N}$.

Define $w_{k}=s_{k} \mu$. Combining $c_{k}\leq s_{k}\leq a_{k}$, (4.1), and (4.2), we deduce that $w_{k}\in \mathscr{U}_{k}$. Since $\lim_{k\to \infty } s_{k}=0$, when k is large enough, we have

$$ \begin{aligned}[b] J_{\lambda }(w_{k}) ={}&\frac{1}{2}s_{k}^{2} \Vert \mu \Vert _{\mathscr{X}_{0}} ^{2}+\frac{\lambda }{2}s_{k}^{2} \Vert \mu \Vert _{2}^{2} - \int _{\varOmega } \alpha (x)F(s_{k} \mu )\,dx \\ \leq&{} \biggl(\frac{1}{2}+\frac{\lambda }{2 \lambda _{1}} \biggr)s_{k}^{2} \Vert \mu \Vert _{\mathscr{X}_{0}}^{2} - \int _{\varOmega \backslash D}\alpha (x)F(s_{k} \mu )\,dx \\ &{}- \int _{D} \alpha (x)F(s _{k}\mu )\,dx \\ \leq{}& \biggl(\frac{1}{2}+\frac{\lambda }{2 \lambda _{1}} \biggr)s_{k}^{2} \Vert \mu \Vert _{\mathscr{X}_{0}}^{2} +l_{0} s_{k}^{2} \Vert \alpha \Vert _{1} -L_{0} s_{k}^{2} \int _{D} \alpha (x)\,dx \\ ={}&s_{k}^{2} \biggl[ \biggl(\frac{1}{2}+ \frac{\lambda }{2 \lambda _{1}} \biggr) \Vert \mu \Vert _{ \mathscr{X}_{0}}^{2} +l_{0} \Vert \alpha \Vert _{1}-L_{0} \int _{D} \alpha (x)\,dx \biggr] \\ < {}&0, \end{aligned} $$

(4.8)

where the first inequality comes from (4.5), the second inequality follows from (4.4) and (4.7), and the last inequality follows from (4.6). Hence, by (4.3) and (4.8), we obtain

$$ J_{\lambda }(u_{k})\leq J_{\lambda }(w_{k})< 0, $$

(4.9)

when k is large enough.

Secondly, we prove that $\lim_{k\to \infty }J_{\lambda }(u _{k})=0$.

By using Lebourg’s mean value theorem and conditions $(\bar{f_{1}})$, $(\bar{f_{2}})$ again, there exist $\theta _{k} \in (0,1)$ and $v_{k}^{*}\in \partial F(v_{k})$ with $v_{k} =\theta _{k} u_{k} \in \mathscr{U}_{k}$ such that

$$ \begin{aligned} F(u_{k}) & \leq \bigl\vert F(u_{k})-F(0) \bigr\vert = \bigl\vert \bigl\langle v_{k}^{*}, u_{k}\bigr\rangle \bigr\vert \\ &\leq C_{0}\bigl(1+ \vert v_{k} \vert ^{q-1}\bigr) \vert a_{k}-c_{k} \vert \\ &\leq C_{0} \bigl(1+(a_{k}-c _{k})^{q-1} \bigr) (a_{k}-c_{k}). \end{aligned} $$

The above inequality, together with $\lim_{k\to \infty }a_{k}= \lim_{k\to \infty }c_{k}=0$ and (4.9), yields that

$$ 0>J_{\lambda }(u_{k})\geq - \int _{\varOmega }\alpha (x)F(u_{k})\,dx \geq -C _{0} \bigl(1+(a_{k}-c_{k})^{q-1} \bigr) (a_{k}-c_{k}) \Vert \alpha \Vert _{1}\to 0, $$

as $k\to \infty$, i.e., $\lim_{k\to \infty }J_{\lambda }(u_{k})=0$.

At last, from the definition of $J_{\lambda }$, we have

$$ \frac{1}{2} \Vert u_{k} \Vert _{\mathscr{X}_{0}}^{2} \leq \int _{\varOmega }\alpha (x)F(u _{k})\,dx \leq C_{0} \bigl(1+(a_{k}-c_{k})^{q-1} \bigr) (a_{k}-c_{k}) \Vert \alpha \Vert _{1} \to 0, $$

as $k\to \infty$. Thus $\lim_{k\to \infty }\|u_{k}\|_{ \mathscr{X}_{0}}=0$. The proof is complete. □

Remark 4.1

Functions satisfying all the conditions in Theorem 4.1 exist. For instance, let $\varOmega \subset \mathbb{R}^{3}$, the function $F(u): \mathbb{R}\to \mathbb{R}$ be defined by

$$ F(u)= \textstyle\begin{cases} u(\sin \frac{1}{u}+1), & \text{if } 0< u< +\infty , \\ u^{3}, & \text{if } -\infty < u\le 0. \end{cases} $$

Obviously, $F( u)$ satisfies $(\bar{f}_{1})$, $(\bar{f}_{2})$, $\liminf_{u\to 0} \frac{F(u)}{u^{2}}=0$, and $\limsup_{u\to 0} \frac{F(u)}{u^{2}}=+\infty$. Take $\hat{u}_{k}=\frac{1}{2k \pi }$, then $\lim_{k\to \infty }\hat{u}_{k} =0$ and $F'(\hat{u}_{k}) =1-2k\pi <0$. Take $\bar{u}_{k}\in (-\infty ,0)$ and $\lim_{k\to \infty }\bar{u}_{k} =0$, then $F'(\bar{u}_{k})=3 \bar{u}_{k}^{2}>0$. That is, $F( u)$ satisfies $(F_{1}^{0})$ and $(F_{2}^{0})$.

Theorem 4.2

Let $\lambda >0$and $\alpha (x)$satisfy condition $(A)$. Assume that $F(u)$satisfies $(\bar{f_{1}})$, $(\bar{f_{2}})$, and the following conditions:

$(F_{1}^{\infty })$:

$-\infty <\liminf_{s\to {+\infty }} \frac{F(s)}{s^{2}}\leq \limsup_{s\to {+\infty }} \frac{F(s)}{s ^{2}}=+\infty$;

$(F_{2}^{\infty })$:

There exists a sequence $\{\tilde{u}_{k}\} \subset (0,+\infty )$with $\lim_{k\to \infty }\tilde{u}_{k} =+\infty$and $u_{0}\in (-\infty ,0)$such that

$$ \tilde{u}_{k}^{*} < 0 \quad \textit{for all } k\in \mathbb {N}\quad \textit{and}\quad u_{0}^{*} \geq 0, $$

where $\tilde{u}_{k}^{*} \in \partial F(\tilde{u}_{k})$, $u_{0}^{*} \in \partial F(u_{0})$.

Then there exists a sequence $\{u_{k}\}$of distinct weak solutions of problem $(\bar{P}_{\lambda })$such that

$$ \lim_{k\to \infty }J_{\lambda }(u_{k})=-\infty . $$

Proof

By virtue of the first inequality in condition $(F_{1}^{\infty })$, there exist $l_{\infty }>0$ and $\rho _{\infty } >0$ such that

$$ F(s)\geq -l_{\infty }s^{2}, \quad \forall s> \rho _{\infty }. $$

(4.10)

Due to condition $(A)$, we can choose $L_{\infty }>0$ large enough so that

$$ \biggl(\frac{1}{2}+\frac{\lambda }{2 \lambda _{1}}\biggr) \Vert \mu \Vert _{\mathscr{X}_{0}} ^{2}+l_{\infty } \Vert \alpha \Vert _{1}< L_{\infty } \int _{D} \alpha (x)\,dx, $$

(4.11)

where μ is defined by (4.1), $\lambda _{1}$ is the first eigenvalue of $(-\Delta )^{s}$. The last equality of condition $(F_{1}^{\infty })$ ensures the existence of a sequence $\{\hat{s} _{k}\}\subset (0,+\infty )$ with $\lim_{k\to \infty }\hat{s}_{k}=+ \infty$ such that

$$ F(\hat{s}_{k}) >L_{\infty }{ \hat{s}_{k}}^{2} $$

(4.12)

for large enough $k\in \mathbb {N}$. By condition $(F_{2}^{\infty })$, $\lim_{k\to \infty } \tilde{u}_{k}=+\infty$. Let us take an increasing subsequence $\{\tilde{u}_{m_{k}}\}$ of $\{\tilde{u}_{k}\}$ such that

$$ \hat{s}_{k} \leq \tilde{u}_{m_{k}} \quad \text{for all } k\in \mathbb {N}. $$

(4.13)

In addition, according to condition $(F_{2}^{\infty })$, we can choose two sequences $\{a_{k}'\},\{b_{k}'\}\subset (0,+\infty )$ such that, for all $k\in \mathbb {N}$,

$$ a_{k}'< \tilde{u}_{m_{k}}< b_{k}'< a_{k+1}' \quad \text{and}\quad u^{*} \leq 0, \quad \forall u\in \bigl[a_{k}',b_{k}'\bigr] \quad \text{and}\quad u^{*}\in \partial F(u); $$

and two numbers $c'$ and $d'$ with $d'< c'\leq 0$ such that

$$ u^{*}\geq 0, \quad \forall u\in \bigl[d',c' \bigr] \quad \text{and}\quad u^{*}\in \partial F(u). $$

Let

$$ \mathscr{U}_{k}'=\bigl\{ u\in \mathscr{X}_{0}: d'\leq u(x)\leq b_{k}', \text{a.e. } x\in \mathbb {R}^{N}\bigr\} . $$

By Theorem 3.1 and Theorem 3.2, there exists a weak solution $u_{k} \in \mathscr{U}_{k}'$ of problem $(\bar{P}_{\lambda })$ such that $c'\leq u_{k}(x)\leq a_{k}'$ for almost every $x\in \mathbb {R}^{N}$ and

$$ J_{\lambda }(u_{k})=\inf_{\substack{u\in \mathscr{U}_{k}'}}J_{\lambda }(u). $$

Let $w_{k}=\hat{s}_{k} \mu$. Then (4.1), (4.13), and $\tilde{u}_{m_{k}}< b_{k}'$ show that $w_{k}\in \mathscr{U}_{k}'$. Besides, by Lebourg’s mean value theorem and conditions $(\bar{f_{1}})$, $(\bar{f_{2}})$, for $w_{k} <\rho _{\infty }$, there exist $\theta _{k} \in (0,1)$ and $v_{k}^{*}\in \partial F(v_{k})$ with $v_{k} =\theta _{k} w_{k} \in \mathscr{U}_{k}'$ such that

$$ \bigl\vert F(w_{k}) \bigr\vert \leq \bigl\vert v_{k}^{*} \bigr\vert \vert w_{k} \vert \leq C_{8}\bigl(1+ \vert v_{k} \vert ^{q-1}\bigr)\leq C _{9}, $$

where $C_{8}$ and $C_{9}$ are positive constants. Owing to (4.5), (4.10), (4.12), and the above inequality, we get

$$ \begin{aligned} J_{\lambda }(w_{k}) ={}& \frac{1}{2}{\hat{s}_{k}}^{2} \Vert \mu \Vert _{ \mathscr{X}_{0}}^{2}+\frac{\lambda }{2}{ \hat{s}_{k}}^{2} \Vert \mu \Vert _{2} ^{2} - \int _{\varOmega }\alpha (x)F(\hat{s}_{k} \mu )\,dx \\ \leq{}& \biggl( \frac{1}{2}+\frac{\lambda }{2 \lambda _{1}} \biggr){ \hat{s}_{k}}^{2} \Vert \mu \Vert _{\mathscr{X}_{0}}^{2} - \int _{D} \alpha (x)F(\hat{s}_{k})\,dx \\ &{} - \int _{({\varOmega \backslash D})\cap \{w_{k} >\rho _{\infty }\}}\alpha (x)F( \hat{s}_{k} \mu )\,dx - \int _{({\varOmega \backslash D})\cap \{w_{k} \leq \rho _{\infty }\}} \alpha (x)F(w_{k})\,dx \\ \leq{}& \biggl(\frac{1}{2}+\frac{\lambda }{2 \lambda _{1}} \biggr){ \hat{s}_{k}}^{2} \Vert \mu \Vert _{\mathscr{X}_{0}}^{2} -L _{\infty }{\hat{s}_{k}}^{2} \int _{D} \alpha (x)\,dx+l_{\infty }{\hat{s} _{k}}^{2} \Vert \alpha \Vert _{1} +C_{9} \Vert \alpha \Vert _{1} \\ ={}&{\hat{s}_{k}}^{2} \biggl[ \biggl( \frac{1}{2}+\frac{\lambda }{2 \lambda _{1}} \biggr) \Vert \mu \Vert _{ \mathscr{X}_{0}}^{2} +l_{\infty } \Vert \alpha \Vert _{1}-L_{\infty } \int _{D} \alpha (x)\,dx \biggr]+C_{9} \Vert \alpha \Vert _{1}. \end{aligned} $$

Thanks to $\lim_{k\to \infty } {\hat{s}_{k}}^{2}=+\infty$ and(4.11), we obtain that

$$ \lim_{k\to \infty }J_{\lambda }(u_{k})\leq \lim _{k\to \infty }J_{ \lambda }(w_{k})=-\infty . $$

Therefore, $\lim_{k\to \infty }J_{\lambda }(u_{k})=-\infty$. The proof is complete. □

Remark 4.2

Functions satisfying all the conditions in Theorem 4.2 exist. For instance, let $\varOmega \subset \mathbb{R}^{3}$, the function $F(x,u):\varOmega \times \mathbb{R}\to \mathbb{R}$ be defined by

$$ F(u)= \textstyle\begin{cases} u^{3}(\cos u+1), & \text{if } 0\le u< +\infty , \\ u, & \text{if } {-}\infty < u< 0. \end{cases} $$

It is easy to check that $F( u)$ satisfies $(\bar{f}_{1})$, $(\bar{f} _{2})$, $\liminf_{u\to \infty } \frac{F(u)}{u^{2}}=0$, and $\limsup_{u\to \infty } \frac{F(u)}{u^{2}}=+\infty$. Take $\hat{u}_{k}= 2k \pi +\frac{\pi }{2}$, then $\lim_{k\to \infty } \hat{u}_{k} =\infty$ and $F'(\hat{u}_{k}) =3-2k\pi - \frac{\pi }{2} <0$. Take $\bar{u}_{0}=-1\in (-\infty ,0)$, then $F'(\bar{u}_{0})=1>0$. That is, $F( u)$ satisfies $(F_{1}^{\infty })$ and $(F_{2}^{\infty })$.

Remark 4.3

In Theorem 4.1, we obtain the property of solutions on problem $(\bar{P}_{\lambda })$ which satisfy $\lim_{k\to \infty }\|u_{k}\|_{\mathscr{X}_{0}}=0$. In Theorem 4.2, if we suppose that $\alpha (x)\in L^{\infty }(\varOmega )$ instead of $\alpha (x)\in L^{2}( \varOmega )$ in condition $(A)$, we can also obtain

$$ \lim_{k\to \infty } \Vert u_{k} \Vert _{\mathscr{X}_{0}}=\infty . $$

Remark 4.4

(1) In [22], we obtained two multiplicity results of solutions for the following hemivariational inequality:

$$ (P_{\lambda , \mu }) \textstyle\begin{cases} -\mathcal {L}_{K}u\in \lambda \partial F(x,u)+\mu \partial G(x,u) & \text{in } \varOmega, \\ u=0 & \text{in }\mathbb {R}^{n} \backslash \varOmega , \end{cases} $$

according to the choice of the positive parameters λ, μ and appropriate assumptions on the nonsmooth potentials $F(x,u),G(x,u):\varOmega \times \mathbb {R}\rightarrow \mathbb {R}$, where $\mathcal {L}_{K}$ is the integrodifferential operator including the fractional Laplace operator ${-(-\Delta )^{s}}$ as its typical example. In fact, the first multiplicity result of [22, Theorem 3.1] was obtained by the coerciveness of the functional corresponding to problem $(P_{\lambda , \mu })$ and the nonsmooth mountain pass theorem, the second multiplicity result of [22, Theorem 3.2] was got by using an extended nonsmooth three-critical-points theorem due to Iannizzotto [26].

(2) In the present paper, we see that the functional $J_{\lambda }$ corresponding to problem $(P_{\lambda })$ or $(\bar{P}_{\lambda })$ may not be coercive. Instead of using the nonsmooth mountain pass theorem and the nonsmooth three-critical-points theorem, we first construct a special set U (defined by (3.1)) in $X_{0}$ and prove that $J_{\lambda }$ achieves its minimum on U at some $u_{0}\in U$ (see Theorem 3.1). In order to show that $u_{0}$ is actually a weak solution of problem $(P_{\lambda })$, we construct several sets, such as A, $A_{1}$, $A_{2}$ (defined by (3.5)) and $B_{1}(\varepsilon )$, $B_{2}(\varepsilon )$, $B_{3}(\varepsilon )$ (defined by (3.14)). By using the definitions of the fractional Laplace operator, $\lambda _{1}$ (the first eigenvalue of $(-\Delta )^{s}$), and these sets, we derive a lot of estimate equations and inequalities which are essential in the proof of our main results. We obtain the existence of a nontrivial solution of problem $(P_{\lambda })$ (see Theorem 3.2). Moreover, when F is autonomous, by employing the results obtained in Theorems 3.1 and 3.2, we obtain the existence of infinitely many solutions of this problem when the nonsmooth potentials F have suitable oscillating behavior in any neighborhood of the origin (respectively the infinity) and discuss the properties of the solutions (see Theorems 4.1 and 4.2). The methods of the proofs of results in the present paper are different from the ones obtained in [13,14,15,16, 22].