A new upper bound for the largest growth rate of linear Rayleigh–Taylor instability

Abstract

We investigate the effect of (interface) surface tensor on the linear Rayleigh–Taylor (RT) instability in stratified incompressible viscous fluids. The existence of linear RT instability solutions with largest growth rate Λ is proved under the instability condition (i.e., the surface tension coefficient ϑ is less than a threshold $\vartheta _{\mathrm{c}}$) by the modified variational method of PDEs. Moreover, we find a new upper bound for Λ. In particular, we directly observe from the upper bound that Λ decreasingly converges to zero as ϑ goes from zero to the threshold $\vartheta _{\mathrm{c}}$.

1 Introduction

Considering two completely plane-parallel layers of stratified (immiscible) fluids, the heavier one on top of the lighter one and both subject to the earth’s gravity, it is well known that such an equilibrium state is unstable to sustain small disturbances, and this unstable disturbance will grow and lead to a release of potential energy as the heavier fluid moves down under the gravitational force and the lighter one is displaced upwards. This phenomenon was first studied by Rayleigh [28] and then Taylor [29], and it is therefore called the Rayleigh–Taylor (RT) instability. In the last decades, this phenomenon has been extensively investigated from mathematical, physical, and numerical aspects, see [2, 10, 31] for instance. It has been also widely investigated how the RT instability evolves under the effects of other physical factors, such as elasticity [3, 13, 23, 25, 32], rotation [2, 6], (internal) surface tension [11, 16, 36], magnetic fields [18–22, 24, 33, 34], and so on. In this article, we are interested in the effect of surface tension on the linear RT instability in stratified incompressible viscous fluids. To conveniently introduce relevant mathematical progress and our main results, next we mathematically formulate our problem in detail.

1.1 Motion equations in Eulerian coordinates

Let us first recall a mathematical model, which describes the horizontally periodic motion of stratified incompressible viscous fluids in an infinity layer domain [23]:

$$ \textstyle\begin{cases} \rho _{\pm }(\partial _{t} v_{\pm }+v_{\pm }\cdot \nabla v_{\pm })+ \operatorname{div}\mathcal{S}_{\pm }=-g\rho _{\pm }e_{3}&\mbox{in } \Omega _{\pm }(t), \\ \operatorname{div}v_{\pm }=0 &\mbox{on }\Omega _{\pm }(t) , \\ d_{t}+v_{1} \partial _{1}d+v_{2} \partial _{2}d=v_{3} &\mbox{on } \Sigma (t), \\ [\!\![v_{\pm }]\!\!]=0,\qquad [\!\![\mathcal{S}_{\pm }\nu ]\!\!]=\vartheta \mathcal{C} \nu &\mbox{on }\Sigma (t) , \\ v_{\pm }=0 & \mbox{on }\Sigma _{\pm }, \\ v_{\pm }|_{t=0}=v^{0}_{\pm }& \mbox{in } \Omega _{\pm }(0), \\ d|_{t=0}=d^{0} & \mbox{on } \Sigma (0). \end{cases} $$

(1.1)

The momentum equations in (1.1)₁ describe the motion of both upper heavier and lower lighter viscous fluids driven by the gravitational field along the negative $x_{3}$-direction, which occupy the two time-dependent disjoint open subsets $\Omega _{+}(t)$ and $\Omega _{-}(t)$ at time t, respectively. Moreover, the fluids are incompressible due to (1.1)₂. The two fluids interact with each other by the motion equation of a free interface (1.1)₃ and the interfacial jump conditions in (1.1)₄. The first jump condition in (1.1)₄ represents that the velocity is continuous across the interface. The second jump in (1.1)₄ represents that the jump in the normal stress is proportional to the mean curvature of the surface multiplied by the normal to the surface. The non-slip boundary condition of the velocities on both upper and lower fixed flat boundaries are described by (1.1)₅. (1.1)₆ and (1.1)₇ represent the initial status of the two fluids. Next we further explain the notations in (1.1) in detail.

The subscripts + resp. − in the notations $f_{+}$ resp. $f_{-}$ mean that functions, parameters, or domains $f_{+}$ resp. $f_{-}$ are relevant to the upper resp. lower fluids. For each given $t>0$, $d:=d(x_{\mathrm{h}},t):\mathbb{T}\mapsto (-h_{-},h_{+})$ is a height function of a point at the interface of stratified fluids, where $h_{-}, h_{+}>0$, $\mathbb{T}:=\mathbb{T}_{1}\times \mathbb{T}_{2}$, $\mathbb{T}_{i}=2\pi L_{i}(\mathbb{R}/\mathbb{Z})$, and $2\pi L_{i}$ ($i=1, 2$) are the periodicity lengths. The domains $\Omega _{\pm }(t)$ and the interface $\Sigma (t)$ are defined as follows:

$$ \begin{aligned} &\Omega _{+}(t):=\bigl\{ (x_{\mathrm{h}},x_{3}) \mid x_{\mathrm{h}}:=(x_{1},x_{2}) \in \mathbb{T} , d(x_{\mathrm{h}},t)< x_{3}< h_{+} \bigr\} , \\ & \Omega _{-}(t):=\bigl\{ (x_{\mathrm{h}},x_{3}) \mid x_{\mathrm{h}}\in \mathbb{T} , -h_{-}< x_{3}< d(x_{\mathrm{h}},t) \bigr\} , \\ & \Sigma (t):=\bigl\{ (x_{\mathrm{h}},x_{3}) \mid x_{\mathrm{h}} \in \mathbb{T}, x_{3}:=d(x_{\mathrm{h}},t)\bigr\} . \end{aligned} $$

In addition, $\Sigma _{+}=\mathbb{T}\times \{h_{+}\}$, $\Sigma _{-}=\mathbb{T}\times \{-h_{-}\}$, and we call $\Omega :=\mathbb{T}\times (-h_{-},h_{+})$ the domain of stratified fluids.

For given $t>0$, $v_{\pm }(x,t): \Omega _{\pm }(t)\mapsto \mathbb{R}^{3}$ are the velocities of the two fluids, and $\mathcal{S}_{\pm }$ are the stress tensors enjoying the following expression:

$$ \mathcal{S}_{\pm }:=p_{\pm }I-\mu _{\pm }\mathbb{D}v_{\pm }\quad \mbox{with } \mathbb{D}v_{\pm }= \nabla v_{\pm }+\nabla v_{\pm }^{\mathrm{T}} . $$

(1.2)

In the above expression the superscript T means matrix transposition and I is the $3\times 3$ identity matrix. $\rho _{\pm }$ are the density constants, and the constants $\mu _{\pm }>0$ are the shear viscosity coefficients. g and ϑ represent the gravitational constant and the surface tension coefficient, resp. In addition, $e_{3}:=(0,0,1)^{\mathrm{T}}$.

For a function f defined on $\Omega (t)$, we define $[\!\![f_{\pm }]\!\!]:=f_{+}|_{\Sigma (t)} -f_{-}|_{\Sigma (t)}$, where $f_{\pm }|_{\Sigma (t)}$ are the traces of the quantities $f_{\pm }$ on $\Sigma (t)$. ν is the unit outer normal vector at boundary $\Sigma (t)$ of $\Omega _{-}(t)$, and $\mathcal{C}$ is the twice of the mean curvature of the internal surface $\Sigma (t)$, i.e.,

$$ \mathcal{C}:= \frac{\Delta _{\mathrm{h}}d+(\partial _{1} d)^{2}\partial _{2}^{2} d+(\partial _{2}d)^{2}\partial _{1}^{2}d-2\partial _{1}d\partial _{2}d\partial _{1}\partial _{2}d}{(1+(\partial _{1}d)^{2}+(\partial _{2}d)^{2})^{3/2}}. $$

Now we further introduce the indicator function $\chi _{\Omega _{\pm }(t)}$ and denote

$$ \begin{aligned} &\rho =\rho _{+}\chi _{\Omega _{+}(t)} + \rho _{-}\chi _{\Omega _{-}(t)},\qquad \mu =\mu _{+} \chi _{\Omega _{+}(t)} +\mu _{-} \chi _{\Omega _{-}(t)}, \\ & v=v_{+}\chi _{\Omega _{+}(t)} +v_{-}\chi _{\Omega _{-}(t)},\qquad p=p_{+}\chi _{\Omega _{+}(t)} +p_{-} \chi _{\Omega _{-}(t)}, \\ & v^{0}=v_{+}^{0}\chi _{\Omega _{+}(0)} +v_{-}^{0}\chi _{\Omega _{-}(0)} , \qquad \mathcal{S} :=p I-\mu \mathbb{D} v, \end{aligned} $$

then model (1.1) can be rewritten as follows:

$$ \textstyle\begin{cases} \rho ( v_{t}+v\cdot \nabla v)+\operatorname{div}\mathcal{S} =-g\rho e_{3} &\mbox{in } \Omega (t), \\ \operatorname{div} v=0&\mbox{in } \Omega (t), \\ d_{t}+v_{1} \partial _{1}d+v_{2} \partial _{2}d=v_{3} &\mbox{on } \Sigma (t), \\ [\!\![v ]\!\!]=0,\qquad [\!\![\mathcal{S} \nu ]\!\!]= \vartheta \mathcal{C} \nu &\mbox{on }\Sigma (t) , \\ v =0 &\mbox{on }\Sigma _{-}^{+}, \\ v |_{t=0}= v^{0} &\mbox{in } \Omega (0), \\ d|_{t=0}=d^{0} &\mbox{on } \Sigma (0), \end{cases} $$

(1.3)

where we have defined that $\Omega (t):=\Omega _{+}(t)\cup \Omega _{-}(t)$, $\Sigma _{-}^{+}:=\Sigma _{-}\cup \Sigma _{+}$ and omitted the subscript ± in $[\!\![f_{\pm }]\!\!]$ for simplicity.

1.2 Reformulation in Lagrangian coordinates

Next we adopt the transformation of Lagrangian coordinates so that the interface and the domains stay fixed in time.

We define that

$$\begin{aligned} &\Omega _{+}:=\bigl\{ (y_{\mathrm{h}},y_{3})\in \mathbb{R}^{3} \mid y_{ \mathrm{h}} \in \mathbb{T}, 0< y_{3}< h_{+}\bigr\} , \\ &\Omega _{-}:=\bigl\{ (y_{\mathrm{h}},y_{3})\in \mathbb{R}^{3} \mid y_{ \mathrm{h}} \in \mathbb{T}, -h_{-}< y_{3}< 0\bigr\} , \end{aligned}$$

and assume that there exist invertible mappings

$$ \zeta _{\pm }^{0}:\Omega _{\pm } \rightarrow \Omega _{\pm }(0) $$

such that

$$ \Sigma (0)=\zeta _{\pm }^{0}(\Sigma ),\qquad \Sigma _{\pm }=\zeta _{\pm }^{0}(\Sigma _{\pm })\quad \mbox{and}\quad \det \nabla \zeta _{\pm }^{0}= 1. $$

(1.4)

We further define $\zeta ^{0}:=\zeta _{+}^{0}\chi _{\Omega _{+}(0)}+\zeta _{-}^{0}\chi _{ \Omega _{-}(0)}$ and the flow map ζ as the solution to

$$ \textstyle\begin{cases} \partial _{t} \zeta (y,t)=v(\zeta (y,t),t)&\mbox{in }\Omega _{-}^{+}, \\ \zeta (y,0)=\zeta ^{0}(y)&\mbox{in }\Omega _{-}^{+}, \end{cases} $$

(1.5)

where $\Omega _{-}^{+}:=\Omega _{+}\cup \Omega _{-}$. We denote the Eulerian coordinates by $(x,t)$ with $x=\zeta (y,t)$, whereas the fixed $(y,t)\in \Omega _{-}^{+}\times \mathbb{R}^{+}$ stand for the Lagrangian coordinates.

In order to switch back and forth from Lagrangian to Eulerian coordinates, we assume that $\zeta _{\pm }(\cdot ,t)$ are invertible and $\Omega _{\pm }(t)=\zeta _{\pm }(\Omega _{\pm },t)$, and since $v_{\pm }$ and $\zeta _{\pm }^{0}$ are all continuous across Σ, we have $\Sigma (t)=\zeta _{\pm }(\Sigma ,t)$. In view of the non-slip boundary condition $v|_{\Sigma _{-}^{+}}=0$, we have

$$ y=\zeta (y, t)\quad \mbox{on }\Sigma _{-}^{+}. $$

Now we set the Lagrangian unknowns

$$ (u,\sigma ) (y,t)=(v,p+g\rho x_{3}) \bigl(\zeta (y,t),t\bigr) \quad \mbox{for } (y,t) \in \Omega _{-}^{+} \times (0,\infty ), $$

then problem (1.3) can be rewritten as an initial-boundary value problem with an interface for $(\zeta ,u)$ in Lagrangian coordinates:

$$ \textstyle\begin{cases} \zeta _{t}=u &\mbox{in } \Omega _{-}^{+}, \\ \rho u_{t}+\nabla _{\mathcal{A}}\sigma -\mu \Delta _{\mathcal{A}}u =0& \mbox{in } \Omega _{-}^{+}, \\ \operatorname{div}_{\mathcal{A}}u=0 &\mbox{ in } \Omega _{-}^{+}, \\ [\!\![\zeta ]\!\!]= [\!\![u ]\!\!]=0 , \qquad [\!\![((\sigma -g\rho \zeta _{3}) I -\mu \mathbb{D}_{\mathcal{A}}u) \vec{n} ]\!\!]= \vartheta \mathcal{H}\vec{n} &\mbox{on }\Sigma , \\ (\zeta ,u)=(y,0) &\mbox{on }\Sigma _{-}^{+}, \\ (\zeta , u)|_{t=0}=(\zeta ^{0}, u^{0}) &\mbox{in } \Omega _{-}^{+}, \end{cases} $$

(1.6)

where we have defined that

$$ \begin{aligned} & \Sigma :=\mathbb{T} \times \{0\}, \qquad \vec{n}:= { \mathcal{A}e_{3}}/{ \vert \mathcal{A}e_{3} \vert },\qquad \mathbb{D}_{\mathcal{A}} u= \nabla _{\mathcal{A}} u + \nabla _{\mathcal{A}} u^{\mathrm{T}} , \\ &\mathcal{H}:=\bigl( \vert \partial _{1} \zeta \vert ^{2}\partial _{2}^{2}\zeta -2( \partial _{1}\zeta \cdot \partial _{2}\zeta )\partial _{1}\partial _{2} \zeta + \vert \partial _{2}\zeta \vert ^{2}\partial _{1}^{2} \zeta \bigr)\cdot \vec{n}/\bigl( \vert \partial _{1}\zeta \vert ^{2} \vert \partial _{2}\zeta \vert ^{2}- \vert \partial _{1}\zeta \cdot \partial _{2}\zeta \vert ^{2}\bigr). \end{aligned} $$

We shall introduce the notations involving $\mathcal{A}$. The matrix $\mathcal{A}:=(\mathcal{A}_{ij})_{3\times 3}$ is defined via

$$ \mathcal{A}^{\mathrm{T}}=(\nabla \zeta )^{-1}:=(\partial _{j} \zeta _{i})^{-1}_{3\times 3}, $$

where $\partial _{j}$ denotes the partial derivative with respect to the jth components of variables y. $\tilde{\mathcal{A}}:= \mathcal{A}-I$, and I is the $3\times 3$ identity matrix. The differential operator $\nabla _{\mathcal{A}}$ is defined by

$$ \nabla _{\mathcal{A}}w:=(\nabla _{\mathcal{A}}w_{1},\nabla _{ \mathcal{A}}w_{2},\nabla _{\mathcal{A}}w_{3})^{\mathrm{T}} \quad \mbox{and}\quad \nabla _{\mathcal{A}}w_{i}:=(\mathcal{A}_{1k} \partial _{k}w_{i}, \mathcal{A}_{2k}\partial _{k}w_{i},\mathcal{A}_{3k}\partial _{k}w_{i})^{\mathrm{T}} $$

for a vector function $w:=(w_{1},w_{2},w_{3})$, and the differential operator $\operatorname{div}_{\mathcal{A}}$ is defined by

$$ \operatorname{div}_{\mathcal{A}}\bigl(f^{1},f^{2},f^{3} \bigr):=\bigl(\operatorname{div}_{ \mathcal{A}}f^{1}, \operatorname{div}_{\mathcal{A}}f^{2}, \operatorname{div}_{ \mathcal{A}}f^{3} \bigr)^{\mathrm{T}} \quad \mbox{and}\quad \operatorname{div}_{ \mathcal{A}}f^{i}:= \mathcal{A}_{lk}\partial _{k} f_{l}^{i} $$

for a vector function $f^{i}:=(f_{1}^{i},f_{2}^{i},f_{3}^{i})^{\mathrm{T}}$. It should be noted that we have used the Einstein convention of summation over repeated indices. In addition, we define $\Delta _{\mathcal{A}}X:=\operatorname{div}_{\mathcal{A}}\nabla _{ \mathcal{A}}X$.

1.3 Linearized motion

We choose a constant $\bar{d}\in (-h_{-},h_{+})$. Without loss of generality, we assume that $\bar{d}=0$. Then we consider an RT equilibrium state

$$ \textstyle\begin{cases} \nabla \bar{p}_{\pm }=-\bar{\rho }_{\pm }g e_{3} &\mbox{in } \Omega _{\pm }, \\ [\!\![\bar{p}_{\pm }]\!\!]e_{3}=0 &\mbox{on }\Sigma , \end{cases} $$

(1.7)

where ρ satisfies the RT (jump) condition

$$ [\!\![{\rho } ]\!\!]>0\quad \mbox{on }\Sigma .$$

(1.8)

Let $\bar{p}:=\bar{p}_{+}\chi _{\Omega _{+}} +\bar{p}_{-}\chi _{\Omega _{-}}$. Then $(v,p)=(0,\bar{p})$ with $d=0$ is an RT equilibria solution of (1.3).

Denote the perturbation in Lagrangian coordinates

$$ \eta :=\zeta -y, \qquad u=u-0\quad \mbox{and}\quad q=\sigma -\bigl(\bar{p}(\zeta _{3})+g \rho \zeta _{3}\bigr), $$

then subtracting (1.7) from (1.6) yields the perturbation RT problem in Lagrangian coordinates:

$$ \textstyle\begin{cases} \eta _{t}= u &\mbox{in }\Omega _{-}^{+}, \\ {\rho } u_{t}+ \nabla q-\mu \Delta u =\mathcal{N}_{1}& \mbox{in } \Omega _{-}^{+}, \\ \operatorname{div}u=\mathcal{N}_{2}& \mbox{in }\Omega _{-}^{+}, \\ [\!\![\eta ]\!\!]= [\!\![u ]\!\!]=0, \qquad [\!\![((q-g\rho \eta _{3})I-\mu \mathbb{D}u) e_{3} ]\!\!]= \vartheta \Delta _{\mathrm{h}}\eta _{3}e_{3}+\mathcal{N}_{3} &\mbox{on }\Sigma , \\ (\eta ,u)= 0 &\mbox{on }\Sigma _{-}^{+}, \\ (\eta , u)|_{t=0}= (\eta ^{0}, u^{0}) &\mbox{in } \Omega _{-}^{+},\end{cases} $$

(1.9)

where $\Delta _{\mathrm{h}}:=\partial _{1}^{2}+\partial _{2}^{2}$ and the nonlinear terms $\mathcal{N}_{1}$–$\mathcal{N}_{3}$ are defined as follows:

$$\begin{aligned} & \mathcal{N}_{1}=\mu \operatorname{div}_{\tilde{\mathcal{A}}}\nabla _{ \mathcal{A}} u+\operatorname{div}\nabla _{\tilde{\mathcal{A}}} u-\nabla _{ \tilde{\mathcal{A}}}q,\qquad \mathcal{N}_{2}=- \operatorname{div}_{ \tilde{\mathcal{A}}} u, \\ &\mathcal{N}_{3}= \mu \mathbb{D}_{\tilde{\mathcal{A}}} u+\vartheta \mathcal{H}\vec{n} -\vartheta \Delta _{\mathrm{h}}\eta _{3}e_{3} . \end{aligned}$$

Omitting the nonlinear terms in (1.9), we get a linearized RT problem:

$$ \textstyle\begin{cases} \eta _{t}= u &\mbox{in }\Omega _{-}^{+}, \\ \rho u_{t}+ \nabla q-\mu \Delta u =0 & \mbox{in }\Omega _{-}^{+}, \\ \operatorname {div}u =0 &\mbox{in }\Omega _{-}^{+}, \\ [\!\![\eta ]\!\!]= [\!\![u ]\!\!]=0&\mbox{on } \Sigma , \\ [\!\![((q-g\rho \eta _{3})I-\mu \mathbb{D}u) e_{3} ]\!\!]= \vartheta \Delta _{\mathrm{h}}\eta _{3}e_{3} &\mbox{on }\Sigma , \\ (\eta ,u)= 0 &\mbox{ on }\Sigma _{-}^{+}, \\ (\eta ,u)|_{t=0}= (\eta ^{0},u^{0}) &\mbox{in } \Omega _{-}^{+}.\end{cases} $$

(1.10)

Of course, the motion equations of stratified viscous fluids in a linear stage can be approximatively described by (1.10).

The inhibition of RT instability by surface tension was first analyzed by Bellman and Phennington [1] based on a linearized two-dimensional (2D) motion equations of stratified incompressible inviscid fluids defined on the domain $2\pi L_{1}\mathbb{T}_{1}\times (-h_{-},h_{+})$ (i.e., $\mu =0$ in the corresponding 2D case of (1.10)) in 1953. More precisely, they proved that the linear 2D stratified incompressible inviscid fluids are stable, resp. unstable for $\vartheta >g[\!\![\rho ]\!\!]L_{1}^{2}$, resp. $\vartheta < g[\!\![\rho ]\!\!]L_{1}^{2}$. The value $g[\!\![\rho ]\!\!]L_{1}^{2}$ is a threshold of surface tension coefficient for linear stability and linear instability. Similar result was also found in the 3D viscous case; for example, Guo and Tice proved that $\vartheta _{\mathrm{c}}:={ g[\!\![\rho ]\!\!]\max \{L_{1}^{2},L_{2}^{2} \} }$ is a threshold of surface tension coefficient for stability and instability in the linearized 3D stratified compressible viscous fluids defined on Ω [11]. Next we further review the mathematical progress for the nonlinear case.

Prüess and Simonett first proved that the RT equilibria solution of the stratified incompressible viscid fluids defined on the domain $\mathbb{R}^{3}$ is unstable based on a Henry instability method [27]. Later Wang, Tice, and Kim verified that the RT equilibria solution of stratified incompressible viscous fluids defined on Ω is stable, resp. unstable for $\vartheta >\vartheta _{\mathbb{T}}$, resp. $\vartheta \in [0, \vartheta _{\mathbb{T}})$ [35, 36]. Jang, Wang, and Tice further obtained the same results of stability and instability in the corresponding compressible case [15, 16]. Recently, Wilke also proved there exists a threshold $\vartheta _{\mathrm{c}}$ for the stability and instability of stratified viscous fluids (with heavier fluid over lighter fluid) defined on a cylindrical domain with finite height [37]. Finally, we mention that the results of nonlinear RT instability in inhomogeneous fluid (without interface) were obtained based on the classical bootstrap instability method, see [14], resp. [17] for inviscid, resp. viscous cases.

2 Main result

In this paper, we investigate the effect of surface tension on the linear RT instability by the linearized motion (1.10). Wang and Tice used a discrete Fourier transformation and the modified variational method of ODEs to prove the existence of growth solutions with a largest growth rate $\Lambda _{\vartheta }$ for (1.10) with $h_{+}=1$ under the condition $\vartheta \in (0,\vartheta _{c})$ [35]. Moreover, they provided an upper bound for $\Lambda _{\vartheta }$:

$$ \Lambda _{\vartheta }\leqslant {h_{-}g [\!\![\rho ]\!\!]}/{4 \mu _{-}}\quad \mbox{for }h_{+}=1. $$

In this paper, we exploit the modified variational method of PDEs and the existence theory of stratified (steady) Stokes problem to prove the existence of growth solutions with a largest growth rate $\Lambda _{\vartheta }$ for (1.10) under the instability condition $\vartheta \in [0,\vartheta _{\mathbb{T}})$. Moreover, we find a new upper bound:

$$ \Lambda _{\vartheta }\leqslant m:=\min \biggl\{ \frac{ (\vartheta _{\mathrm{c}}-\vartheta ) }{4\max \{L_{1}^{2},L_{2}^{2}\}} \min \biggl\{ \frac{h_{+}}{\mu _{+}},\frac{{h}_{-}}{\mu _{-}} \biggr\} , \biggl( \frac{(g[\!\![\rho ]\!\!](\vartheta _{\mathrm{c}}-\vartheta ))^{2} }{4\vartheta _{\mathrm{c}}^{2} \max \{\rho _{+}\mu _{+} , \rho _{-}\mu _{-}\}} \biggr)^{\frac{1}{3}} \biggr\} . $$

(2.1)

It is easy to see that

$$ m \leqslant {h_{-}g [\!\![\rho ]\!\!]}/{4\mu _{-}}. $$

Therefore our upper bound is more precise than Wang and Tice’s one. Moreover, we directly see from (2.1) that

$$ \Lambda _{\vartheta }\to 0\quad \mbox{as }\vartheta \to \vartheta _{ \mathrm{c}}. $$

(2.2)

We mention that, in the classical Rayleigh–Taylor (RT) experiments [8, 12], the phenomenon of that the instability growth is limited by surface tension during the linear stage, where the growth is exponential in time, has been shown. Obviously, the convergence behavior (2.2) mathematically verifies the phenomenon.

Before stating our main results in detail, we introduce some simplified notations used throughout this article.

• (1) Basic notations: $I_{T}:=(0,T)$. $\mathbb{R}^{+}:=(0,\infty )$, $\mathbb{R}^{+}_{0}:=[0,\infty )$. The jth difference quotient of size h is $D_{j}^{h} w:=(w(y+h e_{j})-w(y))/h$ for $j=1$ and 2, and $D^{h}_{\mathrm{h}}w:=(D_{1}^{h}w_{1},D_{2}^{h}w_{2})$, where $|h|\in (0,1)$. ℜf, resp. ℑf denote the real, resp. imaginary parts of the complex function f. $\nabla _{\mathrm{h}}^{k} f$ denotes a $(k+1)\times (k+1)$ matrix $(\partial _{1}^{i}\partial _{2}^{j} f)_{ij}$ for $k\geqslant 0$. $a\lesssim b$ means that $a\leqslant cb$ for some constant $c>0$, where the positive constant c may depend on the domain Ω and known parameters such as $\rho _{\pm }$, $\mu _{\pm }$, g, and ϑ, and may vary from line to line.

• (2) Simplified notations of Sobolev spaces:

  $$\begin{aligned} &L^{p}:=L^{p} \bigl(\Omega _{-}^{+} \bigr)=W^{0,p}\bigl(\Omega _{-}^{+}\bigr),\qquad W^{i,2}:=W^{i,2}\bigl(\Omega _{-}^{+} \bigr), \qquad {H}^{i}:=W^{i,2} , \\ & H^{\infty }:=\bigcap_{j=1}^{\infty }H^{j},\qquad \underline{H}^{i}:= \biggl\{ w \in {H}^{i} \Bigm| \int _{\Omega _{-}^{+}}w\,\mathrm{d}y=0 \biggr\} , \\ &H^{1}_{\sigma }:=\bigl\{ w\in {H}^{1}(\Omega ) \mid w \vert _{\Sigma _{-}^{+}}=0 \mbox{ in the sense of trace}, \operatorname{div}w=0\bigr\} , \\ & H_{\sigma }^{i}:=H_{\sigma }^{1} \cap H^{i}, \qquad H_{\sigma ,\Sigma }^{1}:= \bigl\{ w\in {H}_{\sigma }^{1} \mid w_{3} \vert _{\Sigma }\in H^{1}( \mathbb{T}) \bigr\} , \\ &H^{1}_{\sigma ,3}:= \bigl\{ w\in H^{1}_{\sigma ,\vartheta } \mid w_{3}\neq 0 \mbox{ on }\Sigma \bigr\} , \qquad H_{\sigma , \vartheta }^{1}= \textstyle\begin{cases} H_{\sigma ,\Sigma }^{1} & \mbox{if }\vartheta \neq 0, \\ H_{\sigma }^{1} &\mbox{if }\vartheta = 0, \end{cases}\displaystyle \\ & \mathcal{A}: = \bigl\{ w\in {H}_{\sigma ,\vartheta }^{1} \mid \Vert \sqrt{ {\rho }}w \Vert ^{2}_{L^{2}}=1 \bigr\} ,\qquad H^{-1}_{\sigma }= \mbox{the dual space of }H^{1}_{\sigma }, \end{aligned}$$

  where $1< p\leqslant \infty$ and $i\geqslant 0$ is an integer. Sometimes, we denote $\mathcal{A}$ by $\mathcal{A}_{\vartheta }$ to emphasize the dependence of ϑ. In addition, to prove the existence of unstable classical solutions of the linearized RT problem, we introduce a function space

  $$ H_{\sigma , \vartheta }^{1,k}:= \textstyle\begin{cases} \{w\in H_{\sigma ,\vartheta }^{1} \mid\nabla _{\mathrm{h}}^{j} w \in H^{1}\mbox{ and }w_{3}|_{\Sigma }\in H^{k+1}(\mathbb{T}) \mbox{ for } j\leqslant k \} & \mbox{if }\vartheta \neq 0, \\ \{w\in H_{\sigma }^{1} \mid \nabla _{\mathrm{h}}^{j}w \in H^{1} \mbox{ for } j\leqslant k \} &\mbox{if }\vartheta = 0, \end{cases} $$

  where $k\geqslant 0$ is an integer. It should be noted that $H_{\sigma ,\vartheta }^{1,0}=H_{\sigma , \vartheta }^{1}$.

• (3) Simplified norms: $\|\cdot \|_{i} :=\|\cdot \|_{W^{i,2}}$, $|\cdot |_{s} := \|\cdot |_{\Sigma } \|_{H^{s}(\mathbb{T})}$, where s is a real number and i is a nonnegative integer.

• (4) Functionals: $\mathcal{E}(w):= \vartheta |\nabla _{\mathrm{h}}w_{3}|_{0}^{2}-g [\!\![\rho ]\!\!]|w_{3}|_{0}^{2}$ and $F(w,s):= -(\mathcal{E}(w) +s\|\sqrt{\mu } \mathbb{D}w\|^{2}_{0}/2)$.

In addition, we give the definition of the largest growth rate of RT instability in the linearized RT problem.

Definition 2.1

We call $\Lambda >0$ the largest growth rate of RT instability in the linearized RT problem (1.10) if it satisfies the following two conditions:

• (1) For any strong solution $(\eta , u)\in C^{0}([0,T),H^{3}\cap H^{2}_{\sigma })\cap L^{2}(I_{T},H^{3} \cap H^{3}_{\sigma })$ of the linearized RT problem with q enjoying the regularity $q\in C^{0}([0,T),H^{1})\cap L^{2}(I_{T},H^{2})$, we have, for any $t\in [ 0,T)$,

  $$ \bigl\Vert (\eta ,u) \bigr\Vert _{1}^{2}+ \Vert u_{t} \Vert ^{2}_{0 }+ \int _{0}^{t} \bigl\Vert u(s) \bigr\Vert ^{2}_{1} \,\mathrm{d}s\lesssim e^{2\Lambda t}\bigl( \bigl\Vert \eta ^{0} \bigr\Vert _{3}^{2}+ \bigl\Vert u^{0} \bigr\Vert _{2}^{2}\bigr). $$

  (2.3)

• (2) There exists a strong solution $(\eta , u)$ of the linearized RT problem in the form

  $$ (\eta , u):=e^{\Lambda t}( \tilde{\eta }, \tilde{u} ), $$

  where $( \tilde{\eta }, \tilde{u} )\in H^{2}$.

Now we state the first result on the existence of the largest growth rate in the linearized RT problem.

Theorem 2.1

Let $g>0$, $\rho >0$, and $\mu >0$ be given. Then, for any given

$$ \vartheta \in \bigl[0, \vartheta _{\mathrm{c}}:=g[\!\![\rho ]\!\!]{\max \bigl\{ L_{1}^{2},L_{2}^{2} \bigr\} }\bigr), $$

(2.4)

there is an unstable solution

$$ ( \eta , u,q):=e^{\Lambda t}( w/\Lambda ,w,\beta ) $$

to the linearized RT problem (1.10), where $(w,\beta )\in H^{\infty }$ solves the boundary value problem

$$ \textstyle\begin{cases} \Lambda ^{2}\rho w= \Lambda (\mu \Delta w-\nabla \beta ) &\textit{in } \Omega _{-}^{+}, \\ \operatorname{div}w=0 & \textit{in }\Omega _{-}^{+}, \\ [\!\![w ]\!\!]= 0, \qquad [\!\![\Lambda (\beta I-\mu \mathbb{D}w)e_{3} - g \rho w_{3} e_{3} ]\!\!]= \vartheta \Delta _{\mathrm{h}}w_{3} e_{3} &\textit{on } \Sigma , \\ w = {0}& \textit{on }\Sigma _{-}^{+} \end{cases} $$

(2.5)

with the largest growth rate $\Lambda >0$ satisfying

$$ \Lambda ^{2}= \sup_{\varpi \in \mathcal{A} }F(\varpi , \Lambda )=F(w, \Lambda ) . $$

(2.6)

Moreover,

$$ w_{3}\neq 0, \qquad \partial _{3} w_{3}\neq 0,\qquad \operatorname{div}_{ \mathrm{h}}w_{\mathrm{h}}\neq 0\quad \textit{in }\Omega _{-}^{+}, \qquad \vert w_{3} \vert \neq 0\quad \textit{on }\Sigma . $$

(2.7)

Next we briefly introduce how to prove Theorem 2.1 by the modified variational method of PDEs and the regularity theory of stratified (steady) Stokes problem. The detailed proof is given in Sect. 4.

We assume a growing mode ansatz to the linearized problem

$$ \eta (x,t)=\tilde{\eta }( x)e^{\Lambda t},\qquad u(x,t)=w( x)e^{\Lambda t},\qquad q( x,t)=\beta ( x)e^{\Lambda t} $$

for some $\Lambda >0$. Substituting this ansatz into the linearized RT problem (1.10), we get a spectrum problem

$$ \textstyle\begin{cases} \Lambda \tilde{\eta }=w &\mbox{in }\Omega _{-}^{+}, \\ \Lambda \rho w= \mu \Delta w-\nabla \beta &\mbox{in }\Omega _{-}^{+}, \\ \operatorname{div}w=0 & \mbox{in }\Omega _{-}^{+}, \\ [\!\![w ]\!\!]= 0,\qquad [\!\![(\beta I-\mu \mathbb{D}w)e_{3} - g \rho \tilde{\eta }_{3} e_{3} ]\!\!]= \vartheta \Delta _{\mathrm{h}}\tilde{\eta }_{3} e_{3} & \mbox{on }\Sigma , \\ w = {0}& \mbox{on }\Sigma _{-}^{+}, \end{cases} $$

and then eliminating η̃ by using the first equation, we arrive at the boundary value problem (2.5) for w and β. Obviously, the linearized RT problem is unstable if there exists a solution $(w,\beta )$ to the boundary value problem (2.5) with $\Lambda >0$.

To look for the solution, we use a modified variational method of PDEs and thus modify (2.5) as follows:

$$ \textstyle\begin{cases} \alpha \rho w= s(\mu \Delta w-\nabla \beta ) &\mbox{in }\Omega _{-}^{+}, \\ \operatorname{div}w=0 & \mbox{in }\Omega _{-}^{+}, \\ [\!\![w ]\!\!]= 0,\qquad [\!\![s (\beta I-\mu \mathbb{D}w)e_{3} - g \rho w_{3} e_{3} ]\!\!]= \vartheta \Delta _{\mathrm{h}}w_{3} e_{3} &\mbox{on } \Sigma , \\ w = {0}& \mbox{on }\Sigma _{-}^{+}, \end{cases} $$

(2.8)

where $s>0$ is a parameter. To emphasize the dependence of s upon α and ϑ, we write $\alpha (s,\vartheta )=\alpha$.

Note that the modified problem (2.8) enjoys the following variational identity:

$$ \alpha (s,\vartheta ) \Vert \sqrt{ {\rho }}w \Vert _{0}^{2}= {F}(w,s). $$

Thus, by a standard variational approach, there exists a maximizer $w\in \mathcal{A}$ of the functional F defined on $\mathcal{A}$; moreover, w is just a weak solution to (2.8) with α defined by the relation

$$ \alpha ( s ,\vartheta ) =\sup_{w\in \mathcal{A} } {F}(w,s )\in \mathbb{R},$$

(2.9)

see Proposition 4.1. Then we further use the method of difference quotients and the existence theory of the stratified (steady) Stokes problem to improve the regularity of the weak solution, and thus prove that $(w,\beta )\in H^{\infty }$ is a classical solution to the boundary value problem (2.8), see Proposition 4.2.

In view of the definition of $\alpha (s ,\vartheta )$ and the instability condition (2.4), we can infer that, for given ϑ, the function $\alpha (s,\cdot )$ on the variable s enjoys some good properties (see Proposition 4.3), which imply that there exists Λ satisfying the fixed point relation

$$ \Lambda =\sqrt{\alpha (\Lambda ,\cdot )} \in (0, \mathfrak{S}_{\vartheta }). $$

(2.10)

Then we obtain a nontrivial solution $(w,\beta )\in H^{\infty }$ to (2.5) with Λ defined by (2.10), and therefore the linear instability follows. Moreover, Λ is the largest growth rate of RT instability in the linearized RT problem (see Proposition 4.4), and thus we get Theorem 2.1.

Next we turn to introduce the second result on the properties of the largest growth rate constructed by (2.10).

Theorem 2.2

The largest growth rate $\Lambda _{\vartheta }:=\Lambda$ in Theorem 2.1enjoys the estimate (2.1). Moreover,

$$ \Lambda _{\vartheta}\textit{ strictly decreases and is continuous with respect to } \vartheta \in [0,\vartheta _{\mathrm{c}}). $$

(2.11)

The proof of Theorem 2.2 is presented in Sect. 5. Here we briefly mention the idea of the proof. We find that, for fixed s, $\alpha (\cdot ,\vartheta )$ defined by (2.9) strictly decreases and is continuous with respect to ϑ (see Proposition 5.1). Thus, by the fixed point relation (2.10) and some analysis based on the definition of continuity, we can show that $\Lambda _{\vartheta }:=\Lambda$ also inherits the monotonicity and continuity of $\alpha (\cdot ,\vartheta )$. Finally, we derive (2.1) from (2.6) by some estimate techniques.

3 Preliminary

This section is devoted to introducing some preliminary lemmas, which will be used in the next two sections.

Lemma 3.1

Difference quotients and weak derivatives: Let D be Ω or $\mathbb{T}$.

• (1) Suppose $1\leqslant p<\infty$ and $w\in W^{1,p}(D)$. Then $\|D^{h}_{\mathrm{h}} w\|_{L^{p}(D) }\lesssim \|\nabla _{\mathrm{h}}w \|_{L^{p}(D)}$.

• (2) Assume $1< p<\infty$, $w\in L^{p}(D)$, and there exists a constant c such that $\|D^{h}_{\mathrm{h}} w\|_{L^{p}(D)}\leqslant c$. Then $\nabla _{\mathrm{h}} w\in L^{p}(D)$ satisfies $\|\nabla _{\mathrm{h}}w\|_{L^{p}(D)}\leqslant c$ and $D^{-h_{k}}_{\mathrm{h}} w \rightharpoonup \nabla _{\mathrm{h}}w$ in $L^{p}(D)$ for some subsequence $-h_{k}\to 0$.

Proof

Following the argument of [7, Theorem 3] and using the periodicity of w, we can easily get the desired conclusions. □

Lemma 3.2

Existence theory of a stratified (steady) Stokes problem (see [36, Theorem 3.1]): Let $k\geqslant 0$, $f^{\mathrm{S},1}\in H^{k}$, and $f^{\mathrm{S},2}\in H^{k+1/2}$, then there exists a unique solution $(u,q)\in H^{k+2}\times \underline{H}^{k+1}$ satisfying

$$ \textstyle\begin{cases} \nabla q-\mu \Delta u =f^{\mathrm{S},1}&\textit{in } \Omega , \\ [\!\![u ]\!\!]=0, \qquad [\!\![(q I-\mathbb{D}u )e_{3} ]\!\!]=f^{\mathrm{S},2} &\textit{on }\Sigma , \\ u=0 &\textit{on }\Sigma _{-}^{+}. \end{cases} $$

(3.1)

Moreover,

$$ \Vert u \Vert _{\mathrm{S},k} \lesssim \bigl\Vert f^{\mathrm{S},1} \bigr\Vert _{k} + \bigl\vert f^{ \mathrm{S},2} \bigr\vert _{k+1/2} . $$

(3.2)

Lemma 3.3

Equivalent form of instability condition: the instability condition (2.4) is equivalent to the following integral version of instability condition:

$$ g[\!\![\rho ]\!\!]\vert w_{3} \vert _{0}^{2}-\vartheta \vert \nabla _{ \mathrm{h}}w_{3} \vert _{0}^{2}>0 \quad \textit{for some } w\in H^{1}_{\sigma ,3}. $$

(3.3)

Proof

The conclusion in Lemma 3.3 is obvious if we have the assertion

$$ a:=\sup_{w\in H_{\sigma ,3}^{1} } \frac{ \vert w_{3} \vert _{0}^{2}}{ \vert \nabla _{\mathrm{h}}w_{3} \vert _{0}^{2}}= {\max \bigl\{ L_{1}^{2},L_{2}^{2}\bigr\} } \quad \mbox{for }\vartheta \neq 0. $$

(3.4)

Next we verify (3.4) by two steps. Without loss of generality, we assume that $L_{1}^{2}={\max \{L_{1}^{2},L_{2}^{2}\}}$.

(1) We first prove that $a\geqslant L_{1}^{2}$. We choose a nonzero function $\psi \in H_{0}^{2}(-h_{-},h_{+})$ such that $\psi (0)\neq 0$. We denote

$$ {w}=\bigl(\psi ' (y_{3})\cos \bigl( L^{-1}_{1}y_{1} \bigr),0, L^{-1}_{1}\psi (y_{3}) \sin \bigl(L^{-1}_{1} y_{1} \bigr)\bigr), $$

then ${w} \in H^{1}_{\sigma ,\vartheta }$ and

$$ \frac{ \vert {w}_{3} \vert ^{2}_{0}}{ \vert \nabla _{\mathrm{h}} { w}_{3} \vert ^{2}_{0}} = \frac{ \int _{0}^{2\pi L_{1}} \sin ^{2} ( L^{-1}_{1} y_{1} )\,\mathrm{d}y_{1} }{L^{-2}_{1} \int _{0}^{2\pi L_{1}} \cos ^{2}( L^{-1}_{1} y_{1}) \,\mathrm{d}y_{1} }=L_{1}^{2}, $$

which yields $a\geqslant L_{1}^{2}$.

(2) We turn to the proof of $a\leqslant L_{1}^{2}$. It should be noted that

$$ \vert \nabla _{\mathrm{h}}w_{3} \vert _{0}^{2}=0\quad \mbox{if and only if} \quad w_{3}=0\quad \mbox{for any given }w\in H_{\sigma ,\vartheta }^{1}. $$

(3.5)

In fact, let $w\in H_{\sigma ,3}^{1}$. Since $\operatorname{div}w=0$, we have

$$ - \int _{\Sigma }w_{3}\,\mathrm{d}y_{\mathrm{h}}= \int _{\mathbb{T} \times (0,h_{+})}\operatorname{div}w\,\mathrm{d}y=0. $$

Thus, using Poincare’s inequality, we have

$$ \vert w_{3} \vert _{0}\lesssim \vert \nabla _{\mathrm{h}}w_{3} \vert _{0}, $$

which immediately implies assertion (3.5).

Let $w\in H_{\sigma ,3}^{1}$, then $|\nabla _{\mathrm{h}} w_{3} |_{0}^{2}\neq 0$. Let $\hat{w}_{3}(\xi ,y_{3})$ be the horizontal Fourier transform of $w_{3}(y)$, i.e.,

$$ \hat{w}_{3}(\xi ,y_{3})= \int _{\Sigma }w_{3}( y_{\mathrm{h}},y_{3})e^{- \mathrm{i}y_{\mathrm{h}}\cdot \xi } \,\mathrm{d}x_{\mathrm{h}}, $$

where $\xi =(\xi _{1},\xi _{2})$, then $\widehat{\partial _{3} w_{3}} = \partial _{3} \widehat{w}_{3}$. We denote $\psi (\xi ,y_{3}):= \psi _{1}(\xi ,y_{3}) + \mathrm{i}\psi _{2}( \xi ,y_{3}):=\hat{w}_{3}(\xi ,y_{3})$, where $\psi _{1}$ and $\psi _{2}$ are real functions. Noting that $\psi (0)=0$, by Parseval’s theorem (see [9, Proposition 3.1.16]), we have

$$ \vert \nabla _{\mathrm{h}}{w}_{3} \vert ^{2}_{0}= \frac{1}{4\pi ^{2} L_{1}L_{2}}\sum _{\xi \in (L^{-1}_{1}\mathbb{Z} \times L^{-2}_{2}\mathbb{Z})} \vert \xi \vert ^{2} \bigl\vert \psi (\xi ,0) \bigr\vert ^{2} \geqslant L^{-1}_{1} \vert w_{3} \vert ^{2}_{0}, $$

which immediately yields that $a\leqslant L^{2}_{1}$. The proof is complete. □

Lemma 3.4

Friedrichs’s inequality (see [26, Lemma 1.42]): Let $1\leqslant p<\infty$ and D be a bounded Lipschitz domain. Let a set $\Gamma \subset \partial D$ be measurable with respect to the $(N-1)$-dimensional measure ${\mu }:=\mathrm{meas}_{N-1}$ defined on ∂D, and let $\mathrm{meas}_{N-1}(\Gamma )>0$. Then

$$ \Vert w \Vert _{W^{1,p}(D)}\lesssim \Vert \nabla w \Vert _{L^{p}(D)}$$

for all $u\in W^{1,p}(D)$ satisfying that the trace of u on Γ is equal to 0 a.e. with respect to the $(N-1)$-dimensional measure μ.

Remark 3.1

By Friedrichs’s inequality and the fact

$$ \Vert \nabla w \Vert _{0}^{2}= \Vert \mathbb{D}w \Vert _{0}^{2}/2 \quad \mbox{for any }w \in H_{\sigma }^{1}, $$

(3.6)

we get the Korn’s inequality

$$ \Vert w \Vert _{1} \lesssim \Vert \mathbb{D}w \Vert _{0}\quad \mbox{for any }w\in H_{\sigma }^{1}.$$

(3.7)

Lemma 3.5

Trace estimates:

$$\begin{aligned} & \vert w \vert _{0} \leqslant \Vert w \Vert _{1} \quad \textit{for any }w\in H^{1}_{\sigma }, \end{aligned}$$

(3.8)

$$\begin{aligned} & \vert w \vert _{0} \leqslant \sqrt{h_{\pm }/2} \Vert \mathbb{D} w \Vert _{L^{2}(\Omega _{\pm })}/2 \quad \textit{for any }w\in H^{1}_{\sigma }. \end{aligned}$$

(3.9)

Proof

See [24, Lemma 9.7] for (3.8). Since $C_{\sigma }^{\infty }:=C^{\infty }_{0}(\mathbb{R}^{2}\times (-h_{-},h_{+})) \cap H_{\sigma }^{1}$ is dense in $H^{1}_{\sigma }$, it suffices to prove that (3.9) holds for any $w\in C_{\sigma }^{\infty }$ by (3.8).

Let ŵ be the horizontal Fourier transformed function of $w\in C_{\sigma }^{\infty }$, and

$$ \varphi (\xi ,y_{3})=\mathrm{i} \hat{w}_{1}(\xi ,y_{3}),\qquad \theta (\xi ,y_{3})=\mathrm{i} \hat{w}_{2}(\xi ,y_{3}),\qquad \psi (\xi ,y_{3})= \hat{w}_{3}(\xi ,y_{3}). $$

Then

$$ \xi _{1}\varphi +\xi _{2}\theta +\psi '=0 $$

(3.10)

and $\psi (\cdot , y_{3})\in H^{2}_{0}(-h_{-},h_{+})$, because of $\operatorname{div} {w}=0$ and $w|_{\Sigma _{-}^{+}}=0$. Moreover,

$$ \widehat{\nabla {w}}=(\widehat{\partial _{i} w_{j}})= \begin{pmatrix} \xi _{1}\varphi & \xi _{2}\varphi & -\mathrm{i}\varphi ' \\ \xi _{1}\theta & \xi _{2}\theta &-\mathrm{i}\theta ' \\ \mathrm{i}\xi _{1}\psi & \mathrm{i}\xi _{2}\psi & \psi ' \end{pmatrix}. $$

In addition, we can deduce from (3.10) that

$$ \psi (0,y_{3})=0\quad \mbox{for }\xi =0. $$

(3.11)

By (3.11) and Fubini’s and Parseval’s theorems, one has

$$ \vert w_{3} \vert ^{2}_{0} = \frac{1}{4\pi ^{2} L_{1}L_{2}}\sum_{\xi \in (L^{-1}_{1} \mathbb{Z}\times L^{-1}_{2}\mathbb{Z})\backslash \{0\}} \bigl\vert \psi (\xi ,0) \bigr\vert ^{2} $$

(3.12)

and

$$\begin{aligned} \frac{1}{2} \Vert \mathbb{D} {w} \Vert ^{2}_{L^{2}(\Omega _{-})}&= \frac{1}{8\pi ^{2} L_{1}L_{2}}\sum_{\xi \in (L^{-1}_{1}\mathbb{Z} \times L^{-1}_{2}\mathbb{Z})} \sum _{1\leqslant i,j\leqslant 3} \int _{-h_{-}}^{0} \vert \widehat{\partial _{i} {w}_{j}}+\widehat{\partial _{j} {w}_{i}} \vert ^{2} \,\mathrm{d}y_{3} \\ &=\frac{1}{4\pi ^{2} L_{1}L_{2}}\sum_{\xi \in (L^{-1}_{1}\mathbb{Z} \times L^{-1}_{2}\mathbb{Z})\backslash \{0\}}M_{1}^{\xi }( \varphi , \theta ,\psi ) \\ &\quad {} + \frac{1}{4\pi ^{2} L_{1}L_{2}} \int _{-h_{-}}^{0} \bigl( \bigl\vert \varphi '(0,y_{3}) \bigr\vert ^{2}+ \bigl\vert \theta '(0,y_{3}) \bigr\vert ^{2} \bigr) \,\mathrm{d}y_{3}, \end{aligned}$$

(3.13)

where

$$ \begin{aligned} M_{1}^{\xi }( \varphi ,\theta ,\psi ):={}& \int _{-h_{-}}^{0} \bigl( \vert \xi \vert ^{2}\bigl( \vert \varphi \vert ^{2}+ \vert \theta \vert ^{2}+ \vert \psi \vert ^{2}\bigr) \\ &{} +2\Re {\psi }'' \Re {\psi } + 2\Im {\psi }'' \Im {\psi }+ \bigl\vert \varphi ' \bigr\vert ^{2}+ \bigl\vert \theta ' \bigr\vert ^{2}+3 \bigl\vert \psi ' \bigr\vert ^{2} \bigr)\,\mathrm{d}y_{3}. \end{aligned} $$

Using (3.10), we find that

$$ \begin{aligned} & \bigl\vert \psi ' \bigr\vert ^{2} = \xi _{1}^{2} \vert \varphi \vert ^{2}+\xi _{2}^{2} \vert \theta \vert ^{2}+ 2 \xi _{1}\xi _{2}(\Re {\varphi } \Re {\theta } + \Im {\varphi } \Im { \theta }) \leqslant \vert \xi \vert ^{2}\bigl( \vert \varphi \vert ^{2}+ \vert \theta \vert ^{2}\bigr), \\ & \bigl\vert \psi '' \bigr\vert ^{2} \leqslant \vert \xi \vert ^{2}\bigl( \bigl\vert \varphi ' \bigr\vert ^{2}+ \bigl\vert \theta ' \bigr\vert ^{2}\bigr) , \end{aligned} $$

which imply that

$$ \int _{-h_{-}}^{0} \bigl(4 \bigl\vert {\psi }' \bigl\vert ^{2}+ \bigr\vert \vert \xi \vert {\psi }+ {\psi }''/ \vert \xi \vert \bigr\vert ^{2}\bigr) \,\mathrm{d}y_{3}\leqslant M_{1}^{\xi }(\varphi ,\theta ,\psi ) $$

(3.14)

for given $\xi \in (L^{-1}_{1}\mathbb{Z}\times L^{-1}_{2}\mathbb{Z}) \backslash \{0\}$. Employing (3.12)–(3.14) and the relation

$$ \phi ^{2}(0)\leqslant h_{-} \bigl\Vert \phi ' \bigr\Vert _{L^{2}(-h_{-},0)}^{2}\quad \mbox{for any } \phi \in H^{1}_{0}(-h_{-},h_{+}), $$

we obtain

$$\begin{aligned} \vert w_{3} \vert ^{2}_{0} &= \frac{1}{4\pi ^{2} L_{1}L_{2}}\sum_{\xi \in (L^{-1}_{1} \mathbb{Z}\times L^{-1}_{2}\mathbb{Z})\backslash \{0\}} \bigl\vert \psi ( \xi ,0) \bigr\vert ^{2} \\ &\leqslant \frac{h_{-}}{16 \pi ^{2} L_{1}L_{2}} \sum_{\xi \in (L^{-1}_{1} \mathbb{Z}\times L^{-1}_{2}\mathbb{Z})\backslash \{0\}} \int _{-h_{-}}^{0} \bigl(4 \bigl\vert {\psi }' \bigr\vert ^{2}+\bigl( \vert \xi \vert {\psi }+ {\psi }''/ \vert \xi \vert \bigr)^{2}\bigr) \,\mathrm{d}y_{3} \\ &\leqslant \frac{h_{-}}{16 \pi ^{2} L_{1}L_{2}}\sum_{\xi \in (L^{-1}_{1} \mathbb{Z}\times L^{-1}_{2}\mathbb{Z})\backslash \{0\}} M_{1}^{\xi }( \varphi ,\theta ,\psi ) \leqslant {h_{-}} \Vert \mathbb{D} w \Vert ^{2}_{L^{2}( \Omega _{-})}/8. \end{aligned}$$

(3.15)

Similarly, we also have

$$ \vert w_{3} \vert ^{2}_{0} \leqslant {h_{+}} \Vert \mathbb{D} w \Vert ^{2}_{L^{2}(\Omega _{+})} /8, $$

which, together with (3.15), yields the desired conclusion. This completes the proof. □

Remark 3.2

From the derivation of (3.9), we easily see that

$$ \Vert \partial _{3} w_{3} \Vert _{L^{2}(\Omega _{\pm })}^{2} \leqslant \Vert \mathbb{D} w \Vert _{L^{2}(\Omega _{\pm })}^{2}/8\quad \mbox{for any }w\in H^{1}_{\sigma }. $$

(3.16)

Lemma 3.6

Negative trace estimate:

$$ \vert u_{3} \vert _{-1/2}\lesssim \Vert u \Vert _{0}+ \Vert \operatorname{div}u \Vert _{0}\quad \textit{for any }u:=(u_{1},u_{2},u_{3}) \in H_{0}^{1}. $$

(3.17)

Proof

Estimate (3.17) can be derived by integration by parts and an inverse trace theorem [26, Lemma 1.47]. □

Lemma 3.7

Let X be a given Banach space with dual $X^{*}$, and let u and w be two functions belonging to $L^{1}((a,b),X)$. Then the following two conditions are equivalent:

• (1) For each test function $\phi \in C_{0}^{\infty }(a,b)$,

  $$ \int _{a}^{b} u(t)\phi '(t) \,\mathrm{d}t=- \int _{a}^{b} w(t)\phi (t) \,\mathrm{d}t. $$

• (2) For each $\eta \in X^{*}$,

  $$ \frac{\mathrm{d}}{\mathrm{d}t}\langle u,\eta \rangle_{X\times X^{*}}=\langle w,\eta \rangle_{X \times X^{*}}, $$

  in the scalar distribution sense, on $(a,b)$, where $\langle \cdot ,\cdot \rangle_{X\times X^{*}}$ denotes the dual pair between X and $X^{*}$.

Proof

See Lemma 1.1 in Chap. 3 in [30]. □

4 Linear instability

In this section, we use the modified variational method to construct unstable solutions for the linearized RT problem. The modified variational method was firstly used by Guo and Tice to construct unstable solutions to a class of ordinary differential equations arising from a linearized RT instability problem [11]. This idea had been also used in [4, 5]. In this paper, we directly apply Guo and Tice’s modified variational method to the partial differential equations (2.5) and thus obtain a linear instability result of the RT problem by further using an existence theory of the stratified Stokes problem. Next we prove Theorem 2.1 by four subsections.

4.1 Existence of weak solutions to the modified problem

In this subsection, we consider the existence of weak solutions to the modified problem

$$ \textstyle\begin{cases} s(\mu \Delta w-\nabla q) = \alpha (s,\vartheta ) {\rho } w, \qquad \operatorname{div}w=0&\mbox{in }\Omega _{-}^{+}, \\ [\!\![w ]\!\!]=0,\qquad [\!\![((sq-g\rho w_{3}) I- s \mu \mathbb{D} w)e_{3} ]\!\!]= \vartheta \Delta _{\mathrm{h}}w_{3} e_{3}&\mbox{on }\Sigma , \\ w = {0}& \mbox{on }\Sigma _{-}^{+}, \end{cases} $$

(4.1)

where $s>0$ is any given. To prove the existence of weak solutions of the above problem, we consider the variational problem of the functional $F(\varpi ,s )$:

$$ \alpha (s,\vartheta ): =\sup_{\varpi \in \mathcal{A}} F( \varpi ,s ) $$

(4.2)

for given $s>0$, where we have defined that

$$ F(\varpi ,s):= -\bigl(\mathcal{E}(\varpi )+s \Vert \sqrt{\mu }\mathbb{D}\varpi \Vert _{0}^{2}/2\bigr). $$

Sometimes, we denote $\alpha (s,\vartheta )$ and $F(\varpi ,s)$ by α (or $\alpha (s)$) and $F(\varpi )$, resp., for simplicity. Then we have the following conclusions.

Proposition 4.1

Let $s>0$ be any given.

• 1. In the variational problem (4.2), $F(\varpi )$ achieves its supremum on $\mathcal{A}$.

• 2. Let w be a maximizer and $\alpha := \sup_{\varpi \in \mathcal{A}}F(\varpi )$, the w is a weak solution of the boundary problem (4.1) with given α.

Proof

Noting that

$$ \vert v \vert _{0}^{2} \lesssim \Vert v \Vert _{0} \Vert \partial _{3} v \Vert _{0}\quad \mbox{for any }v \in H_{0}^{1}, $$

(4.3)

thus, by Young’s inequality and Korn’s inequality (3.7), we see that $\{F(\varpi )\}_{\varpi \in \mathcal{A}}$ has an upper bound for any $\varpi \in \mathcal{A}$. Hence there exists a maximizing sequence $\{w^{n}\}_{n=1}^{\infty }\subset \mathcal{A}$, which satisfies $\alpha =\lim_{n\to \infty } F(w_{n})$. Moreover, making use of (4.3), the fact $\|\sqrt{ {\rho }}w^{n}\|_{0}=1$, trace estimate (3.9), and Young’s and Korn’s inequalities, we have $\|w^{n}\|_{1}+\vartheta |\nabla _{\mathrm{h}}w^{n}_{3}|_{0} \leqslant c_{1}$ for some constant $c_{1}$, which is independent of n. Thus, by the well-known Rellich–Kondrachov compactness theorem and (4.3), there exist a subsequence, still labeled by $w^{n}$, and a function $w\in \mathcal{A}$ such that

$$ \begin{aligned} & w^{n}\rightharpoonup w\quad \mbox{in }H^{1}_{\sigma },\qquad w^{n}\to w \quad \mbox{in }L^{2},\qquad w^{n}|_{y_{3}=0}\to w|_{y_{3}=0}\quad \mbox{in }L^{2}( \mathbb{T}), \\ & w^{n}_{3}|_{y_{3}=0}\rightharpoonup w_{3}|_{y_{3}=0}\quad \mbox{in }H^{1}( \mathbb{T}) \mbox{ if }\vartheta \neq 0. \end{aligned} $$

Exploiting the above convergence results and the lower semicontinuity of weak convergence, we have

$$ -\alpha =\liminf_{n\to \infty }\bigl(-F\bigl(w^{n} \bigr)\bigr) \geqslant -F(w) \geqslant - \alpha .$$

Hence w is a maximum point of the functional $F(\varpi )$ with respect to $\varpi \in \mathcal{A}$.

Obviously, w constructed above is also a maximum point of the functional $F(\varpi )/ \|\sqrt{\rho }\varpi \|^{2}_{0}$ with respect to $\varpi \in H_{\sigma , \vartheta }^{1}$. Moreover, $\alpha = F(w)/\|\sqrt{\rho }w\|^{2}_{0}$. Thus, for any given $\varphi \in H^{1}_{\sigma ,\vartheta }$, the point $t=0$ is the maximum point of the function

$$ I(t):=F(w+t\varphi )- \int \alpha {\rho } \vert w+t\varphi \vert ^{2} \,\mathrm{d}y \in C^{1}(\mathbb{R}). $$

Then, by computing out $I'(0)=0$, we have the weak form

$$ \frac{s}{2} \int \mu \mathbb{D} w:\mathbb{D}\varphi \,\mathrm{d}y + \vartheta \int _{\Sigma }\nabla _{\mathrm{h}}w_{3}\cdot \nabla _{ \mathrm{h}}\varphi _{3} \,\mathrm{d}y_{\mathrm{h}} = g [\!\![\rho ]\!\!]\int _{\Sigma }w_{3}\varphi _{3} \,\mathrm{d}y_{ \mathrm{h}}- \alpha \int \rho w\cdot \varphi \,\mathrm{d}y. $$

(4.4)

Note that (4.4) is equivalent to

$$ s \int \mu \mathbb{D} w:\nabla \varphi \,\mathrm{d}y+ \vartheta \int _{\Sigma }\nabla _{\mathrm{h}}w_{3}\cdot \nabla _{\mathrm{h}}\varphi _{3} \,\mathrm{d}y_{\mathrm{h}} = g [\!\![\rho ]\!\!]\int _{\Sigma }w_{3}\varphi _{3} \,\mathrm{d}y_{\mathrm{h}}-\alpha \int \rho w \cdot \varphi \,\mathrm{d}y. $$

This means that w is a weak solution of the modified problem (4.1). □

4.2 Improving the regularity of weak solution

By Proposition 4.1, the boundary value problem (4.1) admits a weak solution $w\in H^{1}_{\sigma ,\vartheta }$. Next we further improve the regularity of w.

Proposition 4.2

Let w be a weak solution of the boundary value problem (4.1). Then $w\in H^{\infty }$.

Proof

To begin with, we establish the following preliminary conclusion:

For any $i\geqslant 0$ , we have

$$ w \in H^{1,i}_{\sigma ,\vartheta } $$

(4.5)

and

$$\begin{aligned} &\frac{1}{2} \int s\mu \mathbb{D} \partial _{\mathrm{h}}^{i} w : \mathbb{D}\varphi \,\mathrm{d}y + \vartheta \int _{\Sigma }\nabla _{ \mathrm{h}} \partial _{\mathrm{h}}^{i} w_{3}\cdot \nabla _{ \mathrm{h}} \varphi _{3}\,\mathrm{d}y_{\mathrm{h}} \\ &\quad = g[\!\![\rho ]\!\!]\int _{\Sigma }\partial _{\mathrm{h}}^{i} w_{3}\varphi _{3}\,\mathrm{d}y_{\mathrm{h}}-\alpha \int \rho \partial _{\mathrm{h}}^{i} w\cdot \varphi \,\mathrm{d}y. \end{aligned}$$

(4.6)

Obviously, by induction, the above assertion reduces to verifying the following recurrence relation:

For given $i\geqslant 0$, if $w \in H^{1,i}_{\sigma ,\vartheta }$ satisfies (4.6) for any $\varphi \in H^{1}_{\sigma ,\vartheta }$, then

$$ w \in H^{1,i+1}_{\sigma ,\vartheta } $$

(4.7)

and w satisfies

$$\begin{aligned} &\frac{s}{2} \int \mu \mathbb{D} \partial _{\mathrm{h}}^{i+1} w : \mathbb{D} \varphi \,\mathrm{d}y+ \vartheta \int _{\Sigma }\nabla _{ \mathrm{h}} \partial _{\mathrm{h}}^{i+1} w_{3}\cdot \nabla _{ \mathrm{h}} \varphi _{3}\,\mathrm{d}y_{\mathrm{h}} \\ &\quad = g[\!\![\rho ]\!\!]\int _{\Sigma }\partial _{\mathrm{h}}^{i+1} w_{3}\varphi _{3} \,\mathrm{d}y_{\mathrm{h}}-\alpha \int \rho \partial _{\mathrm{h}}^{i+1} w\cdot \varphi \,\mathrm{d}y. \end{aligned}$$

(4.8)

Next we verify the above recurrence relation by the method of difference quotients.

Now we assume that $w \in H^{1,i}_{\sigma ,\vartheta }$ satisfies (4.6) for any $\varphi \in H^{1}_{\sigma ,\vartheta }$. Noting that $\partial _{\mathrm{h}}^{i}w \in H^{1}_{\sigma ,\vartheta }$, we can deduce from (4.6) that, for $j=1$ and 2,

$$\begin{aligned} & \frac{s}{2} \int \mu \mathbb{D} \partial _{\mathrm{h}}^{i} w : \mathbb{D} D_{j}^{h} \varphi \,\mathrm{d}y+ \vartheta \int _{\Sigma }\nabla _{\mathrm{h}} \partial _{\mathrm{h}}^{i} w_{3} \cdot \nabla _{ \mathrm{h}} D_{j}^{h} \varphi _{3} \,\mathrm{d}y_{\mathrm{h}} \\ &\quad = g[\!\![\rho ]\!\!]\int _{\Sigma }\partial _{\mathrm{h}}^{i} w _{3} D_{j}^{h} \varphi _{3} \,\mathrm{d}y_{\mathrm{h}}-\alpha \int \rho \partial _{\mathrm{h}}^{i} w\cdot D_{j}^{h} \varphi \,\mathrm{d}y \end{aligned}$$

and

$$\begin{aligned} & \frac{s}{2} \int \mu \mathbb{D} \partial _{\mathrm{h}}^{i} w : \mathbb{D} D_{j}^{-h} D_{j}^{h} \partial _{\mathrm{h}}^{i} w \,\mathrm{d}y+ \vartheta \int _{\Sigma }\nabla _{\mathrm{h}}\partial _{ \mathrm{h}}^{i} w_{3}\cdot \nabla _{\mathrm{h}} D_{j}^{-h} D_{j}^{h} \partial _{\mathrm{h}}^{i} w_{3} \,\mathrm{d}y_{\mathrm{h}} \\ &\quad = g[\!\![\rho ]\!\!]\int _{\Sigma }\partial _{\mathrm{h}}^{i} w_{3} D_{j}^{-h} D_{j}^{h} \partial _{\mathrm{h}}^{i} w_{3} \,\mathrm{d}y_{\mathrm{h}}-\alpha \int \rho \partial _{\mathrm{h}}^{i} w\cdot D_{j}^{-h} D_{j}^{h} \partial _{\mathrm{h}}^{i} w \,\mathrm{d}y, \end{aligned}$$

which yield that

$$\begin{aligned} &\frac{s}{2} \int \mu \mathbb{D} D_{j}^{-h} \partial _{\mathrm{h}}^{i} w : \mathbb{D} \varphi \,\mathrm{d}y +\vartheta \int _{\Sigma }\nabla _{ \mathrm{h}} D_{j}^{-h} \partial _{\mathrm{h}}^{i} w_{3}\cdot \nabla _{\mathrm{h}} \varphi _{3} \,\mathrm{d}y_{\mathrm{h}} \\ &\quad = g[\!\![\rho ]\!\!]\int _{\Sigma }D_{j}^{-h} \partial _{ \mathrm{h}}^{i} w_{3} \varphi _{3} \,\mathrm{d}y_{\mathrm{h}}-\alpha \int \rho D_{j}^{-h}\partial _{\mathrm{h}}^{i} w\cdot \varphi \,\mathrm{d}y , \end{aligned}$$

(4.9)

and

$$\begin{aligned} & \bigl\Vert \sqrt{s\mu }\mathbb{D} D_{j}^{h} \partial _{\mathrm{h}}^{i} w \bigr\Vert ^{2}_{0}/2+ \vartheta \bigl\vert D_{j}^{h}\nabla _{\mathrm{h}} \partial _{\mathrm{h}}^{i} w_{3} \bigr\vert ^{2}_{0} \\ &\quad \lesssim g[\!\![\rho ]\!\!]\bigl\vert D_{j}^{h} \partial _{ \mathrm{h}}^{i} w_{3} \bigr\vert ^{2}_{0} + \vert \alpha \vert \bigl\Vert \sqrt{ \rho } D_{j}^{h} \partial _{\mathrm{h}}^{i} w \bigr\Vert ^{2}_{0}, \end{aligned}$$

(4.10)

resp.

By Korn’s inequality,

$$ \bigl\Vert D_{j}^{h} \partial _{\mathrm{h}}^{i} w \bigr\Vert ^{2}_{1}\lesssim \bigl\Vert \sqrt{s \mu }\mathbb{D} D_{j}^{h} \partial _{\mathrm{h}}^{i} w \bigr\Vert ^{2}_{0}, $$

thus, using (4.3), Young’s inequality, and the first conclusion in Lemma 3.1, we further deduce from (4.10) that

$$ \bigl\Vert D^{h}_{\mathrm{h}} \partial _{\mathrm{h}}^{i} w \bigr\Vert ^{2}_{1}+ \vartheta \bigl\vert D^{h}_{\mathrm{h}}\nabla _{\mathrm{h}} \partial _{ \mathrm{h}}^{i} w_{3} \bigr\vert ^{2}_{0} \lesssim \bigl\Vert D^{h}_{\mathrm{h}} \partial _{\mathrm{h}}^{i} w \bigr\Vert ^{2}_{0} \lesssim \bigl\Vert \nabla _{ \mathrm{h}} \partial _{\mathrm{h}}^{i} w \bigr\Vert ^{2}_{0}\lesssim 1. $$

Thus, using (4.3), trace estimate (3.9), and the second conclusion in Lemma 3.1, there exists a subsequence of $\{-h\}_{h\in \mathbb{R}}$ (still denoted by −h) such that

$$ \textstyle\begin{cases} D^{-h}_{\mathrm{h}} \partial _{\mathrm{h}}^{i} w \rightharpoonup \nabla _{\mathrm{h}}\partial _{\mathrm{h}}^{i} w \quad \mbox{in }H^{1}_{\sigma },\qquad D^{-h}_{\mathrm{h}} \partial _{\mathrm{h}}^{i} w \to \nabla _{\mathrm{h}}\partial _{\mathrm{h}}^{i} w \quad \mbox{in }L^{2}, \\ D^{-h}_{\mathrm{h}} \partial _{\mathrm{h}}^{i} w|_{y_{3}=0} \to \nabla _{\mathrm{h}}\partial _{\mathrm{h}}^{i} w |_{y_{3}=0} \quad \mbox{in }L^{2}(\mathbb{T}), \\ D^{-h}_{\mathrm{h}}\partial _{\mathrm{h}}^{i} w_{3}|_{\Sigma } \rightharpoonup \nabla _{\mathrm{h}}\partial _{\mathrm{h}}^{i} w_{3}|_{ \Sigma }\quad \mbox{in }H^{1}(\mathbb{T})\mbox{ if }\vartheta \neq 0. \end{cases} $$

(4.11)

Using the regularity of w in (4.11) and the fact $w\in H_{\sigma ,\vartheta }^{1,i}$, we have (4.7). In addition, exploiting the limit results in (4.11), we can deduce (4.8) from (4.9). This completes the proof of the recurrence relation, and thus (4.5) holds.

With (4.5) in hand, we can consider a stratified Stokes problem:

$$ \textstyle\begin{cases} s \nabla \beta ^{k} -s\mu \Delta \omega ^{k} =-\alpha \rho \partial _{ \mathrm{h}}^{k} w &\mbox{in } \Omega , \\ \operatorname{div}\omega ^{k}=0&\mbox{in } \Omega , \\ [\!\![\omega ^{k} ]\!\!]=0,\qquad [\!\![( s \beta ^{k} I - s\mu \mathbb{D}\omega ^{k})e_{3} ]\!\!]= \partial _{\mathrm{h}}^{k}\mathcal{L}^{1}&\mbox{on } \Sigma , \\ \omega ^{k} =0 &\mbox{on }\Sigma _{-}^{+},\end{cases} $$

(4.12)

where $k\geqslant 0$ is a given integer, and we have defined that

$$ \mathcal{L}^{1} :=g [\!\![\rho ]\!\!]w_{3}e_{3} + \vartheta \Delta _{\mathrm{h}}w_{3} e_{3}. $$

Recalling the regularity (4.5) of w, we see that $\partial _{\mathrm{h}}^{k}w\in L^{2}$ and $\partial _{\mathrm{h}}^{k}\mathcal{L}^{1} \in H^{1}(\mathbb{T})$. Applying the existence theory of the stratified Stokes problem (see Lemma 3.2), there exists a unique strong solution $(\omega ^{k},\beta ^{k})\in H^{2}\times \underline{H}^{1}$ of the above problem (4.12).

Multiplying (4.12)₁ by $\varphi \in H^{1}_{\sigma ,\vartheta }$ in $L^{2}$ (i.e., taking the inner product in $L^{2}$) and using the integration by parts and (4.12)₂–(4.12)₄, we have

$$\begin{aligned} &\frac{s }{2} \int \mu \mathbb{D}\omega ^{k}:\mathbb{D}\varphi \,\mathrm{d}y \\ &\quad = g[\!\![\rho ]\!\!]\int _{\Sigma }\partial _{\mathrm{h}}^{k} w_{3}\varphi _{3} \,\mathrm{d}y_{\mathrm{h}} - \int _{\Sigma }\vartheta \partial _{\mathrm{h}}^{k} \nabla _{\mathrm{h}}w_{3} \cdot \nabla _{\mathrm{h}}\varphi _{3}\,\mathrm{d}y_{\mathrm{h}} - \int \alpha \rho \partial _{\mathrm{h}}^{k} w\varphi \,\mathrm{d}y. \end{aligned}$$

(4.13)

Subtracting the two identities (4.6) and (4.13) yields that

$$ s \int \mu \mathbb{D}\bigl(\partial _{\mathrm{h}}^{k} w- \omega ^{k}\bigr): \mathbb{D} \varphi \,\mathrm{d}y=0. $$

Taking $\varphi := \partial _{\mathrm{h}}^{k} w-\omega ^{k}\in H^{1}_{ \sigma ,\vartheta }$ in the above identity and using the Korn’s inequality, we find that $\omega ^{k}= \partial _{\mathrm{h}}^{k} w$. Thus we immediately see that

$$ \partial _{\mathrm{h}}^{k} w\in H^{2}\quad \mbox{for any }k\geqslant 0, $$

(4.14)

which implies $\partial _{\mathrm{h}}^{k}w\in H^{1}$ and $\partial _{\mathrm{h}}^{k}\mathcal{L}^{1}\in H^{2}(\mathbb{T})$ for any $k\geqslant 0$. Thus, applying the stratified Stokes estimate (3.2) to (4.12), we have

$$ \partial _{\mathrm{h}}^{k} w\in H^{3} \quad \mbox{for any }k\geqslant 0. $$

(4.15)

Obviously, by induction, we can easily follow the improving regularity method from (4.14) to (4.15) to deduce that $w\in H^{\infty }$. In addition, we have $\beta :=\beta ^{0}\in H^{\infty }$; moreover, $\beta ^{k}$ in (4.12) is equal to $\partial ^{k}_{\mathrm{h}}\beta$.

Finally, recalling the embedding $H^{k+2}\hookrightarrow C^{0}(\overline{\Omega })$ for any $k\geqslant 0$, we easily see that $(w,\beta )$ constructed above is indeed a classical solution to the modified problem (4.1). □

4.3 Some properties of the function $\alpha (s)$

In this subsection, we derive some properties of the function $\alpha (s)$, which ensure the existence of a fixed point of $\sqrt{\alpha (s)}$ in $\mathbb{R}^{+}$.

Proposition 4.3

For given $\vartheta \in \mathbb{R}^{+}_{0}$, we have

$$\begin{aligned} &\alpha (s_{2}) < \alpha (s_{1}) \quad \textit{for any }s_{2} >s_{1}>0, \end{aligned}$$

(4.16)

$$\begin{aligned} & \alpha (s) \in C^{0,1}_{\mathrm{loc}}\bigl( \mathbb{R}^{+}\bigr), \end{aligned}$$

(4.17)

$$\begin{aligned} & \alpha (s)>0\quad \textit{on some interval }(0,c_{2})\textit{ for } \vartheta \in [0, \vartheta _{\mathrm{c}}), \end{aligned}$$

(4.18)

$$\begin{aligned} &\alpha (s)< 0\quad \textit{on some interval }(c_{3},\infty ). \end{aligned}$$

(4.19)

Proof

To begin with, we verify (4.16). For given $s_{2}>s_{1}$, there exists $v^{s_{2}}\in \mathcal{A}$ such that $\alpha (s_{2}) = F(v^{s_{2}},s_{2})$. Thus, by Korn’s inequality and the fact $\|\sqrt{\rho }v^{s_{2}}\|_{0}=1$,

$$ \alpha (s_{1})\geqslant F\bigl(v^{s_{2}},s_{1} \bigr) = \alpha (s_{2}) + ( s_{2}- {s_{1}}) \bigl\Vert \sqrt{\mu }\mathbb{D}v^{s_{2}} \bigr\Vert _{0}^{2}/2 > \alpha (s_{2}), $$

which yields (4.16).

Now we turn to prove (4.17). Choosing a bounded interval $[c_{4},c_{5}]\subset (0,\infty )$, for any $s\in [c_{4},c_{5}]$, there exists a function $v^{s}$ satisfying $\alpha (s)=F(v^{s},s)$. Thus, by the monotonicity of (4.16), we have

$$ \alpha (c_{5})+ c_{4} \bigl\Vert \sqrt{\mu } \mathbb{D}v^{s} \bigr\Vert _{0}^{2}/4 \leqslant F\bigl(v^{s},s/2\bigr) \leqslant \alpha (s/2) \leqslant \alpha ( c_{4}/2), $$

which yields

$$ \bigl\Vert \sqrt{\mu } \mathbb{D}v^{s} \bigr\Vert _{0}^{2}/2 \leqslant 2\bigl(\alpha (c_{4}/2)- \alpha (c_{5})\bigr)/c_{4}=:\xi \quad \mbox{for any }s\in [c_{4},c_{5}]. $$

Thus, for any $s_{1}$, $s_{2}\in [c_{4},c_{5}]$,

$$ \alpha (s_{1})-\alpha (s_{2})\leqslant F\bigl(v^{s_{1}},s_{1} \bigr)-F\bigl(v^{s_{1}},s_{2}\bigr) \leqslant \xi \vert {s_{2}}-s_{1} \vert $$

and

$$ \alpha (s_{2})-\alpha (s_{1})\leqslant \xi \vert {s_{2}}-s_{1} \vert , $$

which immediately imply $|\alpha (s_{1})-\alpha (s_{2})|\leqslant \xi | {s_{2}}-s_{1}|$. Hence (4.17) holds.

Finally, (4.19) can be deduced from the definition of α by using Korn’s inequality and (4.3), while (4.18) is obvious by the definition of α and Lemma 3.3. □

4.4 Construction of an interval for a fixed point

Let $\mathfrak{I}:=\sup \{\text{all the real constant }s\text{, which satisfy that } \alpha ( \tau )>0\text{ for any }\tau \in (0,s)\}$. By virtue of (4.18) and (4.19), $\mathfrak{I}\in \mathbb{R}^{+}$. Moreover, $\alpha (s)>0$ for any $s\in (0,\mathfrak{I})$ and, by the continuity of $\alpha (s)$,

$$ \alpha ( \mathfrak{I})=0. $$

(4.20)

Using the monotonicity and the upper boundedness of $\alpha (s)$, we see that

$$ \lim_{s\rightarrow 0}\alpha (s)=\varsigma\quad \mbox{for some positive constant }\varsigma . $$

(4.21)

Now, exploiting (4.20), (4.21), and the continuity of ${\alpha }(s)$ on $(0,\mathfrak{I})$, we find by a fixed point argument on $(0,\mathfrak{I})$ that there is unique $\Lambda \in (0,\mathfrak{I})$ satisfying

$$ \Lambda =\sqrt{ \alpha (\Lambda )}= \sqrt{\sup _{\varpi \in \mathcal{A}}F(\varpi , \Lambda )}\in (0,\mathfrak{I}). $$

(4.22)

Thus we get a classical solution $(w,\beta ) \in H^{\infty }$ to the boundary problem (2.5) with Λ constructed by (4.22). Moreover,

$$ \Lambda = \sqrt{ F(w, \Lambda )}>0. $$

(4.23)

In addition, (2.7) directly follows (4.23) and the fact $w\in H_{\sigma }^{1}$.

4.5 Largest growth rate

Next we prove that Λ constructed in the previous section is the largest growth rate of RT instability in the linearized RT problem and thus complete the proof of Theorem 2.1.

Proposition 4.4

Under the assumptions of Theorem 2.1, $\Lambda >0$ constructed by (4.22) is the largest growth rate of RT instability in the linearized RT problem.

Proof

Recalling the definition of largest growth rate, it suffices to prove that Λ enjoys the first condition in Definition 2.1.

Let u be a strong solution to the linearized RT problem. Then we derive that, for a.e. $t\in I_{T}$ and all $w\in H_{\sigma }^{1}$,

$$\begin{aligned} \int \rho u_{t}\cdot w\,\mathrm{d}y&= \int (\mu \Delta u-\nabla q) \cdot w\,\mathrm{d}y \\ &= \int _{\Sigma }\bigl(g[\!\![\rho ]\!\!]\eta _{3}w_{3}+\vartheta \Delta _{\mathrm{h}}\eta _{3} w_{3} \bigr)\,\mathrm{d}y_{\mathrm{h}}- \int \mu \mathbb{D}u: \nabla w\,\mathrm{d}y. \end{aligned}$$

(4.24)

Thus,

$$ \frac{\mathrm{d}}{\mathrm{d}t} \int \rho u_{t}\cdot w\,\mathrm{d}y = \int _{\Sigma }\bigl(g[\!\![\rho ]\!\!]u_{3}w_{3}+\vartheta \Delta _{\mathrm{h}}u_{3}w_{3} \bigr)\,\mathrm{d}y_{\mathrm{h}}- \int \mu \mathbb{D}u_{t}: \nabla w\,\mathrm{d}y. $$

(4.25)

Using the regularity of $(\eta ,u)$, we can show that the right-hand side of (4.25) is bounded above by $A(t)(\|w\|_{1}+|w|_{1})$ for some positive function $A(t)\in L^{2}(I_{T})$. Then there exists $f\in L^{2}(I_{T}, H^{-1}_{\sigma })$ such that, for a.e. $t\in I_{T}$,

$$ \langle f, w\rangle _{H^{-1}_{\sigma }\times H^{1}_{\sigma }}:= \int _{\Sigma }\bigl(g [\!\![\rho ]\!\!]u_{3}w_{3}+\vartheta \Delta _{\mathrm{h}}u_{3} w_{3} \bigr)\,\mathrm{d}y_{\mathrm{h}}- \int \mu \mathbb{D}u_{t}: \nabla w \,\mathrm{d}y. $$

(4.26)

Hence it follows from Lemma 3.7 that

$$ (\rho u_{t})_{t}=f\in L^{2} \bigl(I_{T},H^{-1}_{\sigma }\bigr). $$

In addition, by a classical regularization method (referring to Theorem 3 in Chap. 5.9 in [7] and Lemma 6.5 in [26]), we have

$$\begin{aligned} &\frac{1}{2}\frac{\mathrm{d}}{\mathrm{d}t} \int \rho \vert u_{t} \vert ^{2} \,\mathrm{d}y= \bigl\langle \partial _{t}(\rho u_{t}), u_{t}\bigr\rangle _{H^{-1}_{\sigma } \times H^{1}_{\sigma }}, \\ & \int _{\Sigma }\Delta _{\mathrm{h}}u_{3} \partial _{t} u_{3} \,\mathrm{d}y_{\mathrm{h}}=- \frac{1}{2} \frac{\mathrm{d}}{\mathrm{d}t} \int _{\Sigma } \vert \nabla _{\mathrm{h}}u_{3} \vert ^{2} \,\mathrm{d}y_{\mathrm{h}}. \end{aligned}$$

Therefore, we can derive from (4.26) and the above two identities that

$$ \frac{\mathrm{d}}{\mathrm{d}t} \bigl( \Vert \sqrt{\rho } u_{t} \Vert ^{2}_{0}+ \mathcal{E}(u ) \bigr)+ \Vert \sqrt{\mu } \mathbb{D} u _{t} \Vert _{0}^{2} =0. $$

Then, integrating the above identity in time from 0 to t yields that

$$ \Vert \sqrt{\rho } u_{t} \Vert ^{2}_{0}+ \mathcal{E}(u )+ \int _{0}^{t} \Vert \sqrt{ \mu } \mathbb{D} u _{s} \Vert _{0}^{2}\,\mathrm{d}s = I^{0}:= \mathcal{E}(u |_{t=0})+ \Vert \sqrt{\rho }u_{t}|_{t=0} \Vert ^{2}_{0}. $$

(4.27)

Using Newton–Leibniz’s formula and Young’s inequality, we find that

$$\begin{aligned} &\Lambda \bigl\Vert \sqrt{\mu }\mathbb{D} u(t) \bigr\Vert _{0}^{2} \\ &\quad =\Lambda \bigl\Vert \sqrt{\mu } \mathbb{D} u^{0} \bigr\Vert _{0}^{2} + 2\Lambda \int _{0}^{t} \int \mu \mathbb{D} u(s):\mathbb{D} u_{s} \,\mathrm{d} y \,\mathrm{d}s \\ &\quad \leqslant \Lambda \bigl\Vert \sqrt{\mu } \mathbb{D} u^{0} \bigr\Vert _{0}^{2} + \int _{0}^{t} \Vert \sqrt{\mu } \mathbb{D} u_{s} \Vert _{0}^{2} \,\mathrm{d}s +\Lambda ^{2} \int _{0}^{t} \bigl\Vert \sqrt{\mu } \mathbb{D} u (s) \bigr\Vert _{0}^{2} \,\mathrm{d}s. \end{aligned}$$

(4.28)

In addition, by (2.6), we have

$$ -\mathcal{E}(u) \leqslant {\Lambda ^{2}} \Vert \sqrt{\rho } {u} \Vert ^{2}_{0} + \frac{\Lambda }{2} \Vert \sqrt{\mu } \mathbb{D} u \Vert ^{2}_{0}. $$

(4.29)

Thus, we infer from (4.27)–(4.29) that

$$\begin{aligned} &\frac{1}{\Lambda } \Vert \sqrt{ {\rho }} u_{t} \Vert ^{2}_{0}+ \frac{1}{2} \bigl\Vert \sqrt{\mu } \mathbb{D} u (t) \bigr\Vert _{0}^{2} \\ &\quad \leqslant {\Lambda } \bigl\Vert \sqrt{ {\rho }} u (t) \bigr\Vert ^{2}_{0}+ {\Lambda } \int _{0}^{t} \bigl\Vert \sqrt{\mu } \mathbb{D} u(s) \bigr\Vert _{0}^{2} \,\mathrm{d}s + \frac{I^{0}+ \Lambda \Vert \sqrt{\mu }\mathbb{D} u^{0} \Vert _{0}^{2} }{\Lambda }. \end{aligned}$$

(4.30)

Recalling that

$$ \Lambda \frac{\mathrm{d}}{\mathrm{d}t} \Vert \sqrt{ {\rho }} u \Vert ^{2}_{0} = 2\Lambda \int \rho u(t)\cdot u_{t}\,\mathrm{d} y \leqslant \Vert \sqrt{ \rho } u_{t} \Vert ^{2}_{0} +\Lambda ^{2} \bigl\Vert \sqrt{ {\rho }} u(t) \bigr\Vert ^{2}_{0}, $$

we further deduce from (4.30) the differential inequality

$$ \begin{aligned} & \frac{\mathrm{d}}{\mathrm{d}t} \Vert \sqrt{ {\rho }} u \Vert ^{2}_{0}+ \frac{1}{2} \bigl\Vert \sqrt{\mu } \mathbb{D} u(t) \bigr\Vert _{0}^{2} \\ &\quad \leqslant 2\Lambda \biggl( \bigl\Vert \sqrt{\rho } u(t) \bigr\Vert ^{2}_{0} + \frac{1}{2} \int _{0}^{t} \bigl\Vert \sqrt{\mu } \mathbb{D} u(s) \bigr\Vert _{0}^{2} \,\mathrm{d}s \biggr) + \frac{I^{0} + \Lambda \Vert \sqrt{\mu } \mathbb{D} u^{0} \Vert _{0}^{2} }{\Lambda }. \end{aligned} $$

Applying Gronwall’s inequality [26, Lemma 1.2] to the above inequality, one concludes

$$\begin{aligned} & \bigl\Vert \sqrt{ {\rho }} u (t) \bigr\Vert ^{2}_{0}+ \frac{1}{2} \int _{0}^{t} \bigl\Vert \sqrt{ \mu } \mathbb{D} u(s) \bigr\Vert ^{2}_{0} \,\mathrm{d}s \\ &\quad \leqslant \biggl( \bigl\Vert \sqrt{ {\rho }}u^{0} \bigr\Vert _{0}^{2}+ \frac{ I^{0}+ \Lambda \Vert \sqrt{\mu } \mathbb{D} u^{0} \Vert _{0}^{2} }{2\Lambda ^{2}} \biggr) e^{2\Lambda t}, \end{aligned}$$

(4.31)

which, together with (4.30), yields

$$\begin{aligned} \frac{1}{\Lambda } \bigl\Vert \sqrt{ {\rho }} u_{t} (t) \bigr\Vert ^{2}_{0} + \frac{1}{2} \bigl\Vert \sqrt{\mu } \mathbb{D} u (t) \bigr\Vert _{0}^{2} \leqslant{} & 2 \biggl(\Lambda \bigl\Vert \sqrt{ {\rho }}u^{0} \bigr\Vert _{0}^{2}+ \frac{ I^{0}+ \Lambda \Vert \sqrt{\mu } \mathbb{D} u^{0} \Vert _{0}^{2} }{2\Lambda } \biggr) e^{2\Lambda t} \\ &{} + \frac{I^{0}+ \Lambda \Vert \sqrt{\mu } \mathbb{D} u^{0} \Vert _{0}^{2} }{\Lambda }. \end{aligned}$$

(4.32)

Multiplying (1.10)₂ by $u_{t}$ in $L^{2}$ and using the integral by parts, we get

$$ \int \rho \vert u_{t} \vert ^{2}\,\mathrm{d}y= \int _{\Sigma } [\!\![q ]\!\!]\partial _{t}u_{3} \,\mathrm{d}y_{\mathrm{h}} + \int \mu \Delta u\cdot u_{t}\,\mathrm{d}y. $$

(4.33)

Exploiting (3.17), we can estimate that

$$ \int _{\Sigma } [\!\![q ]\!\!]\partial _{t}u_{3} \,\mathrm{d}y_{ \mathrm{h}} \lesssim \bigl\vert [\!\![q ]\!\!]\bigr\vert _{1/2} \vert \partial _{t} u_{3} \vert _{-1/2}\lesssim \bigl\vert [\!\![q ]\!\!]\bigr\vert _{1/2} \Vert u_{t} \Vert _{0}. $$

In addition, using (1.10)₅ and trace estimate (3.9), we have

$$ \bigl\vert [\!\![q ]\!\!]\bigr\vert _{1/2}\lesssim \Vert \eta \Vert _{3}+ \Vert u \Vert _{2}. $$

Using the above two estimates, we can derive from (4.33) that

$$ \Vert u_{t} \Vert ^{2}_{0}\lesssim \Vert \eta \Vert _{3}+ \Vert u \Vert _{2} , $$

which implies that

$$ \Vert \sqrt{\rho }u_{t}|_{t=0} \Vert ^{2}_{0} \lesssim \bigl\Vert \bigl(\eta ^{0},u^{0}\bigr) \bigr\Vert _{3}. $$

By the above estimate and Korn’s inequality, we derive from (4.31) and (4.32) that

$$ \Vert u \Vert _{1}^{2}+ \Vert u_{t} \Vert ^{2}_{0 }+ \int _{0}^{t} \bigl\Vert u(s) \bigr\Vert ^{2}_{1} \,\mathrm{d}s\lesssim e^{2\Lambda t}\bigl( \bigl\Vert \eta ^{0} \bigr\Vert _{3}^{2}+ \bigl\Vert u^{0} \bigr\Vert _{2}^{2}\bigr). $$

Finally, from (1.10)₁ we get

$$\begin{aligned} \bigl\Vert \eta (t) \bigr\Vert _{1} &\lesssim \bigl\Vert \eta ^{0} \bigr\Vert _{1} + \int _{0}^{t} \Vert \eta _{s} \Vert _{1} \,\mathrm{d}s \lesssim \bigl\Vert \eta ^{0} \bigr\Vert _{1} + \int _{0}^{t} \bigl\Vert u(s) \bigr\Vert _{1}\,\mathrm{d}s \\ &\lesssim e^{\Lambda t}\bigl( \bigl\Vert \eta ^{0} \bigr\Vert _{3}+ \bigl\Vert u^{0} \bigr\Vert _{2}\bigr). \end{aligned}$$

By the two estimates above, we see that Λ satisfies the first condition in Definition 2.1. The proof is completed. □

5 Effect of surface tension

5.1 Properties of $\alpha (s,\vartheta )$ with respect to ϑ

To emphasize the dependence of Λ and $\mathcal{G}$ upon ϑ, we denote them by $\Lambda _{\vartheta }$ and $\mathcal{G}_{\vartheta }$, respectively. To prove Theorem 2.2, we further derive relations (2.1) and (2.11) of the surface tension coefficient and the largest growth rate. To this end, we need the following auxiliary conclusions.

Proposition 5.1

Let $g>0$, $\rho >0$, and $\mu >0$ be given.

• (1) Strict monotonicity: if $\vartheta _{1}$ and $\vartheta _{2}$ are constants satisfying $0\leqslant \vartheta _{1}<\vartheta _{2}$, then

  $$ \alpha (s,\vartheta _{2})< \alpha (s,\vartheta _{1}) $$

  (5.1)

  for any given $s>0$. Moreover, if $\vartheta _{2}$ further satisfies $\vartheta _{2}<\vartheta _{\mathrm{c}}$,

  $$ \mathcal{G}_{\vartheta _{\mathrm{1}}}>\mathcal{G}_{\vartheta _{ \mathrm{2}}}, $$

  (5.2)

  where

  $$ \mathcal{G}_{\vartheta _{i}}:=\sup \bigl\{ s\in \mathbb{R} \mid \alpha (\tau , \vartheta _{i})>0\textit{ for any } \tau \in (0,s)\bigr\} \quad \textit{and}\quad \alpha ( \mathcal{G}_{\vartheta _{i}},\vartheta _{i})=0. $$

  (5.3)

• (2) Continuity: for given $s>0$, $\alpha (s,\vartheta )\in C^{0,1}_{\mathrm{loc}}(\mathbb{R}^{+})$ with respect to the variable ϑ.

Proof

(1) Let $s>0$ be fixed and $0\leqslant \vartheta _{1}<\vartheta _{2}$. Then there exist functions ${w}^{\vartheta _{i}}\in H^{\infty }\cap \mathcal{A}_{\vartheta _{i}}$, $i=1,2$, such that

$$ \alpha (s,\vartheta _{i})= {E} \bigl( {w} ^{\vartheta _{i}}\bigr)-\vartheta _{i} \bigl\vert \nabla _{\mathrm{h}} {w}_{3}^{\vartheta _{i}} \bigr\vert _{0}^{2}, $$

where ${E}( {w}^{\vartheta _{i}}):=g[\!\![\rho ]\!\!]| {w}^{ \vartheta _{i}}_{3}|_{0}^{2} -{s}\|\sqrt{\mu } \mathbb{D} {w}^{ \vartheta _{i}}\|_{0}^{2}/2$. Since ${w}^{\vartheta _{i}}\in \mathcal{A}_{\vartheta _{i}}$, by virtue of (2.7) and (3.5), we have

$$ 0< \bigl\vert {w}^{\vartheta _{2}} \bigr\vert _{0}\lesssim \bigl\vert \nabla _{\mathrm{h}} {w}^{ \vartheta _{2}} \bigr\vert _{0}, $$

and thus

$$ \alpha (s,\vartheta _{2})\leqslant \alpha (s,\vartheta _{1})+ {( \vartheta _{1}-\vartheta _{2}) } \bigl\vert \nabla _{\mathrm{h}} {w}_{3}^{ \vartheta _{2}} \bigr\vert _{0}^{2}< \alpha (s,\vartheta _{1}). $$

This yields the desired conclusion (5.1).

Next we prove (5.2) by contradiction. If $\mathcal{G}_{\vartheta _{\mathrm{1}}}<\mathcal{G}_{\vartheta _{ \mathrm{2}}}$, then we get from (5.1) and the strict monotonicity of $\alpha (s,\cdot )$ with respect to s that

$$ 0=\alpha (\mathcal{G}_{\vartheta _{\mathrm{2}}},\vartheta _{2}) < \alpha (\mathcal{G}_{\vartheta _{\mathrm{2}}},\vartheta _{1}) < \alpha ( \mathcal{G}_{\vartheta _{\mathrm{1}}},\vartheta _{1})=0, $$

which is a paradox. If $\mathcal{G}_{\vartheta _{\mathrm{1}}}=\mathcal{G}_{\vartheta _{ \mathrm{2}}}$, exploiting (5.1), we have

$$ 0=\alpha (\mathcal{G}_{\vartheta _{\mathrm{2}}},\vartheta _{2})< \alpha (\mathcal{G}_{\vartheta _{\mathrm{2}}},\vartheta _{1})= \alpha ( \mathcal{G}_{\vartheta _{\mathrm{1}}},\vartheta _{1})=0, $$

which is also a paradox. Thus we immediately get the desired conclusion.

(2) Let $s>0$ be fixed. We choose a bounded interval $[b_{1},b_{2}]\subset \mathbb{R}^{+}$. Then, for any given $\theta \in [ b_{1},b_{2}]$, there is a function ${w}^{\theta }\in \mathcal{A}_{\vartheta }$ satisfying $\alpha (s,\theta )={E}( {w}^{\theta }) -\theta |\nabla _{\mathrm{h}} {w}_{3}^{ \theta }|_{0}^{2}$. Thus, in view of the monotonicity of $\alpha (\cdot ,\theta )$, we know that

$$\begin{aligned} \alpha (s,b_{2})+b_{1} \bigl\vert \nabla _{\mathrm{h}} {w}_{3}^{\theta } \bigr\vert _{0}^{2}/2 &\leqslant \alpha (s,\theta )+\theta \bigl\vert \nabla _{\mathrm{h}} {w}_{3}^{ \theta } \bigr\vert _{0}^{2}/2 \\ &\leqslant \alpha (s,\theta /2 ) \leqslant \alpha (s,b_{1}/2), \end{aligned}$$

(5.4)

which yields

$$ \bigl\vert \nabla _{\mathrm{h}} {w}_{3}^{\vartheta } \bigr\vert _{0}^{2} \leqslant 2 \bigl( \alpha (s,b_{1}/2)-\alpha (s,b_{2}) \bigr)/b_{1} :=K(s) \quad \mbox{for any } \vartheta \in [b_{1},b_{2}]. $$

Thus, for any $\vartheta _{1}$, $\vartheta _{2}\in [b_{1},b_{2}]$,

$$ \begin{aligned} \alpha (s,\vartheta _{1})-\alpha (s, \vartheta _{2})&\leqslant {E}\bigl( {w}^{ \vartheta _{1}}\bigr)- \vartheta _{1} \bigl\vert \nabla _{\mathrm{h}} {w}_{3}^{ \vartheta _{1} } \bigr\vert _{0}^{2} - \bigl({E}\bigl( {w}^{\vartheta _{1}}\bigr)-\vartheta _{2} \bigl\vert \nabla _{\mathrm{h}} {w}_{3}^{\vartheta _{1} } \bigr\vert _{0}^{2}\bigr) \\ &\leqslant K(s) \vert \vartheta _{2}-\vartheta _{1} \vert . \end{aligned} $$

Reversing the role of indices 1 and 2 in the derivation of the above inequality, we obtain the same boundedness with the indices switched. Therefore, we deduce that

$$ \bigl\vert \alpha (s,\vartheta _{1})- \alpha (s,\vartheta _{2}) \bigr\vert \leqslant K(s) \vert \vartheta _{1}-\vartheta _{2} \vert , $$

which yields $\alpha (s,\vartheta )\in C^{0,1}_{\mathrm{loc}}(\mathbb{R}^{+})$. This completes the proof. □

5.2 Proof of Theorem 2.2

First, we verify the monotonicity of $\Lambda _{\vartheta }$ with respect to the variable $\vartheta \in [0, \vartheta _{\mathrm{c}})$.

For given two constants $\vartheta _{1}$ and $\vartheta _{2}$ satisfying $0\leqslant \vartheta _{1}<\vartheta _{2}< \vartheta _{\mathrm{c}}$, there exist two associated curve functions $\alpha (s,\vartheta _{1})$ and $\alpha (s,\vartheta _{2})$ defined in $(0,\vartheta _{\mathrm{c}})$. By the first assertion in Proposition 5.1,

$$ \alpha (s,\vartheta _{1})>\alpha (s,\vartheta _{2}). $$

On the one hand, the fixed point $\Lambda _{\vartheta _{i}}$ satisfying $\Lambda _{\vartheta _{i}}=\sqrt{\alpha (\Lambda _{\vartheta _{i}})}$ can be obtained from the intersection point of the two curves $y=s$ and $y=\sqrt{\alpha (s,\vartheta _{i})}$ on $(0,\mathcal{G}_{\vartheta _{i}})$ for $i=1$ and 2. Thus we can immediately observe the monotonicity

$$ \Lambda _{\vartheta _{1}}>\Lambda _{\vartheta _{2}}\quad \mbox{for }0 \leqslant \vartheta _{1}< \vartheta _{2}< \vartheta _{\mathrm{c}}. $$

(5.5)

Second, we prove the continuity for $\Lambda _{\vartheta }$.

We choose a constant $\vartheta _{0}>0$ and an associated function $\alpha (s,\vartheta _{0})$. Noting that $\alpha (\Lambda _{\vartheta _{0}},\vartheta _{0})=\Lambda ^{2}_{ \vartheta _{0}}>0$ and $\alpha (\cdot ,\vartheta )\in C^{0,1}_{\mathrm{loc}}[0,\vartheta _{ \mathrm{c}})$ are strictly decreasing and continuous with respect to ϑ, for any given $\varepsilon >0$, there exists a constant $\delta >0$ such that

$$ \begin{aligned} & (\vartheta _{0}-\delta ,\vartheta _{0}+\delta ) \subset (0, \vartheta _{\mathrm{c}} ), \qquad \alpha ( \Lambda _{\vartheta _{0}}, \vartheta _{0}+\delta )>0, \\ &0< \sqrt{\alpha (\Lambda _{\vartheta _{0}},\vartheta _{0})}-\sqrt{ \alpha (\Lambda _{\vartheta _{0}},\vartheta _{0}+\delta )}< \varepsilon \end{aligned} $$

and

$$ 0< \sqrt{\alpha (\Lambda _{\vartheta _{0}},\vartheta _{0}-\delta )}- \sqrt{\alpha (\Lambda _{\vartheta _{0}},\vartheta _{0})}< \varepsilon . $$

In particular, we have

$$ \Lambda _{\vartheta _{0}}-\varepsilon < \sqrt{\alpha (\Lambda _{ \vartheta _{0}},\vartheta _{0}+\delta )}\quad \mbox{and}\quad \sqrt{\alpha ( \Lambda _{\vartheta _{0}},\vartheta _{0}-\delta )}< \Lambda _{ \vartheta _{0}}+\varepsilon .$$

By the monotonicity of $\Lambda _{\vartheta }$ with respect to ϑ, we get

$$ \Lambda _{\vartheta _{0}-\delta }>\Lambda _{\vartheta _{0}}>\Lambda _{ \vartheta _{0}+\delta }. $$

Thus, using the monotonicity of $\alpha (s,\cdot )$ with respect to s, we obtain

$$ \sqrt{\alpha (\Lambda _{\vartheta _{0}},\vartheta _{0}+\delta )}< \sqrt{\alpha (\Lambda _{\vartheta _{0}+\delta },\vartheta _{0}+\delta )}= \Lambda _{\vartheta _{0}+\delta } $$

and

$$ \sqrt{\alpha (\Lambda _{\vartheta _{0}},\vartheta _{0}-\delta )}> \sqrt{\alpha (\Lambda _{\vartheta _{0}-\delta },\vartheta _{0}-\delta )}= \Lambda _{\vartheta _{0}-\delta }. $$

Chaining the five inequalities above, we immediately get

$$ \Lambda _{\vartheta _{0}}-\varepsilon < \Lambda _{\vartheta _{0}+ \delta }< \Lambda _{\vartheta _{0}-\delta }< \Lambda _{\vartheta _{0}}+ \varepsilon . $$

Then, for any $\vartheta \in (\vartheta _{0}-\delta ,\vartheta _{0}+\delta )$, we arrive at $\Lambda _{\vartheta _{0}}-\varepsilon <\Lambda _{\vartheta }<\Lambda _{ \vartheta _{0}}+\varepsilon$. Hence

$$ \Lambda _{\vartheta }\mbox{ is a continuous function of } \vartheta \in (0,\vartheta _{\mathrm{c}}). $$

(5.6)

Now we study the limit of $\Lambda _{\vartheta }$ as $\vartheta \to 0$. For any $\varepsilon >0$, there exists ${w}\in \mathcal{A}_{0}$ such that

$$ w_{3}\neq 0\quad \mbox{on }\Sigma \quad \mbox{and}\quad \Lambda _{0} -\varepsilon < \sqrt{ g[\!\![\rho ]\!\!]\vert {w}_{3} \vert _{0}^{2} - \Lambda _{0} \Vert \sqrt{\mu } \mathbb{D} {w} \Vert _{0}^{2}/2 }=\Lambda _{0}. $$

(5.7)

In addition,

$$ \Lambda _{\vartheta }< \Lambda _{0}. $$

(5.8)

Thus, making use of (2.6), (5.7), and (5.8), there exists a sufficiently small constant $\vartheta _{1}\in (0,\vartheta _{\mathrm{c}})$ such that, for any $\vartheta \in (0,\vartheta _{1})$,

$$ \Lambda _{0} -\varepsilon < \sqrt{ g[\!\![\rho ]\!\!]\vert {w}_{3} \vert _{0}^{2} - \Lambda _{\vartheta } \Vert \sqrt{\mu } \mathbb{D} {w} \Vert _{0}^{2}/2- \vartheta \vert \nabla _{\mathrm{h}}w_{3} \vert ^{2}_{0}} \leqslant \Lambda _{\vartheta }< \Lambda _{0}. $$

(5.9)

Hence we get

$$ \lim_{\vartheta \rightarrow 0}\Lambda _{\vartheta }= \Lambda _{0}, $$

which, together with (5.6), yields that

$$ \Lambda _{\vartheta }\mbox{ is a continuous function of } \vartheta \in [0,\vartheta _{\mathrm{c}}). $$

(5.10)

Finally, we derive the upper bound (2.1) for $\Lambda _{\vartheta }$.

Recalling the definition of $\vartheta _{\mathrm{c}}$, we see from (3.4) that

$$ g[\!\![\rho ]\!\!]\vert {w}_{3} \vert _{0}^{2}\leqslant \vartheta _{ \mathrm{c}} \vert \nabla _{\mathrm{h}} {w}_{3} \vert _{0}^{2} \quad \mbox{for any } {w}\in H^{1}_{\sigma ,3}. $$

Hence, by virtue of (2.6), for any given $\vartheta \in [0,\vartheta _{\mathrm{c}})$, there exists ${w}^{\vartheta }\in \mathcal{A}_{\vartheta }$ such that

$$ 0 \leqslant \Lambda _{\vartheta }^{2}= F\bigl(w^{\vartheta }, \Lambda _{\vartheta }\bigr)\leqslant g[\!\![\rho ]\!\!](\vartheta _{ \mathrm{c}}-\vartheta ) \bigl\vert {w}^{\vartheta }_{3} \bigr\vert _{0}^{2}/\vartheta _{ \mathrm{c}}- \frac{\Lambda _{\vartheta }}{2} \bigl\Vert \sqrt{\mu }\mathbb{D} {w}^{ \vartheta } \bigr\Vert _{0}^{2}, $$

which yields that

$$ \Lambda _{\vartheta }^{2}+ \frac{\Lambda _{\vartheta }}{2} \bigl\Vert \sqrt{\mu } \mathbb{D} {w}^{\vartheta } \bigr\Vert _{0}^{2} \leqslant g[\!\![\rho ]\!\!](\vartheta _{\mathrm{c}}-\vartheta ) \bigl\vert {w}^{\vartheta }_{3} \bigr\vert _{0}^{2}/ \vartheta _{\mathrm{c}}. $$

(5.11)

By (3.6) and trace estimate (3.9), we can estimate that

$$ \bigl\vert {w}^{\vartheta }_{3} \bigr\vert _{0}^{2} \leqslant \frac{h_{+}}{8 \mu _{+}} \bigl\Vert \sqrt{\mu } \mathbb{D} {w}^{\vartheta } \bigr\Vert _{0}^{2} . $$

Similarly, we also have

$$ \bigl\vert {w}^{\vartheta }_{3} \bigr\vert _{0}^{2} \leqslant \frac{ h_{-}}{8 \mu _{-}} \bigl\Vert \sqrt{\mu } \mathbb{D} {w}^{\vartheta } \bigr\Vert _{0}^{2} . $$

By the above two estimates, we derive from (5.11) that

$$ \Lambda _{\vartheta } \bigl\Vert \sqrt{\mu } \mathbb{D} {w}^{\vartheta } \bigr\Vert _{0}^{2} \leqslant \frac{ g[\!\![\rho ]\!\!](\vartheta _{\mathrm{c}}-\vartheta )}{ 4\vartheta _{\mathrm{c}} } \min \biggl\{ \frac{h_{+}}{\mu _{+}}, \frac{h_{-}}{\mu _{-}} \biggr\} \bigl\Vert \sqrt{\mu } \mathbb{D} {w}^{\vartheta } \bigr\Vert _{0}^{2}, $$

which yields that

$$ \Lambda _{\vartheta }\leqslant \frac{ (\vartheta _{\mathrm{c}}-\vartheta )}{ 4\max \{L_{1}^{2},L_{2}^{2}\} } \min \biggl\{ \frac{h_{+}}{\mu _{+}},\frac{h_{-}}{\mu _{-}} \biggr\} . $$

(5.12)

Noting that $\|\sqrt{\rho }{w}^{\vartheta } \|_{0}=1$, by (3.16), we have

$$ \bigl\vert {w}^{\vartheta }_{3} \bigr\vert _{0}^{2} \leqslant \frac{2}{\sqrt{\rho _{-}}} \bigl\Vert \sqrt{\rho _{-}} {w}^{\vartheta }_{3} \bigr\Vert _{L^{2}(\Omega _{-})} \bigl\Vert \partial _{3}{w}^{\vartheta }_{3} \bigr\Vert _{L^{2}(\Omega _{-})}\leqslant \frac{ \Vert \sqrt{\mu } \mathbb{D}{w}^{\vartheta } \Vert _{0}}{\sqrt{2\rho _{-}\mu _{-}}}. $$

Putting the above estimate into (5.11) and then using Young’s inequality, we get

$$ \Lambda _{\vartheta }^{2} \leqslant \frac{(g[\!\![\rho ]\!\!](\vartheta _{\mathrm{c}}-\vartheta ))^{2} }{4\vartheta _{\mathrm{c}}^{2} \rho _{-}\mu _{-} \Lambda _{\vartheta }}, $$

which yields that

$$ \Lambda _{\vartheta }^{3} \leqslant \frac{ (g[\!\![\rho ]\!\!](\vartheta _{\mathrm{c}}-\vartheta ))^{2} }{4\vartheta _{\mathrm{c}}^{2} \rho _{-}\mu _{-} }. $$

(5.13)

Similarly, we also have

$$ \Lambda _{\vartheta }^{3} \leqslant \frac{ (g[\!\![\rho ]\!\!](\vartheta _{\mathrm{c}}-\vartheta ))^{2} }{4\vartheta _{\mathrm{c}}^{2} \rho _{+}\mu _{+} }. $$

(5.14)

Summing up the above two estimates yields that

$$ \Lambda _{\vartheta }\leqslant \biggl( \frac{ (g[\!\![\rho ]\!\!](\vartheta _{\mathrm{c}}-\vartheta ))^{2} }{4\vartheta _{\mathrm{c}}^{2} \max \{\rho _{+}\mu _{+} , \rho _{-}\mu _{-}\}} \biggr)^{\frac{1}{3}}, $$

(5.15)

which, together with (5.12), implies that

$$ \Lambda _{\vartheta }\leqslant m. $$

(5.16)

Consequently we complete the proof of Theorem 2.2 from (5.5), (5.10), and (5.16).

6 Conclusion

We investigate the effect of (interface) surface tension on the linear Rayleigh–Taylor (RT) instability in stratified incompressible viscous fluids. The existence of linear RT instability solutions with largest growth rate Λ is proved under the instability condition (i.e., the surface tension coefficient ϑ is less than a threshold $\vartheta _{\mathrm{c}}$) by the modified variational method of PDEs. Moreover, we find a new upper bound for Λ. In particular, we directly observe from the upper bound that Λ decreasingly converges to zero as ϑ goes from zero to the threshold $\vartheta _{\mathrm{c}}$. However, we have no idea to provide a numerical example, since it is very difficult for us to do the numerical experiment for the motion of stratified fluids in three dimensions. We will further investigate this topic in the future.