Skip to main content

Stochastic linear quadratic control problem of switching systems with constraints


This paper is devoted to the optimal control problem for stochastic linear switching systems with a quadratic cost functional. A necessary and sufficient condition of optimality for mentioned linear control systems under endpoint constraints is obtained. A linear quadratic controller is simply constructed via a set of stochastic backward Riccati equations.

1 Introduction

The Linear Quadratic (LQ) problem was mathematically formulated and solved, as well as the filtering one, in the 1960s by Kalman [1]. An important advantage of the LQ theory is the existence of explicit feedback forms for optimal state control and the optimal cost value through the Riccati equations. The deterministic Riccati equation was essentially solved by Wonham [2] by applying Bellman’s principle of quasilinearization [3]. A detailed research of stochastic LQ control problems has been performed by Bismut [4]. The existence of a unique solution for the associated Riccati equations was studied in [5].

Switching systems are more advantageous models to describe the noninvariant phenomena with the continuous law of movement and they have gained considerable attention in science and engineering. Examples of these systems include many evolutionary processes, robotics, integrated circuit design, multimedia, manufacturing, power electronics, chaos generators, and air traffic management systems [6, 7]. Optimization problems have also received growing interest among the researchers of deterministic and for stochastic switching control systems [814].

Manifold problems of stochastic optimal control theory have been considered in [1522]. Optimal control problems of switching systems have attracted considerable attention, due to the advantages, for instance, in modeling and improving the transient response on highly complex systems and systems with large uncertainties. The stochastic maximum principle via backward stochastic differential equations is derived in [2327]. The necessary conditions of optimality for stochastic switching systems earlier have been obtained in [2830]. In [31] the linear quadratic control problem has been investigated for a special type of stochastic systems.

In this paper, the LQ problem of stochastic switching systems with restrictions is considered. Ekeland’s variational principle [32] has been used to establish the necessary and sufficient conditions of optimality for a given problem.

2 Statement of main problem

Unless specified otherwise, throughout the paper we use the same notations as in [30].

Consider the following stochastic linear control system:

$$\begin{aligned}& \begin{aligned}[b] dx^{l}(t) ={}& \bigl[ A^{l}(t)x^{l}(t)+B^{l}(t)u^{l}(t)+g^{l}(t) \bigr]\,dt\\ &{}+ \bigl[ C^{l}(t)x^{l}(t)+D^{l}(t)u^{l}(t)+f^{l}(t) \bigr]\,dw^{l}(t),\quad t \in( {t_{l - 1} ,t_{l} }], \end{aligned} \end{aligned}$$
$$\begin{aligned}& x^{l}(t_{l - 1}) = \Phi^{l - 1}(t_{l - 1})x^{l - 1}(t_{l - 1})+K^{l - 1}(t_{l - 1}), \quad l = \overline{2,r} ; x^{1}(t_{0}) = x_{0}, \end{aligned}$$
$$\begin{aligned}& u^{l}(t) \in U_{\partial}^{l} \equiv \bigl\{ {u^{l} ( {\cdot,\cdot} )\in \mathrm{L}_{F^{l}}^{2} \vert u^{l} ( {t,\cdot} )\in U^{l}\subset\mathrm{R}^{m_{l} }, \mbox{a.c.}} \bigr\} . \end{aligned}$$

The elements of \(U_{\partial}^{l} \), are called admissible controls.

Our goal is to find an optimal solution \((x,u)= ( {x^{1},x^{2},\ldots,x^{r},u^{1},u^{2},\ldots,u^{r}} )\) and a switching sequence \(\mathbf{t}=(t_{1} ,t_{2} ,\ldots,t_{r}) \), that minimize the cost functional:

$$ J(u) = E\sum_{l = 1}^{r} \biggl[ \bigl\langle G^{l}x^{l}(t_{l}),x^{l}(t_{l}) \bigr\rangle + \int_{t_{l-1} }^{t_{l} } \bigl( \bigl\langle M^{l}(t)x^{l}(t),x^{l}(t) \bigr\rangle + \bigl\langle N^{l}(t)u^{l}(t),u^{l}(t) \bigr\rangle \bigr)\,dt \biggr], $$

on the decisions of the system (1)-(3) under the conditions:

$$ \textit{E} \bigl\langle q^{l},x^{l}(t_{l}) \bigr\rangle \in Q^{l}, \quad l = 1,\ldots,r, $$

where \(Q^{1},\ldots,Q^{r} \) are a closed convex sets in \(R^{1}\). The elements of matrices \(A^{l}\), \(B^{l}\), \(C^{l}\), \(D^{l}\), \(\Phi^{l}\), \(M^{l}\), \(N^{l}\) and vectors \(G^{l}\), \(K^{l}\), \(g^{l}\), \(f^{l} \) are continuous, bounded functions. \(G^{l}\), \(M^{l}\) are a positively semi-defined matrices, and \(N^{l}\) are positively defined matrices.

\(A_{i}\) represents the set of elements \(\pi^{i} = (t_{0} ,t_{1} ,t_{i} ,x^{1}(t) ,x^{2}(t) ,\ldots,x^{i}(t),u^{1},u^{2},\ldots,u^{i})\) for each \(i=1,\ldots,r \). To describe the main result we need to introduce some concepts, such as a solution of linear switching systems, admissible element of control problem and optimal solution for LQ problem of stochastic switching systems. For a detailed account we refer the reader to [29, 30].

3 Stochastic LQ problem of switching systems

This section is devoted to the investigation of optimal control problems for linear stochastic switching systems with constraints. The LQ problem belongs to a special class of convex control problems for which the maximum principle is a necessary as well as sufficient condition of optimality. The next theorem provides necessary and sufficient conditions of the optimality of stochastic linear switching systems.

Theorem 1

Let \(A_{r}^{0} \) be a set of admissible elements. The elements

$$\pi^{r}=\bigl(t_{0} ,t_{1}, t_{r} ,x^{1}(t) ,x^{2}(t) ,\ldots, x^{r}(t), u^{1}, u^{2}, \ldots, u^{r}\bigr) \in A_{r}^{0} $$

are an optimal solution of problem (1)-(5) if and only if:

  1. (a)

    there exist random processes \((\psi^{l}(t) ,\beta^{l}(t))\in{L}_{F}^{2} (t_{l-1} ,t_{l} ;{R}^{n_{l} })\times{L}_{F}^{2} (t_{l-1} ,t_{l} ;{R}^{n_{l}\times n_{l}})\) which are the solutions of the following stochastic backward equations:

    $$ \left \{ \textstyle\begin{array}{@{}l} d\psi^{l}(t) =- [A^{l*}(t)\psi^{l}(t)+C^{l*}(t)\beta ^{l}(t)-M^{l}(t)x(t) ] \\ \hphantom{d\psi^{l}(t) =}{}+\beta^{l}(t)\,dw^{l}(t) ,\quad t_{l-1} \le t< t_{l} , \\ \psi^{l}(t_{l}) =-\lambda^{l}_{0}G^{l} x^{l}(t_{l})-\lambda^{l}_{1}q^{l} +\psi ^{l+1}(t_{l}) \Phi^{l} (t_{l} ),\quad l=1,\ldots,r-1 , \\ \psi^{r}(t_{r}) =-\lambda^{r}_{0}G^{r} x^{r}(t_{r})-\lambda^{r}_{1}q^{r} ; \end{array}\displaystyle \right . $$
  2. (b)

    the candidate optimal controls \({u}^{l}\in U^{l}\), \(l=\overline{1,r}\), are defined by

    $$ N^{l*}(t)u^{l}(t)=B^{l*}(t) \psi^{l}(t)+D^{l*}(t)\beta^{l}(t), \quad\textit{a.e. }\theta\in[t_{l-1} , t_{l} ] ; $$
  3. (c)

    the following transversality conditions hold:

    $$ \psi^{l+1}(t_{l}) \bigl(\Phi_{t}^{l*} \bigl(t^{l}\bigr) x^{l}(t_{l})+K_{t}^{l*}(t_{l}) \bigr)=0, \quad \textit{a.c.}, l=1,\ldots,r-1 . $$


First we investigate given optimal control problem without endpoint constraints (5). Let \(u^{l}(t)\) and \(\bar{u}^{l}(t)\), \(l = 1,\ldots,r \) be some admissible controls and \(x^{l}(t)\), \(\bar{x}^{l}(t) \) be corresponding trajectories. \(\Delta\bar{u}^{l}(t)\) represents the admissible increment of the control \(u^{l}(t)\). \(\mathbf{t}=(t_{0}, t_{1}, \ldots t_{r} )\) and \(\bar{\mathbf{t}}=(\bar{ t_{0}}, \bar{t_{1}}, \ldots, \bar{t_{r}})\) denote different switching laws. The increment of the cost functional (4) along the admissible control \(\bar{ \mathbf {u}}=(\bar{u}^{1}(t), \bar{u}^{2}(t), \ldots, \bar{u}^{r}(t)) \) looks like

$$\begin{aligned} \bigl\langle J^{\prime}(\mathbf{u}), \bar{\mathbf{u}}- \mathbf{u} \bigr\rangle ={}& E\sum_{l = 1}^{r} \biggl[ \bigl\langle G^{l}x^{l}(t_{l}), \bar{x}^{l}(t_{l})-x^{l}(t_{l}) \bigr\rangle \\ &{}+\int_{t_{l-1} }^{t_{l} } \bigl( \bigl\langle M^{l}(t)x^{l}(t),\bar {x}^{l}(t)-x^{l}(t) \bigr\rangle + \bigl\langle N^{l}(t)u^{l}(t),\bar {u}^{l}(t)-u^{l}(t) \bigr\rangle \bigr)\,dt \biggr]. \end{aligned}$$

By (1)-(2) the increments of the trajectories are defined as

$$ \left \{ \textstyle\begin{array}{@{}l} d(\bar{x}^{l}(t)-x^{l}(t)) \\ \quad= [ A^{l}(t)(\bar{x}^{l}(t)-x^{l}(t))+B^{l}(t)(\bar {u}^{l}(t)-u^{l}(t)) ]\,dt \\ \qquad{}+ [ C^{l}(t)(\bar{x}^{l}(t)-x^{l}(t))+B^{l}(t)(\bar{x}^{l}(t)-x^{l}(t)) ]\,dw^{l}(t),\quad t \in ( \bar{t_{l - 1}} ,\bar{t_{l}} ],\\ \Delta x^{l+1}(t_{l})=\bar{x}^{l+1}(\bar{t_{l}})-x^{l+1}(t_{l})=\Phi^{l}(\bar {t_{l}})\bar{x}^{l}(\bar{t_{l}})-\Phi^{l}(t_{l})x^{l}(t_{l}). \end{array}\displaystyle \right . $$

Let us introduce the stochastic processes \(\psi^{l}(t) \), \(l=1,\ldots,r \), as the solution of the following stochastic backward differential equations:

$$ \left \{ \textstyle\begin{array}{@{}l} d\psi^{l}(t) =- [A^{l*}(t)\psi^{l}(t)+C^{l*}(t)\beta ^{l}(t)-M^{l}(t)x(t) ] +\beta^{l}(t)\,dw^{l}(t) ,\quad t_{l-1} \le t< t_{l} , \\ \psi^{l}(t_{l}) =-G^{l} x^{l}(t_{l}) +\psi^{l+1}(t_{l}) \Phi^{l} (t_{l} ),\quad l=1,\ldots,r-1 , \\ \psi^{r}(t_{r}) =-G^{r} x^{r}(t_{r}). \end{array}\displaystyle \right . $$

According to the Ito formula for each \(l=1,\ldots,r \) the following identity is satisfied:

$$\begin{aligned} &d \bigl\langle \psi^{l}(t_{l}),(\bar{x}^{l}(t_{l})-x^{l}(t_{l}) (\Delta{t_{l}} \bigr\rangle \\ &\quad= \bigl\langle \,d\psi^{l}(t), \bigl(\bar{x}^{l}(t)-x^{l}(t)\bigr)\Delta{t_{l}} \bigr\rangle + \bigl\langle \psi^{l}(t),d\bigl(\bar{x}^{l}(t)-x^{l}(t) \bigr)\Delta{t_{l}} \bigr\rangle \\ &\qquad{}+ \bigl\langle \beta^{l}(t),C^{l}(t) \bigl(\bar{x}^{l}(t)-x^{l}(t)\bigr)\Delta {t_{l}}+D^{l}(t) \bigl(\bar{u}^{l}(t)-u^{l}(t) \bigr)\Delta{t_{l}} \bigr\rangle \,dt. \end{aligned}$$

Integrating the aforementioned equality and taking the expectation of both sides into account in (10) it follows

$$\begin{aligned} &E \bigl\langle \psi^{l}(t_{l}),(\bar{x}^{l}(t_{l})-x^{l}(t_{l}) (\Delta{t_{l}} \bigr\rangle - \bigl\langle \psi^{l}(t_{l-1}), \bigl(\bar {x}^{l}(t_{l-1})-x^{l}(t_{l-1}) \bigr)\Delta{t_{l-1}} \bigr\rangle \\ &\quad=E \int_{t_{l-1} }^{t_{l} } \bigl\langle \,d\psi^{l}(t)+A^{l*}(t) \psi ^{l}(t)+C^{l*}(t)\beta^{l}(t),\bigl( \bar{x}^{l}(t)-x^{l}(t)\bigr)\Delta t_{l} \bigr\rangle \\ &\qquad{}+E \int_{t_{l-1} }^{t_{l} } \bigl\langle B^{l*}(t)\psi ^{l}(t)+D^{l*}(t)\beta^{l}(t),\bigl( \bar{u}^{l}(t)-u^{l}(t)\bigr)\Delta t_{l} \bigr\rangle \,dt. \end{aligned}$$

Due to this equality equation (9) can be rewritten as

$$\begin{aligned} \bigl\langle J^{\prime}(\mathbf{u}),\bar{\mathbf{u}}- \mathbf{u} \bigr\rangle ={}& E\sum_{l = 1}^{r} \biggl[ \bigl\langle G^{l}x^{l}(t_{l}),\bar {x}^{l}(t_{l})-x^{l}(t_{l}) \bigr\rangle + \bigl\langle \psi^{l}(t_{l}),\bar {x}^{l}(t_{l})-x^{l}(t_{l}) \bigr\rangle \\ &{}- E \int_{t_{l-1} }^{t_{l} } \bigl\langle \,d\psi^{l}(t)+A^{l*}(t) \psi ^{l}(t)+C^{l*}(t)\beta^{l}(t),\bigl( \bar{x}^{l}(t)-x^{l}(t)\bigr)\Delta t_{l} \bigr\rangle \\ &{}-E \int_{t_{l-1} }^{t_{l} } \bigl\langle B^{l*}(t)\psi ^{l}(t)+D^{l*}(t)\beta^{l}(t),\bigl( \bar{u}^{l}(t)-u^{l}(t)\bigr)\Delta t_{l} \bigr\rangle \,dt \\ &{}+\int_{t_{l-1} }^{t_{l} } \bigl( \bigl\langle M^{l}(t)x^{l}(t),\bar {x}^{l}(t)-x^{l}(t) \bigr\rangle + \bigl\langle N^{l}(t)u^{l}(t),\bar {u}^{l}(t)-u^{l}(t) \bigr\rangle \bigr)\,dt \biggr]. \end{aligned}$$

Further, using equation (11) we get a more succinct expression:

$$ \bigl\langle J^{\prime}(\mathbf{u}), \bar{\mathbf{u}}- \mathbf{u} \bigr\rangle = E\sum_{l = 1}^{r} \int_{t_{l-1}}^{t_{l}} \bigl\langle N^{l}(t)u^{l}(t)-B^{l*}(t) \psi^{l}(t)-D^{l*}(t)\beta^{l}(t),\bigl(\bar {u}^{l}(t)-u^{l}(t)\bigr)\Delta{t_{l}} \bigr\rangle \,dt. $$

It is well known that a necessary and sufficient condition of optimality for the convex functional is given by \(J^{\prime}(\mathbf{u})=0\). The validity of (7) and (8), hence the necessary conditions of optimality for the considered unrestricted problem (1)-(4) follows from equations (11) and (13). At last, according to the independence of the increments \(\Delta\bar{x}^{l}(t)\), \(\Delta\bar {u}^{l}(t)\), \(\Delta\bar{t_{l}}\), sufficiency follows from equation (12).

To construct the optimality condition of LQ problem (1)-(4) with the right endpoint constraints (5), the above mentioned problem by using Ekeland’s variational principle [32] is converted into a sequence of unconstrained problems. Based on the results already obtained for problem (1)-(4), necessary and sufficient conditions for the sequence of switching systems are established.

To apply Ekeland’s variational principle we introduce the following approximating functional:

$$I_{j} ( {\mathbf{u}} ) = \min_{(c,y^{l}) \in\varepsilon}\sqrt {\sum_{l=1}^{r}{\bigl\vert c^{l} - \varepsilon^{l}_{j} - ES^{l} ( x ,u ,t )\bigr\vert ^{2}} + \sum_{l=1}^{r} \bigl\vert y^{l}- {Eq^{l}x^{l}(t_{l})} \bigr\vert ^{2}}. $$

Here \(S^{l} ( x ,u ,t )= \langle G^{l}x^{l}(t_{l}),x^{l}(t_{l}) \rangle+\int_{t_{l-1} }^{t_{l} } ( \langle M^{l}(t)x^{l}(t),x^{l}(t) \rangle+ \langle N^{l}(t)u^{l}(t),u^{l}(t) \rangle )\,dt\); \(\lim_{j \to\infty}\varepsilon^{l}_{j}=0 \); \(c=c^{1}+\cdots+c^{r}\); \(\varepsilon= \{ {c:c \le J^{0},y^{l}\in Q^{l}} \}\); let \(J^{0}\) be a minimal value of the functional in the problem (1)-(5).

Let \(V^{l} \equiv(U_{\partial}^{l} ,d)\) be the space of controls obtained by means of the following metric:

$$d\bigl(u^{l},v^{l}\bigr) = ( {l \otimes P} ) \bigl\{ { ( {t, \omega} ) \in [ {t_{l - 1} ,t_{l} } ]\times\Omega: \nu_{t}^{l} \ne u_{t}^{l} } \bigr\} . $$

For each \(l = 1,\ldots,r \), let \(V^{l}\) be a complete metric space [32]. □

For the following fact it is significant that we can provide a relation between the sequence of controls from the metric space \(V^{l}\) and the sequence of corresponding trajectories of system (1)-(2).

Lemma 1

([30], Lemma 4.3)

Let \((v^{1,n},\ldots,v^{r,n}) \) be the sequence of admissible controls from \((V^{1},\ldots,V^{r})\), and \((x^{1,n},\ldots,x^{r,n})\) be the sequence of corresponding trajectories of the system (1)-(2). Let for each \(l=1,\ldots,r\) the condition \(d(v^{l,n} ,u ^{l}) \to0\) when \(n\to \infty\) be met. Then

$$\lim_{n \to\infty} \Bigl\{ \sup_{t_{l - 1} \leq t \leq t_{l} } E\bigl\vert {x^{l,n}(t) - x ^{l}(t) } \bigr\vert ^{2} \Bigr\} = 0, $$

where \(x ^{l}(t) \) is a trajectory corresponding to admissible controls \(u ^{l}(t)\), \(l = 1,\ldots,r\).

Due to Ekeland’s variational principle, we see that \((x^{1,j}(t) ,\ldots,x^{r,j}(t), u^{1,j}(t) ,\ldots,u^{r,j}(t) ) \) is a solution of the following problem:

$$ \left \{ \textstyle\begin{array}{@{}l} J_{j}(u) = I_{j} (\mathbf{u})+E\sum_{l = 1}^{r} \sqrt{\varepsilon^{l} _{j}} \int_{t_{l - 1} }^{t_{l} } {\delta(u ^{l}(t) ,u^{l,j}(t) )\,dt} \\ \hphantom{J_{j}(u)}\to\min dx^{l,j}(t) = [ A^{l}(t)x^{l,j}(t)+B^{l}(t)u^{l,j}(t)+g^{l}(t) ]\,dt\\ \hphantom{J_{j}(u)=}{}+ [ C^{l}(t)x^{l,j}(t)+D^{l,j}(t)u^{l}(t) ]\,dw^{l}(t),\quad t \in ( {t_{l - 1} ,t_{l} } ], \\ x^{l,j}(t_{l - 1}) = \Phi^{l - 1}(t_{l - 1})x^{l - 1,j}(t_{l - 1})+K^{l - 1}(t_{l - 1}),\quad l = 2,\ldots,r, \\ x^{1,j}(t_{0}) = x_{0}, \\ u ^{l,j}(t) \in U_{\partial}^{l} . \end{array}\displaystyle \right . $$

\(\delta(u,v)\) is the characteristic function of the set \(\{ u,v \in V^{l}:u \ne v \}\).

Based on (11), it is found that, if \((x^{1,j}(t) ,\ldots,x^{r,j}(t),u^{1,j}(t) ,\ldots,u^{r,j}(t))\) is an optimal solution of problem (14), there exist the random processes \((\psi^{l,j}(t), \beta^{l,j}(t)) \in L_{F^{l}}^{2} (t_{l - 1} ,t_{l} ;R^{n_{l} })\times L_{F^{l}}^{2} (t_{l - 1} ,t_{l} ;R^{n_{l} \times n_{l} })\), which are solutions of the following system:

$$ \left \{ \textstyle\begin{array}{@{}l} d\psi^{l,j}(t) =- [A^{l*}(t)\psi^{l,j}(t)+C^{l*}(t)\beta ^{l,j}(t)-M^{l}(t)x^{l,j}(t) ] \\ \hphantom{d\psi^{l,j}(t) =}{}+\beta^{l}(t)\,dw^{l}(t) , \quad t_{l-1} \le t< t_{l} , \\ \psi^{l,j}(t_{l}) =-\lambda_{0}^{l,j}G^{l} x^{l,j}(t_{l}) -\lambda _{1}^{l,j}q^{l}+\psi^{l+1,j}(t_{l}) \Phi^{l} (t_{l} ),\quad l=1,\ldots,r-1 , \\ \psi^{r,j}(t_{r}) =-\lambda_{0}^{r,j}G^{r} x^{r,j}(t_{r})-\lambda_{1}^{r,j}q^{r}, \end{array}\displaystyle \right . $$

where the non-zero \((\lambda_{0}^{l,j} ,\lambda_{1}^{l,j})\), \(l = 1,\ldots,r\), are defined as

$$ \bigl( \lambda_{0}^{l,j},\lambda_{1}^{l,j} \bigr)= \frac{ ( -c^{l}+\varepsilon^{l}_{j}+ S^{l}(x^{j},u^{j},t) ,-y^{l}+ Eq^{l}x^{l,j}(t_{l}) )}{J_{j}^{0}}; $$


$$J_{j}^{0} = \Biggl( \sum_{l = 1}^{r} \bigl\vert y^{l}-{Eq^{l}x^{l,j}(t_{l})} \bigr\vert ^{2} + \Biggl\vert \sum_{l = 1}^{r} \bigl[{c^{l} - \varepsilon^{l}_{j} - E{S^{l}\bigl(x^{j},u^{j},t\bigr)}}\bigr] \Biggr\vert ^{2} \Biggr)^{1/2}. $$

On the one hand, due to (15), \(\forall \tilde{u}^{l} \in V^{l} \) the following necessary and sufficient condition of optimality for the unconstrained problem (14) holds:

$$ N^{l*}(t)u^{l,j}(t)=B^{l*}(t) \psi^{l,j}(t)+D^{l*}(t)\beta^{l,j}(t), \quad\textit{a.e. }t \in[t_{l-1}, t_{l} ], \textit{a.c.} $$

Besides, based on (15) and (17) we see that the optimal decision of the problem (14) satisfies the transversality condition:

$$ \psi^{l+1,j}(t_{l}) \bigl(\Phi_{t}^{l*}(t_{l}) x^{l,j}(t_{l})+K_{t}^{l*}(t_{l}) \bigr)=0, \quad \textit{a.c.}, l=1,\ldots,r-1. $$

Since \(\sum_{l = 1}^{r} |\lambda_{0}^{l,j}|^{2}+|\lambda_{1}^{l,j} |^{2} = 1 \) exists by (16) \((\lambda_{0}^{l,j} ,\lambda_{1}^{l,j} ) \to(\lambda_{0}^{l} ,\lambda _{1}^{l} )\) if \(j \to\infty\).

The truth of (6) is based upon the following lemma, which can be proved by the same method as the proof of Lemma 4 [33].

Lemma 2

Let \(\psi^{l}(t_{l} ) \) be a solution of system (6), \(\psi^{l,j}(t_{l} )\) be a solution of system (15). If the sequence of controls \((u^{1,j}(t) ,\ldots,u ^{r,j}(t) )\) satisfies the assumptions of the Lemma  1, then

$$E \int_{t_{l - 1} }^{t_{l} } {\bigl\vert \psi^{l,j}(t) - \psi^{l}(t) \bigr\vert ^{2}\,dt + } E \int_{t_{l - 1} }^{t_{l} } {\bigl\vert \beta^{l,j}(t) - \beta^{l}(t) \bigr\vert ^{2}\,dt \to0} ,\quad l = 1,\ldots,r. $$

Based on Lemma 2, we can pass to the weak limit in system (15) and obtain the fulfillment of (6). Following a similar scheme, we take the limits in (17) and (18), and justifications of (7), (8) are derived. Theorem 1 is proved.

4 Riccati equations for switching systems

In the theory of LQ problem, it is very natural to connect the LQ problem with the Riccati equation for the possible feedback design. In this section the optimal control is determined explicitly via a set of stochastic Riccati equations. First, we investigate the feedback design problem in the case \(g^{l}(t)\equiv0\); \(f^{l}(t)\equiv0\) and we search a relation in the form:

$$ \psi^{l}(t)=-p^{l}(t)x^{l}(t), \quad l = 1,\ldots,r, \mbox{a.c.} $$

To determine the stochastic processes \(p^{l}(t)\) we introduce the following theorem.

Theorem 2

Let \(\psi^{l}(t ) \) be a solution of system (6), \(p ^{l}(t) \) be a stochastic process that satisfies (19), defined as the solution of the following differential equation:

$$\begin{aligned} dp^{l}(t) ={}&{-} \bigl[p^{l}(t)A^{l}(t)+A^{l*}(t)p^{l}(t)+ \gamma ^{l}(t)C^{l}(t)+C^{l*}(t) \gamma^{l}(t)+C^{l*}p^{l}(t)C^{l} \\ &{}+M^{l} -\bigl(p^{l}(t)B^{l}(t)+\gamma ^{l}(t)D^{l}(t)+C^{l*}(t)p^{l}(t)D^{l}(t) \bigr) \bigl(N^{l}(t)+D^{l*}(t)p^{l}(t)D^{l}(t) \bigr) \\ &{}\times\bigl(B^{l*}(t)p^{l}(t)+D^{l*}(t) \gamma ^{l}(t)+D^{l*}(t)p^{l}(t)C^{l}(t) \bigr) \bigr] \\ &{}+\gamma^{l}(t)\,dw^{l}(t) ,\quad t_{l-1} \le t< t_{l} . \end{aligned}$$


Suppose that the differential of random processes \(p^{l}(t)\) is defined as

$$dp^{l}(t)=\alpha^{l}(t)\,dt+\gamma^{l}(t)\,dw^{l}(t), \quad l=1,\ldots,r. $$

According to the Ito formula:

$$d\psi^{l}(t)=-dp^{l}(t)x^{l}(t)-p^{l}(t)\,dx^{l}(t)- \gamma ^{l}(t)\bigl[C^{l}(t)x^{l}(t)+ D^{l}(t)u^{l}(t)\bigr]\,dt,\quad l = 1,\ldots,r,\mbox{a.c.} $$

Using (1) and (6) we have

$$\begin{aligned} &\bigl[-A^{l*}\psi^{l}(t)-C^{l*} \beta^{l}(t)+M^{l}(t)x^{l}(t)\bigr]+\beta ^{l}(t)\,dw^{l}(t) \\ &\quad=-\bigl[\alpha^{l}(t)x^{l}(t)\,dt +\gamma ^{l}(t)x^{l}(t)\,dw^{l}(t)+p^{l}(t)A^{l}(t)x^{l}(t)\,dt \\ &\qquad{}+p^{l}(t)B^{l}(t)u^{l}(t)\,dt+p^{l} \bigl(C^{l}(t)x^{l}(t) \\ &\qquad{}+D^{l}(t)u^{l}(t)\bigr)\,dw^{l}(t)+ \gamma^{l}(t) \bigl(C^{l}(t)x^{l}(t)+ D^{l}(t)u^{l}(t)\bigr)\,dt\bigr],\quad l = 1,\ldots,r. \end{aligned}$$

Taking expectation from both side we obtain the following expression for \(\beta^{l}(t)\), \(l = 1,\ldots,r\):

$$ \beta^{l}(t)=-\gamma ^{l}(t)x^{l}-p^{l}(t)C^{l}(t)x^{l}(t)-p^{l}(t)D^{l}(t)u^{l}(t), \quad t\in [t_{l-1},t_{l} ]. $$

By means of simple transformations taking into account (22) equation (21) can be rewritten as follows:

$$\begin{aligned} & \bigl[\alpha^{l}(t)+p^{l}(t)A^{l}(t)+A^{l*}(t)p^{l}(t)+ \gamma ^{l}(t)C^{l}(t) \\ &\quad{}+C^{l*}(t)\gamma^{l}(t)+C^{l*}p^{l}(t)C^{l}+ M^{l} \bigr]x^{l}(t) \\ &\quad{}+\bigl[p^{l}(t)B^{l}(t)+ \gamma ^{l}(t)D^{l}(t)+C^{l*}(t)p^{l}(t)D^{l}(t) \bigr]u^{l}(t)=0. \end{aligned}$$

Considering (19) in equation (6) the optimal control can be defined explicitly:

$$\begin{aligned} &\bigl[ N^{l}(t)+D^{l*}(t)p^{l}(t)D^{l}(t) \bigr]u^{l}(t) \\ &\quad{}+ \bigl[B^{l*}(t)p^{l}(t)+D^{l*}(t) \gamma ^{l}(t)+D^{l*}(t)p^{l}(t)C^{l}(t) \bigr]x^{l}(t)=0. \end{aligned}$$

Hence, Theorem 2 is proved. □

Finally, the feedback design for LQ problem (1)-(5) is obtained by means of the next theorem.

Theorem 3

Let \(\psi^{l}(t )\) be a solution of system (6), \(p ^{l}(t)\), \(\nu^{l}(t) \) be a stochastic processes satisfy to \(\psi^{l}(t)=-[p^{l}(t)x^{l}(t)+\nu^{l}(t)]\), a.c. Then for each l let the random process \(\nu^{l}(t) \) be a solution of the following differential equation:

$$\begin{aligned} & \bigl[\bigl(p^{l}(t)B^{l}(t)+\gamma ^{l}(t)D^{l}(t)+C^{l*}p^{l}(t)D^{l} \bigr) \bigl(N^{l}(t)+D^{l*}(t)p^{l}(t)D^{l}(t) \bigr)^{-1}B^{l*}(t)-A^{l*}(t) \bigr] \nu^{l}(t)\,dt \\ &\quad{} + \bigl[\bigl(p^{l}(t)B^{l}(t)+\gamma ^{l}(t)D^{l}(t)+C^{l*}p^{l}(t)D^{l} \bigr) \bigl(N^{l}(t)+D^{l*}(t)p^{l}(t)D^{l}(t) \bigr)^{-1}D^{l*}(t)-C^{l*}(t) \bigr] \\ &\qquad{}\times \bigl[p^{l*}(t)f^{l}(t)+ \varphi^{l}(t)\bigr]\,dt \\ &\quad= d\nu ^{l}(t) +\bigl[p^{l*}(t)g^{l}(t)- \gamma^{l*}(t)f^{l}(t)\bigr]\,dt+\varphi ^{l}(t)\,dw^{l}(t), \quad \nu^{l}(t_{l})=0 . \end{aligned}$$

Here for each l stochastic process \(p^{l}(t)\) is the solution of (20).


Suppose that random processes \(\nu^{l}(t)\) are defined in the following way:

$$\nu^{l}(t_{l})-\nu^{l}(t_{l-1})= \int_{t_{l-1}}^{t_{l}}\kappa ^{l}(t)\,dt+ \int_{t_{l-1}}^{t_{l}}\varphi^{l}(t)\,dw^{l}(t), \quad l=1,\ldots,r. $$

According to Ito’s formula for each l:

$$\begin{aligned} &d\psi^{l}(t)+\kappa^{l}(t)\,dt+\varphi ^{l}(t)\,dw^{l}(t) \\ &\quad=-dp^{l}(t)x^{l}(t)-p^{l}(t)\,dx^{l}(t) -\gamma^{l*}(t)\bigl[C^{l}(t)x^{l}(t)+ D^{l}(t)u^{l}(t)+f^{l}(t)\bigr]\,dt,\quad \mbox{a.c.} \end{aligned}$$

In view of (6) we obtain the following expression:

$$\beta^{l}(t)=-\gamma ^{l}(t)x^{l}(t)-p^{l}(t)C^{l}(t)x^{l}(t)+p^{l}(t)D^{l}(t)u^{l}(t) +p^{l*}(t)f^{l}(t)+\varphi ^{l}(t), \quad l = 1, \ldots,r,\mbox{a.c.} $$

Substituting this expression for \(\beta^{l}(t)\) into (10), in view of \([N^{l}(t)+D^{l*}(t)p^{l}(t)D^{l}(t)]\) being a positively defined matrix, we have

$$\begin{aligned} u^{l}(t)={}&{-}\bigl[N^{l}(t)+D^{l*}(t)p^{l}(t)D^{l}(t) \bigr]^{-1} \bigl\{ B^{l*}(t)\nu^{l}(t) +p^{l}(t)D^{l}(t)f^{l}(t) \\ &{} +D^{l*}(t)\varphi^{l}(t)+ \bigl[(p^{l}(t)B^{l}(t)+D^{l*}(t) \gamma ^{l}(t)+p^{l}(t)D^{l}(t)C^{l}(t) \bigr] x^{l}(t) \bigr\} . \end{aligned}$$

Integrating both sides of (26) and using equation (27) of \(u^{l}(t)\), bearing in mind that stochastic processes \(p^{l}(t)\), \(\gamma^{l}(t)\) are the solutions of differential equation (20):

$$\begin{aligned} \kappa^{l}(t)={}& \bigl[ \bigl(p^{l}(t)B^{l}(t)+ \gamma ^{l}(t)D^{l}(t)+C^{l*}(t)p^{l}(t)D^{l}(t) \bigr) \bigl(N^{l}(t)+D^{l*}(t)p^{l}(t)D^{l}(t) \bigr)^{-1} \\ &{}\times B^{l*}(t)-A^{l*}(t)\bigr] \nu^{l}(t)+ \bigl[\bigl(p^{l}(t)B^{l}(t)+\gamma ^{l}(t)D^{l}(t)+C^{l*}(t)p^{l}(t)D^{l}(t) \bigr) \\ &{}\times \bigl(N^{l}(t)+D^{l*}(t)p^{l}(t)D^{l}(t) \bigr)^{-1}D^{l*}(t)-C^{l*}(t)\bigr] \bigl[p^{l*}(t)f^{l}(t)+\varphi ^{l}(t) \bigr]-p^{l*}(t)g^{l}(t) \\ &{}-\gamma^{l*}(t)f^{l}(t). \end{aligned}$$

Therefore, the assertion of the theorem is true. □

5 Conclusion

There are a lot relevant applications of LQ problems in fields such as aerospace, biology, economics, management sciences, etc. [3438].

Switching systems provide a natural and convenient theoretical account for mathematical modeling of many complex real phenomena and practical applications. A broad spectrum of the latest research is concerned with optimal control problems of stochastic switching systems [3941].

The LQ problem of switching systems in which the endpoint restrictions are defined with the help of convex closed sets has been investigated. The objective of the present research is to give an explicit solution to the LQ problem of stochastic switching systems of which drift and diffusion coefficients comprise non-homogeneous terms. The results developed in this study can be viewed as an extension of the problems formulated in [42, 43] for stochastic switching systems.


  1. Kalman, RE: Contributions to the theory of optimal control. Bol. Soc. Mat. Mexicana 5, 102-119 (1960)

    MathSciNet  MATH  Google Scholar 

  2. Wonham, WM: On a matrix Riccati equation of stochastic control. SIAM J. Control Optim. 6, 312-326 (1968)

    Article  MathSciNet  MATH  Google Scholar 

  3. Bellman, R: Functional equations in the theory of dynamic programming, positivity and quasilinearity. Proc. Natl. Acad. Sci. USA 41, 743-746 (1955)

    Article  MATH  Google Scholar 

  4. Bismut, JM: Linear quadratic optimal stochastic control with random coefficients. SIAM J. Control 14, 419-444 (1976)

    Article  MathSciNet  MATH  Google Scholar 

  5. Bismut, JM: An introductory approach to duality in optimal stochastic control. SIAM Rev. 20, 62-78 (1978)

    Article  MathSciNet  MATH  Google Scholar 

  6. Boukas, E: Stochastic Switching Systems: Analysis and Design. Birkhäuser, Basel (2006)

    MATH  Google Scholar 

  7. Liberzon, D: Switching in Systems and Control. Birkhäuser, Basel (2003)

    Book  MATH  Google Scholar 

  8. Bengea, SC, Raymond, AC: Optimal control of switching systems. Automatica 41, 11-27 (2005)

    MathSciNet  MATH  Google Scholar 

  9. Capuzzo, DI, Evans, LC: Optimal switching for ordinary differential equations. SIAM J. Control Optim. 22, 143-161 (1984)

    Article  MathSciNet  MATH  Google Scholar 

  10. Seidmann, TI: Optimal control for switching systems. In: Proceedings of the 21st Annual Conference on in Formations Science and Systems, pp. 485-489 (1987)

    Google Scholar 

  11. Hoek, J, Elliott, RJ: American option prices in a Markov chain model. Appl. Stoch. Models Bus. Ind. 28, 35-39 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  12. Azevedo, N, Pinheiro, D, Weber, G-W: Dynamic programming for a Markov-switching jump-diffusion. J. Comput. Appl. Math. 267, 1-19 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  13. Temocin, B, Weber, G-W: Optimal control of stochastic hybrid system with jumps: a numerical approximation. J. Comput. Appl. Math. 259, 443-451 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  14. Xu, X, Antsaklis, PJ: Results and perspectives on computational methods for optimal control of switched systems. In: Maler, O, Pnueli, A (eds.) Hybrid Systems: Computation and Control. Lecture Notes in Computer Science, vol. 2623, pp. 540-556 (2003)

    Chapter  Google Scholar 

  15. Haussman, UG: General necessary conditions for optimal control of stochastic systems. In: Stochastic Systems: Modeling, Identification and Optimization, II. Mathematical Programming Studies, vol. 6, pp. 30-48 (1976)

    Chapter  Google Scholar 

  16. Kek, SL, Teo, LK, Ismail, AM: An integrated optimal control algorithm for discrete-time nonlinear stochastic system. Int. J. Control 83(12), 2536-2545 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  17. Kushner, HJ: Necessary conditions for continuous parameter stochastic optimization problems. SIAM J. Control 10, 550-565 (1976)

    Article  MathSciNet  MATH  Google Scholar 

  18. Pham, H: On some recent aspects of stochastic control and their applications. Probab. Surv. 2, 506-549 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  19. Hafayed, M, Abbas, S, Abba, A: On mean-field partial information maximum principle of optimal control for stochastic systems with Lévy processes. J. Optim. Theory Appl. 167, 1051-1069 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  20. Shen, Y, Siu, TK: The maximum principle for a jump-diffusion mean-field model and its application to the mean-variance problem. Nonlinear Anal. 86, 58-73 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  21. Wang, G, Zhang, C, Zhang, W: Stochastic maximum principle for mean-field type optimal control under partial information. IEEE Trans. Autom. Control 59(2), 522-528 (2014)

    Article  MathSciNet  Google Scholar 

  22. Yong, J, Zhou, XY: Stochastic Controls: Hamiltonian Systems and HJB Equations. Springer, New York (1999)

    Book  MATH  Google Scholar 

  23. Borkar, V: Controlled diffusion processes. Probab. Surv. 2(4), 213-244 (2005)

    MathSciNet  MATH  Google Scholar 

  24. Makhmudov, NI: General necessary optimality conditions for stochastic systems with controllable diffusion. In: Statistics and Control of Random Processes, pp. 135-138. Nauka, Moscow (1989)

    Google Scholar 

  25. Peng, S: A general stochastic maximum principle for optimal control problem. SIAM J. Control Optim. 28, 966-979 (1990)

    Article  MathSciNet  MATH  Google Scholar 

  26. Hafayed, M, Veverka, P, Abbas, S: On near-optimal necessary and sufficient conditions for forward-backward stochastic systems with jumps, with applications to finance. Appl. Math. 59(4), 407-440 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  27. Hafayed, M, Abba, A, Boukaf, S: On Zhou’s maximum principle for near-optimal control of mean-field forward-backward stochastic systems with jumps and its applications. Int. J. Model. Identif. Control 25(1), 1-16 (2016)

    Article  Google Scholar 

  28. Aghayeva, C, Abushov, Q: The maximum principle for the nonlinear stochastic optimal control problem of switching systems. J. Glob. Optim. 56(2), 341-352 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  29. Abushov, Q, Aghayeva, C: Stochastic maximum principle for the nonlinear optimal control problem of switching systems. J. Comput. Appl. Math. 259, 371-376 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  30. Aghayeva, C: Necessary conditions of optimality for stochastic switching control systems. Dyn. Syst. Appl. 24(3), 243-258 (2015)

    Google Scholar 

  31. Aghayeva, C, Abushov, Q: Linear-square stochastic optimal control problem with variable delay on control and state. Transactions ANAS, math.- ph. series, Informatics and Control Problems 25(3), 204-208 (2005)

    Google Scholar 

  32. Ekeland, I: On the variational principle. J. Math. Anal. Appl. 47, 324-353 (1974)

    Article  MathSciNet  MATH  Google Scholar 

  33. Aghayeva, C, Abushov, Q: The maximum principle for some nonlinear stochastic control system with variable structure. Theory Stoch. Process. 32(1), 1-11 (2010)

    MathSciNet  MATH  Google Scholar 

  34. Bensoussan, A, Delfour, MC, Mitter, SK: The linear quadratic optimal control problem for infinite dimensional systems over an infinite horizon; survey and examples. In: Proceedings of the IEEE Conference on Decision and Control and the 15th Symposium on Adaptive Processes, pp. 746-751. Inst. Electr. Electron. Engrs., New York (1976)

    Google Scholar 

  35. Kohlmann, M, Zhou, X: Relationship between backward stochastic differential equations and stochastic controls: a linear-quadratic approach. SIAM J. Control Optim. 38, 1392-1407 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  36. Turan, O: An exergy way to quantify sustainability metrics for a high bypass turbofan engine. Energy 86, 722-736 (2015)

    Article  Google Scholar 

  37. Yuan, C, Wu, F: Hybrid control for switched linear systems with average dwell time. IEEE Trans. Autom. Control 60(1), 240-245 (2015)

    Article  MathSciNet  Google Scholar 

  38. Sahin, O: Optimum arrival routes for flight efficiency. J. Power Energy Eng. 3, 449-452 (2015)

    Article  Google Scholar 

  39. Azhmyakov, V, Attia, SA, Raisch, J: On the maximum principle for impulsive hybrid system. Lect. Notes Comput. Sci. 4981, 30-42 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  40. Ghomanjani, F, HadiFarahi, M: Optimal control of switched systems based on Bezier control points. Int. J. Intell. Syst. Appl. 7, 16-22 (2012)

    Google Scholar 

  41. Wang, S, Wu, Z: Maximum principle for optimal control problems of forward-backward regime-switching systems involving impulse controls. Math. Probl. Eng. 2015, 892304 (2015)

    MathSciNet  Google Scholar 

  42. Debrabant, K, Jakobsen, ER: Semi-Lagrangian schemes for linear and fully nonlinear diffusion equations. Math. Comput. 82, 1433-1462 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  43. Su, Y, Huang, J: Stability of a class of linear switching systems with applications to two consensus problems. IEEE Trans. Autom. Control 57(6), 1420-1430 (2012)

    Article  MathSciNet  Google Scholar 

Download references


The author thanks the two anonymous reviewers whose comments and suggestions helped improve this manuscript. The research underlying this paper is supported by the Scientific Research Project No. 1505F202 of Anadolu University, Turkey.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Charkaz Aghayeva.

Additional information

Competing interests

The author declares to have no competing interests.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aghayeva, C. Stochastic linear quadratic control problem of switching systems with constraints. J Inequal Appl 2016, 100 (2016).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: