Open Access

Stochastic linear quadratic control problem of switching systems with constraints

Journal of Inequalities and Applications20162016:100

DOI: 10.1186/s13660-016-1046-8

Received: 26 January 2016

Accepted: 18 March 2016

Published: 31 March 2016


This paper is devoted to the optimal control problem for stochastic linear switching systems with a quadratic cost functional. A necessary and sufficient condition of optimality for mentioned linear control systems under endpoint constraints is obtained. A linear quadratic controller is simply constructed via a set of stochastic backward Riccati equations.


stochastic linear system conditions of optimality switching systems transversality conditions

1 Introduction

The Linear Quadratic (LQ) problem was mathematically formulated and solved, as well as the filtering one, in the 1960s by Kalman [1]. An important advantage of the LQ theory is the existence of explicit feedback forms for optimal state control and the optimal cost value through the Riccati equations. The deterministic Riccati equation was essentially solved by Wonham [2] by applying Bellman’s principle of quasilinearization [3]. A detailed research of stochastic LQ control problems has been performed by Bismut [4]. The existence of a unique solution for the associated Riccati equations was studied in [5].

Switching systems are more advantageous models to describe the noninvariant phenomena with the continuous law of movement and they have gained considerable attention in science and engineering. Examples of these systems include many evolutionary processes, robotics, integrated circuit design, multimedia, manufacturing, power electronics, chaos generators, and air traffic management systems [6, 7]. Optimization problems have also received growing interest among the researchers of deterministic and for stochastic switching control systems [814].

Manifold problems of stochastic optimal control theory have been considered in [1522]. Optimal control problems of switching systems have attracted considerable attention, due to the advantages, for instance, in modeling and improving the transient response on highly complex systems and systems with large uncertainties. The stochastic maximum principle via backward stochastic differential equations is derived in [2327]. The necessary conditions of optimality for stochastic switching systems earlier have been obtained in [2830]. In [31] the linear quadratic control problem has been investigated for a special type of stochastic systems.

In this paper, the LQ problem of stochastic switching systems with restrictions is considered. Ekeland’s variational principle [32] has been used to establish the necessary and sufficient conditions of optimality for a given problem.

2 Statement of main problem

Unless specified otherwise, throughout the paper we use the same notations as in [30].

Consider the following stochastic linear control system:
$$\begin{aligned}& \begin{aligned}[b] dx^{l}(t) ={}& \bigl[ A^{l}(t)x^{l}(t)+B^{l}(t)u^{l}(t)+g^{l}(t) \bigr]\,dt\\ &{}+ \bigl[ C^{l}(t)x^{l}(t)+D^{l}(t)u^{l}(t)+f^{l}(t) \bigr]\,dw^{l}(t),\quad t \in( {t_{l - 1} ,t_{l} }], \end{aligned} \end{aligned}$$
$$\begin{aligned}& x^{l}(t_{l - 1}) = \Phi^{l - 1}(t_{l - 1})x^{l - 1}(t_{l - 1})+K^{l - 1}(t_{l - 1}), \quad l = \overline{2,r} ; x^{1}(t_{0}) = x_{0}, \end{aligned}$$
$$\begin{aligned}& u^{l}(t) \in U_{\partial}^{l} \equiv \bigl\{ {u^{l} ( {\cdot,\cdot} )\in \mathrm{L}_{F^{l}}^{2} \vert u^{l} ( {t,\cdot} )\in U^{l}\subset\mathrm{R}^{m_{l} }, \mbox{a.c.}} \bigr\} . \end{aligned}$$
The elements of \(U_{\partial}^{l} \), are called admissible controls.
Our goal is to find an optimal solution \((x,u)= ( {x^{1},x^{2},\ldots,x^{r},u^{1},u^{2},\ldots,u^{r}} )\) and a switching sequence \(\mathbf{t}=(t_{1} ,t_{2} ,\ldots,t_{r}) \), that minimize the cost functional:
$$ J(u) = E\sum_{l = 1}^{r} \biggl[ \bigl\langle G^{l}x^{l}(t_{l}),x^{l}(t_{l}) \bigr\rangle + \int_{t_{l-1} }^{t_{l} } \bigl( \bigl\langle M^{l}(t)x^{l}(t),x^{l}(t) \bigr\rangle + \bigl\langle N^{l}(t)u^{l}(t),u^{l}(t) \bigr\rangle \bigr)\,dt \biggr], $$
on the decisions of the system (1)-(3) under the conditions:
$$ \textit{E} \bigl\langle q^{l},x^{l}(t_{l}) \bigr\rangle \in Q^{l}, \quad l = 1,\ldots,r, $$
where \(Q^{1},\ldots,Q^{r} \) are a closed convex sets in \(R^{1}\). The elements of matrices \(A^{l}\), \(B^{l}\), \(C^{l}\), \(D^{l}\), \(\Phi^{l}\), \(M^{l}\), \(N^{l}\) and vectors \(G^{l}\), \(K^{l}\), \(g^{l}\), \(f^{l} \) are continuous, bounded functions. \(G^{l}\), \(M^{l}\) are a positively semi-defined matrices, and \(N^{l}\) are positively defined matrices.

\(A_{i}\) represents the set of elements \(\pi^{i} = (t_{0} ,t_{1} ,t_{i} ,x^{1}(t) ,x^{2}(t) ,\ldots,x^{i}(t),u^{1},u^{2},\ldots,u^{i})\) for each \(i=1,\ldots,r \). To describe the main result we need to introduce some concepts, such as a solution of linear switching systems, admissible element of control problem and optimal solution for LQ problem of stochastic switching systems. For a detailed account we refer the reader to [29, 30].

3 Stochastic LQ problem of switching systems

This section is devoted to the investigation of optimal control problems for linear stochastic switching systems with constraints. The LQ problem belongs to a special class of convex control problems for which the maximum principle is a necessary as well as sufficient condition of optimality. The next theorem provides necessary and sufficient conditions of the optimality of stochastic linear switching systems.

Theorem 1

Let \(A_{r}^{0} \) be a set of admissible elements. The elements
$$\pi^{r}=\bigl(t_{0} ,t_{1}, t_{r} ,x^{1}(t) ,x^{2}(t) ,\ldots, x^{r}(t), u^{1}, u^{2}, \ldots, u^{r}\bigr) \in A_{r}^{0} $$
are an optimal solution of problem (1)-(5) if and only if:
  1. (a)
    there exist random processes \((\psi^{l}(t) ,\beta^{l}(t))\in{L}_{F}^{2} (t_{l-1} ,t_{l} ;{R}^{n_{l} })\times{L}_{F}^{2} (t_{l-1} ,t_{l} ;{R}^{n_{l}\times n_{l}})\) which are the solutions of the following stochastic backward equations:
    $$ \left \{ \textstyle\begin{array}{@{}l} d\psi^{l}(t) =- [A^{l*}(t)\psi^{l}(t)+C^{l*}(t)\beta ^{l}(t)-M^{l}(t)x(t) ] \\ \hphantom{d\psi^{l}(t) =}{}+\beta^{l}(t)\,dw^{l}(t) ,\quad t_{l-1} \le t< t_{l} , \\ \psi^{l}(t_{l}) =-\lambda^{l}_{0}G^{l} x^{l}(t_{l})-\lambda^{l}_{1}q^{l} +\psi ^{l+1}(t_{l}) \Phi^{l} (t_{l} ),\quad l=1,\ldots,r-1 , \\ \psi^{r}(t_{r}) =-\lambda^{r}_{0}G^{r} x^{r}(t_{r})-\lambda^{r}_{1}q^{r} ; \end{array}\displaystyle \right . $$
  2. (b)
    the candidate optimal controls \({u}^{l}\in U^{l}\), \(l=\overline{1,r}\), are defined by
    $$ N^{l*}(t)u^{l}(t)=B^{l*}(t) \psi^{l}(t)+D^{l*}(t)\beta^{l}(t), \quad\textit{a.e. }\theta\in[t_{l-1} , t_{l} ] ; $$
  3. (c)
    the following transversality conditions hold:
    $$ \psi^{l+1}(t_{l}) \bigl(\Phi_{t}^{l*} \bigl(t^{l}\bigr) x^{l}(t_{l})+K_{t}^{l*}(t_{l}) \bigr)=0, \quad \textit{a.c.}, l=1,\ldots,r-1 . $$


First we investigate given optimal control problem without endpoint constraints (5). Let \(u^{l}(t)\) and \(\bar{u}^{l}(t)\), \(l = 1,\ldots,r \) be some admissible controls and \(x^{l}(t)\), \(\bar{x}^{l}(t) \) be corresponding trajectories. \(\Delta\bar{u}^{l}(t)\) represents the admissible increment of the control \(u^{l}(t)\). \(\mathbf{t}=(t_{0}, t_{1}, \ldots t_{r} )\) and \(\bar{\mathbf{t}}=(\bar{ t_{0}}, \bar{t_{1}}, \ldots, \bar{t_{r}})\) denote different switching laws. The increment of the cost functional (4) along the admissible control \(\bar{ \mathbf {u}}=(\bar{u}^{1}(t), \bar{u}^{2}(t), \ldots, \bar{u}^{r}(t)) \) looks like
$$\begin{aligned} \bigl\langle J^{\prime}(\mathbf{u}), \bar{\mathbf{u}}- \mathbf{u} \bigr\rangle ={}& E\sum_{l = 1}^{r} \biggl[ \bigl\langle G^{l}x^{l}(t_{l}), \bar{x}^{l}(t_{l})-x^{l}(t_{l}) \bigr\rangle \\ &{}+\int_{t_{l-1} }^{t_{l} } \bigl( \bigl\langle M^{l}(t)x^{l}(t),\bar {x}^{l}(t)-x^{l}(t) \bigr\rangle + \bigl\langle N^{l}(t)u^{l}(t),\bar {u}^{l}(t)-u^{l}(t) \bigr\rangle \bigr)\,dt \biggr]. \end{aligned}$$
By (1)-(2) the increments of the trajectories are defined as
$$ \left \{ \textstyle\begin{array}{@{}l} d(\bar{x}^{l}(t)-x^{l}(t)) \\ \quad= [ A^{l}(t)(\bar{x}^{l}(t)-x^{l}(t))+B^{l}(t)(\bar {u}^{l}(t)-u^{l}(t)) ]\,dt \\ \qquad{}+ [ C^{l}(t)(\bar{x}^{l}(t)-x^{l}(t))+B^{l}(t)(\bar{x}^{l}(t)-x^{l}(t)) ]\,dw^{l}(t),\quad t \in ( \bar{t_{l - 1}} ,\bar{t_{l}} ],\\ \Delta x^{l+1}(t_{l})=\bar{x}^{l+1}(\bar{t_{l}})-x^{l+1}(t_{l})=\Phi^{l}(\bar {t_{l}})\bar{x}^{l}(\bar{t_{l}})-\Phi^{l}(t_{l})x^{l}(t_{l}). \end{array}\displaystyle \right . $$
Let us introduce the stochastic processes \(\psi^{l}(t) \), \(l=1,\ldots,r \), as the solution of the following stochastic backward differential equations:
$$ \left \{ \textstyle\begin{array}{@{}l} d\psi^{l}(t) =- [A^{l*}(t)\psi^{l}(t)+C^{l*}(t)\beta ^{l}(t)-M^{l}(t)x(t) ] +\beta^{l}(t)\,dw^{l}(t) ,\quad t_{l-1} \le t< t_{l} , \\ \psi^{l}(t_{l}) =-G^{l} x^{l}(t_{l}) +\psi^{l+1}(t_{l}) \Phi^{l} (t_{l} ),\quad l=1,\ldots,r-1 , \\ \psi^{r}(t_{r}) =-G^{r} x^{r}(t_{r}). \end{array}\displaystyle \right . $$
According to the Ito formula for each \(l=1,\ldots,r \) the following identity is satisfied:
$$\begin{aligned} &d \bigl\langle \psi^{l}(t_{l}),(\bar{x}^{l}(t_{l})-x^{l}(t_{l}) (\Delta{t_{l}} \bigr\rangle \\ &\quad= \bigl\langle \,d\psi^{l}(t), \bigl(\bar{x}^{l}(t)-x^{l}(t)\bigr)\Delta{t_{l}} \bigr\rangle + \bigl\langle \psi^{l}(t),d\bigl(\bar{x}^{l}(t)-x^{l}(t) \bigr)\Delta{t_{l}} \bigr\rangle \\ &\qquad{}+ \bigl\langle \beta^{l}(t),C^{l}(t) \bigl(\bar{x}^{l}(t)-x^{l}(t)\bigr)\Delta {t_{l}}+D^{l}(t) \bigl(\bar{u}^{l}(t)-u^{l}(t) \bigr)\Delta{t_{l}} \bigr\rangle \,dt. \end{aligned}$$
Integrating the aforementioned equality and taking the expectation of both sides into account in (10) it follows
$$\begin{aligned} &E \bigl\langle \psi^{l}(t_{l}),(\bar{x}^{l}(t_{l})-x^{l}(t_{l}) (\Delta{t_{l}} \bigr\rangle - \bigl\langle \psi^{l}(t_{l-1}), \bigl(\bar {x}^{l}(t_{l-1})-x^{l}(t_{l-1}) \bigr)\Delta{t_{l-1}} \bigr\rangle \\ &\quad=E \int_{t_{l-1} }^{t_{l} } \bigl\langle \,d\psi^{l}(t)+A^{l*}(t) \psi ^{l}(t)+C^{l*}(t)\beta^{l}(t),\bigl( \bar{x}^{l}(t)-x^{l}(t)\bigr)\Delta t_{l} \bigr\rangle \\ &\qquad{}+E \int_{t_{l-1} }^{t_{l} } \bigl\langle B^{l*}(t)\psi ^{l}(t)+D^{l*}(t)\beta^{l}(t),\bigl( \bar{u}^{l}(t)-u^{l}(t)\bigr)\Delta t_{l} \bigr\rangle \,dt. \end{aligned}$$
Due to this equality equation (9) can be rewritten as
$$\begin{aligned} \bigl\langle J^{\prime}(\mathbf{u}),\bar{\mathbf{u}}- \mathbf{u} \bigr\rangle ={}& E\sum_{l = 1}^{r} \biggl[ \bigl\langle G^{l}x^{l}(t_{l}),\bar {x}^{l}(t_{l})-x^{l}(t_{l}) \bigr\rangle + \bigl\langle \psi^{l}(t_{l}),\bar {x}^{l}(t_{l})-x^{l}(t_{l}) \bigr\rangle \\ &{}- E \int_{t_{l-1} }^{t_{l} } \bigl\langle \,d\psi^{l}(t)+A^{l*}(t) \psi ^{l}(t)+C^{l*}(t)\beta^{l}(t),\bigl( \bar{x}^{l}(t)-x^{l}(t)\bigr)\Delta t_{l} \bigr\rangle \\ &{}-E \int_{t_{l-1} }^{t_{l} } \bigl\langle B^{l*}(t)\psi ^{l}(t)+D^{l*}(t)\beta^{l}(t),\bigl( \bar{u}^{l}(t)-u^{l}(t)\bigr)\Delta t_{l} \bigr\rangle \,dt \\ &{}+\int_{t_{l-1} }^{t_{l} } \bigl( \bigl\langle M^{l}(t)x^{l}(t),\bar {x}^{l}(t)-x^{l}(t) \bigr\rangle + \bigl\langle N^{l}(t)u^{l}(t),\bar {u}^{l}(t)-u^{l}(t) \bigr\rangle \bigr)\,dt \biggr]. \end{aligned}$$
Further, using equation (11) we get a more succinct expression:
$$ \bigl\langle J^{\prime}(\mathbf{u}), \bar{\mathbf{u}}- \mathbf{u} \bigr\rangle = E\sum_{l = 1}^{r} \int_{t_{l-1}}^{t_{l}} \bigl\langle N^{l}(t)u^{l}(t)-B^{l*}(t) \psi^{l}(t)-D^{l*}(t)\beta^{l}(t),\bigl(\bar {u}^{l}(t)-u^{l}(t)\bigr)\Delta{t_{l}} \bigr\rangle \,dt. $$

It is well known that a necessary and sufficient condition of optimality for the convex functional is given by \(J^{\prime}(\mathbf{u})=0\). The validity of (7) and (8), hence the necessary conditions of optimality for the considered unrestricted problem (1)-(4) follows from equations (11) and (13). At last, according to the independence of the increments \(\Delta\bar{x}^{l}(t)\), \(\Delta\bar {u}^{l}(t)\), \(\Delta\bar{t_{l}}\), sufficiency follows from equation (12).

To construct the optimality condition of LQ problem (1)-(4) with the right endpoint constraints (5), the above mentioned problem by using Ekeland’s variational principle [32] is converted into a sequence of unconstrained problems. Based on the results already obtained for problem (1)-(4), necessary and sufficient conditions for the sequence of switching systems are established.

To apply Ekeland’s variational principle we introduce the following approximating functional:
$$I_{j} ( {\mathbf{u}} ) = \min_{(c,y^{l}) \in\varepsilon}\sqrt {\sum_{l=1}^{r}{\bigl\vert c^{l} - \varepsilon^{l}_{j} - ES^{l} ( x ,u ,t )\bigr\vert ^{2}} + \sum_{l=1}^{r} \bigl\vert y^{l}- {Eq^{l}x^{l}(t_{l})} \bigr\vert ^{2}}. $$
Here \(S^{l} ( x ,u ,t )= \langle G^{l}x^{l}(t_{l}),x^{l}(t_{l}) \rangle+\int_{t_{l-1} }^{t_{l} } ( \langle M^{l}(t)x^{l}(t),x^{l}(t) \rangle+ \langle N^{l}(t)u^{l}(t),u^{l}(t) \rangle )\,dt\); \(\lim_{j \to\infty}\varepsilon^{l}_{j}=0 \); \(c=c^{1}+\cdots+c^{r}\); \(\varepsilon= \{ {c:c \le J^{0},y^{l}\in Q^{l}} \}\); let \(J^{0}\) be a minimal value of the functional in the problem (1)-(5).
Let \(V^{l} \equiv(U_{\partial}^{l} ,d)\) be the space of controls obtained by means of the following metric:
$$d\bigl(u^{l},v^{l}\bigr) = ( {l \otimes P} ) \bigl\{ { ( {t, \omega} ) \in [ {t_{l - 1} ,t_{l} } ]\times\Omega: \nu_{t}^{l} \ne u_{t}^{l} } \bigr\} . $$
For each \(l = 1,\ldots,r \), let \(V^{l}\) be a complete metric space [32]. □

For the following fact it is significant that we can provide a relation between the sequence of controls from the metric space \(V^{l}\) and the sequence of corresponding trajectories of system (1)-(2).

Lemma 1

([30], Lemma 4.3)

Let \((v^{1,n},\ldots,v^{r,n}) \) be the sequence of admissible controls from \((V^{1},\ldots,V^{r})\), and \((x^{1,n},\ldots,x^{r,n})\) be the sequence of corresponding trajectories of the system (1)-(2). Let for each \(l=1,\ldots,r\) the condition \(d(v^{l,n} ,u ^{l}) \to0\) when \(n\to \infty\) be met. Then
$$\lim_{n \to\infty} \Bigl\{ \sup_{t_{l - 1} \leq t \leq t_{l} } E\bigl\vert {x^{l,n}(t) - x ^{l}(t) } \bigr\vert ^{2} \Bigr\} = 0, $$
where \(x ^{l}(t) \) is a trajectory corresponding to admissible controls \(u ^{l}(t)\), \(l = 1,\ldots,r\).
Due to Ekeland’s variational principle, we see that \((x^{1,j}(t) ,\ldots,x^{r,j}(t), u^{1,j}(t) ,\ldots,u^{r,j}(t) ) \) is a solution of the following problem:
$$ \left \{ \textstyle\begin{array}{@{}l} J_{j}(u) = I_{j} (\mathbf{u})+E\sum_{l = 1}^{r} \sqrt{\varepsilon^{l} _{j}} \int_{t_{l - 1} }^{t_{l} } {\delta(u ^{l}(t) ,u^{l,j}(t) )\,dt} \\ \hphantom{J_{j}(u)}\to\min dx^{l,j}(t) = [ A^{l}(t)x^{l,j}(t)+B^{l}(t)u^{l,j}(t)+g^{l}(t) ]\,dt\\ \hphantom{J_{j}(u)=}{}+ [ C^{l}(t)x^{l,j}(t)+D^{l,j}(t)u^{l}(t) ]\,dw^{l}(t),\quad t \in ( {t_{l - 1} ,t_{l} } ], \\ x^{l,j}(t_{l - 1}) = \Phi^{l - 1}(t_{l - 1})x^{l - 1,j}(t_{l - 1})+K^{l - 1}(t_{l - 1}),\quad l = 2,\ldots,r, \\ x^{1,j}(t_{0}) = x_{0}, \\ u ^{l,j}(t) \in U_{\partial}^{l} . \end{array}\displaystyle \right . $$
\(\delta(u,v)\) is the characteristic function of the set \(\{ u,v \in V^{l}:u \ne v \}\).
Based on (11), it is found that, if \((x^{1,j}(t) ,\ldots,x^{r,j}(t),u^{1,j}(t) ,\ldots,u^{r,j}(t))\) is an optimal solution of problem (14), there exist the random processes \((\psi^{l,j}(t), \beta^{l,j}(t)) \in L_{F^{l}}^{2} (t_{l - 1} ,t_{l} ;R^{n_{l} })\times L_{F^{l}}^{2} (t_{l - 1} ,t_{l} ;R^{n_{l} \times n_{l} })\), which are solutions of the following system:
$$ \left \{ \textstyle\begin{array}{@{}l} d\psi^{l,j}(t) =- [A^{l*}(t)\psi^{l,j}(t)+C^{l*}(t)\beta ^{l,j}(t)-M^{l}(t)x^{l,j}(t) ] \\ \hphantom{d\psi^{l,j}(t) =}{}+\beta^{l}(t)\,dw^{l}(t) , \quad t_{l-1} \le t< t_{l} , \\ \psi^{l,j}(t_{l}) =-\lambda_{0}^{l,j}G^{l} x^{l,j}(t_{l}) -\lambda _{1}^{l,j}q^{l}+\psi^{l+1,j}(t_{l}) \Phi^{l} (t_{l} ),\quad l=1,\ldots,r-1 , \\ \psi^{r,j}(t_{r}) =-\lambda_{0}^{r,j}G^{r} x^{r,j}(t_{r})-\lambda_{1}^{r,j}q^{r}, \end{array}\displaystyle \right . $$
where the non-zero \((\lambda_{0}^{l,j} ,\lambda_{1}^{l,j})\), \(l = 1,\ldots,r\), are defined as
$$ \bigl( \lambda_{0}^{l,j},\lambda_{1}^{l,j} \bigr)= \frac{ ( -c^{l}+\varepsilon^{l}_{j}+ S^{l}(x^{j},u^{j},t) ,-y^{l}+ Eq^{l}x^{l,j}(t_{l}) )}{J_{j}^{0}}; $$
$$J_{j}^{0} = \Biggl( \sum_{l = 1}^{r} \bigl\vert y^{l}-{Eq^{l}x^{l,j}(t_{l})} \bigr\vert ^{2} + \Biggl\vert \sum_{l = 1}^{r} \bigl[{c^{l} - \varepsilon^{l}_{j} - E{S^{l}\bigl(x^{j},u^{j},t\bigr)}}\bigr] \Biggr\vert ^{2} \Biggr)^{1/2}. $$
On the one hand, due to (15), \(\forall \tilde{u}^{l} \in V^{l} \) the following necessary and sufficient condition of optimality for the unconstrained problem (14) holds:
$$ N^{l*}(t)u^{l,j}(t)=B^{l*}(t) \psi^{l,j}(t)+D^{l*}(t)\beta^{l,j}(t), \quad\textit{a.e. }t \in[t_{l-1}, t_{l} ], \textit{a.c.} $$
Besides, based on (15) and (17) we see that the optimal decision of the problem (14) satisfies the transversality condition:
$$ \psi^{l+1,j}(t_{l}) \bigl(\Phi_{t}^{l*}(t_{l}) x^{l,j}(t_{l})+K_{t}^{l*}(t_{l}) \bigr)=0, \quad \textit{a.c.}, l=1,\ldots,r-1. $$

Since \(\sum_{l = 1}^{r} |\lambda_{0}^{l,j}|^{2}+|\lambda_{1}^{l,j} |^{2} = 1 \) exists by (16) \((\lambda_{0}^{l,j} ,\lambda_{1}^{l,j} ) \to(\lambda_{0}^{l} ,\lambda _{1}^{l} )\) if \(j \to\infty\).

The truth of (6) is based upon the following lemma, which can be proved by the same method as the proof of Lemma 4 [33].

Lemma 2

Let \(\psi^{l}(t_{l} ) \) be a solution of system (6), \(\psi^{l,j}(t_{l} )\) be a solution of system (15). If the sequence of controls \((u^{1,j}(t) ,\ldots,u ^{r,j}(t) )\) satisfies the assumptions of the Lemma  1, then
$$E \int_{t_{l - 1} }^{t_{l} } {\bigl\vert \psi^{l,j}(t) - \psi^{l}(t) \bigr\vert ^{2}\,dt + } E \int_{t_{l - 1} }^{t_{l} } {\bigl\vert \beta^{l,j}(t) - \beta^{l}(t) \bigr\vert ^{2}\,dt \to0} ,\quad l = 1,\ldots,r. $$

Based on Lemma 2, we can pass to the weak limit in system (15) and obtain the fulfillment of (6). Following a similar scheme, we take the limits in (17) and (18), and justifications of (7), (8) are derived. Theorem 1 is proved.

4 Riccati equations for switching systems

In the theory of LQ problem, it is very natural to connect the LQ problem with the Riccati equation for the possible feedback design. In this section the optimal control is determined explicitly via a set of stochastic Riccati equations. First, we investigate the feedback design problem in the case \(g^{l}(t)\equiv0\); \(f^{l}(t)\equiv0\) and we search a relation in the form:
$$ \psi^{l}(t)=-p^{l}(t)x^{l}(t), \quad l = 1,\ldots,r, \mbox{a.c.} $$

To determine the stochastic processes \(p^{l}(t)\) we introduce the following theorem.

Theorem 2

Let \(\psi^{l}(t ) \) be a solution of system (6), \(p ^{l}(t) \) be a stochastic process that satisfies (19), defined as the solution of the following differential equation:
$$\begin{aligned} dp^{l}(t) ={}&{-} \bigl[p^{l}(t)A^{l}(t)+A^{l*}(t)p^{l}(t)+ \gamma ^{l}(t)C^{l}(t)+C^{l*}(t) \gamma^{l}(t)+C^{l*}p^{l}(t)C^{l} \\ &{}+M^{l} -\bigl(p^{l}(t)B^{l}(t)+\gamma ^{l}(t)D^{l}(t)+C^{l*}(t)p^{l}(t)D^{l}(t) \bigr) \bigl(N^{l}(t)+D^{l*}(t)p^{l}(t)D^{l}(t) \bigr) \\ &{}\times\bigl(B^{l*}(t)p^{l}(t)+D^{l*}(t) \gamma ^{l}(t)+D^{l*}(t)p^{l}(t)C^{l}(t) \bigr) \bigr] \\ &{}+\gamma^{l}(t)\,dw^{l}(t) ,\quad t_{l-1} \le t< t_{l} . \end{aligned}$$


Suppose that the differential of random processes \(p^{l}(t)\) is defined as
$$dp^{l}(t)=\alpha^{l}(t)\,dt+\gamma^{l}(t)\,dw^{l}(t), \quad l=1,\ldots,r. $$
According to the Ito formula:
$$d\psi^{l}(t)=-dp^{l}(t)x^{l}(t)-p^{l}(t)\,dx^{l}(t)- \gamma ^{l}(t)\bigl[C^{l}(t)x^{l}(t)+ D^{l}(t)u^{l}(t)\bigr]\,dt,\quad l = 1,\ldots,r,\mbox{a.c.} $$
Using (1) and (6) we have
$$\begin{aligned} &\bigl[-A^{l*}\psi^{l}(t)-C^{l*} \beta^{l}(t)+M^{l}(t)x^{l}(t)\bigr]+\beta ^{l}(t)\,dw^{l}(t) \\ &\quad=-\bigl[\alpha^{l}(t)x^{l}(t)\,dt +\gamma ^{l}(t)x^{l}(t)\,dw^{l}(t)+p^{l}(t)A^{l}(t)x^{l}(t)\,dt \\ &\qquad{}+p^{l}(t)B^{l}(t)u^{l}(t)\,dt+p^{l} \bigl(C^{l}(t)x^{l}(t) \\ &\qquad{}+D^{l}(t)u^{l}(t)\bigr)\,dw^{l}(t)+ \gamma^{l}(t) \bigl(C^{l}(t)x^{l}(t)+ D^{l}(t)u^{l}(t)\bigr)\,dt\bigr],\quad l = 1,\ldots,r. \end{aligned}$$
Taking expectation from both side we obtain the following expression for \(\beta^{l}(t)\), \(l = 1,\ldots,r\):
$$ \beta^{l}(t)=-\gamma ^{l}(t)x^{l}-p^{l}(t)C^{l}(t)x^{l}(t)-p^{l}(t)D^{l}(t)u^{l}(t), \quad t\in [t_{l-1},t_{l} ]. $$
By means of simple transformations taking into account (22) equation (21) can be rewritten as follows:
$$\begin{aligned} & \bigl[\alpha^{l}(t)+p^{l}(t)A^{l}(t)+A^{l*}(t)p^{l}(t)+ \gamma ^{l}(t)C^{l}(t) \\ &\quad{}+C^{l*}(t)\gamma^{l}(t)+C^{l*}p^{l}(t)C^{l}+ M^{l} \bigr]x^{l}(t) \\ &\quad{}+\bigl[p^{l}(t)B^{l}(t)+ \gamma ^{l}(t)D^{l}(t)+C^{l*}(t)p^{l}(t)D^{l}(t) \bigr]u^{l}(t)=0. \end{aligned}$$
Considering (19) in equation (6) the optimal control can be defined explicitly:
$$\begin{aligned} &\bigl[ N^{l}(t)+D^{l*}(t)p^{l}(t)D^{l}(t) \bigr]u^{l}(t) \\ &\quad{}+ \bigl[B^{l*}(t)p^{l}(t)+D^{l*}(t) \gamma ^{l}(t)+D^{l*}(t)p^{l}(t)C^{l}(t) \bigr]x^{l}(t)=0. \end{aligned}$$
Hence, Theorem 2 is proved. □

Finally, the feedback design for LQ problem (1)-(5) is obtained by means of the next theorem.

Theorem 3

Let \(\psi^{l}(t )\) be a solution of system (6), \(p ^{l}(t)\), \(\nu^{l}(t) \) be a stochastic processes satisfy to \(\psi^{l}(t)=-[p^{l}(t)x^{l}(t)+\nu^{l}(t)]\), a.c. Then for each l let the random process \(\nu^{l}(t) \) be a solution of the following differential equation:
$$\begin{aligned} & \bigl[\bigl(p^{l}(t)B^{l}(t)+\gamma ^{l}(t)D^{l}(t)+C^{l*}p^{l}(t)D^{l} \bigr) \bigl(N^{l}(t)+D^{l*}(t)p^{l}(t)D^{l}(t) \bigr)^{-1}B^{l*}(t)-A^{l*}(t) \bigr] \nu^{l}(t)\,dt \\ &\quad{} + \bigl[\bigl(p^{l}(t)B^{l}(t)+\gamma ^{l}(t)D^{l}(t)+C^{l*}p^{l}(t)D^{l} \bigr) \bigl(N^{l}(t)+D^{l*}(t)p^{l}(t)D^{l}(t) \bigr)^{-1}D^{l*}(t)-C^{l*}(t) \bigr] \\ &\qquad{}\times \bigl[p^{l*}(t)f^{l}(t)+ \varphi^{l}(t)\bigr]\,dt \\ &\quad= d\nu ^{l}(t) +\bigl[p^{l*}(t)g^{l}(t)- \gamma^{l*}(t)f^{l}(t)\bigr]\,dt+\varphi ^{l}(t)\,dw^{l}(t), \quad \nu^{l}(t_{l})=0 . \end{aligned}$$
Here for each l stochastic process \(p^{l}(t)\) is the solution of (20).


Suppose that random processes \(\nu^{l}(t)\) are defined in the following way:
$$\nu^{l}(t_{l})-\nu^{l}(t_{l-1})= \int_{t_{l-1}}^{t_{l}}\kappa ^{l}(t)\,dt+ \int_{t_{l-1}}^{t_{l}}\varphi^{l}(t)\,dw^{l}(t), \quad l=1,\ldots,r. $$
According to Ito’s formula for each l:
$$\begin{aligned} &d\psi^{l}(t)+\kappa^{l}(t)\,dt+\varphi ^{l}(t)\,dw^{l}(t) \\ &\quad=-dp^{l}(t)x^{l}(t)-p^{l}(t)\,dx^{l}(t) -\gamma^{l*}(t)\bigl[C^{l}(t)x^{l}(t)+ D^{l}(t)u^{l}(t)+f^{l}(t)\bigr]\,dt,\quad \mbox{a.c.} \end{aligned}$$
In view of (6) we obtain the following expression:
$$\beta^{l}(t)=-\gamma ^{l}(t)x^{l}(t)-p^{l}(t)C^{l}(t)x^{l}(t)+p^{l}(t)D^{l}(t)u^{l}(t) +p^{l*}(t)f^{l}(t)+\varphi ^{l}(t), \quad l = 1, \ldots,r,\mbox{a.c.} $$
Substituting this expression for \(\beta^{l}(t)\) into (10), in view of \([N^{l}(t)+D^{l*}(t)p^{l}(t)D^{l}(t)]\) being a positively defined matrix, we have
$$\begin{aligned} u^{l}(t)={}&{-}\bigl[N^{l}(t)+D^{l*}(t)p^{l}(t)D^{l}(t) \bigr]^{-1} \bigl\{ B^{l*}(t)\nu^{l}(t) +p^{l}(t)D^{l}(t)f^{l}(t) \\ &{} +D^{l*}(t)\varphi^{l}(t)+ \bigl[(p^{l}(t)B^{l}(t)+D^{l*}(t) \gamma ^{l}(t)+p^{l}(t)D^{l}(t)C^{l}(t) \bigr] x^{l}(t) \bigr\} . \end{aligned}$$
Integrating both sides of (26) and using equation (27) of \(u^{l}(t)\), bearing in mind that stochastic processes \(p^{l}(t)\), \(\gamma^{l}(t)\) are the solutions of differential equation (20):
$$\begin{aligned} \kappa^{l}(t)={}& \bigl[ \bigl(p^{l}(t)B^{l}(t)+ \gamma ^{l}(t)D^{l}(t)+C^{l*}(t)p^{l}(t)D^{l}(t) \bigr) \bigl(N^{l}(t)+D^{l*}(t)p^{l}(t)D^{l}(t) \bigr)^{-1} \\ &{}\times B^{l*}(t)-A^{l*}(t)\bigr] \nu^{l}(t)+ \bigl[\bigl(p^{l}(t)B^{l}(t)+\gamma ^{l}(t)D^{l}(t)+C^{l*}(t)p^{l}(t)D^{l}(t) \bigr) \\ &{}\times \bigl(N^{l}(t)+D^{l*}(t)p^{l}(t)D^{l}(t) \bigr)^{-1}D^{l*}(t)-C^{l*}(t)\bigr] \bigl[p^{l*}(t)f^{l}(t)+\varphi ^{l}(t) \bigr]-p^{l*}(t)g^{l}(t) \\ &{}-\gamma^{l*}(t)f^{l}(t). \end{aligned}$$

Therefore, the assertion of the theorem is true. □

5 Conclusion

There are a lot relevant applications of LQ problems in fields such as aerospace, biology, economics, management sciences, etc. [3438].

Switching systems provide a natural and convenient theoretical account for mathematical modeling of many complex real phenomena and practical applications. A broad spectrum of the latest research is concerned with optimal control problems of stochastic switching systems [3941].

The LQ problem of switching systems in which the endpoint restrictions are defined with the help of convex closed sets has been investigated. The objective of the present research is to give an explicit solution to the LQ problem of stochastic switching systems of which drift and diffusion coefficients comprise non-homogeneous terms. The results developed in this study can be viewed as an extension of the problems formulated in [42, 43] for stochastic switching systems.



The author thanks the two anonymous reviewers whose comments and suggestions helped improve this manuscript. The research underlying this paper is supported by the Scientific Research Project No. 1505F202 of Anadolu University, Turkey.

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors’ Affiliations

Department of Industrial Engineering, Anadolu University
Institute of Control Systems, ANAS


  1. Kalman, RE: Contributions to the theory of optimal control. Bol. Soc. Mat. Mexicana 5, 102-119 (1960) MathSciNetMATHGoogle Scholar
  2. Wonham, WM: On a matrix Riccati equation of stochastic control. SIAM J. Control Optim. 6, 312-326 (1968) MathSciNetView ArticleMATHGoogle Scholar
  3. Bellman, R: Functional equations in the theory of dynamic programming, positivity and quasilinearity. Proc. Natl. Acad. Sci. USA 41, 743-746 (1955) View ArticleMATHGoogle Scholar
  4. Bismut, JM: Linear quadratic optimal stochastic control with random coefficients. SIAM J. Control 14, 419-444 (1976) MathSciNetView ArticleMATHGoogle Scholar
  5. Bismut, JM: An introductory approach to duality in optimal stochastic control. SIAM Rev. 20, 62-78 (1978) MathSciNetView ArticleMATHGoogle Scholar
  6. Boukas, E: Stochastic Switching Systems: Analysis and Design. Birkhäuser, Basel (2006) MATHGoogle Scholar
  7. Liberzon, D: Switching in Systems and Control. Birkhäuser, Basel (2003) View ArticleMATHGoogle Scholar
  8. Bengea, SC, Raymond, AC: Optimal control of switching systems. Automatica 41, 11-27 (2005) MathSciNetMATHGoogle Scholar
  9. Capuzzo, DI, Evans, LC: Optimal switching for ordinary differential equations. SIAM J. Control Optim. 22, 143-161 (1984) MathSciNetView ArticleMATHGoogle Scholar
  10. Seidmann, TI: Optimal control for switching systems. In: Proceedings of the 21st Annual Conference on in Formations Science and Systems, pp. 485-489 (1987) Google Scholar
  11. Hoek, J, Elliott, RJ: American option prices in a Markov chain model. Appl. Stoch. Models Bus. Ind. 28, 35-39 (2012) MathSciNetView ArticleMATHGoogle Scholar
  12. Azevedo, N, Pinheiro, D, Weber, G-W: Dynamic programming for a Markov-switching jump-diffusion. J. Comput. Appl. Math. 267, 1-19 (2014) MathSciNetView ArticleMATHGoogle Scholar
  13. Temocin, B, Weber, G-W: Optimal control of stochastic hybrid system with jumps: a numerical approximation. J. Comput. Appl. Math. 259, 443-451 (2014) MathSciNetView ArticleMATHGoogle Scholar
  14. Xu, X, Antsaklis, PJ: Results and perspectives on computational methods for optimal control of switched systems. In: Maler, O, Pnueli, A (eds.) Hybrid Systems: Computation and Control. Lecture Notes in Computer Science, vol. 2623, pp. 540-556 (2003) View ArticleGoogle Scholar
  15. Haussman, UG: General necessary conditions for optimal control of stochastic systems. In: Stochastic Systems: Modeling, Identification and Optimization, II. Mathematical Programming Studies, vol. 6, pp. 30-48 (1976) View ArticleGoogle Scholar
  16. Kek, SL, Teo, LK, Ismail, AM: An integrated optimal control algorithm for discrete-time nonlinear stochastic system. Int. J. Control 83(12), 2536-2545 (2010) MathSciNetView ArticleMATHGoogle Scholar
  17. Kushner, HJ: Necessary conditions for continuous parameter stochastic optimization problems. SIAM J. Control 10, 550-565 (1976) MathSciNetView ArticleMATHGoogle Scholar
  18. Pham, H: On some recent aspects of stochastic control and their applications. Probab. Surv. 2, 506-549 (2005) MathSciNetView ArticleMATHGoogle Scholar
  19. Hafayed, M, Abbas, S, Abba, A: On mean-field partial information maximum principle of optimal control for stochastic systems with Lévy processes. J. Optim. Theory Appl. 167, 1051-1069 (2015) MathSciNetView ArticleMATHGoogle Scholar
  20. Shen, Y, Siu, TK: The maximum principle for a jump-diffusion mean-field model and its application to the mean-variance problem. Nonlinear Anal. 86, 58-73 (2013) MathSciNetView ArticleMATHGoogle Scholar
  21. Wang, G, Zhang, C, Zhang, W: Stochastic maximum principle for mean-field type optimal control under partial information. IEEE Trans. Autom. Control 59(2), 522-528 (2014) MathSciNetView ArticleGoogle Scholar
  22. Yong, J, Zhou, XY: Stochastic Controls: Hamiltonian Systems and HJB Equations. Springer, New York (1999) View ArticleMATHGoogle Scholar
  23. Borkar, V: Controlled diffusion processes. Probab. Surv. 2(4), 213-244 (2005) MathSciNetMATHGoogle Scholar
  24. Makhmudov, NI: General necessary optimality conditions for stochastic systems with controllable diffusion. In: Statistics and Control of Random Processes, pp. 135-138. Nauka, Moscow (1989) Google Scholar
  25. Peng, S: A general stochastic maximum principle for optimal control problem. SIAM J. Control Optim. 28, 966-979 (1990) MathSciNetView ArticleMATHGoogle Scholar
  26. Hafayed, M, Veverka, P, Abbas, S: On near-optimal necessary and sufficient conditions for forward-backward stochastic systems with jumps, with applications to finance. Appl. Math. 59(4), 407-440 (2014) MathSciNetView ArticleMATHGoogle Scholar
  27. Hafayed, M, Abba, A, Boukaf, S: On Zhou’s maximum principle for near-optimal control of mean-field forward-backward stochastic systems with jumps and its applications. Int. J. Model. Identif. Control 25(1), 1-16 (2016) View ArticleGoogle Scholar
  28. Aghayeva, C, Abushov, Q: The maximum principle for the nonlinear stochastic optimal control problem of switching systems. J. Glob. Optim. 56(2), 341-352 (2013) MathSciNetView ArticleMATHGoogle Scholar
  29. Abushov, Q, Aghayeva, C: Stochastic maximum principle for the nonlinear optimal control problem of switching systems. J. Comput. Appl. Math. 259, 371-376 (2014) MathSciNetView ArticleMATHGoogle Scholar
  30. Aghayeva, C: Necessary conditions of optimality for stochastic switching control systems. Dyn. Syst. Appl. 24(3), 243-258 (2015) Google Scholar
  31. Aghayeva, C, Abushov, Q: Linear-square stochastic optimal control problem with variable delay on control and state. Transactions ANAS, math.- ph. series, Informatics and Control Problems 25(3), 204-208 (2005) Google Scholar
  32. Ekeland, I: On the variational principle. J. Math. Anal. Appl. 47, 324-353 (1974) MathSciNetView ArticleMATHGoogle Scholar
  33. Aghayeva, C, Abushov, Q: The maximum principle for some nonlinear stochastic control system with variable structure. Theory Stoch. Process. 32(1), 1-11 (2010) MathSciNetMATHGoogle Scholar
  34. Bensoussan, A, Delfour, MC, Mitter, SK: The linear quadratic optimal control problem for infinite dimensional systems over an infinite horizon; survey and examples. In: Proceedings of the IEEE Conference on Decision and Control and the 15th Symposium on Adaptive Processes, pp. 746-751. Inst. Electr. Electron. Engrs., New York (1976) Google Scholar
  35. Kohlmann, M, Zhou, X: Relationship between backward stochastic differential equations and stochastic controls: a linear-quadratic approach. SIAM J. Control Optim. 38, 1392-1407 (2000) MathSciNetView ArticleMATHGoogle Scholar
  36. Turan, O: An exergy way to quantify sustainability metrics for a high bypass turbofan engine. Energy 86, 722-736 (2015) View ArticleGoogle Scholar
  37. Yuan, C, Wu, F: Hybrid control for switched linear systems with average dwell time. IEEE Trans. Autom. Control 60(1), 240-245 (2015) MathSciNetView ArticleGoogle Scholar
  38. Sahin, O: Optimum arrival routes for flight efficiency. J. Power Energy Eng. 3, 449-452 (2015) View ArticleGoogle Scholar
  39. Azhmyakov, V, Attia, SA, Raisch, J: On the maximum principle for impulsive hybrid system. Lect. Notes Comput. Sci. 4981, 30-42 (2008) MathSciNetView ArticleMATHGoogle Scholar
  40. Ghomanjani, F, HadiFarahi, M: Optimal control of switched systems based on Bezier control points. Int. J. Intell. Syst. Appl. 7, 16-22 (2012) Google Scholar
  41. Wang, S, Wu, Z: Maximum principle for optimal control problems of forward-backward regime-switching systems involving impulse controls. Math. Probl. Eng. 2015, 892304 (2015) MathSciNetGoogle Scholar
  42. Debrabant, K, Jakobsen, ER: Semi-Lagrangian schemes for linear and fully nonlinear diffusion equations. Math. Comput. 82, 1433-1462 (2013) MathSciNetView ArticleMATHGoogle Scholar
  43. Su, Y, Huang, J: Stability of a class of linear switching systems with applications to two consensus problems. IEEE Trans. Autom. Control 57(6), 1420-1430 (2012) MathSciNetView ArticleGoogle Scholar


© Aghayeva 2016