 Research
 Open Access
 Published:
Stochastic linear quadratic control problem of switching systems with constraints
Journal of Inequalities and Applications volume 2016, Article number: 100 (2016)
Abstract
This paper is devoted to the optimal control problem for stochastic linear switching systems with a quadratic cost functional. A necessary and sufficient condition of optimality for mentioned linear control systems under endpoint constraints is obtained. A linear quadratic controller is simply constructed via a set of stochastic backward Riccati equations.
Introduction
The Linear Quadratic (LQ) problem was mathematically formulated and solved, as well as the filtering one, in the 1960s by Kalman [1]. An important advantage of the LQ theory is the existence of explicit feedback forms for optimal state control and the optimal cost value through the Riccati equations. The deterministic Riccati equation was essentially solved by Wonham [2] by applying Bellman’s principle of quasilinearization [3]. A detailed research of stochastic LQ control problems has been performed by Bismut [4]. The existence of a unique solution for the associated Riccati equations was studied in [5].
Switching systems are more advantageous models to describe the noninvariant phenomena with the continuous law of movement and they have gained considerable attention in science and engineering. Examples of these systems include many evolutionary processes, robotics, integrated circuit design, multimedia, manufacturing, power electronics, chaos generators, and air traffic management systems [6, 7]. Optimization problems have also received growing interest among the researchers of deterministic and for stochastic switching control systems [8–14].
Manifold problems of stochastic optimal control theory have been considered in [15–22]. Optimal control problems of switching systems have attracted considerable attention, due to the advantages, for instance, in modeling and improving the transient response on highly complex systems and systems with large uncertainties. The stochastic maximum principle via backward stochastic differential equations is derived in [23–27]. The necessary conditions of optimality for stochastic switching systems earlier have been obtained in [28–30]. In [31] the linear quadratic control problem has been investigated for a special type of stochastic systems.
In this paper, the LQ problem of stochastic switching systems with restrictions is considered. Ekeland’s variational principle [32] has been used to establish the necessary and sufficient conditions of optimality for a given problem.
Statement of main problem
Unless specified otherwise, throughout the paper we use the same notations as in [30].
Consider the following stochastic linear control system:
The elements of \(U_{\partial}^{l} \), are called admissible controls.
Our goal is to find an optimal solution \((x,u)= ( {x^{1},x^{2},\ldots,x^{r},u^{1},u^{2},\ldots,u^{r}} )\) and a switching sequence \(\mathbf{t}=(t_{1} ,t_{2} ,\ldots,t_{r}) \), that minimize the cost functional:
on the decisions of the system (1)(3) under the conditions:
where \(Q^{1},\ldots,Q^{r} \) are a closed convex sets in \(R^{1}\). The elements of matrices \(A^{l}\), \(B^{l}\), \(C^{l}\), \(D^{l}\), \(\Phi^{l}\), \(M^{l}\), \(N^{l}\) and vectors \(G^{l}\), \(K^{l}\), \(g^{l}\), \(f^{l} \) are continuous, bounded functions. \(G^{l}\), \(M^{l}\) are a positively semidefined matrices, and \(N^{l}\) are positively defined matrices.
\(A_{i}\) represents the set of elements \(\pi^{i} = (t_{0} ,t_{1} ,t_{i} ,x^{1}(t) ,x^{2}(t) ,\ldots,x^{i}(t),u^{1},u^{2},\ldots,u^{i})\) for each \(i=1,\ldots,r \). To describe the main result we need to introduce some concepts, such as a solution of linear switching systems, admissible element of control problem and optimal solution for LQ problem of stochastic switching systems. For a detailed account we refer the reader to [29, 30].
Stochastic LQ problem of switching systems
This section is devoted to the investigation of optimal control problems for linear stochastic switching systems with constraints. The LQ problem belongs to a special class of convex control problems for which the maximum principle is a necessary as well as sufficient condition of optimality. The next theorem provides necessary and sufficient conditions of the optimality of stochastic linear switching systems.
Theorem 1
Let \(A_{r}^{0} \) be a set of admissible elements. The elements
are an optimal solution of problem (1)(5) if and only if:

(a)
there exist random processes \((\psi^{l}(t) ,\beta^{l}(t))\in{L}_{F}^{2} (t_{l1} ,t_{l} ;{R}^{n_{l} })\times{L}_{F}^{2} (t_{l1} ,t_{l} ;{R}^{n_{l}\times n_{l}})\) which are the solutions of the following stochastic backward equations:
$$ \left \{ \textstyle\begin{array}{@{}l} d\psi^{l}(t) = [A^{l*}(t)\psi^{l}(t)+C^{l*}(t)\beta ^{l}(t)M^{l}(t)x(t) ] \\ \hphantom{d\psi^{l}(t) =}{}+\beta^{l}(t)\,dw^{l}(t) ,\quad t_{l1} \le t< t_{l} , \\ \psi^{l}(t_{l}) =\lambda^{l}_{0}G^{l} x^{l}(t_{l})\lambda^{l}_{1}q^{l} +\psi ^{l+1}(t_{l}) \Phi^{l} (t_{l} ),\quad l=1,\ldots,r1 , \\ \psi^{r}(t_{r}) =\lambda^{r}_{0}G^{r} x^{r}(t_{r})\lambda^{r}_{1}q^{r} ; \end{array}\displaystyle \right . $$(6) 
(b)
the candidate optimal controls \({u}^{l}\in U^{l}\), \(l=\overline{1,r}\), are defined by
$$ N^{l*}(t)u^{l}(t)=B^{l*}(t) \psi^{l}(t)+D^{l*}(t)\beta^{l}(t), \quad\textit{a.e. }\theta\in[t_{l1} , t_{l} ] ; $$(7) 
(c)
the following transversality conditions hold:
$$ \psi^{l+1}(t_{l}) \bigl(\Phi_{t}^{l*} \bigl(t^{l}\bigr) x^{l}(t_{l})+K_{t}^{l*}(t_{l}) \bigr)=0, \quad \textit{a.c.}, l=1,\ldots,r1 . $$(8)
Proof
First we investigate given optimal control problem without endpoint constraints (5). Let \(u^{l}(t)\) and \(\bar{u}^{l}(t)\), \(l = 1,\ldots,r \) be some admissible controls and \(x^{l}(t)\), \(\bar{x}^{l}(t) \) be corresponding trajectories. \(\Delta\bar{u}^{l}(t)\) represents the admissible increment of the control \(u^{l}(t)\). \(\mathbf{t}=(t_{0}, t_{1}, \ldots t_{r} )\) and \(\bar{\mathbf{t}}=(\bar{ t_{0}}, \bar{t_{1}}, \ldots, \bar{t_{r}})\) denote different switching laws. The increment of the cost functional (4) along the admissible control \(\bar{ \mathbf {u}}=(\bar{u}^{1}(t), \bar{u}^{2}(t), \ldots, \bar{u}^{r}(t)) \) looks like
By (1)(2) the increments of the trajectories are defined as
Let us introduce the stochastic processes \(\psi^{l}(t) \), \(l=1,\ldots,r \), as the solution of the following stochastic backward differential equations:
According to the Ito formula for each \(l=1,\ldots,r \) the following identity is satisfied:
Integrating the aforementioned equality and taking the expectation of both sides into account in (10) it follows
Due to this equality equation (9) can be rewritten as
Further, using equation (11) we get a more succinct expression:
It is well known that a necessary and sufficient condition of optimality for the convex functional is given by \(J^{\prime}(\mathbf{u})=0\). The validity of (7) and (8), hence the necessary conditions of optimality for the considered unrestricted problem (1)(4) follows from equations (11) and (13). At last, according to the independence of the increments \(\Delta\bar{x}^{l}(t)\), \(\Delta\bar {u}^{l}(t)\), \(\Delta\bar{t_{l}}\), sufficiency follows from equation (12).
To construct the optimality condition of LQ problem (1)(4) with the right endpoint constraints (5), the above mentioned problem by using Ekeland’s variational principle [32] is converted into a sequence of unconstrained problems. Based on the results already obtained for problem (1)(4), necessary and sufficient conditions for the sequence of switching systems are established.
To apply Ekeland’s variational principle we introduce the following approximating functional:
Here \(S^{l} ( x ,u ,t )= \langle G^{l}x^{l}(t_{l}),x^{l}(t_{l}) \rangle+\int_{t_{l1} }^{t_{l} } ( \langle M^{l}(t)x^{l}(t),x^{l}(t) \rangle+ \langle N^{l}(t)u^{l}(t),u^{l}(t) \rangle )\,dt\); \(\lim_{j \to\infty}\varepsilon^{l}_{j}=0 \); \(c=c^{1}+\cdots+c^{r}\); \(\varepsilon= \{ {c:c \le J^{0},y^{l}\in Q^{l}} \}\); let \(J^{0}\) be a minimal value of the functional in the problem (1)(5).
Let \(V^{l} \equiv(U_{\partial}^{l} ,d)\) be the space of controls obtained by means of the following metric:
For each \(l = 1,\ldots,r \), let \(V^{l}\) be a complete metric space [32]. □
For the following fact it is significant that we can provide a relation between the sequence of controls from the metric space \(V^{l}\) and the sequence of corresponding trajectories of system (1)(2).
Lemma 1
([30], Lemma 4.3)
Let \((v^{1,n},\ldots,v^{r,n}) \) be the sequence of admissible controls from \((V^{1},\ldots,V^{r})\), and \((x^{1,n},\ldots,x^{r,n})\) be the sequence of corresponding trajectories of the system (1)(2). Let for each \(l=1,\ldots,r\) the condition \(d(v^{l,n} ,u ^{l}) \to0\) when \(n\to \infty\) be met. Then
where \(x ^{l}(t) \) is a trajectory corresponding to admissible controls \(u ^{l}(t)\), \(l = 1,\ldots,r\).
Due to Ekeland’s variational principle, we see that \((x^{1,j}(t) ,\ldots,x^{r,j}(t), u^{1,j}(t) ,\ldots,u^{r,j}(t) ) \) is a solution of the following problem:
\(\delta(u,v)\) is the characteristic function of the set \(\{ u,v \in V^{l}:u \ne v \}\).
Based on (11), it is found that, if \((x^{1,j}(t) ,\ldots,x^{r,j}(t),u^{1,j}(t) ,\ldots,u^{r,j}(t))\) is an optimal solution of problem (14), there exist the random processes \((\psi^{l,j}(t), \beta^{l,j}(t)) \in L_{F^{l}}^{2} (t_{l  1} ,t_{l} ;R^{n_{l} })\times L_{F^{l}}^{2} (t_{l  1} ,t_{l} ;R^{n_{l} \times n_{l} })\), which are solutions of the following system:
where the nonzero \((\lambda_{0}^{l,j} ,\lambda_{1}^{l,j})\), \(l = 1,\ldots,r\), are defined as
here
On the one hand, due to (15), \(\forall \tilde{u}^{l} \in V^{l} \) the following necessary and sufficient condition of optimality for the unconstrained problem (14) holds:
Besides, based on (15) and (17) we see that the optimal decision of the problem (14) satisfies the transversality condition:
Since \(\sum_{l = 1}^{r} \lambda_{0}^{l,j}^{2}+\lambda_{1}^{l,j} ^{2} = 1 \) exists by (16) \((\lambda_{0}^{l,j} ,\lambda_{1}^{l,j} ) \to(\lambda_{0}^{l} ,\lambda _{1}^{l} )\) if \(j \to\infty\).
The truth of (6) is based upon the following lemma, which can be proved by the same method as the proof of Lemma 4 [33].
Lemma 2
Let \(\psi^{l}(t_{l} ) \) be a solution of system (6), \(\psi^{l,j}(t_{l} )\) be a solution of system (15). If the sequence of controls \((u^{1,j}(t) ,\ldots,u ^{r,j}(t) )\) satisfies the assumptions of the Lemma 1, then
Based on Lemma 2, we can pass to the weak limit in system (15) and obtain the fulfillment of (6). Following a similar scheme, we take the limits in (17) and (18), and justifications of (7), (8) are derived. Theorem 1 is proved.
Riccati equations for switching systems
In the theory of LQ problem, it is very natural to connect the LQ problem with the Riccati equation for the possible feedback design. In this section the optimal control is determined explicitly via a set of stochastic Riccati equations. First, we investigate the feedback design problem in the case \(g^{l}(t)\equiv0\); \(f^{l}(t)\equiv0\) and we search a relation in the form:
To determine the stochastic processes \(p^{l}(t)\) we introduce the following theorem.
Theorem 2
Let \(\psi^{l}(t ) \) be a solution of system (6), \(p ^{l}(t) \) be a stochastic process that satisfies (19), defined as the solution of the following differential equation:
Proof
Suppose that the differential of random processes \(p^{l}(t)\) is defined as
According to the Ito formula:
Taking expectation from both side we obtain the following expression for \(\beta^{l}(t)\), \(l = 1,\ldots,r\):
By means of simple transformations taking into account (22) equation (21) can be rewritten as follows:
Considering (19) in equation (6) the optimal control can be defined explicitly:
Hence, Theorem 2 is proved. □
Finally, the feedback design for LQ problem (1)(5) is obtained by means of the next theorem.
Theorem 3
Let \(\psi^{l}(t )\) be a solution of system (6), \(p ^{l}(t)\), \(\nu^{l}(t) \) be a stochastic processes satisfy to \(\psi^{l}(t)=[p^{l}(t)x^{l}(t)+\nu^{l}(t)]\), a.c. Then for each l let the random process \(\nu^{l}(t) \) be a solution of the following differential equation:
Here for each l stochastic process \(p^{l}(t)\) is the solution of (20).
Proof
Suppose that random processes \(\nu^{l}(t)\) are defined in the following way:
According to Ito’s formula for each l:
In view of (6) we obtain the following expression:
Substituting this expression for \(\beta^{l}(t)\) into (10), in view of \([N^{l}(t)+D^{l*}(t)p^{l}(t)D^{l}(t)]\) being a positively defined matrix, we have
Integrating both sides of (26) and using equation (27) of \(u^{l}(t)\), bearing in mind that stochastic processes \(p^{l}(t)\), \(\gamma^{l}(t)\) are the solutions of differential equation (20):
Therefore, the assertion of the theorem is true. □
Conclusion
There are a lot relevant applications of LQ problems in fields such as aerospace, biology, economics, management sciences, etc. [34–38].
Switching systems provide a natural and convenient theoretical account for mathematical modeling of many complex real phenomena and practical applications. A broad spectrum of the latest research is concerned with optimal control problems of stochastic switching systems [39–41].
The LQ problem of switching systems in which the endpoint restrictions are defined with the help of convex closed sets has been investigated. The objective of the present research is to give an explicit solution to the LQ problem of stochastic switching systems of which drift and diffusion coefficients comprise nonhomogeneous terms. The results developed in this study can be viewed as an extension of the problems formulated in [42, 43] for stochastic switching systems.
References
 1.
Kalman, RE: Contributions to the theory of optimal control. Bol. Soc. Mat. Mexicana 5, 102119 (1960)
 2.
Wonham, WM: On a matrix Riccati equation of stochastic control. SIAM J. Control Optim. 6, 312326 (1968)
 3.
Bellman, R: Functional equations in the theory of dynamic programming, positivity and quasilinearity. Proc. Natl. Acad. Sci. USA 41, 743746 (1955)
 4.
Bismut, JM: Linear quadratic optimal stochastic control with random coefficients. SIAM J. Control 14, 419444 (1976)
 5.
Bismut, JM: An introductory approach to duality in optimal stochastic control. SIAM Rev. 20, 6278 (1978)
 6.
Boukas, E: Stochastic Switching Systems: Analysis and Design. Birkhäuser, Basel (2006)
 7.
Liberzon, D: Switching in Systems and Control. Birkhäuser, Basel (2003)
 8.
Bengea, SC, Raymond, AC: Optimal control of switching systems. Automatica 41, 1127 (2005)
 9.
Capuzzo, DI, Evans, LC: Optimal switching for ordinary differential equations. SIAM J. Control Optim. 22, 143161 (1984)
 10.
Seidmann, TI: Optimal control for switching systems. In: Proceedings of the 21st Annual Conference on in Formations Science and Systems, pp. 485489 (1987)
 11.
Hoek, J, Elliott, RJ: American option prices in a Markov chain model. Appl. Stoch. Models Bus. Ind. 28, 3539 (2012)
 12.
Azevedo, N, Pinheiro, D, Weber, GW: Dynamic programming for a Markovswitching jumpdiffusion. J. Comput. Appl. Math. 267, 119 (2014)
 13.
Temocin, B, Weber, GW: Optimal control of stochastic hybrid system with jumps: a numerical approximation. J. Comput. Appl. Math. 259, 443451 (2014)
 14.
Xu, X, Antsaklis, PJ: Results and perspectives on computational methods for optimal control of switched systems. In: Maler, O, Pnueli, A (eds.) Hybrid Systems: Computation and Control. Lecture Notes in Computer Science, vol. 2623, pp. 540556 (2003)
 15.
Haussman, UG: General necessary conditions for optimal control of stochastic systems. In: Stochastic Systems: Modeling, Identification and Optimization, II. Mathematical Programming Studies, vol. 6, pp. 3048 (1976)
 16.
Kek, SL, Teo, LK, Ismail, AM: An integrated optimal control algorithm for discretetime nonlinear stochastic system. Int. J. Control 83(12), 25362545 (2010)
 17.
Kushner, HJ: Necessary conditions for continuous parameter stochastic optimization problems. SIAM J. Control 10, 550565 (1976)
 18.
Pham, H: On some recent aspects of stochastic control and their applications. Probab. Surv. 2, 506549 (2005)
 19.
Hafayed, M, Abbas, S, Abba, A: On meanfield partial information maximum principle of optimal control for stochastic systems with Lévy processes. J. Optim. Theory Appl. 167, 10511069 (2015)
 20.
Shen, Y, Siu, TK: The maximum principle for a jumpdiffusion meanfield model and its application to the meanvariance problem. Nonlinear Anal. 86, 5873 (2013)
 21.
Wang, G, Zhang, C, Zhang, W: Stochastic maximum principle for meanfield type optimal control under partial information. IEEE Trans. Autom. Control 59(2), 522528 (2014)
 22.
Yong, J, Zhou, XY: Stochastic Controls: Hamiltonian Systems and HJB Equations. Springer, New York (1999)
 23.
Borkar, V: Controlled diffusion processes. Probab. Surv. 2(4), 213244 (2005)
 24.
Makhmudov, NI: General necessary optimality conditions for stochastic systems with controllable diffusion. In: Statistics and Control of Random Processes, pp. 135138. Nauka, Moscow (1989)
 25.
Peng, S: A general stochastic maximum principle for optimal control problem. SIAM J. Control Optim. 28, 966979 (1990)
 26.
Hafayed, M, Veverka, P, Abbas, S: On nearoptimal necessary and sufficient conditions for forwardbackward stochastic systems with jumps, with applications to finance. Appl. Math. 59(4), 407440 (2014)
 27.
Hafayed, M, Abba, A, Boukaf, S: On Zhou’s maximum principle for nearoptimal control of meanfield forwardbackward stochastic systems with jumps and its applications. Int. J. Model. Identif. Control 25(1), 116 (2016)
 28.
Aghayeva, C, Abushov, Q: The maximum principle for the nonlinear stochastic optimal control problem of switching systems. J. Glob. Optim. 56(2), 341352 (2013)
 29.
Abushov, Q, Aghayeva, C: Stochastic maximum principle for the nonlinear optimal control problem of switching systems. J. Comput. Appl. Math. 259, 371376 (2014)
 30.
Aghayeva, C: Necessary conditions of optimality for stochastic switching control systems. Dyn. Syst. Appl. 24(3), 243258 (2015)
 31.
Aghayeva, C, Abushov, Q: Linearsquare stochastic optimal control problem with variable delay on control and state. Transactions ANAS, math. ph. series, Informatics and Control Problems 25(3), 204208 (2005)
 32.
Ekeland, I: On the variational principle. J. Math. Anal. Appl. 47, 324353 (1974)
 33.
Aghayeva, C, Abushov, Q: The maximum principle for some nonlinear stochastic control system with variable structure. Theory Stoch. Process. 32(1), 111 (2010)
 34.
Bensoussan, A, Delfour, MC, Mitter, SK: The linear quadratic optimal control problem for infinite dimensional systems over an infinite horizon; survey and examples. In: Proceedings of the IEEE Conference on Decision and Control and the 15th Symposium on Adaptive Processes, pp. 746751. Inst. Electr. Electron. Engrs., New York (1976)
 35.
Kohlmann, M, Zhou, X: Relationship between backward stochastic differential equations and stochastic controls: a linearquadratic approach. SIAM J. Control Optim. 38, 13921407 (2000)
 36.
Turan, O: An exergy way to quantify sustainability metrics for a high bypass turbofan engine. Energy 86, 722736 (2015)
 37.
Yuan, C, Wu, F: Hybrid control for switched linear systems with average dwell time. IEEE Trans. Autom. Control 60(1), 240245 (2015)
 38.
Sahin, O: Optimum arrival routes for flight efficiency. J. Power Energy Eng. 3, 449452 (2015)
 39.
Azhmyakov, V, Attia, SA, Raisch, J: On the maximum principle for impulsive hybrid system. Lect. Notes Comput. Sci. 4981, 3042 (2008)
 40.
Ghomanjani, F, HadiFarahi, M: Optimal control of switched systems based on Bezier control points. Int. J. Intell. Syst. Appl. 7, 1622 (2012)
 41.
Wang, S, Wu, Z: Maximum principle for optimal control problems of forwardbackward regimeswitching systems involving impulse controls. Math. Probl. Eng. 2015, 892304 (2015)
 42.
Debrabant, K, Jakobsen, ER: SemiLagrangian schemes for linear and fully nonlinear diffusion equations. Math. Comput. 82, 14331462 (2013)
 43.
Su, Y, Huang, J: Stability of a class of linear switching systems with applications to two consensus problems. IEEE Trans. Autom. Control 57(6), 14201430 (2012)
Acknowledgements
The author thanks the two anonymous reviewers whose comments and suggestions helped improve this manuscript. The research underlying this paper is supported by the Scientific Research Project No. 1505F202 of Anadolu University, Turkey.
Author information
Affiliations
Corresponding author
Additional information
Competing interests
The author declares to have no competing interests.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Aghayeva, C. Stochastic linear quadratic control problem of switching systems with constraints. J Inequal Appl 2016, 100 (2016). https://doi.org/10.1186/s1366001610468
Received:
Accepted:
Published:
Keywords
 stochastic linear system
 conditions of optimality
 switching systems
 transversality conditions