Optimal control of stochastic singular affine systems with Markovian jumps

*Correspondence: wangxinhuster@163.com 1School of Science, Qingdao University of Technology, Qingdao 266000, Shandong, P.R. China Abstract We consider an optimal control problem for a class of stochastic singular affine systems with Markovian jumps. We establish the existence and uniqueness of the solution to stochastic singular affine systems with Markovian jumps for the first time. Via square completion technique and the generalized Itô’s formula, we derive new kinds of generalized differential Riccati equations (GDREs) and generalized backward differential equations (GBDEs), which give sufficient conditions for the well-posedness of the optimal control problem, and present an explicit representation of optimal control. Also, we discuss the solvability of the GDREs in two cases. As an application, we present a leader-follower differential game to demonstrate the practicability of our results.


Introduction
Optimal control theory is the science of finding an optimal solution from all possible control schemes. It is the core content of modern control theory. Its essence is developing optimal control law or control strategy under given performance objectives and constraints. It is worth mentioning that the ideological basis of control theory can be traced back to the book [19] by Wiener, which laid to the foundation of control theory.
Stochastic optimal control problems include various physical, biological, and electronic systems, just to mention a few. In view of stochastic differential equations, even in the nonlinear case, the theory is relatively mature [3,9,10,16,20]. In [15] the authors researched linear quadratic (LQ) control problems for stochastic affine systems, which give the open-loop and closed-loop solvabilities; for additional details, we refer the readers to the book [21] and references therein. Recently, due to a better description of physical systems than regular ones, a lot of researchers have focused on singular systems. However, the research related to stochastic singular control systems is still in its fancy; only few papers can be obtained until now. Zhang and Xing [23] studied the optimal control and stability of stochastic singular systems with state-and control-dependent multiplica-tive noise. They established a kind of GDREs, which are harder for solvability due to their symmetry. In [24] the LQ problem for stochastic singular systems with state-dependent multiplicative noise is discussed. The authors presented a new kind of Lyapunov functional, which made the new GDREs easy to solve. In [17], we have discussed the stochastic singular optimal control systems with state-and control-dependent multiplicative noise.
Moreover, for our new kinds of GDREs, we established the solvability in definite, singular, and indefinite cases.
On the other side, the parameter system model with Markovian jump provides an expedient mathematical model to depict systematic dynamics in situation where the system undergoes frequent unpredictable parameter changes. Researches on the stochastic linear jump systems can be at least dated back to the work of Krasosvkii and Lidskii [6]. During the last decades, the jump parameter LQ control systems have been extensively investigated (see, for example, [4,5,11,12,22] and references therein). For the case of stochastic Markov jump differential equation with state-and control-dependent multiplicative noise, Li et al. [8] discussed the infinite time domain control problem with indefinite state and control cost weighting matrices, whereas Li and Zhou [7] investigated the same control problem with finite time domain. However, it is worth mentioning that the papers mentioned are all concerned with stochastic Markov jump differential equations, whereas for the optimal control problem of stochastic singular Markov jump systems, to the best of our knowledge, there is no existing literature. Meanwhile, it is necessary to investigate the affine systems due to the development of the leader-follower differential games. Therefore it is a natural question of the optimal control problem of stochastic singular affine systems with Markovian jumps, which is of particular mathematical interest.
Directly inspired by the works mentioned, the purpose of this paper is investigating the optimal control of stochastic singular affine systems with Markovian jumps. The main content is as follows. We study the stochastic singular affine LQ control systems with Markovian jumps for the first time, which generalizes the result in [17]. Then to get its well-posedness, we introduce new kinds of GDREs and GBDEs. This is quite different from [17] because of the affine character of our new system. Moreover, the solvability of the GDREs is established. As a direct application, results of this paper also enrich the contents related to the theory of leader-follower game, which is one of the most important differential games. This paper comprises several sections. Preliminaries are provided in Sect. 2. In Sect. 3, we derive sufficient conditions for the well-posedness of the LQ control problem in finiteand infinite-time horizons. Section 4 focuses on the solvability of GDREs via applying matrix decomposition. In Sect. 5, as an application, we consider a leader-follower differential game. Finally, the conclusions are given in Sect. 6.

Preliminaries
Let ( , F, {F t } t≥0 , P) be a filtered probability space, where there is a one-dimensional standard Wiener process {w(t)} t≥0 and a right-continuous homogeneous Markov chain {r t } t≥0 with state space ψ = {1, 2, . . . , l}, and F t is the smallest σ -algebra generated by the processes w(s) and r s , 0 ≤ s ≤ t, i.e., F t = σ {w(s), r s |0 ≤ s ≤ t}. We assume that {r t } is indepen-dent of {w(t)} and has the transition probabilities where π ij ≥ 0 for i = j and π ii = -l j=1,j =i π ij . Consider the stochastic singular system with Markovian jump where x(t) ∈ R n is the state variable, x 0 ∈ R n is a given initial function, f (t) and g(t) are both inhomogeneous terms, E ∈ R n×n is a known singular matrix with rank(E) = r ≤ n, A(t, r t ) = A i (t) and C(t, r t ) = C i (t) when r t = i, and A i (t), C i (t), i ∈ ψ, are specified matrices of suitable sizes. For each i ∈ ψ, A i , C i ∈ L ∞ (0, T; R n×n ) and f , g ∈ L 2 (0, T; R n ). Here the Lebesgue space L p (0, T; X) consists measurable functions φ : and X is the real Banach space R n or R n×n . Here we only consider the one-dimensional Wiener process, since the multiplicative noise case can be easily generalized.
For the existence and uniqueness of a solution without impulse to system (2.1), we carry out the following assumptions for every i ∈ ψ: where C i,1 (t) ∈ R r×r , C i,2 (t) ∈ R r×(n-r) , and g i,1 (t) ∈ R r . Accordingly, define , 22 (t), f i,1 (t), and f i,2 (t) are of appropriate dimensions. Let where ξ 1 (t) ∈ R r , ξ 2 (t) ∈ R n-r . Via the above transformations, system (2.1) can be rewritten as On the other side, by assumption (H.2.1) we have the rank relation that is, the matrix A i, 22 (t) has full row rank. Then there is nonsingular matrix F i (t) such that for every i ∈ ψ, A i, 22 Then we can transform system (2.2) as where F i (t)ξ 2 (t) = ξ 2 (t).
For every i ∈ ψ, the first equation in system (2.3) is a stochastic ordinary differential equation. According to Theorem 6.14 in [21], it has a unique solution ξ 1 (t) on [0, T], and so does (2.3). This completes the proof.

Optimal control problem
In this section, we consider the LQ optimal control problem for stochastic singular systems with Markovian jumps in finite-and infinite-time horizons.

Finite-time horizon case
Taking into account the stochastic singular equations with Markovian jumps, we have where x(t) ∈ R n and u(t) ∈ R m represent the state and control vectors, respectively. An admissible control u is an F t -adapted R m -valued measurable process on [0, T]. The set of all admissible controls is denoted by U ad ≡ L 2 F (0, T; R m ), and the others are defined similarly as before.
As in Sect. 2, we impose the following basic assumptions for each i ∈ ψ, which just correspond to the existence and uniqueness of the solution to system (3.1): For all (0, x 0 , i) and u ∈ U ad , the related cost functional is defined as (3.2) and the objective of this control problem is minimizing the cost functional J(0, x 0 , i; u(·)) for a given (0, The following assumption will go into effect throughout this section: where S m denotes the set of all m × m symmetric matrices. A well-posed problem is called attainable with respect to (0, x 0 , i) if there exists a control u * ∈ U ad that achievesV (0, x 0 , i). In such a case, the control u * is called optimal with respect to (0, x 0 , i).

Lemma 3.1 ([14]) For a symmetric matrix S, we can get
where " †" denotes the Moore-Penrose pseudoinverse of a matrix.

has a solution X if and only if
In addition, any solution to (3.3) can be written as where S is a matrix of appropriate size.
Moreover, the set of all optimal controls with respect to the initial (0, Furthermore, the value function is presented bȳ Proof We prove this result by using the completion of squares. First, define a Lyapunov- Then applying generalized Itô's formula [1] to V 1 (t, x, i) and V 2 (t, x, i), after some manip- and Hence by equations (3.8)-(3.9) and the cost function (3.2) we get Now let Y i ∈ L 2 F (0, T; R m×n ) and z i ∈ L 2 F (0, T; R m ) be given for every i ∈ ψ. Set Applying Lemma 2.1 and the above discussion, for k = 1, 2, we have Thus we can represent the cost functional as follows: Hence J(0, x 0 , i; u(·)) is minimized by the control (3.6) with value function (3.7).
On the other hand, we show that any optimal control can be expressed by (3.6) for some Y i and z i . Let u * be an optimal control; thus we know that the integrand on the right-hand side of (3.11) must be zero almost everywhere in t. This gives Applying Lemma 3.2 to solve the above equation in u * , we get (3.6). This completes the proof.
whose number is determined by the solutions of (3.4)-(3.5).

In particular, if the coefficient matrices in (3.1)-(3.2) are the form A(t, r t ) = A(t), B(t, r t ) = B(t), etc., then we can get a sufficient condition for the corresponding optimal control problem and the display expression for optimal control as well.
Corollary 3.2 Consider the following optimal control problem:

E T P(T) = H, E T P(t) = P T (t)E;
(3.14) Then the optimal control problem (3.12)-(3.13) is well-posed and can be represented as and the optimal value of the cost functional can be represented by

Definition 3.2 System (3.16) is called mean-square stabilizable if there is a feedback control
where K 1 , . . . , K l are given matrices, that is stabilizing with respect to any initial state (x 0 , i).
For a given (x 0 , i) ∈ R n × ψ, we define the set of admissible controls The control problem in this subsection is to find a control that minimizes the following quadratic cost associated with (3.16): The value function is similarly defined as In such a case the optimal control can be expressed as and the value function is presented bȳ Proof Suppose that there exists a solution P satisfying equation (3.18). Set the Lyapunov-Krasovskii functional Applying generalized Itô's formula to system (3.16), we derive where From assumption (H.3.5) we have E[V (∞)] = 0. Then extending the integral interval to [0, ∞) for equation (3.19) and combining with (3.17), we eventually obtain From (3.20) we can obviously obtain the optimal control and the value function. This completes the proof.

The solvability of GDREs
We give some conditions under which the GDREs are solvable in this section. Due to limited capacity, we only deal with the following case: (4.1) First, we declare some transformations that will be used later: where P i,11 (t) ∈ R r×r , P i,12 (t) ∈ R r×(n-r) , P i,21 (t) ∈ R (n-r)×r , P i, 22 (t) ∈ R (n-r)×(n-r) . For discussing the solvability conditions for GDREs (4.1), we make a necessary assumption.
Under this assumption,Q i ≥ 0 and H i,11 ≥ 0 for each i ∈ ψ, and thus there exist two matrices F i,1 ∈ R n×r and F i,2 ∈ R n×(n-r) such that Next, we can directly use the above transformations to GDREs (4.1), which can be divided into - with the boundary condition P i,11 (T) = H i, 11 . Next, from the equality E T P i (t) = P T i (t)E we have P i,11 (t) = P T i,11 (t) and P i,12 (t) = 0. Substituting them into (4.2), after some manipulations, we obtain the following equations: Moreover, substituting P i,21 (t) into equation (4.4a), after some calculations, we havė where Similarly to the proof of Theorem 1 in [18], we can prove that there exist C i,0 and D i,0 such that Under these assumptions, by Theorem 5.6.15 in [2] there exists a unique positive semidefinite and bounded solution P i,11 (t), which is also stabilizing.  11 (t)C i,1 -l j=1 π ij P j,11 (t) Under assumption (H.4.2 ), equation (4.5c) has a unique solution P i,2 (t) for any given i ∈ ψ [24]. Then similarly to case 1, we get a unique positive semidefinite and bounded solution P i,11 (t) for equation (4.5a). Substituting P i,11 (t) into (4.5b), we get that Then the solution P i (t) can also be obtained.
Remark 4.1 For GBDEs (3.15), using the results for linear BDEs, we can deal with it by a similar method, but we omit it here for simplicity.

Application
In this section, we study a stochastic LQ leader-follower differential game, where the state equation is an Itô-type linear stochastic singular equations with Markovian jump, and the cost function is quadratic. Consider the following stochastic singular system with Markovian jumps: where u k (t) ∈ R m k is the control process of player k, for which we denote the admissible control set by U k [0, T] L 2 F (0, T; R m k ), k = 1, 2. For player k, the cost functional is defined by We give an assumption imposed throughout this section.
To summarize, we need to solve two LQ optimal control problems, the optimal control u 1 for the first one and the optimal controlū 2 for the second one. However, when a control is given, the corresponding optimal control problem is precisely what we have dealt with in Section 3. So we can directly use the developed conclusions.

Conclusion
We separately study an optimal control problem for a kind of stochastic singular affine systems with Markovian jumps in finite-and infinite-time horizons. By generalized Itô's formula and square completion technique we establish a sufficient condition for the wellposedness of control problem. In particular, we also obtain the solvability of GDREs by applying some matrix decomposition. As a typical application, we discuss a leader-follower differential game. Due to the considerable application potential of this class of stochastic affine singular systems, it will receive more research attention. We also need to pay attention to the fact that it is necessary to calculate an explicit representation of the solutions to GDREs and GBDEs. Therefore we will leave these issues for research in the future.