Updating QR factorization procedure for solution of linear least squares problem with equality constraints
- Salman Zeb^{1}Email author and
- Muhammad Yousaf^{1}
https://doi.org/10.1186/s13660-017-1547-0
© The Author(s) 2017
Received: 5 July 2017
Accepted: 16 October 2017
Published: 13 November 2017
Abstract
In this article, we present a QR updating procedure as a solution approach for linear least squares problem with equality constraints. We reduce the constrained problem to unconstrained linear least squares and partition it into a small subproblem. The QR factorization of the subproblem is calculated and then we apply updating techniques to its upper triangular factor R to obtain its solution. We carry out the error analysis of the proposed algorithm to show that it is backward stable. We also illustrate the implementation and accuracy of the proposed algorithm by providing some numerical experiments with particular emphasis on dense problems.
Keywords
MSC
1 Introduction
The solution of LSE problem (1) can be obtained using direct elimination, the nullspace method and method of weighting. In direct elimination and nullspace methods, the LSE problem is first transformed into unconstrained linear least squares (LLS) problem and then it is solved via normal equations or QR factorization methods. In the method of weighting, a large suitable weighted factor γ is selected such that the weighted residual \(\gamma(d-Bx)\) remains of the same size as that of the residual \(b-Ax\) and the constraints is satisfied effectively. Then the solution of the LSE problem is approximated by solving the weighted LLS problem. In [9], the author studied the method of weighting for LSE problem and provided a natural criterion of selecting the weighted factor γ such that \(\gamma\geq \Vert A \Vert _{2}/ \Vert B \Vert _{2} \epsilon _{M}\), where \(\epsilon_{M}\) is the rounding unit. For further details as regards methods of solution for LSE problem (1), we refer to [2, 3, 7, 10–13].
Updating is a process which allow us to approximate the solution of the original problem without solving it afresh. It is useful in applications such as in solving a sequence of modified related problems by adding or removing data from the original problem. Stable and efficient methods of updating are required in various fields of science and engineering such as in optimization and signal processing [14], statistics [15], network and structural analysis [16, 17] and discretization of differential equations [18]. Various updating techniques based on matrix factorization for different kinds of problems exist in the literature [3, 11, 13, 19–22]. Hammarling and Lucas [23] discussed updating the QR factorization with applications to LLS problem and presented updating algorithms which exploited LEVEL 3 BLAS. Yousaf [24] studied repeated updating based on QR factorization as a solution tool for LLS problem. Parallel implementation on GPUs of the updating QR factorization algorithms presented in [23] is performed by Andrew and Dingle [25]. Zhdanov and Gogoleva [26] studied augmented regularized normal equations for solving LSE problems. Zeb and Yousaf [27] presented an updating algorithm by repeatedly updating both factors of the QR factorization for the solutions of LSE problems.
We organized this article as follows. Section 2 contains preliminaries related to our main results. In Section 3, we present the QR updating procedure and algorithm for solution of LSE problem (1). The error analysis of the proposed algorithm is provided in Section 4. Numerical experiments and comparison of solutions is given in Section 5, while our conclusion is given in Section 6.
2 Preliminaries
This section contains important concepts which will be used in the forthcoming sections.
2.1 The method of weighting
This method is based on the observations that while solving LSE problem (1) we are interested that some equations are to be exactly satisfied. This can be achieved by multiplying large weighted factor γ to those equations. Then we can solve the resulting weighted LLS problem (3). The method of weighting is useful as it allows for the use of subroutines for LLS problems to approximate the solution of LSE problem. However, the use of the large weighted factor γ can compromise the conditioning of the constrained matrix. In particular, the method of normal equations when applied to problem (3) fails for large values of γ in general. For details, see [3, 11–13].
2.2 QR factorization and householder reflection
3 Updating QR factorization procedure for solution of LSE problem
Now, we calculated the QR factorization of the incomplete subproblem (13) is order to reduce it to the upper triangular matrix \(R_{2}\) using the following algorithm (Algorithm 2).
Here house denotes the Householder algorithm and the Householder vectors are calculated using Algorithm 1 and V is a self-explanatory intermediary variable.
Here the term triu denotes the upper triangular part of the concerned matrix and V is the intermediary variable.
Here qr is the MATLAB command of QR factorization and V is a self-explanatory intermediary variable.
The solution of problem (7) can then be obtained by applying the MATLAB built-in command backsub for a back-substitution procedure.
The description of QR updating algorithm in compact form is given as follows (Algorithm 5).
Here, Algorithm 5 for a solution of LSE problem (1) calls upon the partition process, Algorithms 2, 3, 4 and MATLAB command backsub for back-substitution procedure, respectively.
4 Error analysis
In this section, we will study the backward stability of our proposed Algorithm 5. The mainstay in our presented algorithm for the solution of LSE problem is the updating procedure. Therefore, our main concern is to study the error analysis of the updating steps. For others, such as the effect of using the weighting factor, finding the QR factorization and for the back-substitution procedure, we refer the reader to [28, 31]. Here, we recall some important results without giving their proofs and refer the reader to [28].
Lemma 4.1
([28])
Lemma 4.2
([28])
Lemma 4.3
([28])
Lemma 4.4
([28])
Lemma 4.5
([29])
Lemma 4.6
([28])
Theorem 4.7
([28])
Lemma 4.8
([28])
4.1 Backward error analysis of proposed algorithm
To appreciate the backward stability of our proposed Algorithm 5, we first need to carry out the error analysis of Algorithms 3 and 4. For this purpose, we present the following.
Theorem 4.9
Proof
Theorem 4.10
Proof
Theorem 4.11
Proof
5 Numerical experiments
Description of test problems
Problem | Size of (A) | κ ( A ) | \(\boldsymbol {\Vert A \Vert _{F}}\) | Size of (B) | κ ( B ) | \(\boldsymbol { \Vert B \Vert _{F}}\) |
---|---|---|---|---|---|---|
1. | 10×8 | 1.3667e+02 | 2.0006e+02 | 6×8 | 7.4200e+01 | 1.0216e+02 |
2. | 100×90 | 2.9303e+03 | 2.1395e+03 | 90×90 | 3.3687e+03 | 1.3735e+03 |
3. | 800×700 | 6.2106e+03 | 1.6872e+04 | 600×700 | 1.6164e+03 | 9.9000e+03 |
4. | 1,000×500 | 1.1602e+03 | 1.5943e+04 | 500×500 | 1.2883e+05 | 7.6360e+03 |
5. | 2,000×1,000 | 1.6727e+03 | 3.1884e+04 | 1,000×1,000 | 1.7430e+06 | 1.5272e+04 |
Results comparison
Problem | Size of (E) | γ | Size of SP | err1 | err |
---|---|---|---|---|---|
1. | 16×8 | 8.9564e+15 | 3×3 | 1.3222e−14 | 1.4585e−15 |
2. | 190×90 | 7.1258e+15 | 3×3 | 1.2628e−13 | 5.5294e−14 |
3. | 1,400×700 | 7.7993e+15 | 3×3 | 1.2821e−12 | 4.2522e−13 |
4. | 1,500×500 | 9.5551e+15 | 3×3 | 1.9377e−12 | 1.3559e−12 |
5. | 3,000×1,000 | 9.5549e+15 | 3×3 | 1.0828e−10 | 8.5181e−12 |
Backward error analysis results
Problem | \(\boldsymbol {\frac{ \Vert E-\tilde{Q} \tilde{R} \Vert _{F}}{ \Vert E \Vert _{F}}}\) | \(\boldsymbol {\Vert I-\tilde{Q}^{T} \tilde{Q} \Vert _{F}}\) |
---|---|---|
1. | 4.4202e−16 | 1.3174e−15 |
2. | 4.7858e−16 | 9.0854e−15 |
3. | 1.0450e−15 | 4.9428e−14 |
4. | 9.0230e−16 | 3.8711e−14 |
5. | 9.9304e−16 | 6.4026e−14 |
6 Conclusion
The solution of linear least squares problems with equality constraints is studied by updated techniques based on QR factorization. We updated only the R factor of the QR factorization of the small subproblem in order to obtain the solution of our considered problem. Numerical experiments are provided which illustrated the accuracy of the presented algorithm. We also showed that the algorithm is backward stable. The presented approach is suitable for dense problems and also applicable where QR factorization of a problem matrix is available and we are interested in the solution after adding new data to the original problem. In the future, it will be of interest to study the updating techniques for sparse data problems and for those where the linear least squares problem is fixed and the constraint system is changing frequently.
Declarations
Authors’ contributions
All authors contributed equally to this work. All authors read and approved the final manuscript.
Competing interests
The authors declare that they have no competing interests.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Authors’ Affiliations
References
- Auton, JR, Van Blaricum, ML: Investigation of procedures for automatic resonance extraction from noisy transient electromagnetics data. Final Report, Contract N00014-80-C-029, Office of Naval Research Arlington, VA 22217 (1981) Google Scholar
- Barlow, JL, Handy, SL: The direct solution of weighted and equality constrained least-squares problems. SIAM J. Sci. Stat. Comput. 9, 704-716 (1988) View ArticleMATHMathSciNetGoogle Scholar
- Lawson, CL, Hanson, RJ: Solving Least Squares Problems. SIAM, Philadelphia (1995) View ArticleMATHGoogle Scholar
- Eldén, L: Perturbation theory for the least squares problem with linear equality constraints. SIAM J. Numer. Anal. 17, 338-350 (1980) View ArticleMATHMathSciNetGoogle Scholar
- Leringe, O, Wedin, PÅ: A comparison between different methods to compute a vector x which minimizes ∥ A x − b ∥ 2 $\Vert Ax-b \Vert _{2}$ when G x = h $Gx = h$ . Tech. Report, Department of computer science, Lund University (1970) Google Scholar
- Van Loan, CF: A generalized SVD analysis of some weighting methods for equality constrained least squares. In: Kågström, B, Ruhe, A (eds.) Matrix Pencils. Lecture Notes in Mathematics, vol. 973, pp. 245-262. Springer, Heidelberg (1983) View ArticleGoogle Scholar
- Van Loan, CF: On the method of weighting for equality-constrained least-squares problems. SIAM J. Numer. Anal. 22, 851-864 (1985) View ArticleMATHMathSciNetGoogle Scholar
- Wei, M: Algebraic properties of the rank-deficient equality-constrained and weighted least squares problems. Linear Algebra Appl. 161, 27-43 (1992) View ArticleMATHMathSciNetGoogle Scholar
- Stewart, GW: On the weighting method for least squares problems with linear equality constraints. BIT Numer. Math. 37, 961-967 (1997) View ArticleMATHMathSciNetGoogle Scholar
- Anderson, E, Bai, Z, Dongarra, J: Generalized QR factorization and its applications. Linear Algebra Appl. 162, 243-271 (1992) View ArticleMATHMathSciNetGoogle Scholar
- Björck, Å: Numerical Methods for Least Squares Problems. SIAM, Philadelphia (1996) View ArticleMATHGoogle Scholar
- Golub, GH, Van Loan, CF: Matrix Computations. Johns Hopkins University Press, Baltimore (1996) MATHGoogle Scholar
- Stewart, GW: Matrix Algorithms. SIAM, Philadelphia (1998) View ArticleMATHGoogle Scholar
- Alexander, ST, Pan, CT, Plemmons, RJ: Analysis of a recursive least squares hyperbolic rotation algorithm for signal processing. Linear Algebra Appl. 98, 3-40 (1988) View ArticleMATHMathSciNetGoogle Scholar
- Chambers, JM: Regression updating. J. Am. Stat. Assoc. 66, 744-748 (1971) View ArticleGoogle Scholar
- Haley, SB, Current, KW: Response change in linearized circuits and systems: computational algorithms and applications. Proc. IEEE 73, 5-24 (1985) View ArticleGoogle Scholar
- Haley, SB: Solution of modified matrix equations. SIAM J. Numer. Anal. 24, 946-951 (1987) View ArticleMATHMathSciNetGoogle Scholar
- Lai, SH, Vemuri, BC: Sherman-Morrison-Woodbury-formula-based algorithms for the surface smoothing problem. Linear Algebra Appl. 265, 203-229 (1997) View ArticleMATHMathSciNetGoogle Scholar
- Björck, Å, Eldén, L, Park, H: Accurate downdating of least squares solutions. SIAM J. Matrix Anal. Appl. 15, 549-568 (1994) View ArticleMATHMathSciNetGoogle Scholar
- Daniel, JW, Gragg, WB, Kaufman, L, Stewart, GW: Reorthogonalization and stable algorithms for updating the Gram-Schmidt QR factorization. Math. Comput. 30, 772-795 (1976) MATHMathSciNetGoogle Scholar
- Gill, PE, Golub, GH, Murray, W, Saunders, M: Methods for modifying matrix factorizations. Math. Comput. 28, 505-535 (1974) View ArticleMATHMathSciNetGoogle Scholar
- Reichel, L, Gragg, WB: Algorithm 686: FORTRAN subroutines for updating the QR decomposition. ACM Trans. Math. Softw. 16, 369-377 (1990) View ArticleMATHGoogle Scholar
- Hammarling, S, Lucas, C: Updating the QR factorization and the least squares problem. Tech. Report, The University of Manchester (2008). http://www.manchester.ac.uk/mims/eprints
- Yousaf, M: Repeated updating as a solution tool for linear least squares problems. Dissertation, University of Essex (2010) Google Scholar
- Andrew, R, Dingle, N: Implementing QR factorization updating algorithms on GPUs. Parallel Comput. 40, 161-172 (2014) View ArticleMathSciNetGoogle Scholar
- Zhdanov, AI, Gogoleva, SY: Solving least squares problems with equality constraints based on augmented regularized normal equations. Appl. Math. E-Notes 15, 218-224 (2015) MATHMathSciNetGoogle Scholar
- Zeb, S, Yousaf, M: Repeated QR updating algorithm for solution of equality constrained linear least squares problems. Punjab Univ. J. Math. 49, 51-61 (2017) MathSciNetMATHGoogle Scholar
- Higham, NJ: Accuracy and Stability of Numerical Algorithms. SIAM, Philadelphia (2002) View ArticleMATHGoogle Scholar
- Wilkinson, JH: Error analysis of transformations based on the use of matrices of the form \(I-2ww^{H}\). In: Rall, LB (ed.) Error in Digital Computation, vol. 2, pp. 77-101. Wiley, New York (1965) Google Scholar
- Parlett, BN: Analysis of algorithms for reflections in bisectors. SIAM Rev. 13, 197-208 (1971) View ArticleMATHMathSciNetGoogle Scholar
- Cox, AJ: Stability of algorithms for solving weighted and constrained least squares problems. Dissertation, University of Manchester (1997) Google Scholar