The modified accelerated Bregman method for regularized basis pursuit problem
© Xie et al.; licensee Springer. 2014
Received: 9 November 2013
Accepted: 18 March 2014
Published: 31 March 2014
In this paper, a modified accelerated Bregman method (MABM) for solving the regularized basis pursuit problem is considered and analyzed in detail. This idea is based on the fact that the linearized Bregman method (LBM) proposed by Osher et al. (Multiscale Model. Simul. 4(2):460-489, 2005) is equivalent to a gradient descent method applied to a certain dual formulation which converges to the solution of the regularized basis pursuit problem. The proposed method is based on an extrapolation technique which is used in accelerated proximal gradient methods presented by Nesterov (Dokl. Akad. Nauk SSSR 269:543-547, 1983). It is verified that the modified accelerated Bregman method (MABM) is equivalent to the corresponding accelerated augmented Lagrangian method (AALM). The theoretical results confirm that the method has a rapid convergence rate of .
where , , denotes the number of nonzero elements.
where the linear map , is a given vector.
Particularly in , putting in the term in (1.7) yields the tractable object function, which is a strictly convex function. Thus, the linearly constrained basis pursuit problem has a sole solution and its dual problem is smooth. When μ is sufficiently small, the solution to (1.7) is also the solution of (1.1). This exact regularization property of (1.7) was studied in [19, 20]. Problem (1.7) has a 2-norm in the regularizer term, thus it is considered less sensitive to noise than the basis pursuit problem (1.1).
To solve (1.1), a linearized Bregman iteration method was presented in , which was motivated by . The idea of the linearized Bregman method is to combine a fixed point iteration with the Bregman method in [23, 24], but the linearized Bregman method is a bit slow in the convergence result. Hence, many accelerated schemes have been brought up in various theses. For example, in  the kicking technique was introduced. Yin  verified that the linearized Bregman method is equivalent to a gradient descent method applied to the Lagrangian dual problem of (1.7). They improved the linearized Bregman method, utilizing the Barzilai-Borwein line of searching , nonlinear conjugate gradient methods, and the method of limited memory BFGS . Huang et al.  proposed an accelerated linearized Bregman method which is based on the fact that the linearized Bregman method is equivalent to the gradient descent method applied to a Lagrangian dual of problem (1.7) and the extrapolation technique, which is adopted in the accelerated proximal gradient methods  proposed by Nesterov et al. To solve problem (1.7), Goldstein et al.  used an alternating split technique and its Lagrange dual problem.
Based on these studies, we extend the accelerated Bregman method to solve (1.7) in which the object function might be not differentiable but have the ‘good’ performance (convex and continuous). We put forward a new improvement formula based on the accelerated Bregman method. It can be proved to have the property that the modified Bregman method is equivalent to the corresponding accelerated augmented Lagrangian method, and the latter has a rapid convergence rate which can be deemed to be an improvement of .
The rest of this article is organized as follows. In Section 2, we sketch the original Bregman method and the linearized Bregman method which are useful for the subsequent analysis. In Section 3, we introduce the accelerated augmented Lagrangian method (AALM), and we present our modified accelerated Bregman method (MABM). Section 4 is devoted to the convergence of the regularized basis pursuit problem and here we analyze the error bound of the MABM in detail. In Section 5, we give some conclusions and discuss the research plans for our future work.
2 The original Bregman method and its linearized form
starting with , .
For the sake of the requirement for the whole theoretical analysis, in the following, we propose some significant equivalence results, and we give the detailed proofs.
holds, where λ is a certain positive constant.
therefore, we complete the proof. □
From the discussion above, we can give a crucial conclusion.
Theorem 2.2 The original Bregman iterative scheme (2.5) is equivalent to its variant (2.6).
Proof By induction, in fact, we only need to verify that (2.7) holds.
If , , (2.5) holds by the initial conditions and .
where the first equality is from the second term of (2.5), and the third equality is from the second term of (2.6). Therefore, the original Bregman iterative scheme (2.5) is equivalent to its variant (2.6). □
Noting that [24, 30, 31] and the references therein, it was argued that the Bregman iterative method is equivalent to the augmented Lagrangian method. The significance of this statement will be demonstrated in our later analysis in Section 4.
where and is the penalty term.
starting from .
The constant of the objective function does not affect the optimum point, so by comparing with the above equations, we get the conclusion. □
Theorem 2.4 The augmented Lagrangian iterative scheme (2.9) is equivalent to the Bregman iterative method variant (2.6).
Proof It is not difficult to see that the proof is the same as that of Theorem 2.2. Similarly, by the mathematical induction, we simply show that (2.7) holds.
where the first equality is from the second term of (2.6), the second equality has its roots in induction, and the third equality is derived from the second term of (2.9). Therefore, the augmented Lagrangian iterative scheme (2.9) is equivalent to the Bregman iterative method variant (2.6). Moreover, we can get the equivalence of (2.5) and (2.9) from the two theorems above. □
It has appeared in the accelerated Bregman algorithm in recent years, such as in etc. But we shall argue that the accelerated Bregman method has large room to advance. We give some valid improvements on these accelerated Bregman methods. It can be verified that the theoretical result as regards the proposed method has a rapid convergence rate of .
3 The modified accelerated Bregman method
In , the linearized Bregman algorithm could be written as follows.
Algorithm 3.1 (Linearized Bregman method (LBM))
Step 0. Input: ; initial point: , .
Step 1. Initialize: , let and .
Step 2. Compute .
Step 3. Set .
Step 4. Set , go to step 2.
For the iterative scheme above, . Next, we give its equivalence form from the following lemma.
starting from , where .
where the second equality is from (3.2). So, step 2 and step 3 in Algorithm 3.1 can be rewritten as (3.1). □
Yin et al. presented all kinds of techniques, for example, line search, and L-BFGS and BB steps, to accelerate the linearized Bregman method. It is interesting that for the latter in  the accelerated linearized Bregman algorithm is argued for as follows.
Algorithm 3.3 (Accelerated linearized Bregman method (ALBM))
Step 0. Initialize: , , , .
Step 1. Compute .
Step 2. Set .
Step 3. Set .
Step 4. Set .
Step 5. Set , go to step 1.
Algorithm 3.4 (Modified accelerated Bregman method (MABM))
Step 0. Initialize: , , , .
Step 1. Compute .
Step 2. Set .
Step 3. Set .
Step 4. Set .
Step 5. Set , go to step 2.
The basic idea of the equivalence between MABA and the corresponding AALM can be traced back to . Especially, we can see that our updated iterative for is obviously better than the pre-iterative , since we consider a sufficient amount of information about the former iterative. In this way, most of the better iterative efficiency could be expected, which is just our purpose in improving the method. Then we will be dependent on a series of transformations in preparation for the convergence proof in Section 4.
starting with .
Proof Recalling (2.7) and noting Theorem 2.2, we can prove that (2.7) holds for all k by induction.
where the first equality is directly derived from step 2 of Algorithm 3.4, the first equality is derived from the induction hypothesis, the fourth equality utilizes the second step of (3.3).
where the first equality is from step 3 of Algorithm 3.4, the second term is from (3.4) and the induction hypothesis, the fourth equality is from the third term of (3.3), so (2.7) holds for all k. Namely, the MABM in Algorithm 3.4 is equal to (3.3). □
starting from , where is the Lagrangian multiplier.
Proof It is not difficult to see that the idea has likeness to Theorem 2.4, we are just required to verify that (2.7) holds. To this end, we proceed by mathematical induction.
where the first equality is from the second term of (3.3), the second equality stems from induction, and the third equality is derived from the second term of (3.5).
Therefore, AALM (3.5) is equivalent to iterative scheme (3.3). Moreover, we can get the equivalence of the MABM in Algorithm 3.4 to AALM from the above two lemmas. □
Theorem 3.7 The MABM in Algorithm 3.4 is equivalent to the corresponding AALM (3.5).
4 The convergence analysis
A practical challenge for the regularized basis pursuit problem is to offer an efficient method to solve the non-smooth optimization problems. Many algorithms have been proposed in recent years . In these methods, some schemes of approximation that have to do with the non-smooth norm term are usually employed. However, a fast global convergence is difficult to guarantee. Due to the non-smooth nature of the 1-norm, a simple method to solve these problems is the subgradient approach , which converges only as , where k is the iteration counter.
In this paper, we present an efficient method with fast global convergence rate to solve the regularized basis pursuit problem. Particularly, we verify that this result is an extended gradient algorithm with the convergence rate of , like that for smooth problems. Following the Nesterov method for accelerating the gradient method [34, 35], we show that the MABM can be further accelerated to converge as .
A series of lemmas in the following are to ensure the convergence rate of the MABM.
for any and holds.
Compared with (4.3), inequality (4.2) holds for , which completes the proof. □
The second inequality is derived from (4.2), and the third equality is from (2.9). □
where we exploit the fact that .
which completes the proof. □
Theorem 4.4 Let be generated by the augmented Lagrangian iterative scheme (2.9), then .
This completes the proof. □
Comparing (2.9) with (3.5), we have the following two lemmas by replacing with .
where the second equality is derived from the fact that .
Thus, we complete the convergence proof. □
Remark The right formula of (4.18) exploits the fact that the choice of the penalty factor λ can be seen as a monotonically increasing sequence (such as ) that depends on the selection of in Algorithm 3.4. In this way, we are not only able to guarantee the convergence in the formula of divided by , but also to play a critical role of punishment to the constraint condition.
In this paper, we put forward the modified accelerated Bregman method (MABM) for solving the regularized basis pursuit problem. We give some beneficial improvement tasks on the basis of some recent literature on the accelerated Bregman method, and we perform the theoretical feasibility analysis in detail. It can be showed that the proposed MABM has a rapid convergence rate of . We will devote our future study to combining the advantages of LBM with our MABM as regards theory and numerical results.
The project is supported by the Scientific Research Special Fund Project of Fujian University (Grant No. JK2013060), Fujian Natural Science Foundation (Grant No. 2013J01006) and the National Natural Science Foundation of China (Grant No. 11071041).
- Kashin B: The widths of certain finite dimensional sets and classes of smooth functions. Izv. Akad. Nauk SSSR, Ser. Mat. 1977, 41: 334-351.MATHMathSciNetGoogle Scholar
- Candés E, Romberg J, Tao T: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 2006, 52: 489-509.MATHView ArticleGoogle Scholar
- Candés E, Romberg J, Tao T: Stable signal recovery from incomplete and inaccurate measurements. Commun. Pure Appl. Math. 2006, 59: 1207-1223. 10.1002/cpa.20124MATHView ArticleGoogle Scholar
- Candés E, Tao T: Decoding by linear programming. IEEE Trans. Inf. Theory 2005, 51: 4203-4215. 10.1109/TIT.2005.858979MATHView ArticleGoogle Scholar
- Donoho D: Compressed sensing. IEEE Trans. Inf. Theory 2006, 52: 1289-1306.MATHMathSciNetView ArticleGoogle Scholar
- Candés E, Wakin M: An introduction to compressive sampling. IEEE Signal Process. Mag. 2008, 21: 21-30.View ArticleGoogle Scholar
- Cai J, Osher S, Shen Z: Linearized Bregman iterations for compressed sensing. Math. Comput. 2009, 78: 1515-1536. 10.1090/S0025-5718-08-02189-3MATHMathSciNetView ArticleGoogle Scholar
- Cai J, Candés E, Shen Z: A singular value thresholding algorithm for matrix completion. SIAM J. Optim. 2010, 20: 1956-1982. 10.1137/080738970MATHMathSciNetView ArticleGoogle Scholar
- Candés E, Recht B: Exact matrix completion via convex optimization. Found. Comput. Math. 2009, 9: 717-772. 10.1007/s10208-009-9045-5MATHMathSciNetView ArticleGoogle Scholar
- Candés, E, Li, X, Ma, Y, Wright, J: Robust principal component analysis. Preprint (2009)Google Scholar
- Figueiredo MAT, Nowak RD, Wright SJ: Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE J. Sel. Top. Signal Process. 2007,1(4):586-597.View ArticleGoogle Scholar
- Beck A, Teboulle M: Fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2009, 2: 183-202. 10.1137/080716542MATHMathSciNetView ArticleGoogle Scholar
- Peng Y, Ganesh A, Wright J, Xu W, Ma Y: RASL: robust alignment by sparse and low-rank decomposition for linearly correlated images. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2010.Google Scholar
- Ghaoui LE, Gahinet P: Rank minimization under LMI constraints: a framework for output feedback problems. Proceedings of the European Control Conference 1993.Google Scholar
- Fazel M, Hindi H, Boyd SP: A rank minimization heuristic with application to minimum order system approximation. 6. Proceedings of the American Control Conference 2001, 4734-4739.Google Scholar
- Linial N, London E, Rabinovich Y: The geometry of graphs and some of its algorithmic applications. Combinatorica 1995, 15: 215-245. 10.1007/BF01200757MATHMathSciNetView ArticleGoogle Scholar
- Kang M, Yun S, Woo H, Kang M:Accelerated Bregman method for linearly constrained minimization. J. Sci. Comput. 2012, 56: 515-534.MathSciNetView ArticleGoogle Scholar
- Huang B, Ma SQ, Goldfarb D: Accelerated linearized Bregman method. J. Sci. Comput. 2013, 54: 428-453. 10.1007/s10915-012-9592-9MATHMathSciNetView ArticleGoogle Scholar
- Yin W: Analysis and generalizations of the linearized Bregman method. SIAM J. Imaging Sci. 2010, 3: 856-877. 10.1137/090760350MATHMathSciNetView ArticleGoogle Scholar
- Friedlander M, Tseng P: Exact regularization of convex programs. SIAM J. Optim. 2007, 18: 1326-1350.MATHMathSciNetView ArticleGoogle Scholar
- Osher S, Mao Y, Dong B, Yin W: Fast linearized Bregman iteration for compressive sensing and sparse denoising. Commun. Math. Sci. 2010, 8: 93-111. 10.4310/CMS.2010.v8.n1.a6MATHMathSciNetView ArticleGoogle Scholar
- Darbon, J, Osher, S: Fast discrete optimization for sparse approximations and deconvolutions. Preprint (2007)Google Scholar
- Osher S, Burger M, Goldfarb D, Xu J, Yin W: An iterative regularization method for total variation-based image restoration. Multiscale Model. Simul. 2005,4(2):460-489. (electronic). MR2162864 (2006c:49051) 10.1137/040605412MATHMathSciNetView ArticleGoogle Scholar
- Yin W, Osher S, Goldfarb D, Darbon J:Bregman iterative algorithms for -minimization with applications to compressed sensing. SIAM J. Imaging Sci. 2008,1(1):143-168. 10.1137/070703983MATHMathSciNetView ArticleGoogle Scholar
- Barzilai J, Borwein J: Two point step size gradient methods. IMA J. Numer. Anal. 1988, 8: 141-148. 10.1093/imanum/8.1.141MATHMathSciNetView ArticleGoogle Scholar
- Liu D, Nocedal J: On the limited memory method for large scale optimization. Math. Program., Ser. B 1989, 45: 503-528. 10.1007/BF01589116MATHMathSciNetView ArticleGoogle Scholar
- Bennett J, Lanning S: The Netflix prize. Proceedings of KDD Cup and Workshop 2007.Google Scholar
- Goldstein T, Osher S: The split Bregman method for L1-regularized problems. SIAM J. Imaging Sci. 2009,2(2):323-343. 10.1137/080725891MATHMathSciNetView ArticleGoogle Scholar
- Bregman L: The relaxation method of finding the common points of convex sets and its application to the solution of problems in convex programming. USSR Comput. Math. Math. Phys. 1967, 7: 200-217.View ArticleGoogle Scholar
- Powell MJD: A method for nonlinear constraints in minimization problems. In Optimization Edited by: Fletcher R. 1972, 283-298.Google Scholar
- Rockafellar RT: Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Math. Oper. Res. 1976, 1: 97-116. 10.1287/moor.1.2.97MATHMathSciNetView ArticleGoogle Scholar
- Hale ET, Yin W, Zhang Y:Fixed-point continuation for minimization: methodology and convergence. SIAM J. Optim. 2008, 19: 1107-1130. 10.1137/070698920MATHMathSciNetView ArticleGoogle Scholar
- Berg EVD, Friedlander MP: Probing the Pareto frontier for basis pursuit solutions. SIAM J. Sci. Comput. 2008, 31: 890-912.MATHMathSciNetView ArticleGoogle Scholar
- Nesterov YE:A method for unconstrained convex minimization problem with the rate of convergence . Dokl. Akad. Nauk SSSR 1983, 269: 543-547.MathSciNetGoogle Scholar
- Nesterov YE 87. Introductory Lectures on Convex Optimization 2004, 220-236.View ArticleGoogle Scholar
- Rennie JDM, Srebro N: Fast maximum margin matrix factorization for collaborative prediction. Proceedings of the International Conference on Machine Learning 2005, 713-719.Google Scholar
- Bertsekas D: Nonlinear Programming. Athena Scientific, Nashua; 1999.MATHGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.