In this paper, a hybrid splitting method is proposed for solving a smoothing Tikhonov regularization problem. At each iteration, the proposed method solves three subproblems. First of all, two subproblems are solved in a parallel fashion, and the multiplier associated to these two block variables is updated in a rapid sequence. Then the third subproblem is solved in the sense of an alternative fashion with the former two subproblems. Finally, the multiplier associated to the last two block variables is updated. Global convergence of the proposed method is proven under some suitable conditions. Some numerical experiments on the discrete ill-posed problems (DIPPs) show the validity and efficiency of the proposed hybrid splitting method.
Keywords
Tikhonov regularizationaugmented Lagrangian methodparallel splitting methodalternative direction method of multipliers
1 Introduction
In this paper, we consider a smoothing Tikhonov regularization problem, which is an unconstrained minimization of the form [1]
where \(A\in R^{m\times n}\), \(b\in R^{m}\), and \(x\in\mathcal{X} \subset R^{n} \), and \(\|\cdot\|\) denotes the Euclid norm. The parameters \(\delta \geq0\) and \(\eta\geq0\) are used to control the smoothness and size of the approximate solution. Matrix Δ is a (tridiagonal, Toeplitz) matrix, Δx represents a measure of the variation or smoothness of x, where
The smoothing regularization problem (1.1) has numerous applications in many fields, including mathematical programs with vanishing constraints [2], maximum-likelihood estimation problem [3], language modeling [4], and so on.
The last term of the problem (1.1), \(\eta\|x\|^{p}_{p}\), is a regularization term. As a common regularization method, \(\ell_{1}\) regularization (\(p=1\)) problem has many good properties since it is a convex programming problem. In recent years, there has been an increasing interest in the \(\ell_{1}\) regularizer. The \(\ell_{1}\) regularization model can reconstruct the original signal with less observed signals, when the original signal is spare or the observed signal contains noise. Especially, the \(\ell_{1}\) formulation suits significantly better for denoising data containing so-called outliers, i.e., observations containing large measurement errors [5]. Therefore, mathematical models and large-scale fast algorithms associated with \(\ell_{1}\) regularization can be seen everywhere in compressed sensing, signal/image processing, machine learning, statistics, and many other fields [6–9].
By \(\ell_{1}\) regularization, the problem (1.1) reduces to
Indeed, there are many methods for solving the problem (1.4) in the literature. Among these methods, the parallel splitting augmented Lagrangian method and the alternating direction method of multipliers are two power tools. The recent research indicates that, due to the separable convex optimization with three block variables, the direct extension of the alternating direction method of multipliers is not necessarily convergent [10]. Thus, some hybrid splitting methods can be found in the literature. For example, see He [11], Peng and Wu [12], and Glowinski and Le Tallec [13].
The saddle point of a Lagrange function associated to the convex optimization problem (1.4), \(w^{*}=(x^{*},y^{*},\lambda^{*},z^{*},\mu^{*})\in{\mathcal{W}}\), satisfies the following variational inequalities:
In this paper, we will propose a new splitting method to solve the separable convex programming problem (1.4) as well as the structured variational inequalities (1.6). The proposed method, referred to as the hybrid splitting method (HSM), will combine a parallel splitting (augmented Lagrangian) method and an alternating directions method of multipliers. In the HSM, the predictor of the new iterate, \(\tilde{w}^{k}=(\tilde{x}^{k}, \tilde{y}^{k}, \tilde{\lambda }^{k}, \tilde{z}^{k}, \tilde{\mu}^{k} )\), is got in the following way: find \((\tilde{x}^{k},\tilde{y}^{k})\) in a parallel manner, and update \(\tilde{\lambda}^{k}\). Then compute \(\tilde{z}^{k}\) alternately with \((\tilde{x}^{k},\tilde{y}^{k})\), and update \(\tilde{\mu}^{k}\) at the last. The new iterate is produced by a correct operator. The global convergence of the HSM is proven under some wild assumptions.
The rest of this paper is organized as follows. In Section 2, we describe the proposed hybrid splitting method. Section 3 is devoted to showing that the sequence \(\{ w^{k} \}\) generated by the HSM is Fejér monotone with respect to the solution set. Then, the convergence of the HSM is proved. In Section 4, some preliminary numerical results are presented which indicate the feasibility and efficiency of the proposed method. Finally, some concluding remarks are made in Section 5.
2 The hybrid splitting method
In this section, we first propose a hybrid splitting method for the problem (1.6), and then we give some remarks on the described method.
Algorithm 2.1
(The hybrid splitting method (HSM))
For a given \(w^{k}=(x^{k}, y^{k}, \lambda^{k}, z^{k}, u^{k}) \in{\mathcal{W}}\), \(\beta_{k}>0 \), and \(\rho_{k}>0 \), the HSM produces the new iterate \({w}^{k+1}=(x^{k+1}, y^{k+1}, \lambda^{k+1} ,z^{k+1}, u^{k+1})\in {\mathcal{W}}\) by the following scheme:
S1.
Produce \(\tilde{w}^{k}=(\tilde{x}^{k},\tilde{y}^{k},\tilde{\lambda}^{k},\tilde {z}^{k},\tilde{\mu}^{k})\) by s1.1 to s1.5.
s1.1.
Find \(\tilde{x}^{k}\in\mathcal{X}\) (with fixed \(y^{k}\), \(\lambda^{k}\), \(z^{k}\), \(\mu^{k}\), \(\beta_{k}\), \(\rho_{k}\)) such that
Convergence verification: for a given small \(\varepsilon >0\), if \(\| w^{k}-\tilde{w}^{k}\|_{\infty}<\varepsilon\) then stop, and accept \(w^{k}\) to be the approximate solution. Else, go to S3.
The parameters \(\beta_{k}\) and \(\rho_{k}\) are updated in the same style as proposed in He et al. [14]. By Remark 5.1 in [14], the sequences \(\{\beta _{k}\}\) and \(\{\rho_{k}\}\) are bounded and finally constants. Thus, there is a \(\kappa>0\) such that \(\|M\|^{2}_{G}:=\sum\|M(:, j)\| ^{2}_{G}\le\kappa\).
Remark 2.2
It is easy to show, \(\|M(w-\tilde{w})\|_{G}^{2}\le\| M\|_{G}^{2}\cdot\|w-\tilde{w}\|^{2}_{G}\). Thus by (2.8) we have
$$ \alpha_{k}^{*}\ge\frac{1}{\|M\| ^{2}_{G}}\ge\frac{1}{\kappa}, \quad \forall k. $$
(2.10)
For convenience in the analysis, the following notations are useful:
To prove the convergence of the HSM, we will show first in this section that the sequence \(\{w^{k}\}\) generated by the HSM is Fejér monotone with respect to the solution set \(\mathcal{W}^{*}\) of the problem (2.12).
Due to (1.6), multiplying both sides of the third inequality by 2, and multiplying both sides of the last inequality by 3, respectively, we get
$$ \bigl(w'-w^{*}\bigr)^{T} H F\bigl(w^{*}\bigr) \ge0,\quad \forall w'\in\mathcal{W}. $$
(3.2)
Lemma 3.1
For a given\(w^{k}=(x^{k},y^{k},\lambda^{k},z^{k},\mu^{k})\), if\(\tilde{w}^{k}=(\tilde{x}^{k},\tilde{y}^{k}, \tilde{\lambda}^{k},\tilde {z}^{k},\tilde{\mu}^{k})\)is generated by (2.1)-(2.5), then we have
$$\bigl(\tilde{w}^{k}-w^{*}\bigr)^{T} HF\bigl( \tilde{w}^{k}\bigr)\ge\bigl(\tilde{w}^{k}-w^{*} \bigr)^{T} H F\bigl(w^{*}\bigr)\ge0. $$
□
Lemma 3.2
For a given\(w^{k}=(x^{k},y^{k},\lambda^{k},z^{k},\mu^{k})\), if\(\tilde{w}^{k}=(\tilde {x}^{k},\tilde{y}^{k}, \tilde{\lambda}^{k},\tilde{z}^{k},\tilde{\mu}^{k})\)is generated by (2.1)-(2.5), then we have
For a given\(w^{k}=(x^{k},y^{k},\lambda^{k},z^{k},\mu^{k})\), if\(\tilde{w}^{k}=(\tilde {x}^{k},\tilde{y}^{k}, \tilde{\lambda}^{k},\tilde{z}^{k},\tilde{\mu}^{k})\)is generated by (2.1)-(2.5), then we have
The assertion (3.9) is just a compact form of (3.10). □
Theorem 3.1
For a given\(w^{k}=(x^{k},y^{k},\lambda^{k},z^{k},\mu^{k})\), if\(\tilde{w}^{k}=(\tilde {x}^{k},\tilde{y}^{k}, \tilde{\lambda}^{k},\tilde{z}^{k},\tilde{\mu}^{k})\)is generated by (2.1)-(2.5), then for any\(w^{*}=(x^{*}, y^{*}, \lambda^{*},z^{*},\mu^{*}) \in{\mathcal{W}}^{*}\)we have
$$ \bigl(\tilde{w}^{k}-w^{*}\bigr)^{T}\bigl[ H F \bigl(\tilde{w}^{k}\bigr)+g\bigl(w^{k},\tilde{w}^{k} \bigr)\bigr]\ge\varphi \bigl(w^{k},\tilde{w}^{k}\bigr)- \bigl(w^{k}-\tilde{w}^{k}\bigr)^{T}d \bigl(w^{k},\tilde{w}^{k}\bigr). $$
(3.11)
Proof
Note \(d(w^{k},\tilde{w}^{k})=M(w^{k}-\tilde{w}^{k})\), by (3.5) we get
If\(\tilde{w}^{k}=(\tilde{x}^{k},\tilde {y}^{k}, \tilde{\lambda}^{k},\tilde{z}^{k},\tilde{\mu}^{k})\)is generated by (2.1)-(2.5) from a given\(w^{k}=(x^{k},y^{k}, \lambda^{k},z^{k},\mu^{k})\), then for any\(w^{*}=(x^{*}, y^{*}, \lambda^{*},z^{*},\mu^{*}) \in{\mathcal{W}}^{*}\), we have
$$ \bigl(w^{k}-w^{*}\bigr)^{T} G d \bigl(w^{k},\tilde{w}^{k}\bigr) \ge\varphi \bigl(w^{k},\tilde{w}^{k}\bigr). $$
(3.13)
Proof
We omit the proof of Lemma 3.4 here. A similar proof can be found in [15]. □
Theorem 3.2
For a given\(w^{k}=(x^{k},y^{k},\lambda^{k},z^{k},\mu^{k})\), if\(\tilde{w}^{k}=(\tilde {x}^{k},\tilde{y}^{k}, \tilde{\lambda}^{k},\tilde{z}^{k},\tilde{\mu}^{k})\)is generated by (2.1)-(2.5), then for any\(w^{*}=(x^{*}, y^{*}, \lambda^{*},z^{*},\mu^{*}) \in{\mathcal{W}}^{*}\)we have
which results in both \(\{w^{k}\}\) and \(\{\tilde{w}^{k}\}\) being bounded sequences and having cluster points. Let \(w^{\infty}\) be a cluster point of \(\{\tilde{w}^{k}\}\) and \(\{\tilde{w}^{k_{j}}\}\) be a subsequence converging to \(w^{\infty}\).
By the HSM, \(\tilde{w}^{k_{j}}\) is a solution of (2.12), thus
The limit (3.15) also results in \(\lim_{k_{j}\rightarrow \infty} g (w^{k_{j}}, \tilde{w}^{k_{j}})=0\) and \(\lim_{k_{j}\rightarrow\infty} d (w^{k_{j}}, \tilde{w}^{k_{j}})=M (w^{k_{j}}-\tilde{w}^{k_{j}})=0\) by positive-definiteness of G and M. Taking the limit on both sides of (3.16) we have
which implies \(w^{\infty}\) is a solution of the problem (1.4) by the optimality condition (2.11).
Note that (3.14) holds for all solutions of (1.4), we get
$$ \bigl\Vert w^{k+1}-w^{\infty}\bigr\Vert ^{2}_{G}\le\bigl\Vert w^{k}-w^{\infty}\bigr\Vert ^{2}_{G}, \quad \forall k. $$
(3.18)
Since \(\tilde{w}^{k_{j}} \rightarrow w^{\infty}\) and \(\|w^{k_{j}}-\tilde {w}^{k_{j}}\|^{2}_{G}\rightarrow0\) as \(k_{j} \rightarrow\infty\), for \(\forall\varepsilon>0\), there exists an integer \(k_{l}>0\) such that for all k and \(k_{j}\) larger than \(k_{l}\), we have
Thus, the sequence \(\{w^{k}\}\) converges to \(w^{\infty}\), which is a solution of (1.4), or equivalently, of the problem (1.1).
4 Numerical results
The main computational cost of the HSM is in solving the subproblems (2.1)-(2.2) and (2.4). For generality of the HSM, we ignore the special structure of the matrices A and Δ, and employ the existing efficient method to solve those subproblems. In this paper, the subproblems (2.1) and (2.2) are solved by the projected Barzilai-Borwein method proposed by Dai and Fletcher [16], and the subproblem (2.4) is solved by a shrinkage-thresholding algorithm [17]. The test problems are discrete ill-posed problems selected from Hansen [18]. All tests are done on a laptop with Core(TM) i7 M620@D, 2.67 GHz, 4 GB RAM and Matlab R2009a.
A classical example of an ill-posed problem is a Fredholm integral equation of the first kind with a square integrable kernel, which has the form
where \(u = \pi\times(\sin(s) + \sin(t))\). Both K and f are discretized by simple quadrature to produce A and x. Then the discrete right-hand side is produced as \(b = Ax\). In our test, the constants are assigned as follows: \(a_{1} = 2\), \(c_{1} = 6\), \(t_{1} = 0.8\); \(a_{2} = 1\), \(c_{2} = 2\), \(t_{2} = -0.5\). The numerical result is displayed in Figure 2(a).
Figure 2
Numerical results on Examples4.2and4.3: ‘∗’ is the true solution and ‘∘’ denotes the approximate solution.
Example 4.3
Baart test problem: the kernel K and right-hand side g of the discretization of a first-kind Fredholm integral equation are given by
where \(s \in[0, \pi/2]\), \(t \in[0, \pi]\). The solution is given by \(f(t) = \sin(t)\). The numerical result is displayed in Figure 2(b).
We can draw the conclusion from the above computational results: the proposed hybrid splitting method is valid and efficient for the smoothing Tikhonov regularization problem.
5 Conclusions
We have proposed a hybrid splitting method for solving the smoothing Tikhonov regularization problem, which is to minimize the sum of three convex functions over a simple closed convex set. The problem can be reformulated as a convex minimization problem with three separable block variables, and we get the variational inequalities formulation by the KKT conditions of the separable convex minimization problem. By the convexity of the problem, the solution of the resulting variational inequalities is the same as the solution of the convex minimization problem. At each iteration of the proposed method, three subproblems are solved and two multipliers are updated. The former two subproblems are first solved in a parallel fashion, and immediately, the multipliers associated to these are updated; then the third subproblem is solved in the sense of an alternative fashion with the former two subproblems, and the multipliers associated to the last two block variables are updated. The proposed method is essentially to a hybrid splitting method since it combines the parallel splitting method and the alternating direction method, which are two power tools for the convex optimization problem with a separable structure. Under suitable assumptions, the global convergence of the hybrid splitting method is proved. The numerical results on the discrete ill-posed problems show that the proposed method has validity and efficiency.
Declarations
Acknowledgements
This work is supported by Natural Science Foundation of China (11571074), Natural Science Foundation of Fujian Province (2015J01010), Science and Technology Project of Hunan Province (2014SK3235), and Scientific Research Fund of Hunan Provincial Education Department (2015277).
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Authors’ Affiliations
(1)
College of Mathematics and Econometrics, Hunan University, Changsha, China
(2)
College of Mathematics and Computational Science, Hunan First Normal University, Changsha, China
(3)
College of Mathematics and Computer Science, Fuzhou University, Fuzhou, China
References
Stephen, B, Lieven, V: Convex Optimization. Cambridge University Press, London (2004)
MATHGoogle Scholar
Achtziger, W, Hoheisel, T, Kanzow, C: A smoothing-regularization approach to mathematical programs with vanishing constraints. Comput. Optim. Appl. 55(3), 733-767 (2013)
View ArticleMathSciNetMATHGoogle Scholar
Lusem, AN, Svaiter, BF: A new smoothing-regularization approach for a maximum-likelihood estimation problem. Appl. Math. Optim. 29(3), 225-241 (1994)
View ArticleMathSciNetGoogle Scholar
Chen, SF, Goodman, J: An empirical study of smoothing techniques for language modeling. Comput. Speech Lang. 13(4), 359-394 (1999)
View ArticleGoogle Scholar
Candes, EJ, Tao, T: Near optimal signal recovery from random projections: universal encoding strategies. IEEE Trans. Inf. Theory 52(12), 5406-5425 (2006)
View ArticleMathSciNetMATHGoogle Scholar
Zhang, Y: Theory of compressive sensing via \(\ell_{1}\)-minimization: a non-RIP analysis and extensions. J. Oper. Res. Soc. China 1(1), 79-105 (2013)
View ArticleMATHGoogle Scholar
Chen, CH, He, BS, Ye, YY, Yuan, XM: The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent. Math. Program. (2014). doi:10.1007/s10107-014-0826-5Google Scholar
Glowinski, R, Le Tallec, P: Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics. SIAM Studies in Applied Mathematics. SIAM, Philadelphia (1989)
View ArticleMATHGoogle Scholar
He, BS, Liao, LZ, Qian, MJ: Alternating projection based prediction-correction methods for structured variational inequalities. J. Comput. Math. 24(6), 693-710 (2006)
MathSciNetMATHGoogle Scholar
He, BS, Xu, MH: A general framework of contraction methods for monotone variational inequalities. Pac. J. Optim. 4(2), 195-212 (2008)
MathSciNetMATHGoogle Scholar
Amir, B, Teboulle, M: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183-202 (2009)
View ArticleMathSciNetMATHGoogle Scholar
Hansen, PC: Regularization tools: a Matlab package for analysis and solution of discrete ill-posed problems. Numer. Algorithms 6(1), 1-35 (1994)
View ArticleMathSciNetMATHGoogle Scholar
