A primal-dual algorithm framework for convex saddle-point optimization

In this study, we introduce a primal-dual prediction-correction algorithm framework for convex optimization problems with known saddle-point structure. Our unified frame adds the proximal term with a positive definite weighting matrix. Moreover, different proximal parameters in the frame can derive some existing well-known algorithms and yield a class of new primal-dual schemes. We prove the convergence of the proposed frame from the perspective of proximal point algorithm-like contraction methods and variational inequalities approach. The convergence rate \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$O(1/t)$\end{document}O(1/t) in the ergodic and nonergodic senses is also given, where t denotes the iteration number.


Introduction
We consider the following model that arises from various signal and image processing applications: where B is a continuous linear operator, and f  and f  are proper convex lower-semicontinuous functions. We can easily write problem () in its primal-dual formulation through Fenchel duality []: where X ∈ R N and V ∈ R M are two finite-dimensional vector spaces, and f *  is the convex conjugate function of f  defined as As analyzed in [, ], the saddle-point problem () can be regarded as the primal-dual formulation, and more and more scholars have proposed some primal-dual algorithms. Zhu and Chan [] proposed the famous primal-dual hybrid gradient (PDHG) algorithm with adaptive stepsize. Though the algorithm is quite fast, the convergence is not proved. He, You, and Yuan [] showed that PDHG with constant step sizes is indeed convergent if one of the functions of the saddle-point problem is strongly convex. Chambolle and Pock [] gave a primal-dual algorithm with convergence rate O(/k) for the complete class of these problems. They further showed accelerations of the proposed algorithm to yield improved rates on problems with some degree of smoothness. In particular, they showed that the algorithm can achieve the O(/k  ) convergence in problems where the primal or the dual objective is uniformly convex, and the method can show linear convergence, that is, O(ς k ) (for some ς ∈ (, )), on smooth problems. Bonettini and Ruggiero [] established the convergence of a general primal-dual method for nonsmooth convex optimization problems and showed that the convergence of the scheme can be considered as an -subgradient method on the primal formulation of the variational problem when the steplength parameters are a priori selected sequences. He and Yuan [] did a novel study on these primal-dual algorithms from the perspective of contraction perspective. Their method simplified the existing convergence analysis. Cai proposed a simple primal-dual method for total-variation image restoration problems and showed that their iterative scheme has the O(/k) convergence rate in the ergodic sense. When we had finished this paper, we found the algorithm proposed in [], where convergence analysis was similar to our proposed frame. However, the algorithm proposed in [] is actually a particular case of our unified framework when the precondition matrix in our frame is fixed.
More specifically, the iterative schemes of existing primal-dual algorithms for the problem () can be unified as the following procedure: where γ , τ >  and θ ∈ R. and give a primal-dual algorithm framework such that it can be well adopted in different imaging applications.
The organization of this paper is as follows. In Section , we propose the primal-dualbased contraction algorithm framework in prediction-correction fashion. In Section , we present convergence analysis. The iteration complexity in the ergodic and nonergodic senses is established in Sections  and . In Section , connections with well-known methods, and some new schemes are discussed. Finally, a conclusion is given.

Proposed frame
Problem () can be reformulated as the following monotone variational inequality (VI): where ∂ denotes the subdifferential operator of a convex function. By denoting the VI () can be written as follows (denoted VI( , F)): Note that the monotonicity of the variational inequality is guaranteed by the convexity of the function ∂f *  and ∂f  . Recall that the primal-dual algorithm for () presented in [] (θ = ) is We can easily verify that the iteration (v k+ , x k+ ) generated by () can be characterized as follows: The convergence of iteration () was proved in [] with the condition on the stepsize γ τ B T B < . Motivated by the idea in [], the scheme () can be considered as a prediction step. So, in the following, we propose a primal-dual-based contraction method for problem (). To present new methods in the prediction-correction fashion, we denote the iterationũ k = (ṽ k ,x k ) generated by the following primal-dual procedure (), where the prediction step can be redescribed as where P is a positive definite matrix to be selected properly in different applications. Then, the new iteration is yielded by correctingũ k via where  < ρ < . Similarly to (), the predictor scheme () can also be written in the VI form as follows: and using the notation in (), we have the following compact form of (): So, we can prove the convergence of the proposed algorithm in the form of proximal point algorithm [, ]. Next, we use this idea to prove that the scheme ()-() converges.

Convergence analysis
In this section, we show the convergence of the proposed frame. Convergence results easily follow from proximal point algorithm-like contraction methods [] and VI approach [].
Lemma  Let B be the given operator, let γ , τ > , and let Q be defined by (). Then Q is positive definite if where p >  is the minimal eigenvalue of P.
Proof For any nonzero vectors s and t, we have where we used the Cauchy-Schwarz inequality. The proof is completed.
In the following, we give an important inequality for the output of the scheme ()-().

Lemma  For iteration sequences {u
Proof Since () holds for any u ∈ , we set u = u * , where u * is an arbitrary solution, and obtain Thus () leads to Note that the mapping F(u) is monotone. We thus have and also Replacing u * -ũ k by (u *u k ) + (u k -ũ k ) in () and using (), we get the assertion.
Lemma  The sequence {u k } generated by the proposed scheme ()-() satisfies Proof Using () and (), by a simple manipulation we obtain The assertion is proved.
The following theorem states that the proposed iterative scheme converges to an optimal primal-dual solution.
Theorem  If Q in () is positive definite, then any sequence generated by the scheme ()-() converges to a solution of the minimax problem ().
Proof From () we know that the norm u ku * Q is nonincreasing. We also can get that u k is bounded and u k -ũ k Q → . Inequality () implies that the sequence {u k } has at least one cluster point. We denote it by u ∞ . Let {u k j } be a subsequence converging to u ∞ . Thus we have Due to the facts () and (), we have Because {u k j } converges to u ∞ , this inequality becomes Thus, the cluster point u ∞ satisfies the optimality condition of (). Note that inequality () is true for all solution points of VI( , F). Hence we have and thus the sequence u k converges to u ∞ . This proof is completed.

Convergence rate in an ergodic sense
In the following, using proximal point algorithm-like contraction methods for convex optimization [], the convergence rate in the ergodic and nonergodic senses is given. First, we prove a lemma, which is the base for the proofs of the convergence rate in the ergodic sense.
Lemma  The sequence {u k } is generated by the proposed scheme ()-(). Then we have Proof Using (), the right-hand side of () can be written as For the right-hand side of (), taking and applying the identity we obtain For the last term of the right-hand side of (), we have Substituting () and () into (), we get Using the property of the mapping F, we have Substituting it into (), the lemma is proved.
Theorem  Let {u k } be the sequence generated by the scheme ()-(), and letũ t be defined byũ Then, for any integer t > , we have thatũ t ∈ and Proof By the convexity of it is clear thatũ t ∈ . Summing () over k = , , . . . , t, we have By the definition ofũ t , the assertion of the theorem directly follows.

Convergence rate in a nonergodic sense
In this section, we show that a worst-case O(/t) convergence rate in a nonergodic sense can also be established for the proposed algorithm frame. We first prove the following lemma.
Lemma  Let the sequence {u k } be generated by the proposed scheme ()-(). Then we have Proof Setting u =ũ k+ in (), we get Note that () is also true for k := k + , and we have Setting u =ũ k in this inequality, we obtain Adding () and () and using the monotonicity of F, we obtain Adding the term to both sides of (), we have Substituting u ku k+ = ρ(u k -ũ k ) into the left-hand side of the inequality, we obtain the lemma.
Next, we are ready to prove the key inequality of this section.

Lemma  Let the sequence {u k } be generated by the proposed scheme ()-(). Then we have
Since inequality () holds, we obtain The assertion directly follows from this inequality.

Theorem  Let {u k } be the sequence generated by the scheme ()-(). Then, for any integer t > , we have
Assertion () immediately follows from () and ().

Connections with existing methods
In this section, we focus on a specific version of problem () , which arises in imaging processing, where f  (x) =   Axb  is quadratic. For discrete total-variation regularization, B is the gradient operator, and A is a possibly large and ill-conditioned matrix representing a linear transform. If A is the identity matrix, then problem () is the well-known Rudin-Osher-Fatemi denoising model []. Because totalvariation regularization can preserve sharp discontinuities in an image for removing noise, the above problem has received a lot of attention by most scholars in image processing, including computerized tomography [] and parallel magnetic resonance imaging [].
In the following, we establish connections of the proposed frame to the well-known methods for solving (). There are other types methods designed to solve problem (). Among them, the split Bregman method proposed by Goldstein and Osher [] is very popular for imaging applications. This method has been proved to be equivalent to the alternating direction of multiplier method. In [], based on proximal forward-backward splitting and Bregman iteration, a split inexact Uzawa (SIU) method is proposed to maximally decouple the iterations, so that each iteration is explicit in this algorithm. Also, the authors gave an algorithm based on Bregman operator splitting (BOS) when A is not diagonalizable. Recently, Tian and Yuan [] proposed a linearized primal-dual method for linear inverse problems with total-variation regularization and showed that this variant yields significant computational benefits. Next, we show that different P in () can induce the following well-known methods: the linearized primal-dual method, SIU, BOS, and split Bregman methods and some new primal-dual algorithms with the correction step ().

Linearized primal-dual method
The linearized primal-dual method in [] can be directly induced by setting P = Iτ A T A and We can also easily show that the positive definiteness of the matrices P and Q in () In this situation, the scheme () can be written as follows: and the scheme () can be expressed as The idea is also similar to that of [, ], which uses the symmetric positive semi-definite matrix instead of the identity matrix in the proximal term. But their methods [, ] do not have overrelaxation or correction step. In [, ], the authors developed firstorder splitting algorithm for solving jointly the primal and dual formulations of large-scale convex minimization problems involving the sum of a smooth function with Lipschitzian gradient, a nonsmooth proximable function, and linear composite functions. Actually, the linearized primal-dual method () and () is a particular case where a nonsmooth proximable function is missing in [, ].
When ρ = , we can see that there is no correction step, that is, (x k+ , v k+ ) = (x k ,ṽ k ). In the following subsection, we focus on the scheme ()-() with different P and Q when ρ = , that is, If P = I, the the CP method is a particular case of () as discussed in []. We also find that different P in () can induce some existing famous algorithms.

Split inexact Uzawa method
For f  (x) =   Axb  , the explicit SIU algorithm can be described as follows: where γ > ,  < τ < / A T A + γ B T B , and So, the scheme () can be expressed as Using the relation prox γ f *  = (I + γ ∂f *  ) - and changing the order of these equations, the scheme () is equivalent to By introducing the variable d k+ = prox  γ f  (Bx k+ + v k γ ), the scheme () can be further expressed as () Substituting () into the first equation of (), the scheme () is equivalent to We can see that the method () is equivalent to the SIU. Obviously, the explicit SIU method is a particular case of the proposed frame with P = Iτ A T A and ρ = . If ρ = , then a linearized primal-dual method is presented in Section .. So, the algorithm in [] can be considered as a relaxed SIU method.

Bregman operator splitting
The BOS algorithm for solving problem () was recently introduced in [] based on the primal dual formulation of the model. It can be described as Using relation (), we arrive at () Now, we can see that the scheme () is the method (). Clearly, the iterative scheme () is a particular case of the frame with P = Iτ A T A + τ γ B T B and ρ = . Also, when ρ = , we can get a new primal-dual method for solving () as follows: In fact, the scheme () can be considered as a relaxed BOS algorithm. If f  (·) = ·  , then we can deduce that (I + ∂f *  ) - (v) = proj(v), where proj is the projection operator. If the image satisfies periodic boundary conditions and if we use total-variation regularization, then the matrix B T B is block circulant; hence, it can be diagonalized by the Fourier transform matrix as noted in []. So, the new algorithm () can be computed efficiently and does not need the inner iteration to solve the subproblem.

Split Bregman
In this subsection, we identify the split Bregman algorithm as a particular case of the proposed algorithm. Firstly, we reformulate model () as an equivalent constrained minimization problem The split Bregman algorithm for solving this constrained problem is as follows: where γ > . The difficulty of implementing the scheme () is mainly due to that the inverse of the matrix is not easy to obtain. Next, we show that our method can induce the scheme (). Let where τ = , γ > . We see that the matrix is not positive. But the scheme () with this Q can induce the famous split Bregman algorithm. In this situation, the scheme () can be expressed as Using the relation v kv k- = v k + γ (Bx kd k ), the scheme () can be further expressed as The method () is the split Bregman scheme (). So, the split Bregman algorithm can be identified as a particular case of our proposed algorithm framework with P = γ B T B and ρ = . If ρ = , then, for P = γ B T B and τ = , a new primal-dual scheme can be described as follows: where α is a positive number, and θ is a number between  and . Using a similar derivation as before, the modified split Bregman algorithm is ⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ x k+ = (A T A + γ θB T B + α(θ )I) - ((θ )(αγ B T B)x k Checking the convergence condition of the theorem, if α γ > B   , α > , and θ ∈ [, ), we can easily get that the sequence x k generated from () converges to a solution of problem (). We remark that when θ = , the scheme () reduces to the split Bregman method (). When θ = , the scheme () is the preconditioned alternating method of multipliers as discussed in [, ]. Also, when Q is defined by () and ρ = , the new primal-dual method can be expressed as ()