A semi-smoothing augmented Lagrange multiplier algorithm for low-rank Toeplitz matrix completion

Abstract

The smoothing augmented Lagrange multiplier (SALM) algorithm is a generalization of the augmented Lagrange multiplier algorithm for completing a Toeplitz matrix, which saves computational cost of the singular value decomposition (SVD) and approximates well the solution. However, the communication of numerous data is computationally demanding at each iteration step. In this paper, we propose an accelerated scheme to the SALM algorithm for the Toeplitz matrix completion (TMC), which will reduce the extra load coming from data communication under reasonable smoothing. It has resulted in a semi-smoothing augmented Lagrange multiplier (SSALM) algorithm. Meanwhile, we demonstrate the convergence theory of the new algorithm. Finally, numerical experiments show that the new algorithm is more effective/economic than the original algorithm.

1 Introduction

Completing a low-rank matrix from a subset of its entries has been a hot problem recently, first introduced by [8], that has arisen in a wide variety of practical contexts across all disciplines of engineering and computational science such as model reduction [19], machine learning [1, 2], control [22], pattern recognition [12], imaging inpainting [3], video denoising [16], computer vision [28], and so on. Despite matrix completion (MC) requiring the global solution of a non-convex objective, there are many computational efficient algorithms which are effective for a broad class of matrices. The problem has received intensive research from both theoretical and algorithmic aspects, see, e.g., [4,5,6,7,8,9,10,11, 13,14,15,16,17,18, 21, 23, 26, 27, 29,30,31,32, 34, 35], and the references therein for partial review. It is well known that the mathematical model of the MC problem is of the following form:

$$ \begin{aligned} &\min_{A\in \mathbb{R}^{m\times n}} \Vert A \Vert _{\ast }, \\ &\text{subject to } \mathcal{P}_{\varOmega }(A)=\mathcal{P}_{\varOmega }(M), \end{aligned} $$

(1.1)

where the matrix $M\in \mathbb{R}^{m\times n}$ is an underlying matrix to be completed, Ω is a random subset of indices for the available entries, and $\mathcal{P}_{\varOmega }$ is the associated sampling orthogonal projection operator which acquires only the entries indexed by $\varOmega \subset \{1,2,\ldots ,m\}\times \{1,2,\ldots ,n\}$.

In the current MC problems, the matrix M is of special structure in general. Therefore, much attention has been paid to the completion of Toeplitz and Hankel matrices in recent years [20, 24, 25, 30, 32]. Many scholars have conducted in-depth research on the special structure, property, and application of the Toeplitz and Hankel matrices; for example, nuclear norm minimization for the low-rank Hankel matrix reconstruction problem under the random Gaussian sampling model is investigated in [7]. In addition, Hankel matrix reconstitution in the sense of minimizing the nuclear norm with non-uniform sampling of entries is researched in [11]. To make full use of the special structure of a Toeplitz matrix, a mean value algorithm is presented in [30]; the modified Lagrange multiplier (MALM) algorithm [31] and the smoothing augmented Lagrange multiplier (SALM) algorithm [34] are also proposed. Therefore, the Toeplitz matrix completion (TMC) is one of the most important MC problems and has attracted a large amount of attention recently. As is well known, an $n\times n$ Toeplitz matrix is of the following form:

$$T={(t_{j-i})}_{i,j=1}^n=\left(\begin{array}{ccccc}{t}_0& t_1& \cdots & t_{n-2}& t_{n-1}\\ t_{-1}& t_0& \cdots & t_{n-3}& t_{n-2}\\ ⋮& ⋮& \ddots & ⋮& ⋮\\ t_{-n+2}& t_{-n+3}& \cdots & t_0& t_1\\ t_{-n+1}& t_{-n+2}& \cdots & t_{-1}& t_0\end{array}\right)\in {\mathbb{T}}^{n\times n}\subset {\mathbb{R}}^{n\times n},$$

(1.2)

which is determined by $2n-1$ entries, say, the first row and the first column. Explicitly seeking the lowest rank Toeplitz matrix consistent with the known entries is mathematically considered as

$$ \begin{aligned}& \min_{A\in \mathbb{T}^{n\times n}} \Vert A \Vert _{\ast }, \\ &\text{subject to } H\circ A=M, \end{aligned}$$

(1.3)

where “∘” is the Hadamard product, $H=(H_{ij})\in \mathbb{R} ^{n\times n}$ is the weighted matrix with entries $H_{ij}=1$ for $j-i\in \varOmega \subset \{-n+1,\ldots ,n-1\}$ and $H_{ij}=0$ for any other $(i,j)$, $M=(M_{ij})\in \mathbb{T}^{n\times n}$ is the underlying Toeplitz matrix to be completed, namely $M_{ij}=0$ for $j-i\notin \varOmega$.

The SALM algorithm switches the iteration matrix into the Toeplitz structure at each iteration step by the smoothing operator, which saves computational cost of the singular value decomposition and approximates well the solution. Unfortunately, numerous data have to be shifted at each iteration step in the process of implementing this algorithm. However, there is a cost, sometimes relatively great, associated with the moving of data. The control of memory traffic is crucial to performance in many computers.

These factors motivated us to reduce the traffic jam of data, resulting in a semi-smoothing augmented Lagrange multiplier (SSALM) algorithm based on the selecting technique of the optimal parameter $\omega ^{(k)}$ at each of the five iteration steps in [33]. Compared with the SALM algorithm, the new algorithm either saves computation cost or reduces data communication. Two aspects are taken into account, which results in more practical or economic implementation. The new algorithm not only overcomes the slowness of the SVD for the original ALM algorithm, but also reduces the greatness of the data communication for the ALM algorithm. We can see that the CPU of SSALM algorithm is reduced to 30.44% from the numerical experiments.

The rest of this paper is organized as follows. Some preliminaries are provided in Sect. 2. Section 3 presents the semi-smoothing augmented Lagrange multiplier (SSALM) algorithm after giving an outline of the ALM algorithm, the dual approach, and the SALM algorithm. The convergence property of the SSALM algorithm is constructed in Sect. 4. We report the numerical results to indicate the effectiveness of the SSALM algorithm in Sect. 5. Finally, we end the paper with the concluding remarks in Sect. 6.

2 Preliminaries

This section is devoted to some of the necessary notations and preliminaries. $\mathbb{R}^{m\times n}$ denotes the set of $m\times n$ real matrices, $\mathbb{T}^{n\times n}$ is the set of $n\times n$ real Toeplitz matrices. The nuclear norm of a matrix A is denoted by $\|A\|_{\ast }$, and the Frobenius norm $\|A\|_{F}$ is the maximum absolute value of the matrix entries of a matrix A. $A^{T}$ is used to express the transpose of a matrix $A\in \mathbb{R}^{n\times n}$, $\operatorname{rank}(A)$ is equal to the rank of a matrix A, and $\operatorname{tr}(A)$ represents the trace of A. The standard inner product between two matrices is denoted by $\langle X,Y\rangle = \operatorname{tr}(X^{T}Y)$. For $A=(a_{ij}) \in \mathbb{R}^{m\times n}$, $B=(b_{ij})\in \mathbb{R}^{m\times n}$, their Hadamard product $A\circ B$ is an $m\times n$ matrix whose $(i,j)$ entry is the $a_{ij}b_{ij}$, i.e., $A\circ B=(a_{ij}b_{ij})\in \mathbb{R} ^{m\times n}$.

The singular value decomposition (SVD) of a matrix $A\in \mathbb{R} ^{m\times n}$ of r-rank is defined by

$$ A=U\varSigma _{r} V^{T}, \qquad \varSigma _{r}= \operatorname{diag}(\sigma _{1}, \sigma _{2}, \ldots , \sigma _{r}), $$

where $U\in \mathbb{R}^{m\times r}$ and $V \in \mathbb{R}^{n\times r}$ are column orthonormal matrices, and $\sigma _{1} \geq \sigma _{2} \geq \cdots \geq \sigma _{r}>0$.

Definition 2.1

(Singular value thresholding operator [6])

For each $\tau \geq 0$, the singular value thresholding operator $\mathcal{D}_{\tau }$ is defined as follows:

$$ \mathcal{D}_{\tau }(A):=U\mathcal{D}_{\tau }(\varSigma )V^{T},\qquad \mathcal{D}_{\tau }(\varSigma )={\operatorname{diag}}\bigl(\{ \sigma _{i}-\tau \}_{+}\bigr), $$

where

$$\begin{aligned}& A=U\varSigma _{r} V^{T}\in \mathbb{R}^{m\times n}, \qquad \{\sigma _{i}-\tau \}_{+}=\textstyle\begin{cases} \sigma _{i}-\tau , & \text{if } \sigma _{i}>\tau, \\ 0, & \text{if } \sigma _{i}\leq \tau. \end{cases}\displaystyle \end{aligned}$$

$I_{n}=(e_{1},e_{2},\ldots ,e_{n})\in \mathbb{R}^{n\times n}$ denotes the $n\times n$ identity matrix and $Z_{n}=(e_{2},e_{3},\ldots ,e_{n},0) \in \mathbb{R}^{n\times n}$ is called the shift matrix. It is clear that

$$Z_n^r=\{\begin{array}{cc}(\begin{array}{cc}O& O\\ I_{n-r}& O\end{array}),\hfill & 1<r<n,\hfill \\ O,\hfill & r\ge n,\hfill \end{array}$$

where “O” stands for a zero-matrix. Thus, a Toeplitz matrix $T \in \mathbb{T} ^{n\times n}$, shown in (1.2), can be written as a linear combination of these shift matrices, that is,

$$ T=\sum_{l=1}^{n-1}t_{-l}Z_{n}^{l}+ \sum_{l=0}^{n-1}t_{l}\bigl(Z _{n}^{T}\bigr)^{l}. $$

$\varOmega \subset \{-n+1,\ldots ,n-1\}$ is an indices set of observed diagonals of a Toeplitz matrix $M\in \mathbb{T}^{n\times n}$, Ω̄ is the complementary set of Ω. For any Toeplitz matrix $A\in \mathbb{T}^{n\times n}$, the vector vec$(A,\alpha )$ denotes the αth diagonal of A, $\alpha =-n+1, -n+2,\ldots ,n-1$, that is to say,

$$\operatorname{vec}(H\circ A,\alpha )=\textstyle\begin{cases} \operatorname{vec}(A,\alpha ) ,& \alpha \in \varOmega, \\ \mathbf{0} ,& \alpha \notin \varOmega, \end{cases}\displaystyle \quad (\mathbf{0}\mbox{ is a zero-vector}). $$

Definition 2.2

(Toeplitz structure smoothing operator [34])

For any matrix $A=(a_{ij})\in \mathbb{R}^{n\times n}$, the Toeplitz structure smoothing operator $\mathcal{T}$ is defined as follows:

$$ \mathcal{T}(A):=\sum_{l=1}^{n-1} \tilde{a}_{-l}Z_{n}^{l}+ \sum _{l=0}^{n-1}\tilde{a}_{l} \bigl(Z_{n}^{T}\bigr)^{l}, $$

(2.1)

where $\tilde{a}_{\alpha }=\frac{_{\alpha }A^{\min }+_{\alpha }A^{ \max }}{2}$, $\alpha =-n+1,-n+2,\ldots , n-1$ with

$$ _{\alpha }A^{\min }=\min_{i,j\in \{1,2,\ldots ,n\}} \{a_{ij},i-j= \alpha \}, \quad \text{and}\quad {} _{\alpha }A^{\max }= \max_{i,j\in \{1,2,\ldots ,n\}}\{a_{ij},i-j=\alpha \}. $$

It is clear that $\mathcal{T}(A)$ is a Toeplitz matrix derived from the matrix A. Namely, any $A\in \mathbb{R}^{n\times n}$ can be changed into a Toeplitz structure via the smoothing operator $\mathcal{T}( \cdot )$.

3 Algorithms

First of all in this section, for completeness as well as for the purpose of comparison, we briefly review and summarize some relative algorithms for approximately minimizing the nuclear norm of a matrix under convex constraints.

Since the matrix completion problem is closely connected to the robust principal component analysis (RPCA) problem, then it can be formulated in the same way as RPCA, an equivalent problem of (1.1) can be considered as follows.

As E will compensate for the unknown entries of M, the unknown entries of M are simply set as zeros. Suppose that the given data are arranged as the columns of a large matrix $M\in \mathbb{R}^{m\times n}$. The mathematical model for estimating the low-dimensional subspace is to find a low-rank matrix $A\in \mathbb{R}^{m\times n}$ such that the discrepancy between A and M is minimized, leading to the following constrained optimization:

$$ \begin{aligned} &\min_{A,E\in \mathbb{R}^{m\times n}} \Vert A \Vert _{\ast }, \\ &\text{subject to } A+E=M, \mathcal{P}_{\varOmega }(E)=0, \end{aligned} $$

(3.1)

where E will compensate for the unknown entries of M, the unknown entries of $M\in \mathbb{R}^{m\times n}$ are simply set as zeros. And $\mathcal{P}_{\varOmega }: \mathbb{R}^{m\times n}\rightarrow \mathbb{R} ^{m\times n}$ is a linear operator that keeps the entries in Ω unchanged and sets those outside Ω (say, in Ω̅) zeros.

3.1 The dual algorithm

The dual algorithm proposed in [10] tackles problem (3.1) via its dual. That is, one first solves the dual problem

$$ \begin{aligned} &\max_{Y} \langle M,Y \rangle , \\ &\text{subject to } J(Y)\leq 1, \end{aligned} $$

(3.2)

for the optimal Lagrange multiplier Y, where

$$ J(Y)=\max \bigl( \Vert Y \Vert _{2},\lambda ^{-1} \Vert Y \Vert _{\infty }\bigr). $$

(3.3)

A steepest ascend algorithm constrained on the surface $\{Y|J(Y)=1\}$ can be adopted to solve (3.2), where the constrained steepest ascend direction is obtained by projecting M onto the tangent cone of the convex body $\{Y|J(Y)\leq 1\}$. It turns out that the optimal solution to the primal problem (3.1) can be obtained during the process of finding the constrained steepest ascend direction.

3.2 The augmented Lagrange multiplier algorithm

The augmented Lagrange multiplier (ALM) algorithm was proposed in [18] for solving a convex optimization (1.1). It should be described subsequently.

It is famous that the partial augmented Lagrangian function of problem (3.1) is

$$ \mathcal{L}(A,E,Y,\mu )= \Vert A \Vert _{\ast }+\langle Y,M-A-E \rangle +\frac{ \mu }{2} \Vert M-A-E \Vert _{F}^{2}. $$

(3.4)

Hence, the augmented Lagrange multiplier algorithm is designed as follows.

Algorithm 3.1

([18])

Given a sampled set Ω, a sampled matrix $D=\mathcal{P}_{\varOmega }(M)$, $\mu _{0}>0$, $\rho >1$. Given also two initial matrices $Y_{0}=0$, $E_{0}=0$. $k:=0$.

• 1. Compute the SVD of the matrix $(D-E_{k}+\mu _{k}^{-1}Y_{k})$:

  $$ [U_{k},\varSigma _{k},V_{k}]=\operatorname{svd}\bigl(D-E_{k}+\mu _{k}^{-1}Y_{k}\bigr); $$

• 2. Set

  $$ A_{k+1}=U_{k}\mathcal{D}_{\mu _{k}^{-1}}(\varSigma _{k})V_{k}^{T}, $$

  solve $E_{k+1}=\arg \min_{\mathcal{P}_{\varOmega }(E)=0} \mathcal{L}(A_{k+1},E,Y_{k},\mu _{k})$,

  $$ E_{k+1}=\mathcal{P}_{\bar{\varOmega }}\bigl(D-A_{k+1}+\mu _{k}^{-1}Y_{k}\bigr); $$

• 3. If $\|D-A_{k+1}-E_{k+1}\|_{F}/\|D\|_{F}<\epsilon _{1}$ and $\mu _{k}\|E_{k+1}-E_{k}\|_{F}/\|D\|_{F}<\epsilon _{2}$, stop; otherwise, go to the next step;

• 4. Set $Y_{k+1}=Y_{k}+\mu _{k}(D-A_{k+1}-E_{k+1})$.

If $\mu _{k}\|E_{k+1}-E_{k}\|_{F}/\|D\|_{F}<\epsilon _{2}$, set $\mu _{k+1}=\rho \mu _{k}$; otherwise, go to Step 1.

Note that the ALM algorithm performs better both in theory and practice than other algorithms with a Q-linear convergence speed globally. It is of much better numerical behavior or higher accuracy. It is reported that the ALM algorithm has been applied to MC problem (1.1). The algorithm, however, has the disadvantage of the penalty function, namely the matrix sequences $\{A_{k}\}$ generated by the ALM algorithm are the accepted solutions but feasible ones.

3.3 The smoothing augmented Lagrange multiplier algorithm

In this subsection, we make mention of a stepped-up scheme for the TMC problem. The smoothing augmented Lagrange multiplier (SALM) approach employs the smoothing operator $\mathcal{T}$ (see (2.1)) to approximate a matrix generated in the kth iteration so that the current approximation is of a Toeplitz structure.

Then our problem can be expressed as the following convex programming:

$$ \begin{aligned} &\min_{A,E\in \mathbb{T}^{n\times n}} \Vert A \Vert _{\ast }, \\ & \text{subject to}\quad A+E=\mathcal{P}_{\varOmega }(M), \qquad \mathcal{P}_{\varOmega }(E)=0, \end{aligned} $$

(3.5)

where $M\in \mathbb{T}^{n\times n}$ is a real Toeplitz matrix, and $\varOmega \subset \{-n+1,\ldots ,n-1\}$.

Let $D=\mathcal{P}_{\varOmega }(M)$. Then the partial augmented Lagrangian function is

$$ \mathcal{L}(A,E,Y,\mu )= \Vert A \Vert _{\ast }+ \langle Y,D-A-E\rangle +\frac{ \mu }{2} \Vert D-A-E \Vert _{F}^{2}, $$

(3.6)

where $Y\in \mathbb{R}^{n\times n}$.

Algorithm 3.2

([34])

Given a sampled set Ω, a sampled matrix D, $\mu _{0}>0$, $\rho >1$. Given also two initial matrices $Y_{0}=0$, $E_{0}=0$. $k:=0$.

• 1. Compute the SVD of the matrix $(D-E_{k}+\mu _{k}^{-1}Y_{k})$ using the Lanczos method

  $$ [U_{k},\varSigma _{k},V_{k}]=\operatorname{lansvd}\bigl(D-E_{k}+\mu _{k}^{-1}Y_{k}\bigr); $$

• 2. Set

  $$ X_{k+1}=U_{k}\mathcal{D}_{\mu _{k}^{-1}}(\varSigma _{k})V_{k}^{T}, $$

  compute for smoothing $\tilde{a}_{\alpha }=\frac{_{\alpha }X_{k+1} ^{\min }+_{\alpha }X_{k+1}^{\max }}{2}$, $\alpha \in \{-n+1,-n+2, \ldots ,n-1\}$, and

  $$\begin{aligned}& A_{k+1}= \mathcal{T}(X_{k+1})=\sum _{l=1}^{n-1}\tilde{a}_{-l}Z _{n}^{l}+\sum_{l=0}^{n-1} \tilde{a}_{l}\bigl(Z_{n}^{T}\bigr)^{l}, \\& E_{k+1}=\mathcal{P}_{\bar{\varOmega }}\bigl(D-A_{k+1}+\mu _{k}^{-1}Y_{k}\bigr); \end{aligned}$$

• 3. If $\|D-A_{k+1}-E_{k+1}\|_{F}/\|D\|_{F}<\epsilon _{1}$ and $\mu _{k}\|E_{k+1}-E_{k}\|_{F}/\|D\|_{F}<\epsilon _{2}$, stop; otherwise, go to the next step;

• 4. Set $Y_{k+1}=Y_{k}+\mu _{k}(D-A_{k+1}-E_{k+1})$.

If $\mu _{k}\|E_{k+1}-E_{k}\|_{F}/\|D\|_{F}<\epsilon _{2}$, set $\mu _{k+1}=\rho \mu _{k}$; otherwise, go to Step 1.

It is reported that the convergence speed of the SALM algorithm is greater than that of the ALM and APG algorithms. A merit of smoothing is that the fast SVD procedure can be utilized to reduce the computation.

As we know, the SVD time is saved at the expense of data communication. Sometimes, this is not worth the candle. This motivated us to come up with the following algorithm.

3.4 The semi-smoothing augmented Lagrange multiplier algorithm

In this subsection, we propose a semi-smoothing augmented Lagrange multiplier algorithm based on the ALM and SALM algorithms for the TMC problem. The new algorithm consists of two stages: one is $\ell -1$ iterations by the ALM scheme, which is free moving of data; another is the ℓth smoothing by the SALM procedure, which is keeping the iteration matrix as a Toeplitz structure.

Now, the semi-smoothing augmented Lagrange multiplier (SSALM) algorithm will be presented in the following.

Algorithm 3.3

(SSALM algorithm)

Input: A sampled set Ω, a sampled matrix D, $Y_{0,0}=0$, $E_{0,0}=0$; parameters $\mu _{0}>0$, $\rho >1$, $\ell >1$, $\epsilon _{1}$, $\epsilon _{2}$.

Let $k:=0$, $q:=1$, $q=1,2,\ldots ,\ell -1$.

Repeat:

• 1. $\ell -1$ iterations.

  (1) :

  Compute the SVD of the matrix $(D-E_{k,q}+\mu _{k,q} ^{-1}Y_{k,q})$:

  $$ [U_{k,q},\varSigma _{k,q},V_{k,q}]=\operatorname{lansvd}\bigl(D-E_{k,q}+\mu _{k,q} ^{-1}Y_{k,q} \bigr); $$

  (2) :

  Set

  $$\begin{aligned}& X_{k+1,q+1}=U_{k,q}\mathcal{D}_{\mu _{k,q}^{-1}}(\varSigma _{k,q})V_{k,q} ^{T}, \\& E_{k+1,q+1}=\mathcal{P}_{\bar{\varOmega }}\bigl(D-X_{k+1,q+1}+\mu _{k,q}^{-1}Y _{k,q}\bigr); \end{aligned}$$

  (3) :

  If $\|D-X_{k+1,q+1}-E_{k+1,q+1}\|_{F}/\|D\|_{F}<\epsilon _{1}$ and $\mu _{k,q}\|E_{k+1,q+1}-E_{k,q}\|_{F}/\|D\|_{F}<\epsilon _{2}$, stop; otherwise, go to the next step;

  (4) :

  Set $Y_{k+1,q+1}=Y_{k,q}+\mu _{k,q}(D-X_{k+1,q+1}-E_{k+1,q+1})$, $\mu _{k+1,q+1}=\rho \mu _{k,q}$; otherwise, go to step (1);

• 2. ℓth smoothing.

  (1) :

  Compute the SVD of the matrix $(D-E_{k,\ell }+\mu _{k, \ell }^{-1}Y_{k,\ell })$:

  $$ [U_{k,\ell },\varSigma _{k,\ell },V_{k,\ell }]=\operatorname{svd}\bigl(D-E_{k, \ell }+\mu _{k,\ell }^{-1}Y_{k,\ell }\bigr); $$

  (2) :

  Set

  $$ X_{k+1,\ell }=U_{k,\ell }\mathcal{D}_{\mu _{k,\ell }^{-1}}(\varSigma _{k, \ell })V_{k,\ell }^{T}, $$

  compute the factors of smoothing $\tilde{a}_{\alpha }=\frac{_{\alpha }X_{k+1,\ell }^{\min }+_{\alpha }X_{k+1,\ell }^{\max }}{2}$, $\alpha =-n+1,-n+2,\ldots , n-1$,

  and set

  $$ A_{k+1,\ell }= \mathcal{T}(X_{k+1,\ell })=\sum _{l=1}^{n-1} \tilde{a}_{-l}Z_{n}^{l}+ \sum_{l=0}^{n-1}\tilde{a}_{l} \bigl(Z_{n} ^{T}\bigr)^{l}. $$

  Update

  $$ E_{k+1,\ell }=\mathcal{P}_{\bar{\varOmega }}\bigl(D-A_{k+1,\ell }+\mu _{k, \ell }^{-1}Y_{k+1,\ell }\bigr); $$

• 3. If $\|D-A_{k+1,\ell }-E_{k+1,\ell }\|_{F}/\|D\|_{F}<\epsilon _{1}$ and $\mu _{k,\ell }\|E_{k+1,\ell }-E_{k,\ell }\|_{F}/\|D\|_{F}< \epsilon _{2}$, stop; otherwise, go to the next step;

• 4. Set $Y_{k+1,q+1}=Y_{k,q}+\mu _{k,q}(D-A_{k+1,q+1}-E_{k+1,q+1})$.

If $\mu _{k,q}\|E_{k+1,q+1}-E_{k,q}\|_{F}/\|D\|_{F}< \epsilon _{2}$, set $\mu _{k+1,q+1}=\rho \mu _{k,q}$; otherwise, go to Step 1.

In fact, Algorithm 3.3 includes the above Algorithm 3.2 as a special case if $\ell =1$. Obviously, the algorithm is an acceleration of the SALM algorithm in [34].

4 Convergence analysis

This section will analyze the convergence of Algorithm 3.3.

Let $(\ddot{A},\ddot{E})$ be the solution of model (3.5) and Ÿ be that of the optimal problem (3.2). We provide some lemmas in the following firstly.

Lemma 4.1

([8])

Let $A\in \mathbb{R}^{m\times n}$ be an arbitrary matrix and $U\varSigma V^{T}$ be its SVD. Then the set of subgradients of the nuclear norm of A is given by

$$ \partial \Vert A \Vert _{*}=\bigl\{ UV^{T}+W:W\in \mathbb{R}^{m\times n}, U^{T}W=0,WV=0, \Vert W \Vert _{2}\leq 1\bigr\} . $$

Lemma 4.2

([18])

If $\mu _{k}$ is nondecreasing, then each entry of the following series is nonnegative and their sum is finite, i.e.,

$$\begin{aligned}& \sum_{k=1}^{+\infty }{\mu _{k}}^{-1} \bigl(\langle Y_{k+1}-Y_{k},E _{k+1}-E_{k} \rangle +\langle A_{k+1}-\ddot{A},\hat{Y}_{k+1}-\ddot{Y} \rangle +\langle E_{k+1}-\ddot{E},Y_{k+1}-\ddot{Y}\rangle \bigr) \\& \quad < +\infty . \end{aligned}$$

(4.1)

Lemma 4.3

([18])

The sequences $\{\ddot{Y}_{k}\}$, $\{Y_{k}\}$, and $\{\hat{Y}_{k}\}$ are all bounded, where $\hat{Y} _{k}=Y_{k+1}+\mu _{k-1}(D-A_{k}-E_{k-1})$.

Lemma 4.4

The sequence $\{Y_{k,q}\}$ generated by Algorithm 3.3 is bounded.

Proof

Let $B=\mu _{k,q}(D-E_{k,q}+\mu _{k,q}^{-1}Y_{k,q}-X _{k+1,q+1})$, $\mathcal{T}(B)=\sum_{l=1}^{n-1}\tilde{b}_{-l}Z _{n}^{l}+\sum_{l=0}^{n-1}\tilde{b}_{l}(Z_{n}^{T})^{l}$, defined as (2.1).

First of all, we indicate that $Y_{k,q}$, $E_{k,q}$, $k=1,2,\ldots$ , $q=1,2, \ldots ,\ell -1$, are all the Toeplitz matrices. Clearly, $Y_{0,0}=0$, $E _{0,0}=0$ are both smoothed into a Toeplitz structure. Suppose that $Y_{k,q}$, $E_{k,q}$ are both the Toeplitz matrices, so is $E_{k+1,q+1}= \mathcal{P}_{\bar{\varOmega }}(D-X_{k+1,q+1}+\mu _{k,q}^{-1}Y_{k,q})$. Thus, $Y_{k+1,q+1}$ is a Toeplitz matrix also because of Step 4 in Algorithm 3.3.

$$\begin{aligned} Y_{k+1,q+1} =& Y_{k,q}+\mu _{k,q}(D-A_{k+1,q+1}-E_{k+1,q+1}) \\ =& Y_{k,q}+ \mu _{k,q}(D-A_{k+1,q+1}-E_{k,q})+ \mu _{k,q}(E_{k,q}-E_{k+1,q+1}). \end{aligned}$$

Moreover,

$$\begin{aligned} \mu _{k,q}(E_{k,q}-E_{k+1,q+1}) =& \mu _{k,q} \mathcal{P}_{\bar{\varOmega }}\bigl(E_{k,q}-\bigl(D-A_{k+1,q+1}+\mu _{k,q}^{-1}Y_{k,q}\bigr)\bigr) \\ =& \mu _{k,q}\mathcal{P}_{\bar{\varOmega }}\bigl(E_{k,q}-\bigl(D- \mathcal{T}(X_{k+1,q+1})+ \mu _{k,q}^{-1}Y_{k,q} \bigr)\bigr) \\ =& \mu _{k,q}\mathcal{P}_{\bar{\varOmega }}\bigl(E_{k,q}-\bigl(D- \mathcal{T}\bigl(D-E _{k,q}+\mu _{k,q}^{-1}Y_{k,q}- \mu _{k,q}^{-1}B\bigr)+\mu _{k,q}^{-1}Y_{k,q} \bigr)\bigr) \\ =& \mu _{k,q}\mathcal{P}_{\bar{\varOmega }}\bigl(E_{k,q}-\bigl(D- \bigl(D-E_{k,q}+\mu _{k,q}^{-1}Y_{k,q}-\mu _{k,q}^{-1}\mathcal{T}(B)\bigr)+\mu _{k,q}^{-1}Y_{k,q} \bigr)\bigr) \\ =& \mathcal{P}_{\bar{\varOmega }}\mathcal{T}(B). \end{aligned}$$

It is clear that by Steps 1–2 in Algorithm 3.3 we have

$$ D-E_{k,q}+\mu _{k,q}^{-1}Y_{k,q}= \check{U}_{k,q}\check{\varSigma }_{k,q} \check{V}_{k,q}^{T}+ \tilde{U}_{k,q}\tilde{\varSigma }_{k,q}\tilde{V}_{k,q} ^{T}, $$

where $\check{U}_{k,q}$, $\check{V}_{k,q}$ are the singular vectors associated with singular values that are more than $\frac{1}{\mu _{k,q}}$ and $\tilde{U}_{k,q}$, $\tilde{V}_{k,q}$ are those associated with singular values that are less than or equal to $\frac{1}{ \mu _{k,q}}$, the elements of the diagonal matrix $\check{\varSigma } _{k,q}$ are more than $\frac{1}{\mu _{k,q}}$, and those of the diagonal matrix $\tilde{\varSigma }_{k,q}$ are less than or equal to $\frac{1}{ \mu _{k,q}}$. Hence, it is drawn that $A_{k+1,q+1}=\check{U}_{k,q}( \check{\varSigma }_{k,q}-\mu _{k,q}^{-1}I)\check{V}_{k,q}^{T}$, and

$$\begin{aligned} \Vert B \Vert _{F} =& \bigl\Vert \mu _{k,q} \bigl(D-E_{k,q}+\mu _{k,q}^{-1}Y_{k,q}-A_{k+1,q+1} \bigr) \bigr\Vert _{F} \\ =& \bigl\Vert \mu _{k,q}\bigl(\mu _{k,q}^{-1} \check{U}_{k,q}\check{V}_{k,q}^{T}+ \tilde{U}_{k,q}\tilde{\varSigma }_{k,q}\tilde{V}_{k,q}^{T} \bigr) \bigr\Vert _{F} \\ =& \bigl\Vert \check{U}_{k,q}\check{V}_{k,q}^{T}+ \mu _{k,q}\tilde{U}_{k,q} \tilde{\varSigma }_{k,q} \tilde{V}_{k,q}^{T} \bigr\Vert _{F} \\ \leq & \sqrt{n}. \end{aligned}$$

We can obtain hence that $Y_{k,q}+\mu _{k,q}(D-A_{k+1,q+1}-E_{k,q}) \in \partial \|A_{k+1,q+1}\|_{*}$ from Lemmas 4.2 and 4.3.

It is known that, for $A_{k+1,q+1}=U\varSigma V^{T}$, by Lemma 4.1,

$$ \partial \Vert A_{k+1,q+1} \Vert _{*}=\bigl\{ UV^{T}+W:W\in \mathbb{R}^{n\times n},U ^{T}W=0,WV=0, \Vert W \Vert _{2}\leq 1\bigr\} . $$

We also have

$$ \bigl\Vert UV^{T}+W \bigr\Vert _{F}^{2}=\operatorname{tr}\bigl(\bigl(UV^{T}+W\bigr)^{T}\bigl(UV^{T}+W\bigr) \bigr)= \operatorname{tr}\bigl(VV^{T}+W^{T}W\bigr)\leq n. $$

Therefore, the following inequalities can be obtained:

$$ \bigl\Vert Y_{k,q}+\mu _{k,q}(D-A_{k+1,q+1}-E_{k,q}) \bigr\Vert _{F}\leq \sqrt{n} $$

and

$$ \bigl\Vert \mathcal{P}_{\bar{\varOmega }}\bigl(\mathcal{T}(B)\bigr) \bigr\Vert _{F}\leq \big\| \mathcal{T}(B))\big\| _{F} \leq \Vert B \Vert _{F}\leq \sqrt{n}. $$

It is evident that the sequence $\{Y_{k,q}\}$ is bounded. □

Theorem 4.1

Suppose that $\langle A_{k+1,q+1}-A_{k,q},D-A _{k+1,q+1}-E_{k,q}\rangle \geq 0$, then the sequence $\{A_{k,q}\}$ converges to the solution of (3.5) when $\mu _{k,q}\rightarrow \infty$ and $\sum_{k=1}^{+\infty }\mu _{k,q}^{-1}=+\infty$.

Proof

It is true that

$$ \lim_{k\rightarrow \infty }(D-A_{k+1}-E_{k+1})=0 $$

since $\mu _{k,q}^{-1}(Y_{k+1,q+1}-Y_{k,q})=D-A_{k+1,q+1}-E_{k+1,q+1}$ and Lemma 4.4.

Let $(\ddot{A},\ddot{E})$ be the solution of (3.5). Then $A _{k+1,q+1},Y_{k+1,q+1},E_{k+1,q+1}$, $k=1,2,\ldots$ , are all Toeplitz matrices since $\ddot{A}+\ddot{E}=D$. We prove first that

$$\begin{aligned}& \Vert E_{k+1,q+1}-\ddot{E} \Vert _{F}^{2}+ \mu _{k,q}^{-2} \Vert Y_{k+1,q+1}-\ddot{Y} \Vert _{F}^{2} \\& \quad = \Vert E_{k,q}-\ddot{E} \Vert _{F}^{2}- \Vert E_{k+1,q+1}-E_{k,q} \Vert _{F}^{2}+ \mu _{k,q}^{-2} \Vert Y_{k,q}-\ddot{Y} \Vert _{F}^{2} \\& \qquad {}- \mu _{k,q}^{-2} \Vert Y_{k+1,q+1}-Y_{k,q} \Vert _{F}^{2}-2\mu _{k,q}^{-1}\langle A_{k+1,q+1}-\ddot{A},\hat{Y}_{k+1,q+1}-\ddot{Y}\rangle , \end{aligned}$$

(4.2)

where $\hat{Y}_{k+1,q+1}=Y_{k,q}+\mu _{k,q}(D-A_{k+1,q+1}-E_{k,q})$, Ÿ is the optimal solution to the dual problem (3.2).

$$\begin{aligned} \Vert E_{k,q}-\ddot{E} \Vert _{F}^{2} =& \bigl\Vert \mathcal{P}_{\bar{\varOmega }}(E_{k,q}- \ddot{E}) \bigr\Vert _{F}^{2} \\ =& \bigl\Vert \mathcal{P}_{\bar{\varOmega }}(E_{k+1,q+1}- \ddot{E}-E_{k+1,q+1}+E _{k,q}) \bigr\Vert _{F}^{2} \\ =& \bigl\Vert \mathcal{P}_{\bar{\varOmega }}(E_{k+1,q+1}-\ddot{E}) \bigr\Vert _{F}^{2}+ \bigl\Vert \mathcal{P}_{\bar{\varOmega }}(E_{k+1,q+1}-E_{k,q}) \bigr\Vert _{F}^{2} \\ & {}-2\bigl\langle \mathcal{P}_{\bar{\varOmega }}(E_{k+1,q+1}-\ddot{E}), \mathcal{P}_{\bar{\varOmega }}(E_{k+1,q+1}-E_{k,q})\bigr\rangle \\ =& \Vert E_{k+1,q+1}-\ddot{E} \Vert _{F}^{2}+ \Vert E_{k+1,q+1}-E_{k,q} \Vert _{F}^{2} \\ &{} +2\mu _{k,q}^{-1}\bigl\langle \mathcal{P}_{\bar{\varOmega }}(A_{k+1,q+1}- \ddot{A}),\hat{Y}_{k+1,q+1}-\ddot{Y}\bigr\rangle . \end{aligned}$$

We obtain the following result with the same analysis:

$$\begin{aligned} \mu _{k,q}^{-2} \Vert Y_{k,q}-\ddot{Y} \Vert _{F}^{2} =& \mu _{k,q}^{-2} \bigl\Vert \mathcal{P}_{\varOmega }(Y_{k,q}-\ddot{Y}) \bigr\Vert _{F}^{2} \\ =&\mu _{k,q}^{-2} \bigl\Vert \mathcal{P}_{\varOmega }(Y_{k+1,q+1}- \ddot{Y}) \bigr\Vert _{F} ^{2}+\mu _{k,q}^{-2} \bigl\Vert \mathcal{P}_{\varOmega }(Y_{k+1,q+1}-Y_{k,q}) \bigr\Vert _{F}^{2} \\ &{} -2\mu _{k,q}^{-1}\bigl\langle \mathcal{P}_{\varOmega }(Y_{k+1,q+1}- \ddot{Y}), \mathcal{P}_{\varOmega }(Y_{k+1,q+1}-Y_{k,q})\bigr\rangle \\ =&\mu _{k,q}^{-2} \Vert Y_{k+1,q+1}-\ddot{Y} \Vert _{F}^{2}+\mu _{k,q}^{-2} \Vert Y _{k+1,q+1}-Y_{k,q} \Vert _{F}^{2} \\ & {}+2\mu _{k,q}^{-1}\bigl\langle \mathcal{P}_{\varOmega }(A_{k+1,q+1}- \ddot{A}), \hat{Y}_{k+1,q+1}-\ddot{Y}\bigr\rangle . \end{aligned}$$

Then

$$ \min_{A\in \mathbb{T}^{m\times n}} \mu \Vert A \Vert _{*}+\frac{1}{2} \big\Vert \mathcal{P}_{\varOmega }(A-M) \big\Vert _{F}^{2} $$

(4.3)

holds true.

The sum $\sum_{k=1}^{\infty }\mu _{k,q}^{-1}\langle A_{k,q}- \ddot{A},\hat{Y}_{k,q}-\ddot{Y}\rangle <+\infty$ and $\|E_{k,q}- \ddot{E}\|^{2}+\mu _{k,q}^{-2}\|Y_{k,q}-\ddot{Y}\|^{2}$ is nonincreasing since $\|A\|_{\ast }$ is a convex function and $\hat{Y}_{k+1,q+1} \in \partial \|A\|_{\ast }$, $\langle A_{k+1,q+1}-\ddot{A}, \hat{Y} _{k+1,q+1}-\ddot{Y}\rangle \geq 0$. On the other hand, the following are true by Algorithm 3.3:

$$ \langle Y_{k,q},E_{k,q}-\ddot{E}\rangle =0 \quad \text{and} \quad \langle \ddot{Y},E_{k,q}-\ddot{E}\rangle =0. $$

Therefore, by the same idea of Theorem 2 in [18], it is obtained that Ä is the solution of (3.5). □

Theorem 4.2

Let $X=(x_{ij})\in \mathbb{R}^{n\times n}$, $\mathcal{T}(X)=(\tilde{x}_{ij})\in \mathbb{T}^{n\times n}$ be the Toeplitz matrix derived from X, introduced in (2.1). Then, for any Toeplitz matrix $Y=(y_{ij})\in \mathbb{T}^{n\times n}$, $\langle X-\mathcal{T}(X),Y\rangle =0$.

Proof

By the definition of $\mathcal{T}(X)$, we have $\sum_{i,j}(x_{ij}-\tilde{x}_{ij})=0$, $i,j=1,2,\ldots ,n$. Since Y is a Toeplitz matrix and $y_{\alpha }=y_{ij}$, $\alpha =i-j$, $i,j=1,2,\ldots , n$, then

$$\begin{aligned} \bigl\langle X-\mathcal{T}(X),Y\bigr\rangle =& \sum _{i=1}^{n}\sum_{j=1}^{n}(x_{ij}- \tilde{x}_{ij})y_{ji} \\ =& \sum_{l=1}^{n-1}{y}_{-l}(x_{ij}- \tilde{x}_{ij})+\sum_{l=0}^{n-1}{y}_{l}(x_{ij}- \tilde{x}_{ij}) \\ =& 0. \end{aligned}$$

 □

Theorem 4.3

In Algorithm 3.3, $A_{k,q}$ is a Toeplitz matrix generated by $X_{k,q}$, then

$$ \Vert A_{k,q}-\ddot{A} \Vert _{F}< \Vert X_{k,q}-\ddot{A} \Vert _{F} $$

is true with Ä being the solution of (3.5).

Proof

$$\begin{aligned} \Vert X_{k,q}-\ddot{A} \Vert _{F}^{2} =& \Vert X_{k,q}-A_{k,q}+A_{k,q}-\ddot{A} \Vert _{F}^{2} \\ =& \langle X_{k,q}-A_{k,q},X_{k,q}-A_{k,q} \rangle +2\langle X_{k,q}-A _{k,q},A_{k,q}-\ddot{A} \rangle \\ & {}+\langle A_{k,q}-\ddot{A},A_{k,q}-\ddot{A}\rangle \\ =& \Vert X_{k,q}-A_{k,q} \Vert _{F}^{2}+ \Vert A_{k,q}-\ddot{A} \Vert _{F}^{2}. \end{aligned}$$

Thus, $\|A_{k,q}-\ddot{A}\|_{F}<\|X_{k,q}-\ddot{A}\|_{F}$ holds true. □

5 Numerical experiments

In this section we report some original numerical results of two algorithms (SALM, SSALM) for some $n\times n$ matrices with different ranks. All the experiments are conducted on the same workstation with an Intel(R) Core(TM) i7-6700 CPU @ 3.40 GHz that has 16 GB memory and 16-bit operating system, running Windows 7 and Matlab (vision 2016a). We analyze and compare iteration numbers (IT), computing time in seconds (time(s)), deviation (error 1, error 2), and ratio which are defined in the following. It can be seen that the SSALM algorithm proposed in this study is highly effective compared with the SALM algorithm.

$$\begin{aligned}& {\text{\textsc{error }}} 1 =\frac{ \Vert A+E-D \Vert _{F}}{ \Vert D \Vert _{F}}, \qquad {\text{\textsc{error }}} 2 =\frac{ \Vert A-M \Vert _{F}}{ \Vert M \Vert _{F}}, \\& {\text{\textsc{ratio }}}=\frac{{\text{the CPU of the SSALM algorithm}}}{ {\text{the CPU of the SALM algorithm}}}\times 100\%. \end{aligned}$$

In the experiments, $M\in \mathbb{T}^{n\times n}$ represents an uncompleted Toeplitz matrix, the sampling density $p= m/(2n-1)$, where m is the number of the known diagonal entries. By the way, $p\in \{0.3,0.4,0.5,0.6\}$ here. Due to the special structure of a Toeplitz matrix, we have $0\leq m \leq 2n-1$. The SALM and SSALM algorithms share the same values of all parameters, say, $\tau _{0}=1/ \|D\|_{2}$, $\delta =1.2172 + {1.8588m}/{n^{2}}$, $\epsilon _{1}=10^{-9}$, $\epsilon _{2}=5\times 10^{-6}$. In the test of the SSALM algorithm, $\ell =2$ or $\ell =3$ as a rule of semi-smoothing.

The experimental results of two algorithms are presented in Tables 1–6. From the tables, two algorithms can successfully compute an approximate solution of the prescriptive stop condition for all the test matrix M. And computing time of our SSALM algorithm in far less than that of the SALM algorithm. In particular, compared with the cost of the SALM algorithm, we can find that the cost of the SSALM algorithm is decreased up to 30.44%. The “ratio” in Tables 5–6 can show this effectiveness.

Table 1 Comparison between SALM and SSALM for $p=0.6$

Table 2 Comparison between SALM and SSALM for $p=0.5$

Table 3 Comparison between SALM and SSALM for $p=0.4$

Table 4 Comparison between SALM and SSALM for $p=0.3$

Table 5 The values of ratio for $\ell =2$

Table 6 Comparison between SALM and SSALM for $\ell =3$

6 Concluding remarks

As is known to all, matrix completion usually means to reconstruct a matrix from a subset of the items of a matrix by taking advantage of low-rank structure matrix interdependencies between the entries. It is well known but NP-hard in general. In recent years, the Toeplitz matrix completion has attracted widespread attention and has become one of the most important completion problems. In order to solve such problems, we proposed an accelerated technique of the SALM algorithm in this study, namely the SSALM algorithm, and the theory of the SSALM algorithm convergence is established. Theoretical analysis and numerical results have shown that the SSALM scheme is an effective algorithm for solving the TMC problem. In particular, the CPU of the SSALM algorithm is consistently reduced up to 30.44% in all cases. The SSALM algorithm overcomes the original ALM algorithm both complexity of the singular value decomposition and surmounts the property of the extra load of the SALM algorithm. The reason is that data communication congestion is far more expensive than computing in many computers. Therefore, the SSALM algorithm has better numerical behavior for solving the TMC problem than the SALM algorithm (Tables 1–6).