A modified exact smooth penalty function for nonlinear constrained optimization

Abstract

In this paper, a modified simple penalty function is proposed for a constrained nonlinear programming problem by augmenting the dimension of the program with a variable that controls the weight of the penalty terms. This penalty function enjoys improved smoothness. Under mild conditions, it can be proved to be exact in the sense that local minimizers of the original constrained problem are precisely the local minimizers of the associated penalty problem.

MSC:47H20, 35K55, 90C30.

1 Introduction

Merit function has always taken an important role in optimization problem. It is traditionally constructed to solve nonlinear programs by augmenting the objective function or a corresponding Lagrange function some penalty or barrier terms with respect of the constraints. Then it can be optimized by some unconstrained or bounded constrained optimization softwares or sequential quadratic programming (SQP) techniques. No matter what kind of techniques are involved, the merit function always depends on a small parameter ε or large parameter $\rho ={\epsilon }^{-1}$. As $\epsilon \to 0$, the minimizer of a merit function such as a barrier function or the quadratic penalty function, converges to a minimizer of the original problem. By using some exact penalty function such as $l_1$ penalty function (see [1, 9, 20–22]), the minimizer of the corresponding penalty problem must be a minimizer of the original problem when ε is sufficiently small. There are some nonsmooth penalty functions for nonsmooth optimization problems, such as the exact penalty function using the distance function for the nonsmooth variational inequality problem in Hilbert spaces [18] and the one in [19].

The traditional exact penalty functions [8] are always nonsmooth. When it is used as a merit function to accept a new iterate in an SQP method, it may cause the Maratos effect [13]. On the other hand, a traditional smooth penalty function like the quadratic penalty function cannot be an exact one. So we must compute a sequence of minimization subproblem as $\epsilon \to 0$. At that time, ill-conditioning may occur when the penalty parameter is too large or small, which also brings difficulty of computation. In [3] and [6], some kinds of augmented Lagrangian penalty functions have been proposed with improved exactness under strong conditions. In [11], exact penalty functions via regularized gap function for variational inequalities have also been given. All these functions enjoy some smoothness, but at the very beginning, to use this smoothness we need second-order or third-order derivative information of the problem function that is difficult to estimate in practice. Besides, all the above kinds of penalty functions (see [2–4, 7, 16] for summary) may be unbounded below even when the constrained problem is bounded, which may make it difficult to locate a minimizer.

In the paper [10], a new penalty function is proposed for the constrained optimization problem. By augmenting the dimension of the program with an additional variable ε that controls the weight of the penalty terms, this new penalty function enjoys properties of smoothness and exactness, and remains bounded below under reasonable conditions. Its important new idea is that the penalty function is considered as a function of variable x and the additional variable ε simultaneously. Under proper assumptions, the minimizer $(x^{\ast },{\epsilon }_{\ast })$ of the merit function satisfies ${\epsilon }_{\ast }=0$, and $x^{\ast }$ is a minimizer of the original problem. However, the penalty function given in [10] is not smooth in a small neighborhood of $(x^{\ast },0)$, where the minimizer of the original constrained problem lies. In this paper, we give a penalty function which enjoys the properties of the penalty function given in [10] and has improved smoothness.

The rest of this paper is organized as follows. In Section 2, a penalty function is introduced for a smooth nonlinear optimization problem with equality constraints and bounded constraints. The smoothness of this penalty function is discussed, as well as other properties, including being bounded below under mild assumptions. Section 3 shows the exactness of our penalty function in the sense that under certain conditions, local minimizer of our penalty function has the form $(x^{\ast },{\epsilon }_{\ast })$ with ${\epsilon }_{\ast }=0$ and $x^{\ast }$ is a local minimizer of the original problem, and a converse result holds.

Notation Throughout this paper, we use the Euclidean norm $\parallel x\parallel =\sqrt{\sum x_k^2}$. The subvector of x indexed by the indices in J is denoted by $x_J$. We denote sets of the form

$$[\underline{x},\overline{x}]:=x\in R^n|\{\underline{x}\le x\le \overline{x}\},$$

where the lower bound $\underline{x}\in {(R\cup \{-\mathrm{\infty }\})}^n$ and the upper bound $\overline{x}\in {(R\cup \{\mathrm{\infty }\})}^n$ are vectors containing proper or infinite bounds on the components of x and $[\underline{x},\overline{x}]$ is referred to an n-dimensional box.

2 New penalty function

We consider the smooth nonlinear optimization problem with equality constraints and bound constraints:

$$(P)\begin{array}{l}minf(x)\\ \text{s.t.}x\in [u,v],F(x)=0,\end{array}$$

(2.1)

where $[u,v]$ is a box in ${\mathrm{\Re }}^n$ with nonempty interior, $f:D\to \mathrm{\Re }$ and $F:D\to {\mathrm{\Re }}^m$ are continuously differentiable in an open set D containing $[u,v]$ and $m<n$. We fix $w\in {\mathrm{\Re }}^m$ and consider the equivalent problem:

$$(\overline{P})\begin{array}{l}minf(x)\\ \text{s.t.}{F}_j(x)={\epsilon }^{\gamma }{w}_j,j=1,\dots ,m,\\ x\in [u,v],\epsilon =0,\end{array}$$

(2.2)

where $\gamma >0$.

Let $\overline{\epsilon }>0$ be an upper bound of the parameter ε. Then the corresponding penalty function $f_{\sigma }$ on $D\times [0,\overline{\epsilon }]$ for ($\overline{P}$) is given as follows:

$$f_{\sigma }(x,\epsilon )=\{\begin{array}{ll}f(x)& \text{if }\epsilon =\mathrm{△}(x,\epsilon )=0,\\ f(x)+\frac{1}{{\epsilon }^{\alpha }}\frac{\mathrm{△}(x,\epsilon )}{1-q\mathrm{△}(x,\epsilon )}+\sigma {\epsilon }^{\beta }& \text{if }\epsilon >0,\mathrm{△}(x,\epsilon )<q^{-1},\\ +\mathrm{\infty }& \text{otherwise}\end{array}$$

(2.3)

with the constraint violation measure

$$\mathrm{△}(x,\epsilon ):={\parallel F(x)-{\epsilon }^{\gamma }w\parallel }^2,$$

where, in addition, $\gamma >\alpha \ge 2\beta \ge 2$ and $q>0$, are all fixed numbers, $\sigma >0$ is a penalty parameter and $\parallel \cdot \parallel$ is a Euclidean norm, with $\parallel x\parallel =\sqrt{x^{T}x}$ for any vector x.

Obviously, $\epsilon =\mathrm{△}(x,\epsilon )=0$ if and only if $\epsilon =0$, $F_j(x)=0$, $j=1,\dots ,m$. The corresponding penalty problem then reads

$$(P_{\sigma })\begin{array}{l}min{f}_{\sigma }(x,\epsilon )\\ \text{s.t.}(x,\epsilon )\in [u,v]\times [0,\overline{\epsilon }].\end{array}$$

(2.4)

The main difference between (2.3) and the penalty function given in [10] is that in (2.3), $\beta (\epsilon )={\epsilon }^{\beta }$, which does not have the property that ${\beta }^{\prime }(\epsilon )\to +\mathrm{\infty }$, as $\epsilon \to 0^{+}$.

It is easy to see that $f_{\sigma }(x,\epsilon )$ is continuously differentiable with respect to $(x,\epsilon )$ on

$$D_q:=\{(x,\epsilon )\in D\times (0,\overline{\epsilon }]|0\le \mathrm{△}(x,\epsilon )<\frac{1}{q}\}.$$

2.1 Boundedness of the penalty function

If $F(x)=0$, $(x,\epsilon )\in D_q$, then

$$f_{\sigma }(x,\epsilon )=f(x)+\frac{{\epsilon }^{2\gamma -\alpha }{\parallel w\parallel }^2}{1-q{\epsilon }^{2\gamma }{\parallel w\parallel }^2}+\sigma {\epsilon }^{\beta }\ge f_{\sigma }(x,0)=f(x).$$

(2.5)

Therefore, $f_{\sigma }(x,\epsilon )$ is bounded below on the set

$$D^{\prime }=\{x\in [u,v]|\parallel F(x)\parallel \le q^{-1/2}+{\overline{\epsilon }}^{\gamma }\parallel w\parallel \},$$

whenever $f(x)$ is bounded below on the set $D^{\prime }$. This is a reasonable condition since it usually holds when f is bounded below on the feasible set, $\overline{\epsilon }$ is small enough, and q is large enough.

The denominator $1-q\mathrm{△}(x,\epsilon )$ is included since it forces the level sets of $f_{\sigma }$ to remain in the set $\{(x,\epsilon )\in {\mathrm{\Re }}^{n+1}|\mathrm{△}(x,\epsilon )<q^{-1}\}$, hence in some sense does not go far away from the feasible set of (P).

Now we see a simple example:

$$\begin{array}{l}min{x}^7\\ \text{s.t.}{x}^2-1=0,\\ x\in \mathrm{\Re }.\end{array}$$

It has a bounded feasible domain, a global minimizer at $x^{\ast }=-1$ with $f(x^{\ast })=-1$, and a local minimizer $x=1$. The traditional quadratic penalty function for this problem

$$P(x)=x^7+\frac{1}{\epsilon }{(x^2-1)}^2,$$

is unbounded below for all penalty parameters $\epsilon >0$ since, e.g., $p(x)\to -\mathrm{\infty }$ for $x=-s$, $s\to +\mathrm{\infty }$. It is also the case for traditional penalty functions, including multiplier penalty functions that use an additional term $+\lambda (x^2-1)$. On the other hand, our new penalty function is bounded below. Set $w=1$, it reads

$$f_{\sigma }(x,\epsilon )=\{\begin{array}{ll}{x}^7& \text{if }\epsilon =r=0,\\ x^7+\frac{1}{{\epsilon }^{\alpha }}\frac{r^2}{1-q{r}^2}+\sigma {\epsilon }^{\beta }& \text{if }\epsilon >0,|r|<q^{-1/2},\\ +\mathrm{\infty }& \text{otherwise,}\end{array}$$

where $r=1+{\epsilon }^{\gamma }-x^2$. Since $f_{\sigma }(x,\epsilon )=+\mathrm{\infty }$, if $|x|\ge \sqrt{q^{-1/2}+1+{\epsilon }^{\gamma }}$, it is obvious that our penalty function is bounded below. See Figure 1 for the display of the contour of the penalty function on this example.

Figure 1

$\mathit{\beta }\mathbf{=}\mathbf{1}$ , $\mathit{\gamma }\mathbf{=}\mathbf{3}$ , $\mathit{\alpha }\mathbf{=}\mathbf{2}$ , and $\mathit{\sigma }\mathbf{=}\mathbf{2}\mathbf{\text{,}}\mathbf{000}$ .

3 Exactness of the penalty function

In this section, we show that our penalty function is exact in the sense that under certain conditions, local minimizer of our penalty function has the form $(x^{\ast },{\epsilon }_{\ast })$ in which ${\epsilon }_{\ast }=0$ and $x^{\ast }$ is a local minimizer of the original problem and a converse proposition holds.

Firstly, recall the Mangasarian-Fromovitz condition. We say that the Mangasarian-Fromovitz condition (see [12]) for Problem (P) holds at $x\in [u,v]$ if $F^{\prime }(x)$ has full rank and there is a vector $p\in {\mathrm{\Re }}^n$ with $F^{\prime }(x)p=0$ and

$$p_i\{\begin{array}{ll}>0& \mathit{\text{if }}{x}_i=u_i,\\ <0& \mathit{\text{if }}{x}_i=v_i.\end{array}$$

(3.1)

Theorem 3.1 Assume that the set $D^{\prime }$ defined in Section  2 is bounded, and the Mangasarian-Fromovitz condition holds at each $x^{\prime }\in D^{\prime }$. Let $\gamma >\alpha \ge 2\beta \ge 2$ in ($P_{\sigma }$). If σ is sufficiently large, then there is no Kuhn-Tucker point $(x,\epsilon )$ of ($P_{\sigma }$) with $\epsilon >0$.

Proof Let the Lagrangian function of ($P_{\sigma }$) for $\sigma >0$ be

$$L(x,\epsilon ,y,z)=f_{\sigma }(x,\epsilon )+\sum _{i=1}^n{y}_i(u_i-x_i)+\sum _{i=1}^n{z}_i(x_i-v_i)+y_{n+1}(-\epsilon )+z_{n+1}(\epsilon -\overline{\epsilon }),$$

where $y_i,z_i\in \mathrm{\Re }$, $i=1,\dots ,n+1$ are the Lagrangian multipliers. If $(x,\epsilon )$ is a Kuhn-Tucker point of ($P_{\sigma }$) with $\epsilon >0$, then there exist vectors $y,z\in {\mathrm{\Re }}^{n+1}$ such that

$$\begin{array}{c}{\mathrm{\nabla }}_{(x,\epsilon )}L(x,\epsilon ,y,z)=0,\hfill \\ u_i-x_i\le 0,x_i-v_i\le 0,i=1,\dots ,n,\hfill \\ -\epsilon <0,\epsilon -\overline{\epsilon }\le 0,\hfill \\ y_i(u_i-x_i)=0,z_i(v_i-x_i)=0,z_{n+1}(\epsilon -\overline{\epsilon })=0,y_{n+1}\epsilon =0,\hfill \end{array}$$

then

$${\mathrm{\nabla }}_{(x,\epsilon )}{f}_{\sigma }(x,\epsilon )=y-z,$$

and

$$\begin{array}{c}inf(y_i,x_i-u_i)=inf(z_i,v_i-x_i)=0,i=1,\dots ,n,\hfill \\ y_{n+1}=inf(z_{n+1},\overline{\epsilon }-\epsilon )=0,\hfill \end{array}$$

where ${\mathrm{\nabla }}_{(x,\epsilon )}{f}_{\sigma }(x,\epsilon )$ is the gradient of $f_{\sigma }$ with respect to $(x,\epsilon )$. The assertion of the theorem is proved by contradiction. □

Assume that there exists a sequence $\{(x^k,{\epsilon }_k,{\sigma }_k)\}$ with ${\epsilon }_k>0$ for all k, ${\sigma }_k\to +\mathrm{\infty }$ as $k\to \mathrm{\infty }$, where $(x^k,{\epsilon }_k)$ is a Kuhn-Tucker point of ($P_{{\sigma }_k}$). We use the abbreviation ${\mathrm{△}}_k:=\mathrm{△}(x^k,{\epsilon }_k)$. The point $x^k$ satisfies

$$\parallel F\left(x^k\right)\parallel \le {\mathrm{△}}_k^{1/2}+{\epsilon }_k^{\gamma }\parallel w\parallel \le q^{-1/2}+{\overline{\epsilon }}^{\gamma }\parallel w\parallel ,$$

hence, $x^k\in D^{\prime }=\{x\in [u,v]|\parallel F(x)\parallel \le q^{-1/2}+{\overline{\epsilon }}^{\gamma }\parallel w\parallel \}$. Since $D^{\prime }$ is closed and bounded, we may restrict ourselves to a subsequence if necessary and assume that

$$\underset{k\to \mathrm{\infty }}{lim}{\epsilon }_k={\epsilon }_{\ast }\in [0,\overline{\epsilon }]\text{and}\underset{k\to \mathrm{\infty }}{lim}{x}^k=x^{\ast }\in D^{\prime }.$$

The condition $\frac{\partial }{\partial \epsilon }{f}_{\sigma }(x^k,{\epsilon }_k)\le 0$ yields

$$\begin{array}{r}\alpha q{\mathrm{△}}_k^2+(2\gamma -\alpha ){\epsilon }_k^{2\gamma }{\parallel w\parallel }^2+2(\alpha -\gamma ){\epsilon }_k^{\gamma }\sum _{j=1}^m{F}_j\left(x^k\right)w_j\\ +\beta {\epsilon }_k^{\alpha +\beta }{(1-q{\mathrm{△}}_k)}^2{\sigma }_k\le \alpha \sum _{j=1}^m{F}_j^2\left(x^k\right)\end{array}$$

(3.2)

with equality in the case ${\epsilon }_k\ne \overline{\epsilon }$. When ${\sigma }_k\to \mathrm{\infty }$ and because $\alpha {\sum }_{j=1}^m{F}_j^2(x^k)$ - the right side of (3.2) is a finite number, we have

$$\underset{k\to \mathrm{\infty }}{lim}{\epsilon }_k={\epsilon }_{\ast }=0\text{or}\underset{k\to \mathrm{\infty }}{lim}\mathrm{△}(x^k,{\epsilon }_k)={\mathrm{△}}^{\ast }=q^{-1},$$

(3.3)

where ${\mathrm{△}}^{\ast }=\mathrm{△}(x^{\ast },{\epsilon }_{\ast })$. On the other side, the derivative the penalty function $f_{\sigma }$ with respect to x is given as

$$\begin{array}{rcl}\frac{\partial }{\partial x_i}{f}_{\sigma }(x^k,{\epsilon }_k)& =& \frac{\partial }{\partial x_i}f\left(x^k\right)+\frac{1}{{\epsilon }_k^{\alpha }}\frac{\frac{\partial }{\partial x_i}\mathrm{△}(x^k,{\epsilon }_k)}{1-q\mathrm{△}(x^k,{\epsilon }_k)}+\frac{1}{{\epsilon }_k^{\alpha }}\frac{q\mathrm{△}(x^k,{\epsilon }_k)\frac{\partial }{\partial x_i}\mathrm{△}(x^k,{\epsilon }_k)}{{(1-q\mathrm{△}(x^k,{\epsilon }_k))}^2}\\ =& \frac{\partial }{\partial x_i}f\left(x^k\right)+\frac{1}{{\epsilon }_k^{\alpha }}\frac{\frac{\partial }{\partial x_i}\mathrm{△}(x^k,{\epsilon }_k)}{{(1-q\mathrm{△}(x^k,{\epsilon }_k))}^2}\{\begin{array}{ll}\ge 0& \text{if }{x}_i^k=u_i,\\ =0& \text{if }{u}_i<x_i^k<v_i,\\ \le 0& \text{if }{x}_i^k=v_i\end{array}\end{array}$$

(3.4)

(3.4) is equivalent to

$$\begin{array}{r}{\epsilon }_k^{\alpha }{(1-q\mathrm{△}(x^k,{\epsilon }_k))}^2\frac{\partial }{\partial x_i}f\left(x^k\right)+\frac{\partial }{\partial x_i}\mathrm{△}(x^k,{\epsilon }_k)\\ ={\epsilon }_k^{\alpha }{(1-q\mathrm{△}(x^k,{\epsilon }_k))}^2\frac{\partial }{\partial x_i}f\left(x^k\right)+2{\left(F^{\prime }{\left(x^k\right)}^T(F\left(x^k\right)-{\epsilon }_k^{\gamma }w)\right)}_i\\ \{\begin{array}{ll}\ge 0& \text{if }{x}_i^k=u_i,\\ =0& \text{if }{u}_i<x_i^k<v_i,\\ \le 0& \text{if }{x}_i^k=v_i.\end{array}\end{array}$$

(3.5)

Let $k\to +\mathrm{\infty }$, since ${\epsilon }_{\ast }=0$ or ${\mathrm{△}}^{\ast }=q^{-1}$, we have

$${\left(F^{\prime }{\left(x^{\ast }\right)}^T(F\left(x^{\ast }\right)-{\epsilon }_{\ast }^{\gamma }w)\right)}_i\{\begin{array}{ll}\ge 0& \text{if }{x}_i^{\ast }=u_i,\\ =0& \text{if }{u}_i<x_i^{\ast }<v_i,\\ \le 0& \text{if }{x}_i^{\ast }=v_i.\end{array}$$

(3.6)

Because $x^{\ast }\in D^{\prime }$, thus Mangasarian-Fromovitz condition holds at $x^{\ast }$ and there exists some vector $p\in {\mathrm{\Re }}^n$ such that $F^{\prime }(x^{\ast })p=0$, where p satisfies (3.1). Let $I_1:=\{i|x_i^{\ast }=u_i\}$, $I_2:=\{i|x_i^{\ast }=v_i\}$ and $\overline{{\mathrm{△}}^{\ast }}:=F(x^{\ast })-{\epsilon }_{\ast }^{\gamma }w$. Then by (3.6), we have

$$0={\left(F^{\prime }\left(x^{\ast }\right)p\right)}^T\overline{{\mathrm{△}}^{\ast }}=\sum _{i\in I_1}{p}_i{\left(F^{\prime }{\left(x^{\ast }\right)}^T\overline{{\mathrm{△}}^{\ast }}\right)}_i+\sum _{i\in I_2}{p}_i{\left(F^{\prime }{\left(x^{\ast }\right)}^T\overline{{\mathrm{△}}^{\ast }}\right)}_i.$$

(3.7)

(3.7) and the Mangasarian-Fromovitz condition (3.1) imply ${(F^{\prime }{(x^{\ast })}^T\overline{{\mathrm{△}}^{\ast }})}_i=0$ for $i\in I_1\cup I_2$. Thus, $F^{\prime }{(x^{\ast })}^T\overline{{\mathrm{△}}^{\ast }}=0$. Now the fact that $F^{\prime }(x^{\ast })$ has full rank yields $\overline{{\mathrm{△}}^{\ast }}=0$, i.e.,

$$F\left(x^{\ast }\right)-{\epsilon }_{\ast }^{\gamma }w=0,$$

and by $\overline{{\mathrm{△}}^{\ast }}={lim}_{k\to \mathrm{\infty }}(F(x^k)-{\epsilon }_k^{\gamma }w)=0$, it follows that

$$\underset{k\to \mathrm{\infty }}{lim}\mathrm{△}(x^k,{\epsilon }_k)=\underset{k\to \mathrm{\infty }}{lim}{\parallel F\left(x^k\right)-{\epsilon }_k^{\gamma }w\parallel }^2={\mathrm{△}}^{\ast }=0.$$

By (3.3), we obtain

$$\underset{k\to \mathrm{\infty }}{lim}{\epsilon }_k={\epsilon }_{\ast }=0.$$

Thus, ${lim}_{k\to \mathrm{\infty }}F(x^k)=F(x^{\ast })=0$.

Furthermore, by (3.2), it holds that

$$\begin{array}{r}\frac{q}{{\epsilon }_k^{\alpha +\beta }}{\mathrm{△}}^2(x^k,{\epsilon }_k)+\frac{2\frac{\gamma }{\alpha }-1}{{\epsilon }_k^{\alpha +\beta -2\gamma }}{\parallel w\parallel }^2+2(1-\frac{\gamma }{\alpha }){\epsilon }_k^{\gamma -(\alpha +\beta )}\sum _{i=1}^m{F}_i\left(x^k\right)w_i\\ +\frac{\beta }{\alpha }{(1-q\mathrm{△}(x^k,{\epsilon }_k))}^2{\sigma }_k\le \frac{{\sum }_{i=1}^m{F}_i^2(x^k)}{{\epsilon }_k^{\alpha +\beta }}.\end{array}$$

Let $k\to \mathrm{\infty }$, the last term on the left-hand side tend to +∞. Thus, the vectors $y^k=\frac{F(x^k)}{{\epsilon }_k^{\frac{\alpha +\beta }{2}}}$ satisfies $\parallel y^k\parallel \to +\mathrm{\infty }$. The vectors $z^k=\frac{y^k}{\parallel y^k\parallel }$ have norm 1, and (3.5) implies that the numbers ${\mu }_i^k$ ($i=1,\dots ,n$), defined by

$$\begin{array}{rcl}{\mu }_i^k& =& \frac{1}{\parallel y^k\parallel }\frac{\partial }{\partial x_i}f\left(x^k\right)+\frac{1}{\parallel y^k\parallel }\frac{2}{{\epsilon }_k^{\alpha }{(1-q\mathrm{△}(x^k,{\epsilon }_k))}^2}{\left(F^{\prime }{\left(x^k\right)}^T(F(x_k)-{\epsilon }_k^{\gamma }w)\right)}_i\\ =& \frac{1}{\parallel y^k\parallel }\frac{\partial }{\partial x_i}f\left(x^k\right)+\frac{2}{{\epsilon }_k^{\frac{\alpha -\beta }{2}}{(1-q\mathrm{△}(x^k,{\epsilon }_k))}^2}{\left(F^{\prime }{\left(x^k\right)}^T(z^k-\frac{{\epsilon }_k^{\gamma -\frac{\alpha +\beta }{2}}w}{\parallel y^k\parallel })\right)}_i,\end{array}$$

satisfy

$${\mu }_i^k\{\begin{array}{ll}\ge 0& \text{if }{x}_i^k=u_i,\\ =0& \text{if }{u}_i<x_i^k<v_i,\\ \le 0& \text{if }{x}_i^k=v_i.\end{array}$$

If we pick a convergent subsequence $z^{n_k}$ with the limit $z^{\ast }$ and pass to the limit we obtain

$${\left(F^{\prime }\left(x^{\ast }\right)z^{\ast }\right)}_i\{\begin{array}{ll}\ge 0& \text{if }{x}_i^{\ast }=u_i,\\ =0& \text{if }{u}_i<x_i^{\ast }<v_i,\\ \le 0& \text{if }{x}_i^{\ast }=v_i.\end{array}$$

Now similarly as above, it yields $z^{\ast }=0$, which is a contradiction with $\parallel z^{\ast }\parallel =1$. Thus such a sequence $\{(x^k,{\epsilon }_k,{\sigma }_k)\}$ cannot exist, and for sufficiently large $\sigma >0$, all Kuhn-Tucker points of ($P_{\sigma }$) are of the form $(x,0)$.

Theorem 3.2 Assume that $(x^{\ast },{\epsilon }_{\ast })$ is a local minimizer of minimizer of ($P_{\sigma }$) with finite $f_{\sigma }(x^{\ast },{\epsilon }_{\ast })$, where $\sigma >0$ is sufficiently large. If the hypotheses of Theorem  3.1 are fulfilled, then $x^{\ast }$ is a local minimizer of (P).

Proof Now let $(x^{\ast },{\epsilon }_{\ast })$ be a local minimizer of ($P_{\sigma }$) with finite $f_{\sigma }(x^{\ast },{\epsilon }_{\ast })$ and $\sigma >0$ is sufficiently large. If ${\epsilon }_{\ast }>0$, then $(x^{\ast },{\epsilon }_{\ast })$ must be a Kuhn-Tucker point of ($P_{\sigma }$), which is a contradiction with Theorem 3.1. Therefore, ${\epsilon }_{\ast }=0$, and since $f_{\sigma }(x^{\ast },{\epsilon }_{\ast })$ is finite, $\mathrm{△}(x^{\ast },{\epsilon }_{\ast })=0$. It implies that $F(x^{\ast })=0$, and by (2.5) there is a neighborhood $N(x^{\ast })$ of $x^{\ast }$ where $f(x)\ge f(x^{\ast })$ for feasible x. Therefore, $x^{\ast }$ is a local minimizer of (P). □

We now show a converse result of Theorem 3.2, which will use the following lemmas.

Lemma 3.1 Suppose $F(x^{\ast })=0$, and $F^{\prime }(x^{\ast })$ has full rank. Then there exist a neighborhood $N_0(x^{\ast })$ of $x^{\ast }$ and a constant ${\kappa }_0>0$ such that for each $x\in N_0(x^{\ast })$, and each subset J of $\{1,2,\dots ,m\}$, there exists a vector $y=y(x)\in N_0(x^{\ast })$ with $F_i(y)=0$, for $i\in J$ and $F_i(y)=F_i(x)$, $i\in K=\{1,2,\dots ,m\}\setminus J$, such that

$$\parallel x-y\parallel \le {\kappa }_0\parallel F_J(x)\parallel .$$

Proof Since $F(x^{\ast })=0$ and $F^{\prime }(x^{\ast })$ has full rank, there exists a matrix $B\in {\mathrm{\Re }}^{(n-m)\times n}$ such that the augmented matrix $\left(\begin{array}{c}{F}^{\prime }(x^{\ast })\\ B\end{array}\right)$ is nonsingular. By the continuity of $F^{\prime }(\cdot )$ at $x^{\ast }$, there exists a neighborhood $N_1(x^{\ast })\subset D$ of $x^{\ast }$ such that $\left(\begin{array}{c}{F}^{\prime }(x)\\ B\end{array}\right)$ is nonsingular, for any $x\in N_1(x^{\ast })$. Take for $\mathcal{A}$ the closed convex hull of $\{F^{\prime }(x)|x\in N_1(x^{\ast })\}$, then for all $A\in \mathcal{A}$, the matrix $\left(\begin{array}{c}A\\ B\end{array}\right)$ is nonsingular. We now show that for any $x,y\in N_1(x^{\ast })$, there exists a matrix $A\in \mathcal{A}$ such that

$$F(x)-F(y)=A(x-y).$$

(3.8)

In fact, given $x,y\in N_1(x^{\ast })$, it follows from the mean value theorem that

$$\begin{array}{rl}F(x)-F(y)& ={\int }_0^1{F}^{\prime }(y+s(x-y))(x-y)ds\\ =A_{x,y}(x-y),\end{array}$$

where $A_{x,y}={\int }_0^1{F}^{\prime }(y+s(x-y))ds\in \mathcal{A}$, so (3.8) holds. Set the mapping $H(z):=\left(\begin{array}{c}F(z)\\ B(z-x^{\ast })\end{array}\right)$, for $z\in N_1(x^{\ast })$. By the proof in [10], Theorem 4.5], we have that there exists a neighborhood $N_0(x^{\ast })\subset N_1(x^{\ast })$ of $x^{\ast }$ such that for each $x\in N_0(x^{\ast })$, and each subset J of $\{1,2,\dots ,m\}$, there exists a vector $y=y(x)\in N_0(x^{\ast })$ with

$$H(y)=\left(\begin{array}{c}F(y)\\ B(y-x^{\ast })\end{array}\right)=\left(\begin{array}{c}0\\ F_K(x)\\ B(x-x^{\ast })\end{array}\right),$$

so $F_i(y)=0$, for $i\in J$ and $F_i(y)=F_i(x)$, $i\in K$.

For $x,y\in N_0(x^{\ast })$, we have

$$H(x)-H(y)=\left(\begin{array}{c}A\\ B\end{array}\right)(x-y),$$

for some $A\in \mathcal{A}$. On the other side, we have

$$\parallel H(x)-H(y)\parallel =\parallel F_J(x)\parallel .$$

(3.9)

Therefore, combining (3.8) with (3.9), we have

$$\parallel x-y\parallel \le \parallel {\left(\begin{array}{c}A\\ B\end{array}\right)}^{-1}\parallel \parallel F_J(x)\parallel \le {\kappa }_0\parallel F_J(x)\parallel ,$$

where

$${\kappa }_0:=\parallel \underset{A\in \mathcal{A}}{sup}{\left(\begin{array}{c}A\\ B\end{array}\right)}^{-1}\parallel <+\mathrm{\infty },$$

and this complete the proof. □

Lemma 3.2 Assume that $x^{\ast }$ is a local minimizer of problem (2.1) with $u_i\le x_i^{\ast }\le v_i$, $i\in \{1,\dots ,p\}$, where $m\le p\le n$. Suppose that $F^{\prime }(x^{\ast })$ has full rank. Then there exists a constant ${\kappa }_1>0$ such that

$$f(x)\ge f\left(x^{\ast }\right)-{\kappa }_1\parallel F(x)\parallel ,\mathit{\text{for all }}x\in N_0\left(x^{\ast }\right),$$

where $N_0(x^{\ast })$ is defined in Lemma  3.1.

Proof From Lemma 3.1, let $N_0(x^{\ast })$ and ${\kappa }_0$ be the one in Lemma 3.1. Let $x\in N_0(x^{\ast })$, then by Lemma 3.1 with $J=\{1,\dots ,m\}$, there exists an $y=y(x)$ with $F(y)=0$ and $y_i=x_i^{\ast }$, $i=m+1,\dots ,n$ such that

$$\parallel x-y\parallel \le {\kappa }_0\parallel F(x)\parallel .$$

(3.10)

So y is a feasible point of problem (2.1), and $f(y)\ge f(x^{\ast })$. Since f is continuously differentiable, for any $x,y\in N_0(x^{\ast })$, there exists a vector $\xi \in {\mathrm{\Re }}^n$ such that

$$f(x)-f(y)=\mathrm{\nabla }f{(\xi )}^T(x-y),$$

where ξ lies in the segment between x and y. Set $L:={sup}_{z\in N(x^{\ast })}\parallel \mathrm{\nabla }f(z)\parallel$, we have

$$\begin{array}{rl}|f(x)-f(y)|& \le \parallel \mathrm{\nabla }f(\xi )\parallel \parallel x-y\parallel \\ \le L\parallel x-y\parallel \\ \le L{\kappa }_0\parallel F(x)\parallel ,\end{array}$$

where the last inequality holds from (3.10).

Let ${\kappa }_1=L{\kappa }_0$, then

$$f(x)=f(x)-f(y)+f(y)\ge f\left(x^{\ast }\right)-{\kappa }_1\parallel F(x)\parallel ,$$

which complete the proof. □

Theorem 3.3 If $x^{\ast }$ is a local minimizer of problem (P) with $u_i\le x_i^{\ast }\le v_i$, $i\in \{1,\dots ,p\}$, where $m\le p\le n$, and $F^{\prime }(x^{\ast })$ has full rank, then for sufficiently large $\sigma >0$, there are a neighborhood $N(x^{\ast })$ of $x^{\ast }$ and a ${\epsilon }^{\prime }\in (0,\overline{\epsilon }]$ such that

$$f_{\sigma }(x,\epsilon )>f_{\sigma }(x^{\ast },0)=f\left(x^{\ast }\right),\mathit{\text{for all }}(x,\epsilon )\in N\left(x^{\ast }\right)\times (0,{\epsilon }^{\prime }].$$

In particular, $(x^{\ast },0)$ is a local minimizer of $f_{\sigma }$.

Proof Let $N(x^{\ast })\subset N_0(x^{\ast })$ is a neighborhood of $x^{\ast }$ such that

$$\underset{x\in N(x^{\ast })}{sup}(f\left(x^{\ast }\right)-f(x))<1,$$

(3.11)

where $N_0(x^{\ast })$ is defined in Lemma 3.1.

For $(x,\epsilon )\in N(x^{\ast })\times (0,{\epsilon }^{\prime }]$, where ${\epsilon }^{\prime }\in (0,\overline{\epsilon }]$ and ${\epsilon }^{\prime }\le 1$, we distinguish two cases.

Case 1. $\mathrm{△}(x,\epsilon )={\parallel F(x)-{\epsilon }^{\gamma }w\parallel }^2\ge {\epsilon }^{\alpha }$. For this case, we have

$$\begin{array}{rl}{f}_{\sigma }(x,\epsilon )& \ge f(x)+1+\sigma {\epsilon }^{\beta }\\ \ge f\left(x^{\ast }\right)+\sigma {\epsilon }^{\beta }\\ >f\left(x^{\ast }\right)=f_{\sigma }(x^{\ast },0),\end{array}$$

where the second inequality is by (3.11).

Case 2. $\mathrm{△}(x,\epsilon )<{\epsilon }^{\alpha }$. Then $\parallel F(x)\parallel \le {\mathrm{△}}^{\frac{1}{2}}+{\epsilon }^{\gamma }\parallel w\parallel \le {\epsilon }^{\frac{\alpha }{2}}+{\epsilon }^{\gamma }\parallel w\parallel$, and

$$\begin{array}{rl}{f}_{\sigma }(x,\epsilon )& \ge f(x)+\sigma {\epsilon }^{\beta }\\ \ge f\left(x^{\ast }\right)-{\kappa }_1\parallel F(x)\parallel +\sigma {\epsilon }^{\beta }\\ \ge f\left(x^{\ast }\right)-{\kappa }_1({\epsilon }^{\frac{\alpha }{2}}+{\epsilon }^{\gamma }\parallel w\parallel )+\sigma {\epsilon }^{\beta }\\ \ge f\left(x^{\ast }\right)+(\sigma -{\kappa }_1(1+{\epsilon }^{\gamma -\frac{\alpha }{2}})\parallel w\parallel ){\epsilon }^{\frac{\alpha }{2}}.\end{array}$$

(3.12)

The last inequality holds since $\beta \le \frac{\alpha }{2}$.

Let $\sigma >{\kappa }_1(1+{\epsilon }^{\gamma -\frac{\alpha }{2}})\parallel w\parallel$, we get

$$f_{\sigma }(x,\epsilon )\ge f\left(x^{\ast }\right)=f_{\sigma }(x^{\ast },0).$$

From Case 1 and Case 2, we obtain the conclusion. □

4 Conclusion remarks

In this paper, a modified exact penalty function for equality constrained nonlinear programming problem is constructed by augmenting a new variable that controls the constraint violence. This function enjoys smoothness, and with very mild conditions it is proved to be an exact penalty function.

Since in practice, a lot of applied problems are nonsmooth, it is a meaningful work to extend the results in this paper to the nonsmooth case. By using the limiting subgradients that is presented in two books written by Mordukhovich [14, 15], as well as Clarke’s generalized gradients in [5], we can extend the penalty function with the mentioned good properties to nonsmooth optimization problems, just as that has been done in [17–19]. That will be our future research direction.