Bifurcation of a feed forward neural network with delay and application in image contrast enhancement

1.   Introduction

The theories and applications of neural networks have been extensively developed after the works of Cohen ^[1] and Hopfield ^[2]. The research of neural network is quite extensive, which reflects the characteristics of the multi discipline cross technology field. Due to the finite propagating speed in the signal switching and transmission between the neurons, time delay is inevitable in the neural network and thus should be incorporated in the mathematical model. Different types of neural network systems with time delays have been proposed and developed. In these models, various types of dynamical behaviors including stability, chaos, and bifurcation were investigated (see ^[3,4,5,6,7]). Furthermore, the wide applications including pattern recognition of speech and images, associative memory, signal processing have been done (see ^[8]).

In the classical Hopf bifurcation theorem for ordinary differential equations, as a pair of complex-conjugate simple eigenvalues crosses the imaginary axis, there is born a unique branch of periodic orbits near an equilibrium point. This paper investigates the Hopf-zero singularity case of feed-forward neural networks model with delay, in which the purely imaginary eigenvalues at criticality and zero coexist. In this case, the Hopf-pitchfork can occur. Hopf-pitchfork bifurcation has been investigated for a long time as an important dynamical behavior ^{[9,10,11,12,13,14]}. In particular, there are some works on Hopf-pitchfork bifurcation in systems with delay ^[16,17,18]. For example, in 2015, Wang et al. ^[15] investigated Hopf-pitchfork bifurcation in a two-neuron system with discrete and distributed delays. In 2012, Dong et al. ^[7] studied Hopf-pitchfork bifurcation in an inertial two-neuron system with time delay.

This paper partly deals with a simple case of feed-forward system. We consider the following two-neuron feed-forward neural network model

There are many interesting properties in the feed-forward chain model which have great potential application prospect. For example, signal propagation in the feed-forward neural network has been discussed by some researchers, see ^[19,20,21]. In this paper, we will see that the simple model (1.1) can show the complex dynamic properties. The rest of the article is organized as follows. In Section 2, we derive the existence condition of the Hopf-pitchfork bifurcation with interaction coefficient and delay as two parameters. In Section 3, we obtain and analyze the normal form and the unfolding for Hopf-pitchfork bifurcation in the feed-forward neural network system with time delay, as well as Hopf-pitchfork diagrams.

Studies have found that some neurons may exhibit forced vibrator behavior. This characteristic of the system can detect and amplify signals of a specific frequency. M. Golubitsky ^[19] et al. constructed a nonlinear feed-forward network coupled oscillator mode, and utilized the inherent nonlinear response near Hopf branch point of the oscillator to achieve a significant amplification effect on a small frequency band width signal. In section 4 and 5, we discuss the existence and normal forms of Hopf bifurcation. Through numerical simulation, we find that when Hopf branches occur, the model shows properties similar to the conclusion of M. Golubitsky ^[19] et al..

Image enhancement is to highlight and strengthen the target area of the original image purposefully, suppress the features that are not of interest, enhance the recognition of the image and improve the visual effect of the image. Image enhancement allows the enhanced image to be inconsistent with the original image. The final purpose of image enhancement is to facilitate the further analysis and processing of images. Tiger and other animal images captured in the forest often have low contrast due to natural factors such as trees, grass and equipment, which will further affect the effect of image processing. In order to improve the sharpness of animal image and highlight its edge information. In section 6, we use the feed forward neural network model to enhance the image. With the help of dynamic properties, this algorithm can achieve a good signal amplification effect. Therefore, the image enhancement effect is obvious, and it can improve the image clarity and contrast. In the final section, we give some conclusions and future works.

2.   The existence of Hopf-pitchfork bifurcation

It is clear that (0, 0) is an equilibrium point of Eq (1.1). The linearization of Eq (1.1) at the origin leads to

The associated characteristic equation of Eq (2.1) takes the form

In the following, if the characteristic Eq (2.2) has a simple root 0 and a pair of purely imaginary roots $\pm i\omega$ and all other roots of the characteristic equation have negative real parts, then the Hopf-pitchfork bifurcation will occur. We make the following assumptions:

Lemma 2.1 If the assumption $(H_{1}), (H_{2})$ are satisfied, then all the roots of Equation (2.2) have negative real parts except a single zero root and a pair of purely imaginary roots when

Further, the transversality condition is satisfied at $\tau = \tau_{j}, (j = 0, 1, \cdots)$

Proof. Clearly, $\lambda = 0$ is a root of Eq (2.2), if $a+b = 1$. If $\tau = 0$, then Eq (2.2) becomes $\lambda = 0$ or $\lambda = a-1$. There is no purely imaginary root for $\Delta_{1} = 0$. We only need to consider $\Delta_{2} = 0$. Let $i \omega$ be a root of $\Delta_{2} = 0$, then

Separating the real and imaginary parts we have the values of $\omega$ and $\tau$ are given by

Through simple calculation, the transversality conditions are shown as follows:

Based on the work above, we can obtain the following Theorem.

Theorem 2.1 The system (1.1) undergoes Hopf-pitchfork bifurcation when the assumption $(H_{1})$ and $(H_{2})$ are satisfied and $\tau = \tau_{j}(j = 0, 1, \cdots)$.

3.   Normal form for Hopf-pitchfork bifurcation

In this section, Center manifold theory and normal form method ^[12,22] are used to study Hopf-pitchfork bifurcation. After scaling $t\rightarrow\frac{t}{\tau}$, system (1.1) can be written as

Suppose that the system (3.1) undergoes Hopf-pitchfork bifurcation at the critical point $b = b_{0} = 1-a, \tau = \tau_{0}$, with a pair of eigenvalues $\pm i \tau_{0} \omega$ and one zero, and all other roots have negative real parts. Let $\tau = \tau_0+\mu_1, b = 1-a+\mu_2$, $\mu_1$ and $\mu_2$ are bifurcation parameters, choosing the phase space $C = C([-1, 0];R^{2})$ with supreme norm and $U_{t}$ is defined by $U_t(\theta) = U(t+\theta), -1 \leq \theta \leq 0$. Then system (3.1) can be written as

where

Thus, system (3.2) becomes an abstract ODE in the space BC

where $\mathcal{A}$ is defined by

and

and the bilinear form on $C^{*}\times C$ ($^*$stand for adjoint) is

with $\varphi\in C, \psi\in C^{*}$.

Because $L(0)$ has a simple 0 and a pair of purely imaginary eigenvalues $\pm i \omega$ and all other eigenvalues have negative real parts. Let $\Lambda = (i\omega\tau_0, -i\omega\tau_0, 0)$ and $P$ can be the generalized eigenspace associated with $\Lambda$ and $P^{*}$ the space adjoint with $P$. Then the $C$ can be decomposed as $C = P \oplus Q$ where $Q = \{\varphi\in C|(\psi, \varphi) = 0\:for\:all\:\psi \in P^{*}\}$.

Choose the bases $\Phi$ and $\Psi$ for $P$ and $P^{*}$ such that $(\Psi(s), \Phi(\theta)) = I, \dot{\Phi} = \Phi J$, and $\dot{\Psi} = -J \Psi$, where

By calculating we choose

and

with $\bar{D}_{11} = \frac{1}{1+\tau_{0}+i\omega\tau_{0}}, D_{21} = \frac{1}{1+\tau_{0}}$.

Let $u = \Phi x+y$, that is

Then Eq (3.3) is therefore decomposed into the system

Using the idea of Faria ^[22], we know that Eq (3.4) can be written as

where $f_{j}(x, y, \mu)$ are homogeneous polynomials of degree $j$ in $(x, y, \mu)$ with coefficients in $C^{3}$.

$S_1$ is spanned by

with $e_{1} = (1, 0, 0)^{T}, e_{2} = (0, 1, 0)^{T}, e_{3} = (0, 0, 1)^{T}$.

$S_2$ is spanned by

On the center manifold, Eq (3.5) can be transform as the following normal form

where

and $a_{12} = \bar{D}_{11}i\omega, a_{21} = D_{21}\tau_{0} = \frac{\tau_{0}}{1+\tau_{0}}$.

and $b_{11} = -\bar{D}_{11}\tau_{0}(i\omega +1), b_{12} = b_{11}, b_{22} = -\frac{\tau_0}{3(1+\tau_0)}$.

Hence Eq (3.5) can be written as

Through the change of variables $x_{1} = rcos\theta-irsin\theta, x_{2} = rcos\theta+irsin\theta, x_{3} = Z$, the system (3.6) becomes

Let $\hat{Z} = Z\sqrt{|b_{22}|}, \hat{r} = r\sqrt{|Re(b_{11})|}, (Re(b_{11}) < 0, b_{22} < 0)$, after dropping the hats, the equation becomes:

where $c_{1} = Re(a_{12})\mu_{2}, c_{2} = a_{21}\mu_{1}, \sigma = \frac{Re(b_{12})}{b_{22}}$.

In Eq (3.7), $M_{0} = (r, Z) = (0, 0)$ is always an equilibrium and the other equilibria are

We obtain five distinct types of unfolding with respect to different signs in the system (3.7). Similar to the work in ^[21], we have the following Theorem:

Theorem 3.1 The detailed dynamics of system (3.7) in $D_{1}-D_{5}$ near the original parameters $b_{0}, \tau_{0}$ are as follows: (1) In $D_{1}$, Eq (3.7) has only one trivial equilibrium $M_{0}$, which is a sink.

(2) In $D_{2}$, the trivial equilibrium (corresponding to $M_{0}$) becomes a saddle from a sink, and a stable periodic orbit (corresponding to $M_{1}$) appears.

(3) In $D_{3}$, Eq (3.7) has a pair of stable periodic orbits (corresponding to $M_{3}^{\pm}$), a pair of unstable semitrivial equilibria (corresponding to $M_{2}^{\pm}$), an unstable periodic oribi(corresponding to $M_{1}$), and an unstable trivial equilibrium (corresponding to $M_{0}$).

(4) In $D_{4}$, the unstable periodic orbits (corresponding to $M_{3}^{\pm}$) disappear, the periodic orbit (corresponding to $M_{1}$) becomes stable, and the semitrivial equilibria (corresponding to $M_{2}^{\pm}$) become stable.

(5) In $D_{5}$, the periodic orbit (corresponding to $M_{1}$) disappears, the trivial equilibrium (corresponding to $M_{0}$) becomes a saddle from a source, and the semitrivial equilibria(corresponding to $M_{2}^{\pm}$) remains stable.

Choosing $a = -2, b = 3$. By direct calculation, we obtain $\omega = 1.732, \tau_0 = 1.209, D_{11} = 0.2384+0.2260i, D_{21} = 0.4527$.

(1) Choosing $\mu_1 = -0.05, \mu_2 = -0.1$, In this case, $c_1 = Re(a_{12})\mu_2 = -0.0391, c_2 = a_{21}\mu_1 = -0.0274$. By Theorem 3.1, Figure 3 shows $M_0$ is the only equilibrium in $D_1$, which is a sink.

(2) Choosing $\mu_1 = 0.05, \mu_2 = -0.1$, In this case, $c_1 = Re(a_{12})\mu_2 = -0.0391, c_2 = a_{21}\mu_1 = 0.0274$. By Theorem 3.1, the bifurcation occurs in $D_2$. Figure 4 shows a stable periodic orbits appears.

(3) Choosing $\mu_1 = -0.05, \mu_2 = 0.1$, In this case, $c_1 = Re(a_{12})\mu_2 = 0.0391, c_2 = a_{21}\mu_1 = -0.0274$. By Theorem 3.1, the bifurcation occurs in $D_5$. Figure 5 shows two stable equilibria appear.

4.   The existence of Hopf bifurcation

We make the following assumptions:

Lemma 4.1 If the assumption $(H_{3}), (H_{4})$ are satisfied, then all the roots of Equation (2.2) have negative real parts except a pair of purely imaginary roots when

Further, the transversality condition is satisfied at $\tau = \tau_{j}, (j = 0, 1, \cdots)$

Proof. Under the condition $(H_{3})$, the roots of $\Delta_{1} = 0$ have negative real part. We only need to consider $\Delta_{2} = 0$, then

Under the condition $(H_{4})$, separating the real and imaginary parts we have the values of $\omega$ and $\tau$ are given by

Through simple calculation, the transversality conditions are shown as follows:

Based on the work above, we can obtain the following Theorem.

Theorem 4.1 (1) The system (1.1) undergoes Hopf bifurcation when the assumption $(H_{3})$ and $(H_{4})$ are satisfied and $\tau = \tau_{j}(j = 0, 1, \cdots)$.

(2) Choosing $a = -2, b = 2.5$. By direct calculation, we obtain $\tau_0 = 1.209$, Choosing $\tau = 1 < \tau_0$, then all the oscillators 1 and 2 are stable, see Figure 6.

(3) Choosing $a = -2, b = 2.5$. By direct calculation, we obtain $\tau_0 = 1.209$, Choosing $\tau = 2 > \tau_0$, then the oscillator 1 is stable and oscillator 2 is periodic, see Figure 7.

5.   Normal form for Hopf bifurcation

In this section, Center manifold theory and normal form method ^[22] are used to study Hopf bifurcation. After scaling$t\rightarrow\frac{t}{\tau}$, system (1.1) can be written as

Denote $\tau_{c} = \tau_{j}(j = 0, 1, \cdots)$, suppose that the system (5.1) undergoes Hopf bifurcation at the critical point $\tau = \tau_{c}$, with a pair of eigenvalues $\pm i \omega \tau_{c}$, and all other roots have negative real parts. Choosing the phase space $C = C([-1, 0];R^{2})$ with supreme norm and $U_{t}$ is defined by $U_t(\theta) = U(t+\theta), -1 \leq \theta \leq 0$. Let $\mu = \tau - \tau_c$, then $\mu$ is the bifurcation parameter and the system (5.1) becomes

The linearization of system (5.2) at $(0, 0)$ is

Let

where $\delta(\theta)$ is dirac-delta function and

Define ${\begin{matrix} X = \begin{pmatrix} u_1 \\ u_2\\ \end{pmatrix} \end{matrix}}$ and ${\begin{matrix} F(X, \mu) = \begin{pmatrix} F^1 \\ F^2\\ \end{pmatrix} \end{matrix}}$, where

Then system (5.2) can be transformed into

The bilinear form on $C^{*}\times C$ is

with $\varphi\in C, \psi\in C^{*}$.

Define the infinitesimal generator $\mathcal{A}$

Let $\Lambda = (i\omega, -i\omega)$ and $P$ can be the generalized eigenspace associated with $\Lambda$ and $P^{*}$ the space adjoint with $P$. Then the $C$ can be decomposed as $C = P \oplus Q$ where $\dim p = 2$ and $Q = (\varphi\in C:(\psi, \varphi) = 0\:for\:all\:\psi \in P^{*})$. Choose the base $\Phi$ and $\Psi$ for $P$ and $P^*$ respectively such that

where $I$ is $2\times 2$ identity matrix and ${\begin{matrix} J = \begin{pmatrix} i\omega \tau_c & 0\\ 0 & -i\omega \tau_c\\ \end{pmatrix} \end{matrix}}$.

It can be computed directly that

We use the idea of Faria ^[22], Let

The elements of $BC$ can be expressed as $\psi = \phi+X_0\alpha$ with $\phi\in C$, $\alpha \in R^2$. Define the projection $\pi:BC\rightarrow P$ by

Let $u = \Phi x+y$ with $x\in R^2$ and $y\in Q^1 = \{\varphi\in Q:\dot{\varphi}\in C\}$, namely,

Let ${\begin{matrix} \Psi(0) = \begin{pmatrix} \psi_{11} & \psi_{12}\\ \psi_{21} & \psi_{22}\\ \end{pmatrix} \end{matrix}}$ = ${\begin{matrix} \begin{pmatrix} -D & D\\ -D & D\\ \end{pmatrix} \end{matrix}}$, then system(5.2) can be decomposed as

Which can be rewritten as

Where

${\begin{matrix} f_2^1(x, y, \mu) = \begin{pmatrix} \psi_{11}F_2^1(x, y, \mu)+\psi_{12}F_2^2(x, y, \mu) \\ \psi_{21}F_2^1(x, y, \mu)+\psi_{22}F_2^2(x, y, \mu) \\ \end{pmatrix} \end{matrix}}$,

${\begin{matrix} f_3^1(x, y, \mu) = \begin{pmatrix} \psi_{11}F_3^1(x, y, \mu)+\psi_{12}F_3^2(x, y, \mu) \\ \psi_{21}F_3^1(x, y, \mu)+\psi_{22}F_3^2(x, y, \mu) \\ \end{pmatrix} \end{matrix}}$,

${\begin{matrix} f_2^2(x, y, \mu) = (I-\pi)X_0 \begin{pmatrix} F_2^1(x, y, \mu) \\ F_2^2(x, y, \mu) \\ \end{pmatrix} \end{matrix}}$,

${\begin{matrix} f_3^2(x, y, \mu) = (I-\pi)X_0 \begin{pmatrix} F_3^1(x, y, \mu) \\ F_3^2(x, y, \mu) \\ \end{pmatrix} \end{matrix}}$.

with

$\frac{1}{2!} F_2^1(x, y, \mu) = -\mu y_1(0)+\mu (a+b)y_1(-1)$,

$\frac{1}{2!} F_2^2(x, y, \mu) = -\mu (x_1+x_2+y_2(0)+\mu by_1(-1)+\mu a(e^{-i\omega\tau_c}x_1+e^{i\omega\tau_c}x_2+y_2(-1)$,

$\frac{1}{3!} F_3^1(x, y, \mu) = -\frac{\tau_c(a+b)}{3}(y_1(-1))^3,$

$\frac{1}{3!} F_3^2(x, y, \mu) = -\frac{\tau_cb}{3}(y_1(-1))^3-\frac{\tau_ca}{3}(e^{-i\omega\tau_c}x_1+e^{i\omega\tau_c}x_2+y_2(-1))^3.$

Let $M_2$ denotes the operator defined in $V_2^3(C^2\times Ker\pi)$, with

where $V_2^3(C^2)$ denote the linear space of the second order homogeneous polynomials in three variables$(x_1, x_2, \mu)$, and with coefficients in $C^2$. Then it is easy to check that one may choose the decomposition

The complementary space $Im(M_2^1)^c$ spanned by the elements

Let $M_3$ denotes the operator defined in $V_3^3(C^2\times Ker\pi)$, with

where $V_3^3(C^2)$ denote the linear space of the three order homogeneous polynomials in three variables$(x_1, x_2, \mu)$, and with coefficients in $C^2$. Then it is easy to check that one may choose the decomposition

The complementary space $Im(M_3^1)^c$ spanned by the elements

Then the normal form of system (5.4) on the center manifold of the origin near $\mu = 0$ has the form (see ^[22])

where $a_{11} = D(-1+ae^{-i\omega\tau_c}), a_{12} = D(-\tau_c ae^{-i\omega\tau_c}), a_{21} = \overline{a_{11}}, a_{22} = \overline{a_{12}}$. Since $x_1 = \overline{x_2}$, through the change of variables $x_1 = w_1-iw_2, x_2 = w_1+iw_2$, and then a change to polar coordinates according to $w_1 = r\cos\xi, w_2 = r\sin\xi$, system (5.7) becomes

6.   Image contrast enhancement using the Feed-Forward neural networks

According to the property of system (1.1), neuron 2 can enhance the weak signal input by neuron 1 through coupling action. Using this property, we set up an algorithm for image enhancement:

The following are specific implementation steps of image contrast enhancement algorithm: Contrast enhancement is a phenomenon of increasing gray difference in coherent regions. The image has low contrast, does not present a clear scene, and contains no obvious objects. In a low contrast image, the pixel values (grayscale) of all pixels are very similar. This means that the difference between any two pixels of an image is small. Think of each pixel of an image as one oscillator with properties of sections 4 and 5. The standard coding of images defines a white level for $x = 1$, and a black level for $x = 0$, the other gray levels being included between these two values. The maximal values of oscillators $1, 2$ is 1, that is matching result to describe the pixels dynamics between the range [0, 1]. The input image of The tiger image is loaded (as initial conditions) in oscillator 1. Given the initial value $(x_{1i}, 0),$ where $x_{1i}$ the initial gray level of the pixel and $i$ will traverse all the pixel point.

The Figure 8 present a low-contrast tiger image, the neuron 2 has the effect of enhancing the amplitude of the signal, so it can be used to enhance the contrast of the tiger image. Choosing $a = -2, b = 2.5$, then $\tau_0 = 1.209$. Choosing $\tau = 2 > \tau_0$, then the oscillator 1 is stable and oscillator 2 is periodic, and can be use to enhance the contrast, see Figure 9.

It can be seen from the processed tiger image has a better processing effect and higher image contrast.

7.   Conclusion

In this paper, we have investigated the Hopf-pitchfork bifurcation of a simple feed forward neural network system with time delay. By analyzing the distribution of the eigenvalues of the corresponding transcendental characteristic equation of its linearized equation, we find the critical values for the occurrence of Hopf-pitchfork bifurcation. Using the normal form method and the center manifold theorem, we have derived the normal form of the reduced system on the center manifold and discussed the Hopf- pitchfork bifurcation with the parameter perturbations, and completely determined the stability and bifurcation of the zero solution near the critical value.

In this paper, we also considered the application of Hopf bifurcation in image processing. The results show that the contrast of gray image processed by oscillator 2 is enhanced. This is due to the Hopf bifurcation caused by delay. This paper presents a novel image processing method based on Hopf bifurcation. Numerical experiments show that this method has obvious advantages in processing low-contrast images. Our work is helpful in the application of the complex phenomena of feed forward neural network system.

Acknowledgments

Our deepest gratitude goes to the anonymous reviewers for their careful work and thoughtful suggestions that have helped improve this paper substantially.

Conflict of interest

The authors declare there is no conflict of interest.