A sigmoidal fractional derivative for regularization

1.   Introduction

Fractional calculus has undergone significant developments in recent years and has found use in physics, engineering, economics, etc ^[1,2,3]. Classical results about the Riemann-Liouville and Caputo derivatives as well as fractional differential equations can be found in ^[4,5]. In ^[11] and ^[33], Caputo and Fabrizio suggested a new fractional derivative, whose properties were investigated by Losada and Nieto ^[15]. This fractional derivative was utilized in various applications, including the fractional Nagumo equation in Alqahtani et al. ^[23], coupled systems of time-fractional differential problems in Alsaedi et al. ^[24] and Fischer's reaction-diffusion equation in Atangana et al ^[25]. More applications of the Caputo-Fabrizio fractional derivative can be found in Aydogan et al ^[26]. and Atangana et al ^[27].

For $0 \leq \alpha \leq 1$, $-\infty < a < t$, $f \in H^{1}(a, b)$ and $b > a$, the Caputo fractional derivative is defined by

By replacing the term $\frac{1}{\Gamma(1-\alpha)}$ with the normalization constant $M(\alpha)$ such that $M(0) = M(1) = 1$ and adjusting the kernel $(t-s)^{-\alpha}$, we obtain the Caputo-Fabrizio fractional derivative defined by

The Caputo-Fabrizio fractional derivative of a constant vanishes as does the usual Caputo derivative, however the new kernel $\exp \big(\frac{-\alpha}{\alpha - 1})$ is no longer singular for $s = t$. Caputo and Fabrizio try to extend their definition in ^[11] to functions in $L^{1}$ by

Algahtani et al. ^[23] show that the nonlinear Nagumo equation given by

where $0 < \alpha < 1$ and $\beta, \gamma, \delta$ are constant, subject to the boundary conditions

has an exact solution. The authors show that this PDE can be reformulated in terms of a Lipschitz kernel. Existence of the exact solution is shown using a fixed point approach and uniqueness is provided, given that suitable assumptions are made about the Lipschitz constant. Their study claims that an exponential kernel is in some sense a better kernel than a power function, since the lack of a singularity provides a better filtration effect. In the context of fractional differential equation applications, since the associated functions are not defined in a Banach space, only approximate solutions to certain fractional differential equations can be investigated. The methods used to handle fractional differential problems such as ${^{CF}}D^{\alpha}f(t) = g(t, f(t))$, cannot be extended to the problems resembling ${^{CF}}D^{\alpha}f(t) = g(t, f(t), {^{CF}}D^{\alpha}f(t)$).

In Baleanu et al ^[14], the Caputo-Fabrizio fractional derivative on the Banach space $C_{\mathbb{R}}[0, 1]$ is considered in the context of higher order series-type fractional integrodifferential equations. More precisely, an extended Caputo-Fabrizio type fractional derivative is provided of order $0 \leq \alpha < 1$ on $C_{\mathbb{R}} [0, 1]$ for $b > 0$ by

These authors use a standard fixed point approach to establish uniqueness of solutions to fractional series-type differential problems such as

with initial condition $f(0) = 0$ and $\alpha, \gamma, \delta, \rho \in (0, 1)$.

An extension of this type which is compatible with orders beyond $(0, 1)$ has yet to be provided.

The Caputo-Fabrizio fractional derivative is discussed in the setting of distributions in ^[28]. Other types of fractional derivatives can be found in Katugampola ^[22] and Oliveira et al ^[6]. In de Oliveira ^[12], it is shown that the choice of kernel in a Caputo-type fractional derivative is connected to the Laplace transform via convolution.

Let $\mathscr{I}$ denote the Schwarz class of smooth test functions whose derivatives decay at infinity. Moreover, let $\mathscr{I}'$ denote the space of continuous linear functionals on $\mathscr{I}$. The distributional derivative

{T'} is defined as in ^[32]

for all smooth compactly supported test functions $\phi$ on $\mathbb{R}$.The distributional Laplace transform is given by

where $s = \sigma + i\mu$, $\mu < 0$ and $\phi(t)e^{-\sigma t} \in \mathscr{I}'$. Suppose that $f$ is supported on $(0, \infty)$ such that $\sigma > 0$ and $f(t)e^{-\sigma t} \in \mathscr{I}'$. It follows that the Laplace transform of the derivative is given by

Let $\mathscr{L}$ denote the distributional Laplace transform defined by

One can define a more general fractional derivative as follows. Suppose that $\Phi(s, \alpha)$ is a fractional integrodifferential operator and $K(t, s): \mathbb{R}^2 \to \mathbb{R}$ is a continuous kernel. Let the corresponding operator $\phi(s, \alpha)$ be defined for some fractional derivative $D^{\alpha}$ such that

where $\Phi(s, 1) = s, \Phi(s, -1) = \frac{1}{s}$ and $\Phi(s, 0) = 1$. Then, letting $\Phi(s, \alpha) = s\mathscr{L}(K(s, t, \alpha))$. Proceeding with the Convolution Theorem, we are left with a Caputo-type fractional operator of the form

which is dependent on the choice of kernel $K$. For $f \in H^{1}(a, b)$, and $n \in \mathbb{N}$, we can spot commonly used kernels such as the Caputo kernel $K_{1} = \frac{1}{\Gamma(1-\alpha)} (t-s)^{{\left\lceil \partial \right\rceil } - \alpha - 1}$, the Caputo-Fabrizio kernel $K_{2} = \frac{M(\alpha)}{1-\alpha} \exp(\frac{-\alpha (t-s)}{\alpha-1})$ and the Gaussian kernel $K_{3} = \frac{1}{\sqrt{2\pi \sigma^2}} \exp(\frac{-t^2}{2\sigma^2})$ ^[4,10,13].

The memory principle for fractional derivatives describes the history of $f(t)$ near the terminal point $t = a$. Let $L$ denote the memory length, satisfying $a+L \leq t \leq b$. Define the error in approximating the fractional derivative by

where $_{a} D_{K} ^{\alpha} f(t)$ is as in (5). If $f'(t) \leq M$ for $a < t < b$ and $0 < \alpha < 1$, we have the following error estimate for the Caputo fractional derivative

For all $\epsilon > 0$, if $E_{L, \alpha, a} (t) \leq \epsilon$ with $a+L \leq t \leq b$, we have

Therefore, the Caputo fractional derivative with terminal $a$ can be approximated by the corresponding fractional derivative with lower limit $t-L$, with the level of accuracy described above.

In this work, we propose a different fractional derivative that has a smooth kernel. Our primary interest in defining this fractional derivative is the improvement of machine learning algorithms. Caputo-type fractional derivatives have been applied in machine learning, such as in Pu et al ^[10]. In particular, fractional order gradient methods have been considered in order to improve the performance of the integer order methods. For example, suppose that $f: \mathbb{R}^{n} \to \mathbb{R}$ is convex and differentiable with a Lipschitz gradient, then the integer order gradient method defined by

has a linear convergence rate. Improving the performance of the integer-order gradient method is critical in optimization problems. In recent literature, fractional calculus has been thought to improve the integer order gradient method due to nonlocality and the memory principle. Fractional order gradient methods have been proposed based on the Caputo fractional derivative that offer competitive convergence rates. For example, in ^[19], a Caputo fractional gradient method is proposed that is shown to be monotone and exhibit strong convergence.

Fractional derivatives were used in the backpropagation algorithm for feedforward neural networks and convolutional neural networks in ^[20,31]. In both studies, the rate of convergence was shown to exceed the rate of integer-order methods. Fractional-order methods have been used to investigate complex-valued neural networks in ^[17] and recurrent neural network models in ^[30]. In ^[19] and ^[16], gradients based on the Caputo fractional derivative are used to update parameters while integer order gradients are used to handle backpropagation allowing for simpler computation. The experiments therein are shown to improve the accuracy of the neural network's performance compared to integer-order methods while being equally costly.

In the training of machine learning models, one often needs to obtain weights of the features which optimize the training data. In the case of maximum likelihood training, regularization is typically needed so that the model does not overfit the training data. In $\ell_p$ regularization, the weight vector is penalized by its $\ell_p$ norm. While the case for $p = 1$ and $p = 2$ are very common and result in similar levels of accuracy, $\ell_1$ regularization is much more practical. Due to its sparsity, $\ell_1$ regularization is less memory intensive and more time-effective than $\ell_2$ regularization. On the other hand, $\ell_1$ regularization is problematic in that during the update process, the gradient of the regularization term is not differentiable at the origin as the error function given below

has classical derivative

A typical remedy to this problem is to use the stochastic gradient descent method, which approximates the gradient using the training data. Although time efficient for training, when the dimension of the feature space is large, the update process slows down significantly. Furthermore, the model becomes less sparse after training the data. The discontinuity induced by the regularizer proves to be problematic as it adjusts the direction of descent. The use of sigmoids in regularization problems has been previously explored as in Krutikov ^[29], but not in the context of fractional derivatives. Another remedy to the aforementioned problem is the use of fractional gradients over the classical descent methods. These methods are still in their infancy and problematic in that convergence to the local optimum is not always guaranteed, even when the algorithm converges. Furthermore, these methods often require an adjustment to the fractional derivative by truncation as in ^[9], variable order techniques as in ^[18], and methods based on the memory principle (6) due to the computational expense and the failure of the Caputo kernel to be smooth.

We would also like our operator to be nonlocal. In ^[13], it is shown that unlike the Caputo derivative, the Caputo-Fabrizio fractional derivative is not a nonlocal operator. The linear fractional differential equation

is shown to reduce to a first-order ordinary differential equation. This means that the Caputo-Fabrizio derivative cannot sufficiently describe processes with nonlocality and memory. With the correct choice of kernel, this complication can be avoided.

2.   Main results

In this section, we define a new left-sided fractional derivative. We show that the proposed fractional derivative reduces to the $H^{1}$ derivative as the order approaches 1. In the results to follow, for $0 < \alpha \leq 1$, we will let $C_1(\alpha)$ denote a normalization constant $\frac{C(\alpha)}{\Gamma(2-\alpha)}$ satisfying $C(\alpha)\Gamma(1-\alpha) \to \frac{1}{2}$ as $\alpha \to 1^{-}$.

Definition 2.1. (Left sigmoidal fractional derivative) Let $0 < \alpha \leq 1$, $f \in H^{1}((a, b))$, $t > a$ and $\{ f(t) \}'$ denotes the $H^{1}$ distributional derivative as in (4). We define a new fractional derivative by

Now, we show that the left sigmoidal fractional derivative reduces to the $H^{1}$ derivative.

Theorem 2.1. (Reduction to classical derivative) Suppose $f \in H^{1}(a, b)$, then

Proof.

where the last result follows from the observation that

$\delta(t)$ is the Dirac distribution.

In the following theorem, we show that this left sigmoidal fractional derivative is commutative with respect to the classical derivative.

Theorem 2.2. Suppose that $f$ is at least twice continuously differentiable and ${^{\sigma}} D^{\alpha}_{a} f(t)$ is differentiable. If $f'(a) = 0$, then

where $0 < \alpha < 1$.

Proof. From (8), integrating by parts yields

so we have

appealing to the Leibniz integral rule

From (11) and (12), the desired result is obtained.

In the next theorem, we show that the left sigmoidal fractional derivative does not satisfy the memory principle in the sense of (6). More precisely, the next theorem implies that we show that the left sigmoidal fractional derivative can be approximated by the corresponding fractional derivative with lower limit $t-L$ with increased accuracy for orders in which $C_1(\alpha)$ is large.

Theorem 2.3. (Memory principle) Suppose that $f$ is differentiable on $(a, b)$, $a+L \leq t \leq b$ and $0 < \alpha < 1$. For every $\epsilon > 0$, if there exists $C_0 > 0$ such that $f'(t) \leq C_0$, then

Proof. Making use of the inequality

we have

and the result follows.

In the theorem below, we show that our new fractional derivative provides a sigmoidal approximation to functions that have a piecewise linear $H^{1}$ distributional derivative. For instance, the proposed left sigmoidal fractional derivative is compatible with $\ell_{1}$-regularization. In the case of the $\ell_{1}$ norm, it can be used to define a fractional gradient, which approximates its classical gradient via a family of sigmoids as $\alpha$ approaches 1. This is promising in the context of gradient descent algorithms.

Theorem 2.4 (Norm-1 compatibility) ${^{\sigma}} D^{\alpha}_{a}$ provides a smooth approximation to the $\ell_{1}$ norm defined by

as $\alpha \to 1$ in the sense that for the error function $E$ given in (7), ${^{\sigma}} D^{\alpha}_{a} E_{\ell_{1}}(x_j)$ is given by

where $a > 0$.

Proof. The result follows from the observation that

where $H(t)$ is the Heaviside function.

Theorem 2.5. (Mittag-Leffler function). Suppose that $\gamma, \eta > 0$ and $0 < a < t$. Then

where $E_{\gamma, \eta} (z) = \sum\limits_{k = 0}^{\infty} \frac{z^{k}}{\Gamma(\gamma k + \eta)}$ is the two-parameter Mittag-Leffler function

Proof.

Theorem 2.6. Suppose that $f \geq 0$, $1 < p < \infty$, $0 < \alpha < 1$ and $0 < t \leq T$. If $f \geq 0$ is differentiable with $f' \in L^{p}(\mathbb{R})$ and $M$ is the maximal operator of $f$ given by

then

(a) ${^{\sigma}} D^{\alpha}_{-t} f(t) \leq 2TC_1(\alpha)M(|f'|)(0)$

(b) ${^{\sigma}} D^{\alpha}_{a} f(t)$ is integrable on $\mathbb{R}$.

Proof. (a) Since

(b) From Young's convolution inequality, $\left\|{ f \star g }\right\|_{L^{r}} \leq \left\|{f}\right\|_{L^{p}} \left\|{g} \right\|_{L^{\frac{pr}{p + r(p-1)}}}$.

The next theorem describes the effect of the Laplace and Fourier transforms, which can extend to distributions as in de Oliveira ^[6]. The Convolution Theorem connects our choice of kernel as in (5) via the operator $\Phi(s, \alpha) = s\mathscr{L} (K(s, t, \alpha))$. In this case, $\Phi(s, \alpha)$ depends on the digamma function $\Psi(z) = \frac{\Gamma'(z)}{\Gamma(z)}$. This shows that the left-sigmoidal fractional derivative does not reduce to the left-sided Riemann-Liouville fractional derivative.

Theorem 2.7. (Transformations) Suppose that $0 < \alpha < 1$, $Re(s) > 0$, $\omega \in \mathbb{R}$, $a \in \mathbb{R}$ and $f$ is a differentiable function of exponential order such that $f(0) = 0$. If $T_1(s), T_2(\omega)$ are defined by

then

(a) $\mathscr{L}({^{\sigma}} D^{\alpha}_{0} f(t))(s) = C_1(\alpha)(s(\alpha - 1))^2 {T_1\big({(\alpha - 1)s} \big) \mathscr{L}(f)(s) }$

(b) $\mathscr{F}({^{\sigma}} D^{\alpha}_{0})(\omega) = -C_1(\alpha) \omega^2 | {\alpha-1}|(\alpha-1) {T_2\big((\alpha - 1)s} \big) \mathscr{F}(f)(\omega),$

where $\mathscr{L}(f)(s)$ denotes the Laplace transform of $f$ and $\mathscr{F}(f)(\omega)$ denotes the Fourier transform of $f$.

Proof. (a) follows from a standard application of the Convolution theorem. By using the dilation property $\mathscr{L}(f(at)) = \frac{F(\frac{s}{a})}{a}$, we have

The transform $\mathscr{L}(\tanh t)$ is handled as follows

Because of the absolute convergence of the monotone decreasing sum $\sum\limits_{k = 0}^{\infty} (-1)^ke^{-2kt} dt$ and the nondecreasing nature of its partial sums, we can exchange integration and summation using the Lebesgue Monotone Convergence Theorem. Continuing, we have

The identity

used above comes from the Lerch transcendent, defined by

where $|z| < 1$, $a \neq 0, -1, -2, ...$ and using the dilation property once more, the result follows.

(b) We proceed as in (a).

To finish the proof, we recall the result

Theorem 2.8. Suppose that $f$ is differentiable and $0 < \alpha < 1$. Then

Proof. Using the inequality

we have

which results in the leftmost inequality. Noticing that $\cosh^2{x} \geq 1 + x^2$, we have that

finishing the last three inequalities.

Theorem 2.9. The problem

has the solution

where $G(t) = \int_{0}^{t} g(s) ds$.

Proof. Differentiating the differential equation above, the problem above reduces to

which can be integrated to obtain the result.

Theorem 2.10. Let $0 < \alpha < 1$ and let $g: (a, b) \times \mathbb{R}^2$ be a continuous function such that there exists a constant $C_0 > 0$ satisfying

for all $t \in (a, b)$ and $x_1, x_2, y_1, y_2 \in \mathbb{R}$ and $|(\alpha -1)C(\alpha)C_0| < 1$. Then, the problem

has a unique solution.

Proof.

Since $(\alpha -1)C(\alpha)C_0 < 1$, the map $F: H^{1}(a, b) \to H^{1} (a, b)$ defined by

is a contraction. By the Banach fixed-point theorem, it has a unique fixed point, finishing the proof.

We note that this result is advantageous in that the analogous existence and uniqueness result as in fractional differential systems defined by the Caputo derivative is highly dependent on initial conditions imposed on the primary function of interest and its classical derivatives ^[4].

We now shift our attention to a gradient descent method. Suppose that $f(x)$ has a bounded derivative and unique critical point $t^*$ such that $f'(t^*) = 0$. For $a \leq t \leq b$, $0 < \alpha < 1$, define the scalar left sigmoidal fractional gradient descent method by

where $0 < \mu < 1$ is the learning rate.

Theorem 2.11 (Fractional Gradient Descent). Let $f$ be as in (13). Then, the left-sigmoidal fractional-order gradient method (13) converges to the true critical point $t^{*}$.

Proof. Denote the Lipschitz constant of $f$ by $L$. For $k \geq N$,

Repeating this process, it follows that the $t_k$ form a Cauchy sequence, guaranteeing convergence. To show that the sequence converges to the critical point, suppose for contradiction that the sequence $(t_k)_{k = 0}^{\infty}$ converges to a point $\hat{t} \neq t^{*}$. Then, for every $\epsilon > 0$, there exists $N \in \mathbb{N}$ such that for all $k \geq N$, $|f'(t_k)| > 0$ and

As a consequence of $(13)$ and Theorem 2.8, we have

where

On the other hand, we have the inequality

Choosing $\epsilon < \frac{1}{2(M_{1} M_{2})^{2}}$ yields $M_1 M_2 > |t_{k+1} - t_{k}|^{\frac{-1}{2}}$, which implies that $|t_{k+1} - t_{k}| > |t_{k} - t_{k-1}|$, contradicting the assumption that the sequence $(t_{k})$ is convergent.

3.   Conclusion

In this paper, we defined a new sigmoidal fractional derivative, which is compatible with certain weakly differentiable functions. We showed that this fractional derivative satisfies some forms of classical properties and is compatible with the $\ell_{1}$ norm by a sigmoidal approximation. For further research, we will investigate this operator in optimization and machine learning. We note that the left-sigmoidal fractional derivative can be applied in the context of gradient descent, which has applications in optimization and machine learning ^[7,8]. Recently, backpropagation and convolution neural networks have been studied in the context of fractional derivatives, typically of the Caputo-type are being used for gradient descent. This idea is still novel and needs to see improvements. For example, the gradient descent method has been handled by Sheng et al. ^[20,21], Wang et al. ^[19], Wei et al. ^[9] and Bao et al ^[16]. These methods are still early in development. The following topics still need to be fully addressed: convergence to an extreme point, extending the available range of fractional order, more complicated neural networks, loss function compatibility and the usage of the chain rule.

Conflict of interest

The authors declare that there is no conflict of interest.