On the symmetries in the dynamics of wide two-layer neural networks

1.   Introduction

We consider a family of scalar conservation laws defined on an oriented graph $\Gamma$ consisting of $m$ incoming and $n$ outgoing edges $\Omega_\ell$, $\ell = 1, \ldots m+n$ joining at a single vertex. Incoming edges are parametrized by $x\in(-\infty,0]$ while outgoing edges by $x\in[0,\infty)$ in such a way that the junction is always located at $x = 0$. We use the index $i$, $i = 1,\ldots,m$, to refer to incoming edges and $j$, $j = m+1, \ldots, m+n$, for the outgoing ones.

On the edge $\Omega_\ell$ we introduce a scalar conservation law, describing the evolution of a density $\rho_\ell$. Then on the incoming edges we have

and on the outgoing ones

The fluxes $f_1,...,f_{m+n}$, differ in general, however we assume that they are bell-shaped (unimodal), Lipschitz and non-degenerate nonlinear, i.e.

(H.1) for each $\ell\in\{1,...,m+n\}$, $f_\ell\in C^{2}\left([0,1]\right)$, $f(0) = f(1) = 0$, $f_\ell\ge0$, and there exist $\overline{\rho}_\ell\in (0,1)$ such that $f_\ell'(\rho)\; (\overline{\rho}_\ell-\rho)>0$ for every $\rho \in [0,1]\setminus \{\overline{\rho}_\ell\}$;

(H.2) for any $\ell\in\{1,...,m+n\}$, $\left\vert \left\{ \rho \: : \: f''_\ell(\rho) = 0 \right\}\right\vert = 0$.

We augment (1) and (2) with the initial conditions

assuming that

(H.3) $\rho_{1,0},...,\rho_{m,0}\in L^1(-\infty,0)\cap BV(-\infty,0)$,

$\rho_{m+1,0},...,\rho_{m+n,0}\in L^1(0,\infty)\cap BV(0,\infty)$

and $0\le \rho_{1,0},...,\,\rho_{m+n,0}\le 1$.

Finally, we introduce the necessary conservation assumption at the node, which transforms our family of independent equations into a single problem

Questions related to existence, uniqueness and stability of solutions for problems of this kind have been extensively investigated in recent years, mainly in relation with traffic modeling. The interested reader can refer to [7,13] for an overview of the subject. Here our point of view is different, as we do not focus on a specific model. We consider a parabolic regularization of the problem, similarly to what has been done in [11,10], but instead of enforcing a continuity condition at the node for the regularized solutions, we introduce a more general set of transmission conditions on the parabolic fluxes.

In this work we adopt the following definition of weak solution for the problem (1), (2), and (3). We stress that this definition is for sure not sufficient to ensure uniqueness. On the contrary it fix somehow a minimal set of properties that any reasonable solution is expected to satisfy, see [3] and references therein for a more detailed discussion on this point.

Definition 1.1. Let $\rho_1,...,\rho_m:[0,\infty)\times(-\infty,0]\to \mathbb{R}$ and $\rho_{m+1},...,\rho_{m+n}:[0,\infty)\times[0,\infty)\to \mathbb{R}$ be functions. We say that $(\rho_1,....,\rho_{m+n})$ is a weak solution of (1), (2), and (3) if

(D.1) $f_1(\rho_1),...,f_m(\rho_m)\in BV_{loc}((0,\infty)\times(-\infty,0))$ and $f_{m+1}(\rho_{m+1}),..., f_{m+n}(\rho_{m+n})\in BV_{loc}((0,\infty)\times(0,\infty))$;

(D.2) for every $i\in\{1,...,m\}$, every $c\in \mathbb{R}$ and every nonnegative test function $\varphi\in C^\infty( \mathbb{R}\times (-\infty,0))$ with compact support

(D.3) for every $j\in\{m+1,...,m+n\}$, every $c\in \mathbb{R}$ and every nonnegative test function $\varphi\in C^\infty( \mathbb{R}\times (0,\infty))$ with compact support

(D.4) $\sum\limits_{i = 1}^m f_i( \rho_i(t,0-)) = \sum\limits_{j = m+1}^{m+n}f_j( \rho_j(t,0+))$ for a.e. $t\ge0$.

In [10] the authors approximated (1), (2), and (3) in the following way

where $i\in\{1,...,m\}$ and $j\in\{m+1,...,m+n\}$ and $\rho_{i,0, \varepsilon},\, \rho_{j,0, \varepsilon}$ are smooth approximations of $\rho_{i,0},\, \rho_{j,0}$. In this setting they showed that

where $(\rho_1,....,\rho_{m+n})$ is a weak solution of (1), (2), (3), in the sense of Definition 1.1.

In this paper we modify the transmission condition of (4) and inspired by [14] we consider the following viscous approximation of (1), (2), and (3)

where, of course,

The additional assumptions we make on the functions $\beta_\ell$ and on the initial conditions $\rho_{\ell,0, \varepsilon}$ are postposed to the next section.

The main result of the paper is the following.

Theorem 1.2. Assume(H.1), (H.2), and (H.3). There exist a sequence $\{ \varepsilon_k\}_{k\in \mathbb{N}}\subset(0,\infty)$, $\varepsilon_k\to 0$, and a solution $(\rho_{1}, \ldots, \rho_{m+n})$ of (1), (2), and (3), in the sense of Definition 1.1, such that

for every $1\le p<\infty,\, i\in\{1,..,m\},\,j\in\{m+1,..,m+n\},$ where $( \rho_{1, \varepsilon_k},..., \rho_{m+n, \varepsilon_k})$ is the corresponding solution of (5).

It worth mentioning that a complete characterization of the limit solution obtained from (4) as $\varepsilon\to0$ is given in [3], where the authors adapt to a star shaped graph setting some ideas and techniques originally developed for conservation laws with discontinous flux, see in particular [2,4,5].

At the moment we are not able to formulate a similar characterization of the limit of (5). In general, however, the limits coming from parabolic regularization subject to the two different kinds of transmission conditions are different.

To show this consider the simple case of a junction with one incoming and one outgoing edges. So we have the conservation law

on the incoming edge and

on the outgoing one. Assume that

Consider the simplified version of (5)

The unique solution of (13) is

Therefore, as $\varepsilon\to0$ we get the solution of (10)-(11)

This stationary solution is not admissible in the sense of the classical vanishing viscosity germ, see [5,Sec. 5], as it consists of a nonclassical shock. However, when dealing with conservation laws with discontinuous flux, it is well known that infinitely many $L^1$ contractive semigroups of solutions exist, also in relation with different physical applications. In particular, when the right and left fluxes are bell-shaped, as we assume in condition (H.1), each of those notions of admissible solution is uniquely determined by the choice of a $(A,B)$-connection, see [1,5,9,12] for precise definitions and exemples. In the exemple above the couple $(\hat\rho, \check\rho)$ is a connection.

It is worth noticing that entropy solutions admissible in the sense of a $(A,B)$-connection can be obtained as limits of a sequence of parabolic approximations made with adapted viscosities but a classical condition of continuity at the interface, see [5,Sec. 6.2] for a general result, but also [2,15] for an application to the Buckley-Leverett equation.

It is difficult, however, to establish a direct equivalence between the aforementioned results and the one we put forward in this paper. In particular, in the present case we miss information on the boundary layers at the parabolic level and we do not know how the transmission conditions we impose on the parabolic fluxes translates into a condition for the hyperbolic problem.

This means in particular that we have little information on the germ associated to the family of limit solutions obtained in Theorem 1.2 and, so far, we have not been able to prove that this germ is $L^1$-dissipative. We conjecture, however, that this is due to a technical obstruction and that uniqueness of the limit solutions holds.

The paper is organized as follows: Section 2 contains the precise list of assumptions on the initial and transmission conditions in the parabolic problem (5). In Section 3 we present the proofs of all necessary a priori estimates on (5). Finally, in Section 4 we detail the proof of Theorem 1.2.

2.   Initial and transmission conditions for the parabolic problem

The initial conditions $\rho_{\ell, 0}$, $\ell = 1,\ldots, m+n$, on the hyperbolic problem (1), (2), and (3) satisfy (H.3).

Once the functions $\rho_{\ell, 0}$ are fixed, we impose on (5) initial conditions $\rho_{\ell,0, \varepsilon}$ such that

for some constant $C>0$ independent on $\varepsilon$.

The functions $\beta_\ell$ appearing in the transmission conditions in (5) take the form

for $i\in\{1,\ldots, m\}$, and for $j\in\{m+1,\ldots, m+n\}$

The functions $G_{i,j}(u,v)\in C^\infty( \mathbb{R}^2)$, $i\in\{1,\ldots, m\}$, $j\in\{m+1,\ldots, m+n\}$, and $K_{h,\ell}(u,v)\in C^\infty( \mathbb{R}^2)$, $h,\,\ell\in\{1,\ldots, m+n\}$, satisfy

In particular, (19) implies

where $\chi_{(-\infty,0)}$ is the characteristic function of the set $(-\infty,0)$.

This specific form of transmission conditions is reminiscent of the parabolic transmission conditions considered in [14,8], which were originally inspired from the Kedem-Katchalsky conditions for membrane permeability introduced in [16]

for some constants $\mathfrak{c}_{h,\ell}>0$. Our conditions are more general and in particular we can notice that the function ${\mathcal{G}_{h,\ell}}$ above satisfies

that allows the authors in [14] to get the $L^2$ conservation (see Lemma 3.3 below).

We can observe that the equality (6) holds as

and analogously

3.   A priori estimates

This section is devoted to establish a priori estimates, uniform with respect to $\varepsilon$, which are necessary toward the proof of our main convergence result in the next section.

For every $\varepsilon > 0$, let $(\rho_{1, \varepsilon},...,\rho_{m+n, \varepsilon})$ be a solution of (5) satisfying (16).

Lemma 3.1 ($L^\infty$ estimate). We have that

Proof. Consider the function

Since

using (19) we obtain

where $\delta_{\{ \rho_{i,\varepsilon} = 0\}}$ and $\delta_{\{ \rho_{j,\varepsilon} = 0\}}$ are the Dirac deltas concentrated on the sets $\{ \rho_{i,\varepsilon} = 0\}$ and $\{ \rho_{j,\varepsilon} = 0\}$, respectively and we apply [6,Lemma 2] to compute the value of the integrals as a limit. Integrating over $(0,t)$ and using (16) we get

and then

that proves the lower bounds in (25). The upper bounds in (25) can be proved in the same way using the function $\xi\mapsto (\xi-1)\chi_{(1,\infty)}(\xi)$.

Lemma 3.2 ($L^1$ estimate). We have that

Proof. Thanks to (5), (23), (24), and (25), we have that

Integrating over $(0,t)$ and using (16) we get (26).

Lemma 3.3 ($L^2$ estimate). We have that

for every $t\ge0$.

Proof. Thanks to (5), we have that

Integrating over $(0,t)$ and using (16) we get (27).

Lemma 3.4 ($BV$ estimate). We have that

for every $t\ge0.$

Proof. From (5) we get

Thanks to (20), we have that

where $\delta_{\{ \partial_t \rho_{i,\varepsilon} = 0\}}$ and $\delta_{\{ \partial_t \rho_{j,\varepsilon} = 0\}}$ are the Dirac deltas concentrated on the sets $\{ \partial_t \rho_{i,\varepsilon} = 0\}$ and $\{ \partial_t \rho_{j,\varepsilon} = 0\}$, respectively and we apply [6,Lemma 2].

Integrating over $(0,t)$ and using (16), (25) we get

that is (28).

Lemma 3.5 (Stability estimate). Let $(\rho_{1, \varepsilon},...,\rho_{m+n, \varepsilon})$ and $(\overline{\rho}_{1, \varepsilon},...,\overline{\rho}_{m+n, \varepsilon})$ be two solutions of (5). The following estimate holds

Proof. From (5) we get

Thanks to (5), (20), and (25), we have that

where we use [6,Lemma 2] and we denote by $\delta_{\{ \rho_{i,\varepsilon} = \overline{\rho}_{i,\varepsilon}\}}$ and $\delta_{\{ \rho_{j,\varepsilon} = \overline{\rho}_{j,\varepsilon}\}}$ respectively the Dirac deltas concentrated on the sets $\{ \rho_{i,\varepsilon} = \overline{\rho}_{i,\varepsilon}\}$ and $\{ \rho_{j,\varepsilon} = \overline{\rho}_{j,\varepsilon}\}$.

Integrating over $(0,t)$ we get (29).

4.   Proof of Theorem 1.2

The well-posedness of smooth solutions for (5) can be proved following the argument used in [10,Theorem 1.2] to establish the well-posedness of smooth solutions for (4). Indeed, the existence of a linear semigroup of solutions in the linear case (i.e., when $f_\ell\equiv 0$) is shown in [14]. Then the Duhamel Formula, estimates similar to the ones in the previous section and a fixed point argument lead to the result.

The main result of this section is the following.

Lemma 4.1. Let $( \rho_{1, \varepsilon},...,\rho_{m+n, \varepsilon})$ be the solution of (5). There exist a sequence $\{ \varepsilon_k\}_{k\in \mathbb{N}} \subset (0,\infty),\, \varepsilon_k\to0$, and $m+n$ maps $\rho_{1},...,\rho_{m+n}$ such that

for every $1\le p<\infty,\, i\in\{1,..,m\},\, j\in\{m+1,..,m+n\}.$ Moreover, we have that

Thanks to the genuine nonlinearity of $f_1,...,f_{m+n}$, we can use the Tartar compensated compactness method [18] to obtain strong convergence of a subsequence of viscosity approximations. The notation $\mathfrak{R}$ can stand for $(0,\infty)$ or $(-\infty, 0)$.

Theorem 4.2 (Tartar). Let $\{v_\nu\}_{\nu>0}$ be a family of functions defined on $(0,\infty)\times\mathfrak{R}$ such that

and the family

is compact in $H^{-1}_{loc}((0,\infty)\times\mathfrak{R})$, for every convex $\eta\in C^2( \mathbb{R})$, where $q_\ell' = f_\ell'\eta'$. Then there exist a sequence $\{\nu_n\}_{n\in \mathbb{N}}\subset(0,\infty),\,\nu_n\to 0,$ and a map $v\in L^\infty((0,T)\times\mathfrak{R}),\,T>0,$ such that

The following compact embedding of Murat [17] is useful.

Theorem 4.3 (Murat). Let $\Omega$ be a bounded open subset of $\mathbb{R}^N$, $N\ge 2$. Suppose the sequence $\left\{{\mathcal L}_n\right\}_{n\in \mathbb{N}}$ of distributions is bounded in $W^{-1,\infty}(\Omega)$. Suppose also that

where $\left\{{\mathcal L}_{1,n}\right\}_{n\in \mathbb{N}}$ lies in a compact subset of $H_{\mathrm{loc}}^{-1}(\Omega)$ and $\left\{ {\mathcal L}_{2,n}\right\}_{n\in \mathbb{N}}$ lies in a bounded subset of $L^1_{loc}(\Omega)$. Then $\left\{{\mathcal L}_n\right\}_{n\in \mathbb{N}}$ lies in a compact subset of $H_{\mathrm{loc}}^{-1}(\Omega)$.

Proof of Lemma 4.1. Let us fix $i\in\{1,...,m\}$ and prove the lemma for the incoming edges, as the proof for the outgoing ones is analogous.

Let $\eta: \mathbb{R}\to \mathbb{R}$ be any convex $C^2$ entropy function, and let $q_i: \mathbb{R}\to \mathbb{R}$ be the corresponding entropy flux defined by $q_i' = \eta'f_i'$. By multiplying $i-$th equation in (5) by $\eta'( \rho_{i,\varepsilon})$ and using the chain rule, we get

We claim that

Indeed, (25) and (27) imply

Due to (16), (39) follows. Therefore, Theorems 4.3 and 4.2 give the existence of a subsequence $\{ \rho_{i,\varepsilon_k}\}_{k\in \mathbb{N}}$ and a limit function $\rho_i$ satisfying (30) such that as $k\to \infty$

that guarantees (32) and (33).

Finally, thanks to Lemmas 3.2, 3.3, and 3.4 we have (35), (36), and (37).

Proof of Theorem 1.2.. The first part of the statement related to the convergence of vanishing viscosity approximations has been proved in Lemma 4.1.

Let us fix $i \in \{ 1, \ldots, m \}$ and prove (9) for the incoming edges, the case of the outgoing ones is analogous.

Thanks to (3.4) and (33), for all $\varphi \in C^\infty((0,\infty)\times(-\infty,0))$ with compact support, we have

therefore

where $\mathcal{M}((0,\infty)\times(-\infty,0))$ is the set of all Radon measures on $(0,\infty)\times(-\infty,0).$ Moreover, from the equations in (1) and (2) we have also

Clearly (41) and (42) give (9) and so the trace at the junction $f(\rho_{i}(t,0-))$ exists for a.e. $t > 0$.

We prove now that the identity

holds for a.e. $t > 0$; consequently the functions $\rho_1,\ldots, \rho_{m+n}$ provide a solution to (1), (2), and (3) in the sense of Definition 1.1.

Let $\varphi \in C^1([0,\infty)), \, \varphi(0) = 0$ with compact support. Consider the sequence $\{r_\nu\}_{\nu\in \mathbb{N}\setminus\{0\}}\subset C^2([0,\infty))$ of cut-off functions satisfying

for every $x \ge 0$ and $\nu \ge 1$. Moreover, for every $\nu \ge 1$, we define the sequence $\{\tilde r_\nu\}_{\nu\in \mathbb{N}\setminus\{0\}}\subset C^2((-\infty,0])$ by writing $\tilde r_\nu(x) = r_\nu(-x)$ for every $x \le 0$.

From (5) we have that

As $k\to\infty$, due to (27), (33), and (34),

Finally, sending $\nu\to \infty$,

that gives (43).