We investigate the neural dynamics in an exicitatory-inhibitory neural nerwork (E-I network) with ReLU activation and Hebbian learning. By local bifurcation analysis, we find there are ten possible neural dynamical patterns between one excitatory neuron and one inhibitory neuron with all four types of synaptic connections. The unstable node never exists in this system, and when there is no external input, Hebbian learning makes the network dynamic converge to a stable star. We give sufficient and necessary conditions for the exisitence of limit cycles in an E-I network.

E-I Networks

An E-I nerwork consists of two populations of neurons: the excitatory neurons (E) and the inhibitory neurons (I). The E-neurons encourage the firing of the post-synaptic neurons, while the I-neurons resist the firing of the post-synaptic neurons. There are four types of synaptic connections: E-E, E-I, I-E, I-I, with different synaptic weights. We constrain all E-E and I-I connections to be symmetric, and E-I and I-E connections to be anti-symmetric. The weights and neural activity are non-negative. The change of weights is subject to Hebb’s rule. See the general formuation in the above illustration.

Summary

  • There are ten possible dynamical patterns for the neural activity of linear neural network by investigating the local bifurcation.
    • All the possible spirals and circles are anticlockwise.
    • The unstable node never exist.
  • If there is no external input, in the long term, the fixed point of linear network becomes a stable star.
  • The positive fixed point of a linear neural network is also a positive fixed point of its corresponding rectified linear neural network, vise versa.
  • There are ten possible dynamical patterns for the neural activity of rectified linear neural network:
    • :
      • a. if , i.e., , the fixed point is still an unstable spiral, but there exist one stable limit cycle.
      • b. if , i.e., , the fixed point is still a center of circles, but there exist one stable limit cycle.
      • c. if , i.e., , the fixed point is still a stable spiral.
    • :
      • a. if , i.e., , the fixed point is an unstable star.
      • b. if , i.e., , the fixed point is a stable star.
    • :
      • a. if , i.e., ,the original fixed point is a saddle point, but there is one boundary stable node.
      • b. if , i.e., , we can show that , then the fixed point is a stable node.
    • : There is no isolated fixed points.
      • a. if , i.e., , there is a line repeller.
      • b. if , i.e., , there is a line attractor, and one stable node on the boundary.
      • c. if , i.e., , there is no fixed points.
  • In a rectified linear neural network, if we add two time constants and , a limit cycle exists if and only if all the following conditions hold:
      (1) a positive fixed point exists;
      (2) ; and
      (3) .
    The behavior of the system when varying is similar to supercritical Hopf bifurcation.
  • When the inhibition is much faster than the excitation, i.e., , it is impossible to produce limit cycles in an E-I network.

Analysis

CASE 1. the simplest case , w/o nonnegative connenction constraints in network dynamics.

Fast Movement (Neural Activity)

The dynamics can be expressed as

where , , and . Then the trace of Jacobian , the determinant of Jacobian . Assume . Using Cramer’s rule, we can rewrite this linear dynamic system as

The external input control the position of fixed point

The discriminant of this linear system is .

Thus, when

  • :
    • a. if , i.e., , the fixed point is an unstable spiral.
    • b. if , i.e., , the fixed point is a center of circles.
    • c. if , i.e., , the fixed point is a stable spiral.
  • :
    • a. if , i.e., , the fixed point is an unstable star.
    • b. if , i.e., , the fixed point is a stable star
  • :
    • a. if , i.e., , the fixed point is a saddle point.
    • b. if , i.e., , we can show that , then the fixed point is a stable node.
  • : There is no isolated fixed points.
    • a. if , i.e., , there is a line of unstable nodes.
    • b. if , i.e., , there is a line of stable nodes.
    • c. if , i.e., , there is no fixed points.

All the spiral and circles are anticlockwise. This can be verified by investigating the velocity at points at vertical line using the fact . And (1,1) is always the only eigenvector for stars in situations 2.a and 2.b. This observation is very helpful for proving the existence of limit cycles in a rectified linear neural network. We restate it as a lemma.

Lemma 1. All the spiral and circles of a linear neural network are anticlockwise. Formally, we can write it with cross product .

Note that when , and ,

always holds. Thus the unstable node cannot exist in this system. If we temporarily ignore the existence of boundary fixed points ( or ), our discussion covers all 10 possible situations. The boundary fixed points may exist because we want to ensure . However, these points are not interesting by now since they indicate dead neurons. We visualize all 9 situations below except for 4.c, which is less interesting.

Slow Movement (Plasticity)

In the long term, the change of affects the location of fixed point as well as the stability of this linear system. Since , and , as long as is small enough and initial , is alway nonnegative. So are and . The trace of Jacobian will chage with

since this movement slow compared with movement of , then approximately is the fixed point of the linear system parameterized by and external input . And the determinant of Jacobian varies by

Assume . The slow dynamical process is captured by

Note that fixed point is a function of but cannot be determine by only and . If there is no external input, i.e., , then this is a perfect 2D linear system, where and , the fixed point becomes a stable star. But in a more general situation, the external inputs are non-zero, and the equations become very complicated

where .

CASE 2. , w/ nonnegative connenction constraints in network dynamics.

The dynamics can be expressed as

where , , and . Suppose is the gradient in case 1 (without nonnegativity constraints). Then current gradient can be seen as some clipped gradient

Thus we know the behavior of this system should be similar to case 1, because the only difference is now and cannot decrease too fast.

Theorem 1. The positive fixed point of the linear system in case 1 is also the only positive fixed point of the network with nonnegative connection constraints, where representing the neural activity of excitatory neuron and inhibitory neuron.

Proof. Because is a fixed point of linear system,

Observe that

Since , it guarantees . is also a fixed point of the system with nonnegativity constraints. Furthermore, since , if is a fixed point of the system with nonnegativity constraints,

must hold. As we know that when the fixed point of the linear dynamical system in case 1 exists, and it is the only solution to the above linear system. Hence, we have shown that the positive fixed point of the linear system in case 1 is also the only positive fixed point of the network with nonnegative connection constraints. Q.E.D.

Eprically, we investigate the following six categories:

  • :
    • a. if , i.e., , the fixed point is still an unstable spiral, but there exist one stable limit cycle.
    • b. if , i.e., , the fixed point is still a center of circles, but there exist one stable limit cycle.
    • c. if , i.e., , the fixed point is still a stable spiral.
  • :
    • a. if , i.e., , the fixed point is an unstable star.
    • b. if , i.e., , the fixed point is a stable star.
  • :
    • a. if , i.e., ,the original fixed point is a saddle point, but there exist one extra stable node on the boundary.
    • b. if , i.e., , we can show that , then the fixed point is a stable node.
  • : There is no isolated fixed points.
    • a. if , i.e., , there is a line repeller.
    • b. if , i.e., , there is a line attractor, and one stable node on the boundary.
    • c. if , i.e., , there is no fixed points.

Fast Movement (Neural Activity) - Cont’d

There might be some “extra” stable node on the boundary, or limit cycles after applying nonnegative connection constraints to the neural network dynamics. Those boundary fixed points exists in the linear neural network, but now they are more stable since they are fixed points even when we allow to be negative. But more interestingly, we want to analyze those emergent limit cycles — what is the condition for a limit cycle to exist?

First, according to Theorem 1, the positive fixed point of a rectified neural network is also the positive fixed point of the corresponding linear neural network. Also, all the points in the region

have the identical movement to the linear networks. The positive fixed point is always in this region . Therefore, the neighborhood is unaffected by the nonnegative connection constraints. We illustrate this region in gray. If there exists a limit cycle, then the Hairy Ball Theorem guarantees that there must be a fixed point in the interior region of the limit cycle. Since is the only positive fixed point, there must be at least part of the limit cycle lies in . When , all the trajectories in either diverge from or converge to . When and , all the trajectories in also converge to . These rule out the possibility of the existence of limit cycles. Thus, a necessary condition for limit cycle to exist is and .

Theorem 2. (Necessary and Sufficient Condition for the Existence of Limit Cycles) In a rectified neural network, a limit cycle exists if and only if all the following conditions hold: (1) a positive fixed point exists; (2) ; and (3) .

Proof. The necessity has been justified as above. Now we prove this is actually a sufficient condition. Let us divide the first quadrant into four regions . We have discussed in the above context, and the remaining three are defined as follows. Consider another region illustrated in red

The velocity at any point is exact . Vice versea, if we know the velocity at some point is , then . When , by the AM-GM inequality,

there must exist a region

at the upper right part of the first quadrant between regions and . The horizontal velocity at any points is exact , and the vertical velocity is greater than , and any point having this velocity property is in .

And when , there exis an additional region

at the lower left part of the first quadrant between regions and . The vertical velocity at any points is exact , and the horizontal velocity is greater than , and any point having this velocity property is in .

We continue our proof from a geometric point of view.

is the original point. is the positive fixed point. Line intersects -axis at point , and line intersects -axis at point . is the intersection of these two lines. Since , are both on the half-line , and , . When (1) a positive fixed point exists; (2) ; and (3) , there are two cases:

  • , illustrated in Figure (a).
  • , illustrated in Figure (b). In this case, we need to ensure the positive fixed point exists.

In both cases, there are three adjacent regions , , . In the region , as we discussed in the linear case (Lemma 1), the velocity field is anticlockwise around the fixed point . If find all the points in where the velocity is parallel to the line :

where , they all lie on a line

going through the fixed point .

Now we argue that this line intersect with line at point , which always lies on the boundary between and .

Since at any points on line , including point , then at should be . Solving

we get the coordinates of is . In case (a), the -coordinate of is , which is smaller than when holds. Since the positive fixed point exists and , we can verify must hold. Therefore, point is above on the line . is always on the boundary between and .

In case (b), the -coordinate of is always positive because

and . Hence, point is above and always on the boundary between and . Point splits the boundary between and into two parts: on the line segment (or in case (b)), the vertical velocity ; on the half-line above point , the vertical velocity . In other words, trajectories always come into region through the line segment , (or ), and come out from region to region through the half-line above point .

Now we verify the following four claims:

[C1] Every trajectory comes in the region through line segment will come out from the half-line above point into region .
[C2] Every trajectory in the region will go through line segment into region .
[C3] Every trajectory in the region will go through and finally in .
[C4] Every trajectory in the region will finally go into or .

If a trajectory comes across into (no matter from or ), it has two fates: either staying in or leave from somewhere. As we know that the velocity field in is identical to that of the linear network of cases 1(a) and 1(b), this trajectory cannot stay in forever. If it is case 1(a), this trajectory is a part of the arm of an unstable spiral and will go out finally; if it is case 1(b), this trajectory has to be a closed circle in , contradicting with the fact that it comes into through . Thus, this trajectory has to leave from somewhere. But where?

In the case illustrated in figure (b), the only possibility is half-line above on the boundary between and . In figure (a) case, we show that it cannot be line segment (the boundary between and ). The vertical velocity at points on is always . When discussing the horizontal velocity, line is a good auxiliary. On the line , the horizontal velocity . For region below line , the horizontal velocity . Therefore, at point , the velocity is exactly , pointing towards , which is pointing the interior of or along the boundary when . Moreover, the velocities at other points on are all pointing interior , as they all have greater horizontal component towards . It means in figure (a) case, and trajectories cannot go out through . Hence, our first claim is true.

The trajectories in region all have horizontal velocity . In the long term, they have . It indicates they cannot stay in forever. In figure (a) case, it asks for trajectories to exceed point in the horizontal direction, whose -coordinate is non-zero constant as must hold. Then they are no longer in the region . It is a contradiction. Similarly, in figure (b) case, trajectories approch infinitly close to the line segment . But we know that when close to line the vertical velocity for (it must hold in figure (b) case otherwise and no positive fixed point exists.) and any sufficient small . It derives , the trajectories will exceed point in the vertical direction. It leads a contradiction. Then we know that all the trajectories in leave this region from somewhere.

There is actually only one option for those trajectories to go. Since the velocity at any point on the boundary between and is pointing original point , when , it is always from to ; when , it is along the line and moving towards point . The velocity at the point and on the line segment has been discussed above. Thus, all trajectories all finally go across into . The second claim is valid. The last two claims are relatively easier to verify. does not contain fixed point so that goes through into . Then following the similar argument we made for the first claim, these trajectories will come out the half-line above point into region , proving the third claim. The velocity in region is . If the trajectories are confined in , then as time goes to infinite which leads a contradiction since . Thus the trajectories will go into , , or through the point into then go to . The fourth claim holds.

So far, we make it clear that any trajectory in the plane, either is a closed orbit in the region , or eventually shuttles in and out at the boundary between and . It is the time to give our proof a final hit — there must exist a stable limit cycle!

Case i. When , there is a closed trajectory in which tangentially intersects with line at point . We argue this closed circle is the limit cycle. In the trajectories inner this circle are all closed circles around the fixed point . Now we show that the outer trajectories spiral into it as time approaches infinity. Consider the trajectory comes into at point . According to Claim C1, this trajectory will come out at some point on the half-line above on the boundary between and . By Claim C4, it will come back to again through some point on the line segment (or if it is in figure (b) situation. The discussion for these two situations are essentially the same so let us focus on the figure (a) situation, as illustrated above). It cannot go through the endpoint again to make itself closed, otherwise another trajectory in has to intersect with it to be in through on , which violates the existence and uniqueness theorem. Thus, it revisits through another point on line segment , . Then, it comes to through point on the open line segment , and goes back to through the point on the open line segment . Actually, if a trajectory enters at point on line segment , the next time it will reenter at on the open line segment if . It can be easily verified by calculating the analytical solution of the coordinates of point . An intuitive explanation is without nonnegativity constaint, the trajectory will go back to , but now region sets a “speed limit” for the horizontal velocity but remains the vertical velocity the same. It is like when we throw a ball on a hill, it falls closer when there is horizontal air drag. Therefore, all the trajectories will spiral into the tangent closed circle. It is a stable limit cycle!

Case ii. When , there is an unstable spiral trajectory in which tangentially intersects with line at the point and goes to at the point . This trajectory reenters through point on the line segment by Claim C2 ( can be ), and then exits through point with . If , this suggests there exists an open line segment on line segment such that any trajectory enters through some point on (), will reenter through some point on the open line segment . Also, there exists an open line segment on the half-line above such that any trajectory leaves through some point () on , will leave next time through some point on the open line segment . However, if we investigate trajectory coming into through point , following the same argument we have made in case i, it will spiral inward. Thus, it suggests any trajectory enters through some point () on , will reenter through some point on the open line segment . Also, any trajectory leaves through some point (), will leave next time through some point on the open line segment . All these arguments can be verified by calculating coordinates of given . If there is a trajectory passes through then it will pass , and revisit again. Otherwise, it will intersect with adjacent orbits. This has to be a closed orbit. Therefore, all the trajectories spiral to approach a closed orbit going through and as time go to infinity!

To sum up, when (1) a positive fixed point exists; (2) ; and (3) , there is a stable limit cycle in if , and a limit cycle across and if . Combined with the necessity arguments, these three conditions are necessary and sufficient for the existence of limit cycle.

Q.E.D.

CASE 3: Time Constants (n=r=1)

We add two time constants and to contral the speed of excitatory and inhibitory synapses. The dynamics of the neural network is defined by

where , , and . Adding times constants does not change the division of regions , , , and . Also, it does not change the existence and the location of positive fixed point . It will change the trace and the determinant of Jacobian

respectively. The discriminant becomes

We update Theorem 2 to include the time constants: in a rectified linear neural network, a limit cycle exists if and only if all the following conditions hold:

  (1) a positive fixed point exists;
  (2) ; and
  (3) .

When the inhibition is much faster than the excitation, i.e., , it is impossible to produce limit cycles in an E-I network.

Runzhe Yang
Please contact the author to cite this work.