![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/03\/att2_hi.gif?ssl=1\")
Neural networks have become ubiquitous in our current Artificial Intelligence<\/a> era, typically serving as powerful tools for representation extraction and pattern recognition. However, these systems can also be viewed through another fascinating lens: as dynamical systems that evolve and converge to a manifold of states over time. When implemented with feedback loops, even simple neural networks can produce strikingly beautiful attractors, ranging from limit cycles to chaotic structures.<\/p>\n While neural networks in general sense are most commonly known for embedding extraction tasks, they can also be viewed as dynamical systems. A dynamical system describes how points in a state space evolve over time according to a fixed set of rules or forces. In the context of neural networks, the state space consists of the activation patterns of neurons, and the evolution rule is determined by the network\u2019s weights, biases, activation functions, and other tricks.<\/p>\n Traditional NNs are optimized via gradient descent to find its endstate of convergence. However, when we introduce feedback \u2014 connecting the output back to the input \u2014 the network becomes a recurrent system with a different kind of temporal dynamic. These dynamics can exhibit a wide range of behaviors, from simple convergence to a fixed point to complex chaotic patterns.<\/p>\n An attractor is a set of states toward which a system tends to evolve from a wide variety of starting conditions. Once a system reaches an attractor, it remains within that set of states unless perturbed by an external force. Attractors<\/a> are indeed deeply involved in forming memories [1], oscillating chemical reactions [2], and other nonlinear dynamical systems.\u00a0<\/p>\n Dynamical Systems<\/a> can exhibit several types of attractors, each with distinct characteristics:<\/p>\n In the following section, we will dive deeper into an example of a very simple NN architecture capable of said behavior, and demonstrate some pretty examples. We will touch on Lyapunov exponents, and provide implementation for those who wish to experiment with generating their own Neural Network<\/a> attractor art (and not in the generative AI sense).<\/p>\n We will use a grossly simplified one-layer NN with a feedback loop. The architecture consists of:<\/p>\n The critical aspects that make this network generate attractors:<\/p>\n In order to investigate how prone the system is to chaos, we will calculate the Lyapunov exponents for different sets of parameters. Lyapunov exponent is a measure of the instability of a dynamical system<\/strong>\u2026<\/p>\n \u2026where nt<\/sub>\u200b is a number of time steps, \u0394yk<\/sub> \u200bis a distance between the states y(xi<\/sub>) and y(xi<\/sub>+\u03f5) at a point in time; \u0394Z(0) represents an initial infinitesimal (very small) separation between two nearby starting points, and \u0394Z(t) is the separation after time t. For stable systems converging to a fixed point or a stable attractor this parameter is less than 0, for unstable (diverging, and, therefore, chaotic systems) it is greater than 0.<\/p>\n Let\u2019s code it up! We will only use NumPy and default Python libraries for the implementation.<\/p>\n We initialize the Next, we will define the iteration logic. We start every iteration with the feedback loop \u2014 we implement the shift register circuit by shifting all y elements to the right, and compute the most recent y0<\/sub> output to place it into the first element of the input.<\/p>\n \n Now, we define the function that will iterate our network map over the tmax number of time steps and output the states of the first two hidden neurons for visualization. We can use any hidden neurons, and we could even visualize 3D state space, but we will limit our imagination to two dimensions.<\/p>\n This is the gist of the system. Now, we will just define some line and segment magic for pretty visualizations.<\/p>\n \n The functions written above will take the generated state space trajectories and visualize them. Because the state space may be densely filled, we will skip every 8th, 16th or 32th time point to sparsify<\/strong> our vectors. We also don\u2019t want to plot these in one solid color, therefore we are coding the color as a periodic function<\/strong> (np.sin(1.6 * y_plot + 0.4 * x_plot)<\/em>) based on the x and y coordinates of the figure axis. The multipliers for the coordinates are arbitrary and happen to generate nice smooth color maps, to your liking.<\/p>\n \n After defining the NN and iteration parameters, we can generate the state space trajectories. If we spend enough time poking around with parameters, we will find something cool (I promise!). If manual parameter grid search labor is not exactly our thing, we could add a function that checks what proportion of the state space is covered over time<\/strong>. If after t = 100,000 iterations (except the initial 1,000 \u201cwarm up\u201d time steps) we only touched a narrow range of values of the state space, we are likely stuck in a point. Once we found an attractor that is not so shy to take up more state space, we can plot it using default plotting params:<\/p>\n One of the stable types of attractors is the limit cycle attractor<\/strong> (parameters: N = 4, D = 32, s = 0.22, seed = 174658140). It looks like a single, closed loop trajectory in phase space. The orbit follows a regular, periodic path over time series. I will not include the code for Lyapunov exponent calculation here to focus on the visual aspect of the generated attractors more, but one can find it under this link<\/a>, if interested. The Lyapunov exponent for this attractor (\u03bb=\u22123.65) is negative, indicating stability<\/strong>: mathematically, this exponent will lead to the state of the system decaying, or converging, to this basin of attraction over time.<\/p>\n If we keep increasing the scaling factor, we are more likely to tune up the values in the circuit, and perhaps more likely to find something interesting.<\/p>\n Here is the toroidal (quasiperiodic) attractor<\/strong> (parameters: N = 4, D = 32, s = 0.55, seed = 3160697950). It still has an ordered structure of sheets that wrap around in organized, quasiperiodic patterns. The Lyapunov exponent for this attractor has a higher value, but is still negative (\u03bb=\u22120.20).<\/p>\n As we further increase the scaling factor s, the system becomes more prone to chaos. The strange (chaotic) attractor<\/strong> emerges with the following parameters: N = 4, D = 16, s = 1.4, seed = 174658140). It is characterized by an erratic, unpredictable pattern of trajectories that never repeat. The Lyapunov exponent for this attractor is positive (\u03bb=0.32), indicating instability (divergence from an initially very close state over time) and chaotic behavior. This is the \u201cbutterfly effect\u201d attractor.<\/p>\n As we further increase the scaling factor s, the system becomes more prone to chaos. The strange (chaotic) attractor emerges with the following parameters: N = 4, D = 16, s = 1.4, seed = 174658140. It is characterized by an erratic, unpredictable pattern of trajectories that never repeat. The Lyapunov exponent for this attractor is positive (\u03bb=0.32), indicating instability (divergence from an initially very close state over time) and chaotic behavior. This is the \u201cbutterfly effect\u201d attractor.<\/p>\n Just another confirmation that aesthetics can be very mathematical, and vice versa. The most visually compelling attractors often exist at the edge of chaos \u2014 think about it for a second! These structures are complex enough to exhibit intricate behavior, yet ordered enough to maintain coherence. This resonates with observations from various art forms, where balance between order and unpredictability often creates the most engaging experiences.<\/p>\n An interactive widget to generate and visualize these attractors is available here<\/a>. The source code is available<\/a>, too, and invites further exploration. The ideas behind this project were largely inspired by the work of J.C. Sprott [3].\u00a0<\/p>\n [1] B. Poucet and E. Save, Attractors in Memory (2005), Science DOI:10.1126\/science.1112555<\/a>.<\/p>\n [2] Y.J.F. Kpomahou et al., Chaotic Behaviors and Coexisting Attractors in a New Nonlinear Dissipative Parametric Chemical Oscillator (2022), Complexity DOI:10.1155\/2022\/9350516<\/a>.<\/p>\n [3] J.C. Sprott, Artificial Neural Net Attractors (1998), Computers & Graphics DOI:10.1016\/S0097-8493(97)00089-7<\/a>.<\/p>\n \n The post Attractors in Neural Network Circuits:\u00a0Beauty and Chaos<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\nNeural Networks as Dynamical Systems<\/strong><\/h3>\n
Understanding Attractors<\/strong><\/h3>\n
Types of Attractors<\/strong><\/h3>\n
\n
Setup<\/strong><\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/03\/neural-attractor-schematic-1024x640.png?resize=1024%2C640&ssl=1\")
\n
\n
\n
\n
\n
\n
\n
\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/03\/Screenshot-2025-03-25-at-3.26.52%25E2%2580%25AFPM-1024x248.png?resize=1024%2C248&ssl=1\")
<\/figure>\nimport numpy as np\nfrom typing import Tuple, List, Optional\n\n\nclass NeuralAttractor:\n \"\"\"\n \n N : int\n Number of neurons in the hidden layer\n D : int\n Dimension of the input vector\n s : float\n Scaling factor for the output\n\n \"\"\"\n \n def __init__(self, N: int = 4, D: int = 16, s: float = 0.75, seed: Optional[int] = \nNone):\n self.N = N\n self.D = D\n self.s = s\n \n if seed is not None:\n np.random.seed(seed)\n \n # Initialize weights and biases\n self.w = 2.0 * np.random.random((N, D)) - 1.0 # Uniform in [-1, 1]\n self.b = s * np.random.random(N) # Uniform in [0, s]\n \n # Initialize state vector structures\n self.x = np.zeros(N) # Neuron states\n self.y = np.zeros(D) # Input vector<\/code><\/pre>\nNeuralAttractor<\/code> class with some basic parameters \u2014 number of neurons in the hidden layer, number of elements in the input array, scaling factor for the output, and random seed. We proceed to initialize the weights and biases randomly, and x and y states. These weights and biases will not be optimized \u2014 they will stay put, no gradient descent this time.<\/p>\n def reset(self, init_value: float = 0.001):\n \"\"\"Reset the network state to initial conditions.\"\"\"\n self.x = np.ones(self.N) * init_value\n self.y = np.zeros(self.D)\n \n def iterate(self) -> np.ndarray:\n \"\"\"\n Perform one iteration of the network and return the neuron outputs.\n \n \"\"\"\n # Calculate the output y0\n y0 = np.sum(self.b * self.x)\n \n # Shift the input vector\n self.y[1:] = self.y[:-1]\n self.y[0] = y0\n \n # Calculate the neuron inputs and apply activation fn\n for i in range(self.N):\n u = np.sum(self.w[i] * self.y)\n self.x[i] = np.tanh(u)\n \n return self.x.copy()<\/code><\/pre>\n def generate_trajectory(self, tmax: int, discard: int = 0) -> Tuple[np.ndarray, \nnp.ndarray]:\n \"\"\"\n Generate a trajectory of the states for tmax iterations.\n \n -----------\n tmax : int\n Total number of iterations\n discard : int\n Number of initial iterations to discard\n\n \"\"\"\n self.reset()\n \n # Discard initial transient\n for _ in range(discard):\n self.iterate()\n \n x1_traj = np.zeros(tmax)\n x2_traj = np.zeros(tmax)\n \n for t in range(tmax):\n x = self.iterate()\n x1_traj[t] = x[0]\n x2_traj[t] = x[1]\n \n return x1_traj, x2_traj<\/code><\/pre>\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.collections as mcoll\nimport matplotlib.path as mpath\nfrom typing import Tuple, Optional, Callable\n\n\ndef make_segments(x: np.ndarray, y: np.ndarray) -> np.ndarray:\n \"\"\"\n Create list of line segments from x and y coordinates.\n \n -----------\n x : np.ndarray\n X coordinates\n y : np.ndarray\n Y coordinates\n\n \"\"\"\n points = np.array([x, y]).T.reshape(-1, 1, 2)\n segments = np.concatenate([points[:-1], points[1:]], axis=1)\n return segments\n\n\ndef colorline(\n x: np.ndarray,\n y: np.ndarray,\n z: Optional[np.ndarray] = None,\n cmap = plt.get_cmap(\"jet\"),\n norm = plt.Normalize(0.0, 1.0),\n linewidth: float = 1.0,\n alpha: float = 0.05,\n ax = None\n):\n \"\"\"\n Plot a colored line with coordinates x and y.\n \n -----------\n x : np.ndarray\n X coordinates\n y : np.ndarray\n Y coordinates\n\n \"\"\"\n if ax is None:\n ax = plt.gca()\n \n if z is None:\n z = np.linspace(0.0, 1.0, len(x))\n \n segments = make_segments(x, y)\n lc = mcoll.LineCollection(\n segments, array=z, cmap=cmap, norm=norm, linewidth=linewidth, alpha=alpha\n )\n ax.add_collection(lc)\n \n return lc\n\n\ndef plot_attractor_trajectory(\n x: np.ndarray,\n y: np.ndarray,\n skip_value: int = 16,\n color_function: Optional[Callable] = None,\n cmap = plt.get_cmap(\"Spectral\"),\n linewidth: float = 0.1,\n alpha: float = 0.1,\n figsize: Tuple[float, float] = (10, 10),\n interpolate_steps: int = 3,\n output_path: Optional[str] = None,\n dpi: int = 300,\n show: bool = True\n):\n \"\"\"\n Plot an attractor trajectory.\n \n Parameters:\n -----------\n x : np.ndarray\n X coordinates\n y : np.ndarray\n Y coordinates\n skip_value : int\n Number of points to skip for sparser plotting\n\n \"\"\"\n fig, ax = plt.subplots(figsize=figsize)\n \n if interpolate_steps > 1:\n path = mpath.Path(np.column_stack([x, y]))\n verts = path.interpolated(steps=interpolate_steps).vertices\n x, y = verts[:, 0], verts[:, 1]\n \n x_plot = x[::skip_value]\n y_plot = y[::skip_value]\n \n if color_function is None:\n z = abs(np.sin(1.6 * y_plot + 0.4 * x_plot))\n else:\n z = color_function(x_plot, y_plot)\n \n colorline(x_plot, y_plot, z, cmap=cmap, linewidth=linewidth, alpha=alpha, ax=ax)\n \n ax.set_xlim(x.min(), x.max())\n ax.set_ylim(y.min(), y.max())\n \n ax.set_axis_off()\n ax.set_aspect('equal')\n \n plt.tight_layout()\n \n if output_path:\n fig.savefig(output_path, dpi=dpi, bbox_inches='tight')\n\n return fig<\/code><\/pre>\nN = 4\nD = 32\ns = 0.22\nseed=174658140\n\ntmax = 100000\ndiscard = 1000\n\nnn = NeuralAttractor(N, D, s, seed=seed)\n\n# Generate trajectory\nx1, x2 = nn.generate_trajectory(tmax, discard)\n\nplot_attractor_trajectory(\n x1, x2,\n output_path='trajectory.png',\n)<\/code><\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/03\/N_4_D_16_s_0.22_limcyc.png?ssl=1\")
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/03\/N_4_D_32_s_0.55_toroidal.png?ssl=1\")
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/03\/N_4_D_16_s_1.4_strange.png?ssl=1\")
References<\/h3>\n