{"id":1637,"date":"2025-02-04T07:02:58","date_gmt":"2025-02-04T07:02:58","guid":{"rendered":"https:\/\/mailitics.com\/index.php\/2025\/02\/04\/neural-networks-intuitively-and-exhaustively-explained-0153f85c1007\/"},"modified":"2025-02-04T07:02:58","modified_gmt":"2025-02-04T07:02:58","slug":"neural-networks-intuitively-and-exhaustively-explained-0153f85c1007","status":"publish","type":"post","link":"https:\/\/mailitics.com\/index.php\/2025\/02\/04\/neural-networks-intuitively-and-exhaustively-explained-0153f85c1007\/","title":{"rendered":"Neural Networks \u2013 Intuitively and Exhaustively Explained"},"content":{"rendered":"

Neural Networks \u2013 Intuitively and Exhaustively Explained
\n \t

\n
<\/BR>
\n
\n \t

\n
<\/BR><\/p>\n

\n

An in-depth exploration of the most fundamental architecture in modern AI<\/h3>\n
\"The Thinking Part\" by Daniel Warfield using MidJourney. All images by the author unless otherwise specified. Article originally made available on Intuitively and Exhaustively Explained.
“The Thinking Part” by Daniel Warfield using MidJourney. All images by the author unless otherwise specified. Article originally made available on Intuitively and Exhaustively Explained<\/a>.<\/figcaption><\/figure>\n

In this article we\u2019ll form a thorough understanding of the neural network, a cornerstone technology underpinning virtually all cutting edge AI<\/a> systems. We\u2019ll first explore neurons in the human brain, and then explore how they formed the fundamental inspiration for neural networks in AI. We\u2019ll then explore back-propagation, the algorithm used to train neural networks to do cool stuff. Finally, after forging a thorough conceptual understanding, we\u2019ll implement a Neural Network ourselves from scratch and train it to solve a toy problem.<\/p>\n

\n

Who is this useful for?<\/strong> Anyone who wants to form a complete understanding of the state of the art of AI.<\/p>\n

How advanced is this post?<\/strong> This article is designed to be accessible to beginners, and also contains thorough information which may serve as a useful refresher for more experienced readers.<\/p>\n

Pre-requisites:<\/strong> None<\/p>\n

\n

Inspiration From the Brain<\/h2>\n

Neural networks take direct inspiration from the human brain, which is made up of billions of incredibly complex cells called neurons.<\/p>\n

\"The
The Neuron, source<\/a><\/figcaption><\/figure>\n

The process of thinking within the human brain is the result of communication between neurons. You might receive stimulus in the form of something you saw, then that information is propagated to neurons in the brain via electrochemical signals.<\/p>\n

\"eye
eye image generated with Midjourney<\/figcaption><\/figure>\n

The first neurons in the brain receive that stimulus, then each neuron may choose whether or not to “fire” based on how much stimulus it received. “Firing”, in this case, is a neurons decision to send signals to the neurons it\u2019s connected to.<\/p>\n

\"Imagine
Imagine the signal from the eye directly feeds into three neurons, and two decide to fire.<\/figcaption><\/figure>\n

Then the neurons which those Neurons are connected to may or may not choose to fire.<\/p>\n

\"Neurons
Neurons receive stimulus from previous neurons and then choose whether or not to fire based on the magnitude of the stimulus.<\/figcaption><\/figure>\n

Thus, a “thought” can be conceptualized as a large number of neurons choosing to, or not to fire based on the stimulus from other neurons.<\/p>\n

As one navigates throughout the world, one might have certain thoughts more than another person. A cellist might use some neurons more than a mathematician, for instance.<\/p>\n

\"Different
Different tasks require the use of different neurons. Images generated with Midjourney<\/figcaption><\/figure>\n

When we use certain neurons more frequently, their connections become stronger, increasing the intensity of those connections. When we don\u2019t use certain neurons, those connections weaken. This general rule has inspired the phrase “Neurons that fire together, wire together”, and is the high-level quality of the brain which is responsible for the learning process.<\/p>\n

\"The
The process of using certain neurons strengthens their connections.<\/figcaption><\/figure>\n

I\u2019m not a neurologist, so of course this is a tremendously simplified description of the brain. However, it\u2019s enough to understand the fundamental idea of a neural network.<\/p>\n

The Intuition of Neural Networks<\/h2>\n

Neural networks are, essentially, a mathematically convenient and simplified version of neurons within the brain. A neural network is made up of elements called “perceptrons”, which are directly inspired by neurons.<\/p>\n

\"A
A perceptron, on the left, vs a neuron, on the right. [source](https:\/\/en.wikipedia.org\/wiki\/Neuron#\/media\/File:Blausen_0657_MultipolarNeuron.png) 1<\/a>, source 2<\/figcaption><\/figure>\n

Perceptrons take in data, like a neuron does,<\/p>\n

\"Perceptrons
Perceptrons in AI work with numbers, while Neurons within the brain work with electrochemical signals.<\/figcaption><\/figure>\n

aggregate that data, like a neuron does,<\/p>\n

\"Perceptrons
Perceptrons aggregate numbers to come up with an output, while neurons aggregate electrochemical signals to come up with an output.<\/figcaption><\/figure>\n

then output a signal based on the input, like a neuron does.<\/p>\n

\"Perceptrons
Perceptrons output numbers, while neurons output electrochemical signals.<\/figcaption><\/figure>\n

A neural network can be conceptualized as a big network of these perceptrons, just like the brain is a big network of neurons.<\/p>\n

\"A
A neural network (left) vs the brain (right). src1<\/a> src2<\/a><\/figcaption><\/figure>\n

When a neuron in the brain fires, it does so as a binary decision. Or, in other words, neurons either fire or they don\u2019t. Perceptrons, on the other hand, don\u2019t “fire” per-se, but output a range of numbers based on the perceptrons input.<\/p>\n

\"Perceptrons
Perceptrons output a continuous range of numbers, while Neurons either fire or they don\u2019t.<\/figcaption><\/figure>\n

Neurons within the brain can get away with their relatively simple binary inputs and outputs because thoughts exist over time. Neurons essentially pulse at different rates<\/a>, with slower and faster pulses communicating different information.<\/p>\n

So, neurons have simple inputs and outputs in the form of on or off pulses, but the rate at which they pulse can communicate complex information. Perceptrons only see an input once per pass through the network, but their input and output can be a continuous range of values. If you\u2019re familiar with electronics, you might reflect on how this is similar to the relationship between digital and analogue signals.<\/p>\n

The way the math for a perceptron actually shakes out is pretty simple. A standard neural network consists of a bunch of weights connecting the perceptron\u2019s of different layers together.<\/p>\n

\"A
A neural network, with the weights leading into and out of a particular perceptron highlighted.<\/figcaption><\/figure>\n

You can calculate the value of a particular perceptron by adding up all the inputs, multiplied by their respective weights.<\/p>\n

\"An
An example of how the value of a perceptron might be calculated. (0.3\u00d70.3) + (0.7\u00d70.1) +(-0.5\u00d70.5)=-0.09<\/figcaption><\/figure>\n

Many Neural Networks also have a “bias” associated with each perceptron, which is added to the sum of the inputs to calculate the perceptron\u2019s value.<\/p>\n

\"An
An example of how the value of a perceptron might be calculated when a bias term is included in the model. (0.3\u00d70.3) + (0.7\u00d70.1) +(-0.5\u00d70.5) + 0.01 =-0.08<\/figcaption><\/figure>\n

Calculating the output of a neural network, then, is just doing a bunch of addition and multiplication to calculate the value of all the perceptrons.<\/p>\n

Sometimes data scientists refer to this general operation as a “linear projection”, because we\u2019re mapping an input into an output via linear operations (addition and multiplication). One problem with this approach is, even if you daisy chain a billion of these layers together, the resulting model will still just be a linear relationship between the input and output because it\u2019s all just addition and multiplication.<\/p>\n

This is a serious problem because not all relationships between an input and output are linear. To get around this, data scientists employ something called an “activation function”. These are non-linear functions which can be injected throughout the model to, essentially, sprinkle in some non-linearity.<\/p>\n

\"Examples
Examples of a variety of functions which, given some input, produce some output. The top three are linear, while the bottom three are non-linear.<\/figcaption><\/figure>\n

by interweaving non-linear activation functions between linear projections, neural networks are capable of learning very complex functions,<\/p>\n

\"By
By placing non-linear activation functions within a neural network, neural networks are capable of modeling complex relationships.<\/figcaption><\/figure>\n

In AI there are many popular activation functions, but the industry has largely converged on three popular ones: ReLU, Sigmoid, and Softmax, which are used in a variety of different applications. Out of all of them, ReLU is the most common due to its simplicity and ability to generalize to mimic almost any other function.<\/p>\n

\"The
The ReLU activation function, where the output is equal to zero if the input is less than zero, and the output is equal to the input if the input is greater than zero.<\/figcaption><\/figure>\n

So, that\u2019s the essence of how AI models make predictions. It\u2019s a bunch of addition and multiplication with some nonlinear functions sprinkled in between.<\/p>\n

Another defining characteristic of neural networks is that they can be trained to be better at solving a certain problem, which we\u2019ll explore in the next section.<\/p>\n

Back Propagation<\/h2>\n

One of the fundamental ideas of AI is that you can “train” a model. This is done by asking a neural network (which starts its life as a big pile of random data) to do some task. Then, you somehow update the model based on how the model\u2019s output compares to a known good answer.<\/p>\n

\"The
The fundamental idea of training a neural network. You give it some data where you know what you want the output to be, compare the neural networks output with your desired result, then use how wrong the neural network was to update the parameters so it\u2019s less wrong.<\/figcaption><\/figure>\n

For this section, let\u2019s imagine a neural network with an input layer, a hidden layer, and an output layer.<\/p>\n

A neural network with two inputs and a single output, with a hidden layer in-between allowing the model to make more complex predictions.<\/figcaption><\/figure>\n

Each of these layers are connected together with, initially, completely random weights.<\/p>\n

\"The
The neural network, with random weights and biases defined.<\/figcaption><\/figure>\n

And we\u2019ll use a ReLU activation function on our hidden layer.<\/p>\n

We\u2019ll apply the ReLU activation function to the value of our hidden perceptrons.<\/figcaption><\/figure>\n

Let\u2019s say we have some training data, in which the desired output is the average value of the input.<\/p>\n

\"An
An example of the data that we\u2019ll be training off of.<\/figcaption><\/figure>\n

And we pass an example of our training data through the model, generating a prediction.<\/p>\n

Calculating the value of the hidden layer and output based on the input, including all major intermediary steps.<\/figcaption><\/figure>\n

To make our neural network better at the task of calculating the average of the input, we first compare the predicted output to what our desired output is.<\/p>\n

\"The
The training data has an input of 0.1 and 0.3, and the desired output (the average of the input) is 0.2. The prediction from the model was -0.1. Thus, the difference between the output and the desired output is 0.3.<\/figcaption><\/figure>\n

Now that we know that the output should increase in size, we can look back through the model to calculate how our weights and biases might change to promote that change.<\/p>\n

First, let\u2019s look at the weights leading immediately into the output: w\u2087, w\u2088, w\u2089. Because the output of the third hidden perceptron was -0.46, the activation from ReLU was 0.00.<\/p>\n

\"The
The ultimate, activated output of the third perceptron, 0.00<\/figcaption><\/figure>\n

As a result, there\u2019s no change to w\u2089 that could result us getting closer to our desired output, because every value of w\u2089 would result in a change of zero in this particular example.<\/p>\n

The second hidden neuron, however, does have an activated output which is greater than zero, and thus adjusting w\u2088 will have an impact on the output for this example.<\/p>\n

\"\"<\/figure>\n

The way we actually calculate how much w\u2088 should change is by multiplying how much the output should change, times the input to w\u2088.<\/p>\n

How we calculate how the weight should change. Here the symbol \u0394(delta) means \"change in\", so \u0394w\u2088 means the \"change in w\u2088\"
How we calculate how the weight should change. Here the symbol \u0394(delta) means “change in”, so \u0394w\u2088 means the “change in w\u2088”<\/figcaption><\/figure>\n

The easiest explanation of why we do it this way is “because calculus”, but if we look at how all weights get updated in the last layer, we can form a fun intuition.<\/p>\n

\"Calculating
Calculating how the weights leading into the output should change.<\/figcaption><\/figure>\n

Notice how the two perceptrons that “fire” (have an output greater than zero) are updated together. Also, notice how the stronger a perceptrons output is, the more its corresponding weight is updated. This is somewhat similar to the idea that “Neurons that fire together, wire together” within the human brain.<\/p>\n

Calculating the change to the output bias is super easy. In fact, we\u2019ve already done it. Because the bias is how much a perceptrons output should change, the change in the bias is just the changed in the desired output. So, \u0394b\u2084=0.3<\/p>\n

\"how
how the bias of the output should be updated.<\/figcaption><\/figure>\n

Now that we\u2019ve calculated how the weights and bias of the output perceptron should change, we can “back propagate” our desired change in output through the model. Let\u2019s start with back propagating so we can calculate how we should update w\u2081.<\/p>\n

First, we calculate how the activated output of the of the first hidden neuron should change. We do that by multiplying the change in output by w\u2087.<\/p>\n

Calculating how the activated output of the first hidden neuron should have changed by multiplying the desired change in the output by w\u2087.<\/figcaption><\/figure>\n

For values that are greater than zero, ReLU simply multiplies those values by 1. So, for this example, the change we want the un-activated value of the first hidden neuron is equal to the desired change in the activated output<\/p>\n

How much we want to change the un-activated value of the first hidden perceptron, based on back-propagating from the output.<\/figcaption><\/figure>\n

Recall that we calculated how to update w\u2087 based on multiplying it\u2019s input by the change in its desired output. We can do the same thing to calculate the change in w\u2081.<\/p>\n

Now that we\u2019ve calculated how the first hidden neuron should change, we can calculate how we should update w\u2081 the same way we calculated how w\u2087 should be updated previously.<\/figcaption><\/figure>\n

It\u2019s important to note, we\u2019re not actually updating any of the weights or biases throughout this process. Rather, we\u2019re taking a tally of how we should update each parameter, assuming no other parameters are updated.<\/p>\n

So, we can do those calculations to calculate all parameter changes.<\/p>\n

\"By
By back propagating through the model, using a combination of values from the forward passes and desired changes from the backward pass at various points of the model, we can calculate how all parameters should change<\/figcaption><\/figure>\n

A fundamental idea of back propagation is called “Learning Rate”, which concerns the size of the changes we make to neural networks based on a particular batch of data. To explain why this is important, I\u2019d like to use an analogy.<\/p>\n

Imagine you went outside one day, and everyone wearing a hat gave you a funny look. You probably don\u2019t want to jump to the conclusion that wearing hat = funny look<\/code> , but you might be a bit skeptical of people wearing hats. After three, four, five days, a month, or even a year, if it seems like the vast majority of people wearing hats are giving you a funny look, you may start considering that a strong trend.<\/p>\n

Similarly, when we train a neural network, we don\u2019t want to completely change how the neural network thinks based on a single training example. Rather, we want each batch to only incrementally change how the model thinks. As we expose the model to many examples, we would hope that the model would learn important trends within the data.<\/p>\n

After we\u2019ve calculated how each parameter should change as if it were the only parameter being updated, we can multiply all those changes by a small number, like 0.001<\/code> , before applying those changes to the parameters. This small number is commonly referred to as the “learning rate”, and the exact value it should have is dependent on the model we\u2019re training on. This effectively scales down our adjustments before applying them to the model.<\/p>\n

At this point we covered pretty much everything one would need to know to implement a neural network. Let\u2019s give it a shot!<\/p>\n

\"Join
Join IAEE<\/a><\/figcaption><\/figure>\n

Implementing a Neural Network from Scratch<\/h2>\n

Typically, a data scientist would just use a library like PyTorch<\/code> to implement a neural network in a few lines of code, but we\u2019ll be defining a neural network from the ground up using NumPy, a numerical computing library.<\/p>\n

First, let\u2019s start with a way to define the structure of the neural network.<\/p>\n

\"\"\"Blocking out the structure of the Neural Network\n\"\"\"\n\nimport numpy as np\n\nclass SimpleNN:\n    def __init__(self, architecture):\n        self.architecture = architecture\n        self.weights = []\n        self.biases = []\n\n        # Initialize weights and biases\n        np.random.seed(99)\n        for i in range(len(architecture) - 1):\n            self.weights.append(np.random.uniform(\n                low=-1, high=1,\n                size=(architecture[i], architecture[i+1])\n            ))\n            self.biases.append(np.zeros((1, architecture[i+1])))\n\narchitecture = [2, 64, 64, 64, 1]  # Two inputs, two hidden layers, one output\nmodel = SimpleNN(architecture)\n\nprint('weight dimensions:')\nfor w in model.weights:\n    print(w.shape)\n\nprint('nbias dimensions:')\nfor b in model.biases:\n    print(b.shape)<\/code><\/pre>\n
\"The
The weight and bias matrix defined in a sample neural network.<\/figcaption><\/figure>\n
While we typically draw neural networks as a dense web in reality we represent the weights between their connections as matrices. This is convenient because matrix multiplication, then, is equivalent to passing data through a neural network.<\/p>\n
\"Thinking
Thinking of a dense network as weighted connections on the left, and as matrix multiplication on the right. On the right hand side diagram, the vector on the left would be the input, the matrix in the center would be the weight matrix, and the vector on the right would be the output. Only a portion of values are included for readability. From my article on LoRA<\/a>.<\/figcaption><\/figure>\n
We can make our model make a prediction based on some input by passing the input through each layer.<\/p>\n
\"\"\"Implementing the Forward Pass\n\"\"\"\n\nimport numpy as np\n\nclass SimpleNN:\n    def __init__(self, architecture):\n        self.architecture = architecture\n        self.weights = []\n        self.biases = []\n\n        # Initialize weights and biases\n        np.random.seed(99)\n        for i in range(len(architecture) - 1):\n            self.weights.append(np.random.uniform(\n                low=-1, high=1,\n                size=(architecture[i], architecture[i+1])\n            ))\n            self.biases.append(np.zeros((1, architecture[i+1])))\n\n    @staticmethod\n    def relu(x):\n        #implementing the relu activation function\n        return np.maximum(0, x)\n\n    def forward(self, X):\n        #iterating through all layers\n        for W, b in zip(self.weights, self.biases):\n\n            #applying the weight and bias of the layer\n            X = np.dot(X, W) + b\n\n            #doing ReLU for all but the last layer\n            if W is not self.weights[-1]:\n                X = self.relu(X)\n\n        #returning the result\n        return X\n\n    def predict(self, X):\n        y = self.forward(X)\n        return y.flatten()\n\n#defining a model\narchitecture = [2, 64, 64, 64, 1]  # Two inputs, two hidden layers, one output\nmodel = SimpleNN(architecture)\n\n# Generate predictions\nprediction = model.predict(np.array([0.1,0.2]))\nprint(prediction)<\/code><\/pre>\n
\"the
the result of passing our data through the model. Our model is randomly defined, so this isn\u2019t a useful prediction, but it confirms that the model is working.<\/figcaption><\/figure>\n
We need to be able to train this model, and to do that we\u2019ll first need a problem to train the model on. I defined a random function that takes in two inputs and results in an output:<\/p>\n
\"\"\"Defining what we want the model to learn\n\"\"\"\nimport numpy as np\nimport matplotlib.pyplot as plt\n\n# Define a random function with two inputs\ndef random_function(x, y):\n    return (np.sin(x) + x * np.cos(y) + y + 3**(x\/3))\n\n# Generate a grid of x and y values\nx = np.linspace(-10, 10, 100)\ny = np.linspace(-10, 10, 100)\nX, Y = np.meshgrid(x, y)\n\n# Compute the output of the random function\nZ = random_function(X, Y)\n\n# Create a 2D plot\nplt.figure(figsize=(8, 6))\ncontour = plt.contourf(X, Y, Z, cmap='viridis')\nplt.colorbar(contour, label='Function Value')\nplt.title('2D Plot of Objective Function')\nplt.xlabel('X-axis')\nplt.ylabel('Y-axis')\nplt.show()<\/code><\/pre>\n
\"The
The modeling objective. Given two inputs (here plotted as x and y), the model needs to predict an output (here represented as color). This is a completely arbitrary function<\/figcaption><\/figure>\n
In the real world we wouldn\u2019t know the underlying function. We can mimic that reality by creating a dataset consisting of random points:<\/p>\n
import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\n# Define a random function with two inputs\ndef random_function(x, y):\n    return (np.sin(x) + x * np.cos(y) + y + 3**(x\/3))\n\n# Define the number of random samples to generate\nn_samples = 1000\n\n# Generate random X and Y values within a specified range\nx_min, x_max = -10, 10\ny_min, y_max = -10, 10\n\n# Generate random values for X and Y\nX_random = np.random.uniform(x_min, x_max, n_samples)\nY_random = np.random.uniform(y_min, y_max, n_samples)\n\n# Evaluate the random function at the generated X and Y values\nZ_random = random_function(X_random, Y_random)\n\n# Create a dataset\ndataset = pd.DataFrame({\n    'X': X_random,\n    'Y': Y_random,\n    'Z': Z_random\n})\n\n# Display the dataset\nprint(dataset.head())\n\n# Create a 2D scatter plot of the sampled data\nplt.figure(figsize=(8, 6))\nscatter = plt.scatter(dataset['X'], dataset['Y'], c=dataset['Z'], cmap='viridis', s=10)\nplt.colorbar(scatter, label='Function Value')\nplt.title('Scatter Plot of Randomly Sampled Data')\nplt.xlabel('X-axis')\nplt.ylabel('Y-axis')\nplt.show()<\/code><\/pre>\n
\"This
This is the data we\u2019ll be training on to try to learn our function.<\/figcaption><\/figure>\n
Recall that the back propagation algorithm updates parameters based on what happens in a forward pass. So, before we implement backpropagation itself, let\u2019s keep track of a few important values in the forward pass: The inputs and outputs of each perceptron throughout the model.<\/p>\n
import numpy as np\n\nclass SimpleNN:\n    def __init__(self, architecture):\n        self.architecture = architecture\n        self.weights = []\n        self.biases = []\n\n        #keeping track of these values in this code block\n        #so we can observe them\n        self.perceptron_inputs = None\n        self.perceptron_outputs = None\n\n        # Initialize weights and biases\n        np.random.seed(99)\n        for i in range(len(architecture) - 1):\n            self.weights.append(np.random.uniform(\n                low=-1, high=1,\n                size=(architecture[i], architecture[i+1])\n            ))\n            self.biases.append(np.zeros((1, architecture[i+1])))\n\n    @staticmethod\n    def relu(x):\n        return np.maximum(0, x)\n\n    def forward(self, X):\n        self.perceptron_inputs = [X]\n        self.perceptron_outputs = []\n\n        for W, b in zip(self.weights, self.biases):\n            Z = np.dot(self.perceptron_inputs[-1], W) + b\n            self.perceptron_outputs.append(Z)\n\n            if W is self.weights[-1]:  # Last layer (output)\n                A = Z  # Linear output for regression\n            else:\n                A = self.relu(Z)\n            self.perceptron_inputs.append(A)\n\n        return self.perceptron_inputs, self.perceptron_outputs\n\n    def predict(self, X):\n        perceptron_inputs, _ = self.forward(X)\n        return perceptron_inputs[-1].flatten()\n\n#defining a model\narchitecture = [2, 64, 64, 64, 1]  # Two inputs, two hidden layers, one output\nmodel = SimpleNN(architecture)\n\n# Generate predictions\nprediction = model.predict(np.array([0.1,0.2]))\n\n#looking through critical optimization values\nfor i, (inpt, outpt) in enumerate(zip(model.perceptron_inputs, model.perceptron_outputs[:-1])):\n    print(f'layer {i}')\n    print(f'input: {inpt.shape}')\n    print(f'output: {outpt.shape}')\n    print('')\n\nprint('Final Output:')\nprint(model.perceptron_outputs[-1].shape)<\/code><\/pre>\n
\"The
The values throughout various layers of the model as a result of the forward pass. This will allow us to compute the necessary changes to update the model.<\/figcaption><\/figure>\n
Now that we have a record stored of critical intermediary value within the network, we can use those values, along with the error of a model for a particular prediction, to calculate the changes we should make to the model.<\/p>\n
import numpy as np\n\nclass SimpleNN:\n    def __init__(self, architecture):\n        self.architecture = architecture\n        self.weights = []\n        self.biases = []\n\n        # Initialize weights and biases\n        np.random.seed(99)\n        for i in range(len(architecture) - 1):\n            self.weights.append(np.random.uniform(\n                low=-1, high=1,\n                size=(architecture[i], architecture[i+1])\n            ))\n            self.biases.append(np.zeros((1, architecture[i+1])))\n\n    @staticmethod\n    def relu(x):\n        return np.maximum(0, x)\n\n    @staticmethod\n    def relu_as_weights(x):\n        return (x > 0).astype(float)\n\n    def forward(self, X):\n        perceptron_inputs = [X]\n        perceptron_outputs = []\n\n        for W, b in zip(self.weights, self.biases):\n            Z = np.dot(perceptron_inputs[-1], W) + b\n            perceptron_outputs.append(Z)\n\n            if W is self.weights[-1]:  # Last layer (output)\n                A = Z  # Linear output for regression\n            else:\n                A = self.relu(Z)\n            perceptron_inputs.append(A)\n\n        return perceptron_inputs, perceptron_outputs\n\n    def backward(self, perceptron_inputs, perceptron_outputs, target):\n        weight_changes = []\n        bias_changes = []\n\n        m = len(target)\n        dA = perceptron_inputs[-1] - target.reshape(-1, 1)  # Output layer gradient\n\n        for i in reversed(range(len(self.weights))):\n            dZ = dA if i == len(self.weights) - 1 else dA * self.relu_as_weights(perceptron_outputs[i])\n            dW = np.dot(perceptron_inputs[i].T, dZ) \/ m\n            db = np.sum(dZ, axis=0, keepdims=True) \/ m\n            weight_changes.append(dW)\n            bias_changes.append(db)\n\n            if i > 0:\n                dA = np.dot(dZ, self.weights[i].T)\n\n        return list(reversed(weight_changes)), list(reversed(bias_changes))\n\n    def predict(self, X):\n        perceptron_inputs, _ = self.forward(X)\n        return perceptron_inputs[-1].flatten()\n\n#defining a model\narchitecture = [2, 64, 64, 64, 1]  # Two inputs, two hidden layers, one output\nmodel = SimpleNN(architecture)\n\n#defining a sample input and target output\ninput = np.array([[0.1,0.2]])\ndesired_output = np.array([0.5])\n\n#doing forward and backward pass to calculate changes\nperceptron_inputs, perceptron_outputs = model.forward(input)\nweight_changes, bias_changes = model.backward(perceptron_inputs, perceptron_outputs, desired_output)\n\n#smaller numbers for printing\nnp.set_printoptions(precision=2)\n\nfor i, (layer_weights, layer_biases, layer_weight_changes, layer_bias_changes)\nin enumerate(zip(model.weights, model.biases, weight_changes, bias_changes)):\n    print(f'layer {i}')\n    print(f'weight matrix: {layer_weights.shape}')\n    print(f'weight matrix changes: {layer_weight_changes.shape}')\n    print(f'bias matrix: {layer_biases.shape}')\n    print(f'bias matrix changes: {layer_bias_changes.shape}')\n    print('')\n\nprint('The weight and weight change matrix of the second layer:')\nprint('weight matrix:')\nprint(model.weights[1])\nprint('change matrix:')\nprint(weight_changes[1])<\/code><\/pre>\n
\"\"<\/figure>\n
This is probably the most complex implementation step, so I want to take a moment to dig through some of the details. The fundamental idea is exactly as we described in previous sections. We\u2019re iterating over all layers, from back to front, and calculating what change to each weight and bias would result in a better output.<\/p>\n
# calculating output error\ndA = perceptron_inputs[-1] - target.reshape(-1, 1)\n\n#a scaling factor for the batch size.\n#you want changes to be an average across all batches\n#so we divide by m once we've aggregated all changes.\nm = len(target)\n\nfor i in reversed(range(len(self.weights))):\n  dZ = dA #simplified for now\n\n  # calculating change to weights\n  dW = np.dot(perceptron_inputs[i].T, dZ) \/ m\n  # calculating change to bias\n  db = np.sum(dZ, axis=0, keepdims=True) \/ m\n\n  # keeping track of required changes\n  weight_changes.append(dW)\n  bias_changes.append(db)\n  ...<\/code><\/pre>\n
Calculating the change to bias is pretty straight forward. If you look at how the output of a given neuron should have impacted all future neurons, you can add up all those values (which are both positive and negative) to get an idea of if the neuron should be biased in a positive or negative direction.<\/p>\n
The way we calculate the change to weights, by using matrix multiplication, is a bit more mathematically complex.<\/p>\n
dW = np.dot(perceptron_inputs[i].T, dZ) \/ m<\/code><\/pre>\n
Basically, this line says that the change in the weight should be equal to the value going into the perceptron, times how much the output should have changed. If a perceptron had a big input, the change to its outgoing weights should be a large magnitude, if the perceptron had a small input, the change to its outgoing weights will be small. Also, if a weight points towards an output which should change a lot, the weight should change a lot.<\/p>\n
There is another line we should discuss in our back propagation implement.<\/p>\n
dZ = dA if i == len(self.weights) - 1 else dA * self.relu_as_weights(perceptron_outputs[i])<\/code><\/pre>\n
In this particular network, there are activation functions throughout the network, following all but the final output. When we do back propagation, we need to back-propagate through these activation functions so that we can update the neurons which lie before them. We do this for all but the last layer, which doesn\u2019t have an activation function, which is why dZ = dA if i == len(self.weights) - 1<\/code> .<\/p>\n
In fancy math speak we would call this a derivative, but because I don\u2019t want to get into calculus, I called the function relu_as_weights<\/code> . Basically, we can treat each of our ReLU activations as something like a tiny neural network, who\u2019s weight is a function of the input. If the input of the ReLU activation function is less than zero, then that\u2019s like passing that input through a neural network with a weight of zero. If the input of ReLU is greater than zero, then that\u2019s like passing the input through a neural netowork with a weight of one.<\/p>\n
\"Recall
Recall the ReLU activation function.<\/figcaption><\/figure>\n
This is exactly what the relu_as_weights<\/code> function does.<\/p>\n
def relu_as_weights(x):\n        return (x > 0).astype(float)<\/code><\/pre>\n
Using this logic we can treat back propagating through ReLU just like we back propagate through the rest of the neural network.<\/p>\n
Again, I\u2019ll be covering this concept from a more robust mathematical prospective soon, but that\u2019s the essential idea from a conceptual perspective.<\/p>\n
Now that we have the forward and backward pass implemented, we can implement training the model.<\/p>\n
import numpy as np\n\nclass SimpleNN:\n    def __init__(self, architecture):\n        self.architecture = architecture\n        self.weights = []\n        self.biases = []\n\n        # Initialize weights and biases\n        np.random.seed(99)\n        for i in range(len(architecture) - 1):\n            self.weights.append(np.random.uniform(\n                low=-1, high=1,\n                size=(architecture[i], architecture[i+1])\n            ))\n            self.biases.append(np.zeros((1, architecture[i+1])))\n\n    @staticmethod\n    def relu(x):\n        return np.maximum(0, x)\n\n    @staticmethod\n    def relu_as_weights(x):\n        return (x > 0).astype(float)\n\n    def forward(self, X):\n        perceptron_inputs = [X]\n        perceptron_outputs = []\n\n        for W, b in zip(self.weights, self.biases):\n            Z = np.dot(perceptron_inputs[-1], W) + b\n            perceptron_outputs.append(Z)\n\n            if W is self.weights[-1]:  # Last layer (output)\n                A = Z  # Linear output for regression\n            else:\n                A = self.relu(Z)\n            perceptron_inputs.append(A)\n\n        return perceptron_inputs, perceptron_outputs\n\n    def backward(self, perceptron_inputs, perceptron_outputs, y_true):\n        weight_changes = []\n        bias_changes = []\n\n        m = len(y_true)\n        dA = perceptron_inputs[-1] - y_true.reshape(-1, 1)  # Output layer gradient\n\n        for i in reversed(range(len(self.weights))):\n            dZ = dA if i == len(self.weights) - 1 else dA * self.relu_as_weights(perceptron_outputs[i])\n            dW = np.dot(perceptron_inputs[i].T, dZ) \/ m\n            db = np.sum(dZ, axis=0, keepdims=True) \/ m\n            weight_changes.append(dW)\n            bias_changes.append(db)\n\n            if i > 0:\n                dA = np.dot(dZ, self.weights[i].T)\n\n        return list(reversed(weight_changes)), list(reversed(bias_changes))\n\n    def update_weights(self, weight_changes, bias_changes, lr):\n        for i in range(len(self.weights)):\n            self.weights[i] -= lr * weight_changes[i]\n            self.biases[i] -= lr * bias_changes[i]\n\n    def train(self, X, y, epochs, lr=0.01):\n        for epoch in range(epochs):\n            perceptron_inputs, perceptron_outputs = self.forward(X)\n            weight_changes, bias_changes = self.backward(perceptron_inputs, perceptron_outputs, y)\n            self.update_weights(weight_changes, bias_changes, lr)\n\n            if epoch % 20 == 0 or epoch == epochs - 1:\n                loss = np.mean((perceptron_inputs[-1].flatten() - y) ** 2)  # MSE\n                print(f\"EPOCH {epoch}: Loss = {loss:.4f}\")\n\n    def predict(self, X):\n        perceptron_inputs, _ = self.forward(X)\n        return perceptron_inputs[-1].flatten()<\/code><\/pre>\n
The train<\/code> function:<\/p>\n