In a neural network trained using backpropagation, computations occur in the Input, Hidden, and Output Layers. Let's walk through the process step by step while considering the sigmoidal activation function and weight adjustments.
1. **Forward Pass (Feedforward):**
- Input Layer: The input values are passed directly to the Input Layer neurons.
- Hidden Layers: Neurons in each Hidden Layer receive inputs from the previous layer, apply the sigmoidal activation function, and pass the result to the next layer.
- For a single neuron in a Hidden Layer, the computation would look like this:
- Calculate the weighted sum of inputs: `weighted_sum = Σ(input * weight) + bias`
- Apply the sigmoidal activation function: `output = 1 / (1 + e^(-weighted_sum))`, where `e` is the base of the natural logarithm.
- Output Layer: The Output Layer follows the same process as the Hidden Layers.
2. **Calculate Error (Loss):**
- After the feedforward pass, you compare the network's output to the expected output to calculate the error or loss. The choice of loss function depends on the specific problem (e.g., Mean Squared Error for regression, Cross-Entropy for classification).
3. **Backpropagation (Error Backpropagation):**
- The goal is to minimize the error, and this is done by adjusting the weights in a way that reduces the error in subsequent iterations.
- Starting from the Output Layer and moving backward through the network:
- Calculate the gradient of the loss with respect to the output of each neuron in the Output Layer. This gradient is often computed based on the chosen loss function.
- Propagate this gradient backward through the network to compute the gradients for each neuron in the Hidden Layers.
- For each neuron in a layer, the weight adjustment is determined by the gradient of the error with respect to that neuron's output. This is typically done using the chain rule of calculus.
- Adjust each weight by subtracting the gradient times a learning rate (a small positive value that controls the step size of weight updates) and update the bias as well. The learning rate prevents weight updates that are too large and ensures convergence.
Mathematically, the weight update for a single weight `w` associated with a neuron would be:
```
Δw = -learning_rate * gradient * output_of_neuron
new_weight = old_weight + Δw
```
4. **Repeat:**
- Steps 1 to 3 are repeated for multiple iterations or epochs until the network's error converges to an acceptable level.
This process of forward pass, error calculation, and backpropagation is iterated upon many times during training to optimize the neural network's weights and biases. The sigmoidal activation function is used to introduce non-linearity into the network, allowing it to model complex relationships in the data. Adjusting the weights based on the calculated gradients is the key to training the network to make accurate predictions.