Backpropagation

The following are my notes I prepared for the students as part of tutoring the Neural Networks class at Saarland University in fall 2023. Originally typset in LaTeX I decided that they are also a nice addition to my website. Most of the content is a more detailed discussion of the backpropagation section in Christopher Bishops excellent PRML Bishop, 2006 book with the addition of a few examples.

Consider a fully connected feed-forward neural network with one hidden layer with $M$ neurons, $D$ inputs and $K$ output units. A network of this is from depicted in Figure 1. A forward pass involves the following calculations.

Here $\mathbf{x} \in \mathbb{R}^{D}, \mathbf{a}^{(1)}, \mathbf{z} \in \mathbb{R}^{M}, \mathbf{a}^{(2)}, \mathbf{\hat{y}} \in \mathbb{R}^{K}$ and $\boldsymbol{W}^{(1)} \in \mathbb{R}^{M \times D}, \boldsymbol{W}^{(2)} \in \mathbb{R}^{K \times M}$. And $h^{(1)}$ is the activation function after the first set of activations $\mathbf{a}^{(1)}$ and $h^{(2)}$ is the output activation function. Now we will consider a separable loss or error function $\mathrm{J}$.

To minimize the loss we are interested in gradients of the weight matrices. From now on we will only consider the loss $\mathrm{J}_n$ of a single sample $(\mathbf{x}_n, \mathbf{\hat{y}}_n)$. Let’s first consider the loss w.r.t. entry $\boldsymbol{W}^{(2)}_{ji}$. Because $\boldsymbol{W}^{(2)}_{ji}$ influences the output $\mathbf{\hat{y}}$ only through activation $a_j^{(2)}$, by applying the chain rule we may also write:

We can further simplify by again applying the chain rule, this time to the first term:

Here $\frac{\partial \mathrm{J}_n}{\partial \hat{y}_j}$ depends on the loss function. The derivatives of e.g. the squared or the cross-entropy error function read:

The second term $\frac{\partial \hat{y}_j}{\partial a_j^{(2)}}$ is determined solely by the output activation function $h^{(2)}$ and reduces to 1 if it is the identity function $h^{(2)}\left(\mathbf{a^{(2)}} \right) = \mathbf{a^{(2)}}$ (as is usually the case in regression).
The remaining term is trivial to calculate:

Therefore the partial derivative for entry $ij$ of $\boldsymbol{W}^{(2)}$ reads:

We denote the term with $\delta^{(2)}_j := \frac{\partial \mathrm{J}_n}{\partial a_j^{(2)}}$ and call it the error term. The calculation of the partial derivatives of $\boldsymbol{W}^{(1)}_{ji}$ is a little more involved because it requires careful considerations, which terms are influenced by $\boldsymbol{W}^{(1)}_{ji}$ in later layers.
As before, $\boldsymbol{W}^{(1)}_{ji}$ influences $\mathbf{\hat{y}}$ only through $a^{(1)}_j$, hence we may write:

To begin with, we focus on the first term. Notice that $\mathrm{J}_n$ depends on $a^{(1)}_j$ only through $z_j$:

This time however, $z_j$ is involved in the calculation of all output units $y_k$ by $a^{(2)}_k = \sum_{m=1}^M \boldsymbol{W}^{(2)}_{km}z_m$. Therefore, to express $\frac{\partial \mathrm{J}_n}{\partial z_j}$ in terms of $a^{(2)}_k$, we need to sum over all $k=1,\dots, K$:

By definition, we know $\frac{\partial \mathrm{J}_n}{\partial a^{(2)}_k} = \delta^{(2)}_k$. It’s easy to verify that $\frac{\partial z_j}{\partial a^{(1)}_j} = h^{(1)\prime}(a^{(1)}_j)$ and similarly by writing out the definition of $\frac{\partial a^{(2)}_k}{\partial z_j}$ it’s clear that the result of that term is $\boldsymbol{W}^{(2)}_{kj}$. Therefore we obtain:

Only the partial derivative $\frac{\partial a^{(1)}_j}{\partial \boldsymbol{W}^{(1)}_{ji}}$ is left, however the calculation is trivial:

Consequently, every entry $ij$ of $\boldsymbol{W}^{(1)}$ has partial derivative:

Note, although we assumed a single hidden layer neural network, the partial derivatives for deeper layers take the same form as for $\boldsymbol{W}^{(1)}$, this can be seen by defining $\mathbf{x} = \mathbf{z}^{(0)} = h^{(0)}\left(\mathbf{a}^{(0)}\right)$ as the output of the previous layer, with $\mathbf{a}^{(0)} = \boldsymbol{W}^{0}\mathbf{z}^{(-1)}$.