Residual Blocks

The building blocks of ResNets, which consists of layers connected by a skip connection and the layers in between.

These residual blocks mitigate the Vanishing Gradient Problem by having Skip Connections which adds input from a shallow layer to a deeper layer. This makes the gradients of the deeper layers have a higher value and do not vanish to zero before the gradient of the shallow layer does.


Examples

Our “network” will consist of just one residual block for simplicity. We define our residual block function as: $f(x)=a{x}^2+b$ where:

• $x$ is the input to the block,
• $a$ and $b$ are parameters of the function (similar to weights in a convolution layer),
• and the residual block’s output is $f(x)+x$.

We use the mean squared error (MSE) as our loss function: $L=\frac{1}{2}(f(x)+x-y{)}^2$where $y$ is the true value we’re trying to predict. Optimisation with Gradient Descent We will optimise the parameters $a$ and $b$ using gradient descent. We need to calculate the gradients of $L$ with respect to $a$ and $b$, and then update $a$ and $b$ in the direction of the steepest decrease of $L$.

Calculate the Gradients:

• Gradient of $L$ with respect to $a$ : $\frac{\partial L}{\partial a}=\frac{\partial L}{\partial (f(x)+x)}\cdot \frac{\partial (f(x)+x)}{\partial f(x)}\cdot \frac{\partial f(x)}{\partial a}=(f(x)+x-y)\cdot 1\cdot x^2$
• Gradient of $L$ with respect to $b$: $\frac{\partial L}{\partial b}=\frac{\partial L}{\partial (f(x)+x)}\cdot \frac{\partial (f(x)+x)}{\partial f(x)}\cdot \frac{\partial f(x)}{\partial b}=(f(x)+x-y)\cdot 1\cdot 1$

This example illustrates how residual blocks help mitigate the Vanishing Gradient Problem by adding the input $x$ to the gradients of early layers. We also see that the network is minimising the loss function with regard to $f(x)+x$ instead of $f(X)$.