Activation Function

The activation function $g$ is a non-linear function, which ensures that the neuron can not strictly pass linear transformations of the inputted examples through the network, which would otherwise make it impossible to learn non-linear functions.

While the activation function for the hidden layers is a design choice, the ones for the output layer are defined by the learning task: - If the NN is to perform regression then the activation function (or none at all) should allow the output space to be $\mathbb{R}$. - If the NN is to perform classification then we are typically interested in an output that represents the probability of an example belonging to a class, meaning that we want the mapping $\mathbb{R}\to [0,1]$: - for binary classification, a sigmoid function is used. - for multi-class classification, the soft-max function is used.

Examples

• Logistic sigmoid $$g(a)=\sigma (a)\equiv \frac{1}{1+e^{-a}}$$
• Softmax for $K$ classes $$g(a{)}_i=\frac{e^{a_i}}{{\sum }_{j=1}^K{e}^{a_j}}\text{for}i=1,\dots ,K$$
• Rectified linear unit (ReLU) $$g(a)=\max \left(0,a\right)$$ where $a$ is the activation value calculated by the neuron using $\mathbf{w}$ and $\mathbf{b}$.