A Neural network is a machine learning algorithm inspired by the biological neural network of the brain. A neural network is a collection of nodes, called Neurons, that transmit and process signals to and from other Neurons connected to it, just like the biological neural network.
The most simple neural networks are called feedforward neural networks since the signals flow in one direction through the Neurons. These Neurons are arranged in layers to indicate which Neurons receive and send data to which other Neurons. One can therefore mathematically describe such a model as a chain of functions for a NN with hidden layers, where is the first layer, i.e. the input layer, which passes its calculation to and so on finally on to the output layer .
The number of hidden layers and the amount of Neurons in each hidden layer is not specified by theory and is a design choice.
A feedforward NN with is can be seen below:
The gradient for the loss function is not readily available in Neural Networks as the loss is calculated for the output while weights and bias are used throughout the network. So to train neural networks using Stochastic Gradient Descent we need to trace the information back through the network, which is done using Backpropagation proposed by Rummelhart (1986).