Is a phenomenon that happens in deep Feedforward Neural Networks and Recurrent Neural Networks when gradients are communicated through the many layers by Backpropagation.
Since Backpropagation is an application of the chain rule of calculus through the network, we have that the calculation of gradients at the first layers becomes a large multiplication of the gradients of the deeper layers. This means that as the model is trained and gradients become close to zero it is possible to experience that the multiplication of many close-to-zero values results in gradients of the first layers quickly becoming a negligible value than when calculated.
This makes the gradients of the first layer have limited influence on the training which can result in: - Slow convergence. - Limited learning capacity as the network can stop learning completely in the worst case. - Difficulty in capturing long-term dependencies in Recurrent Neural Networks.