Gradient through many steps
How much the state at the last step T depends on the state at an earlier step t. The chain rule makes it a product of one factor per step, which is why RNN gradients vanish or explode.
The chain rule says the derivative through a sequence of functions is the product of each function's derivative. Across recurrent steps, that is matrices multiplied together.
If each factor shrinks things (roughly, a size ), the product shrinks like . The gradient reaching step is then effectively zero, and the model cannot learn that step affected step . If each factor is above 1, the product grows without limit instead. Keeping every factor near exactly 1 over long sequences is very hard. This is a main reason attention replaced recurrence: attention connects any two positions in a single step.
Example
Fifty steps with a factor slightly below or above 1.
Where the course uses it
Related terms
- ∂ — partial derivativeRead "partial L by partial θ". It is how much the top quantity changes when you nudge the bottom one slightly, keeping every other input fixed.
- ∏ — productRead "the product of". The big pi works like ∑, but it multiplies the terms instead of adding them.
- ∇ — gradientRead "nabla" or "the gradient of". It is the list of every partial derivative at once, and it points in the direction that increases the function fastest.