D-DL4CV-Lec12c-VanillaRNNs

What is Vanilla RNNs?

$$\begin{array}{rl}{h}_t& =\tanh (W_{hh}{h}_{t-1}+W_{xh}{x}_t+b_h)\\ y_t& =W_{hy}{h}_t+b_y\end{array}$$

---

Vanilla RNNs Gradient Flow

For computation efficiency, we simplify $h_t$ calculation with:

$$\begin{array}{rl}{h}_t& =\tanh (W_{hh}{h}_{t-1}+W_{xh}{x}_t+b_h)\\ & =\tanh \left(\left(\begin{array}{cc}{W}_{hh}& W_{hx}\end{array}\right)\left(\begin{array}{c}{h}_{t-1}\\ x_t\end{array}\right)+b_h\right)\\ & =\tanh \left(W\left(\begin{array}{c}{h}_{t-1}\\ x_t\end{array}\right)+b_h\right)\end{array}$$

---

Problem with Vanilla RNNs Gradient Flow

Repeated $\tanh$ and $W$ in backpropagation

When doing backpropagation, we can observe from the computational graph that we’ll need to repeatedly multiply the gradient by $d\tanh (x)/dx$ and $dy/dW$

Problem with $d\tanh (x)/dx$

• Saturated neurons “kill” the gradients

Problem with $dy/dW$

Largest singular value > 1 → Exploding Gradients Largest singular value < 1 → Vanishing Gradients

Solution

We give up Vanilla RNNs and use LSTM instead