Sequence Models (RNN, LSTM)

Sequence Models

Recurrent Neural Networks (RNN)

Process sequences by maintaining hidden state: h_t = tanh(W_h × h_{t-1} + W_x × x_t)

Problem: Vanishing gradients for long sequences

Long Short-Term Memory (LSTM)

Solves vanishing gradient with gates:

Gated Recurrent Unit (GRU)

Simplified LSTM:

Limitations

This led to → Transformers