Processing sequences one step at a time
Feedforward neural networks take a fixed-size input and produce a fixed-size output. That design works well for images or spreadsheet rows, but text does not come in fixed sizes. A sentence can be five words long or five hundred words long. We need a model that can handle any length.
Recurrent neural networks, or RNNs, solve this by reading one token at a time while maintaining an internal memory called the hidden state. At every timestep the network receives the current token and the previous hidden state, and it produces a new hidden state and optionally an output.
In this equation, x_t is the input vector for the token at position t, h_{t-1} is the hidden state from the previous step, and h_t is the updated hidden state. The non-linearity is typically tanh, which squashes the values into the range (-1, 1). The same weight matrices W_h and W_x are reused at every timestep, a property called weight sharing.
Because the same parameters are used for every position, an RNN can process a sequence of any length. It does not need a separate set of weights for the first word, the tenth word, and the hundredth word. This makes RNNs elegant and compact in terms of parameter count.
The hidden state is not a database
The hidden state is a compressed summary of the sequence so far. It is a fixed-size vector, no matter how long the input has been. The model must learn to throw away irrelevant details and keep the information that will be useful for future predictions.
Sequence modeling tasks
RNNs are not limited to language modeling. Depending on how inputs and outputs are paired, the same recurrent cell supports several sequence tasks.
- Many-to-one: an entire sequence is compressed into a single output. Sentiment analysis and document classification use this shape. The final hidden state becomes the input to a classifier.
- One-to-many: a single input vector is expanded into a sequence. Image captioning starts with a vector from a CNN and generates a sentence token by token.
- Many-to-many (synchronized): an output is produced for every input. Named-entity recognition and part-of-speech tagging align labels with each token.
- Many-to-many (encoder-decoder): input and output sequences have different lengths. Machine translation and summarization map a source sentence to a target sentence.
The choice of shape changes the loss and the inference procedure, but the underlying RNN cell stays the same. This flexibility is one reason RNNs became so widely used before Transformers.
Unrolling through time
Training an RNN means applying backpropagation to an unrolled version of the network. If a sentence has one hundred tokens, the computational graph has one hundred copies of the RNN cell connected in a chain. Mentally, you can picture the network laid out from left to right, one cell per word.
This process is called backpropagation through time, or BPTT. It is just the chain rule, but instead of flowing backward through layers, the gradients flow backward through time steps. The gradient at the last token is propagated all the way back to the first token, passing through every intermediate cell along the way.
The unrolled picture makes it clear why RNNs are hard to train. A one-hundred-token sentence produces a one-hundred-layer network. Deep feedforward networks already struggle with gradients; here the depth is the length of the input itself.
The vanishing gradient
During backpropagation, the loss gradient travels backward from the final hidden state toward the earliest hidden state. At every timestep it gets multiplied by another Jacobian matrix: the derivative of the current hidden state with respect to the previous one.
If the largest singular value of each Jacobian is less than one, repeated multiplication makes the gradient shrink exponentially. After fifty or one hundred steps the gradient becomes so small that the parameters at the beginning of the sequence receive almost no update signal.
This is the vanishing gradient problem. It means that a plain RNN cannot reliably learn long-range dependencies. Information from the first word of a long sentence has very little influence on the loss by the time we reach the last word, so the model cannot learn to connect them.
The sentence dependency problem
Consider the sentence "The cat, which was black and fluffy and rather proud of itself, sat on the mat." By the time the network reaches "sat", the subject "cat" has traveled through many timesteps. In a plain RNN that gradient has faded, so the verb has a weak connection to its subject.
Bidirectional RNNs
A single RNN reads the sentence from left to right, so the hidden state at word t only sees words 1 through t. That makes it blind to future context, which humans use heavily when disambiguating language. A bidirectional RNN runs two RNNs in parallel: one left-to-right and one right-to-left.
The two final hidden states are concatenated into a single vector that captures both past and future context. This is especially useful for tasks like named-entity recognition, where the word Paris is a location if surrounded by travel verbs and a name if surrounded by person verbs.
From RNN to gated memory
Plain RNNs are elegant, but the vanishing gradient limits how far back they can remember. The next chapter introduces Long Short-Term Memory networks, which add a protected cell state and gating mechanisms to create a gradient highway across long sequences. That improvement leads naturally to sequence-to-sequence models and, eventually, to attention.
Key takeaway
RNNs process variable-length sequences by reusing the same weights at every timestep and maintaining a hidden state. Weight sharing makes them compact, but the repeated matrix multiplications in backpropagation through time cause gradients to vanish, limiting their ability to capture long-range dependencies.
What comes next
The next chapter adds gating to the RNN cell, turning it into an LSTM. The LSTM cell state acts like a protected conveyor belt, allowing information to travel many timesteps with only minimal change.