LSTMs & GRUs in Python

A SimpleRNN theoretically remembers everything from the start of a sequence, but in practice it forgets almost immediately. Each timestep squashes the hidden state through a tanh activation and multiplies it against a weight matrix, and after a few dozen steps the gradients used to update those early weights either vanish toward zero or explode toward infinity. The network effectively suffers from amnesia: it can't connect a word at the start of a paragraph to one at the end.

The LSTM (Long Short-Term Memory), introduced by Hochreiter and Schmidhuber, fixes this with a dedicated memory pathway that isn't repeatedly squashed by an activation function on every step. from tensorflow.keras.layers import LSTM and model.add(LSTM(64)) is literally a drop-in replacement for SimpleRNN(64) — same input shape (timesteps, features), same general usage — but internally it manages two separate memory signals instead of one.

This is why LSTMs (and their close relative, the GRU) replaced SimpleRNN in the vast majority of real sequence-modeling work: text generation, machine translation, and time-series forecasting all depend on carrying information across dozens or hundreds of timesteps, which a plain RNN simply cannot do reliably.

Where a SimpleRNN passes a single hidden state from timestep to timestep, an LSTM maintains two parallel signals. The Cell State is the long-term memory: it behaves like a conveyor belt running the length of the sequence, and information can travel along it largely unchanged unless a gate deliberately modifies it. The Hidden State is the short-term, working memory: it's the filtered, immediately-relevant view of the Cell State that gets used for the current prediction and passed to the next timestep.

This separation is the actual engineering trick. In a SimpleRNN, every timestep forces the entire memory through a nonlinear squashing function, which is exactly what causes gradients to vanish over long sequences. The Cell State, by contrast, is updated with mostly additive operations (add this, remove that) rather than being rewritten from scratch every step, so a gradient can flow backward through many timesteps without shrinking to zero.

Think of the Cell State as the notebook the network keeps for the whole sequence, and the Hidden State as the sticky note it glances at to decide what to output right now.

During backpropagation through time, a SimpleRNN repeatedly multiplies the same weight matrix and derivative of tanh at every timestep. Multiply a number slightly less than 1 by itself fifty times and it collapses to zero — that's the vanishing gradient problem, and it's why plain RNNs can't learn long-range dependencies.

The Cell State sidesteps this because its update rule is additive rather than purely multiplicative: cell_state = (forget_gate * old_cell_state) + (input_gate * candidate_values). The gradient flowing backward through the addition doesn't get crushed the way it does when passing through a chain of matrix multiplications and tanh squashes. The 'Gates' (Forget, Input, Output) are what control this addition — they're small sigmoid-based neural networks that decide, at each step, how much of the old memory to keep and how much new information to write in.

This is the core architectural insight of the LSTM: separate the 'what to remember' decision (the gates) from the 'how memory is carried' mechanism (the mostly-additive Cell State), and long-range gradients survive the trip.

The LSTM controls its memory using mathematical 'Gates' — small feed-forward layers that apply a sigmoid activation, squashing their output into the range 0 to 1. A gate output of 0 means 'block everything,' 1 means 'let everything through,' and anything in between is a partial, learned filter. Each gate takes the previous Hidden State and the current input, concatenates them, and passes the result through its own weight matrix and sigmoid.

The first gate in the pipeline is the Forget Gate. It looks at the new input together with the previous Hidden State and decides what to discard from the Cell State — for example, when a new sentence subject appears, the network can learn to forget the grammatical gender of the previous subject because it's no longer relevant.

Crucially, these gates are learned, not hand-coded. During training, backpropagation adjusts each gate's weights so that the network discovers on its own which information matters for long-term memory and which is noise to discard.

The Forget Gate's job is narrow but critical: it outputs a vector of numbers between 0 and 1, one per unit in the Cell State, and that vector is multiplied elementwise against the existing Cell State. A 0 at a given position means 'permanently erase this piece of long-term memory,' a 1 means 'keep it exactly as is.'

Concretely, forget_vector = sigmoid(W_f · [hidden_state, input] + b_f) and then cell_state = forget_vector * cell_state. Because this happens before any new information is added, the Forget Gate is effectively doing memory garbage collection — clearing out space in the Cell State for whatever the Input Gate decides is worth writing next.

Without this gate, the Cell State would just keep accumulating information forever and saturate, since nothing would ever be actively removed. Learning what to forget is just as important to sequence modeling as learning what to remember.

Once the Forget Gate has cleared out irrelevant old information, the Input Gate decides what new information from the current timestep deserves to be written onto the Cell State. It actually works in two parts: a sigmoid layer decides which positions in the Cell State should be updated (the 'gate' itself), and a separate tanh layer generates a vector of candidate values — new information that could be added.

These two pieces are combined elementwise: cell_state = cell_state + (input_gate * candidate_values). The sigmoid controls how much of the tanh's candidate information actually gets written in, position by position, just like the Forget Gate controlled how much old information got erased.

Together, the Forget and Input Gates form a complete read-modify-write cycle on the Cell State every single timestep: forget what's stale, write what's new, and carry the rest forward untouched.

Zooming into the Input Gate's two components: the sigmoid layer, often written i_t = sigmoid(W_i · [hidden_state, input] + b_i), produces a value between 0 and 1 for each Cell State position — this is the actual 'gate.' Separately, a tanh layer produces candidate_t = tanh(W_c · [hidden_state, input] + b_c), a vector of proposed new values scaled between -1 and 1.

The two are multiplied elementwise, i_t * candidate_t, before being added to the Cell State. This two-step design matters: the tanh layer proposes what new information could be relevant, while the sigmoid layer independently decides how much of that proposal is actually trustworthy enough to commit to long-term memory.

Splitting 'what could be added' from 'how much should be added' is exactly the same read-modify-write pattern the Forget Gate uses, just running in the opposite direction — writing instead of erasing.

With the Forget Gate, Input Gate, and Cell State mechanics covered, the natural next question is a practical one: is the full LSTM always the right choice? Every gate in an LSTM has its own weight matrix, which means three full gates (Forget, Input, Output) worth of parameters to learn and compute at every single timestep, on every single sequence, in every batch.

That cost is worth it when a task genuinely needs the LSTM's fine-grained control over memory, but it isn't always necessary. Before reaching for a lighter architecture, it's worth understanding exactly what the LSTM's extra machinery is buying you, so you can judge when a simpler design gives up too little to be worth the trade.

This sets up the comparison in the next section: the GRU (Gated Recurrent Unit), a streamlined cousin of the LSTM that keeps the gating idea but merges some of the machinery together.

The GRU (Gated Recurrent Unit) simplifies the LSTM in two ways. First, it merges the Forget and Input Gates into a single Update Gate that decides, in one step, how much of the old state to keep versus how much new candidate information to write in — instead of computing those as two independent decisions. Second, it drops the separate Cell State entirely and folds long-term and short-term memory into one Hidden State.

Fewer gates means fewer weight matrices, which means fewer parameters to train and less compute per timestep. In Keras this is just as simple to use as the LSTM: from tensorflow.keras.layers import GRU and model.add(GRU(64)) slots in wherever an LSTM layer would go, with the same (timesteps, features) input shape.

In practice, GRUs often match LSTM accuracy on small-to-medium datasets while training noticeably faster, which is why they're a common first thing to try when an LSTM is too slow but a SimpleRNN isn't powerful enough.

This is the decision engineers actually face: training is too slow with an LSTM, but swapping down to a bare SimpleRNN causes the model to fail outright because the task needs longer memory than a SimpleRNN can provide. The GRU is the right move in that situation — it keeps the gating mechanism that lets gradients survive across long sequences, while cutting the parameter count enough to meaningfully speed up training and inference.

Swapping is nearly free in code: replace LSTM(64) with GRU(64) and keep everything else — input shape, surrounding Dense layers, loss function — unchanged. Because both layers accept the same (batch, timesteps, features) input and produce compatible outputs, this is genuinely a drop-in change worth A/B testing before assuming you need custom optimization.

The rule of thumb: reach for GRU when you need long-range memory but are compute- or latency-constrained; reach for the full LSTM when the task's memory requirements are complex enough that the extra Output Gate and separate Cell State demonstrably improve validation performance.

Recapping the full picture: a SimpleRNN forgets long-range context because its gradients vanish through repeated tanh squashing. The LSTM fixes this by splitting memory into a Cell State (long-term, mostly-additive updates) and a Hidden State (short-term, immediately usable output), and it regulates both with learned sigmoid Gates — Forget, Input, and (though not covered gate-by-gate here) Output.

When the full LSTM is more compute than a project can afford, the GRU offers a lighter alternative: it merges the Forget and Input Gates into one Update Gate and drops the separate Cell State, trading a small amount of representational flexibility for meaningfully faster training.

With both architectures in hand, you now have the tools to model sequences — text, time series, sensor streams — where order and long-range dependencies matter, and to choose the right trade-off between accuracy and training cost for the constraints of a given project.

To solve the amnesia of the SimpleRNN, researchers invented the LSTM: Long Short-Term Memory. It is an engineering marvel.

Instead of just one memory state, an LSTM has two lines of communication: The Hidden State (Short-term memory) and the Cell State (Long-term memory).

What architectural addition allows the LSTM to solve the Vanishing Gradient problem and remember data across long sequences?

• →It adds convolutional kernels to the RNN.
• →The introduction of the 'Cell State' (a secondary, long-term memory conveyor belt) protected by 'Gates' that control the flow of information.
• →It connects directly to a SQL database.

The LSTM controls this memory using mathematical "Gates" (mini neural networks powered by Sigmoid functions). The first is the Forget Gate.

What is the purpose of the "Forget Gate" inside an LSTM?

• →It randomly drops out neurons to prevent overfitting.
• →It outputs a number between 0 and 1, deciding mathematically how much of the previous Long-Term memory (Cell State) should be permanently deleted.
• →It erases the hard drive.

Next, the Input Gate decides what NEW information from the current word is important enough to add to the Long-Term memory.

After the Forget Gate cleans up the old memory, what does the "Input Gate" do?

• →It analyzes the current input and decides which new, relevant information should be written onto the Long-Term memory conveyor belt.
• →It downloads more training data.
• →It calculates the final Loss.

Now, prepare yourself. We are about to enter the ADA Defense Protocol. Ensure you understand GRUs vs LSTMs.

LSTMs are powerful but computationally heavy. A newer, slightly faster variant exists called the GRU (Gated Recurrent Unit).

ADA DEFENSE: A senior engineer tells you that the LSTM is running too slowly on the company servers, but a SimpleRNN causes the model to fail. What architectural drop-in replacement should you try to speed up training while keeping the gates?

• →A Conv2D layer.
• →A GRU (Gated Recurrent Unit), which combines the Forget and Input gates into a single Update gate, making it mathematically faster while maintaining long-term memory.
• →A Dense layer.

Threat neutralized. Gated logic validated. Module 04 complete.

Apply a Real Forget Gate. Finish apply_forget_gate(): multiply the cell state element-wise by the forget signal.