Building Neural Networks in Python

Now that you understand tensors, GPU acceleration, and Autograd, it's time to assemble them into an actual trainable architecture. PyTorch's torch.nn module is a library of pre-built, differentiable building blocks — layers, activation functions, loss functions — designed to be composed together rather than written from scratch.

Every layer in torch.nn is itself built on tensors under the hood: a Linear layer stores its weights and biases as tensors with requires_grad=True, so Autograd automatically tracks how the loss depends on them. That's what makes torch.nn layers trainable — you don't manage gradients yourself, you just define the forward computation and PyTorch's Autograd handles the rest.

This module walks through the core building blocks in the order you'll actually use them: the Linear layer as the fundamental unit, nn.Sequential to stack layers into a network, and activation functions as the piece that makes the whole thing capable of learning anything beyond straight lines.

nn.Linear is the most basic learnable layer in PyTorch, and it corresponds to what other frameworks and textbooks call a 'Dense' or 'Fully Connected' layer. Every input feature is connected to every output neuron through a learned weight, plus one learned bias term per output.

Calling nn.Linear(in_features=10, out_features=5) doesn't run any computation yet — it allocates a weight matrix of shape (5, 10) and a bias vector of shape (5,), both initialized randomly and marked as trainable parameters. When you later pass a batch of data through it, the layer computes y = xW^T + b: a matrix multiplication followed by a bias addition.

Every other layer type you'll encounter — convolutional, recurrent, attention — is a variation on this same idea: a set of learnable weights that transform an input tensor into an output tensor of a different (or the same) shape.

The exercise asks what nn.Linear(in_features=10, out_features=5) does 'under the hood,' and the precise answer is linear algebra: it creates a weight matrix of shape (5, 10) and a bias vector of shape (5,), then computes the transformation y = xW^T + b for every input you pass in.

Concretely, if you feed in a batch of vectors each with 10 values, the layer multiplies each one by its weight matrix and adds the bias, producing a new vector of 5 values per input. Those 10 and 5 numbers aren't image dimensions or compression ratios — they're literally the count of input and output features the layer's matrix multiplication expects.

This is why the wrong answers in the exercise are wrong: nn.Linear doesn't draw a graph or compress files — it performs a specific matrix multiply-and-add that the rest of the network relies on having the correct shape.

A single Linear layer can only learn one linear transformation. Real networks stack many layers so that the output of one becomes the input of the next, and nn.Sequential is PyTorch's simplest tool for defining that stack: you pass it a list of layers, in order, and it chains their outputs and inputs together automatically.

nn.Sequential(nn.Linear(10, 20), nn.Linear(20, 5)) creates a two-layer network: the first layer expands 10 input features to 20 'hidden' values, and the second layer compresses those 20 values down to 5 outputs. Calling the model on a batch of data runs it through both layers in the exact order they were listed.

This pattern — increase then reduce, or reduce then increase, dimensionality across a sequence of layers — is the basic shape of nearly every feed-forward neural network, and nn.Sequential is the most direct way to express it in code.

Stacking layers with nn.Sequential isn't free-form — there's a strict mathematical constraint you must respect: the out_features of one layer must exactly equal the in_features of the next layer, because that next layer's weight matrix is only defined for inputs of that specific size.

In nn.Sequential(nn.Linear(10, 20), nn.Linear(20, 5)), the first layer's out_features=20 matches the second layer's in_features=20 exactly. If you instead wrote the second layer as nn.Linear(15, 5), PyTorch would raise a shape-mismatch error the moment you tried to run data through the model, because a (20,)-shaped output can't be multiplied against a weight matrix expecting 15 inputs.

This is one of the most common early PyTorch errors — 'mat1 and mat2 shapes cannot be multiplied' — and the fix is always the same: trace through your layer definitions and confirm each layer's out_features matches the next layer's in_features.

Here's a subtle trap: stacking multiple nn.Linear layers with nothing in between doesn't actually create a more powerful model. Linear transformations composed with other linear transformations collapse mathematically into a single linear transformation — two matrix multiplications in a row is still just one matrix multiplication in disguise.

That means a 'deep' network built purely from Linear layers can only ever learn a straight-line (or flat-hyperplane) relationship between inputs and outputs, no matter how many layers you stack. Real-world data — images, language, physical systems — is almost never linear, so a purely linear network would badly underfit.

The fix is Non-Linearity: inserting an activation function between each pair of Linear layers. Activation functions break the mathematical property that lets consecutive linear layers collapse together, which is what actually gives depth its power.

The exercise's correct answer is precise: without activation functions, a network 'could only learn perfectly straight linear relationships, destroying its ability to understand complex data like images or text.' That's not an exaggeration — it's a direct mathematical consequence of composing linear functions.

An activation function is a small, usually simple, non-linear function applied elementwise after each Linear layer's output — ReLU, Sigmoid, Tanh, and others are all common choices. Because they're non-linear, they prevent consecutive layers from collapsing into one, which lets the overall network approximate arbitrarily complex, curved functions given enough layers and neurons.

This is why every real neural network architecture interleaves Linear (or Convolutional, or Attention) layers with activation functions — removing them entirely, even from a 100-layer network, would leave you with something mathematically equivalent to a single linear regression.

The most widely used activation function in modern deep learning is ReLU — Rectified Linear Unit — and it's deliberately simple: for any input, it outputs the input unchanged if positive, and zero if negative.

That simplicity is a feature, not a limitation. ReLU is cheap to compute (just a comparison against zero), and its gradient is either 0 or 1, which avoids some of the numerical problems (like vanishing gradients) that plagued earlier activation functions like Sigmoid in very deep networks.

Before looking at the exact mechanics, it's worth previewing why this matters in practice: ReLU is the default choice between hidden layers in the vast majority of modern architectures, from simple feed-forward networks to the CNNs and Transformers covered later in this course.

Formally, ReLU is defined as f(x) = max(0, x). If the input is positive, ReLU returns that exact value unchanged. If the input is zero or negative, ReLU returns zero, discarding the original value entirely.

Applied to a whole tensor of neuron outputs, ReLU acts elementwise: each value is independently checked and either passed through or zeroed out. There's no interaction between elements and no learnable parameters — ReLU is a fixed, deterministic function, unlike the Linear layers around it.

This 'kill everything negative' behavior is exactly what introduces the non-linear kink that prevents Linear layers from collapsing together — the function is linear on the positive side, flat at zero on the negative side, and the seam between those two pieces is where the non-linearity lives.

The defense question is a direct application of the ReLU formula: a neuron outputs -5.2, and that value is passed into ReLU. Since ReLU is defined as f(x) = max(0, x), and -5.2 is less than 0, the function returns 0 — the negative value is discarded entirely, not reduced, not made positive, just zeroed out.

It's worth being precise about why the other options are wrong: ReLU never passes negative values through unchanged (ruling out -5.2), and it never flips their sign to make them positive (ruling out 5.2). It has exactly one rule for negative inputs — output zero.

This all-or-nothing behavior on the negative side is also why ReLU neurons can 'die' during training: if a neuron's weights push its output permanently negative for every input in the dataset, it will always output zero and its gradient will always be zero too, effectively removing it from the network.

At this point you can build a working feed-forward network: Linear layers to transform data, nn.Sequential to stack them, and ReLU (or another activation) between them to make the stack capable of learning non-linear patterns. That combination — Linear, activation, Linear, activation — is the basic skeleton of a huge fraction of neural network architectures.

What you've used so far, nn.Sequential, works well for simple, strictly linear stacks of layers. But it can't express architectures with branches, skip connections, or multiple inputs and outputs — the kinds of designs that show up in real production models.

The next step is writing PyTorch models as Python classes using nn.Module directly, which gives you full control over the forward pass and unlocks the flexibility needed for the more advanced, object-oriented architectures the rest of this course builds toward.

Module 05: Building Neural Networks. You now understand Tensors, Gradients, and GPUs. It is time to build the actual architecture.

The most fundamental building block is the Linear Layer, often called a "Dense" or "Fully Connected" layer.

What does nn.Linear(in_features=10, out_features=5) actually do under the hood?

• →It draws a straight line on a graph.
• →It creates a mathematical matrix of 'weights' and 'biases' that multiply the 10 inputs and transform them into 5 outputs.
• →It compresses a 10MB image down to 5MB.

A neural network is just a stack of these layers. We use nn.Sequential to stack them together in a specific order.

When stacking layers in nn.Sequential, what is the strict mathematical rule regarding the inputs and outputs of adjacent layers?

• →They must all be the same number (e.g., 10, 10, 10).
• →The out_features of the previous layer MUST exactly match the in_features of the next layer.
• →The in_features must always be larger than the out_features.

If you just stack Linear layers, the network is just one giant straight line (Regression). To learn complex, curved patterns, we MUST inject Non-Linearity.

Why are "Activation Functions" (like ReLU or Sigmoid) absolutely mandatory in Deep Learning?

• →Without them, the network could only learn perfectly straight linear relationships, destroying its ability to understand complex data like images or text.
• →They activate the GPU hardware.
• →They prevent the code from crashing.

Now, prepare yourself. We are about to enter the ADA Defense Protocol. Ensure you understand ReLU mechanics.

The ReLU function is incredibly simple. If the input is positive, it passes it through. If it is negative, it turns it to zero.

ADA DEFENSE: A neuron outputs the number -5.2. This number is passed into a ReLU activation function. What is the output of the ReLU function?

• →-5.2
• →0 (Zero).
• →5.2

Threat neutralized. Non-linearity confirmed. Proceeding to Object-Oriented PyTorch architectures.

Validate Real Layer Shapes. Finish can_stack_layers(): adjacent layers must match on their shared dimension.