Autograd & Gradients in Python

Training a neural network means adjusting every weight in the network so the model's predictions get closer to the correct answer. To know which direction and how much to adjust each weight, you need the derivative of the loss function with respect to that weight — the gradient. For a network with millions of parameters spread across dozens of layers, that means computing millions of derivatives, correctly, every single training step.

Before automatic differentiation existed, researchers derived and coded these gradient formulas by hand using the chain rule, layer by layer. It was tedious and extremely error-prone: a single sign error or missed term in a hand-derived formula would silently produce a network that never converges, with no obvious error message pointing to the cause — just a loss curve that refuses to go down.

PyTorch's Autograd engine eliminates this manual work entirely. Instead of deriving formulas by hand, you write the forward pass as ordinary Python/tensor operations, and PyTorch automatically builds the machinery needed to compute exact gradients for you. This is the foundation that makes modern deep learning practical — without it, every new architecture would require re-deriving backpropagation from scratch.

PyTorch solves the manual-derivative problem with Autograd, its automatic differentiation engine. When you create a tensor with requires_grad=True, you are telling PyTorch: track every operation performed on this tensor, because I'll need its gradient later. From that point on, any arithmetic involving that tensor — addition, multiplication, matrix multiplication, activation functions — gets recorded rather than just computed and forgotten.

Under the hood, each recorded operation attaches a grad_fn to the resulting tensor, a reference back to the operation that produced it and the inputs it depended on. Chaining these references together as computation proceeds builds up a dynamic computation graph — 'dynamic' because PyTorch constructs it on the fly as your Python code actually executes, rather than requiring you to define the graph structure ahead of time the way older frameworks like TensorFlow 1.x did.

This is why model parameters (weights and biases) are created with requires_grad=True by default when you use nn.Parameter or nn.Linear, while input data typically isn't tracked — you don't need the gradient of the loss with respect to your raw input pixels, only with respect to the weights you're actually going to update.

The computation graph that requires_grad=True builds is not just a bookkeeping trick — it is the literal structure Autograd walks when computing gradients later. Every node in the graph represents either a leaf tensor (something you created directly, like a weight) or the output of an operation, and every edge tracks which inputs fed into which operation.

This graph is built forward, in the same order your code executes: z = x * y first computes z's value, then attaches a MulBackward node recording that z came from multiplying x and y. Nothing about the derivative is computed yet — only the recipe for computing it later is stored. That's the key distinction between forward-mode execution (compute values immediately) and Autograd's reverse-mode differentiation (walk the recorded graph backward once you know the final loss).

Because the graph is rebuilt fresh on every forward pass, you can freely use Python control flow — if statements, loops, recursion — inside your model, and Autograd will still produce a correct graph for whatever path your code actually took that iteration. This flexibility is one of PyTorch's defining advantages over static-graph frameworks.

Because PyTorch builds a computation graph as your forward pass runs, it knows exactly how to walk backward from the final output to every input that contributed to it. You trigger that backward walk by calling .backward() on the final scalar value — typically the loss.

When y.backward() is called on y = x ** 2, PyTorch applies the chain rule automatically, starting at y and propagating derivatives back through every recorded operation until it reaches each leaf tensor with requires_grad=True. For y = x**2, the derivative is dy/dx = 2x; PyTorch doesn't 'know calculus' in a symbolic sense — it stores the local derivative rule for every basic operation (multiplication, power, matmul, etc.) and multiplies these local derivatives together along the graph, exactly implementing the chain rule numerically.

Once .backward() finishes, the computed gradient is stored on the .grad attribute of each leaf tensor that required it — x.grad in this example. Non-leaf tensors (intermediate results like y) don't retain their .grad by default, since in a typical training loop only the leaf parameters need updating.

.backward() is almost always called on a scalar value — a single number, like the loss — because gradients are only unambiguously defined with respect to a scalar output. If you call .backward() on a tensor with more than one element, PyTorch will raise an error unless you explicitly pass a gradient argument telling it how to weight each element, since 'the derivative of a vector with respect to another vector' isn't a single well-defined quantity without more context.

In a real training loop, this scalar is produced by a loss function: loss = criterion(predictions, targets) collapses the model's entire batch of predictions down to one number representing how wrong the model currently is. Calling loss.backward() then propagates that single error signal backward through every layer, populating .grad on every weight and bias tensor that required gradients.

It's worth contrasting this with .forward(), which isn't actually a method you call directly in most cases — calling a model instance like model(x) invokes its forward() method internally via Python's __call__ protocol. .backward(), by contrast, is called explicitly by you, on the loss, once per training step.

A detail that trips up almost every PyTorch beginner: gradients are accumulated, not overwritten. Each time you call .backward(), PyTorch adds the newly computed gradients into whatever is already sitting in .grad, rather than replacing it. Call .backward() five times in a row without resetting in between, and x.grad will hold the sum of all five gradient computations.

This accumulation behavior isn't a bug — it exists deliberately to support cases like RNNs or gradient accumulation across multiple small batches (used to simulate a larger batch size when GPU memory is limited), where you genuinely want gradients from several forward/backward passes to add together before taking a single optimizer step.

But in the standard single-batch training loop, this accumulation is dangerous if left unmanaged: without an explicit reset, each new step's gradient gets contaminated by every previous step's gradient, and the optimizer ends up updating weights based on garbage. That's why optimizer.zero_grad() (or model.zero_grad()) is called at the start of every iteration, before the next .backward() call.

Skipping optimizer.zero_grad() is one of the most common and most insidious bugs in PyTorch code, precisely because it doesn't crash — it just quietly trains a bad model. The loss might even decrease at first, misleading you into thinking training is working, before the accumulated gradients grow so large that updates become erratic and the loss diverges or plateaus at a poor value.

The canonical training-loop pattern is exactly three lines, in this order: optimizer.zero_grad() clears any stale gradients from the previous step; loss.backward() computes fresh gradients for the current batch via the computation graph; optimizer.step() uses those fresh gradients to actually update the weights. Reordering or omitting any of these three lines breaks training in a different way — for example, calling zero_grad() after backward() would erase the gradients you just computed.

Some PyTorch code calls optimizer.zero_grad(set_to_none=True) instead of the default. This sets .grad to None rather than a zero tensor, which is slightly faster and uses less memory, since PyTorch can skip allocating a zero-filled tensor it's about to overwrite anyway.

So far, everything about Autograd has been in service of training: build a graph, compute a loss, walk backward, update weights. But a trained model also needs to be evaluated and used for inference — running the test set through it to measure accuracy, or serving predictions in production — and during those phases, gradient tracking serves no purpose at all.

Python's context manager syntax (with ... :) is the mechanism PyTorch uses to temporarily change Autograd's behavior for a block of code. A context manager guarantees setup code runs on entry and cleanup code runs on exit — even if an exception occurs inside the block — which makes it the natural tool for 'do X differently for this scoped region, then restore normal behavior afterward.'

Understanding this pattern matters because forgetting to use it during evaluation is one of the most common sources of wasted GPU memory in PyTorch scripts: every forward pass through a model still builds a full computation graph by default, even if you never intend to call .backward() on the result.

Every intermediate tensor produced inside a tracked computation graph has to be kept alive in memory, because Autograd might need it later to compute a derivative — for example, the derivative of x * y with respect to x requires knowing the value of y, so y can't simply be discarded after the multiplication. For a deep network processing large batches, this means every activation at every layer stays resident in memory throughout the forward pass.

When you're just running inference — feeding your test set through a trained model to compute accuracy, or generating predictions for a real user — none of that saved history is ever going to be used, because you'll never call .backward(). Building and retaining the graph anyway is pure waste: it inflates memory usage and adds unnecessary bookkeeping overhead to every operation.

This matters most on GPUs, where VRAM is a hard, often tight limit. A model that trains fine on an 8GB GPU can run out of memory during evaluation on the exact same GPU if the evaluation code accidentally leaves gradient tracking enabled, because the 'no gradients needed' savings never kick in.

with torch.no_grad(): is the standard fix. Any tensor operation performed inside that block skips graph construction entirely — no grad_fn, no retained intermediate activations, no accumulated memory overhead. The typical evaluation loop wraps the whole forward pass in it: with torch.no_grad(): predictions = model(test_inputs).

A closely related, newer alternative is torch.inference_mode(), which does everything no_grad() does and goes slightly further by also disabling PyTorch's version-tracking machinery (used to detect in-place modifications that would corrupt a graph). inference_mode() is generally faster and is the recommended choice for pure inference code where you're certain no gradient will ever be needed on the results; no_grad() remains useful when you might still need to interact with tensors in ways that inference_mode()'s stricter guarantees don't allow.

Both are also commonly used as function decorators (@torch.no_grad()) on evaluation functions, which has the same effect as wrapping the function body in a with block but avoids the extra indentation.

Threat neutralized. Memory leaks prevented. Proceeding to Hardware Acceleration.

Putting it all together, a complete PyTorch training step follows one consistent lifecycle built entirely on the concepts covered here: mark the tensors that need gradients with requires_grad=True (or let nn.Parameter do it for you), run the forward pass to build the computation graph, call loss.backward() to walk that graph in reverse and populate .grad on every leaf tensor, then optimizer.step() to apply those gradients — always preceded by optimizer.zero_grad() so the current step isn't contaminated by gradients left over from the last one.

The evaluation half of that lifecycle matters just as much: wrapping inference and validation code in torch.no_grad() (or the stricter torch.inference_mode()) stops PyTorch from building graphs it will never walk backward through, which is often the difference between a model that fits comfortably in GPU memory and one that throws a CUDA out-of-memory error the moment validation runs.

Everything Autograd does — building graphs, computing gradients, tracking memory — happens on whichever device your tensors live on. That's the next piece of the puzzle: moving tensors and models onto a GPU so this same graph-and-backward machinery runs orders of magnitude faster.

Training a Neural Network requires Backpropagation. That means calculating the calculus derivative (gradient) of every single weight relative to the error.

PyTorch solves this with Autograd. When you create a tensor with requires_grad=True, PyTorch starts secretly recording every math operation done to it.

What happens when you set requires_grad=True on a PyTorch Tensor?

• →The tensor is automatically moved to the GPU.
• →PyTorch begins recording every mathematical operation performed on that tensor to build a 'Computation Graph' for calculus.
• →The tensor becomes immutable and cannot be changed.

Because PyTorch builds a "Computation Graph", it knows exactly how to work backward from the output to the input. We trigger this with .backward().

Which method do you call on your final output (e.g., the Error or Loss) to trigger the automatic calculation of all gradients in the network?

• →.backward()
• →.compute_derivatives()
• →.forward()

PyTorch accumulates gradients. If you run a loop and call .backward() 5 times, the gradients add up. You MUST clear them using .zero_grad().

Why is calling optimizer.zero_grad() absolutely critical inside a PyTorch training loop?

• →Because it resets the weights of the neural network.
• →Because PyTorch accumulates (adds) gradients by default. If you don't zero them out, step 2 will contain the gradients of step 1 and step 2 combined, destroying the math.
• →To free up RAM on the GPU.

Now, prepare yourself. We are about to enter the ADA Defense Protocol. Ensure you understand context managers.

Autograd uses memory. When you are just testing your model (not training), you do not want it recording operations.

ADA DEFENSE: You are running your test dataset through the neural network to get an accuracy score. What context manager should you wrap your code in to prevent PyTorch from wasting RAM building computation graphs?

• →with torch.test_mode():
• →with torch.no_grad():
• →with torch.disable_ram():

Threat neutralized. Memory leaks prevented. Proceeding to Hardware Acceleration.

Compute a Real Gradient. Finish gradient_of_x_squared(): the derivative of x^2 is 2x.