Introduction to PyTorch in Python

Everything up to this module operated inside scikit-learn's world: traditional statistical algorithms like Random Forests, SVMs, and KMeans that mostly work with tabular data and handcrafted features. PyTorch is a different kind of tool entirely — the deep learning framework built by Meta (originally Facebook) that underlies most of the modern AI landscape, from image recognition to large language models.

The shift isn't just a new library to import; it's a shift in modeling philosophy. Scikit-learn estimators are largely fixed algorithms you configure with hyperparameters and call .fit() on. PyTorch instead gives you the building blocks — tensors, layers, an automatic differentiation engine — to construct and train custom neural network architectures yourself.

That flexibility is precisely why PyTorch is the tool of choice once a problem outgrows what tabular, feature-engineered statistical models can handle — raw pixels, raw audio, and raw text are exactly the unstructured, high-dimensional inputs neural networks are designed to learn directly from.

Every scikit-learn algorithm covered so far, whether it's a Decision Tree splitting on feature thresholds or a Linear Regression fitting a straight line, has a recognizable mathematical shape you could sketch on paper. Deep learning abandons both of those shapes in favor of Artificial Neural Networks (ANNs): layers of interconnected 'neurons' loosely inspired by how biological brains process signals.

An ANN's structure is a stack of layers — an input layer that receives the raw features, one or more hidden layers that transform the data, and an output layer that produces the prediction. What makes this architecture powerful isn't any single layer, but the way many simple transformations compose together to approximate extremely complex functions that a single tree or line never could.

This is also why neural networks need so much more data and compute than a Random Forest to train well: instead of a handful of interpretable splits, you're learning millions of numeric weights spread across many layers, which only becomes tractable with GPU acceleration — the next topic.

Scikit-Learn's Random Forest, and every other estimator in that library, is a fixed algorithm: you choose hyperparameters, call .fit(), and the library runs a predetermined procedure — splitting on feature thresholds, averaging trees, whatever the algorithm specifies. There is no concept of 'layers' or 'weights' you design yourself; the architecture is baked into the class you imported.

PyTorch flips that completely. A PyTorch model is a graph of tensor operations you assemble from primitives — nn.Linear layers, activation functions, convolutions — stacked in whatever order you choose. Nothing is 'fit' automatically; you write the forward pass, PyTorch tracks the operations for Autograd, and you write the training loop that updates the weights.

That's the real answer to the exercise question: Scikit-Learn gives you a menu of complete, off-the-shelf algorithms, while PyTorch gives you construction material for building a custom multi-layered Artificial Neural Network, with all the extra responsibility (and flexibility) that implies.

Every operation inside a neural network — matrix multiplications between weights and activations, elementwise additions, gradient computations — is the same simple arithmetic repeated millions of times. A CPU executes this arithmetic mostly one instruction at a time across a handful of cores; a GPU has thousands of smaller cores built to do exactly that kind of repetitive parallel math simultaneously.

This is why PyTorch treats the GPU as a first-class citizen rather than an afterthought. A tensor and a model can be moved from CPU to GPU with a single .to('cuda') call, and every subsequent matrix multiplication then runs across thousands of GPU cores instead of a handful of CPU cores — often 10-100x faster for the large matrices deep learning relies on.

Without this hardware acceleration, training large neural networks would be impractical: a model that trains in hours on a GPU could take weeks on a CPU. GPU acceleration is what makes it feasible to train models with millions or billions of parameters at all.

Training a model at the scale of ChatGPT involves billions of parameters and trillions of individual arithmetic operations per training step. That scale is only tractable because CUDA lets PyTorch dispatch those operations to a GPU's thousands of cores and run enormous batches of matrix multiplications in parallel, rather than serially on a CPU.

CUDA (Compute Unified Device Architecture) is NVIDIA's platform for running general-purpose code on GPU hardware. PyTorch is built with native CUDA support, so calling .to('cuda') on a tensor or model hands that computation off to the GPU without you writing any low-level GPU code yourself.

This is precisely why the exercise's correct answer highlights parallel processing: it's not just that GPUs are 'faster' in the abstract, it's that they can execute millions of independent multiply-add operations at the same time, which is the specific shape of the workload that training a massive neural network produces.

Training a neural network requires computing the gradient of a loss function with respect to every single weight in the network, so the optimizer knows which direction to nudge each parameter. Doing that calculus by hand for a network with millions of parameters is not realistic — this is where Autograd, PyTorch's automatic differentiation engine, takes over.

As you perform operations on tensors that have requires_grad=True, PyTorch silently builds a computation graph recording every operation. When you call .backward() on the final loss, Autograd walks that graph backward, applying the chain rule automatically to compute the gradient for every weight in a single pass.

This is what makes backpropagation practical for arbitrarily deep, arbitrarily complex architectures: you only have to define the forward pass (the math that produces a prediction), and Autograd derives the entire backward pass for you.

The purpose of Autograd is narrow and specific: it computes derivatives, not predictions. Every tensor operation you perform — multiplication, addition, activation functions — gets recorded onto a computation graph as long as the tensors involved have requires_grad=True.

When you call loss.backward(), Autograd traverses that graph from the loss value back to every parameter that contributed to it, applying the chain rule at each step to compute exactly how much a tiny change in that parameter would change the loss. That value — the gradient — gets stored in the parameter's .grad attribute, ready for an optimizer like SGD or Adam to use.

Without Autograd, you would have to manually derive and code the backward pass for every architecture you design — a task that becomes intractable the moment a network has more than a couple of layers. Autograd is what lets you focus on model architecture instead of calculus.

By the mid-2010s, two deep learning frameworks had emerged as dominant: TensorFlow, built by Google, and PyTorch, built by Meta (then Facebook). Both frameworks solve the same underlying problem — defining neural networks and training them with gradient descent — but they took different early approaches to how a computation graph is built and executed.

TensorFlow originally used 'static' computation graphs: you defined the entire graph structure first, then ran data through it in a separate session. PyTorch instead used 'dynamic' computation graphs from day one — the graph is built on the fly, operation by operation, as your Python code actually executes.

That difference sounds small, but it has large practical consequences for how debuggable and 'Pythonic' each framework feels — which is exactly what the next section explores.

PyTorch's dynamic computation graph means the graph is rebuilt every time your code runs, following your actual Python control flow — if statements, loops, and print statements all work exactly like they would in any other Python script, because the model genuinely is just executing Python.

Early TensorFlow's static graph, by contrast, required you to define the full graph structure up front, separate from the Python code that ran it, and then execute it inside a special Session object. Debugging meant inspecting an abstract graph definition rather than stepping through familiar code — a workflow many researchers found unintuitive.

This is the core of why PyTorch became known as 'Pythonic': researchers could set a breakpoint, print an intermediate tensor's value, and iterate on architecture ideas exactly the way they would with any other Python program, without learning a separate graph-definition API.

By the late 2010s, the majority of new deep learning papers were being published with PyTorch implementations rather than TensorFlow ones — a striking reversal from TensorFlow's early dominance. The reason traces directly back to dynamic computation graphs: PyTorch code runs and fails exactly like normal Python, so researchers could use standard debuggers, insert print statements mid-model, and prototype new architectures without fighting a separate graph-building API.

Experimentation speed matters enormously in research, where architectures change daily and half-finished ideas need to be tested quickly. TensorFlow's original static-graph workflow added friction to that kind of rapid iteration, while PyTorch's 'just write Python' model removed it.

TensorFlow has since adopted eager execution to close this gap, but PyTorch's early advantage in ease-of-debugging is what cemented its dominance in academic and research settings, which is why it also became the framework underlying most modern large language model research.

At this point you understand why PyTorch exists, what makes it fast (GPU acceleration via CUDA), what makes it trainable (Autograd), and why it became the dominant research framework (dynamic, Pythonic computation graphs). Everything from here forward builds directly on those three pillars.

The next step is the tensor itself — the fundamental data structure every PyTorch operation is built on. Tensors are the GPU-accelerated, Autograd-tracked equivalent of a NumPy array, and understanding how to create, reshape, and move them between devices is the prerequisite for building any real model.

Everything you've covered so far was conceptual; tensor mechanics is where you start writing the actual code that GPU acceleration and Autograd operate on.

Module 04: Welcome to Deep Learning. Scikit-Learn was for traditional statistics. PyTorch is the engine of modern AI, built by Meta (Facebook).

Deep Learning abandons Decision Trees and Lines. Instead, it uses Artificial Neural Networks (ANNs) inspired by the human brain.

What is the primary architectural difference between Scikit-Learn algorithms (like Random Forest) and PyTorch models?

• →PyTorch exclusively relies on multi-layered Artificial Neural Networks, whereas Scikit-Learn uses a variety of traditional statistical algorithms.
• →PyTorch uses SQL databases natively.
• →Scikit-Learn uses Neural Networks, but PyTorch uses Trees.

PyTorch has two massive advantages over Scikit-Learn. First: GPU Acceleration. PyTorch can run math on your graphics card, making it 100x faster for massive matrices.

Why do Deep Learning practitioners strongly prefer PyTorch over traditional Python libraries for training massive models like ChatGPT?

• →Because PyTorch natively supports GPU hardware acceleration (CUDA), which allows for parallel processing of millions of mathematical operations simultaneously.
• →Because PyTorch is written in HTML.
• →Because PyTorch automatically cleans missing data in Pandas.

The second massive advantage is Autograd (Automatic Differentiation). PyTorch automatically calculates the complex calculus derivatives needed to train Neural Networks.

What is the purpose of PyTorch's "Autograd" engine?

• →It automatically grades student exams.
• →It automatically calculates derivatives and gradients, completely automating the brutal calculus required for Backpropagation.
• →It automatically generates a Pandas DataFrame.

Now, prepare yourself. We are about to enter the ADA Defense Protocol. Ensure you understand the difference between TensorFlow and PyTorch.

TensorFlow is Google's library. PyTorch is Meta's library. Both do the same thing, but PyTorch dominates academic research because it is "Pythonic".

ADA DEFENSE: Why did PyTorch overtake TensorFlow as the dominant framework in AI research labs worldwide?

• →Because PyTorch feels exactly like writing standard Python (Dynamic Computation Graphs), making it much easier to debug and experiment with.
• →Because PyTorch is the only one that supports GPUs.
• →Because TensorFlow was discontinued in 2015.

Threat neutralized. Landscape understood. Proceeding to Tensor mechanics.

Run a Real Elementwise Tensor Operation. Finish elementwise_multiply(): a Tensor supports the same elementwise math as a NumPy array.