Compiling & Training in Python

In frameworks like raw PyTorch, you write an explicit training loop: iterate over batches, run a forward pass, compute the loss, call .backward(), and step the optimizer — typically ten to fifteen lines of boilerplate repeated in every project. Keras collapses all of that into two calls: model.compile() configures how training should happen, and model.fit() actually runs it.

This isn't magic hiding complexity you should ignore — it's the same gradient descent loop, just implemented once inside TensorFlow's runtime instead of rewritten by hand each time. Because the loop lives inside the framework, TensorFlow can also apply graph-level optimizations, like operation fusion, that a hand-written Python loop typically can't.

The tradeoff is that you lose line-by-line visibility into each training step. If you need custom logic — a non-standard loss computation, gradient clipping — Keras exposes a train_step() override and a low-level GradientTape API, so you don't have to abandon model.fit() entirely to get that control.

model.compile() doesn't train anything — it configures the machinery training will use. Three arguments matter most: optimizer (the algorithm that updates weights, like "adam"), loss (the function that scores how wrong a prediction is), and metrics (values tracked for your benefit, like "accuracy", that don't affect the gradients at all).

Calling compile() is mandatory before fit() — Keras needs to know which loss function to differentiate before it can compute a single gradient. Skip it, and model.fit() raises an error rather than guessing at defaults.

Notice that metrics is deliberately separate from loss. The loss is what the optimizer actually minimizes; accuracy, or any other metric you list, is purely informational and printed during training so you can judge model quality in human terms, even though the optimizer never looks at it directly.

It's worth separating the two jobs compile() does, because they answer different questions. The optimizer answers "how should weights change given a gradient?" — Adam, SGD, and RMSprop all take the same gradient information and update weights differently. The loss function answers "how wrong was this specific prediction?" — that's the value the optimizer is actually trying to drive toward zero.

A common misconception is that compile() runs on the GPU or converts Python to C++. It does neither by itself; it builds the internal computation graph and attaches the chosen optimizer and loss so that when fit() runs, TensorFlow already knows exactly what to compute and differentiate on each batch.

Because compile() is where the loss function is fixed, changing the problem you're solving — say, from binary to multi-class classification — always requires calling compile() again with a different loss argument. You can't swap the loss function mid-training without recompiling.

The loss argument you pass to compile() is not a style choice — it has to match the shape of your problem, or training will either crash or silently learn the wrong thing. Regression problems (predicting a continuous number, like a house price) use mean_squared_error or mean_absolute_error, which measure numeric distance between prediction and target. Classification problems use a crossentropy loss instead, because they're scoring probability distributions, not raw numeric error.

Within classification, the split matters: binary_crossentropy is for exactly two classes (or independent multi-label outputs), while categorical_crossentropy is for mutually exclusive multi-class problems where the labels are one-hot encoded. Picking binary_crossentropy for a 3-class problem doesn't throw an error — Keras will happily compute it — but the gradients it produces don't correspond to the actual classification task, so the model trains toward the wrong objective.

If your labels are integers (0, 1, 2) rather than one-hot vectors ([1,0,0]), Keras also provides sparse_categorical_crossentropy, which computes the same mathematical loss without requiring you to one-hot encode the labels first. Choosing between the two categorical_crossentropy variants is purely about label format, not about the problem itself.

Take a concrete case: an image classifier that outputs one of three labels — "Cat", "Dog", or "Bird". This is a mutually exclusive, multi-class problem, so the loss must be categorical_crossentropy (or sparse_categorical_crossentropy if the labels stay as integers). The final Dense layer needs 3 units with a softmax activation, so the outputs form a valid probability distribution over the three classes that the crossentropy loss can score.

A common beginner mistake is reaching for binary_crossentropy here because it 'sounds' like the general-purpose classification loss. It isn't — binary_crossentropy assumes a single sigmoid output between 0 and 1 and only works correctly for two-class (or multi-label) problems. Using it on a 3-unit softmax output produces a loss value that trains the network in a mathematically inconsistent way.

The fix is mechanical once you see the pattern: count the number of mutually exclusive output classes. Two classes with a single sigmoid output → binary_crossentropy. More than two classes with a softmax output → categorical_crossentropy or sparse_categorical_crossentropy depending on label encoding. Matching loss to output layer shape is one of the most common sources of silently wrong training.

Once compile() has configured the optimizer and loss, model.fit(X_train, y_train, epochs=10, batch_size=32) is what actually runs training. Internally, Keras splits X_train into chunks of batch_size examples, runs a forward pass on each batch, computes the loss, backpropagates gradients, and updates weights — then repeats until every batch has been seen once, which counts as one epoch.

batch_size controls how many examples are processed before a single weight update happens. Smaller batches update weights more often (noisier but sometimes better generalization); larger batches make more efficient use of GPU parallelism but need more memory per step. 32 is a common default that balances both.

model.fit() returns a History object, assigned here to history, that records the loss (and any tracked metrics) after every epoch. That's the object you plot afterward to check whether the model actually learned — a loss curve that's still dropping sharply at epoch 10 suggests you stopped training too early; one that's flat from epoch 2 onward suggests the model has already converged or the learning rate is too low.

epochs=10 tells model.fit() to run 10 complete passes over the entire training dataset, not to stop after 10 individual examples or 10 batches. Within each epoch, the dataset is still processed in batch_size-sized chunks, so the true number of weight updates per epoch is roughly len(X_train) / batch_size.

More epochs means more chances for the optimizer to refine the weights and reduce the loss — but only up to a point. Too few epochs and the model is underfit, having barely learned the patterns in the data. Too many epochs and the model starts memorizing noise specific to the training set rather than general patterns, which is exactly the overfitting behavior validation data is used to detect.

There's no universal correct value for epochs. In practice, it's picked by watching the validation loss during training (or using EarlyStopping) and stopping once validation loss stops improving, rather than hard-coding a number in advance and hoping it's right.

A model's accuracy on X_train tells you how well it fits the data it has already seen — it says nothing about how it will perform on new, unseen inputs. A model can reach 99% training accuracy while being nearly useless in production if it has simply memorized the training examples rather than learning generalizable patterns.

This is why deep learning workflows always hold out a validation set: a slice of labeled data the model never trains on, used purely to measure performance on unseen examples after each epoch. If training accuracy keeps climbing while validation accuracy stalls or drops, that gap is the textbook signature of overfitting.

Without a validation set, overfitting is invisible during training — the only loss number you see is one the model is actively being optimized to minimize, so of course it looks good. Validation loss is the number that hasn't been gamed by the optimizer, which is exactly what makes it trustworthy.

If you call model.fit(X_train, y_train, epochs=10) with no validation data at all, you will only ever see training loss and training accuracy in the output. Both metrics can look excellent right up until the model is deployed, because a large enough network can effectively memorize a training set — driving training loss toward zero — without having learned anything that transfers to new inputs.

The practical danger is that this failure mode is silent. There's no error, no exception, no warning in the console — just a model that looks great on paper and then produces poor predictions the moment it sees real, unseen data. By the time that's discovered, it's usually in production.

The fix is to always monitor validation metrics alongside training metrics from the very first training run, not just when something already seems wrong. Comparing training and validation loss on every epoch is the earliest and cheapest way to catch overfitting, well before it becomes a production incident.

The simplest way to get validation metrics is to add a single keyword argument: model.fit(X_train, y_train, epochs=10, validation_split=0.2). Keras takes the last 20% of X_train/y_train (before shuffling, by default) and holds it back entirely from training, evaluating the model against it after each epoch and reporting val_loss and val_accuracy alongside the training numbers.

This is different from a manual train/test split you create yourself with something like scikit-learn's train_test_split — validation_split operates on the arrays you already passed to fit(), splitting them internally. For more control (for example, a validation set that's already been carefully stratified), you can instead pass validation_data=(X_val, y_val) directly and skip validation_split altogether.

Once validation metrics are visible, the diagnostic pattern is straightforward: training loss falling while validation loss also falls means the model is generalizing well. Training loss falling while validation loss rises or plateaus is the clearest sign of overfitting, and usually the cue to add regularization, reduce model capacity, or stop training earlier.

Putting the pieces together: model.compile(optimizer=..., loss=..., metrics=...) configures how the model will learn, choosing the loss to match the problem shape (regression, binary, or multi-class). model.fit(X_train, y_train, epochs=..., batch_size=..., validation_split=...) then runs the actual training loop, iterating over the data in batches for the given number of epochs while holding back a validation slice to catch overfitting as it happens.

The reason this two-call structure matters is that each call answers a different question that would otherwise get tangled together: compile() is a one-time setup step that says how training should be measured and optimized, while fit() is the (potentially repeated) step that actually executes it. Calling fit() again on an already-compiled model continues training with the same configuration; changing the loss or optimizer requires recompiling first.

With this loop in place, the natural next question is what actually goes inside optimizer and loss beyond the string shortcuts ("adam", "mse") used so far — that's exactly what the next module on Loss Functions and Optimizers covers, including when to reach for a custom loss or a manually configured optimizer instance.

Module 03: Training the Model. In PyTorch, you write a 15-line training loop. In Keras, you literally just call model.fit().

Before training, you MUST call model.compile(). This tells Keras exactly how to measure success (Loss) and how to update weights (Optimizer).

What is the exact purpose of the model.compile() step in Keras?

• →It configures the model for training by specifying the Optimizer (how to learn) and the Loss Function (how to measure error).
• →It runs the training loop on the GPU.
• →It translates the Python code into C++.

The Loss function depends strictly on your problem. Regression = mse. Binary classification = binary_crossentropy. Multi-class = categorical_crossentropy.

If you are building an AI to classify an image as "Cat", "Dog", or "Bird" (3 distinct categories), which Loss Function MUST you use in .compile()?

• →binary_crossentropy
• →mean_squared_error
• →categorical_crossentropy

Once compiled, you train the model using model.fit(). You provide the training data, the labels, and the number of Epochs.

What does the epochs=10 parameter inside model.fit() instruct the Neural Network to do?

• →It limits the dataset to only 10 images.
• →It tells the network to iterate over the entire training dataset exactly 10 times to progressively learn patterns and reduce the loss.
• →It sets the GPU timeout to 10 minutes.

Now, prepare yourself. We are about to enter the ADA Defense Protocol. Ensure you understand Validation Splits.

If you only train on X_train, you will Overfit. You won't know if the model is memorizing the data or actually learning.

ADA DEFENSE: How do you configure model.fit() to automatically test the model on unseen data at the end of every epoch, ensuring it isn't just memorizing the training data?

• →Pass anti_memorize=True.
• →Pass the validation_split=0.2 argument, which automatically holds back 20% of the data exclusively for testing at the end of each epoch.
• →Run model.test() at the same time.

Threat neutralized. Validation metrics secured. Proceeding to Loss Functions and Optimizers.

Choose a Real Loss Function. Finish choose_loss(): the choice is mechanical based on the problem type.