Metrics in Python

Every Keras model juggles two different kinds of numbers during training: a loss and one or more metrics. The loss — binary_crossentropy, mse, categorical_crossentropy — is the value the optimizer actually differentiates and minimizes through backpropagation. It is chosen because it is mathematically well-behaved (smooth, differentiable), not because a human can look at 0.1423 and understand what it means.

Metrics exist purely for the human on the other side of the screen. "Accuracy: 95%" or "Precision: 0.87" is something a product manager, a stakeholder, or you at 2 a.m. can interpret at a glance. Keras computes metrics after every batch or epoch, but — critically — never uses them to update a single weight.

This separation matters because the two numbers can tell very different stories. A model's loss can keep decreasing smoothly while a metric like accuracy stalls or even gets worse on validation data, which is usually the first sign of overfitting.

You declare which metrics to track directly inside model.compile(), alongside the optimizer and loss — for example model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"]).

From that point on, Keras computes every listed metric automatically at the end of each batch during model.fit(), and reports it in the progress bar and the returned History object — no extra code required. You can pass metric names as strings ("accuracy", "precision", "recall") or import the metric classes directly (tf.keras.metrics.Precision()) when you need to configure thresholds or per-class behavior.

Because metrics are decoupled from the loss, you can track as many as you want — accuracy, precision, recall, AUC — simultaneously, without any of them affecting how the model actually learns.

The core distinction to internalize: the loss function is mathematical fuel for backpropagation, while metrics are a read-only dashboard. During training, TensorFlow computes gradients of the loss with respect to every trainable weight and nudges those weights in the direction that reduces the loss — metrics never enter that gradient calculation.

This is why you can pick a loss that's technically 'ugly' to a human (like binary_crossentropy) but perfect for gradient descent (smooth, differentiable everywhere), while simultaneously tracking a metric like accuracy that's intuitive for humans but mathematically unsuitable for optimization (it's a step function — not differentiable, so gradients through it are meaningless).

Getting this backwards is a common beginner mistake: trying to directly optimize accuracy instead of a proper loss function almost never works, because there's no useful gradient signal to follow.

Accuracy — the percentage of predictions that are correct — feels like the obvious metric to optimize for. It becomes dangerously misleading the moment your classes are imbalanced. Imagine a model screening for a disease that affects only 1% of the population. A model that outputs "Healthy" for every single patient, without looking at a single input feature, scores 99% accuracy.

That number looks like a triumph on a dashboard, but the model is medically useless — it has a 0% detection rate for the one class that actually matters. This is the "accuracy paradox": on skewed datasets, accuracy rewards a model for mirroring the majority class rather than for learning anything meaningful about the minority class.

The same trap appears constantly in production ML: fraud detection (fraud is rare), spam filtering, defect detection on a manufacturing line — anywhere the interesting class is a small minority of the data.

The accuracy paradox generalizes to any highly imbalanced classification task, not just medical screening. In fraud detection, where roughly 99.9% of transactions are legitimate, a model that never flags anything as fraud still reports 99.9% accuracy — while catching zero fraud, which is the entire point of building the model in the first place.

The root cause is that accuracy treats every misclassification as equally costly, but in imbalanced problems the errors are wildly asymmetric: missing a fraudulent transaction (a false negative) is far more expensive than flagging a legitimate one for review (a false positive). Accuracy has no way to express that asymmetry — it just counts total correct predictions over total predictions, letting the massive majority class dominate the score.

This is exactly why practitioners reach for Precision, Recall, F1-score, or AUC on imbalanced datasets instead of accuracy alone — each of those metrics is sensitive to how the model performs specifically on the minority class.

Precision answers a very specific question: of everything the model flagged as positive, how much of it was actually positive? Formally, Precision = True Positives / (True Positives + False Positives).

In code, this looks like importing the metric class directly: from tensorflow.keras.metrics import Precision, then passing metrics=[Precision()] inside model.compile().

High precision means very few false alarms — when the model says "fraud", it's almost always right. That makes precision the metric to prioritize whenever a false positive is costly or annoying: falsely flagging a legitimate credit card transaction irritates a customer and burns a human reviewer's time, so a fraud team optimizing for precision wants to be confident every flagged case is worth investigating.

Precision says nothing, however, about how much fraud the model *missed* — that's a completely separate question, answered by recall.

A model with high precision but low recall on cancer detection has a very specific — and dangerous — failure profile. High precision means that whenever it does predict "cancer", it's almost always correct: very few healthy patients get a false alarm. But low recall means it's only catching a small fraction of the actual cancer cases in the dataset — it's being extremely conservative, only flagging the most obvious, unambiguous cases and staying silent on everything else.

In plain terms: the model rarely cries wolf, but it also lets a lot of real wolves through. For a cancer screening tool, that's close to the worst possible trade-off, because the cost of a missed diagnosis (a false negative) is far higher than the cost of an unnecessary follow-up test (a false positive).

This is why precision and recall have to be read together, never in isolation — a single high number can hide a serious weakness in the other.

Before the next check, make sure precision and recall are cleanly separated in your head, because they answer opposite questions. Precision looks at the model's positive predictions and asks how many were correct. Recall is about to flip that lens entirely: it looks at all the real positive cases in the dataset and asks how many the model actually found.

The two metrics trade off against each other. A model can trivially get 100% recall by flagging every single input as positive — it will never miss a real case, but its precision will collapse because of all the false alarms. Likewise, a model can get near-perfect precision by only flagging the single most obvious case — but its recall will be terrible.

Understanding where your application sits on that trade-off is the whole game of choosing metrics for imbalanced problems.

Recall — also called sensitivity or the true positive rate — answers the mirror-image question to precision: of all the actual positive cases that exist in the dataset, how many did the model successfully find? Formally, Recall = True Positives / (True Positives + False Negatives).

High recall means the model misses very few real positives — it casts a wide net and catches almost everything, even if that means occasionally flagging something that turns out to be negative. That makes recall the metric to prioritize whenever a false negative (a missed case) is the expensive, dangerous, or catastrophic outcome: missed fraud, an undetected tumor, an undetected structural defect.

Like precision, recall says nothing on its own about false positives — a model can achieve perfect recall by flagging everything, which is why recall is almost always evaluated alongside precision, never alone.

Self-driving pedestrian detection is the textbook case for choosing recall over precision. A false positive here — the car braking hard because it mistook a shadow or a mailbox for a person — is uncomfortable and annoying for the passengers, but ultimately harmless. A false negative — failing to detect an actual pedestrian in the road — is catastrophic and potentially fatal.

Because the cost of the two error types is wildly asymmetric, the engineering decision is clear: tune the model (and its decision threshold) to maximize recall, accepting a higher rate of false alarms as the price of near-zero missed detections. This is the general pattern for any safety-critical system: identify which error type is unacceptable, and optimize the metric that punishes that error type the hardest.

Contrast this with something like a spam filter, where a false positive (a real email marked as spam) is often worse than a false negative (a spam email reaching the inbox) — the right metric genuinely depends on the cost structure of the application, not a fixed rule.

With loss versus metrics, and precision versus recall, sorted out, the next lesson moves to a closely related topic: Callbacks, the mechanism Keras uses to react to those metrics *during* training — stopping early when validation loss stops improving, saving the best checkpoint automatically, or reducing the learning rate when a metric plateaus.

As a quick recap: never let "Accuracy" be the only number you look at on an imbalanced dataset. Check precision when false positives are costly, check recall when false negatives are costly, and remember that both are metrics for humans — the model itself is still being optimized against the loss function the whole time.

Loss Functions are for the computer. "Metrics" are for humans. The computer optimizes binary_crossentropy, but humans want to see "Accuracy".

You define metrics inside model.compile(). Keras will calculate them automatically during model.fit().

What is the primary difference between a "Loss Function" and a "Metric" in Keras?

• →There is no difference; they are identical.
• →The network uses the Loss Function to mathematically update its weights via backpropagation; Metrics are solely calculated for humans to monitor performance.
• →Metrics update the weights; Loss is for humans.

Accuracy is dangerous. Imagine an AI detecting a rare disease that only affects 1% of the population. If the AI simply guesses "Healthy" every single time, it will have 99% Accuracy.

In highly imbalanced datasets (e.g., fraud detection where 99.9% of transactions are legitimate), why is "Accuracy" a terrible metric?

• →Because accuracy is too hard for the GPU to calculate.
• →Because a model can achieve near-perfect accuracy simply by ignoring the rare class and guessing the majority class every single time.
• →Because accuracy cannot go above 50%.

To solve this, we use specialized metrics. "Precision" asks: Of all the transactions the AI FLAGGED as fraud, how many were ACTUALLY fraud?

If your AI has High Precision but Low Recall in predicting Cancer, what does that mean?

• →It means the AI is perfectly balanced.
• →It means that when the AI does predict Cancer, it is almost always right (few false alarms), but it is probably missing a lot of actual cancer patients.
• →It means the model is broken.

Now, prepare yourself. We are about to enter the ADA Defense Protocol. Ensure you understand Recall.

Recall asks: Of ALL the actual fraud in the dataset, how much did the AI successfully find? (High Recall = Few Missed Targets).

ADA DEFENSE: You are programming the AI for a self-driving car to detect pedestrians. Is it more important to have High Precision (never braking for shadows) or High Recall (never missing a real pedestrian)?

• →High Precision. False alarms are the worst.
• →High Recall is absolutely critical. A False Positive (braking for a shadow) is annoying, but a False Negative (missing a real pedestrian) is catastrophic.
• →Accuracy is the only thing that matters.

Threat neutralized. Precision/Recall trade-off acknowledged. Proceeding to Callbacks.

Compute Real Accuracy. Finish compute_accuracy(): accuracy is just the fraction of exact matches.