Model Evaluation in Python

A model with 99% accuracy can still be completely useless in the real world. Why? Because of imbalanced data — when one class dominates the dataset, a classifier can reach a very high accuracy score simply by always predicting the majority class, without learning anything useful about the minority class you actually care about.

Take a spam filter trained on 100 emails where 99 are legitimate and only 1 is spam. A model that blindly labels every email 'Inbox' scores 99% accuracy while catching zero spam. The same pattern shows up constantly in real datasets: fraud detection, rare disease diagnosis, manufacturing defect detection — in each case the event you're trying to catch is rare, so accuracy alone tells you almost nothing about whether the model is actually useful.

This is why model evaluation in scikit-learn goes well beyond accuracy_score. The rest of this lesson builds up a toolkit — the confusion matrix, precision, recall, F1-score, and cross-validation — specifically designed to expose what a raw accuracy number hides.

To uncover what accuracy hides, we use a confusion matrix. It's a simple grid that breaks every prediction into one of four buckets: True Positives (correctly predicted positive), True Negatives (correctly predicted negative), False Positives (predicted positive but actually negative), and False Negatives (predicted negative but actually positive).

In scikit-learn, confusion_matrix(y_test, predictions) returns this breakdown as a 2D array. For the spam example, it would show that the naive 'always Inbox' model has zero True Positives and one False Negative — it never once correctly flagged the actual spam email — which the single accuracy number completely conceals.

Every other classification metric covered in this lesson — precision, recall, F1-score — is derived directly from these four counts. Reading the confusion matrix first makes it much easier to understand what those downstream metrics are actually measuring.

Why is a simple accuracy score dangerous when dealing with imbalanced datasets, like detecting a rare disease that only 1 in 10,000 people have? Because a model that always predicts 'healthy' scores 99.99% accuracy while missing every single true case — the exact failure mode this lesson opened with, now made concrete with medical stakes.

This matters in practice because accuracy is often the first metric beginners reach for, and in a balanced classroom dataset it works fine. Real-world classification problems — fraud, churn, disease, defect detection — are almost always imbalanced, so relying on accuracy there gives a false sense of confidence right up until the model fails on the cases that matter most.

The fix isn't a different single number; it's looking at the confusion matrix directly, and at precision and recall, which are sensitive to exactly the kind of failure accuracy is blind to.

From the confusion matrix we derive two metrics that matter far more than accuracy on imbalanced problems: precision and recall. Precision answers 'of everything I labeled positive, how much was actually positive?' — it's TP / (TP + FP). Recall answers 'of everything that was actually positive, how much did I find?' — it's TP / (TP + FN).

Using the spam example: precision asks, out of all the emails the model called spam, how many really were spam (low precision means annoying false alarms, legitimate emails sent to the spam folder). Recall asks, out of all the real spam emails that existed, how many did the model actually catch (low recall means spam slipping into the inbox undetected).

These two metrics trade off against each other — a model can trivially get perfect recall by labeling everything positive, tanking precision, or perfect precision by only ever predicting positive when it's extremely confident, tanking recall. Which one to prioritize depends entirely on the cost of each type of mistake in your specific problem.

If you're building a model to detect cancer, which matters more: precision or recall? The answer is recall. A false negative here means telling a person with cancer that they're healthy, sending them home without treatment — a catastrophic, potentially fatal mistake. A false positive means an unnecessary follow-up test, which is costly and stressful but not life-threatening.

This asymmetry is the whole point of choosing metrics deliberately instead of defaulting to accuracy or even a 50/50 balance of precision and recall. In medical screening, fraud detection, and safety-critical systems, missing a true positive (low recall) is usually far more expensive than a false alarm (low precision), so teams often tune the classification threshold to favor recall even at the cost of more false positives.

The opposite is true in other domains — a spam filter that's too aggressive (high recall, low precision) starts burying legitimate emails, which is its own kind of failure. There's no universally 'better' metric; the right choice depends on which type of error costs more in your specific application.

When you need a single number that balances precision and recall rather than picking one over the other, use the F1-score. It's the harmonic mean of the two: 2 * (precision * recall) / (precision + recall). scikit-learn computes it directly with f1_score(y_test, predictions).

The harmonic mean matters here specifically because it punishes extreme imbalance between precision and recall much more harshly than a simple average would. A model with precision 1.0 and recall 0.0 has an arithmetic mean of 0.5, which looks deceptively okay, but its F1-score is 0 — correctly reflecting that the model is useless despite one 'perfect' metric.

F1 is a good default when both false positives and false negatives carry roughly similar costs and you have no strong reason to prioritize one. When the costs are lopsided (as in the cancer example), a weighted variant like fbeta_score with beta > 1 to favor recall, or beta < 1 to favor precision, is usually a better choice than plain F1.

What is the purpose of the F1-score metric, concretely? It exists so you can compare and rank models with one number instead of juggling separate precision and recall values that might disagree about which model is 'better'. A model with precision 0.9/recall 0.5 and another with precision 0.6/recall 0.7 aren't directly comparable on either metric alone — F1 collapses each pair into a single comparable score.

In scikit-learn, f1_score(y_test, predictions) returns this value directly, and classification_report(y_test, predictions) prints precision, recall, and F1 together for every class in one readable table — usually the first thing worth running after predict() on any classifier.

For multi-class problems, F1 can be averaged across classes in different ways (average='macro', 'micro', or 'weighted' in scikit-learn), and the choice matters: macro treats every class equally regardless of size, while weighted accounts for class frequency — picking the wrong one can hide poor performance on a rare but important class.

This section marks a shift from classification metrics to a different evaluation problem entirely: how do you know your evaluation score itself is trustworthy? Everything covered so far — accuracy, precision, recall, F1 — is computed on a single test set, and that single number can be misleading depending on exactly which rows ended up in that test set.

This is the motivation for cross-validation, the technique covered next. Instead of trusting one train/test split, cross-validation repeats the evaluation across multiple different splits of the same data and reports the spread of results, giving a much more honest picture of how the model is likely to perform on new, unseen data.

Understanding this distinction — a single evaluation metric versus a robust, repeated evaluation procedure — is what separates a model score you can defend to stakeholders from one that might just be a lucky roll of the dice.

Relying on a single train_test_split is risky. If you happen to get a 'lucky split' — one where the test set is, by chance, easier than average, or the hard examples all landed in the training set — your reported score will be artificially high and won't hold up once the model meets real, unseen data. An 'unlucky split' has the opposite effect, making a genuinely good model look worse than it is.

scikit-learn solves this with cross-validation, most commonly K-Fold: the dataset is divided into K equal-sized folds, and the model is trained and evaluated K separate times, each time holding out a different fold as the test set and training on the rest. The result is K scores instead of one.

In practice, you rarely write the K-Fold loop by hand — cross_val_score(model, X, y, cv=5) does the splitting, training, and scoring automatically and returns an array of scores, which you typically summarize with their mean and standard deviation to report both expected performance and how much it varies across folds.

What does cross_val_score(model, X, y, cv=5) actually do under the hood? It splits the dataset into 5 roughly equal folds, then runs 5 separate training rounds. In each round, one fold is held out as the test set and the model is trained fresh on the remaining 4 folds, so every data point gets used for testing exactly once and for training four times across the whole procedure.

The function returns an array of 5 scores — one per round — rather than a single number. A tight cluster of scores (e.g. [0.91, 0.89, 0.92, 0.90, 0.88]) tells you the model's performance is stable across different subsets of the data. A wide spread (e.g. [0.95, 0.60, 0.88, 0.72, 0.91]) is a red flag: it usually means the dataset is small, noisy, or has some structure (like class imbalance or a time component) that a plain random split isn't handling well.

One subtlety worth knowing: cross_val_score retrains the model completely from scratch on each fold — it does not carry over learned parameters between rounds — which is exactly what makes each of the 5 scores an independent, fair estimate of generalization performance.

This wraps up the model evaluation toolkit: the confusion matrix to see the raw prediction breakdown, precision and recall to understand the two ways a classifier can be wrong, F1-score to compress both into one comparable number, and cross-validation to make sure that number isn't just a lucky split. Together they replace a single, misleading accuracy figure with a much more honest picture of how a model will actually perform.

In a real pipeline, these tools are used together rather than in isolation: cross_val_score gives you a robust estimate of expected performance, classification_report breaks that down by class into precision/recall/F1, and the confusion matrix pinpoints exactly which classes are being confused with which. Reaching for accuracy alone and stopping there is the single most common evaluation mistake in applied machine learning.

With solid evaluation habits in place, the next step is applying the same rigor to model selection and tuning — comparing different algorithms and hyperparameters using these same metrics rather than a single train/test score, which is where scikit-learn's grid search and pipeline tools come in.

A model with 99% accuracy can still be completely useless in the real world. Why? Because of Imbalanced Data.

To uncover this, we use a Confusion Matrix. It shows exactly where the model got confused, breaking down predictions into True Positives, False Positives, etc.

Why is a simple "Accuracy Score" dangerous when dealing with imbalanced datasets (e.g., detecting a rare disease that only 1 in 10,000 people have)?

• →Because a model that simply predicts 'Healthy' 100% of the time will technically have 99.99% accuracy, while entirely failing to detect the disease.
• →Because Accuracy is a metric only used for Regression, not Classification.
• →Because Scikit-Learn calculates Accuracy incorrectly.

From the Confusion Matrix, we derive two advanced metrics: Precision and Recall. Precision focuses on the quality of positive predictions. Recall focuses on finding ALL the positives.

If you are building an AI to detect Cancer, which metric is more important: Precision or Recall?

• →Recall. It is better to falsely flag a healthy person (False Positive) than to miss a person who actually has cancer (False Negative).
• →Precision. You never want to tell a healthy person they are sick.
• →Accuracy.

To balance Precision and Recall, Data Scientists use the F1-Score. It is the harmonic mean of both metrics, providing a single, trustworthy number for imbalanced data.

What is the purpose of the F1-Score metric?

• →It calculates the speed of the training algorithm.
• →It provides a single metric that balances both Precision and Recall, making it much more reliable than Accuracy for imbalanced datasets.
• →It measures Mean Squared Error.

Now, prepare yourself. We are about to enter the ADA Defense Protocol. Ensure you understand Cross-Validation.

Relying on a single train_test_split is risky. If you get a "lucky split", your score will be artificially high. Scikit-Learn solves this with Cross-Validation.

ADA DEFENSE: What does cross_val_score(cv=5) actually do under the hood?

• →It multiplies the accuracy by 5.
• →It splits the data into 5 different chunks, trains and tests the model 5 separate times using different chunks each time, and returns the 5 scores.
• →It runs the model on 5 different GPUs.

Threat neutralized. Model evaluation protocols verified. Proceeding to Deep Learning architectures.

Measure Real Precision and Recall. Finish evaluate(): precision measures correctness of positive predictions, recall measures how many real positives were caught.