Supervised Learning in Python

Supervised Learning is defined by one simple fact: every training example comes with the correct answer already attached. When you call model.fit(X, y), you're handing the algorithm both the features (X) and the ground-truth labels (y) it should have produced, and the algorithm's entire job is to find a mapping from one to the other that generalizes to examples it hasn't seen.

This is what distinguishes supervised learning from unsupervised learning, which only ever sees X and has to find structure without any answer key. Because supervised learning has labels to check itself against, it also gives you a very concrete way to measure success: compare what the model predicts against what the label actually says, using metrics like accuracy or mean squared error.

Almost every practical business use case you'll build in scikit-learn — spam detection, price prediction, churn prediction, medical diagnosis support — is supervised, precisely because historical labeled data (past emails marked spam, past houses with known sale prices) is usually available to train on.

Every supervised learning problem falls into one of exactly two buckets, and figuring out which one you're facing is the first decision you make before choosing an algorithm. Classification predicts a discrete category from a finite set of possibilities — spam or not spam, cat or dog or bird, approved or denied. Regression predicts a continuous number that could, in principle, take infinitely many values — a house price, a temperature, a stock return.

The distinction isn't cosmetic; it determines which algorithms, loss functions, and evaluation metrics are even valid to use. Accuracy makes sense for classification (what fraction of predictions were exactly right) but is meaningless for regression, where a prediction of $301,200 versus an actual price of $300,000 is 'close' rather than simply right or wrong — regression instead uses metrics like mean squared error that reward closeness.

A useful test when you're unsure which bucket a problem falls into: if you can enumerate every possible answer in advance (a fixed list of categories), it's classification. If the answer could be any number on a continuous scale, it's regression.

Predicting tomorrow's exact temperature, like 23.5 degrees, is a textbook regression problem, because temperature can take any value on a continuous scale — 23.4, 23.5, 23.51, and so on, with no natural set of discrete buckets to sort it into. There's no finite list of 'temperature categories' the way there is a finite list of email categories (spam vs. not spam).

It's easy to get tripped up if you think in terms of how the number will be displayed rather than what the model is actually predicting. A weather app might round 23.5 to '24°' for display, but the underlying prediction task — estimating a real-valued quantity — is still regression, not classification, regardless of how the output gets rounded afterward.

The practical consequence: for this task you'd reach for LinearRegression, RandomForestRegressor, or another *Regressor* estimator, and you'd evaluate it with a regression metric like mean absolute error (how many degrees off, on average) rather than accuracy.

Scikit-Learn bakes the classification/regression distinction directly into class names, so you can often tell what a model does just by reading the import line. DecisionTreeClassifier predicts discrete labels; DecisionTreeRegressor predicts continuous numbers — same underlying tree-splitting algorithm, different output layer and different loss function used to grow the tree.

This pattern repeats across almost the entire library: RandomForestClassifier/RandomForestRegressor, KNeighborsClassifier/KNeighborsRegressor, SVC/SVR (Support Vector Classifier/Regressor). Once you recognize the pattern, you can guess an unfamiliar estimator's purpose from its name alone, before ever reading its docstring.

The payoff isn't just readability — every estimator that follows this convention exposes the same fit(X, y) / predict(X) interface, so swapping a Classifier for a Regressor (or trying a different algorithm entirely) is usually a one-line change, not a rewrite of your training pipeline.

Seeing RandomForestClassifier in an import statement is a strong signal on its own: this estimator is built to output discrete categories, not continuous numbers. It works by training many individual decision trees on random subsets of the data and features, then having them vote — the category with the most votes becomes the prediction.

Its sibling, RandomForestRegressor, uses the exact same ensemble-of-trees mechanism, but instead of voting on a category it averages each tree's continuous prediction. The 'Random Forest' part of the name describes the *algorithm*; the 'Classifier' or 'Regressor' suffix tells you the *problem type* it's wired to solve.

This matters in practice because plugging a classification target (like 'Dog'/'Cat') into RandomForestRegressor, or a continuous target (like price) into RandomForestClassifier, either throws an error or silently produces nonsense — scikit-learn won't stop you from making that mistake, so recognizing the suffix is your first line of defense.

Training a model with model.fit(X_train, y_train) only produces something that *can* predict — it says nothing about whether those predictions are any good. To find out, you hold back a test set the model never saw during training, generate predictions on it with model.predict(X_test), and compare those predictions against the real answers in y_test.

accuracy_score(y_test, predictions) does exactly that comparison: it counts what fraction of predictions exactly matched the true label and returns a number between 0 and 1 (or 0% to 100%). A score of 0.95 means the model got 95% of the test examples exactly right.

The critical detail is *which* data you evaluate on. Scoring the model against X_train/y_train tells you how well it memorized data it already saw — evaluating on a held-out X_test/y_test split is what actually tells you how the model will perform on new, unseen data.

The basic workflow for judging a classification model is straightforward: fit on training data, predict on test data, then run accuracy_score(y_test, predictions) to see what fraction of predictions were correct. For a roughly balanced problem — like classifying images into five equally common categories — accuracy alone is a reasonable first signal.

But accuracy can be dangerously misleading when classes are imbalanced. If 99% of transactions in a fraud dataset are legitimate, a model that always predicts 'not fraud' scores 99% accuracy while catching zero actual fraud — that's why real evaluation pipelines also look at precision, recall, and the F1 score, which separate 'how often was it right' from 'did it catch the cases that actually mattered.'

The practical rule: use accuracy_score as a quick sanity check, but before shipping a classifier on an imbalanced or high-stakes dataset, check sklearn.metrics.classification_report for the fuller picture.

Picking the wrong side of the classification/regression divide isn't a cosmetic mistake — it's a design decision baked into which estimator, loss function, and evaluation metric you use, and getting it wrong doesn't always throw a helpful error. Scikit-Learn will happily let you fit LinearRegression on a target that's actually a category; it just quietly produces numbers that don't mean what you think they mean.

This is exactly the kind of mistake that survives code review and unit tests, because the code runs without crashing. The model trains, .predict() returns values, and everything looks fine until someone tries to interpret a regression output like '0.83' as a probability of default, or feeds a continuous prediction into logic that expects a discrete label.

The fix is a habit, not a tool: before you import an estimator, name the problem out loud — 'this is classification' or 'this is regression' — and only then pick the class whose suffix matches.

The word 'regression' trips people up because it sounds like it should mean 'going backward.' In statistics it means something much narrower: predicting a continuous, quantitative variable from other variables. The term traces back to Francis Galton's 19th-century observation that the heights of children tended to 'regress toward the mean' of the population rather than continuing to extremes — the name stuck to the whole family of techniques that model one continuous variable as a function of others.

That historical accident is worth knowing precisely because it's misleading: nothing about LinearRegression or RandomForestRegressor involves moving backward through anything. What they all share is a continuous, real-valued target — a price, a temperature, a duration — rather than a label drawn from a fixed set of categories.

Once you decouple the everyday meaning of the word from its statistical one, the naming convention in scikit-learn stops being confusing: 'Regressor' just means 'predicts a number,' full stop.

Imagine a bank wants to predict whether a customer will 'Default' or 'Not Default' on a loan — a two-category target — and someone imports LinearRegression to do it. The model will still call .fit() and .predict() without complaint, because scikit-learn doesn't inspect your target and refuse to run; it just does the linear algebra it's designed to do.

The problem is what comes out the other end. LinearRegression is built to output any real number, so predictions might come back as -0.3, 0.42, or 1.8 — numbers that were never constrained to represent 'Default' vs 'Not Default' in any principled way, let alone a calibrated probability. Thresholding those outputs at 0.5 to fake a classification decision throws away the statistical guarantees that an actual classifier like LogisticRegression or RandomForestClassifier provides.

The fix is to match the estimator family to the target type: a two-category outcome like loan default calls for LogisticRegression (which despite its name is a classifier — it models the probability of class membership) or another *Classifier* estimator, not a regressor coerced into acting like one.

With the classification/regression split under control, the rest of Module 02 is about applying it: swapping in specific algorithms — decision trees, random forests, k-nearest neighbors, support vector machines — while reusing the same fit(X, y) / predict(X) interface every scikit-learn estimator shares.

That consistency is the payoff for internalizing the naming convention now. Once you can look at KNeighborsClassifier or SVR and immediately know both the algorithm family and the problem type it targets, comparing multiple models on the same dataset becomes a matter of swapping one import line, not rewriting your training pipeline.

The next lessons build directly on this foundation, walking through individual classifiers and regressors and the hyperparameters that control how each one balances underfitting against overfitting.

Module 02: Supervised Learning algorithms. Supervised Learning means we are training the model on data that already has the correct answers (labels).

Supervised Learning is split into two main problems: Classification (predicting a category) and Regression (predicting a continuous number).

If you want an algorithm to predict the exact temperature for tomorrow (e.g. 23.5 degrees), which type of Supervised Learning is this?

• →Classification (because the temperature is a category).
• →Regression (because temperature is a continuous number, not a discrete category).
• →Unsupervised Learning.

Scikit-Learn makes it easy to distinguish them. Classification models usually end in "Classifier", while Regression models end in "Regressor".

If you import RandomForestClassifier, what type of problem is this algorithm designed to solve?

• →Predicting discrete categories (like 'Dog' vs 'Cat' or 'Spam' vs 'Inbox').
• →Predicting continuous numerical values (like Stock Prices).
• →Scaling data.

Once a Supervised model is trained, you evaluate its accuracy by comparing its predictions against the actual hidden answers in the testing set.

How do you evaluate if a Supervised Learning classification model is actually good?

• →By seeing how fast it calls the fit() method.
• →By using functions like accuracy_score() to compare the model's predictions on X_test against the real answers in y_test.
• →By checking if X_train equals y_train.

Now, prepare yourself. We are about to enter the ADA Defense Protocol. Ensure you understand what Regression actually means.

Regression doesn't mean "going backward". In statistics, it refers to predicting continuous quantitative variables.

ADA DEFENSE: A bank wants to predict if a user will "Default" or "Not Default" on a loan. You import LinearRegression. Why will this fail catastrophically?

• →Because LinearRegression requires Deep Learning GPUs.
• →Because 'Default' vs 'Not Default' are two categories (Classification), but LinearRegression is designed to output infinite continuous numbers (like 1.4532...).
• →Because the bank needs Unsupervised Learning.

Threat neutralized. Model architecture understood. Proceeding to algorithm implementation.

Compare Real Classifier and Regressor Output. Finish compare_outputs(): a Classifier predicts a category, a Regressor predicts a number.