Linear Regression in Python

Linear Regression is the simplest and most foundational algorithm in scikit-learn's toolbox. Given a set of numeric features and a continuous target — house price, temperature, salary — it searches for the single straight line (or hyperplane, once you have more than one feature) that best represents the relationship between the inputs and the output. Instantiating LinearRegression() from sklearn.linear_model doesn't do any work by itself; it just creates an untrained estimator object waiting for data.

What makes Linear Regression the natural first stop for a regression problem is that its output is directly interpretable: once fitted, the model literally is the equation of the line. There's no black-box weight matrix to decode — every coefficient tells you exactly how much the prediction moves when that one feature increases by one unit, holding the others constant.

That interpretability is also its biggest constraint. A straight line can only capture relationships that are actually linear. If the true pattern in your data curves, Linear Regression will still dutifully draw its best straight-line approximation — it just won't fit well, which is exactly the failure mode explored later in this lesson.

Calling model.fit(X, y) is where the actual learning happens. Under the hood, scikit-learn's LinearRegression uses Ordinary Least Squares (OLS): it mathematically solves for the slope and intercept values that minimize the sum of the squared vertical distances between the line and every data point.

Squaring the errors before summing them is a deliberate choice, not an accident. It punishes large misses far more than small ones — a prediction that's off by 10 contributes 100 to the error sum, while one off by 1 contributes only 1 — and it also makes the optimization problem solvable with a closed-form matrix equation rather than requiring iterative search.

Because OLS has a closed-form solution, fit() on a small-to-medium dataset is nearly instantaneous: there's no learning rate to tune and no risk of the training loop diverging. That's a big part of why Linear Regression remains the default first model to try before reaching for anything more complex.

The exercise above asks you to name the actual objective a LinearRegression model is optimizing for. The answer is precise: it draws a single straight line that minimizes the total squared distance between that line and every point in your dataset — not a curve that passes through every point, and not a boundary that separates categories.

This distinction matters because it's easy to conflate regression with classification once you've used scikit-learn's consistent .fit() / .predict() API for both. The API looks the same, but the underlying math is completely different: classifiers like SVC search for a separating boundary between discrete classes, while LinearRegression searches for the best-fitting continuous line through numeric targets.

Keeping that separation clear early on prevents a common beginner mistake: reaching for LinearRegression on a categorical target (like 'spam' vs 'not spam') where a classifier, not a regressor, is what the problem actually calls for.

Once fit() completes, the trained model doesn't just sit there as an opaque object — it exposes the exact formula it learned. model.coef_ holds the learned weight for each input feature (the slope), and model.intercept_ holds the constant offset (where the line crosses the y-axis when every feature is zero).

Together, these two attributes reconstruct the prediction formula: y = (coef_ * X) + intercept_. For a model trained on a single feature, coef_ is a single number; with multiple features it becomes an array, one weight per column of X, in the same order the columns appeared during training.

This is what separates Linear Regression from most other scikit-learn estimators: you can print model.coef_ and hand the resulting equation directly to a non-technical stakeholder. A Random Forest or SVM can't offer that same level of direct, human-readable transparency.

This check reinforces where the learned parameters actually live after training. Scikit-Learn stores the per-feature weights in the model.coef_ attribute — not in a .slopes() method, and not by mutating X_train in place. Scikit-Learn's convention is consistent across nearly every estimator: learned parameters are exposed as attributes ending in a trailing underscore (coef_, intercept_, feature_importances_ on tree-based models, and so on), which is the library's signal for 'this value only exists after .fit() has been called'.

Trying to read model.coef_ before calling .fit() raises a NotFittedError — a deliberate guardrail that stops you from accidentally using an untrained model's attributes, which would otherwise fail silently or return garbage.

Recognizing the trailing-underscore convention is a small detail, but it pays off across the whole scikit-learn ecosystem: it's the same pattern you'll rely on later to read feature_importances_ off a Random Forest or support_vectors_ off an SVM.

Evaluating a regressor with plain 'accuracy' doesn't make sense, because accuracy only means something when predictions can be exactly right or exactly wrong. Predicting $300,000 for a house that actually sold for $300,001 is, strictly speaking, wrong — but it's also an excellent prediction. Regression needs a metric that measures how far off a prediction is, not whether it matches exactly.

Mean Squared Error (MSE), imported from sklearn.metrics, does exactly that: it takes the difference between each predicted value and its true value, squares each difference, and averages the results. Squaring again serves double duty here — it makes every error positive (so overshoots and undershoots don't cancel out) and it disproportionately penalizes the predictions that are furthest from the truth.

Because MSE is in squared units (dollars-squared, for a price prediction), many practitioners report its square root instead — RMSE — which brings the error back into the original, interpretable unit of the target variable.

This check tests whether you understand why accuracy is the wrong tool here, not just that it is. accuracy_score counts exact matches between predicted and true labels — a perfectly sensible question for classification, where there's a finite set of discrete classes to get right or wrong. Continuous targets don't have that property: there are effectively infinite possible numeric outputs, so requiring an exact match would make almost every prediction 'wrong' regardless of how close it actually was.

MSE (and its relatives, like Mean Absolute Error and R²) instead measure distance from the truth, which is the only evaluation approach that makes sense for a continuous target. A model that consistently predicts within a few percent of the true value should score well, even though it never lands on the exact number.

Mixing this up — running accuracy_score against a regressor's output — is a real, common scikit-learn mistake, and it will either throw an error or silently produce a meaningless near-zero score, because two floating-point numbers are almost never bit-for-bit equal.

Every algorithm has a breaking point, and Linear Regression's is baked into its name: it can only ever fit relationships that are actually linear. The ADA Defense Protocol that follows is designed to test whether you understand exactly where that boundary sits — not just how to call .fit() and .predict(), but when this model is fundamentally the wrong tool for the job.

This matters in practice because Linear Regression won't warn you when it's the wrong choice. Fit it against clearly curved or cyclical data and it will still return coefficients, a line, and predictions — it just won't be a good line, and the resulting MSE will quietly tell that story if you know to check it.

Understanding this limitation up front is what separates someone who can call scikit-learn functions from someone who can actually choose the right algorithm for a given dataset.

The core assumption baked into every LinearRegression model is linearity: it assumes that, on average, a fixed increase in a feature produces a fixed, constant change in the target — the same relationship whether the feature is small or large. That assumption holds reasonably well for things like 'square footage vs. house price' over a normal range, but it breaks down completely for relationships that curve, plateau, or oscillate.

A classic real-world example of a non-linear relationship is anything governed by physics involving acceleration — the height of a thrown object over time, for instance, follows a parabola, not a straight line. No amount of retraining or more data will make LinearRegression fit a parabola well, because the model literally does not have the mathematical capacity to represent a curve.

Recognizing this assumption is what tells you when to reach for a different tool — polynomial features, a tree-based model, or an SVM with a non-linear kernel — instead of continuing to fight a model that's structurally unsuited to the data.

This is the payoff question for the whole lesson: a bouncing ball's height over time traces a parabolic arc, not a straight line, so LinearRegression is structurally incapable of modeling it well — no matter how much training data you throw at it or how long you let .fit() run (and note that OLS doesn't have 'epochs' at all; it solves the equation in one step). The massive MSE isn't a bug or a fluke, it's the model correctly reporting that a straight line is a poor fit for curved motion.

The fix isn't to tune hyperparameters on LinearRegression — there's essentially nothing to tune. The fix is to give the model the mathematical capacity to represent a curve, either by engineering polynomial features (feeding in X and X² so a linear combination of those two terms can trace a parabola) or by switching to an algorithm that isn't restricted to straight lines in the first place, like a decision tree or an SVM with a non-linear kernel.

This is the pattern to internalize: a large error on a specific, well-understood shape of data (parabolic, cyclical, categorical) is a signal to reconsider the algorithm, not just to reach for more data or more compute.

With the straight-line assumption clearly understood — what it buys you in interpretability, and exactly where it breaks down on curved or non-linear data — you're ready to move past the simplest regression case. The next lessons in this module build on the same scikit-learn workflow (fit, predict, evaluate) but swap in algorithms that don't share Linear Regression's straight-line limitation, starting with tree-based models that split data with a sequence of yes/no questions instead of drawing a single line through it.

The evaluation habits from this lesson carry forward unchanged: you'll still be reaching for mean_squared_error (or its relatives) any time the target is a continuous number, regardless of which algorithm produced the prediction. What changes going forward is the shape of the decision boundary the model is capable of drawing — not how you measure whether it's any good.

Keep the coef_ / intercept_ mental model in your back pocket too: it's the clearest possible illustration of what a trained model actually is, and it makes the more opaque models you're about to meet easier to reason about by contrast.

Linear Regression is the foundation of predictive math. It attempts to draw a perfectly straight mathematical line through your data points.

When you call model.fit(X, y), the algorithm mathematically minimizes the "distance" between the data points and the line it draws.

What is the primary visual/mathematical goal of a standard LinearRegression model when it looks at your data points?

• →To draw a curved line that connects every single dot perfectly.
• →To draw a single, straight line that minimizes the total distance (error) between the line and all the data points.
• →To separate data into discrete categories.

Once trained, the model literally stores the mathematical formula of that line inside model.coef_ (the slopes/weights) and model.intercept_.

After training a LinearRegression model, where does Scikit-Learn store the learned mathematical weights (slopes) for each feature?

• →In the model.slopes() method.
• →In the model.coef_ attribute.
• →In the X_train variable.

To evaluate a Regressor, we cannot use "Accuracy" (because guessing $300,000 for a $300,001 house is functionally 0% accuracy). We use Mean Squared Error (MSE).

Why do we use metrics like mean_squared_error (MSE) instead of accuracy_score to evaluate Regression models?

• →Because Python does not allow accuracy_score on NumPy arrays.
• →Because predicting exact continuous numbers is virtually impossible; we need to measure the average 'distance' or error, rather than exact matches.
• →MSE actually stands for Model Scoring Execution, which is the same as Accuracy.

Now, prepare yourself. We are about to enter the ADA Defense Protocol. Ensure you understand the limitations of straight lines.

A Linear Regression model assumes your data has a linear relationship.

ADA DEFENSE: You are trying to predict the path of a bouncing ball. You use LinearRegression, but the error (MSE) is massive. Why is LinearRegression failing so badly?

• →You didn't run the model for enough epochs.
• →A bouncing ball follows a parabolic (curved) arc. Linear Regression only draws straight lines, so it inherently cannot model complex curves.
• →The ball is moving too fast for Scikit-Learn to process.

Threat neutralized. Linear relationships understood. Proceeding to non-linear algorithms.

Train a Real Linear Regression. Finish train_and_get_slope(): after fit(), the learned slope lives in model.coef_.