Does PCA remove features, or does it create new ones?

It creates new ones. Each principal component is a linear combination (a weighted mix) of all the original features, not a subset of them — so after PCA you no longer have 'Age' or 'Income' as standalone columns, only combined axes like Component 1 and Component 2.

How many principal components should I actually keep?

There's no fixed answer — plot cumulative explained variance against the number of components and pick a threshold (commonly 90-95%) that balances information retention against dimensionality reduction. For pure visualization, 2 or 3 components are typically chosen regardless of variance captured.

Can PCA be used for classification tasks, or only regression?

PCA itself doesn't predict anything — it's a preprocessing step used before either classification or regression. It's commonly used to speed up and denoise the input features for any downstream supervised model, or purely to visualize high-dimensional labeled data in 2D.

HTML MASTER CLASS /// LEARN TAGS /// BUILD STRUCTURE /// SEMANTIC WEB /// HTML MASTER CLASS /// LEARN TAGS ///

Principal Component Analysis in Machine Learning

Learn about Principal Component Analysis in this comprehensive Machine Learning tutorial. Learn how to crush the 'Curse of Dimensionality'. Understand the role of feature scaling, the mechanics of orthogonal projection, and how to interpret explained variance to build efficient, compact machine learning pipelines.

⚡ Total XP: 0|💻 machinelearning XP: 0

Skill Matrix

UNLOCK NODES BY LEARNING NEW TAGS.

Dimension Crush

Reducing features.

Quick Quiz //

What is the primary goal of PCA?

🚀 LEVEL UP TO SENIOR:Unlock 500+ Advanced Practical Challenges & Exercises.

🎓 COURSERA PARTNER:Earn professional Google, Meta, and IBM certificates to supercharge your resume.

Too many features can confuse even the best models. PCA acts as a mathematical lens, focusing on the most important 'directions' of your data.

1The Curse of Dimensionality

As you add more features (dimensions) to a dataset, the space becomes increasingly sparse. This makes it harder for models to find patterns and easier for them to overfit. PCA solves this by projecting high-dimensional data onto a lower-dimensional subspace.

2Variance as Information

In PCA, we assume that features with the most spread (variance) contain the most information. The algorithm identifies the Principal Components—new, independent axes that capture the maximum possible variance from the original features.

3Interpretability Tradeoff

While PCA makes models faster and easier to visualize, it comes at a cost: Interpretability. Principal components are linear combinations of original features (e.g., a mix of 'Age' and 'Income'). You lose the ability to say exactly which original feature caused a specific prediction.

4Step-by-Step Breakdown

PCA is a dimensionality reduction technique. It compresses large datasets into fewer variables while retaining most of the original information.

Before running PCA, you MUST scale your data. Features with larger numerical ranges will disproportionately influence the algorithm.

Checkpoint: Why do we need to standardize features before applying PCA?

→To prevent large-scale features from dominating
→To make the algorithm run faster

Now we instantiate PCA and specify the number of components. These are the new 'directions' our data will be projected onto.

How much information did we keep? The 'Explained Variance Ratio' tells us the percentage of original variance captured by each component.

PCA is commonly used for data visualization (reducing to 2D or 3D) and as a pre-processing step to speed up other ML models.

Checkpoint: If PC1 has an explained variance of 0.60 and PC2 has 0.30, how much total variance is captured by these two components?

→30%
→90%

Dimensionality reduced! You've successfully compressed complex data without losing its soul. You're ready for high-dimensional challenges.

Run Real PCA Dimensionality Reduction. Finish scaling the data and compressing it down to 2 principal components.

Level Up 🚀

Advanced cheat sheets, SEO tricks, and interview prep for this topic.

Browser Support

ChromeSupported

Fully supported.

FirefoxSupported

Fully supported.

SafariSupported

Fully supported.

EdgeSupported

Fully supported.

Accessibility (A11y)

1Report Explained Variance as a Number, Not Only a Scree Plot

The 'elbow' in a scree plot is a visual cue that a screen reader user can't perceive — always state the cumulative explained variance in text (e.g., 'the first 2 components capture 90% of the variance') alongside any chart, so the component-selection reasoning is available without seeing the plot.

<p>Components 1-2 capture 90% of total variance.</p>

SEO Implications

1
Target 'Curse of Dimensionality' and 'Explained Variance' as Distinct Search Intents
Learners arrive at PCA content from very different angles — some searching 'why does my model perform worse with more features' and others searching 'how to interpret explained_variance_ratio_' — covering both phrasings explicitly widens the page's organic search reach beyond just the algorithm's name.

Best Practices

Always Fit the Scaler and PCA Transformer on Training Data Only

Fitting StandardScaler or PCA on the full dataset before splitting leaks information about the test set's distribution into training, inflating validation performance. Fit both on X_train, then use transform() (not fit_transform()) on X_test.

Choose n_components by Cumulative Explained Variance, Not an Arbitrary Round Number

Picking n_components=2 just because it's easy to plot ignores how much information is actually retained. Plot cumulative explained variance against component count and pick the smallest number of components that clears a threshold like 90-95%.

Frequent Bugs

THE BUG

Running PCA on unscaled features, causing components to be dominated entirely by whichever feature happens to have the largest numeric range (e.g., 'income in dollars' overwhelming 'age in years').

THE FIX

Always apply StandardScaler (or another scaler) before PCA — PCA calculates variance directly from the raw magnitude of each feature, so unscaled features with large ranges will mathematically dominate the principal components regardless of their actual real-world importance.

Real-World Examples

Compressing Sensor Data Before Feeding a Classifier

A manufacturing quality-control model ingests 50 correlated sensor readings per unit — PCA reduces those 50 features down to 8-10 principal components that retain over 95% of the variance, cutting both training time and overfitting risk before the compressed features are passed into a downstream classifier.

pca = PCA(n_components=0.95)  # keep enough components for 95% variance
X_reduced = pca.fit_transform(X_scaled)

Interview Prep

?Frequently Asked Questions

Pascual Vila

Frontend Instructor // Code Syllabus

Common Pitfalls & Errors

The Error //

Data Leakage

# Wrong
scaler.fit(X)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

# Correct
scaler.fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

The Solution //

Never use data from the validation or test sets to train your model. This includes fitting scalers or imputers on the entire dataset before splitting.

The Error //

Overfitting on small datasets

// Solution: Use techniques like Dropout, L2 Regularization, or Early Stopping to prevent the model from overfitting the training data.

The Solution //

Training a complex model (like a deep neural network) on a very small dataset usually leads to memorization instead of generalization. Use simpler models or apply strong regularization.