Listen up. If you're going to process data in Python, you need to understand Data Cleaning Fundamentals in Python. This is where data engineers separate themselves from script kiddies. It's about writing code that scales.
1The Three Categories of Bad Data
Real datasets are rarely analysis-ready. Pandas' cleaning tools address three recurring problems: empty cells (missing values, represented as NaN), data stored in the wrong format (a date column read as plain text, a price stored as a string with a currency symbol), and data that's simply wrong (an age of 999, a negative quantity). Each category needs a different fix, and applying the wrong one ā say, blindly filling every NaN with zero ā can quietly distort an analysis instead of correcting it.
For missing values, Pandas gives you two opposing strategies: df.dropna() removes rows (or columns) containing NaN entirely, while df.fillna(value) replaces them with something else, often the column mean, median, or a sentinel value. Which one is right depends on scale and cause: dropping five bad rows out of five million is safe, but dropping a column because 40% of its rows are missing throws away a lot of signal ā imputation is usually the better call there.
The reason this matters beyond tidiness is that most downstream tools, including every standard Scikit-Learn estimator, cannot handle NaN values at all ā feeding a DataFrame with missing data straight into .fit() raises a ValueError rather than silently working around it. Cleaning isn't optional polish; it's a hard prerequisite for anything built on top of the data.
# Example
import pandas as pd
print("Running Pandas...")Data processed and aggregated.
2Step-by-Step Breakdown
Welcome to Module 03: Data Cleaning. In the real world, data is never perfect. It is messy, incomplete, and full of errors.
Data Scientists spend 80% of their time cleaning and organizing data. If you feed garbage data into a Machine Learning model, it will output garbage predictions.
What is the widely accepted acronym in computer science for the concept that flawed input data produces flawed output?
- āGIGO (Garbage In, Garbage Out)
- āDRY (Don't Repeat Yourself)
- āKISS (Keep It Simple Stupid)
Pandas provides a suite of tools to handle the three main types of bad data: Empty cells (NaN), Data in the wrong format, and Completely wrong data.
Which of the following is NOT one of the typical categories of "bad data" that we must clean?
- āMissing or Empty cells (NaN)
- āPerfectly formatted integer arrays
- āData in the wrong format (like a date stored as a string)
Cleaning data often involves deciding whether to delete the bad rows entirely, or try to replace the bad values with an average or default value.
If your dataset has millions of rows and only 5 rows contain missing data, what is generally the safest approach?
- āDelete the 5 rows using dropna()
- āDelete the entire column that contains the missing data
- āLeave them as NaN and hope the model ignores them
Now, prepare yourself. We are about to enter the ADA Defense Protocol. Ensure you understand the consequences of uncleaned data.
ADA DEFENSE: If you pass a DataFrame containing NaN (Not a Number) values directly into a standard Scikit-Learn machine learning model, what will happen?
- āThe model will automatically delete those rows for you.
- āThe model will crash and throw a ValueError.
- āThe model will assume the missing values are zeroes.
Threat neutralized. You understand the necessity of sanitization. We proceed to empty cell handling.
Fill Real Missing Values with the Mean. Finish fill_with_mean(): replace NaN with the column's own average.
Level Up š
Advanced cheat sheets, SEO tricks, and interview prep for this topic.
Browser Support
Fully supported.
Fully supported.
Fully supported.
Fully supported.
Accessibility (A11y)
1Document Imputation Choices
When you fill missing values with a mean or default, leave a code comment or a data-quality note explaining the choice ā anyone consuming exported reports (including via screen readers) needs to know a value was estimated, not observed.
# Imputed: missing 'age' filled with column median (12 of 5,000 rows)
df["age"] = df["age"].fillna(df["age"].median())SEO Implications
- 1
High-Intent Reference Content
Searches like 'pandas dropna vs fillna' and 'pandas handle missing data' are extremely common among people learning data analysis, making a clear, example-driven explanation of the cleaning workflow valuable evergreen content.
Best Practices
Decide Drop vs. Fill Based on Scale, Not Habit
Dropping a handful of NaN rows out of millions is safe; dropping a column because a chunk of it is missing throws away signal. Check what fraction of the data is affected before choosing dropna() or fillna().
Validate Ranges After Cleaning, Not Just Nulls
NaN checks catch missing data but not wrong data (age = 999, negative prices). Add explicit range or sanity checks (e.g. df[df["age"] > 120]) as part of the cleaning step.
Frequent Bugs
Passing a DataFrame with NaN values straight into a Scikit-Learn model or a groupby aggregation and getting a confusing ValueError or silently wrong result.
Explicitly handle missing values with dropna() or fillna() as a dedicated cleaning step before any modeling or aggregation, rather than discovering NaNs downstream.
Real-World Examples
Cleaning a Messy Signup Dataset
A signup form export has a few rows with a missing 'age' field and one row where age was entered as 999 by mistake.
# Fix the obviously wrong value
df.loc[df["age"] > 120, "age"] = pd.NA
# Impute the small number of missing ages with the median
df["age"] = df["age"].fillna(df["age"].median())