Data in the wild is rarely perfect. Missing entries, corrupted records, and null values are the norm. As a data scientist, your first job is to identify these gaps and decide whether to drop them or fill them with intelligent estimates.
1Identifying the Gaps
Pandas provides isna() and isnull() to detect missing data. By chaining these with .sum(), you can quickly see which columns are the most problematic and require your attention.
2Drop or Fill?
You have two main strategies: dropping or imputing. dropna() is fast but loses information. fillna() allows you to replace gaps with zeros, means, or medians, preserving the rest of the row's data for analysis.
3Step-by-Step Breakdown
Real-world datasets are messy. Missing values (NaN) can crash machine learning models. Let's learn how to fix them using Pandas.
First, we must identify where the missing data is. We use isna() chained with sum() to get a count per column.
The console shows us exactly how many Nulls exist in each column.
Checkpoint: Which Pandas method detects missing values, returning a boolean same-sized object?
The simplest solution is dropping rows with missing data using dropna(). This is fine if you have plenty of data.
Dropping data wastes information. Instead, we can IMPUTE (fill) missing values using fillna(). Let's fill NaNs with the column mean.
Checkpoint: Which method is used to replace missing values with a specific number or statistic?
Time to clean up real code. Complete the challenges below to earn your 'Null Hunter' achievement!
Fill Real Missing Values. Finish computing the mean age and filling the missing value with it.
Level Up 🚀
Advanced cheat sheets, SEO tricks, and interview prep for this topic.
Browser Support
Fully supported.
Fully supported.
Fully supported.
Fully supported.
Accessibility (A11y)
1Announce Data Quality Warnings in Text, Not Color Alone
A data-quality dashboard that flags problem columns only by highlighting them red is invisible to colorblind users and screen readers — pair the visual flag with explicit text ('Age: 12% missing, flagged for review') so the warning is conveyed through content, not just styling.
<li>Age: 12% missing — flagged for review</li>SEO Implications
- 1
Cleaned DataFrames Are Pipeline State, Not Page Content
The specific dataset being cleaned in this lesson's examples exists only in a script's runtime memory — this page's indexable value comes from its own explanation of when to drop versus impute missing data, not from any particular dataset's cleaned output.
Best Practices
Investigate Why Data Is Missing Before Choosing How to Handle It
Missing data has different causes — a sensor failure (Missing Completely at Random) versus a survey question people skip when the true answer is sensitive (Missing Not at Random) — and the correct handling strategy depends on which one you're facing. Blindly imputing without understanding the missingness mechanism can introduce systematic bias.
Always Check the Null Count Before and After Cleaning
Run df.isna().sum() before AND after your cleaning step, and assert the after-count matches your expectation. A dropna() or fillna() call with the wrong axis or subset argument can silently clean far more (or less) than intended.
Frequent Bugs
Calling fillna() or dropna() without inplace=True or reassigning the result, expecting the original DataFrame to be modified.
df.fillna(0) by default returns a new DataFrame and leaves the original df unchanged — a common mistake is calling it as a bare statement and then continuing to use the stale, un-cleaned df. Either reassign explicitly (df = df.fillna(0)) or pass inplace=True (though reassignment is generally the safer, more explicit pattern).
Real-World Examples
Handling Missing Survey Responses
A customer satisfaction survey has a 'salary' field that respondents often skip (likely Missing Not at Random, since people with lower salaries may be less willing to disclose it) — rather than filling it with the overall mean (which would bias estimates upward), the analysis team adds a separate 'salary_disclosed' boolean flag and imputes conservatively, preserving the fact that non-disclosure itself might correlate with the target variable.
df['salary_disclosed'] = df['salary'].notna()
df['salary'] = df['salary'].fillna(df['salary'].median())