By default, drop_duplicates() compares every column when deciding whether two rows are duplicates, keeping only the first occurrence of each unique combination and dropping the rest. The subset parameter restricts the duplicate check to specific columns, so rows are considered duplicates if just those columns match, even if other columns differ; keep controls which occurrence survives — 'first', the default, 'last', or False to drop every occurrence of anything that has a duplicate at all, keeping none of them.
1Understanding df.drop_duplicates()
By default, drop_duplicates() compares every column when deciding whether two rows are duplicates, keeping only the first occurrence of each unique combination and dropping the rest. The subset parameter restricts the duplicate check to specific columns, so rows are considered duplicates if just those columns match, even if other columns differ; keep controls which occurrence survives — 'first', the default, 'last', or False to drop every occurrence of anything that has a duplicate at all, keeping none of them.
Use subset=['email'] with drop_duplicates() to remove rows with a duplicate email specifically, even if other columns like name or signup date differ between those rows — the default behavior only catches rows that are duplicates across every single column.
import pandas as pd
df = pd.DataFrame({"name": ["Alice", "Bob", "Alice"], "age": [30, 25, 30]})
print(df.drop_duplicates())2Practical Example
Here is a real-world application of df.drop_duplicates() showing how it is used in production Pandas code.
import pandas as pd
df = pd.DataFrame({"email": ["a@x.com", "b@x.com", "a@x.com"], "name": ["Alice", "Bob", "Alice V2"]})
print(df.drop_duplicates(subset=["email"], keep="last"))3Best Practices
Follow these guidelines when working with df.drop_duplicates():
1. Use subset to define duplicates based on the columns that actually determine uniqueness for your data, like an email or ID, instead of relying on an exact full-row match
2. Check df.duplicated().sum() before dropping, to understand how many rows would actually be removed
3. Choose keep='last' instead of the default 'first' when a later row represents more up-to-date or authoritative information than an earlier one
Tip: Use subset=['email'] with drop_duplicates() to remove rows with a duplicate email specifically, even if other columns like name or signup date differ between those rows — the default behavior only catches rows that are duplicates across every single column.
import pandas as pd
df = pd.DataFrame({"name": ["Alice", "Bob", "Alice"], "age": [30, 25, 30]})
print(df.drop_duplicates())