Pandas dataframe.drop_duplicates()
Last Updated :
13 Jun, 2025
When working with data in Pandas one common task is removing duplicate rows to ensure clean and accurate datasets. The drop_duplicates() method in Pandas is designed to make this process quick and easy. It allows us to remove duplicate rows from a DataFrame based on all columns or specific ones.
By default drop_duplicates() scans the entire DataFrame and retains the first occurrence of each row and removes any duplicates that follow. In this article, we will see how to use the drop_duplicates() method and its examples.
Let's start with a basic example to see how drop_duplicates() works.
Python
import pandas as pd
data = {
"Name": ["Alice", "Bob", "Alice", "David"],
"Age": [25, 30, 25, 40],
"City": ["NY", "LA", "NY", "Chicago"]
}
df = pd.DataFrame(data)
print("Original DataFrame:")
print(df)
df_cleaned = df.drop_duplicates()
print("\nModified DataFrame (no duplicates)")
print(df_cleaned)
Output:
Basic ExampleThis example shows how duplicate rows are removed while retaining the first occurrence using dataframe.drop_duplicates().
Syntax:
DataFrame.drop_duplicates(subset=None, keep='first', inplace=False)
Parameters:
1. subset: Specifies the columns to check for duplicates. If not provided all columns are considered.
2. keep: Finds which duplicate to keep:
- 'first' (default): Keeps the first occurrence, removes subsequent duplicates.
- 'last': Keeps the last occurrence and removes previous duplicates.
- False: Removes all occurrences of duplicates.
3. inplace: If True it modifies the original DataFrame directly. If False (default), returns a new DataFrame.
Return type: Method returns a new DataFrame with duplicates removed unless inplace=True.
Examples
Below are some examples of dataframe.drop_duplicates() method:
1. Dropping Duplicates Based on Specific Columns
We can target duplicates in specific columns using the subset parameter. This is useful when some columns are more relevant for identifying duplicates.
Python
import pandas as pd
df = pd.DataFrame({
'Name': ['Alice', 'Bob', 'Alice', 'David'],
'Age': [25, 30, 25, 40],
'City': ['NY', 'LA', 'SF', 'Chicago']
})
df_cleaned = df.drop_duplicates(subset=["Name"])
print(df_cleaned)
Output:
Dropping Duplicates Based on Specific ColumnsHere duplicates are removed only based on the Name column while Age and City are ignored for the purpose of removing duplicates.
2. Keeping the Last Occurrence of Duplicates
By default drop_duplicates() retains the first occurrence of duplicates. If we want to keep the last occurrence we can use keep='last'.
Python
import pandas as pd
df = pd.DataFrame({
'Name': ['Alice', 'Bob', 'Alice', 'David'],
'Age': [25, 30, 25, 40],
'City': ['NY', 'LA', 'NY', 'Chicago']
})
df_cleaned= df.drop_duplicates(keep='last')
print(df_cleaned)
Output:
Keeping the Last OccurrenceHere the last occurrence of Alice is kept and the first occurrence is removed.
3. Dropping All Duplicates
If we want to remove all rows that are duplicates i.e retain only completely unique rows amd here we can set keep=False.
Python
import pandas as pd
df = pd.DataFrame({
'Name': ['Alice', 'Bob', 'Alice', 'David'],
'Age': [25, 30, 25, 40],
'City': ['NY', 'LA', 'NY', 'Chicago']
})
df_cleaned = df.drop_duplicates(keep=False)
print(df_cleaned)
Output:
Dropping All DuplicatesWith keep=False both occurrences of Alice are removed leaving only the rows with unique values across all columns.
4. Modifying the Original DataFrame Directly
If we'll like to modify the DataFrame in place without creating a new DataFrame set inplace=True.
Python
import pandas as pd
df = pd.DataFrame({
'Name': ['Alice', 'Bob', 'Alice', 'David'],
'Age': [25, 30, 25, 40],
'City': ['NY', 'LA', 'NY', 'Chicago']
})
df.drop_duplicates(inplace=True)
print(df)
Output:
Modifying the Original DataFrameUsing inplace=True directly modifies the original DataFrame saving memory and avoiding the need to assign the result to a new variable.
5. Dropping Duplicates Based on Partially Identical Columns
Sometimes we might encounter situations where duplicates are not exact rows but have identical values in certain columns. For example after merging datasets we may want to drop rows that have the same values in a subset of columns.
Python
import pandas as pd
data = {
"Name": ["Alice", "Bob", "Alice", "David", "Bob"],
"Age": [25, 30, 25, 40, 30],
"City": ["NY", "LA", "NY", "Chicago", "LA"]
}
df = pd.DataFrame(data)
df_cleaned = df.drop_duplicates(subset=["Name", "City"])
print(df_cleaned)
Output:
Dropping Partially Identical ColumnsHere duplicates are removed based on the Name and City columns leaving only unique combinations of Name and City.
By mastering the drop_duplicates() method, we'll ensure that our datasets are clean and reliable which allow us to get accurate insights and make informed decisions.
How to Remove Duplicate Rows in Pandas Dataframe? | Pandas in Python
Similar Reads
Joining two Pandas DataFrames using merge() The merge() function is designed to merge two DataFrames based on one or more columns with matching values. The basic idea is to identify columns that contain common data between the DataFrames and use them to align rows. Let's understand the process of joining two pandas DataFrames using merge(), e
4 min read
Pandas DataFrame.astype()-Python DataFrame.astype() function in pandas cast a pandas object such as a DataFrame or Series to a specified data type. This is especially useful when you need to ensure that columns have the correct type, such as converting strings to integers or floats to strings. For example:Pythonimport pandas as pd
3 min read
Python | Pandas DataFrame.set_index() Pandas set_index() method is used to set one or more columns of a DataFrame as the index. This is useful when we need to modify or add new indices to our data as it enhances data retrieval, indexing and merging tasks. Setting the index is helpful for organizing the data more efficiently, especially
3 min read
Pandas DataFrame.reset_index() The reset_index() method in Pandas is used to manage and reset the index of a DataFrame. It is useful after performing operations that modify the index such as filtering, grouping or setting a custom index. By default reset_index() reverts to a clean, default integer-based index (0, 1, 2, ...) which
4 min read
Python | Pandas Dataframe.at[ ] Python is a great language for doing data analysis, primarily because of the fantastic ecosystem of data-centric Python packages. Pandas is one of those packages and makes importing and analyzing data much easier. Pandas at[] is used to return data in a dataframe at the passed location. The passed l
2 min read
Pandas DataFrame iterrows() Method iterrows() method in Pandas is a simple way to iterate over rows of a DataFrame. It returns an iterator that yields each row as a tuple containing the index and the row data (as a Pandas Series). This method is often used in scenarios where row-wise operations or transformations are required. Exampl
4 min read
Python | Pandas Series.iteritems() Pandas series is a One-dimensional ndarray with axis labels. The labels need not be unique but must be a hashable type. The object supports both integer- and label-based indexing and provides a host of methods for performing operations involving the index. Pandas Series.iteritems() function iterates
2 min read
Pandas.to_datetime()-Python pandas.to_datetime() converts argument(s) to datetime. This function is essential for working with date and time data, especially when parsing strings or timestamps into Python's datetime64 format used in Pandas. For Example:Pythonimport pandas as pd d = ['2025-06-21', '2025-06-22'] res = pd.to_date
3 min read
Python | pandas.to_numeric method Python is a great language for doing data analysis, primarily because of the fantastic ecosystem of data-centric python packages. Pandas is one of those packages and makes importing and analyzing data much easier. pandas.to_numeric() is one of the general functions in Pandas which is used to convert
2 min read
Pandas DataFrame.to_string-Python Pandas is a powerful Python library for data manipulation, with DataFrame as its key two-dimensional, labeled data structure. It allows easy formatting and readable display of data. DataFrame.to_string() function in Pandas is specifically designed to render a DataFrame into a console-friendly tabula
5 min read