Python Pandas – Dropping Missing Data

Handling Missing Data Using dropna() in Pandas – Vista Academy

Missing data is a common issue when working with real-world datasets. The Python Pandas library provides an easy way for removing rows or columns that contain missing values (NaN or NaT) from a dataset using the dropna() method.

The dropna() method in Pandas is a useful tool to handle missing data by dropping rows or columns based on your specific requirements. In this tutorial, we will learn how to use dropna() to clean your dataset by dropping missing data based on various conditions.

The dropna() Method

The Pandas dropna() method allows you to remove missing values from a Pandas data structures such as, Series and DataFrame objects. It offers several options to customize how you drop rows or columns based on the presence of NaN values. This method returns a new Pandas object with missing data dropped or it returns None if inplace parameter is set to True.

Syntax

Following is the syntax −

DataFrame.dropna(*, axis=0, how=<no_default>, thresh=<no_default>, subset=None, inplace=False, ignore_index=False)

Where:

• axis: 0 or ‘index’ (default) to drop rows; 1 or ‘columns’ to drop columns.
• how: By default it is set to ‘any’, which drops that row or column if any missing values are present. If set to ‘all’, then it drops that row or column if all the missing values.
• thresh: Require a minimum number of non-NA values to retain the row or column.
• subset: List of specific columns (if dropping rows) or rows (if dropping columns) to consider.
• inplace: Modify the DataFrame in place (default is False).
• ignore_index: Reset the index of the result (default is False).

Let’s explore how the dropna() method drops the missing data based on various conditions.

Drop Rows with Any Missing Values

By default, the dropna() method removes rows where any missing values are present.

Example

The following example uses the dropna() method to drop the rows that have any missing values.

import pandas as pd
import numpy as np

dataset = {"Student_name": ["Ajay", "Krishna", "Deepak", "Swati"], "Roll_number": [23, 45, np.nan, 18],
           "Major_Subject": ["Maths", "Physics", "Arts", "Political science"], "Marks": [57, np.nan, 98, np.nan]}

df = pd.DataFrame(dataset, index= [1, 2, 3, 4])
print("Original DataFrame:")
print(df)

# Drop the rows that have any missing values
df_cleaned = df.dropna()
print('\nResultant DataFrame after removing row:\n',df_cleaned)
  

Following is the output of the above code −

Original DataFrame:
Student name	Roll number	Major Subject	Marks
1	Ajay	    23.0	      Maths	            57.0
2	Krishna	45.0	      Physics	          NaN
3	Deepak	  NaN	        Arts	              98.0
4	Swati	    18.0	      Political science	NaN

Resultant DataFrame after removing row:
Student name	Roll number	Major Subject	Marks
1	Ajay	    23.0	      Maths	            57.0
  

Drop Rows Where All Values Are Missing

To drop rows where all values are missing, set how='all' parameter in the dropna() method.

Example

The following example demonstrates how to drop the rows where all values are missing in a DataFrame.

import pandas as pd
import numpy as np

dataset = {"Student name": ["Ajay", np.nan, "Deepak", "Swati"], 
"Roll number": [23, np.nan, np.nan, 18],
"Major Subject": ["Maths", np.nan, "Arts", "Political science"], 
"Marks": [57, np.nan, 98, np.nan]}

df = pd.DataFrame(dataset, index= [1, 2, 3, 4])
print("Original DataFrame:")
print(df)

# Drop rows where all values are missing
reslut = df.dropna(how='all')
print('\nResultant DataFrame after removing row:\n',reslut)
  

Following is the output of the above code −

Original DataFrame:
Student name	Roll number	Major Subject	Marks
1	Ajay	    23.0	      Maths	            57.0
2	NaN	      NaN	        NaN	              NaN
3	Deepak	  NaN	        Arts	              98.0
4	Swati	    18.0	      Political science	NaN

Resultant DataFrame after removing row:
Student name	Roll number	Major Subject	Marks
1	Ajay	    23.0	      Maths	            57.0
3	Deepak	  NaN	        Arts	              98.0
4	Swati	    18.0	      Political science	NaN
  

Keep Rows with a Minimum Number of Missing Values

The Pandas dropna() method provides the thresh parameter to specify a minimum threshold of non-missing values for keeping rows with a minimum number of Non-Na values.

Example

This example demonstrates how to keep the rows with a minimum number of missing values.

import pandas as pd
import numpy as np

dataset = {"Student name": ["Ajay", "Krishna", "Deepak", "Swati"], 
"Roll number": [23, np.nan, np.nan, 18],
"Major Subject": ["Maths", np.nan, "Arts", "Political science"], 
"Marks": [57, np.nan, 98, np.nan]}

df = pd.DataFrame(dataset, index= [1, 2, 3, 4])
print("Original DataFrame:")
print(df)

# Drop the rows with a threshold 
result = df.dropna(thresh=2)
print('\nResultant DataFrame after removing row:\n',result)
  

Following is the output of the above code −

Original DataFrame:
Student name	Roll number	Major Subject	Marks
1	Ajay	    23.0	      Maths	            57.0
2	Krishna	NaN	        NaN	              NaN
3	Deepak	  NaN	        Arts	              98.0
4	Swati	    18.0	      Political science	NaN

Resultant DataFrame after removing row:
Student name	Roll number	Major Subject	Marks
1	Ajay	    23.0	      Maths	            57.0
3	Deepak	  NaN	        Arts	              98.0
4	Swati	    18.0	      Political science	NaN
  

Drop Columns with Any Missing Values

To drop columns that contain any missing values, use the axis parameter of the dropna() method to select the columns.

Example

This example shows how the dropna() method removes the entire column where any of the values are missing.

import pandas as pd
import numpy as np

dataset = {"Student_name": ["Ajay", "Krishna", "Deepak", "Swati"], 
"Roll_number": [23, 45, np.nan, 18],
"Major_Subject": ["Maths", "Physics", "Arts", "Political science"], 
"Marks": [57, np.nan, 98, np.nan]}

df = pd.DataFrame(dataset, index= [1, 2, 3, 4])
print("Original DataFrame:")
print(df)

# Drop column with any missing values
result = df.dropna(axis='columns')
print('\nResultant DataFrame after removing columns:\n',result)
  

Following is the output of the above code −

Original DataFrame:
Student name	Roll number	Major Subject	Marks
1	Ajay	    23.0	      Maths	            57.0
2	Krishna	45.0	      Physics	          NaN
3	Deepak	  NaN	        Arts	              98.0
4	Swati	    18.0	      Political science	NaN

Resultant DataFrame after removing columns:
Student name	Major Subject
1	Ajay	    Maths
2	Krishna	Physics
3	Deepak	  Arts
4	Swati	    Political science
  

Drop Rows Based on Missing Data in Specific Columns

You can use the subset parameter of the dropna() method to focus only on those particular columns while dropping rows where data is missing.

Example

This example shows how to remove the rows based on missing data present in the specific column using the subset parameter of the dropna() method.

import pandas as pd
import numpy as np

dataset = {"Student_name": ["Ajay", "Krishna", "Deepak", "Swati"], 
"Roll_number": [23, 45, np.nan, 18],
"Major_Subject": ["Maths", "Physics", np.nan, "Political science"], 
"Marks": [57, np.nan, 98, np.nan]}

df = pd.DataFrame(dataset, index= [1, 2, 3, 4])
print("Original DataFrame:")
print(df)

# Drop Rows Based on Missing Data in Specific Columns
result = df.dropna(subset=['Roll_number', 'Major_Subject'])
print('\nResultant DataFrame after removing rows:\n',result)
  

Following is the output of the above code −

Original DataFrame:
Student name	Roll number	Major Subject	Marks
1	Ajay	    23.0	      Maths	            57.0
2	Krishna	45.0	      Physics	          NaN
3	Deepak	  NaN	        NaN	              98.0
4	Swati	    18.0	      Political science	NaN

Resultant DataFrame after removing rows:
Student name	Roll number	Major Subject	Marks
1	Ajay	    23.0	      Maths	            57.0
2	Krishna	45.0	      Physics	          NaN
4	Swati	    18.0	      Political science	NaN