If you are starting your journey in data cleaning with Python, the first step is understanding what Pandas in Python really is. In this section, we explain the basics of Pandas, DataFrame (df), and why it is the most popular tool for data manipulation and data cleansing in Python.
Answer: Pandas is an open-source Python library used for
data analysis, cleaning, and manipulation. It provides flexible data
structures like Series (1-D) and DataFrame (2-D) to work with
structured data (tables, CSVs, Excel, SQL queries).
Answer: Pandas is mainly used for:
Answer: In simple terms, Pandas are powerful Python tools that help turn raw data into clean, structured tables so analysts can focus on insights instead of messy datasets.
Answer: In Pandas, df is a common variable name used for
a DataFrame. A DataFrame is a 2D table of rows and columns, just like an
Excel sheet or SQL table.
import pandas as pd
df = pd.read_csv("data.csv")
print(df.head()) # displays first 5 rows
Answer: Pandas is the library, while df
(DataFrame) is the data structure created using Pandas to store and
manipulate tabular data.
Answer: Pandas is the industry standard for Python data cleaning because it offers powerful built-in functions for handling missing data, removing duplicates, data type conversion, and filtering outliersโall with simple, readable syntax that saves hours of manual coding.
Learn advanced techniques for handling missing values, duplicates, and outliers in the next section.
Learn data cleaning in python, pandas read csv, and df.head() in python using real datasets like Google Sheets and Superstore dataset inside Google Colab.
import pandas as pdimport pandas as pd
url = "https://docs.google.com/spreadsheets/d/1zZDBuiFCitgMU-D5XA2G7DVvoGmJ2G6KqXM9sOLpGz0/export?format=csv&gid=1140868527"
df = pd.read_csv(url)
df.head()import pandas as pd
url = "https://docs.google.com/spreadsheets/d/1zZDBuiFCitgMU-D5XA2G7DVvoGmJ2G6KqXM9sOLpGz0/export?format=csv&gid=1140868527"
df = pd.read_csv(url)
df.head()df.head(10)df.isnull().sum()
df.dropna()
df.describe()Learn how to filter data using pandas dataframe. Filtering is one of the most important steps in data cleaning in python to select specific rows, remove unwanted data, and analyze patterns.
df.head()# Filter rows where Sales > 500
df[df['Sales'] > 500]# Sales > 500 AND Profit > 50
df[(df['Sales'] > 500) & (df['Profit'] > 50)]df[['Sales', 'Profit']]df.query("Sales > 500 and Profit > 50")# High value customers
high_sales = df[df['Sales'] > 1000]
print(high_sales.head())Sorting is a key step in data cleaning in python. Using pandas sort_values and sort_index(), you can organize your dataset to find top values, trends, and insights.
df.head()# Sort by Sales (low โ high)
df.sort_values(by='Sales')# Sort by Sales (high โ low)
df.sort_values(by='Sales', ascending=False)# Sort by Category then Sales
df.sort_values(by=['Category','Sales'], ascending=[True, False])df.sort_index()# Top 5 highest Sales
top_sales = df.sort_values(by='Sales', ascending=False).head(5)
print(top_sales)Missing data is common in real datasets. In data cleaning in python, we use fillna() and dropna() to handle missing values effectively using Pandas DataFrame.
df.isnull().sum()# Fill all missing values with 0
df.fillna(0, inplace=True)# Fill Sales column with mean
df['Sales'].fillna(df['Sales'].mean(), inplace=True)df.dropna()df.dropna(axis=1)# Check missing values
print(df.isnull().sum())
# Fill missing values
df.fillna(0, inplace=True)
# Verify
print(df.isnull().sum())Before cleaning data, you must detect missing values. In data cleaning in python, we use df.isnull(), sum(), and any() to find missing data in rows and columns.
df.head()df.isnull()df.isnull().sum()df.columns[df.isnull().any()]df[df.isnull().any(axis=1)]# Count missing values
print(df.isnull().sum())
# Show rows with missing values
print(df[df.isnull().any(axis=1)])The pandas describe method is used to summarize numerical data. It helps in data cleaning in python pandas by showing mean, standard deviation, min, max, and quartiles of your dataset.
import pandas as pd
url = "https://raw.githubusercontent.com/plotly/datasets/master/superstore.csv"
df = pd.read_csv(url)
df.head()df.describe()df[['Sales','Profit']].describe()df.describe(include='all')import pandas as pd
url = "https://docs.google.com/spreadsheets/d/1zZDBuiFCitgMU-D5XA2G7DVvoGmJ2G6KqXM9sOLpGz0/export?format=csv&gid=1140868527"
df = pd.read_csv(url)
# Full summary
print(df.describe())
# Specific columns
print(df[['Sales','Profit']].describe())Duplicate data is a common problem in real datasets. In data cleaning in python, we use duplicated() and drop_duplicates() to identify and remove duplicate rows from a Pandas DataFrame.
df.duplicated().sum()df[df.duplicated()]df.drop_duplicates(inplace=True)df.drop_duplicates(subset='Customer Name', inplace=True)df.drop_duplicates(keep='last', inplace=True)# Count duplicates
print(df.duplicated().sum())
# Remove duplicates
df.drop_duplicates(inplace=True)
# Verify
print(df.duplicated().sum())In real datasets, columns often have incorrect data types. In data cleaning in python, we use astype(), to_datetime(), and to_numeric() to fix data types for proper analysis.
df.dtypes# Convert column safely
df['Age'] = df['Age'].astype('Int64')df['Sales'] = pd.to_numeric(df['Sales'], errors='coerce')df['Order Date'] = pd.to_datetime(df['Order Date'], errors='coerce')cols = ['Sales', 'Profit']
df[cols] = df[cols].apply(pd.to_numeric, errors='coerce')# Check types
print(df.dtypes)
# Fix numeric column
df['Sales'] = pd.to_numeric(df['Sales'], errors='coerce')
# Fix date column
df['Order Date'] = pd.to_datetime(df['Order Date'], errors='coerce')
# Verify changes
print(df.dtypes)Outliers are extreme values that can distort analysis. In data cleaning in python, we use the IQR (Interquartile Range) method to detect and remove outliers.
df['Sales'].describe()Q1 = df['Sales'].quantile(0.25)
Q3 = df['Sales'].quantile(0.75)IQR = Q3 - Q1lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQRoutliers = df[(df['Sales'] < lower_bound) | (df['Sales'] > upper_bound)]df = df[(df['Sales'] >= lower_bound) & (df['Sales'] <= upper_bound)]# Calculate quartiles
Q1 = df['Sales'].quantile(0.25)
Q3 = df['Sales'].quantile(0.75)
# Calculate IQR
IQR = Q3 - Q1
# Define bounds
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR
# Filter data
df_clean = df[(df['Sales'] >= lower) & (df['Sales'] <= upper)]
print(df_clean.head())Real datasets often contain messy text like extra spaces, inconsistent casing, or wrong spellings. In data cleaning in python, we use string methods (.str) to clean and standardize text data.
df['City'] = df['City'].str.lower()df['City'] = df['City'].str.strip()df['City'] = df['City'].replace({
'delhi ': 'delhi',
'new delhi': 'delhi'
})df['Name'] = df['Name'].str.replace('[^a-zA-Z ]', '', regex=True)df['City'] = df['City'].str.title()# Clean City column
df['City'] = df['City'].str.lower().str.strip()
# Standardize values
df['City'] = df['City'].replace({
'new delhi': 'delhi'
})
# Format nicely
df['City'] = df['City'].str.title()
print(df['City'].unique())Feature engineering is the process of creating new columns from existing data. It helps transform raw data into meaningful insights for analysis and machine learning.
# Create Profit Ratio
df['Profit_Ratio'] = df['Profit'] / df['Sales']df['Year'] = df['Order Date'].dt.yeardf['Month'] = df['Order Date'].dt.month# High / Low Sales
df['Sales_Category'] = df['Sales'].apply(
lambda x: 'High' if x > 500 else 'Low'
)# Combine City and State
df['Location'] = df['City'] + ", " + df['State']# Create new features
df['Profit_Ratio'] = df['Profit'] / df['Sales']
df['Year'] = df['Order Date'].dt.year
# Categorize data
df['Sales_Category'] = df['Sales'].apply(
lambda x: 'High' if x > 500 else 'Low'
)
print(df.head())GroupBy is used to analyze data by categories. In data analysis using pandas, we use groupby() to summarize data like total sales, average profit, and customer insights.
df.groupby('Category')['Sales'].sum()df.groupby('Category')['Profit'].mean()df.groupby('Category').agg({
'Sales': 'sum',
'Profit': 'mean'
})df.groupby(['Category','Region'])['Sales'].sum()df.groupby('Category')['Sales'].sum().reset_index()# Total sales per category
sales_by_cat = df.groupby('Category')['Sales'].sum()
# Average profit
profit_avg = df.groupby('Category')['Profit'].mean()
print(sales_by_cat)
print(profit_avg)Now it’s time to apply everything you learned. This project covers data cleaning, feature engineering, and data analysis using a real dataset in Google Colab.
Get answers to common questions about data cleaning in python and pandas.