Missing data is one of the most common problems faced by Data Analysts. Real-world datasets rarely arrive perfectly complete. Customer records may have no phone number, a sales transaction may have a missing discount, an employee record may not contain a joining date, a survey respondent may skip a question, or a sensor may fail to record a measurement. Pandas provides a complete set of tools for identifying, understanding, cleaning, replacing, and analyzing missing values.
Handling missing data is not simply about deleting empty cells. The correct approach depends on why the information is missing, how much information is missing, what the column represents, and what the business analysis requires. Replacing every missing value with zero can create incorrect statistics. Dropping every row containing a missing value can unnecessarily remove useful observations. A reliable analyst therefore treats missing data as an analytical problem that requires investigation before transformation.
In Pandas, missing values are commonly represented by values such as NaN, None, and NaT. Pandas provides methods including isna(), notna(), dropna(), fillna(), ffill(), bfill(), and interpolation methods for working with incomplete data.
We will use practical datasets throughout this lesson.
import pandas as pd
import numpy as np
customers = pd.DataFrame({
"Customer_ID": [101, 102, 103, 104, 105, 106],
"Name": ["Aman", "Priya", "Rahul", "Neha", "Karan", "Sonia"],
"City": [
"Dehradun",
"Delhi",
None,
"Haridwar",
"Dehradun",
None
],
"Age": [24, 31, np.nan, 35, 42, 28],
"Purchase": [45000, 85000, 125000, np.nan, 150000, 72000]
})
print(customers)Inspecting the DataFrame is the first step:
print(customers.head())
print(customers.info())The info() method provides useful information about non-null values and data types. It is one of the quickest ways to notice that a column contains missing observations.
To count missing values in every column:
print(customers.isna().sum())The result gives the number of missing values for each column.
You can also calculate the percentage of missing values:
missing_percentage = (
customers.isna().mean() * 100
)
print(missing_percentage)This is often more useful than the raw count because columns can have very different numbers of records.
For example, five missing values in a dataset of ten records represents a very different situation from five missing values in a dataset of one million records.
To check whether the entire DataFrame contains any missing value:
print(customers.isna().any().any())To check which columns contain at least one missing value:
columns_with_missing = customers.columns[
customers.isna().any()
]
print(columns_with_missing)To find rows containing at least one missing value:
rows_with_missing = customers.loc[
customers.isna().any(axis=1)
]
print(rows_with_missing)The axis=1 parameter means that the check is performed across columns for each row.
To find rows where every value is missing:
completely_empty = customers.loc[
customers.isna().all(axis=1)
]
print(completely_empty)Although completely empty rows are uncommon in a carefully structured dataset, this check can be useful after importing spreadsheets or manually collected data.
The opposite of isna() is notna():
complete_city = customers.loc[
customers["City"].notna()
]
print(complete_city)This returns only rows where City contains a non-missing value.
To find records where Purchase is missing:
missing_purchase = customers.loc[
customers["Purchase"].isna()
]
print(missing_purchase)To find records where Purchase is available:
available_purchase = customers.loc[
customers["Purchase"].notna()
]
print(available_purchase)These simple checks are essential before calculating averages, totals, rankings, or machine-learning features.
One common mistake is assuming that a missing value means zero. It usually does not.
Consider a sales column:
sales = pd.DataFrame({
"Customer": ["Aman", "Priya", "Rahul", "Neha"],
"Sales": [50000, np.nan, 75000, 90000]
})A missing Sales value could mean that the sales amount was not recorded. It does not necessarily mean that the customer generated zero sales.
If you replace it with zero without understanding the source, the total and average may be distorted.
This is why missing-data treatment should be based on context.
Pandas provides dropna() for removing missing observations.
To remove rows containing at least one missing value:
cleaned = customers.dropna()
print(cleaned)This is simple, but it can remove many records when even one column is missing.
For example, suppose a customer has a missing City but a perfectly valid Purchase value. Removing the entire customer record may not be appropriate if the analysis concerns revenue rather than location.
You can instead drop rows only when a particular column is missing:
cleaned = customers.dropna(
subset=["Purchase"]
)
print(cleaned)This keeps customers whose Purchase is available while removing records where the key analytical metric is missing.
You can specify multiple required columns:
cleaned = customers.dropna(
subset=["City", "Purchase"]
)
print(cleaned)This removes rows where either City or Purchase is missing.
The how parameter controls whether rows are removed when any or all values are missing.
cleaned = customers.dropna(
how="all"
)
print(cleaned)This removes only rows in which every value is missing.
To remove a row if at least one value is missing:
cleaned = customers.dropna(
how="any"
)This is the default behavior.
You can also work with columns instead of rows by using axis=1:
cleaned = customers.dropna(
axis=1
)
print(cleaned)This removes columns containing missing values. It should be used carefully because a column may contain useful information even if some observations are missing.
For example, removing an important customer-income column simply because 5 percent of values are missing may be a poor analytical decision.
The thresh parameter can retain rows that contain a minimum number of non-missing values:
cleaned = customers.dropna(
thresh=4
)
print(cleaned)This keeps rows containing at least four non-null values.
This can be useful when a dataset has many optional fields but the analyst wants to retain records containing enough information for further processing.
Dropping missing values is only one strategy. Another major strategy is imputation, meaning replacing missing values with an appropriate value.
The Pandas method for this is fillna().
For example:
customers["Age"] = customers["Age"].fillna(
customers["Age"].median()
)The missing age is replaced with the median age of the available observations.
For numerical variables, common replacement strategies include mean, median, or a domain-specific value.
Mean imputation:
mean_age = customers["Age"].mean()
customers["Age"] = customers["Age"].fillna(
mean_age
)Median imputation:
median_age = customers["Age"].median()
customers["Age"] = customers["Age"].fillna(
median_age
)Median is often preferred when the variable contains extreme values because it is less influenced by unusually large or small observations.
Consider salaries:
salaries = pd.Series([
30000,
35000,
40000,
45000,
500000,
np.nan
])
print("Mean:", salaries.mean())
print("Median:", salaries.median())The very large salary affects the mean much more than the median. The appropriate imputation method therefore depends on the distribution and business context.
For categorical data, a common strategy is to use the most frequent category:
customers["City"] = customers["City"].fillna(
customers["City"].mode()[0]
)The mode() method returns the most frequently occurring value.
However, mode imputation can create an artificial concentration in the most common category. It should therefore be used only when it makes sense for the analytical objective.
Another option is to use an explicit label:
customers["City"] = customers["City"].fillna(
"Unknown"
)This approach preserves the information that the value was missing rather than pretending that it belonged to the most common city.
Labels such as “Unknown”, “Not Provided”, or “Not Available” can be useful when missingness itself has meaning.
For example, a customer may have deliberately chosen not to provide a phone number. Replacing the missing value with another person’s phone number would be inappropriate, while an “Unknown” category preserves the distinction.
You can fill a column with a fixed numerical value:
df["Discount"] = df["Discount"].fillna(0)This is appropriate only if a missing discount genuinely means that no discount was applied. If the source system simply failed to record the discount, zero would be misleading.
This illustrates an important rule: a replacement value should represent a justified business interpretation, not merely make the dataset look complete.
Missing values can also be filled using dictionaries:
df = df.fillna({
"City": "Unknown",
"Discount": 0,
"Age": 30
})This allows different columns to receive different replacement values.
For repeated analytical workflows, it can be useful to create a clear cleaning pipeline:
cleaned = customers.copy()
cleaned["Age"] = cleaned["Age"].fillna(
cleaned["Age"].median()
)
cleaned["City"] = cleaned["City"].fillna(
"Unknown"
)
print(cleaned)Keeping a separate cleaned DataFrame makes it easier to compare the source data with the transformed data.
Forward filling is another technique. It uses the previous available value to fill a missing observation.
series = pd.Series([
100,
110,
np.nan,
np.nan,
140
])
filled = series.ffill()
print(filled)The missing values between 110 and 140 are filled with 110.
Forward filling can be useful in time-series data when the most recent known state should remain valid until a new value appears.
For example, suppose a machine status is recorded at irregular intervals. If the status remains valid until the next recorded status, forward filling may be appropriate.
Backward filling works in the opposite direction:
filled = series.bfill()
print(filled)Missing values are filled using the next available observation.
Consider:
series = pd.Series([
np.nan,
np.nan,
120,
130
])
print(series.bfill())The missing values at the beginning are filled using the first available value.
Forward and backward filling should not be used automatically. The analyst should understand whether carrying information forward or backward is logically valid.
For example, using a future product price to fill an earlier missing price could introduce information that was not available at that time. This can create bias in time-dependent analysis.
Interpolation is another approach, particularly for numerical time-series data.
temperature = pd.Series([
20,
21,
np.nan,
25,
26
])
interpolated = temperature.interpolate()
print(interpolated)Pandas estimates the missing value based on surrounding observations.
Interpolation can be useful for measurements that are expected to change gradually, such as temperature, sensor readings, or certain time-series metrics.
It should not be used for every variable. A categorical variable such as City cannot be meaningfully interpolated.
Time-series data provides an important context for missing-data handling.
weather = pd.DataFrame({
"Date": pd.to_datetime([
"2026-01-01",
"2026-01-02",
"2026-01-03",
"2026-01-04",
"2026-01-05"
]),
"Temperature": [
15,
np.nan,
17,
np.nan,
20
]
})
weather = weather.sort_values(
by="Date"
)
weather["Temperature_Interpolated"] = (
weather["Temperature"]
.interpolate()
)
print(weather)Sorting the dates before interpolation is an important step because the surrounding observations should represent the correct chronological sequence.
Missing-data analysis should also consider whether missingness is random or systematic.
Suppose a customer dataset has missing income values. If missing income occurs randomly, one type of treatment may be reasonable. If high-income customers are more likely to refuse to provide income information, simply replacing missing values with the average could distort the analysis.
Therefore, analysts should investigate patterns of missingness.
For example:
customers.isna().sum()
customers.groupby(
customers["Age"].isna()
)["Purchase"].mean()This can help compare purchase behavior between records with and without missing Age values.
Similarly, missingness can be compared by category:
customers.groupby("City")["Purchase"].mean()When City itself contains missing values, you may first create a missingness indicator:
customers["City_Missing"] = (
customers["City"].isna()
)This converts missingness into an explicit Boolean feature.
A missingness indicator can be useful when the fact that a value is missing may itself contain information.
For numerical variables, you can create a similar indicator:
customers["Age_Missing"] = (
customers["Age"].isna()
)Then impute Age separately if necessary.
This produces two pieces of information: an estimated or replaced Age value and an indicator showing that the original value was missing.
For example:
customers["Age_Missing"] = (
customers["Age"].isna()
)
customers["Age"] = customers["Age"].fillna(
customers["Age"].median()
)This approach can be useful in predictive modeling and advanced analytics when missingness may carry predictive information.
Missing values also affect descriptive statistics.
sales = pd.Series([
50000,
60000,
np.nan,
80000
])
print(sales.mean())
print(sales.sum())
print(sales.count())Pandas statistical functions generally skip missing values by default. This behavior is useful, but analysts should understand it before interpreting the results.
For example, count() counts non-missing observations, not the total number of rows.
print("Total rows:", len(sales))
print("Non-missing:", sales.count())This distinction is important when calculating averages or reporting data completeness.
You can explicitly control missing-value behavior in many operations using parameters such as skipna:
print(sales.sum(skipna=True))
print(sales.sum(skipna=False))With skipna=False, the presence of a missing value can cause the result to become missing depending on the operation.
This is useful when you want incomplete data to be visible rather than silently ignored.
Missing data can also affect sorting and ranking.
employees = pd.DataFrame({
"Name": ["Aman", "Priya", "Rahul", "Neha"],
"Performance": [85, np.nan, 92, 78]
})
sorted_employees = employees.sort_values(
by="Performance",
ascending=False,
na_position="last"
)
print(sorted_employees)For ranking:
employees["Rank"] = (
employees["Performance"]
.rank(
ascending=False,
method="dense"
)
)
print(employees)The missing performance value should not automatically be interpreted as the lowest performance. Missing and low are different concepts.
This is an important distinction in employee, student, customer, and operational reports.
Data quality checks can be built around missingness.
missing_report = pd.DataFrame({
"Missing_Count": customers.isna().sum(),
"Missing_Percentage": (
customers.isna().mean() * 100
)
})
print(missing_report)This creates a compact data-quality report.
You can sort it by missing percentage:
missing_report = missing_report.sort_values(
by="Missing_Percentage",
ascending=False
)
print(missing_report)This immediately identifies columns with the greatest missing-data problem.
For larger projects, such a report can be created before the main analysis begins.
For example:
def missing_value_report(df):
report = pd.DataFrame({
"Missing_Count": df.isna().sum(),
"Missing_Percentage": df.isna().mean() * 100,
"Data_Type": df.dtypes.astype(str)
})
return report.sort_values(
by="Missing_Percentage",
ascending=False
)
report = missing_value_report(customers)
print(report)This reusable function provides a simple first-level data-quality assessment.
One common mistake is dropping missing values before measuring the amount of missing data. If you immediately use dropna(), you lose information about the original completeness of the dataset.
A better workflow is:
Inspect
↓
Measure Missingness
↓
Understand the Cause
↓
Choose a Strategy
↓
Transform
↓
Validate
↓
AnalyzeKeep the original DataFrame available whenever possible.
For example:
raw = customers.copy()
cleaned = customers.copy()
cleaned["Age"] = cleaned["Age"].fillna(
cleaned["Age"].median()
)This allows you to compare the original and cleaned datasets.
You can check how many values were changed:
before = customers["Age"].isna().sum()
cleaned["Age"] = cleaned["Age"].fillna(
cleaned["Age"].median()
)
after = cleaned["Age"].isna().sum()
print("Before:", before)
print("After:", after)This provides a simple validation of the cleaning operation.
For categorical replacement:
before = customers["City"].isna().sum()
cleaned["City"] = cleaned["City"].fillna(
"Unknown"
)
after = cleaned["City"].isna().sum()
print("Before:", before)
print("After:", after)It is also useful to compare summary statistics before and after imputation.
print(customers["Age"].describe())
print(cleaned["Age"].describe())If imputation changes the distribution significantly, investigate whether the chosen strategy is appropriate.
For example, replacing a large proportion of missing values with one constant can create an artificial spike in the distribution.
Suppose 60 percent of a salary column is missing. Filling all missing values with the median does not magically create reliable salary information. It creates many identical estimated values. In such a situation, the analyst should investigate the source and determine whether the column should be used for the intended analysis at all.
Missing-data percentage can therefore become a decision criterion.
For example:
missing_rate = (
df["Salary"].isna().mean()
)
print(missing_rate)If the missing rate is extremely high, dropping the column or collecting better data may be more appropriate than aggressive imputation.
The exact threshold should not be treated as a universal rule. It depends on the business context, data source, variable importance, and analytical objective.
Missing data should also be considered before machine-learning model training. Many algorithms cannot directly process missing values, while some models and preprocessing pipelines can handle them differently.
For a Data Analyst, the important lesson is not to blindly fill every missing value. Instead, determine the appropriate preprocessing strategy for the intended analytical method.
For example, a simple reporting analysis may tolerate missing City values while calculating revenue from available transactions. A machine-learning model predicting customer value may require a carefully designed imputation and missingness strategy.
The same dataset can therefore require different missing-data treatment for different use cases.
Text columns can contain missing values alongside empty strings:
names = pd.DataFrame({
"Name": [
"Aman",
"",
None,
"Priya",
" "
]
})
print(names)An empty string is not automatically the same as a Pandas missing value.
You can identify empty or whitespace-only strings separately:
names["Name_Clean"] = (
names["Name"]
.replace(r"^\s*$", np.nan, regex=True)
)
print(names)Now the blank and whitespace-only values can be treated as missing.
This is a common data-cleaning issue when information comes from spreadsheets, forms, or manually entered systems.
Similarly, values such as “N/A”, “NA”, “Unknown”, and “-” may appear in source files as text rather than true missing values.
df = pd.DataFrame({
"Income": [
"50000",
"N/A",
"75000",
"-"
]
})
df["Income"] = df["Income"].replace(
["N/A", "-"],
np.nan
)
df["Income"] = pd.to_numeric(
df["Income"]
)This converts placeholder text into actual missing values and then converts the column into a numerical type.
After cleaning, check:
print(df)
print(df.dtypes)
print(df.isna().sum())This illustrates why missing-data handling is closely connected with data-type cleaning.
Dates can also contain missing values:
orders = pd.DataFrame({
"Order_Date": [
"2026-01-10",
None,
"2026-03-15"
]
})
orders["Order_Date"] = pd.to_datetime(
orders["Order_Date"],
errors="coerce"
)
print(orders)Invalid or missing date values can become NaT, which is Pandas’ representation for missing datetime values.
Check them using:
print(
orders["Order_Date"].isna()
)For date columns, do not fill missing dates with arbitrary dates unless the business logic supports it. An unknown order date is not necessarily January 1 or today’s date.
Sometimes the best strategy is to retain the missing value and create a separate indicator:
orders["Order_Date_Missing"] = (
orders["Order_Date"].isna()
)This preserves the fact that the date is unavailable.
For time-series analysis, missing values can be handled using forward fill, backward fill, interpolation, rolling calculations, or domain-specific methods.
temperature = pd.Series(
[20, 21, np.nan, 24, 25]
)
temperature_ffill = temperature.ffill()
temperature_bfill = temperature.bfill()
temperature_interpolated = (
temperature.interpolate()
)
print(temperature_ffill)
print(temperature_bfill)
print(temperature_interpolated)These three methods produce different results because they make different assumptions about the missing observation.
Forward fill assumes the previous value remains applicable. Backward fill assumes the next value can represent the missing period. Interpolation estimates a value between known observations.
The analyst should select the method based on how the data was generated.
For example, a daily account status may reasonably be forward-filled until the next status update, while a temperature measurement may be better interpolated if the change is expected to be gradual.
Missing-data treatment should also be documented in project reports.
# Age:
# Missing values replaced with median.
# City:
# Missing values labelled Unknown.
# Purchase:
# Missing values retained because absence
# of recorded purchase is not equivalent to zero.Documentation improves reproducibility and allows other analysts to understand the assumptions behind the cleaned dataset.
For an E-E-A-T-oriented analytical workflow, transparency is especially important. A reader should be able to understand how incomplete data was handled and whether that decision could affect the reported result.
A trustworthy Data Analytics report should not hide data-quality limitations.
Suppose a tourism dataset contains district-wise visitor counts and one district is missing several months. Reporting an annual total without discussing the missing months could create a misleading comparison with complete districts.
A simple completeness check can help:
monthly_counts = (
tourism
.groupby("District")["Month"]
.nunique()
)
print(monthly_counts)You can identify districts with fewer expected observations:
incomplete = monthly_counts.loc[
monthly_counts < 12
]
print(incomplete)This demonstrates a broader principle: missingness should be evaluated relative to the expected structure of the dataset.
For example, if twelve monthly observations are expected for every district, a district with eight records has four missing observations. The correct treatment depends on why those records are absent.
Another useful check is completeness by category:
completeness = (
tourism
.groupby("District")
.apply(
lambda x: x.notna().mean()
)
)For large projects, completeness reports can be incorporated into automated data-quality checks.
Missing values can also affect joins and merges. If a key column contains missing values, records may fail to match correctly with another DataFrame.
customers = pd.DataFrame({
"Customer_ID": [101, 102, np.nan, 104],
"Name": ["Aman", "Priya", "Rahul", "Neha"]
})
orders = pd.DataFrame({
"Customer_ID": [101, 102, 103, 104],
"Sales": [50000, 70000, 80000, 90000]
})
merged = customers.merge(
orders,
on="Customer_ID",
how="left"
)
print(merged)The missing Customer_ID cannot provide a reliable matching key. This is why key-column completeness is often more important than completeness in optional descriptive fields.
Before a merge, check:
print(
customers["Customer_ID"].isna().sum()
)In a production workflow, missing identifiers should usually be investigated rather than replaced with arbitrary values.
Missing values also affect duplicate analysis. If a key field is missing, duplicate detection may need to use additional columns.
duplicates = customers.duplicated(
subset=["Name"],
keep=False
)
print(customers.loc[duplicates])This is another reason why data cleaning should be planned around the analytical structure of the dataset.
For practical learning, create a dataset with at least ten rows and deliberately introduce missing values into numerical, categorical, and date columns. Then complete this workflow:
info().dropna() selectively.A useful interview exercise is to explain why the following code may be dangerous:
df = df.fillna(0)The problem is that it treats every missing value as zero regardless of the column’s meaning. Zero sales, unknown sales, zero age, unknown age, and an unknown city are completely different concepts.
A better approach is column-specific treatment:
df["Sales"] = df["Sales"].fillna(
df["Sales"].median()
)
df["City"] = df["City"].fillna(
"Unknown"
)Even this code should only be used after checking whether median imputation and an Unknown category make sense for the business context.
Another common interview question is the difference between isna() and notna().
isna() identifies missing values, while notna() identifies values that are present.
df.loc[df["Age"].isna()]
df.loc[df["Age"].notna()]A common question about dropna() is how to remove rows only when a particular field is missing:
df.dropna(
subset=["Customer_ID"]
)Another common question is how to replace missing numerical values with a median:
df["Salary"] = df["Salary"].fillna(
df["Salary"].median()
)For the most frequent categorical value:
df["City"] = df["City"].fillna(
df["City"].mode()[0]
)For an explicit category:
df["City"] = df["City"].fillna(
"Unknown"
)For forward filling:
df["Value"] = df["Value"].ffill()For backward filling:
df["Value"] = df["Value"].bfill()For interpolation:
df["Value"] = df["Value"].interpolate()The most important interview concept is not memorizing these methods but knowing when each method is appropriate.
For example, if a sales amount is missing because the source system failed, replacing it with zero may understate revenue. If the business definition says a blank discount means no discount, zero may be appropriate. If a temperature reading is missing between two valid readings, interpolation may be reasonable. If a customer city was not supplied, “Unknown” may preserve the meaning better than using the most common city.
Context determines the correct approach.
A professional data-cleaning pipeline may therefore contain separate rules for different data types:
Numerical
↓
Investigate distribution
↓
Mean / Median / Model / Domain Rule
Categorical
↓
Mode / Unknown / Separate Category
Time Series
↓
Forward Fill / Backward Fill / Interpolation
Identifiers
↓
Investigate Missingness
↓
Avoid Arbitrary ReplacementThis structure is useful because it prevents a one-size-fits-all approach.
For a final project exercise, take a customer dataset and create a missing-data audit containing:
Column
Data Type
Total Rows
Missing Count
Missing Percentage
Recommended Treatment
ReasonFor example:
missing_audit = pd.DataFrame({
"Data_Type": df.dtypes.astype(str),
"Missing_Count": df.isna().sum(),
"Missing_Percentage": (
df.isna().mean() * 100
)
})
missing_audit["Total_Rows"] = len(df)
print(missing_audit)You can then add a manual Recommended Treatment column based on the business context.
This type of audit is useful in professional projects because it separates data-quality measurement from the cleaning decision.
Another valuable practice is to keep a record of how many observations were removed:
before_rows = len(df)
cleaned = df.dropna(
subset=["Customer_ID"]
)
after_rows = len(cleaned)
removed_rows = (
before_rows - after_rows
)
print("Rows removed:", removed_rows)This provides transparency.
For imputation, record how many values were originally missing:
missing_before = df["Age"].isna().sum()
median_age = df["Age"].median()
df["Age"] = df["Age"].fillna(
median_age
)
print("Values imputed:", missing_before)Such metadata can be included in a data-quality report.
When working with large datasets, avoid repeatedly scanning the same DataFrame unnecessarily if a complete profiling workflow can calculate several quality metrics together. Clear, reusable functions can make the process easier to maintain.
For example:
def missing_report(df):
return pd.DataFrame({
"Missing_Count": df.isna().sum(),
"Missing_Percentage": (
df.isna().mean() * 100
),
"Non_Missing_Count": df.notna().sum()
}).sort_values(
by="Missing_Percentage",
ascending=False
)
report = missing_report(customers)
print(report)This function can be reused across many datasets.
A more complete quality check can also identify columns with no variation after cleaning:
print(
customers.nunique(dropna=False)
)If a column becomes almost entirely the same value after imputation, that may indicate that the chosen treatment has reduced useful variation.
For example, if 90 percent of ages are missing and all missing values are replaced by the same median, the resulting distribution may not represent the original population well.
This is why imputation should be evaluated rather than accepted automatically.
Visualization can also help investigate missingness. A simple missing-count table can reveal problematic fields, while grouped summaries can show whether missingness is concentrated in particular regions, departments, dates, or customer segments.
For example:
customers["Purchase_Missing"] = (
customers["Purchase"].isna()
)
missing_by_city = (
customers
.groupby("City", dropna=False)["Purchase_Missing"]
.mean()
)
print(missing_by_city)The dropna=False option can preserve missing categories during grouping, allowing the analyst to examine records whose City itself is missing.
This is an advanced but useful technique when missingness needs to be analyzed as a pattern.
Another important issue is the timing of imputation in predictive analysis. If you calculate an imputation value using information from a future test dataset or future period, you may introduce data leakage. For simple descriptive analysis this may not be relevant, but for predictive modeling it is a critical concern.
The general principle is that preprocessing decisions should be based only on information legitimately available for the analytical stage being performed.
For example, in a predictive workflow, a median used to impute a training feature should normally be learned from the training data and then applied to validation or test data rather than recalculated independently using information from the test set.
This lesson focuses on Pandas data handling, but understanding this principle prepares you for later Machine Learning workflows.
Missing data can also affect calculated columns. Suppose:
sales = pd.DataFrame({
"Revenue": [100000, 120000, np.nan],
"Cost": [70000, 85000, 50000]
})
sales["Profit"] = (
sales["Revenue"] -
sales["Cost"]
)
print(sales)The missing Revenue value causes the corresponding Profit to remain missing.
You could choose to calculate Profit only for complete records:
complete = sales.loc[
sales["Revenue"].notna() &
sales["Cost"].notna()
].copy()
complete["Profit"] = (
complete["Revenue"] -
complete["Cost"]
)Alternatively, you might impute Revenue first, but that requires a defensible method.
This example demonstrates why missing-data treatment can affect downstream calculations.
If a missing value appears in a key input, every dependent metric may also become missing. Therefore, missing-data handling should be considered before building derived columns, KPIs, and visualizations.
For dashboard preparation, it is useful to decide explicitly whether missing values should appear as blank, zero, Unknown, or be excluded. The correct display depends on the metric.
For example, a missing sales amount should not automatically display as zero unless the dashboard definition says that zero represents no sales.
A missing customer city may reasonably appear as “Unknown” in a geographic table.
Transparency improves trust in the dashboard.
For E-E-A-T-oriented content and professional reporting, explain the data-quality limitations whenever they could influence interpretation. A report that says “12% of district records have missing monthly observations” provides valuable context to a reader interpreting a district ranking.
This is better than silently imputing the values and presenting the result as though the source data were complete.
At the end of a data-cleaning process, perform validation:
print(cleaned.info())
print(cleaned.isna().sum())
print(cleaned.describe())Check that data types remain appropriate:
print(cleaned.dtypes)Check row counts:
print("Original rows:", len(customers))
print("Cleaned rows:", len(cleaned))Check that required identifiers remain unique where appropriate:
print(
cleaned["Customer_ID"].duplicated().sum()
)Check that numerical values remain within valid business ranges:
print(
cleaned.loc[
cleaned["Age"] < 0
]
)Missing-data cleaning is only one part of data quality. A dataset can have no missing values and still contain incorrect, duplicated, inconsistent, or impossible values.
Therefore, after handling missing values, continue with broader validation.
The complete workflow can be summarized as:
Load Data
↓
Inspect Structure
↓
Identify Missing Values
↓
Measure Missingness
↓
Investigate Patterns
↓
Choose Treatment
↓
Drop / Fill / Interpolate
↓
Validate
↓
Analyze
↓
Document AssumptionsBy following this workflow, you avoid the common mistake of treating missing data as a purely technical problem.
Missing values often tell you something about the data collection process. A high level of missingness in a particular department may indicate a reporting problem. Missing customer income may indicate privacy concerns. Missing sensor values may indicate equipment failure. Missing dates may indicate incomplete records in the source system.
Therefore, missingness itself can sometimes be an analytical signal.
The final principle is simple: Do not ask only “How do I fill this missing value?” Ask “Why is this value missing, and what does that mean for my analysis?”
That question leads to better cleaning decisions, more reliable statistics, and more trustworthy Data Analytics results.
Lesson takeaway: Pandas provides powerful tools for detecting, removing, replacing, carrying forward, carrying backward, and interpolating missing data. The correct method depends on data type, business meaning, missingness pattern, analytical objective, and downstream use. Always measure missingness before cleaning, preserve the original data when possible, validate the transformed result, and document the assumptions used.
Applied Case Study: Customer Data Quality Audit
Consider a customer database containing demographic information, location, purchase activity, and customer status. Before calculating customer lifetime value or creating a marketing segmentation report, the analyst should understand the completeness of every important field.
customer_data = pd.DataFrame({
"Customer_ID": [101,102,103,104,105,106,107,108],
"Name": [
"Aman", "Priya", "Rahul", "Neha",
"Karan", "Sonia", "Arjun", "Meena"
],
"Age": [
24, np.nan, 31, 42,
np.nan, 29, 35, 27
],
"City": [
"Dehradun", "Delhi", None, "Haridwar",
"Dehradun", None, "Delhi", "Dehradun"
],
"Annual_Spend": [
55000, 72000, np.nan, 150000,
68000, 90000, np.nan, 47000
],
"Status": [
"Active", "Active", "Inactive", "Active",
"Active", "Active", "Inactive", "Active"
]
})Start with the missing-value report:
audit = pd.DataFrame({
"Missing_Count": customer_data.isna().sum(),
"Missing_Percentage": (
customer_data.isna().mean() * 100
)
})
print(audit)Suppose the business requirement says Customer_ID and Status are mandatory, while Age and City are optional. Annual_Spend is required for revenue analysis but may be unavailable for some inactive customers.
In that situation, blindly dropping every incomplete row would be inappropriate.
Instead, validate the mandatory fields:
valid_customers = customer_data.dropna(
subset=["Customer_ID", "Status"]
)
print(valid_customers)For Age, the analyst might retain missingness or use an appropriate imputation method depending on the intended analysis. For City, an Unknown category may be useful:
valid_customers["City"] = (
valid_customers["City"]
.fillna("Unknown")
)For Annual_Spend, if the analysis requires observed spending only, filter to non-missing values:
spend_analysis = valid_customers.loc[
valid_customers["Annual_Spend"].notna()
].copy()Now calculate summary statistics:
print(
spend_analysis["Annual_Spend"].describe()
)This approach keeps customers available for general analysis while restricting the spending analysis to records where spending is actually known.
Suppose management instead asks for an estimated average annual spend for all active customers. That is a different analytical question and may justify an imputation strategy, but the method should be documented.
active = valid_customers.loc[
valid_customers["Status"] == "Active"
].copy()
median_spend = active["Annual_Spend"].median()
active["Annual_Spend_Imputed"] = (
active["Annual_Spend"]
.fillna(median_spend)
)
print(active)The report should clearly distinguish observed spend from imputed spend.
You can create an indicator:
active["Spend_Was_Missing"] = (
active["Annual_Spend"].isna()
)This allows the analyst to identify which customers received an estimated value.
Now calculate a simple average:
print(
active["Annual_Spend_Imputed"].mean()
)Compare it with the observed-only average:
print(
active["Annual_Spend"].mean()
)If the two values differ meaningfully, the report should explain why.
This example demonstrates why data cleaning decisions can influence business conclusions.
Suppose the missing spending values belong disproportionately to high-value customers. Median imputation could understate the true average. Therefore, the analyst should investigate whether missingness is associated with customer status, acquisition channel, geography, or another variable.
print(
active.groupby(
active["Annual_Spend"].isna()
)["Age"].mean()
)Additional comparisons can be made across City or Status where appropriate.
The purpose is not to prove a causal relationship but to identify patterns that may influence the cleaning strategy.
This is the kind of reasoning expected in professional Data Analytics work.
A final quality report might contain:
Column
Missing Count
Missing %
Treatment
Reason
ImpactFor example, an analyst might document that City missing values were labelled Unknown because geographic assignment was unavailable, while Annual_Spend missing values were excluded from observed-revenue calculations because missing spend did not mean zero spend.
This level of transparency improves reproducibility and stakeholder confidence.
Another useful exercise is to compare three strategies for a numerical column:
original = customer_data["Annual_Spend"]
mean_filled = original.fillna(
original.mean()
)
median_filled = original.fillna(
original.median()
)
zero_filled = original.fillna(0)
print("Original mean:", original.mean())
print("Mean filled:", mean_filled.mean())
print("Median filled:", median_filled.mean())
print("Zero filled:", zero_filled.mean())The results demonstrate how different assumptions produce different analytical outcomes.
This is why an analyst should never select an imputation method merely because it removes missing values.
The goal is not to make the DataFrame look complete. The goal is to create a dataset that is appropriate for the intended analytical question.
For a final hands-on challenge, create your own dataset with at least 20 records and deliberately introduce missing values into five columns. Produce a missing-data audit, test at least three treatment strategies, compare their statistical effects, and write a short explanation of which strategy you would use and why.
Your final output should include the original dataset, cleaned dataset, missing-value report, treatment decisions, validation checks, and a short data-quality note.
This project will reinforce the central principle of the lesson: data cleaning is an analytical decision, not simply a technical operation.