Reading and exploring data is the first practical step in every Data Science, Data Analytics, and Machine Learning project. Before cleaning data, creating visualizations, or training Machine Learning models, you must first understand the structure and quality of your dataset. Pandas provides powerful functions that allow you to load datasets from different sources and inspect them within seconds.
Real-world datasets often contain thousands or even millions of rows. Instead of manually examining every record, Pandas offers built-in methods that summarize the dataset, display sample records, identify missing values, reveal data types, and calculate descriptive statistics. These techniques help data scientists understand the dataset before performing preprocessing and feature engineering.
In this lesson, you will learn how to read datasets using Pandas, inspect DataFrames, understand rows and columns, check data types, identify missing values, and explore the overall structure of a dataset.
Machine Learning models depend on high-quality data. Before building any predictive model, it is important to understand what information the dataset contains, how many records are available, which columns are useful, and whether the data contains errors.
import pandas as pd
CSV (Comma-Separated Values) files are one of the most common formats used in Data Science and Machine Learning. The read_csv() function loads a CSV file into a Pandas DataFrame.
import pandas as pd
df = pd.read_csv("students.csv")
Many organizations store data in Microsoft Excel spreadsheets. Pandas can read Excel files using the read_excel() function.
import pandas as pd
df = pd.read_excel("students.xlsx")
import pandas as pd
df = pd.read_json("students.json")
import pandas as pd
import sqlite3
connection = sqlite3.connect("school.db")
df = pd.read_sql("SELECT * FROM students", connection)
The head() method displays the first five rows of a dataset by default.
df.head()
Display the first ten rows.
df.head(10)
df.tail()
Display the last ten rows.
df.tail(10)
The shape attribute returns the number of rows and columns.
print(df.shape)
Output
(5000, 8)
This means the dataset contains 5,000 rows and 8 columns.
print(df.columns)
Example Output
Index(['Student_ID',
'Name',
'Age',
'Gender',
'Marks',
'City',
'Course',
'Attendance'])
print(df.dtypes)
This function displays the data type of every column.
| Data Type | Description |
|---|---|
| int64 | Integer values |
| float64 | Decimal values |
| object | Text values |
| bool | True or False |
| datetime64 | Date and Time |
The info() function provides a complete overview of the dataset.
df.info()
The output includes:
The describe() function summarizes numerical columns.
df.describe()
It returns:
df.isnull().sum()
This command counts missing values in every column.
df.duplicated().sum()
This displays the total number of duplicate rows.
df.sample(5)
Displays five random rows from the dataset.
df["Marks"]
df[["Name","Marks","City"]]
df["City"].unique()
df["City"].nunique()
Count the frequency of each category.
df["Course"].value_counts()
Suppose you receive a dataset containing information about 50,000 bank customers. Before predicting whether customers will purchase a financial product, you first inspect the dataset using head(), info(), describe(), shape, and isnull(). These functions help identify missing values, incorrect data types, and unusual patterns before any preprocessing begins.
info() before analysis.head() and tail() to verify the data.read_csv().Reading and exploring data is the foundation of every Data Science and Machine Learning project. Pandas provides simple yet powerful functions to load datasets, inspect their structure, identify missing values, examine data types, and generate statistical summaries. Performing these exploratory steps before data preprocessing ensures that the dataset is well understood and ready for cleaning, feature engineering, and model development.
Exploring a dataset helps identify missing values, incorrect data types, duplicate records, and potential issues that can negatively affect model performance.
head() and tail()?
The head() function displays the first rows of a DataFrame, while tail() displays the last rows.
describe() do?
The describe() function generates summary statistics such as count, mean, standard deviation, minimum, maximum, and quartiles for numerical columns.
Use df.isnull().sum() to count missing values in each column.
In the next lesson, you will learn Handling Missing Values in Pandas, including detecting null values, filling missing data with mean, median, mode, forward fill, backward fill, and advanced imputation techniques used in Machine Learning.
Missing values are one of the most common problems encountered in real-world datasets. Before training a Machine Learning model, it is essential to identify, understand, and handle missing data properly. If missing values are ignored, they can reduce model accuracy, introduce bias, and even cause Machine Learning algorithms to fail.
Pandas provides several built-in functions for detecting, removing, and replacing missing values efficiently. In this lesson, you will learn different techniques for handling missing values, understand when to use each method, and apply them using practical examples.
Missing values represent unavailable or unknown information in a dataset. In Pandas, missing values are usually represented by NaN (Not a Number), None, or NA.
Missing values may occur because data was not collected, users skipped questions, sensors failed to record measurements, or information became unavailable during data collection.
import pandas as pd
import numpy as np
data = {
"Name":["Rahul","Priya","Aman","Neha"],
"Age":[20,np.nan,21,22],
"Marks":[85,90,np.nan,88],
"City":["Delhi","Mumbai",None,"Pune"]
}
df = pd.DataFrame(data)
print(df)
Use the isnull() function to identify missing values.
df.isnull()
df.isnull().sum()
Example Output
Name 0
Age 1
Marks 1
City 1
df.isnull().sum().sum()
df.notnull()
The dropna() function removes rows containing missing values.
df.dropna()
df.dropna(axis=1)
df.dropna(how="all")
df.fillna(0)
Replace missing text values.
df.fillna("Unknown")
The mean is commonly used for numerical columns with normally distributed data.
df["Marks"] = df["Marks"].fillna(df["Marks"].mean())
Median is useful when the dataset contains outliers.
df["Age"] = df["Age"].fillna(df["Age"].median())
Mode is commonly used for categorical variables.
df["City"] = df["City"].fillna(df["City"].mode()[0])
Forward Fill copies the previous valid value into missing cells.
df.fillna(method="ffill")
Backward Fill copies the next valid value into missing cells.
df.fillna(method="bfill")
Interpolation estimates missing numerical values using nearby observations.
df.interpolate()
(df.isnull().sum() / len(df)) * 100
| Situation | Recommended Action |
|---|---|
| Very few missing rows | Remove rows |
| Many missing rows | Fill missing values |
| Important numerical column | Mean or Median |
| Categorical column | Mode |
| Time-series data | Forward Fill or Backward Fill |
Suppose a hospital dataset contains missing patient ages and blood pressure values. Removing all records would waste valuable information. Instead, missing ages can be replaced using the median age, while missing blood pressure values can be estimated using interpolation or the mean.
Handling missing values is an essential part of data preprocessing in Machine Learning. Pandas provides flexible methods such as dropna(), fillna(), forward fill, backward fill, interpolation, and statistical imputation using mean, median, and mode. Choosing the right technique depends on the type of data, the amount of missing information, and the business problem. Proper handling of missing values improves data quality and leads to more accurate Machine Learning models.
NaN stands for “Not a Number” and represents missing numerical values in a Pandas DataFrame.
No. Removing rows is appropriate only when the number of missing records is small. Otherwise, filling missing values is often a better option.
Use the median when numerical data contains outliers because it is less affected by extreme values.
The mode is generally the best choice because it replaces missing values with the most frequently occurring category.
In the next lesson, you will learn Removing Duplicate Data and Detecting Outliers in Pandas, including duplicate detection, duplicate removal, outlier identification using IQR and Z-score methods, and data quality improvement techniques for Machine Learning.
Real-world datasets often contain duplicate records and unusual observations known as outliers. These problems reduce data quality and negatively affect Machine Learning models. Duplicate rows can bias predictions, while outliers can distort statistical calculations and decrease model accuracy. Therefore, identifying and handling duplicates and outliers is an essential step in data preprocessing.
Pandas provides powerful functions to detect and remove duplicate records, while statistical techniques such as the Interquartile Range (IQR) and Z-Score methods help identify outliers. In this lesson, you will learn how to detect duplicates, remove redundant data, identify outliers, and prepare high-quality datasets for Machine Learning.
Duplicate records are rows that contain identical information. They often occur due to repeated data entry, merging datasets, system errors, or importing data multiple times.
import pandas as pd
data = {
"Name":["Rahul","Priya","Rahul","Aman"],
"Age":[21,22,21,20],
"Marks":[85,91,85,78]
}
df = pd.DataFrame(data)
print(df)
The duplicated() method identifies duplicate rows.
df.duplicated()
df.duplicated().sum()
Output
1
duplicates = df[df.duplicated()]
print(duplicates)
df = df.drop_duplicates()
print(df)
df.drop_duplicates(subset=["Name"])
df.drop_duplicates(keep="last")
df.drop_duplicates(keep=False)
Outliers are observations that differ significantly from the rest of the data. They may occur because of measurement errors, data entry mistakes, or naturally rare events.
For example, if the salaries of most employees range between ₹30,000 and ₹90,000 but one record contains ₹25,00,000, that value may be an outlier.
import seaborn as sns
import matplotlib.pyplot as plt
sns.boxplot(x=df["Marks"])
plt.show()
Points outside the whiskers of the box plot are potential outliers.
The Interquartile Range (IQR) method is one of the most common techniques for detecting outliers.
Q1 = df["Marks"].quantile(0.25)
Q3 = df["Marks"].quantile(0.75)
IQR = Q3 - Q1
print(IQR)
lower_limit = Q1 - 1.5 * IQR
upper_limit = Q3 + 1.5 * IQR
print(lower_limit)
print(upper_limit)
outliers = df[
(df["Marks"] < lower_limit) |
(df["Marks"] > upper_limit)
]
print(outliers)
df = df[
(df["Marks"] >= lower_limit) &
(df["Marks"] <= upper_limit)
]
The Z-Score measures how far a value is from the mean in terms of standard deviations.
from scipy.stats import zscore
df["Z_Score"] = zscore(df["Marks"])
print(df)
outliers = df[
abs(df["Z_Score"]) > 3
]
print(outliers)
| Method | Best Used For |
|---|---|
| IQR | Skewed distributions |
| Z-Score | Normally distributed data |
Suppose a bank is building a Machine Learning model to predict customer loan eligibility. During data exploration, several duplicate customer IDs are found because customer information was imported twice. Additionally, one customer's annual income is recorded as ₹500 crore instead of ₹5 lakh due to a typing mistake. Removing duplicates and correcting outliers significantly improves the quality of the training dataset.
Removing duplicate records and handling outliers are critical steps in data preprocessing. Duplicate rows can bias Machine Learning models, while outliers can distort statistical measures and reduce prediction accuracy. Pandas provides simple functions such as duplicated() and drop_duplicates() for managing duplicate data, while statistical methods like the Interquartile Range (IQR) and Z-Score help identify unusual observations. Careful handling of duplicates and outliers leads to cleaner datasets and more reliable Machine Learning models.
A duplicate record is a row that appears more than once in a dataset, often due to repeated data entry or data integration issues.
No. Some outliers represent genuine observations and may contain valuable information. Always investigate the cause before deciding whether to remove or retain them.
The IQR method works well for skewed data, while the Z-Score method is more suitable for normally distributed datasets.
df.duplicated().sum()
In the next lesson, you will learn Encoding Categorical Variables in Machine Learning, including Label Encoding, One-Hot Encoding, Ordinal Encoding, Target Encoding, and best practices for preparing categorical data for Machine Learning algorithms.