After loading a dataset into Python using Pandas, the next step is to understand its structure before performing data cleaning, visualization, or Machine Learning. Data exploration helps you identify the size of the dataset, understand the data types, detect missing values, and summarize numerical information. Without understanding the dataset, it becomes difficult to choose the correct preprocessing techniques or Machine Learning algorithms.
Pandas provides several built-in functions that allow you to quickly inspect your dataset. The most commonly used functions are shape, info(), describe(), head(), tail(), columns, and dtypes. These functions provide valuable insights into the quality and characteristics of your data.
In this lesson, you will learn how to use these functions to explore datasets efficiently and prepare them for Data Analysis and Machine Learning.
Summary statistics provide a quick overview of a dataset without requiring you to examine every row individually. They help answer important questions such as:
import pandas as pd
df = pd.read_csv("students.csv")
The shape attribute returns the number of rows and columns.
df.shape
Example Output
(5000, 8)
This means the dataset contains 5,000 rows and 8 columns.
rows = df.shape[0]
columns = df.shape[1]
print(rows)
print(columns)
df.columns
Example Output
Index([
'Student_ID',
'Name',
'Age',
'Gender',
'Marks',
'City',
'Course',
'Attendance'
])
df.dtypes
Example Output
| Column | Data Type |
|---|---|
| Student_ID | int64 |
| Name | object |
| Marks | float64 |
| Attendance | float64 |
| Data Type | Description |
|---|---|
| int64 | Whole numbers |
| float64 | Decimal values |
| object | Text or strings |
| bool | True or False values |
| datetime64 | Date and time values |
The info() function provides a complete overview of the DataFrame.
df.info()
The output includes:
RangeIndex: 5000 entries
Data columns (total 8 columns)
Age 4985 non-null float64
Marks 5000 non-null float64
City 4990 non-null object
From this output, you can immediately identify missing values because the non-null count is smaller than the total number of rows.
The describe() function generates descriptive statistics for numerical columns.
df.describe()
| Statistic | Meaning |
|---|---|
| count | Total non-missing values |
| mean | Average value |
| std | Standard deviation |
| min | Smallest value |
| 25% | First quartile |
| 50% | Median |
| 75% | Third quartile |
| max | Largest value |
Age Marks
count 4985.00 5000.00
mean 21.42 74.83
std 3.82 11.75
min 17.00 30.00
25% 19.00 67.00
50% 21.00 75.00
75% 24.00 83.00
max 45.00 100.00
Suppose a university wants to analyze student performance. Before building a Machine Learning model to predict exam results, the Data Analyst loads the dataset and immediately checks shape, info(), and describe(). The analyst discovers that the Age column contains missing values and that Marks include unusually high values that may represent outliers. This initial exploration helps determine the preprocessing steps needed before model training.
info().shape, info(), and describe() together.info().describe().
Summary statistics are the foundation of Exploratory Data Analysis. Functions such as shape, info(), and describe() provide valuable information about dataset size, structure, data types, missing values, and numerical distributions. Performing these checks before cleaning or modeling helps identify potential issues early and leads to more reliable Data Analysis and Machine Learning workflows.
df.shape return?It returns a tuple containing the number of rows and columns in the DataFrame.
df.info() provide?It displays column names, data types, non-null counts, memory usage, and the overall structure of the DataFrame.
df.describe()?It generates descriptive statistics such as count, mean, standard deviation, minimum, maximum, and quartiles for numerical columns.
In the next lesson, you will learn Advanced Data Exploration with value_counts(), unique(), nunique(), corr(), and DataFrame indexing to gain deeper insights into your dataset before visualization and Machine Learning.
After understanding the basic structure of a dataset using shape, info(), and describe(), the next step is to perform deeper data exploration. Exploratory Data Analysis (EDA) helps Data Analysts and Machine Learning Engineers understand patterns, identify relationships, detect inconsistencies, and prepare data for feature engineering and model building.
Pandas provides many powerful functions that allow you to examine categorical variables, identify unique values, calculate frequencies, measure relationships between numerical variables, and efficiently access specific rows and columns. These techniques are used in almost every real-world Data Science project.
In this lesson, you will learn how to use value_counts(), unique(), nunique(), corr(), indexing methods, and filtering operations to explore datasets effectively.
The unique() function displays all distinct values present in a column.
df["City"].unique()
Example Output
['Delhi'
'Mumbai'
'Pune'
'Jaipur']
The nunique() function returns the number of distinct values.
df["City"].nunique()
Output
4
The value_counts() function counts how frequently each category appears.
df["Course"].value_counts()
Example Output
Python 420
SQL 315
Power BI 280
Excel 195
df["Course"].value_counts(normalize=True)
Example Output
Python 0.35
SQL 0.26
Power BI 0.23
Excel 0.16
df["City"].value_counts(ascending=True)
df[["Name","Marks","City"]]
df.iloc[0]
df.iloc[0:5]
df.iloc[0:5, 1:4]
The loc() function selects rows and columns using labels.
df.loc[0:4, ["Name","Marks"]]
Pandas allows filtering records based on conditions.
df[df["Marks"] > 80]
df[
(df["Marks"] > 80) &
(df["Attendance"] > 90)
]
df.sort_values("Marks")
df.sort_values(
"Marks",
ascending=False
)
Correlation measures how strongly numerical variables are related.
df.corr(numeric_only=True)
Example Output
| Age | Marks | Attendance | |
|---|---|---|---|
| Age | 1.00 | 0.42 | 0.28 |
| Marks | 0.42 | 1.00 | 0.75 |
| Attendance | 0.28 | 0.75 | 1.00 |
| Correlation | Meaning |
|---|---|
| +1 | Perfect positive relationship |
| 0 | No relationship |
| -1 | Perfect negative relationship |
df.isnull().sum()
df.duplicated().sum()
df.sample(5)
df.nlargest(5,"Marks")
df.nsmallest(5,"Marks")
Suppose a college wants to analyze student performance before building a Machine Learning model to predict final grades. Using value_counts(), the analyst identifies the most popular courses. With corr(), they discover that attendance has a strong positive relationship with marks. By filtering students who scored below 40, they identify learners who may need additional support. These exploratory steps provide valuable insights before data preprocessing and model training.
unique() with value_counts().iloc() and loc().
Advanced data exploration helps uncover meaningful insights before preprocessing and Machine Learning. Functions such as unique(), nunique(), value_counts(), corr(), iloc(), loc(), and filtering operations allow Data Scientists to understand categorical variables, numerical relationships, feature distributions, and hidden patterns. Performing these analyses before visualization and model training leads to better feature selection and more accurate predictive models.
unique() returns all distinct values, while nunique() returns only the number of unique values.
It counts how many times each category appears in a column and is commonly used for categorical data analysis.
The corr() function calculates correlation coefficients between numerical variables to measure the strength and direction of their relationships.
In the next lesson, you will learn Grouping and Aggregating Data with groupby(), agg(), pivot_table(), and Crosstab to summarize and analyze large datasets efficiently.
After exploring a dataset using shape, info(), describe(), value_counts(), and correlation analysis, the next step is to summarize data by categories. In real-world Data Analysis and Machine Learning projects, analysts rarely examine individual records. Instead, they group data based on categories such as department, city, gender, product, or year to identify trends and make data-driven decisions.
Pandas provides several powerful functions for grouping and summarizing data. The most commonly used are groupby(), agg(), pivot_table(), and crosstab(). These functions allow you to calculate averages, totals, counts, minimums, maximums, percentages, and many other statistics for different groups.
This lesson explains how to use these functions with practical examples commonly used in Data Analytics and Machine Learning.
The groupby() function divides a dataset into groups based on one or more columns and then performs calculations for each group.
df.groupby("Department")
df.groupby("Department")["Salary"].mean()
Example Output
| Department | Average Salary |
|---|---|
| HR | 42000 |
| IT | 68000 |
| Finance | 61000 |
df.groupby("Department")["Salary"].agg(
["mean","max","min","sum","count"]
)
df.groupby(
["Department","Gender"]
)["Salary"].mean()
The agg() function allows multiple aggregation operations at the same time.
df.groupby("City").agg({
"Salary":"mean",
"Age":"max",
"Experience":"sum"
})
| Function | Description |
|---|---|
| mean() | Average |
| sum() | Total |
| count() | Count |
| min() | Minimum |
| max() | Maximum |
| median() | Middle value |
| std() | Standard deviation |
A Pivot Table summarizes data similar to Microsoft Excel Pivot Tables.
pd.pivot_table(
df,
values="Salary",
index="Department",
aggfunc="mean"
)
pd.pivot_table(
df,
values=["Salary","Age"],
index="Department",
aggfunc=["mean","max"]
)
The crosstab() function counts relationships between categorical variables.
pd.crosstab(
df["Gender"],
df["Department"]
)
Example Output
| Gender | HR | IT | Finance |
|---|---|---|---|
| Male | 40 | 65 | 35 |
| Female | 55 | 38 | 42 |
df.groupby("Department").agg({
"Salary":["mean","max"],
"Age":["min","max"],
"Experience":"mean"
})
df.groupby("Department")["Salary"]
.mean()
.sort_values(ascending=False)
df["Department"]
.value_counts()
summary = df.groupby("Department")["Salary"].mean()
summary.reset_index()
Suppose an HR department wants to analyze employee salaries. Using groupby(), they calculate the average salary for each department. With pivot_table(), they compare salaries across departments and cities. Using crosstab(), they analyze the gender distribution within each department. These summaries help management make informed hiring and compensation decisions.
agg().
Grouping and aggregation are fundamental skills in Data Analysis. Functions such as groupby(), agg(), pivot_table(), and crosstab() allow analysts to summarize large datasets efficiently and discover meaningful business insights. These techniques are widely used in reporting, dashboard development, feature engineering, and Machine Learning preprocessing.
The groupby() function groups data based on one or more columns and applies calculations such as averages, sums, counts, or maximum values to each group.
groupby() is mainly used for data aggregation in Python code, while pivot_table() creates report-style summaries similar to Excel Pivot Tables.
Use crosstab() when analyzing relationships between categorical variables, such as gender versus department or city versus product category.
In the next lesson, you will learn Exploratory Data Analysis (EDA) with Pandas and Matplotlib, including histograms, box plots, scatter plots, bar charts, distribution analysis, and identifying trends before building Machine Learning models.