In the previous lesson, you learned about the Pandas Series, one of the fundamental data structures used in Python for Data Analytics. A Series stores one-dimensional labeled data. In real-world analytics, however, datasets normally contain multiple columns such as customer name, age, location, sales, product, date, and revenue.
This is where the Pandas DataFrame becomes essential.
A Pandas DataFrame is a two-dimensional labeled data structure that organizes information into rows and columns. It is one of the most widely used structures in Python Data Analytics because it provides a convenient way to work with tabular data.
If you have worked with Microsoft Excel, Google Sheets, or a database table, the basic structure of a DataFrame will feel familiar. You can think of a DataFrame as a programmable data table where each column can contain a particular type of information and each row generally represents an observation or record.
For example, consider a student dataset:
import pandas as pd
data = {
"Name": ["Aman", "Priya", "Rahul", "Neha"],
"Age": [21, 22, 20, 23],
"Marks": [82, 91, 76, 88]
}
df = pd.DataFrame(data)
print(df)The output will look similar to:
Name Age Marks
0 Aman 21 82
1 Priya 22 91
2 Rahul 20 76
3 Neha 23 88Here, Name, Age, and Marks are columns. Each row represents one student, and the numbers on the left represent the DataFrame index.
This simple structure is the foundation for much more advanced analysis.
A useful way to understand a DataFrame is to compare it with a Series.
A Series is one-dimensional:
marks = pd.Series([82, 91, 76, 88])A DataFrame is two-dimensional:
df = pd.DataFrame({
"Name": ["Aman", "Priya", "Rahul", "Neha"],
"Marks": [82, 91, 76, 88]
})The DataFrame contains multiple columns, while a Series normally represents one labeled column of data.
When you select one column from a DataFrame:
df["Marks"]the result is generally a Pandas Series.
When you select multiple columns:
df[["Name", "Marks"]]the result remains a DataFrame.
This relationship is extremely important. You can think of a DataFrame as a collection of aligned Series that share a common row index.
Every DataFrame has two primary dimensions: rows and columns.
Consider:
students = pd.DataFrame({
"Name": ["Aman", "Priya", "Rahul"],
"Course": ["Python", "Data Analytics", "SQL"],
"Score": [85, 92, 78]
})The columns are:
The rows represent individual student records.
Conceptually, the data looks like:
| Index | Name | Course | Score |
|---|---|---|---|
| 0 | Aman | Python | 85 |
| 1 | Priya | Data Analytics | 92 |
| 2 | Rahul | SQL | 78 |
The index is not normally considered a data column in the same way as Name, Course, and Score. It provides labels that identify the rows.
One of the most common ways to create a DataFrame is by using a Python dictionary.
data = {
"Product": ["Laptop", "Mobile", "Tablet", "Monitor"],
"Price": [65000, 25000, 32000, 18000],
"Quantity": [5, 12, 7, 9]
}
df = pd.DataFrame(data)
print(df)Each dictionary key becomes a column name, while the associated list becomes the column’s values.
The lists must generally have compatible lengths because each row requires a value for every column.
For example, this structure is problematic:
data = {
"Name": ["Aman", "Priya", "Rahul"],
"Age": [21, 22]
}The number of values does not match, so Pandas cannot construct the expected rectangular table from these lists.
This is an important concept: a normal DataFrame represents rectangular tabular data where rows and columns are aligned.
Another practical method is to provide a list containing dictionaries.
students = [
{"Name": "Aman", "Age": 21, "Marks": 82},
{"Name": "Priya", "Age": 22, "Marks": 91},
{"Name": "Rahul", "Age": 20, "Marks": 76},
{"Name": "Neha", "Age": 23, "Marks": 88}
]
df = pd.DataFrame(students)
print(df)This approach is particularly useful when records are already structured individually.
Each dictionary represents one record, while the dictionary keys become column names.
It is also useful when data comes from APIs or other systems that return records in a dictionary-like structure.
You can also create a DataFrame from a two-dimensional Python list.
data = [
["Aman", 21, 82],
["Priya", 22, 91],
["Rahul", 20, 76],
["Neha", 23, 88]
]
df = pd.DataFrame(
data,
columns=["Name", "Age", "Marks"]
)
print(df)Here, the columns parameter specifies the names of the columns.
This approach can be useful when the data is already available as a matrix-like structure.
You can also create an empty DataFrame:
df = pd.DataFrame()
print(df)An empty DataFrame contains no rows or columns.
You may also define its columns initially:
df = pd.DataFrame(
columns=["Name", "Age", "Marks"]
)
print(df)Although creating an empty DataFrame is possible, professional analytical workflows generally prefer loading or constructing data from a defined source rather than repeatedly adding individual rows manually.
Once a DataFrame has been created, one of the first tasks should be inspection.
The head() method displays the first few rows:
print(df.head())You can specify how many rows you want:
print(df.head(3))Similarly, tail() displays the last rows:
print(df.tail())These methods are extremely useful when working with large datasets because printing an entire dataset may produce thousands or millions of lines.
An analyst should normally inspect a sample before performing transformations or calculations.
The shape attribute tells you the number of rows and columns.
print(df.shape)If the DataFrame contains 100 rows and 5 columns, the result will be:
(100, 5)The first number represents rows, while the second represents columns.
This is one of the quickest ways to understand the size of a dataset.
For example:
rows = df.shape[0]
columns = df.shape[1]
print("Rows:", rows)
print("Columns:", columns)This distinction becomes useful when checking whether a dataset has loaded completely.
The columns attribute returns the DataFrame’s column labels.
print(df.columns)You can convert them to a Python list:
print(df.columns.tolist())This is useful when you need to understand the exact names used in a dataset.
For example, a dataset may contain:
["customer_id", "customer_name", "city", "sales", "date"]Knowing the exact column names helps prevent errors when selecting or transforming data.
The index attribute displays the row labels.
print(df.index)By default, a DataFrame normally uses a zero-based integer index:
0
1
2
3
...However, the index can also be changed to meaningful labels or dates when appropriate.
For example:
df.index = ["S001", "S002", "S003", "S004"]
print(df)Now the student IDs act as row labels.
However, an index should be chosen carefully. A meaningful index can be useful for selection and alignment, but not every dataset needs a custom index.
The info() method provides a compact overview of the DataFrame.
print(df.info())It typically provides information about:
This makes info() one of the most useful first inspection tools after loading a dataset.
For example, if a dataset contains a Sales column with 10,000 rows but only 9,500 non-null values, the analyst immediately knows that 500 observations require further investigation.
You can inspect the data type of every column using:
print(df.dtypes)A typical dataset might contain:
Name object
Age int64
Marks int64
The exact dtype names can vary depending on the data and Pandas version.
Data types matter because different columns require different analytical operations.
For example, numerical columns can generally be used for mathematical calculations, while text columns may require string operations or categorical analysis.
A date column should ideally be represented using an appropriate datetime representation if you intend to perform date-based analysis.
The real strength of a DataFrame appears when you combine its structure with analytical operations.
Imagine a sales dataset containing:
sales = pd.DataFrame({
"Product": ["Laptop", "Mobile", "Tablet", "Laptop"],
"Region": ["North", "South", "East", "West"],
"Sales": [65000, 45000, 32000, 72000]
})You can calculate total sales:
print(sales["Sales"].sum())Average sales:
print(sales["Sales"].mean())Highest sale:
print(sales["Sales"].max())And select only products with sales above 50,000:
print(sales[sales["Sales"] > 50000])Notice that the Series concepts from Lesson 2 are now being applied to a DataFrame column.
This is why learning Series first was important. Once you understand Series operations, DataFrame analysis becomes much more intuitive.
In a professional Data Analytics project, a DataFrame might represent thousands or millions of records.
For example, a company could have a customer DataFrame with columns such as:
The analyst can use Pandas to inspect the dataset, identify missing values, calculate statistics, filter records, create new columns, sort information, group data, and prepare the cleaned dataset for visualization or further analysis.
This makes the DataFrame one of the central structures in a typical Python-based Data Analytics workflow.
However, creating a DataFrame is only the beginning. The real analytical value comes from understanding the data, selecting the correct records and columns, validating the results, and connecting the analysis to the original business question.
In the next section, we will work extensively with DataFrame rows and columns. You will learn how to select one or multiple columns, select rows using loc and iloc, slice datasets, modify columns, rename columns, and combine row and column selection.
In Part 1, you learned what a Pandas DataFrame is, how it differs from a Series, how to create a DataFrame from Python dictionaries and lists, and how to inspect its basic structure using methods and attributes such as head(), shape, columns, index, dtypes, and info().
Now we move to one of the most important practical skills in Pandas: selecting and working with specific rows and columns.
A real dataset may contain hundreds of columns and millions of rows, but an analytical question usually requires only a small portion of that information. For example, a business manager may ask for only the Product and Sales columns, or an analyst may need only customers from a particular city. Pandas provides several ways to make these selections efficiently.
Understanding selection is also essential before you learn filtering, sorting, grouping, merging, and advanced data transformation.
Suppose we have a student DataFrame:
import pandas as pd
students = pd.DataFrame({
"Name": ["Aman", "Priya", "Rahul", "Neha"],
"Age": [21, 22, 20, 23],
"Course": ["Python", "Data Analytics", "SQL", "Python"],
"Marks": [82, 91, 76, 88]
})
print(students)To select the Marks column, use:
marks = students["Marks"]
print(marks)The result is a Pandas Series.
This is important because selecting one column generally changes the dimensionality from a two-dimensional DataFrame to a one-dimensional Series.
You can also use:
marks = students.MarksAlthough attribute-style access can be convenient, bracket notation such as students["Marks"] is generally safer and more consistent because it works with column names containing spaces, special characters, or names that conflict with DataFrame attributes or methods.
If you need more than one column, provide a list of column names:
selected = students[["Name", "Marks"]]
print(selected)The result remains a DataFrame.
This distinction is worth remembering:
students["Marks"]normally returns a Series.
students[["Marks"]]returns a DataFrame containing one column.
Similarly:
students[["Name", "Course", "Marks"]]returns a DataFrame containing the three requested columns.
Multiple-column selection is useful when preparing a smaller analytical dataset for further processing or visualization.
The loc accessor allows label-based selection.
Consider:
students.loc[1]With the default index, this selects the row whose index label is 1.
The result is generally a Series containing the values from that row.
You can select multiple rows:
students.loc[[0, 2]]This returns the rows with labels 0 and 2.
You can also select a range:
students.loc[0:2]With label-based slicing, the ending label is generally included when the labels are present.
This is different from ordinary Python positional slicing, so it is important to understand whether you are working with labels or positions.
The iloc accessor selects rows based on integer positions.
students.iloc[0]This selects the first row.
The second row can be selected using:
students.iloc[1]Multiple positions can be selected:
students.iloc[[0, 2, 3]]And a positional slice can be used:
students.iloc[0:3]This selects positions 0, 1, and 2 because the ending position is excluded, following Python’s normal positional slicing behavior.
The central rule remains:
loc works with labels; iloc works with positions.
One of the most useful capabilities of loc and iloc is selecting both rows and columns at the same time.
For example:
students.loc[0:2, ["Name", "Marks"]]This means:
Select rows from label 0 through label 2 and return only the Name and Marks columns.
You can use iloc similarly:
students.iloc[0:3, 0:2]This means:
Select the first three row positions and the first two column positions.
This ability becomes extremely valuable when working with large datasets because you can precisely specify the information required for an analysis.
loc can also be combined with conditions.
Suppose you want students whose marks are greater than 80:
high_scores = students.loc[students["Marks"] > 80]
print(high_scores)This returns the complete rows satisfying the condition.
You can also return only selected columns:
high_scores = students.loc[
students["Marks"] > 80,
["Name", "Marks"]
]
print(high_scores)This is a powerful pattern because it separates the two analytical questions:
The first part defines the row condition, while the second part defines the columns.
Suppose you want students who are enrolled in Python and have marks above 80.
result = students.loc[
(students["Course"] == "Python") &
(students["Marks"] > 80)
]
print(result)For an OR condition:
result = students.loc[
(students["Course"] == "Python") |
(students["Marks"] > 90)
]
print(result)Remember that Pandas uses & for element-wise AND and | for element-wise OR.
Parentheses around individual conditions are important.
Suppose you want all students enrolled in Python:
python_students = students.loc[
students["Course"] == "Python"
]
print(python_students)This demonstrates that DataFrame filtering is not limited to numerical values. You can filter based on text, categories, dates, Boolean values, and other data types.
Data Analysts frequently create new columns from existing information.
Suppose we want to classify students according to their marks.
students["Result"] = students["Marks"] >= 50
print(students)This creates a new Boolean column called Result.
The DataFrame may now contain:
Name Age Course Marks Result
0 Aman 21 Python 82 True
1 Priya 22 Data Analytics 91 True
2 Rahul 20 SQL 76 True
3 Neha 23 Python 88 TrueThis is an example of feature creation, where a new analytical field is derived from existing information.
Suppose the academy wants to calculate a bonus score equal to 5% of the marks:
students["Bonus"] = students["Marks"] * 0.05
print(students)Every value in the Marks column is multiplied by 0.05, and the resulting values are stored in the new Bonus column.
You can also calculate a final score:
students["Final Score"] = ( students["Marks"] + students["Bonus"] )This type of vectorized calculation is one of the major advantages of Pandas.
Renaming Columns
Column names sometimes need to be standardized.
For example:
students = students.rename( columns={ "Marks": "Score", "Course": "Program" } ) print(students)Now the columns have clearer or more standardized names.
Renaming is especially useful when datasets come from different sources and use inconsistent naming conventions.
For example, one source may use:
customer_namewhile another uses:
Customer NameA consistent naming strategy makes subsequent analysis easier.
Changing All Column Names
You can replace the entire column index:
students.columns = [ "student_name", "age", "course", "marks" ]This approach requires the number of new names to match the number of existing columns.
For larger projects, consistent column naming can make Python code easier to read and maintain.
Removing a Column
If a column is no longer required, you can remove it using
drop().students = students.drop( columns=["Bonus"] ) print(students)You can remove multiple columns at once:
students = students.drop( columns=["Bonus", "Final Score"] )It is good practice to be deliberate when deleting data. Before removing a column, make sure it is genuinely unnecessary for the analysis or final output.
Removing Rows
You can also remove rows by index label:
students = students.drop(index=2) print(students)This removes the row with index label 2.
Multiple rows can be removed:
students = students.drop(index=[1, 3])In data-cleaning workflows, row removal should be based on a clear rule. For example, you might remove duplicate records or records that fail a defined quality requirement, but arbitrary deletion can introduce bias into an analysis.
Selecting the First and Last Rows
You can use
head()to inspect the beginning of a DataFrame:print(students.head())And
tail()for the end:print(students.tail())You can specify the number of rows:
print(students.head(2)) print(students.tail(2))These methods are particularly useful for quickly checking whether data has loaded correctly.
Selecting Columns with a List
Suppose a sales DataFrame contains many fields:
sales = pd.DataFrame({ "Order_ID": [101, 102, 103, 104], "Customer": ["Aman", "Priya", "Rahul", "Neha"], "Product": ["Laptop", "Mobile", "Tablet", "Laptop"], "Region": ["North", "South", "East", "West"], "Quantity": [2, 5, 3, 1], "Sales": [130000, 125000, 96000, 72000] })If the business team needs only customer, product, region, and sales:
report = sales[ ["Customer", "Product", "Region", "Sales"] ] print(report)This creates a focused DataFrame suitable for further reporting.
Changing the Order of Columns
Column selection can also change their order.
report = sales[ ["Region", "Product", "Customer", "Sales"] ]The data itself is not changed; you are creating a DataFrame view or selection with the requested column arrangement.
This is useful when preparing data for reports, exports, or presentations where a particular column order improves readability.
Using iloc for a DataFrame Matrix
Because
ilocworks with positions, it can be useful when you do not want to refer to column names directly.For example:
sales.iloc[0:3, 1:4]This selects rows at positions 0 through 2 and columns at positions 1 through 3, excluding position 4.
This approach can be useful for positional operations, but for business-readable analytical code, explicit column names are often easier to understand and maintain.
Using loc with Column Labels
Suppose you want the first three rows and specific columns:
sales.loc[ 0:2, ["Customer", "Sales"] ]This is often very readable because someone reviewing the code can immediately understand which columns are being selected.
Practical Business Example
Imagine a company has a sales DataFrame containing 50,000 transactions. Management asks:
“Show me customers from the North region whose sales exceeded ₹50,000, and display only the customer name, product, and sales.”
A Pandas solution could be:
result = sales.loc[ (sales["Region"] == "North") & (sales["Sales"] > 50000), ["Customer", "Product", "Sales"] ] print(result)This is a complete analytical selection.
First, Pandas evaluates the region condition.
Second, it evaluates the sales condition.
Third, the conditions are combined.
Finally, only the requested columns are returned.
This pattern is extremely common in practical Data Analytics.
Why DataFrame Selection Matters
Learning selection is not simply about memorizing Pandas syntax. It is about translating analytical questions into precise operations.
For example:
This way of thinking is more important than memorizing individual commands.
Create this DataFrame:
employees = pd.DataFrame({
"Name": ["Amit", "Priya", "Rahul", "Neha", "Karan", "Sonia"],
"Department": [
"Analytics", "HR", "Analytics",
"Finance", "IT", "Analytics"
],
"Age": [28, 31, 26, 35, 29, 27],
"Salary": [55000, 48000, 62000, 70000, 65000, 58000]
})Complete these tasks:
Annual Salary.Department to Team.Try solving these tasks first without looking at documentation. The goal is to develop practical fluency rather than simply copying syntax.
Using one pair of brackets when multiple columns are required:
df["Name", "Salary"]This is not the normal syntax for selecting multiple DataFrame columns.
Use:
df[["Name", "Salary"]]Confusing loc and iloc:
Remember that loc works with labels and iloc works with integer positions.
Forgetting parentheses with multiple conditions:
df[(df["Age"] > 25) & (df["Salary"] > 50000)]Using Python's and/or instead of Pandas element-wise operators:
For Series conditions, use & and | rather than ordinary and and or.
Changing the dataset without checking the result:
After important transformations, inspect the DataFrame with methods such as head(), shape, or info(). Validation is an essential part of professional analytical work.
loc performs label-based selection.iloc performs position-based selection.In the next part, we will move beyond basic selection and examine DataFrame data types, missing values, statistical summaries, sorting, filtering, and calculated columns. These operations will help you turn a basic DataFrame into a useful analytical dataset.
In the previous parts of this lesson, you learned how to create a Pandas DataFrame and how to select rows and columns using standard indexing, loc, and iloc. These skills allow you to control exactly which part of a dataset you want to work with.
The next step is to understand the condition of the data itself. Before performing serious analysis, a Data Analyst needs to know what types of values exist, how many records are available, whether missing values are present, what the numerical distribution looks like, and whether the data is sorted or filtered correctly.
Pandas provides a collection of methods and attributes that make this inspection process fast and reproducible.
Every column in a DataFrame has a data type. You can inspect all column data types using:
print(df.dtypes)For example:
employees = pd.DataFrame({
"Name": ["Amit", "Priya", "Rahul", "Neha"],
"Age": [28, 31, 26, 35],
"Salary": [55000, 48000, 62000, 70000],
"Department": ["Analytics", "HR", "Analytics", "Finance"]
})
print(employees.dtypes)The result will show that numerical columns are stored using numerical data types while text columns use a text-compatible dtype.
Data types matter because they determine what operations are appropriate. For example, calculating the mean of a salary column makes sense, while calculating the mean of employee names does not.
One of the first commands you should run after loading a dataset is:
employees.info()The info() method provides a compact overview of the DataFrame, including the number of entries, column names, non-null values, data types, and memory usage.
This is especially useful for large datasets because it gives you a quick understanding of the dataset without printing every record.
Suppose a DataFrame contains 10,000 rows but the Salary column has only 9,700 non-null values. You immediately know that 300 salary records are missing and require investigation.
The shape attribute tells you the dimensions of the DataFrame:
print(employees.shape)If the result is:
(1000, 8)the DataFrame contains 1,000 rows and 8 columns.
You can separately obtain the number of rows and columns:
print("Rows:", employees.shape[0])
print("Columns:", employees.shape[1])The column names can be viewed with:
print(employees.columns)And the row index can be viewed with:
print(employees.index)These simple checks are useful for validating whether the dataset loaded as expected.
For numerical columns, describe() provides a statistical summary:
print(employees.describe())Depending on the data, the output can include:
For example, if you analyze employee salaries, describe() can quickly show the average salary, minimum salary, maximum salary, and the distribution's quartiles.
This does not replace deeper statistical analysis, but it is an excellent first step in exploratory data analysis.
You can also apply it to one column:
print(employees["Salary"].describe())This returns the statistical summary specifically for Salary.
For text or categorical data, frequency analysis is often more useful than numerical statistics.
print(employees["Department"].value_counts())This tells you how many employees belong to each department.
You can calculate the proportion of each category:
print(
employees["Department"].value_counts(
normalize=True
) * 100
)This can answer questions such as:
Missing data is common in real-world datasets. Pandas provides isna() to identify missing values.
print(employees.isna())This produces a Boolean DataFrame where missing values are represented by True.
To count missing values in each column:
print(employees.isna().sum())This is one of the most useful data-cleaning checks in Pandas.
You can also calculate the percentage of missing values:
missing_percentage = (
employees.isna().mean() * 100
)
print(missing_percentage)This gives you a column-by-column view of missing-data levels.
The opposite of isna() is notna():
print(employees.notna())You can use it to select records where a particular field exists:
complete_salary = employees.loc[
employees["Salary"].notna()
]
print(complete_salary)This is useful when you need to perform analysis only on records containing valid observations.
Filtering is one of the most frequently used DataFrame operations.
Suppose you want employees earning more than ₹60,000:
high_salary = employees[
employees["Salary"] > 60000
]
print(high_salary)You can use several comparison operators:
employees[employees["Age"] >= 30]
employees[employees["Salary"] < 60000]
employees[employees["Department"] == "Analytics"]Filtering allows an analyst to turn a broad dataset into a targeted subset relevant to a particular question.
Suppose you want employees who are older than 27 and earn more than ₹55,000:
result = employees[
(employees["Age"] > 27) &
(employees["Salary"] > 55000)
]
print(result)For an OR condition:
result = employees[
(employees["Department"] == "Analytics") |
(employees["Salary"] > 65000)
]
print(result)These operations allow you to translate business rules directly into DataFrame filters.
Suppose you want employees whose salary is between ₹50,000 and ₹70,000.
result = employees[
(employees["Salary"] >= 50000) &
(employees["Salary"] <= 70000)
]
print(result)Pandas also provides between():
result = employees[
employees["Salary"].between(50000, 70000)
]
print(result)The between() method can make range-based filtering more readable.
Sorting is useful when you want to identify the highest or lowest values.
To sort employees by salary from lowest to highest:
sorted_df = employees.sort_values(
by="Salary"
)
print(sorted_df)For highest salary first:
sorted_df = employees.sort_values(
by="Salary",
ascending=False
)
print(sorted_df)You can sort using multiple columns:
sorted_df = employees.sort_values(
by=["Department", "Salary"],
ascending=[True, False]
)
print(sorted_df)Here, departments are sorted alphabetically, while salaries within each department are sorted from highest to lowest.
A common analytical requirement is to find the highest-performing records.
For example:
top_employees = employees.sort_values(
by="Salary",
ascending=False
).head(3)
print(top_employees)This returns the three highest-paid employees.
Similarly, the lowest-paid employees can be found using:
bottom_employees = employees.sort_values(
by="Salary"
).head(3)This combination of sort_values() and head() is extremely common in practical Data Analytics.
DataFrames become particularly powerful when you create new columns from existing data.
Suppose the employee salary is monthly salary and you want annual salary:
employees["Annual Salary"] = (
employees["Salary"] * 12
)
print(employees)You can calculate a hypothetical 10% performance bonus:
employees["Bonus"] = (
employees["Salary"] * 0.10
)And total compensation:
employees["Total Compensation"] = (
employees["Annual Salary"] +
employees["Bonus"]
)These calculations are vectorized, meaning Pandas performs the operation across the column efficiently without requiring you to manually write a Python loop for every row.
You can also create a category based on a numerical condition.
employees["Salary Level"] = "Standard"
employees.loc[
employees["Salary"] >= 60000,
"Salary Level"
] = "High"
print(employees)This creates a simple classification based on salary.
In a real project, such categories might represent customer segments, risk levels, performance groups, price ranges, or other business classifications.
Individual DataFrame columns can be analyzed using Series methods.
print(employees["Salary"].max())
print(employees["Salary"].min())
print(employees["Salary"].mean())
print(employees["Salary"].median())The mean represents the arithmetic average, while the median represents the middle value after sorting the observations.
Median can be particularly useful when a dataset contains extreme values that may strongly influence the mean.
For example, if most employees earn between ₹40,000 and ₹70,000 but one executive earns ₹500,000 per month, the mean may be substantially higher than what a typical employee earns. The median may provide a more representative central value.
You can use len() to count rows:
print(len(employees))You can also use:
print(employees.shape[0])For non-missing values in a particular column:
print(employees["Salary"].count())Remember that count() counts non-missing values, not necessarily all rows.
Consider a Data Analyst receiving a customer-sales dataset from a company. A sensible initial workflow might look like this:
df.head()
df.shape
df.info()
df.dtypes
df.isna().sum()
df.describe()After understanding the structure, the analyst might filter the required records:
df[df["Sales"] > 50000]Then sort them:
df.sort_values(
by="Sales",
ascending=False
)Then create an analytical field:
df["Profit Margin"] = (
df["Profit"] / df["Sales"]
) * 100Finally, the analyst can validate the results and prepare the dataset for visualization, reporting, or further analysis.
The important principle is that analysis should be systematic. Do not immediately start creating charts or conclusions before understanding the structure and quality of the underlying data.
Create the following DataFrame:
sales = pd.DataFrame({
"Product": [
"Laptop", "Mobile", "Tablet",
"Laptop", "Mobile", "Monitor"
],
"Region": [
"North", "South", "East",
"North", "West", "South"
],
"Sales": [
85000, 45000, 32000,
92000, 58000, 41000
],
"Quantity": [2, 5, 3, 2, 6, 4]
})Try to answer the following using Pandas:
Do not focus only on getting the code to run. Check whether the results make business sense. A Data Analyst's responsibility is not simply to execute Python commands but to validate and interpret the output.
Always inspect a dataset before analyzing it. Use head(), shape, info(), dtypes, and missing-value checks to understand what you are working with.
Use meaningful column names and maintain consistent naming conventions.
When filtering data, make the condition explicit and readable. Clear code is easier to validate and maintain.
When creating calculated columns, verify the formula before interpreting the result.
Do not delete missing or unusual records simply because they look inconvenient. First determine why they exist and whether removing them could affect the analysis.
Finally, always connect the calculation to the original analytical question. A technically correct calculation can still lead to a poor business decision if the wrong metric or population was analyzed.
dtypes shows the data type of each column.info() provides a compact structural summary.shape tells you the number of rows and columns.describe() provides descriptive statistics for numerical data.isna().sum() is useful for detecting missing values by column.between() simplifies range-based filtering.sort_values() sorts rows according to one or more columns.head() can be combined with sorting to identify top records.At this stage, you can create a DataFrame, inspect its structure, select the required data, filter records, sort information, calculate statistics, identify missing values, and create basic analytical columns.
In the next part, we will bring these skills together through a real-world DataFrame analysis workflow, including practical business questions, data-quality checks, interview questions, FAQs, and a complete lesson summary.
You have now learned how to create a Pandas DataFrame, inspect its structure, select rows and columns, filter records, sort values, identify missing data, calculate statistics, and create new columns. The final step is to combine these skills into a practical analytical workflow.
In professional Data Analytics, the objective is not simply to write Pandas code. The objective is to use data to answer a meaningful question accurately. A good workflow therefore moves from understanding the data to checking data quality, then performing analysis, and finally interpreting the results.
Consider the following sales dataset:
import pandas as pd
sales = pd.DataFrame({
"Order_ID": [1001, 1002, 1003, 1004, 1005, 1006, 1007, 1008],
"Product": [
"Laptop", "Mobile", "Tablet", "Laptop",
"Mobile", "Monitor", "Tablet", "Laptop"
],
"Region": [
"North", "South", "East", "North",
"West", "South", "East", "West"
],
"Quantity": [2, 5, 3, 1, 6, 4, 2, 3],
"Sales": [
130000, 125000, 96000, 72000,
150000, 72000, 64000, 195000
]
})
print(sales)The first step should be inspection:
print(sales.head())
print(sales.shape)
print(sales.info())Now we know how many records and columns exist and what type of information each column contains.
Instead of randomly calculating statistics, imagine that management asks these questions:
These questions give the analysis a clear purpose.
Total sales can be calculated using:
total_sales = sales["Sales"].sum()
print(total_sales)Average order value:
average_order = sales["Sales"].mean()
print(average_order)Highest individual transaction:
highest_sale = sales["Sales"].max()
print(highest_sale)To identify the complete record associated with the highest sale:
highest_order = sales.loc[
sales["Sales"].idxmax()
]
print(highest_order)This is an important analytical pattern. max() gives the value, while idxmax() helps locate the corresponding row.
Suppose management wants to understand the sales value generated per unit:
sales["Revenue Per Unit"] = (
sales["Sales"] / sales["Quantity"]
)
print(sales)Now the DataFrame contains a new analytical variable.
This demonstrates an important principle of Pandas: existing columns can be transformed into new business metrics using vectorized calculations.
Suppose the company defines a high-value order as one with sales greater than ₹100,000.
high_value = sales.loc[
sales["Sales"] > 100000
]
print(high_value)If management wants only the Order ID, Product, and Sales:
high_value = sales.loc[
sales["Sales"] > 100000,
["Order_ID", "Product", "Sales"]
]
print(high_value)This is a practical example of combining filtering with column selection.
To find the transaction with the highest sales:
best_transaction = sales.sort_values(
by="Sales",
ascending=False
).head(1)
print(best_transaction)This approach is easy to understand and can be extended to find the top five or top ten transactions:
top_five = sales.sort_values(
by="Sales",
ascending=False
).head(5)
print(top_five)Before presenting results, check whether important fields contain missing values:
print(sales.isna().sum())If every column returns zero, there are no missing values in this particular example.
In a real project, missing values may require additional investigation. An analyst should determine whether the missing information can be recovered, should be replaced according to a justified rule, or should remain missing.
Another important data-quality check is duplicate detection.
print(sales.duplicated())To count duplicate rows:
print(sales.duplicated().sum())This can prevent the same transaction from being accidentally counted more than once.
Duplicate detection should always be interpreted in context. Two rows with similar values are not automatically duplicates. The business definition of a unique record should determine what constitutes duplication.
Suppose you need only the information required for a management report:
report = sales[
["Order_ID", "Product", "Region", "Sales"]
].copy()
print(report)Using copy() can be useful when you intend to independently modify the selected DataFrame.
You can then sort the report:
report = report.sort_values(
by="Sales",
ascending=False
)Now the report is organized from the highest-value transaction to the lowest.
Suppose the analysis shows that Laptop orders have several of the highest transaction values. That observation does not automatically mean laptops are the most profitable product.
Why?
Sales revenue and profit are different metrics.
A product may generate high revenue but have a low profit margin. Another product may have lower revenue but significantly higher profitability.
This demonstrates an important E-E-A-T-oriented analytical principle: do not make claims that the available data cannot support.
If the dataset contains only Sales, you can discuss sales performance. You cannot confidently claim which product is most profitable without a Profit or Cost field.
1. Ignoring data quality
Running calculations without checking missing values, duplicates, and data types can produce misleading results.
2. Confusing correlation with explanation
If two columns appear related, that does not automatically establish why they are related.
3. Using the wrong metric
Total sales, average sales, quantity sold, revenue per unit, and profit answer different questions.
4. Modifying data without validation
After creating or changing a column, inspect the result and verify that the calculation is logically correct.
5. Removing records without justification
Deleting unusual records simply because they look strange can introduce bias. Investigate them first.
What is a Pandas DataFrame?
A DataFrame is a two-dimensional labeled data structure in Pandas consisting of rows and columns.
What is the difference between a Series and DataFrame?
A Series is one-dimensional, while a DataFrame is two-dimensional and can contain multiple columns.
How do you check the dimensions of a DataFrame?
df.shapeHow do you view the first five rows?
df.head()How do you inspect column data types?
df.dtypesHow do you get a structural summary?
df.info()How do you select one column?
df["Column_Name"]How do you select multiple columns?
df[["Column1", "Column2"]]What is loc?
loc is used primarily for label-based selection.
What is iloc?
iloc is used for integer-position-based selection.
How do you find missing values?
df.isna().sum()How do you sort a DataFrame?
df.sort_values(by="Column_Name")How do you create a calculated column?
df["New Column"] = df["Column1"] * df["Column2"]How do you find duplicate rows?
df.duplicated()Why is the Pandas DataFrame important for Data Analytics?
It provides a flexible structure for working with tabular data and supports selection, filtering, transformation, statistical analysis, cleaning, and preparation for visualization.
Can a DataFrame contain different data types?
Yes. Different columns can contain different types of data, such as numbers, text, Boolean values, and dates.
Should I always use loc instead of brackets?
No. Simple column selection with brackets is concise and appropriate in many situations. loc becomes particularly useful when you need precise row and column selection or conditional filtering.
Should I memorize all Pandas DataFrame methods?
No. Focus on understanding the most commonly used operations and the problems they solve. With practice, you will naturally become familiar with frequently used methods.
Is DataFrame knowledge required for Data Science?
It is highly useful. DataFrames are widely used for data preparation, exploratory analysis, feature engineering, and many other tasks in Python-based Data Science workflows.
Create a DataFrame containing at least 10 sales transactions with these columns:
Then complete the following analysis:
After completing the technical analysis, write three to five sentences explaining what the data tells you. This final interpretation is important because professional Data Analytics involves both technical execution and meaningful communication.
You have now completed Lesson 3: Pandas DataFrame Fundamentals.
A DataFrame is a two-dimensional labeled structure that is central to tabular data analysis in Pandas. You learned how to create DataFrames from dictionaries, lists, and record-based structures and how to understand rows, columns, indexes, and data types.
You learned how to select individual and multiple columns, use loc and iloc, filter records with Boolean conditions, work with multiple conditions, sort rows, create calculated columns, rename columns, remove unnecessary data, and inspect the overall structure of a dataset.
You also learned how to use info(), describe(), shape, dtypes, isna(), and duplicated() as part of a basic data-quality workflow.
Most importantly, you learned that Pandas should be used to answer analytical questions rather than simply execute commands. The quality of an analysis depends on understanding the data, choosing appropriate metrics, validating calculations, and avoiding conclusions that the available data cannot support.
With Series and DataFrame fundamentals now covered, you are ready to move into more advanced Pandas operations used in practical Data Analytics.