Stacking and unstacking in Pandas are the useful techniques for reshaping DataFrames to extract more information in different ways. It works efficiently with multi-level indices also. Whether it’s compressing columns into row levels or expanding rows into columns, these operations are crucial for handling complex datasets.
The Pandas library provides two main methods for these operations −
In this tutorial, we will learn about stacking and unstacking techniques in Pandas along with practical examples, including handling missing data.
Stacking in Pandas is a process of compressing a DataFrame columns into rows. The DataFrame.stack() method in Pandas is used for stacking the levels from column to index. This method pivots a level of column labels (possibly hierarchical) into row labels, and returns a new DataFrame or Series with a multi-level index.
Following example uses the df.stack() method for pivoting the columns into the row index.
import pandas as pd
import numpy as np
# Create MultiIndex
tuples = [["x", "x", "y", "y", "", "f", "z", "z"],
["1", "2", "1", "2", "1", "2", "1", "2"]]
index = pd.MultiIndex.from_arrays(tuples, names=["first", "second"])
# Create a DataFrame
df = pd.DataFrame(np.random.randn(8, 2), index=index, columns=["A", "B"])
# Display the input DataFrame
print('Input DataFrame:\n', df)
# Stack columns
stacked = df.stack()
print('Output Reshaped DataFrame:\n', stacked)
Input DataFrame:
A B
first second
x 1 0.596485 -1.356041
2 -1.091407 0.246216
y 1 0.499328 -1.346817
2 -0.893557 0.014678
1 -0.059916 0.106597
f 2 -0.315096 -0.950424
z 1 1.050350 -1.744569
2 -0.255863 0.539803
Output Reshaped DataFrame:
first second
x 1 A 0.596485
B -1.356041
2 A -1.091407
B 0.246216
y 1 A 0.499328
B -1.346817
2 A -0.893557
B 0.014678
1 A -0.059916
B 0.106597
f 2 A -0.315096
B -0.950424
z 1 A 1.050350
B -1.744569
2 A -0.255863
B 0.539803
dtype: float64
Here, the stack() method pivots the columns A and B into the index, compressing the DataFrame into a long format.
Unstacking reverses the stacking operation by moving the row index level back to the columns. The Pandas DataFrame.unstack() method is used to pivot a level of the row index to become a column, which is useful for converting a long-format DataFrame into a wide format.
The following example demonstrates the working of the df.unstack() method for unstacking a DataFrame.
import pandas as pd
import numpy as np
# Create MultiIndex
tuples = [["x", "x", "y", "y", "", "f", "z", "z"],
["1", "2", "1", "2", "1", "2", "1", "2"]]
index = pd.MultiIndex.from_arrays(tuples, names=["first", "second"])
# Create a DataFrame
df = pd.DataFrame(np.random.randn(8, 2), index=index, columns=["A", "B"])
# Display the input DataFrame
print('Input DataFrame:\n', df)
# Unstack the DataFrame
unstacked = df.unstack()
print('Output Reshaped DataFrame:\n', unstacked)
Input DataFrame:
A B
first second
x 1 -0.407537 -0.957010
2 0.045479 0.789849
y 1 0.751488 -0.474536
2 -1.043122 -0.015152
1 -0.133349 1.094900
f 2 1.681111 2.480652
z 1 0.283679 0.769553
2 -2.034907 0.301275
Output Reshaped DataFrame:
A B
second 1 2 1 2
first
-0.133349 NaN 1.094900 NaN
f NaN 1.681111 NaN 2.480652
x -0.407537 0.045479 -0.957010 0.789849
y 0.751488 -1.043122 -0.474536 -0.015152
z 0.283679 -2.034907 0.769553 0.301275
Unstacking can produce missing values when the reshaped DataFrame has unequal label sets in subgroups. Pandas handles these missing values with NaN by default, but you can specify a custom fill value.
This example demonstrates how to handle missing values when unstacking a DataFrame.
import pandas as pd
import numpy as np
# Create Data
index = pd.MultiIndex.from_product([["bar", "baz", "foo", "qux"], ["one", "two"]],
names=["first", "second"])
columns = pd.MultiIndex.from_tuples([("A", "cat"), ("B", "dog"),
("B", "cat"), ("A", "dog")],
names=["exp", "animal"])
df = pd.DataFrame(np.random.randn(8, 4), index=index, columns=columns)
# Create a DataFrame
df3 = df.iloc[[0, 1, 4, 7], [1, 2]]
print(df3)
# Unstack the DataFrame
unstacked = df3.unstack()
# Display the Unstacked DataFrame
print("Unstacked DataFrame without Filling:\n", unstacked)
unstacked_filled = df3.unstack(fill_value=1)
print("Unstacked DataFrame with Filling:\n", unstacked_filled)
exp B
animal dog cat
first second
bar one -0.556587 -0.157084
two 0.109060 0.856019
foo one -1.034260 1.548955
qux two -0.644370 -1.871248
Unstacked DataFrame without Filling:
exp B
animal dog cat
second one two one two
first
bar -0.556587 0.10906 -0.157084 0.856019
foo -1.034260 NaN 1.548955 NaN
qux NaN -0.64437 NaN -1.871248
Unstacked DataFrame with Filling:
exp B
animal dog cat
second one two one two
first
bar -0.556587 0.10906 -0.157084 0.856019
foo -1.034260 1.00000 1.548955 1.000000
qux 1.000000 -0.64437 1.000000 -1.871248