Skip to main content

Handling Missing Data in pandas

In real-world datasets, missing data is a common issue that can lead to inaccurate analysis or biased results if not handled properly. Pandas provides robust tools to detect, fill, and drop missing data, ensuring your datasets are clean and ready for analysis. In this article, we'll explore the various methods for handling missing data in pandas.

1. Identifying Missing Data​

Before you can handle missing data, you need to identify where it exists in your DataFrame. Pandas uses NaN (Not a Number) to represent missing values.

1.1 Using .isnull() and .notnull()​

The .isnull() method returns a DataFrame of the same shape with Boolean values indicating where the missing data (NaN) is located. The .notnull() method does the opposite, identifying where data is present.

import pandas as pd

# Sample DataFrame with missing data
data = {
    'Name': ['Alice', 'Bob', 'Charlie', 'David'],
    'Age': [25, None, 35, 28],
    'City': ['New York', 'Los Angeles', None, 'Chicago']
}
df = pd.DataFrame(data)

# Identifying missing data
missing_data = df.isnull()
print("Missing data in DataFrame:\n", missing_data)

# Identifying where data is not missing
not_missing_data = df.notnull()
print("Data not missing in DataFrame:\n", not_missing_data)

1.2 Checking for Missing Data in Columns​

You can quickly check if any columns have missing data using the .isnull().sum() method, which sums the number of missing values per column.

# Summing missing values in each column
missing_data_summary = df.isnull().sum()
print("Summary of missing data per column:\n", missing_data_summary)

2. Filling Missing Data​

Filling missing data is one strategy to handle NaN values, ensuring that your dataset remains usable.

2.1 Filling with a Specific Value​

You can fill missing values with a specific value, such as 0, the mean, or a placeholder string, using the .fillna() method.

# Filling missing values with a placeholder
df_filled = df.fillna('Unknown')
print("DataFrame with filled missing data:\n", df_filled)

2.2 Forward and Backward Fill​

Pandas allows you to propagate the last valid observation forward to fill missing data (.ffill()) or backward to fill from the next valid observation (.bfill()).

# Forward fill (propagate the last valid observation)
df_ffill = df.fillna(method='ffill')
print("DataFrame after forward fill:\n", df_ffill)

# Backward fill (propagate the next valid observation)
df_bfill = df.fillna(method='bfill')
print("DataFrame after backward fill:\n", df_bfill)

2.3 Filling with a Statistic (Mean, Median, Mode)​

You can fill missing values with a statistical measure, such as the mean, median, or mode of the column.

# Filling missing 'Age' with the mean value
mean_age = df['Age'].mean()
df['Age'] = df['Age'].fillna(mean_age)
print("DataFrame with 'Age' column filled by mean:\n", df)

3. Dropping Missing Data​

In some cases, it may be appropriate to drop rows or columns with missing data entirely.

3.1 Dropping Rows with Missing Data​

You can drop any rows that contain missing data using the .dropna() method.

# Dropping rows with any missing data
df_dropped_rows = df.dropna()
print("DataFrame after dropping rows with missing data:\n", df_dropped_rows)

3.2 Dropping Columns with Missing Data​

Similarly, you can drop entire columns that contain missing data.

# Dropping columns with any missing data
df_dropped_columns = df.dropna(axis=1)
print("DataFrame after dropping columns with missing data:\n", df_dropped_columns)

3.3 Dropping Rows or Columns Based on a Threshold​

Pandas allows you to drop rows or columns only if a certain number of non-missing values are present, using the thresh parameter in .dropna().

# Dropping rows that have fewer than 2 non-missing values
df_thresh = df.dropna(thresh=2)
print("DataFrame after applying threshold for dropping rows:\n", df_thresh)

4. Interpolating Missing Data​

Interpolation is a method of estimating unknown values by using the known data points. Pandas provides an interpolate() method that can fill missing data based on interpolation.

4.1 Linear Interpolation​

Linear interpolation is the most common method, which assumes that the change between two points is linear.

# Interpolating missing values linearly
df_interpolated = df.interpolate()
print("DataFrame after linear interpolation:\n", df_interpolated)

5. Conclusion​

Handling missing data is a critical step in data preprocessing, as missing values can significantly impact the quality and reliability of your analysis. Pandas provides versatile methods for identifying, filling, dropping, and interpolating missing data, allowing you to clean your datasets effectively. Mastering these techniques will help you maintain data integrity and ensure that your analyses are based on complete and accurate datasets. In the next article, we'll dive into more advanced data manipulation techniques with pandas.