Skip to main content

Data Selection and Filtering in pandas

Selecting and filtering data are essential tasks in data analysis, allowing you to extract meaningful subsets of your dataset for further analysis. Pandas provides powerful tools to accomplish these tasks efficiently. In this article, we'll explore how to select data by labels and positions, and how to filter data using conditions.

1. Selecting Data by Label​

Pandas allows you to select data by using labels, which refer to the index and column names. The primary method for label-based selection is .loc[].

1.1 Selecting a Single Row or Column​

You can select a single row or column using the .loc[] method by specifying the row label and/or column label.

import pandas as pd

# Sample DataFrame
data = {
    'Name': ['Alice', 'Bob', 'Charlie'],
    'Age': [25, 30, 35],
    'City': ['New York', 'Los Angeles', 'Chicago']
}
df = pd.DataFrame(data)

# Selecting a single row
row = df.loc[1]
print("Selected row:\n", row)

# Selecting a single column
column = df.loc[:, 'Age']
print("Selected column:\n", column)

1.2 Selecting Multiple Rows or Columns​

You can select multiple rows or columns by passing a list of labels.

# Selecting multiple rows
rows = df.loc[[0, 2]]
print("Selected rows:\n", rows)

# Selecting multiple columns
columns = df.loc[:, ['Name', 'City']]
print("Selected columns:\n", columns)

1.3 Slicing DataFrames with Labels​

Label-based slicing allows you to select a range of rows and/or columns.

# Slicing rows
row_slice = df.loc[0:1]
print("Row slice:\n", row_slice)

# Slicing rows and columns
subset = df.loc[0:1, 'Name':'Age']
print("Sliced DataFrame:\n", subset)

2. Selecting Data by Position​

When you need to select data by its position (i.e., row number or column number), you can use the .iloc[] method.

2.1 Selecting a Single Row or Column by Position​

You can select a single row or column by specifying its position.

# Selecting a single row by position
row = df.iloc[1]
print("Selected row by position:\n", row)

# Selecting a single column by position
column = df.iloc[:, 1]
print("Selected column by position:\n", column)

2.2 Selecting Multiple Rows or Columns by Position​

Similar to label-based selection, you can select multiple rows or columns by passing a list of positions.

# Selecting multiple rows by position
rows = df.iloc[[0, 2]]
print("Selected rows by position:\n", rows)

# Selecting multiple columns by position
columns = df.iloc[:, [0, 2]]
print("Selected columns by position:\n", columns)

2.3 Slicing DataFrames with Positions​

Position-based slicing allows you to select a range of rows and/or columns by their position.

# Slicing rows by position
row_slice = df.iloc[0:2]
print("Row slice by position:\n", row_slice)

# Slicing rows and columns by position
subset = df.iloc[0:2, 0:2]
print("Sliced DataFrame by position:\n", subset)

3. Filtering Data Using Conditions​

Filtering data using conditions is a powerful way to extract specific subsets of your data based on criteria.

3.1 Filtering Rows Based on a Condition​

You can filter rows based on a condition applied to one or more columns.

# Filtering rows where Age is greater than 30
filtered_df = df[df['Age'] > 30]
print("Filtered DataFrame where Age > 30:\n", filtered_df)

3.2 Combining Multiple Conditions​

You can combine multiple conditions using logical operators like & (and), | (or), and ~ (not).

# Filtering rows where Age is greater than 25 and City is not 'Chicago'
filtered_df = df[(df['Age'] > 25) & (df['City'] != 'Chicago')]
print("Filtered DataFrame with multiple conditions:\n", filtered_df)

3.3 Using np.where() for Conditional Selection​

The np.where() function allows you to apply a condition and return different values based on whether the condition is True or False.

import numpy as np

# Adding a column with conditional values
df['Age_Group'] = np.where(df['Age'] > 30, 'Senior', 'Junior')
print("DataFrame with conditional Age_Group column:\n", df)

4. Practical Examples of Data Selection and Filtering​

4.1 Selecting and Filtering in Real-World Scenarios​

Let's apply the concepts of selection and filtering to a more complex dataset. Consider a DataFrame that contains information about products in an inventory.

# Sample DataFrame with product information
inventory = {
    'Product': ['Laptop', 'Tablet', 'Smartphone', 'Monitor', 'Keyboard'],
    'Price': [1200, 300, 800, 150, 100],
    'Stock': [50, 100, 75, 200, 300]
}
inventory_df = pd.DataFrame(inventory)

# Selecting products priced above $500
expensive_products = inventory_df[inventory_df['Price'] > 500]
print("Products priced above $500:\n", expensive_products)

# Selecting products with low stock (less than 100)
low_stock_products = inventory_df[inventory_df['Stock'] < 100]
print("Products with stock less than 100:\n", low_stock_products)

5. Conclusion​

Data selection and filtering are crucial techniques for data analysis, enabling you to focus on the most relevant parts of your dataset. By mastering label-based and position-based selection, along with conditional filtering, you can efficiently extract and manipulate data in pandas. These skills are foundational for more complex data analysis tasks, including data cleaning, transformation, and aggregation, which we will explore in upcoming articles.