Skip to main content

Basic Operations with pandas DataFrames

Pandas DataFrames are powerful tools for data manipulation, offering a variety of operations to efficiently manage your data. In this article, we'll explore how to perform basic operations on DataFrames, such as adding, modifying, and deleting columns and rows, and handling missing data.

1. Adding Columns to a DataFrame​

Adding columns to a DataFrame is a common task, whether you're calculating new metrics or integrating additional data. You can add a new column by simply assigning a list, Series, or array to a new column name.

1.1 Adding a Column from a List​

import pandas as pd

# Sample DataFrame
data = {
    'Name': ['Alice', 'Bob', 'Charlie'],
    'Age': [25, 30, 35],
    'City': ['New York', 'Los Angeles', 'Chicago']
}
df = pd.DataFrame(data)

# Adding a new column from a list
df['Salary'] = [70000, 80000, 120000]
print("DataFrame with new Salary column:\n", df)

This will output:

      Name  Age         City  Salary
0    Alice   25     New York   70000
1      Bob   30  Los Angeles   80000
2  Charlie   35     Chicago  120000

1.2 Adding a Column from a Calculation​

You can create new columns based on calculations involving other columns.

# Adding a column with calculated values
df['Salary after Tax'] = df['Salary'] * 0.7
print("DataFrame with Salary after Tax column:\n", df)

2. Modifying Data in a DataFrame​

Modifying data within a DataFrame is straightforward. You can update values in an entire column, specific rows, or specific cells.

2.1 Updating an Entire Column​

You can modify the values in a column by assigning new values to it.

# Updating the Age column by adding 1 year to each age
df['Age'] = df['Age'] + 1
print("DataFrame with updated Age column:\n", df)

2.2 Updating Specific Rows or Cells​

To update specific rows or cells, use .loc[] to target the specific data you want to change.

# Updating a specific cell
df.loc[0, 'City'] = 'San Francisco'
print("DataFrame with updated City for Alice:\n", df)

# Updating multiple rows based on a condition
df.loc[df['Name'] == 'Bob', 'Salary'] = 85000
print("DataFrame with updated Salary for Bob:\n", df)

3. Deleting Columns and Rows​

Deleting unnecessary data from your DataFrame can help streamline your dataset and reduce memory usage.

3.1 Dropping Columns​

You can remove columns from your DataFrame using the .drop() method.

# Dropping the 'Salary after Tax' column
df = df.drop('Salary after Tax', axis=1)
print("DataFrame after dropping a column:\n", df)

3.2 Dropping Rows​

Similarly, you can remove rows from your DataFrame by specifying the row index.

# Dropping the first row
df = df.drop(0, axis=0)
print("DataFrame after dropping the first row:\n", df)

4. Handling Missing Data​

Missing data is a common issue in real-world datasets. Pandas provides several methods to handle missing data effectively.

4.1 Identifying Missing Data​

You can detect missing data in your DataFrame using the .isnull() method, which returns a DataFrame of the same shape with Boolean values indicating the presence of missing data.

# Sample DataFrame with missing data
data_with_nan = {
    'Name': ['Alice', 'Bob', 'Charlie'],
    'Age': [25, None, 35],
    'City': ['New York', 'Los Angeles', None]
}
df_nan = pd.DataFrame(data_with_nan)

# Identifying missing data
print("Missing data in DataFrame:\n", df_nan.isnull())

4.2 Filling Missing Data​

You can fill missing values with a specified value using the .fillna() method.

# Filling missing values with a specific value
df_nan_filled = df_nan.fillna('Unknown')
print("DataFrame with filled missing data:\n", df_nan_filled)

4.3 Dropping Missing Data​

Alternatively, you can drop any rows or columns with missing data using .dropna().

# Dropping rows with any missing data
df_nan_dropped = df_nan.dropna()
print("DataFrame after dropping rows with missing data:\n", df_nan_dropped)

5. Conclusion​

Understanding how to perform basic operations on pandas DataFrames is crucial for effective data manipulation. Adding, modifying, and deleting columns and rows, as well as handling missing data, are common tasks that you'll encounter frequently in data science projects. Mastering these operations will provide a solid foundation for more advanced data manipulation techniques, which we'll explore in upcoming articles.