Skip to main content

Introduction to Time Series Data in pandas

Time series data is ubiquitous in many fields, from finance to environmental science, and pandas provides powerful tools for handling and analyzing such data. This article introduces the basics of working with time series data in pandas, including datetime indexing, resampling, and applying rolling windows.

1. Understanding Time Series Data​

Time series data consists of observations or measurements taken at specific time intervals, such as daily stock prices, monthly sales figures, or hourly temperature readings.

1.1 Characteristics of Time Series Data​

  • Temporal Ordering: Time series data is ordered by time.
  • Frequency: Observations can be taken at regular intervals (e.g., hourly, daily, monthly) or irregular intervals.
  • Stationarity: A time series is stationary if its statistical properties (mean, variance) do not change over time.

2. Working with Datetime in pandas​

Pandas makes it easy to work with datetime data. You can convert columns to datetime format and set them as the index to facilitate time series analysis.

2.1 Converting to Datetime​

You can convert a column containing date strings to a datetime format using pd.to_datetime().

import pandas as pd

# Sample DataFrame with date strings
data = {
    'Date': ['2024-01-01', '2024-02-01', '2024-03-01'],
    'Value': [100, 200, 300]
}
df = pd.DataFrame(data)

# Converting the 'Date' column to datetime
df['Date'] = pd.to_datetime(df['Date'])
print("DataFrame with datetime:\n", df)

2.2 Setting the Datetime Index​

Setting a datetime column as the index allows you to leverage pandas' time series functionalities.

# Setting the 'Date' column as the index
df.set_index('Date', inplace=True)
print("DataFrame with datetime index:\n", df)

2.3 Generating Date Ranges​

Pandas can generate a range of dates using pd.date_range(), which is useful for creating time series data.

# Generating a date range
date_range = pd.date_range(start='2024-01-01', periods=10, freq='D')
print("Generated date range:\n", date_range)

3. Resampling Time Series Data​

Resampling is the process of converting a time series to a different frequency, which can involve aggregating or interpolating the data.

3.1 Downsampling​

Downsampling reduces the frequency of the data by aggregating it over a specified time period, such as converting daily data to monthly data.

# Sample time series data
date_range = pd.date_range(start='2024-01-01', periods=100, freq='D')
data = pd.Series(range(100), index=date_range)

# Downsampling to monthly frequency using sum
monthly_data = data.resample('M').sum()
print("Monthly downsampled data:\n", monthly_data)

3.2 Upsampling​

Upsampling increases the frequency of the data, which often requires filling in missing values.

# Upsampling to hourly frequency
hourly_data = data.resample('H').ffill()  # Forward fill to fill missing values
print("Hourly upsampled data:\n", hourly_data.head(10))

3.3 Custom Resampling with apply()​

You can apply custom functions during resampling for more control over how data is aggregated or interpolated.

# Custom resampling using a lambda function
custom_resample = data.resample('W').apply(lambda x: x.mean() + 5)
print("Custom resampled data:\n", custom_resample)

4. Rolling and Expanding Windows​

Rolling and expanding operations allow you to apply functions over a sliding or expanding window of your time series data, which is useful for smoothing or identifying trends.

4.1 Rolling Windows​

A rolling window applies a function to a subset of data defined by a window size, such as calculating a moving average.

# Calculating a 7-day rolling mean
rolling_mean = data.rolling(window=7).mean()
print("7-day rolling mean:\n", rolling_mean.head(10))

4.2 Expanding Windows​

An expanding window includes all prior data points up to the current point in the calculation.

# Calculating an expanding mean
expanding_mean = data.expanding().mean()
print("Expanding mean:\n", expanding_mean.head(10))

5. Shifting and Lagging Data​

Shifting or lagging time series data is a common operation in time series analysis, particularly for calculating differences or creating lagged features.

5.1 Shifting Data​

Shifting moves data forward or backward in time by a specified number of periods.

# Shifting data by one day
shifted_data = data.shift(1)
print("Data shifted by one day:\n", shifted_data.head(10))

5.2 Calculating Differences​

You can calculate the difference between consecutive data points, which is useful for identifying changes over time.

# Calculating the difference between consecutive data points
difference = data.diff()
print("Difference between consecutive data points:\n", difference.head(10))

6. Time Series Analysis Applications​

Time series analysis is essential in various fields, such as finance, economics, and environmental science. Let’s look at a basic application.

6.1 Example: Moving Average in Stock Prices​

Consider a time series of daily stock prices. You can calculate the moving average to smooth out short-term fluctuations and highlight longer-term trends.

# Sample stock price data
stock_prices = pd.Series([150, 152, 153, 155, 157, 160, 162, 165], index=pd.date_range('2024-01-01', periods=8))

# Calculating the 3-day moving average
moving_average = stock_prices.rolling(window=3).mean()
print("3-day moving average of stock prices:\n", moving_average)

7. Conclusion​

Time series data is prevalent in many domains, and pandas provides a comprehensive suite of tools for handling and analyzing such data. By mastering datetime indexing, resampling, rolling windows, and shifting data, you’ll be well-equipped to perform effective time series analysis. In the next article, we'll explore efficient data operations in pandas to optimize your data processing workflows.