Skip to main content

XGBoost with PyTorch Example

In this article, we will implement XGBoost alongside PyTorch using the Pima Indians Diabetes dataset. This example will show how PyTorch can be used to preprocess data and how to train an XGBoost model using the processed data.

1. Dataset Overview​

The Pima Indians Diabetes dataset is a binary classification dataset that consists of several medical features used to predict whether a patient has diabetes (1) or not (0). The dataset contains features such as Pregnancies, Glucose, BloodPressure, BMI, and others.

2. Importing Libraries​

We will import PyTorch for handling the data pipeline and XGBoost for building and training the model. Additionally, we will use pandas to handle data loading.

# Importing necessary libraries
import torch
from torch.utils.data import Dataset, DataLoader
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix
from xgboost import XGBClassifier

3. Loading and Preprocessing the Dataset with PyTorch​

We will load the dataset using pandas and create a custom PyTorch dataset class to handle batching, shuffling, and transformations.

# Load the dataset
url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv"
column_names = ['Pregnancies', 'Glucose', 'BloodPressure', 'SkinThickness', 'Insulin', 'BMI',
                'DiabetesPedigreeFunction', 'Age', 'Outcome']

# Read the dataset into a pandas DataFrame
data = pd.read_csv(url, names=column_names)

# Split data into features (X) and target (y)
X = data.iloc[:, :-1].values  # All columns except the last one (features)
y = data.iloc[:, -1].values   # The last column (target variable)

# Train-test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Next, we will create a PyTorch Dataset class to handle our data.

class DiabetesDataset(Dataset):
    def __init__(self, features, targets):
        self.features = torch.tensor(features, dtype=torch.float32)
        self.targets = torch.tensor(targets, dtype=torch.float32)

    def __len__(self):
        return len(self.features)

    def __getitem__(self, idx):
        return self.features[idx], self.targets[idx]

# Create PyTorch datasets
train_dataset = DiabetesDataset(X_train, y_train)
test_dataset = DiabetesDataset(X_test, y_test)

# DataLoader for batching and shuffling
BATCH_SIZE = 32
train_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=BATCH_SIZE, shuffle=False)

4. Training the XGBoost Model​

Since XGBoost requires NumPy arrays, we will extract the data from the PyTorch DataLoader and convert it back to NumPy arrays before passing it to XGBoost for training.

# Convert PyTorch DataLoader data to NumPy arrays for XGBoost
X_train_np = X_train
y_train_np = y_train

# Initialize the XGBoost classifier
xgb_model = XGBClassifier(learning_rate=0.1, n_estimators=100, max_depth=3, random_state=42)

# Train the XGBoost model
xgb_model.fit(X_train_np, y_train_np)

5. Model Evaluation​

After training, we will evaluate the XGBoost model using the test set and compute the accuracy score and confusion matrix.

# Make predictions on the test set
y_pred = xgb_model.predict(X_test)

# Evaluate the model
accuracy = accuracy_score(y_test, y_pred)
conf_matrix = confusion_matrix(y_test, y_pred)

print(f"Accuracy: {accuracy:.2f}")
print("Confusion Matrix:")
print(conf_matrix)

6. Integrating PyTorch with XGBoost (Advanced)​

If you want to combine PyTorch and XGBoost in a more integrated pipeline (for instance, if you want to use PyTorch for more complex transformations or feature engineering), you can use PyTorch to manage the data and transformations, and then pass the preprocessed data to XGBoost for training.

Here’s an example where you can define a custom transformation in PyTorch and pass the transformed data to XGBoost:

# Define a custom PyTorch transformation (e.g., normalization)
class NormalizeTransform:
    def __call__(self, sample):
        features, target = sample
        features = (features - torch.mean(features)) / torch.std(features)
        return features, target

# Apply transformation during dataset creation
train_dataset_transformed = DiabetesDataset(X_train, y_train)
train_loader_transformed = DataLoader(train_dataset_transformed, batch_size=BATCH_SIZE, shuffle=True)

# Convert transformed PyTorch data to NumPy arrays for XGBoost
X_train_transformed = []
y_train_transformed = []
for batch_features, batch_targets in train_loader_transformed:
    X_train_transformed.extend(batch_features.numpy())
    y_train_transformed.extend(batch_targets.numpy())

X_train_transformed = np.array(X_train_transformed)
y_train_transformed = np.array(y_train_transformed)

# Train XGBoost with transformed data
xgb_model.fit(X_train_transformed, y_train_transformed)

In this way, you can leverage PyTorch's data preprocessing capabilities while using XGBoost for model training and prediction.

7. Hyperparameter Tuning with PyTorch and XGBoost​

For advanced tuning, we can use GridSearchCV with PyTorch's preprocessed data to fine-tune the hyperparameters of the XGBoost model.

from sklearn.model_selection import GridSearchCV

# Define the parameter grid for GridSearchCV
param_grid = {
    'max_depth': [3, 5, 7],
    'learning_rate': [0.01, 0.1, 0.2],
    'n_estimators': [100, 200, 300]
}

# Perform GridSearchCV to find the best parameters
grid_search = GridSearchCV(estimator=XGBClassifier(random_state=42), param_grid=param_grid, cv=3, scoring='accuracy', verbose=1)
grid_search.fit(X_train_np, y_train_np)

# Output the best parameters and accuracy score
print(f"Best Parameters: {grid_search.best_params_}")
print(f"Best Accuracy: {grid_search.best_score_:.2f}")

8. Conclusion​

In this article, we demonstrated how to integrate XGBoost with PyTorch by:

  • Preprocessing data using PyTorch’s Dataset and DataLoader classes.
  • Converting PyTorch data back to NumPy arrays for XGBoost training.
  • Training and evaluating an XGBoost model on the Pima Indians Diabetes dataset.
  • Performing advanced hyperparameter tuning with GridSearchCV.
  • Leveraging PyTorch for custom transformations, such as normalization, before passing the data to XGBoost.

This approach allows you to use the flexible data handling capabilities of PyTorch while benefiting from the power and efficiency of XGBoost for predictive modeling. Let me know if you'd like to proceed with the next section or make any adjustments!