Skip to main content

Common Mistakes & Best Practices for CatBoost

1. Hiding categories behind numeric codes​

Pass categorical names or indices through cat_features, ideally with a DataFrame whose feature names persist. Numeric category codes without that declaration are split as ordered numbers. Keep the same columns, values, and types at inference; represent categorical missing values consistently.

Do not perform global target encoding first. CatBoost's ordered statistics are designed to reduce self-target leakage, but external preprocessing can reintroduce it.

2. Using the test set for the overfitting detector​

eval_set, early_stopping_rounds, and use_best_model select the final tree count. Their data must come from the training partition. Keep a separate final test set or evaluate the full selection process with outer cross-validation.

3. Assuming ordered boosting prevents every leak​

Ordered methods address specific training-statistic biases. They cannot repair future-derived fields, duplicates across folds, patient/customer overlap, post-outcome variables, or feature engineering fitted before splitting. Choose group-aware or temporal boundaries first.

4. Copying depth and iteration defaults​

depth, learning_rate, iterations, l2_leaf_reg, sampling, and categorical-statistic settings interact. Symmetric-tree depth grows leaf count exponentially. Search bounded ranges supported by the chosen grow policy and CPU/GPU mode.

5. Treating imbalance settings as calibrated probabilities​

Class weights alter the loss. Evaluate ranking and probability calibration separately, and select operational thresholds using validation data and explicit costs. Do not report accuracy alone on an imbalanced target.

6. Mixing CPU and GPU results silently​

Supported parameters, numerical behavior, and speed differ by processing unit. Record task_type, devices, threads, package version, and hardware. Reproducible seeds do not guarantee bit-for-bit equality across platforms.

7. Overinterpreting feature importance and SHAP​

Importance explains the fitted predictive model under a particular method and background, not a causal intervention. Correlated features can share or redirect credit. Calculate expensive loss-based importance with appropriate held-out data, and assess stability.

8. Ignoring installation constraints​

CatBoost 1.2.10's Python package supports 64-bit CPython. GPU support and driver requirements are platform-specific. CatBoost is Apache-2.0 licensed, but optional visualization and GPU workflows add dependencies that should be documented separately.

Checklist​

  • Preserve feature names and explicit cat_features.
  • Split by deployment unit before any target-aware work.
  • Stop on validation, evaluate once on test.
  • Log objective, evaluation metric, best_iteration_, seed, and processing unit.
  • Compare against fold-safe one-hot and simple-model baselines.