Building Predictive Models

Python
import pandas as pd
from sklearn.model_selection import train_test_split, StratifiedKFold
from catboost import CatBoostClassifier
import optuna
from sklearn.metrics import roc_auc_score
import shap

# 1. Load & split
df = pd.read_csv("clients.csv")
X, y = df.drop(columns=["default"]), df["default"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.15, stratify=y, random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(
    X_train, y_train, test_size=0.176, stratify=y_train, random_state=42
)  # → 70/15/15 overall

# 2. Baseline model
baseline = CatBoostClassifier(verbose=0, random_state=42)
baseline.fit(X_train, y_train, cat_features=["category_col"])
baseline_auc = roc_auc_score(y_val, baseline.predict_proba(X_val)[:, 1])
print(f"Baseline AUC: {baseline_auc:.3f}")

# 3. Hyperparameter tuning
def objective(trial):
    params = {
        "depth": trial.suggest_int("depth", 4, 10),
        "learning_rate": trial.suggest_float("learning_rate", 0.01, 0.3, log=True),
        "l2_leaf_reg": trial.suggest_float("l2_leaf_reg", 1, 10),
        "iterations": trial.suggest_int("iterations", 200, 1000),
    }
    model = CatBoostClassifier(**params, verbose=0, random_state=42)
    model.fit(X_train, y_train, cat_features=["category_col"])
    preds = model.predict_proba(X_val)[:, 1]
    return roc_auc_score(y_val, preds)

study = optuna.create_study(direction="maximize")
study.study_optimize = study.optimize(objective, n_trials=50)

# 4. Final model + interpretation
best_model = CatBoostClassifier(**study.best_params, verbose=0, random_state=42)
best_model.fit(X_train, y_train, cat_features=["category_col"])
explainer = shap.TreeExplainer(best_model)
shap_values = explainer.shap_values(X_test)

Progress:

• Step 1: EDA — distributions, missing values, class balance, correlations
• Step 2: Preprocessing — impute missing values, scale/normalize numeric features, encode categoricals
• Step 3: Split data 70/15/15 (train/val/test), stratify on target for classification
• Step 4: Train baseline model (no tuning) and pick primary metric
• Step 5: Hyperparameter search (Optuna/GridSearch) on train, score on validation
• Step 6: Cross-validate best config (stratified k-fold, k=5) to confirm stability
• Step 7: Evaluate on held-out test set once, final check for overfitting
• Step 8: Interpret via SHAP/LIME, analyze error cases
• Step 9: Log experiment (MLflow/W&B), package model for deployment

Metric selection by task:

• Imbalanced classification → AUC-ROC, PR-AUC, F1
• Balanced classification → Accuracy, F1
• Regression → RMSE, MAE, R²
• Business-critical → cost-weighted custom metric

Overfitting checks:

• Train metric >> validation metric → overfitting signal
• Learning curves: gap between train/val loss growing with model complexity
• Cross-validation std across folds too high → unstable model
• Use regularization (L2, early stopping, max_depth limits) when gap detected