Titanic Documentation

Titanic Competition Solution

This documentation details the feature engineering pipeline and stacking ensemble used to achieve a top-tier score in the Kaggle Titanic competition.

0.79665
Public Leaderboard Accuracy
839 / 16,563
Competition Rank (Top 5%)
0.84736
Local Stacking CV Score
Complete ML Pipeline

Complete ML Pipeline - End-to-end workflow from data preprocessing to final predictions.

1. Feature Engineering (Script 1)

The first script (process_pipeline_v3) focuses on maximizing information extraction from text fields and group dynamics. The pipeline applies transformations to both Train and Test sets simultaneously to ensure consistency.

Feature Engineering Pipeline

Figure 1: Feature Engineering Pipeline - Data flow from raw CSVs to encoded feature matrices.

Core Feature Groups

Imputation Strategy

Age: Imputed using Medians grouped by Title + Pclass.

Fare: Imputed using Medians grouped by Pclass.

Embarked: Mode imputation.

WomanOrChild (WOC)

A composite binary feature capturing the "Women and Children First" policy.

WOC = (Sex == Female) | (Age <= 16)
Fare Per Person (FarePP)

The raw Fare is often for a shared ticket. We normalize this to find the individual cost.

FarePP = Fare / Max(FamilySize, TicketFreq)
Binning & Interaction

FareBin: Quantile-based binning (Sextiles) of FarePP.

AgeBin: Fixed width bins (0-16, 16-32, etc.).

Pclass_Sex: String interaction (e.g., "1_male").

View Complete Feature Engineering Code (preprocess.py) 
import pandas as pd
import numpy as np
import os
import warnings

warnings.filterwarnings('ignore')

# ========================================================================
# 1. HELPER FUNCTIONS (Imputation and Titles)
# ========================================================================

def calculate_imputation_params(df):
    """Calculates imputation parameters ONLY from the training data."""
    params = {}

    # Embarked: Mode
    try:
        params['embarked_mode'] = df['Embarked'].mode()[0]
    except IndexError:
        params['embarked_mode'] = 'S' # Fallback

    # Fare: Median grouped by Pclass
    params['fare_medians'] = df.groupby('Pclass')['Fare'].median()

    # Age: Requires titles first
    df_copy = df.copy()
    df_copy['Title'] = df_copy['Name'].str.extract(' ([A-Za-z]+)\.', expand=False)
    df_copy = standardize_titles(df_copy)

    params['age_medians'] = df_copy.groupby(['Title', 'Pclass'])['Age'].median()
    params['age_medians_pclass'] = df_copy.groupby('Pclass')['Age'].median()

    return params

def standardize_titles(df):
    """Standardizes and groups rare titles."""
    rare_titles = ['Dr', 'Rev', 'Col', 'Major', 'Capt', 'Jonkheer', 'Sir',
                   'Lady', 'Don', 'Countess', 'Dona']
    df['Title'] = df['Title'].replace(rare_titles, 'Rare')
    df['Title'] = df['Title'].replace(['Mlle', 'Ms'], 'Miss')
    df['Title'] = df['Title'].replace('Mme', 'Mrs')
    return df

# ========================================================================
# 2. ADVANCED FEATURE ENGINEERING (Strict FSR and WOC)
# ========================================================================

def feature_engineer_v3(df, target=None):
    """Applies feature engineering, including the Strict FSR definition."""

    # 1. Title Extraction
    df['Title'] = df['Name'].str.extract(' ([A-Za-z]+)\.', expand=False)
    df = standardize_titles(df)

    # 2. Family Features
    df['FamilySize'] = df['SibSp'] + df['Parch'] + 1
    df['IsAlone'] = (df['FamilySize'] == 1).astype(int)
    df['FamilySizeBin'] = pd.cut(df['FamilySize'], bins=[0, 1, 4, 15],
                                  labels=['Alone', 'Small', 'Large'])

    # 3. Deck Extraction
    df['Deck'] = df['Cabin'].apply(lambda x: x[0] if pd.notnull(x) else 'U')
    df['Deck'] = df['Deck'].replace('T', 'U')

    # 4. Ticket Frequency
    ticket_counts = df['Ticket'].value_counts()
    df['Ticket_Freq'] = df['Ticket'].map(ticket_counts)

    # 5. Strict Family Survival Rate (FSR)
    if target is not None:
        df['Survived_temp'] = target

    df['LastName'] = df['Name'].str.split(',').str[0]
    df['FSR'] = 0.5  # Initialize to neutral

    for group_type in ['Ticket', 'LastName']:
        for identifier, group in df.groupby(group_type):
            if (len(group) > 1) and (not group['Survived_temp'].isnull().all()):

                for index, row in group.iterrows():
                    if group_type == 'LastName' and df.loc[index, 'FSR'] != 0.5:
                        continue

                    others_survival = group.drop(index)['Survived_temp'].dropna()

                    if len(others_survival) > 0:
                        if others_survival.max() == 0:
                            df.loc[index, 'FSR'] = 0
                        elif others_survival.min() == 1:
                            df.loc[index, 'FSR'] = 1

    df = df.drop(['LastName', 'Survived_temp'], axis=1, errors='ignore')
    return df

# ========================================================================
# 3. IMPUTATION AND TRANSFORMATION
# ========================================================================

def impute_and_transform_v3(df, params):
    """Handles imputation and subsequent transformations."""

    # 1. Embarked
    df['Embarked'] = df['Embarked'].fillna(params['embarked_mode'])

    # 2. Fare (Impute BEFORE calculating FarePP)
    if df['Fare'].isnull().any():
        df['Fare'] = df['Fare'].fillna(df['Pclass'].map(params['fare_medians']))

    # 3. Calculate Fare Per Person (FarePP)
    df['GroupSize'] = df[['FamilySize', 'Ticket_Freq']].max(axis=1)
    df['FarePP'] = df['Fare'] / df['GroupSize'].replace(0, 1)

    # 4. Age Imputation
    if df['Age'].isnull().any():
        df_copy = df.copy()
        for (title, pclass), median_val in params['age_medians'].items():
            mask = (df_copy['Title'] == title) & \
                   (df_copy['Pclass'] == pclass) & \
                   (df_copy['Age'].isnull())
            if not pd.isna(median_val):
                df_copy.loc[mask, 'Age'] = median_val

        if df_copy['Age'].isnull().any():
            df_copy['Age'] = df_copy['Age'].fillna(
                df_copy['Pclass'].map(params['age_medians_pclass']))
        df = df_copy

    # 5. Binning and Interactions
    df['AgeBin'] = pd.cut(df['Age'], bins=[0, 16, 32, 48, 64, 100], labels=False)

    # WomanOrChild Feature (WOC)
    df['IsChild'] = (df['Age'] <= 16).astype(int)
    df['Sex'] = df['Sex'].map({'male': 0, 'female': 1}).astype(int)
    df['WomanOrChild'] = ((df['Sex'] == 1) | (df['IsChild'] == 1)).astype(int)

    # FarePP Bins
    try:
        df['FareBin'] = pd.qcut(df['FarePP'], 6, labels=False, duplicates='drop')
    except ValueError:
        df['FareBin'] = pd.cut(df['FarePP'],
                               bins=[-0.001, 7.75, 8.66, 13.0, 26.0, 50.0, 600],
                               labels=False)

    # 6. Pclass and Sex interaction
    df['Pclass_Sex'] = df['Pclass'].astype(str) + '_' + df['Sex'].astype(str)

    return df

def encode_and_drop(df):
    """Applies One-Hot Encoding and drops unnecessary columns."""

    categorical_features = ['Pclass', 'Embarked', 'Title', 'Deck', 'AgeBin',
                           'FareBin', 'FamilySizeBin', 'Pclass_Sex']
    df = pd.get_dummies(df, columns=categorical_features, drop_first=True)

    columns_to_drop = ['Name', 'Ticket', 'Cabin', 'SibSp', 'Parch', 'Age',
                       'Fare', 'FarePP', 'FamilySize', 'Ticket_Freq',
                       'GroupSize', 'IsChild']
    columns_to_drop_existing = [col for col in columns_to_drop if col in df.columns]
    df = df.drop(columns_to_drop_existing, axis=1)
    return df

# ========================================================================
# 4. MAIN PIPELINE
# ========================================================================

def process_pipeline_v3(train_file='train.csv', test_file='test.csv',
                       output_suffix='_v3_stack'):

    if not os.path.exists(train_file) or not os.path.exists(test_file):
        print(f"Error: Missing input files.")
        return

    df_train = pd.read_csv(train_file)
    df_test = pd.read_csv(test_file)

    imputation_params = calculate_imputation_params(df_train.copy())

    target = df_train['Survived']
    test_ids = df_test['PassengerId']

    df_train_features = df_train.drop(['PassengerId', 'Survived'], axis=1)
    df_test_features = df_test.drop('PassengerId', axis=1)

    train_len = len(df_train_features)
    df_combined = pd.concat([df_train_features, df_test_features],
                           axis=0, ignore_index=True)

    combined_target = pd.Series(np.nan, index=df_combined.index)
    combined_target.iloc[:train_len] = target.values

    print("Applying V3 feature engineering...")
    df_combined = feature_engineer_v3(df_combined, target=combined_target)

    print("Applying imputation and transformation...")
    df_combined = impute_and_transform_v3(df_combined, imputation_params)

    print("Applying encoding...")
    df_combined = encode_and_drop(df_combined)

    df_train_processed = df_combined.iloc[:train_len].copy()
    df_test_processed = df_combined.iloc[train_len:].copy()

    df_train_processed.insert(0, 'Survived', target.values)
    df_test_processed.insert(0, 'PassengerId', test_ids.values)

    train_output = f'train_processed{output_suffix}.csv'
    test_output = f'test_processed{output_suffix}.csv'

    df_train_processed.to_csv(train_output, index=False)
    df_test_processed.to_csv(test_output, index=False)

    print(f"Processed data saved to {train_output} and {test_output}")
    print(f"Final features shape: {df_train_processed.shape}")

if __name__ == '__main__':
    process_pipeline_v3(train_file='train.csv', test_file='test.csv')

⭐ Key Feature: Strict Family Survival Rate (FSR)

This is the most impactful feature in the pipeline. It exploits the assumption that families (or ticket groups) often shared the same fate.

The Logic:

  • Groups are identified by Ticket (Strong link) and LastName (Weak link).
  • The algorithm looks at other members of the group (excluding the current passenger to prevent self-leakage).

The Strict Definition:

Let \( S_{others} \) be the set of survival labels for other members in the group.

$$ FSR = \begin{cases} 0 & \text{if } \max(S_{others}) = 0 \text{ (Everyone else died)} \\ 1 & \text{if } \min(S_{others}) = 1 \text{ (Everyone else survived)} \\ 0.5 & \text{Otherwise (Unknown or Mixed)} \end{cases} $$
View FSR Code Snippet 
# Check if group > 1 person AND has known outcomes
if (len(group) > 1) and (not group['Survived_temp'].isnull().all()):

    for index, row in group.iterrows():
        # Get survival outcomes of others in the group
        others_survival = group.drop(index)['Survived_temp'].dropna()

        if len(others_survival) > 0:
            if others_survival.max() == 0:
                df.loc[index, 'FSR'] = 0  # Everyone else died
            elif others_survival.min() == 1:
                df.loc[index, 'FSR'] = 1  # Everyone else survived

2. Model Architecture (Script 2)

The solution uses a 2-Level Stacking Ensemble. Level 0 consists of four diverse gradient boosting/tree models optimized via Optuna. Level 1 is a simple linear meta-learner.

High-Level Pipeline Workflow

Pipeline Workflow - End-to-end execution flow from data loading and hyperparameter optimization to stacking ensemble assembly and final prediction generation.

Stacking Ensemble Architecture

Figure 2: Stacking Ensemble Architecture - Four optimized Level 0 base estimators feeding out-of-fold predictions to the Level 1 meta-estimator (StandardScaler + Logistic Regression).

Hyperparameter Optimization Strategy

A key technique is the integration of Optuna with manual cross-validation loops to enable early stopping for gradient boosting models. This approach prevents overfitting during the tuning phase.

Hyperparameter Optimization Strategy

Figure 3: Hyperparameter Optimization Process - Optuna suggests parameters which are evaluated using 10-fold CV with validation set monitoring to prevent overfitting during tuning.

Level 0: Base Models (Optimized)

Models were tuned using 50 trials of Optuna TPE optimization.

Model Best CV Score Key Characteristics
CatBoost 0.86752 Highest performer. Handles categorical features well.
XGBoost 0.85630 Hist-gradient method, aggressive depth tuning.
Random Forest 0.85516 Bagging approach, provides diversity against boosting models.
LightGBM 0.85292 Fast training, optimized for leaf-wise growth.

Level 1: Meta-Learner

Algorithm: Logistic Regression (Ridge, C=0.1)

Input: Class Probabilities from Level 0 models.

Preprocessing: StandardScaler applied to probabilities before regression.

View Complete Model Training Code (train.py) 
import pandas as pd
import numpy as np
import optuna
import os
import warnings
import time

from sklearn.ensemble import RandomForestClassifier, StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
import xgboost as xgb
import lightgbm as lgb
from catboost import CatBoostClassifier

# --- Configuration ---
N_SPLITS = 10
N_TRIALS = 50
SEED = 42
TARGET_VARIABLE = 'Survived'
SUFFIX = '_v3_stack'
TRAIN_FILE = f'train_processed{SUFFIX}.csv'
TEST_FILE = f'test_processed{SUFFIX}.csv'
OUTPUT_DIR = f'stacking_output{SUFFIX}'
EARLY_STOPPING_ROUNDS = 50
FINAL_N_ESTIMATORS = 800

warnings.filterwarnings("ignore")
optuna.logging.set_verbosity(optuna.logging.WARNING)

if not os.path.exists(OUTPUT_DIR):
    os.makedirs(OUTPUT_DIR)

def load_data():
    """Loads data, ensuring all features are numeric."""
    if not os.path.exists(TRAIN_FILE):
        print(f"File {TRAIN_FILE} not found.")
        return None, None

    df = pd.read_csv(TRAIN_FILE)
    X = df.drop(TARGET_VARIABLE, axis=1)
    y = df[TARGET_VARIABLE]

    for col in X.columns:
        if X[col].dtype == bool:
            X[col] = X[col].astype(int)

    print(f"Data loaded. Shape: {X.shape}")
    return X, y

# ==================================================================
# OPTIMIZATION OBJECTIVES
# ==================================================================

def objective_xgb(trial, X, y):
    param = {
        'objective': 'binary:logistic',
        'eval_metric': 'logloss',
        'verbosity': 0,
        'tree_method': 'hist',
        'seed': SEED,
        'n_estimators': 2000,
        'early_stopping_rounds': EARLY_STOPPING_ROUNDS,
        'max_depth': trial.suggest_int('max_depth_xgb', 3, 8),
        'learning_rate': trial.suggest_float('learning_rate_xgb', 0.005, 0.1, log=True),
        'min_child_weight': trial.suggest_int('min_child_weight_xgb', 1, 10),
        'subsample': trial.suggest_float('subsample_xgb', 0.6, 1.0),
        'colsample_bytree': trial.suggest_float('colsample_bytree_xgb', 0.6, 1.0),
        'gamma': trial.suggest_float('gamma_xgb', 1e-8, 1.0, log=True),
        'reg_alpha': trial.suggest_float('reg_alpha_xgb', 1e-8, 10.0, log=True),
        'reg_lambda': trial.suggest_float('reg_lambda_xgb', 1e-8, 10.0, log=True),
    }

    cv = StratifiedKFold(n_splits=N_SPLITS, shuffle=True, random_state=SEED)
    cv_scores = []

    for train_idx, val_idx in cv.split(X, y):
        X_train, X_val = X.iloc[train_idx], X.iloc[val_idx]
        y_train, y_val = y.iloc[train_idx], y.iloc[val_idx]

        model = xgb.XGBClassifier(**param)
        model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)
        preds = model.predict(X_val)
        cv_scores.append(accuracy_score(y_val, preds))

    return np.mean(cv_scores)

def objective_lgbm(trial, X, y):
    param = {
        'objective': 'binary',
        'metric': 'binary_logloss',
        'verbosity': -1,
        'boosting_type': 'gbdt',
        'random_state': SEED,
        'n_estimators': 2000,
        'learning_rate': trial.suggest_float('learning_rate_lgbm', 0.005, 0.1, log=True),
        'num_leaves': trial.suggest_int('num_leaves_lgbm', 8, 64),
        'max_depth': trial.suggest_int('max_depth_lgbm', 3, 9),
        'min_child_samples': trial.suggest_int('min_child_samples_lgbm', 5, 50),
        'subsample': trial.suggest_float('subsample_lgbm', 0.6, 1.0),
        'colsample_bytree': trial.suggest_float('colsample_bytree_lgbm', 0.6, 1.0),
        'reg_alpha': trial.suggest_float('reg_alpha_lgbm', 1e-8, 10.0, log=True),
        'reg_lambda': trial.suggest_float('reg_lambda_lgbm', 1e-8, 10.0, log=True),
    }

    cv = StratifiedKFold(n_splits=N_SPLITS, shuffle=True, random_state=SEED)
    cv_scores = []

    for train_idx, val_idx in cv.split(X, y):
        X_train, X_val = X.iloc[train_idx], X.iloc[val_idx]
        y_train, y_val = y.iloc[train_idx], y.iloc[val_idx]

        model = lgb.LGBMClassifier(**param)
        callbacks = [lgb.early_stopping(EARLY_STOPPING_ROUNDS, verbose=False)]
        model.fit(X_train, y_train, eval_set=[(X_val, y_val)], callbacks=callbacks)
        preds = model.predict(X_val)
        cv_scores.append(accuracy_score(y_val, preds))

    return np.mean(cv_scores)

def objective_catboost(trial, X, y):
    param = {
        'iterations': 2000,
        'learning_rate': trial.suggest_float('learning_rate_cb', 0.01, 0.1, log=True),
        'depth': trial.suggest_int('depth_cb', 3, 8),
        'l2_leaf_reg': trial.suggest_float('l2_leaf_reg_cb', 1, 10),
        'border_count': trial.suggest_int('border_count_cb', 32, 255),
        'loss_function': 'Logloss',
        'eval_metric': 'Accuracy',
        'verbose': 0,
        'random_seed': SEED,
        'od_type': 'Iter',
        'od_wait': EARLY_STOPPING_ROUNDS
    }

    cv = StratifiedKFold(n_splits=N_SPLITS, shuffle=True, random_state=SEED)
    cv_scores = []

    for train_idx, val_idx in cv.split(X, y):
        X_train, X_val = X.iloc[train_idx], X.iloc[val_idx]
        y_train, y_val = y.iloc[train_idx], y.iloc[val_idx]

        model = CatBoostClassifier(**param)
        model.fit(X_train, y_train, eval_set=(X_val, y_val),
                 use_best_model=True, verbose=False)
        preds = model.predict(X_val)
        cv_scores.append(accuracy_score(y_val, preds))

    return np.mean(cv_scores)

def objective_rf(trial, X, y):
    param = {
        'n_estimators': trial.suggest_int('n_estimators_rf', 100, 800),
        'max_depth': trial.suggest_int('max_depth_rf', 5, 15),
        'min_samples_split': trial.suggest_int('min_samples_split_rf', 2, 20),
        'min_samples_leaf': trial.suggest_int('min_samples_leaf_rf', 1, 10),
        'max_features': trial.suggest_float('max_features_rf', 0.4, 1.0),
        'criterion': trial.suggest_categorical('criterion_rf', ['gini', 'entropy']),
        'random_state': SEED,
        'n_jobs': -1
    }

    model = RandomForestClassifier(**param)
    cv = StratifiedKFold(n_splits=N_SPLITS, shuffle=True, random_state=SEED)
    score = cross_val_score(model, X, y, cv=cv, scoring='accuracy', n_jobs=-1)
    return score.mean()

# ==========================================
# STACKING PIPELINE
# ==========================================
def run_stacking_pipeline():
    X, y = load_data()
    if X is None: return None, None

    studies = {}
    objectives = {
        'RF': objective_rf,
        'LGBM': objective_lgbm,
        'XGB': objective_xgb,
        'CatBoost': objective_catboost
    }

    print("\n--- Optimizing Base Models (Level 0) ---")
    for model_name, objective_func in objectives.items():
        print(f"\nOptimizing {model_name}...")
        study = optuna.create_study(direction='maximize',
                                    sampler=optuna.samplers.TPESampler(seed=SEED))
        study.optimize(lambda t: objective_func(t, X, y),
                      n_trials=N_TRIALS, show_progress_bar=True)

        studies[model_name] = study
        print(f"Best {model_name} Accuracy: {study.best_value:.5f}")

    print("\n--- Initializing Base Models with Best Parameters ---")

    def get_params(study, suffix):
        return {k.replace(f'_{suffix}', ''): v
                for k, v in study.best_params.items() if k.endswith(f'_{suffix}')}

    params_rf = get_params(studies['RF'], 'rf')
    rf_final = RandomForestClassifier(**params_rf, random_state=SEED, n_jobs=-1)

    params_lgbm = get_params(studies['LGBM'], 'lgbm')
    lgbm_final = lgb.LGBMClassifier(**params_lgbm, objective='binary',
                                    verbosity=-1, random_state=SEED,
                                    n_estimators=FINAL_N_ESTIMATORS)

    params_xgb = get_params(studies['XGB'], 'xgb')
    params_xgb.pop('early_stopping_rounds', None)
    xgb_final = xgb.XGBClassifier(**params_xgb, objective='binary:logistic',
                                  eval_metric='logloss', seed=SEED,
                                  n_estimators=FINAL_N_ESTIMATORS)

    params_cb = get_params(studies['CatBoost'], 'cb')
    cb_final = CatBoostClassifier(**params_cb, random_seed=SEED,
                                  verbose=0, iterations=FINAL_N_ESTIMATORS)

    print("\n--- Defining and Training Stacker ---")

    base_estimators = [
        ('rf', rf_final),
        ('lgbm', lgbm_final),
        ('xgb', xgb_final),
        ('catboost', cb_final)
    ]

    meta_model = make_pipeline(
        StandardScaler(),
        LogisticRegression(random_state=SEED, C=0.1, solver='liblinear')
    )

    stacker = StackingClassifier(
        estimators=base_estimators,
        final_estimator=meta_model,
        cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=SEED),
        stack_method='predict_proba',
        n_jobs=-1,
        passthrough=False
    )

    stacker.fit(X, y)

    print("\n--- Evaluating Stacker (Nested CV) ---")
    cv_eval = StratifiedKFold(n_splits=5, shuffle=True, random_state=SEED+1)
    stacker_score = cross_val_score(stacker, X, y, cv=cv_eval,
                                    scoring='accuracy', n_jobs=-1).mean()
    print(f"\n*** Final Stacker CV Accuracy: {stacker_score:.5f} ***")

    return stacker, X.columns

def generate_predictions_v3(stacker_model, train_cols):
    if not os.path.exists(TEST_FILE):
        print("Test file not found.")
        return

    print("\n--- Generating Predictions ---")
    df_test = pd.read_csv(TEST_FILE)
    passenger_ids = df_test['PassengerId']
    X_test = df_test.drop('PassengerId', axis=1, errors='ignore')

    missing_cols = set(train_cols) - set(X_test.columns)
    if missing_cols:
        for c in missing_cols:
            X_test[c] = 0

    X_test = X_test[train_cols]

    for col in X_test.columns:
        if X_test[col].dtype == bool:
            X_test[col] = X_test[col].astype(int)

    final_preds = stacker_model.predict(X_test)

    submission = pd.DataFrame({'PassengerId': passenger_ids,
                              'Survived': final_preds})
    path = os.path.join(OUTPUT_DIR, f"submission_stacking{SUFFIX}.csv")
    submission.to_csv(path, index=False)

    print(f"Submission saved to: {path}")

if __name__ == "__main__":
    if os.path.exists(TRAIN_FILE):
        start_time = time.time()
        stacker_model, cols = run_stacking_pipeline()
        if stacker_model:
            generate_predictions_v3(stacker_model, cols)
        print(f"\nTotal time: {time.time() - start_time:.2f}s")

3. Training Logs & Output

Below is the actual output from the production run. Note the high consistency between the base models.

Data loaded. Shape: (891, 35) --- Optimizing Base Models (Level 0) --- Optimizing RF... Best trial: 3. Best value: 0.855156 Best RF Accuracy: 0.85516 Optimizing LGBM... Best trial: 10. Best value: 0.852921 Best LGBM Accuracy: 0.85292 Optimizing XGB... Best trial: 39. Best value: 0.856305 Best XGB Accuracy: 0.85630 Optimizing CatBoost... Best trial: 41. Best value: 0.867516 Best CatBoost Accuracy: 0.86752 --- Initializing Base Models with Best Parameters --- --- Defining and Training Stacker --- --- Evaluating Stacker (Nested CV - may take time) --- *** Final Stacker CV Accuracy: 0.84736 *** --- Generating Predictions ---