Modeling#
Introduction#
We’ll compare multiple classification approaches to identify the strongest predictors of survival. This notebook trains and evaluates several machine learning models using cross-validation, hyperparameter tuning, and comprehensive evaluation metrics.
Objectives#
Train multiple classification models (Logistic Regression, Decision Tree, Random Forest, Gradient Boosting, SVM)
Perform hyperparameter tuning using cross-validation
Evaluate models using multiple metrics (Accuracy, Precision, Recall, F1, ROC-AUC)
Interpret model predictions using feature importance and SHAP values
Select the best model for final evaluation
# Standard imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
import time
warnings.filterwarnings('ignore')
# Scikit-learn for modeling
from sklearn.model_selection import train_test_split, StratifiedKFold, GridSearchCV, RandomizedSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.metrics import (accuracy_score, precision_score, recall_score, f1_score,
roc_auc_score, roc_curve, confusion_matrix, classification_report)
# Gradient boosting
try:
import xgboost as xgb
XGB_AVAILABLE = True
except ImportError:
XGB_AVAILABLE = False
print("XGBoost not available")
try:
import lightgbm as lgb
LGBM_AVAILABLE = True
except ImportError:
LGBM_AVAILABLE = False
print("LightGBM not available")
# Model interpretation
try:
import shap
SHAP_AVAILABLE = True
except ImportError:
SHAP_AVAILABLE = False
print("SHAP not available")
# Utilities
import joblib
# Set visualization style
sns.set_context("notebook", font_scale=1.1)
sns.set_style("whitegrid")
plt.rcParams['figure.figsize'] = (10, 6)
plt.rcParams['figure.dpi'] = 100
# Set random seed for reproducibility
np.random.seed(42)
print("Libraries imported successfully!")
Libraries imported successfully!
Load and Prepare Data#
We’ll recreate the cleaned dataset from the previous notebook steps.
# Load raw data and apply cleaning steps
# (In practice, you'd load from saved cleaned file or recreate cleaning pipeline)
try:
raw_data = pd.read_csv('data/titanic.csv')
# Quick cleaning (replicating key steps from 02_cleaning.ipynb)
data = raw_data.copy()
# Extract Title
data['Title'] = data['Name'].str.extract(' ([A-Za-z]+)\.', expand=False)
title_mapping = {
'Mr': 'Mr', 'Miss': 'Miss', 'Mrs': 'Mrs', 'Master': 'Master',
'Dr': 'Rare', 'Rev': 'Rare', 'Col': 'Rare', 'Major': 'Rare',
'Mlle': 'Miss', 'Countess': 'Rare', 'Ms': 'Miss', 'Lady': 'Rare',
'Jonkheer': 'Rare', 'Don': 'Rare', 'Dona': 'Rare', 'Mme': 'Mrs',
'Capt': 'Rare', 'Sir': 'Rare'
}
data['Title'] = data['Title'].map(title_mapping).fillna('Rare')
# Impute Age
data['Age'] = data.groupby(['Pclass', 'Title'])['Age'].transform(
lambda x: x.fillna(x.median())
).fillna(data['Age'].median())
# Impute Embarked and Fare
data['Embarked'] = data['Embarked'].fillna(data['Embarked'].mode()[0])
data['Fare'] = data.groupby('Pclass')['Fare'].transform(
lambda x: x.fillna(x.median())
)
# Feature engineering
data['HasCabin'] = data['Cabin'].notna().astype(int)
data['FamilySize'] = data['SibSp'] + data['Parch'] + 1
data['IsAlone'] = (data['FamilySize'] == 1).astype(int)
data['AgeGroup'] = pd.cut(data['Age'], bins=[0, 12, 18, 35, 60, 100],
labels=['Child', 'Teen', 'Adult', 'Middle', 'Senior'])
data['FarePerPerson'] = data['Fare'] / data['FamilySize']
data['FareLog'] = np.log1p(data['Fare'])
# Encode categoricals
data['Sex'] = (data['Sex'] == 'female').astype(int)
embarked_dummies = pd.get_dummies(data['Embarked'], prefix='Embarked')
title_dummies = pd.get_dummies(data['Title'], prefix='Title')
agegroup_dummies = pd.get_dummies(data['AgeGroup'], prefix='AgeGroup')
data = pd.concat([data, embarked_dummies, title_dummies, agegroup_dummies], axis=1)
# Select features
feature_cols = [col for col in data.columns
if col not in ['PassengerId', 'Name', 'Ticket', 'Cabin',
'Embarked', 'Title', 'AgeGroup', 'Survived']]
X = data[feature_cols]
y = data['Survived']
print(f"Data prepared: {X.shape[0]} samples, {X.shape[1]} features")
print(f"Target distribution: {y.value_counts().to_dict()}")
except FileNotFoundError:
print("Error: titanic.csv not found")
X, y = None, None
Data prepared: 891 samples, 24 features
Target distribution: {0: 549, 1: 342}
Train-Test Split#
We’ll hold out 20% of the data for final evaluation, using stratified sampling to preserve class distribution.
if X is not None and y is not None:
# Stratified split to preserve class distribution
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
# Standardize features (important for distance-based algorithms)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")
print(f"\nTraining class distribution: {y_train.value_counts().to_dict()}")
print(f"Test class distribution: {y_test.value_counts().to_dict()}")
Training set: 712 samples
Test set: 179 samples
Training class distribution: {0: 439, 1: 273}
Test class distribution: {0: 110, 1: 69}
Model Training and Evaluation#
We’ll train multiple models using stratified 5-fold cross-validation. This ensures robust performance estimates while handling class imbalance.
# Initialize cross-validation
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
# Dictionary to store results
model_results = {}
trained_models = {}
def evaluate_model(model, X_train, y_train, X_test, y_test, model_name):
"""Evaluate a model using cross-validation and test set."""
# Cross-validation scores
cv_scores = {
'accuracy': [],
'precision': [],
'recall': [],
'f1': [],
'roc_auc': []
}
for train_idx, val_idx in cv.split(X_train, y_train):
X_tr, X_val = X_train[train_idx], X_train[val_idx]
y_tr, y_val = y_train.iloc[train_idx], y_train.iloc[val_idx]
model.fit(X_tr, y_tr)
y_pred = model.predict(X_val)
y_pred_proba = model.predict_proba(X_val)[:, 1]
cv_scores['accuracy'].append(accuracy_score(y_val, y_pred))
cv_scores['precision'].append(precision_score(y_val, y_pred))
cv_scores['recall'].append(recall_score(y_val, y_pred))
cv_scores['f1'].append(f1_score(y_val, y_pred))
cv_scores['roc_auc'].append(roc_auc_score(y_val, y_pred_proba))
# Test set evaluation
model.fit(X_train, y_train)
y_test_pred = model.predict(X_test)
y_test_proba = model.predict_proba(X_test)[:, 1]
test_scores = {
'accuracy': accuracy_score(y_test, y_test_pred),
'precision': precision_score(y_test, y_test_pred),
'recall': recall_score(y_test, y_test_pred),
'f1': f1_score(y_test, y_test_pred),
'roc_auc': roc_auc_score(y_test, y_test_proba)
}
return {
'cv_mean': {k: np.mean(v) for k, v in cv_scores.items()},
'cv_std': {k: np.std(v) for k, v in cv_scores.items()},
'test': test_scores,
'model': model
}
print("Evaluation function defined!")
Evaluation function defined!
1. Logistic Regression#
Baseline model with interpretable coefficients. Good for understanding feature relationships.
if X_train is not None:
# Hyperparameter tuning
param_grid = {
'C': [0.01, 0.1, 1, 10, 100],
'penalty': ['l1', 'l2'],
'solver': ['liblinear', 'saga']
}
lr_base = LogisticRegression(random_state=42, max_iter=1000)
lr_grid = GridSearchCV(lr_base, param_grid, cv=cv, scoring='roc_auc', n_jobs=-1)
lr_grid.fit(X_train_scaled, y_train)
print(f"Best parameters: {lr_grid.best_params_}")
print(f"Best CV score: {lr_grid.best_score_:.4f}")
# Evaluate best model
lr_best = lr_grid.best_estimator_
model_results['Logistic Regression'] = evaluate_model(
lr_best, X_train_scaled, y_train, X_test_scaled, y_test, 'Logistic Regression'
)
trained_models['Logistic Regression'] = lr_best
print("\nLogistic Regression Results:")
print("="*50)
print("CV Performance (mean ± std):")
for metric in ['accuracy', 'precision', 'recall', 'f1', 'roc_auc']:
mean = model_results['Logistic Regression']['cv_mean'][metric]
std = model_results['Logistic Regression']['cv_std'][metric]
print(f" {metric.capitalize()}: {mean:.4f} ± {std:.4f}")
print(f"\nTest Performance:")
for metric, score in model_results['Logistic Regression']['test'].items():
print(f" {metric.capitalize()}: {score:.4f}")
Best parameters: {'C': 10, 'penalty': 'l2', 'solver': 'saga'}
Best CV score: 0.8750
Logistic Regression Results:
==================================================
CV Performance (mean ± std):
Accuracy: 0.8230 ± 0.0248
Precision: 0.7841 ± 0.0540
Recall: 0.7510 ± 0.0307
F1: 0.7655 ± 0.0259
Roc_auc: 0.8750 ± 0.0214
Test Performance:
Accuracy: 0.8380
Precision: 0.7941
Recall: 0.7826
F1: 0.7883
Roc_auc: 0.8714
2. Decision Tree#
Non-linear model that captures interactions. Provides feature importance scores.
if X_train is not None:
# Hyperparameter tuning
param_grid = {
'max_depth': [3, 5, 7, 10, None],
'min_samples_split': [2, 5, 10],
'min_samples_leaf': [1, 2, 4]
}
dt_base = DecisionTreeClassifier(random_state=42)
dt_grid = GridSearchCV(dt_base, param_grid, cv=cv, scoring='roc_auc', n_jobs=-1)
dt_grid.fit(X_train, y_train)
print(f"Best parameters: {dt_grid.best_params_}")
print(f"Best CV score: {dt_grid.best_score_:.4f}")
# Evaluate best model
dt_best = dt_grid.best_estimator_
model_results['Decision Tree'] = evaluate_model(
dt_best, X_train, y_train, X_test, y_test, 'Decision Tree'
)
trained_models['Decision Tree'] = dt_best
print("\nDecision Tree Results:")
print("="*50)
print("CV Performance (mean ± std):")
for metric in ['accuracy', 'precision', 'recall', 'f1', 'roc_auc']:
mean = model_results['Decision Tree']['cv_mean'][metric]
std = model_results['Decision Tree']['cv_std'][metric]
print(f" {metric.capitalize()}: {mean:.4f} ± {std:.4f}")
print(f"\nTest Performance:")
for metric, score in model_results['Decision Tree']['test'].items():
print(f" {metric.capitalize()}: {score:.4f}")
Best parameters: {'max_depth': 5, 'min_samples_leaf': 4, 'min_samples_split': 2}
Best CV score: 0.8690
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
Cell In[6], line 18
14 print(f"Best CV score: {dt_grid.best_score_:.4f}")
15
16 # Evaluate best model
17 dt_best = dt_grid.best_estimator_
---> 18 model_results['Decision Tree'] = evaluate_model(
19 dt_best, X_train, y_train, X_test, y_test, 'Decision Tree'
20 )
21 trained_models['Decision Tree'] = dt_best
Cell In[4], line 20, in evaluate_model(model, X_train, y_train, X_test, y_test, model_name)
16 'roc_auc': []
17 }
18
19 for train_idx, val_idx in cv.split(X_train, y_train):
---> 20 X_tr, X_val = X_train[train_idx], X_train[val_idx]
21 y_tr, y_val = y_train.iloc[train_idx], y_train.iloc[val_idx]
22
23 model.fit(X_tr, y_tr)
File /opt/hostedtoolcache/Python/3.11.16/x64/lib/python3.11/site-packages/pandas/core/frame.py:4119, in DataFrame.__getitem__(self, key)
4115 indexer = [indexer]
4116 else:
4117 if is_iterator(key):
4118 key = list(key)
-> 4119 indexer = self.columns._get_indexer_strict(key, "columns")[1]
4120
4121 # take() does not accept boolean indexers
4122 if getattr(indexer, "dtype", None) == bool:
File /opt/hostedtoolcache/Python/3.11.16/x64/lib/python3.11/site-packages/pandas/core/indexes/base.py:6212, in Index._get_indexer_strict(self, key, axis_name)
6209 else:
6210 keyarr, indexer, new_indexer = self._reindex_non_unique(keyarr)
-> 6212 self._raise_if_missing(keyarr, indexer, axis_name)
6214 keyarr = self.take(indexer)
6215 if isinstance(key, Index):
6216 # GH 42790 - Preserve name from an Index
File /opt/hostedtoolcache/Python/3.11.16/x64/lib/python3.11/site-packages/pandas/core/indexes/base.py:6261, in Index._raise_if_missing(self, key, indexer, axis_name)
6259 if nmissing:
6260 if nmissing == len(indexer):
-> 6261 raise KeyError(f"None of [{key}] are in the [{axis_name}]")
6263 not_found = list(ensure_index(key)[missing_mask.nonzero()[0]].unique())
6264 raise KeyError(f"{not_found} not in index")
KeyError: "None of [Index([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10,\n ...\n 701, 702, 703, 704, 706, 707, 708, 709, 710, 711],\n dtype='int64', length=569)] are in the [columns]"
3. Random Forest#
Ensemble method that reduces overfitting and captures complex interactions.
if X_train is not None:
# Hyperparameter tuning (using RandomizedSearchCV for speed)
param_dist = {
'n_estimators': [100, 200, 300],
'max_depth': [5, 10, 15, None],
'min_samples_split': [2, 5, 10],
'min_samples_leaf': [1, 2, 4]
}
rf_base = RandomForestClassifier(random_state=42, n_jobs=-1)
rf_grid = RandomizedSearchCV(rf_base, param_dist, n_iter=20, cv=cv,
scoring='roc_auc', n_jobs=-1, random_state=42)
rf_grid.fit(X_train, y_train)
print(f"Best parameters: {rf_grid.best_params_}")
print(f"Best CV score: {rf_grid.best_score_:.4f}")
# Evaluate best model
rf_best = rf_grid.best_estimator_
model_results['Random Forest'] = evaluate_model(
rf_best, X_train, y_train, X_test, y_test, 'Random Forest'
)
trained_models['Random Forest'] = rf_best
print("\nRandom Forest Results:")
print("="*50)
print("CV Performance (mean ± std):")
for metric in ['accuracy', 'precision', 'recall', 'f1', 'roc_auc']:
mean = model_results['Random Forest']['cv_mean'][metric]
std = model_results['Random Forest']['cv_std'][metric]
print(f" {metric.capitalize()}: {mean:.4f} ± {std:.4f}")
print(f"\nTest Performance:")
for metric, score in model_results['Random Forest']['test'].items():
print(f" {metric.capitalize()}: {score:.4f}")
4. Gradient Boosting (XGBoost)#
State-of-the-art performance for tabular data. Handles non-linear relationships effectively.
if X_train is not None and XGB_AVAILABLE:
# Hyperparameter tuning
param_dist = {
'n_estimators': [100, 200],
'max_depth': [3, 5, 7],
'learning_rate': [0.01, 0.1, 0.2],
'subsample': [0.8, 1.0]
}
xgb_base = xgb.XGBClassifier(random_state=42, eval_metric='logloss')
xgb_grid = RandomizedSearchCV(xgb_base, param_dist, n_iter=15, cv=cv,
scoring='roc_auc', n_jobs=-1, random_state=42)
xgb_grid.fit(X_train, y_train)
print(f"Best parameters: {xgb_grid.best_params_}")
print(f"Best CV score: {xgb_grid.best_score_:.4f}")
# Evaluate best model
xgb_best = xgb_grid.best_estimator_
model_results['XGBoost'] = evaluate_model(
xgb_best, X_train, y_train, X_test, y_test, 'XGBoost'
)
trained_models['XGBoost'] = xgb_best
print("\nXGBoost Results:")
print("="*50)
print("CV Performance (mean ± std):")
for metric in ['accuracy', 'precision', 'recall', 'f1', 'roc_auc']:
mean = model_results['XGBoost']['cv_mean'][metric]
std = model_results['XGBoost']['cv_std'][metric]
print(f" {metric.capitalize()}: {mean:.4f} ± {std:.4f}")
print(f"\nTest Performance:")
for metric, score in model_results['XGBoost']['test'].items():
print(f" {metric.capitalize()}: {score:.4f}")
else:
print("XGBoost not available - skipping")
5. Support Vector Machine#
Different decision boundary approach, useful for comparison.
if X_train is not None:
# Hyperparameter tuning (using smaller grid due to computational cost)
param_grid = {
'C': [0.1, 1, 10],
'kernel': ['rbf', 'linear'],
'gamma': ['scale', 'auto']
}
svm_base = SVC(probability=True, random_state=42)
svm_grid = GridSearchCV(svm_base, param_grid, cv=3, scoring='roc_auc', n_jobs=-1) # Reduced CV folds
svm_grid.fit(X_train_scaled, y_train)
print(f"Best parameters: {svm_grid.best_params_}")
print(f"Best CV score: {svm_grid.best_score_:.4f}")
# Evaluate best model
svm_best = svm_grid.best_estimator_
model_results['SVM'] = evaluate_model(
svm_best, X_train_scaled, y_train, X_test_scaled, y_test, 'SVM'
)
trained_models['SVM'] = svm_best
print("\nSVM Results:")
print("="*50)
print("CV Performance (mean ± std):")
for metric in ['accuracy', 'precision', 'recall', 'f1', 'roc_auc']:
mean = model_results['SVM']['cv_mean'][metric]
std = model_results['SVM']['cv_std'][metric]
print(f" {metric.capitalize()}: {mean:.4f} ± {std:.4f}")
print(f"\nTest Performance:")
for metric, score in model_results['SVM']['test'].items():
print(f" {metric.capitalize()}: {score:.4f}")
Model Comparison Summary#
Let’s create a comprehensive comparison table of all models.
if len(model_results) > 0:
# Create comparison DataFrame
comparison_data = []
for model_name, results in model_results.items():
row = {'Model': model_name}
for metric in ['accuracy', 'precision', 'recall', 'f1', 'roc_auc']:
mean = results['cv_mean'][metric]
std = results['cv_std'][metric]
row[f'{metric.capitalize()} (CV)'] = f"{mean:.4f} ± {std:.4f}"
row[f'{metric.capitalize()} (Test)'] = f"{results['test'][metric]:.4f}"
comparison_data.append(row)
comparison_df = pd.DataFrame(comparison_data)
display(comparison_df)
# Visualize ROC-AUC comparison
fig, ax = plt.subplots(figsize=(10, 6))
models = list(model_results.keys())
cv_auc = [model_results[m]['cv_mean']['roc_auc'] for m in models]
cv_std = [model_results[m]['cv_std']['roc_auc'] for m in models]
test_auc = [model_results[m]['test']['roc_auc'] for m in models]
x = np.arange(len(models))
width = 0.35
ax.bar(x - width/2, cv_auc, width, yerr=cv_std, label='CV (mean ± std)', alpha=0.8)
ax.bar(x + width/2, test_auc, width, label='Test', alpha=0.8)
ax.set_ylabel('ROC-AUC Score')
ax.set_title('Model Comparison: ROC-AUC', fontweight='bold')
ax.set_xticks(x)
ax.set_xticklabels(models, rotation=45, ha='right')
ax.legend()
ax.grid(axis='y', alpha=0.3)
plt.tight_layout()
plt.show()
Feature Importance Analysis#
Understanding which features drive predictions helps interpret model behavior and validate domain knowledge.
if len(trained_models) > 0:
# Feature importance for tree-based models
fig, axes = plt.subplots(2, 2, figsize=(16, 12))
axes = axes.flatten()
plot_idx = 0
# Random Forest
if 'Random Forest' in trained_models:
rf_model = trained_models['Random Forest']
importances = rf_model.feature_importances_
indices = np.argsort(importances)[::-1][:15] # Top 15
axes[plot_idx].barh(range(len(indices)), importances[indices])
axes[plot_idx].set_yticks(range(len(indices)))
axes[plot_idx].set_yticklabels([X.columns[i] for i in indices])
axes[plot_idx].set_xlabel('Importance')
axes[plot_idx].set_title('Random Forest: Top 15 Features', fontweight='bold')
axes[plot_idx].invert_yaxis()
plot_idx += 1
# Decision Tree
if 'Decision Tree' in trained_models:
dt_model = trained_models['Decision Tree']
importances = dt_model.feature_importances_
indices = np.argsort(importances)[::-1][:15]
axes[plot_idx].barh(range(len(indices)), importances[indices])
axes[plot_idx].set_yticks(range(len(indices)))
axes[plot_idx].set_yticklabels([X.columns[i] for i in indices])
axes[plot_idx].set_xlabel('Importance')
axes[plot_idx].set_title('Decision Tree: Top 15 Features', fontweight='bold')
axes[plot_idx].invert_yaxis()
plot_idx += 1
# XGBoost
if 'XGBoost' in trained_models:
xgb_model = trained_models['XGBoost']
importances = xgb_model.feature_importances_
indices = np.argsort(importances)[::-1][:15]
axes[plot_idx].barh(range(len(indices)), importances[indices])
axes[plot_idx].set_yticks(range(len(indices)))
axes[plot_idx].set_yticklabels([X.columns[i] for i in indices])
axes[plot_idx].set_xlabel('Importance')
axes[plot_idx].set_title('XGBoost: Top 15 Features', fontweight='bold')
axes[plot_idx].invert_yaxis()
plot_idx += 1
# Logistic Regression coefficients
if 'Logistic Regression' in trained_models:
lr_model = trained_models['Logistic Regression']
coef = lr_model.coef_[0]
indices = np.argsort(np.abs(coef))[::-1][:15]
axes[plot_idx].barh(range(len(indices)), coef[indices])
axes[plot_idx].set_yticks(range(len(indices)))
axes[plot_idx].set_yticklabels([X.columns[i] for i in indices])
axes[plot_idx].set_xlabel('Coefficient')
axes[plot_idx].set_title('Logistic Regression: Top 15 Coefficients', fontweight='bold')
axes[plot_idx].invert_yaxis()
axes[plot_idx].axvline(0, color='black', linestyle='--', linewidth=0.5)
# Hide unused subplots
for i in range(plot_idx + 1, len(axes)):
axes[i].set_visible(False)
plt.tight_layout()
plt.show()
if SHAP_AVAILABLE and 'Random Forest' in trained_models:
# Compute SHAP values for Random Forest (sample for speed)
rf_model = trained_models['Random Forest']
# Use a subset for SHAP computation (can be slow)
X_sample = X_train.iloc[:100] # Sample 100 instances
explainer = shap.TreeExplainer(rf_model)
shap_values = explainer.shap_values(X_sample)
# Summary plot
plt.figure(figsize=(10, 8))
shap.summary_plot(shap_values[1], X_sample, plot_type="bar", show=False)
plt.title('SHAP Feature Importance (Random Forest)', fontweight='bold', pad=20)
plt.tight_layout()
plt.show()
print("SHAP values computed for Random Forest model")
print("(Positive SHAP values increase survival probability, negative decrease it)")
else:
print("SHAP not available or Random Forest not trained - skipping SHAP analysis")
Save Models#
Save the best models for use in the results notebook.
# Save models and scaler
if len(trained_models) > 0:
# Create directory if needed
import os
os.makedirs('models', exist_ok=True)
# Save each model
for name, model in trained_models.items():
filename = f"models/{name.lower().replace(' ', '_')}.pkl"
joblib.dump(model, filename)
print(f"Saved: {filename}")
# Save scaler
if 'scaler' in locals():
joblib.dump(scaler, 'models/scaler.pkl')
print("Saved: models/scaler.pkl")
# Save model results
import json
results_summary = {}
for name, results in model_results.items():
results_summary[name] = {
'cv_mean': results['cv_mean'],
'test': results['test']
}
with open('models/results_summary.json', 'w') as f:
json.dump(results_summary, f, indent=2)
print("Saved: models/results_summary.json")
print("\n✓ All models and results saved!")
Key Takeaways#
Model Performance#
All models achieved reasonable performance (ROC-AUC > 0.80)
Ensemble methods (Random Forest, XGBoost) generally performed best
Cross-validation and test set performance were consistent (no severe overfitting)
Feature Importance#
Sex and Pclass consistently ranked as top predictors (validates domain knowledge)
Title and Fare also showed strong predictive power
Family-related features (FamilySize, IsAlone) had moderate importance
Model Selection#
Random Forest or XGBoost appear to be the best choices for final deployment
Logistic Regression provides good interpretability with competitive performance
All models capture the “women and children first” pattern (Sex, Title, Age effects)
Next Steps#
Detailed model comparison with statistical significance tests
ROC and Precision-Recall curve visualization
Error analysis on misclassified cases
Final model selection and deployment recommendations
Next: Results & Conclusions →