Results & Conclusions#
Introduction#
Comparing model performance reveals insights about both the data and algorithmic trade-offs. This notebook provides a comprehensive evaluation of all models, including visualizations, statistical comparisons, error analysis, and final recommendations.
Objectives#
Compare all models side-by-side with comprehensive metrics
Visualize ROC and Precision-Recall curves
Analyze confusion matrices and error patterns
Perform statistical significance testing
Provide final model recommendations and conclusions
# Standard imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
import json
warnings.filterwarnings('ignore')
# Scikit-learn for evaluation
from sklearn.metrics import (roc_curve, auc, precision_recall_curve,
confusion_matrix, classification_report)
from sklearn.model_selection import train_test_split
# Statistical testing
from scipy import stats
# Model loading
import joblib
# Set visualization style
sns.set_context("notebook", font_scale=1.1)
sns.set_style("whitegrid")
plt.rcParams['figure.figsize'] = (10, 6)
plt.rcParams['figure.dpi'] = 100
# Set random seed for reproducibility
np.random.seed(42)
print("Libraries imported successfully!")
Libraries imported successfully!
Load Models and Data#
Load the trained models and prepare test data for final evaluation.
# Load data (replicating cleaning steps)
try:
raw_data = pd.read_csv('data/titanic.csv')
# Quick cleaning (same as modeling notebook)
data = raw_data.copy()
data['Title'] = data['Name'].str.extract(' ([A-Za-z]+)\.', expand=False)
title_mapping = {
'Mr': 'Mr', 'Miss': 'Miss', 'Mrs': 'Mrs', 'Master': 'Master',
'Dr': 'Rare', 'Rev': 'Rare', 'Col': 'Rare', 'Major': 'Rare',
'Mlle': 'Miss', 'Countess': 'Rare', 'Ms': 'Miss', 'Lady': 'Rare',
'Jonkheer': 'Rare', 'Don': 'Rare', 'Dona': 'Rare', 'Mme': 'Mrs',
'Capt': 'Rare', 'Sir': 'Rare'
}
data['Title'] = data['Title'].map(title_mapping).fillna('Rare')
data['Age'] = data.groupby(['Pclass', 'Title'])['Age'].transform(
lambda x: x.fillna(x.median())
).fillna(data['Age'].median())
data['Embarked'] = data['Embarked'].fillna(data['Embarked'].mode()[0])
data['Fare'] = data.groupby('Pclass')['Fare'].transform(
lambda x: x.fillna(x.median())
)
data['HasCabin'] = data['Cabin'].notna().astype(int)
data['FamilySize'] = data['SibSp'] + data['Parch'] + 1
data['IsAlone'] = (data['FamilySize'] == 1).astype(int)
data['AgeGroup'] = pd.cut(data['Age'], bins=[0, 12, 18, 35, 60, 100],
labels=['Child', 'Teen', 'Adult', 'Middle', 'Senior'])
data['FarePerPerson'] = data['Fare'] / data['FamilySize']
data['FareLog'] = np.log1p(data['Fare'])
data['Sex'] = (data['Sex'] == 'female').astype(int)
embarked_dummies = pd.get_dummies(data['Embarked'], prefix='Embarked')
title_dummies = pd.get_dummies(data['Title'], prefix='Title')
agegroup_dummies = pd.get_dummies(data['AgeGroup'], prefix='AgeGroup')
data = pd.concat([data, embarked_dummies, title_dummies, agegroup_dummies], axis=1)
feature_cols = [col for col in data.columns
if col not in ['PassengerId', 'Name', 'Ticket', 'Cabin',
'Embarked', 'Title', 'AgeGroup', 'Survived']]
X = data[feature_cols]
y = data['Survived']
# Same split as modeling notebook
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
# Load scaler if available
try:
scaler = joblib.load('models/scaler.pkl')
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
except:
print("Scaler not found - will scale on the fly if needed")
scaler = None
print(f"Data loaded: {X_test.shape[0]} test samples")
except FileNotFoundError:
print("Error: titanic.csv not found")
X_test, y_test = None, None
Scaler not found - will scale on the fly if needed
Data loaded: 179 test samples
Model Comparison Table#
Comprehensive side-by-side comparison of all models with all evaluation metrics.
# Load model results if available, otherwise compute
try:
with open('models/results_summary.json', 'r') as f:
results_summary = json.load(f)
print("Loaded saved results")
except:
print("Results file not found - would need to retrain models")
results_summary = {}
if results_summary and X_test is not None:
# Create comprehensive comparison table
comparison_data = []
for model_name, results in results_summary.items():
row = {
'Model': model_name,
'Accuracy': f"{results['test']['accuracy']:.4f}",
'Precision': f"{results['test']['precision']:.4f}",
'Recall': f"{results['test']['recall']:.4f}",
'F1-Score': f"{results['test']['f1']:.4f}",
'ROC-AUC': f"{results['test']['roc_auc']:.4f}",
'CV ROC-AUC (mean ± std)': f"{results['cv_mean']['roc_auc']:.4f} ± {results['cv_std']['roc_auc']:.4f}"
}
comparison_data.append(row)
comparison_df = pd.DataFrame(comparison_data)
comparison_df = comparison_df.sort_values('ROC-AUC', ascending=False)
print("\nModel Comparison (Test Set Performance):")
print("="*80)
display(comparison_df)
# Highlight best performer
best_model = comparison_df.iloc[0]['Model']
print(f"\n🏆 Best Model (by ROC-AUC): {best_model}")
Results file not found - would need to retrain models
ROC Curves#
ROC curves show the trade-off between true positive rate and false positive rate across different classification thresholds.
if X_test is not None:
# Load models and compute ROC curves
models_to_plot = {}
model_files = {
'Logistic Regression': 'models/logistic_regression.pkl',
'Decision Tree': 'models/decision_tree.pkl',
'Random Forest': 'models/random_forest.pkl',
'SVM': 'models/svm.pkl'
}
# Try to add XGBoost if available
try:
import xgboost
model_files['XGBoost'] = 'models/xgboost.pkl'
except ImportError:
pass
for name, filepath in model_files.items():
try:
model = joblib.load(filepath)
# Determine if scaling needed
if name in ['Logistic Regression', 'SVM'] and scaler is not None:
X_test_model = X_test_scaled
else:
X_test_model = X_test
y_pred_proba = model.predict_proba(X_test_model)[:, 1]
fpr, tpr, _ = roc_curve(y_test, y_pred_proba)
roc_auc = auc(fpr, tpr)
models_to_plot[name] = {'fpr': fpr, 'tpr': tpr, 'auc': roc_auc}
except FileNotFoundError:
print(f"Model {name} not found - skipping")
# Plot ROC curves
plt.figure(figsize=(10, 8))
colors = ['#1f77b4', '#ff7f0e', '#2ca02c', '#d62728', '#9467bd', '#8c564b']
for i, (name, data) in enumerate(models_to_plot.items()):
plt.plot(data['fpr'], data['tpr'],
label=f"{name} (AUC = {data['auc']:.3f})",
linewidth=2, color=colors[i % len(colors)])
plt.plot([0, 1], [0, 1], 'k--', linewidth=1, label='Random Classifier (AUC = 0.500)')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate', fontsize=12)
plt.ylabel('True Positive Rate', fontsize=12)
plt.title('ROC Curves: Model Comparison', fontsize=14, fontweight='bold', pad=20)
plt.legend(loc="lower right", fontsize=10)
plt.grid(alpha=0.3)
plt.tight_layout()
plt.show()
Model Logistic Regression not found - skipping
Model Decision Tree not found - skipping
Model Random Forest not found - skipping
Model SVM not found - skipping
Model XGBoost not found - skipping
Precision-Recall Curves#
Precision-Recall curves are especially important for imbalanced datasets, showing the trade-off between precision and recall.
if X_test is not None and len(models_to_plot) > 0:
plt.figure(figsize=(10, 8))
colors = ['#1f77b4', '#ff7f0e', '#2ca02c', '#d62728', '#9467bd', '#8c564b']
for i, (name, _) in enumerate(models_to_plot.items()):
try:
model = joblib.load(model_files[name])
if name in ['Logistic Regression', 'SVM'] and scaler is not None:
X_test_model = X_test_scaled
else:
X_test_model = X_test
y_pred_proba = model.predict_proba(X_test_model)[:, 1]
precision, recall, _ = precision_recall_curve(y_test, y_pred_proba)
pr_auc = auc(recall, precision)
plt.plot(recall, precision,
label=f"{name} (AUC = {pr_auc:.3f})",
linewidth=2, color=colors[i % len(colors)])
except:
continue
# Baseline (random classifier)
baseline_precision = y_test.mean()
plt.axhline(y=baseline_precision, color='k', linestyle='--',
label=f'Baseline (P = {baseline_precision:.3f})', linewidth=1)
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('Recall', fontsize=12)
plt.ylabel('Precision', fontsize=12)
plt.title('Precision-Recall Curves: Model Comparison', fontsize=14, fontweight='bold', pad=20)
plt.legend(loc="lower left", fontsize=10)
plt.grid(alpha=0.3)
plt.tight_layout()
plt.show()
Confusion Matrices#
Confusion matrices show the exact breakdown of correct and incorrect predictions for the best model.
if X_test is not None and len(models_to_plot) > 0:
# Plot confusion matrices for top 3 models
top_models = sorted(models_to_plot.items(), key=lambda x: x[1]['auc'], reverse=True)[:3]
fig, axes = plt.subplots(1, 3, figsize=(15, 4))
for idx, (name, _) in enumerate(top_models):
try:
model = joblib.load(model_files[name])
if name in ['Logistic Regression', 'SVM'] and scaler is not None:
X_test_model = X_test_scaled
else:
X_test_model = X_test
y_pred = model.predict(X_test_model)
cm = confusion_matrix(y_test, y_pred)
# Normalize confusion matrix
cm_normalized = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
sns.heatmap(cm_normalized, annot=True, fmt='.2f', cmap='Blues',
xticklabels=['Did Not Survive', 'Survived'],
yticklabels=['Did Not Survive', 'Survived'],
ax=axes[idx], cbar_kws={'label': 'Proportion'})
axes[idx].set_ylabel('True Label', fontsize=10)
axes[idx].set_xlabel('Predicted Label', fontsize=10)
axes[idx].set_title(f'{name}\n(AUC = {models_to_plot[name]["auc"]:.3f})',
fontweight='bold', fontsize=11)
except:
axes[idx].text(0.5, 0.5, 'Model not found', ha='center', va='center')
axes[idx].set_title(name)
plt.tight_layout()
plt.show()
# Print detailed classification report for best model
if top_models:
best_name = top_models[0][0]
try:
model = joblib.load(model_files[best_name])
if best_name in ['Logistic Regression', 'SVM'] and scaler is not None:
X_test_model = X_test_scaled
else:
X_test_model = X_test
y_pred = model.predict(X_test_model)
print(f"\nDetailed Classification Report: {best_name}")
print("="*60)
print(classification_report(y_test, y_pred,
target_names=['Did Not Survive', 'Survived']))
except:
pass
Error Analysis#
Understanding which cases are misclassified can reveal model limitations and data patterns.
if X_test is not None and len(models_to_plot) > 0:
# Analyze misclassifications for best model
best_name = sorted(models_to_plot.items(), key=lambda x: x[1]['auc'], reverse=True)[0][0]
try:
model = joblib.load(model_files[best_name])
if best_name in ['Logistic Regression', 'SVM'] and scaler is not None:
X_test_model = X_test_scaled
else:
X_test_model = X_test
y_pred = model.predict(X_test_model)
# Get misclassified cases
misclassified = X_test[y_test != y_pred].copy()
misclassified['True_Label'] = y_test[y_test != y_pred].values
misclassified['Predicted_Label'] = y_pred[y_test != y_pred]
print(f"Error Analysis for {best_name}:")
print("="*60)
print(f"Total misclassifications: {len(misclassified)} ({len(misclassified)/len(y_test)*100:.1f}%)")
# Analyze false positives (predicted survived, actually did not)
false_positives = misclassified[misclassified['Predicted_Label'] == 1]
false_negatives = misclassified[misclassified['Predicted_Label'] == 0]
print(f"\nFalse Positives (predicted survived, actually did not): {len(false_positives)}")
print(f"False Negatives (predicted did not survive, actually survived): {len(false_negatives)}")
if len(misclassified) > 0:
# Analyze patterns in misclassifications
print("\nCharacteristics of Misclassified Cases:")
print("-" * 60)
# Reconstruct original features for analysis
misclassified_original = data.loc[X_test.index[y_test != y_pred]].copy()
if len(misclassified_original) > 0:
print("\nSex distribution of misclassified:")
print(misclassified_original['Sex'].value_counts())
print("\nPclass distribution of misclassified:")
print(misclassified_original['Pclass'].value_counts())
print("\nAverage age of misclassified:")
print(f" {misclassified_original['Age'].mean():.1f} years")
except Exception as e:
print(f"Error in analysis: {e}")
Final Conclusions#
Key Findings#
Strongest Predictors: Sex and Passenger Class were consistently the most important features across all models, validating the historical “women and children first” evacuation policy.
Model Performance: All models achieved ROC-AUC scores above 0.80, with ensemble methods (Random Forest, XGBoost) performing best. The consistency between cross-validation and test set performance indicates robust generalization.
Feature Engineering Impact: Extracting Title from names and creating family-related features (FamilySize, IsAlone) improved model performance by capturing additional demographic patterns.
Model Selection: Random Forest or XGBoost would be recommended for deployment, balancing performance and interpretability. Logistic Regression provides a good baseline with interpretable coefficients.
Limitations#
Historical Context: The dataset reflects 1912 social norms and evacuation procedures, which may not generalize to modern scenarios.
Missing Data: ~20% of Age values were imputed, which introduces some uncertainty.
Sample Size: With 891 training samples, there are limits to model complexity before overfitting.
Real-World Implications#
If deploying this model for similar survival prediction tasks:
Interpretability: Use Random Forest for feature importance or Logistic Regression for coefficient interpretation
Performance: XGBoost provides best accuracy but requires more computational resources
Monitoring: Track model performance over time and retrain if data distribution shifts
Future Work#
Explore deep learning approaches (neural networks) for comparison
Investigate interaction effects more systematically
Apply similar methodology to other survival analysis datasets
Develop a production pipeline with automated retraining
Back to: Project Overview | Data Exploration