Deep Learning

Deep Learning#

Navigation: ← Previous: Classical Models | Next: Interpretation →

A compact CNN trained from scratch on 546 images, then a frozen ImageNet ResNet-18 with a new head.

Two conv nets, two bets#

Small CNN. Four conv-BN-ReLU blocks with max-pooling, global average pooling, dropout, and a single logit. Trained with binary cross-entropy and a positive-class weight of n_benign / n_malignant, horizontal flips, and early stopping on validation ROC-AUC.

ResNet-18 transfer. ImageNet-pretrained backbone, grayscale repeated to three channels and resized to 224×224, all convolutional weights frozen. Only the final linear head is trained. With a few hundred labels this is usually a better bet than learning filters from scratch.

Weights in artifacts/ were produced by _train_models.py so the Jupyter Book build does not retrain on CPU in CI. This notebook loads those runs and reports held-out metrics.

import sys
from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import warnings
warnings.filterwarnings("ignore")
plt.style.use("seaborn-v0_8-whitegrid")

PROJ_DIR = Path(".").resolve()
if not (PROJ_DIR / "cancer_cv_utils.py").exists():
    PROJ_DIR = Path("projects/cancer-imaging").resolve()
if str(PROJ_DIR) not in sys.path:
    sys.path.insert(0, str(PROJ_DIR))

from cancer_cv_utils import (
    CLASS_NAMES,
    METRIC_ORDER,
    artifacts_dir,
    classification_metrics,
    display_plotly,
    extract_cv_features,
    figures_dir,
    grouped_importance,
    load_metrics,
    load_predictions,
    load_splits,
    metrics_frame,
    overlay_heatmap,
    tune_threshold,
)

SPLITS = load_splits()
FIG = figures_dir()
ART = artifacts_dir()
print("Splits:", {k: v["labels"].shape[0] for k, v in SPLITS.items()})
print("Malignant rates:", {k: f"{v['labels'].mean():.1%}" for k, v in SPLITS.items()})

Splits: {'train': 546, 'val': 78, 'test': 156}
Malignant rates: {'train': '26.9%', 'val': '26.9%', 'test': '26.9%'}

from IPython.display import Image, display
import torch
from cancer_cv_utils import SmallCNN, last_conv

def _load_ckpt(path):
    try:
        return torch.load(path, map_location='cpu', weights_only=False)
    except TypeError:
        return torch.load(path, map_location='cpu')

cnn_ckpt = _load_ckpt(ART / 'small_cnn.pt')
res_ckpt = _load_ckpt(ART / 'resnet18_head.pt')
cnn = SmallCNN()
cnn.load_state_dict(cnn_ckpt['state_dict'])
n_params = sum(p.numel() for p in cnn.parameters())
print(f'Small CNN parameters: {n_params:,}')
print(f"Grad-CAM target: {type(last_conv(cnn)).__name__}")
print('CNN epochs recorded:', cnn_ckpt['history'][-1]['epoch'])
print('ResNet epochs recorded:', res_ckpt['history'][-1]['epoch'])

Small CNN parameters: 97,761
Grad-CAM target: Conv2d
CNN epochs recorded: 16
ResNet epochs recorded: 23

Training curves (validation ROC-AUC)#

fig, ax = plt.subplots(figsize=(7.5, 3.8))
for hist, label, colour in [
    (cnn_ckpt['history'], 'Small CNN', '#8e44ad'),
    (res_ckpt['history'], 'ResNet-18 head', '#c0392b'),
]:
    ax.plot([h['epoch'] for h in hist], [h['val_auc'] for h in hist],
            marker='o', label=label, color=colour)
ax.set_xlabel('Epoch')
ax.set_ylabel('Validation ROC-AUC')
ax.set_title('Early stopping on val AUC — not on test')
ax.legend(frameon=False)
ax.set_ylim(0.45, 1.0)
plt.tight_layout()
plt.show()
../../_images/572a664ce1fdc83e7d3c00c6d65e8ce5a3894e79eaee139d513a9dd497e93b68.png

Held-out metrics#

mets = load_metrics()
metrics_frame({
    'Small CNN': mets['Small CNN'],
    'ResNet-18 transfer': mets['ResNet-18 transfer'],
}).round(3)
accuracy precision recall f1 specificity balanced_accuracy roc_auc pr_auc brier mcc
model
Small CNN 0.846 0.725 0.690 0.707 0.904 0.797 0.833 0.736 0.236 0.603
ResNet-18 transfer 0.776 0.554 0.857 0.673 0.746 0.801 0.869 0.772 0.164 0.542

Thresholds were chosen by validation F1, then frozen for the test set. The small CNN lands on a similar accuracy to the random forest; the ResNet head trades precision for recall (it misses fewer cancers, flags more benign scans). ROC-AUC and PR-AUC, which do not depend on a single threshold, both favour transfer learning.

for fname in ['03_cm_cnn.png', '03_cm_resnet.png']:
    display(Image(str(FIG / fname)))
../../_images/fd4dabd8d17456c1656d477ac6ab708781f559cbda86af2b9394ceb45f6eb952.png ../../_images/1e120e032a60d494f708e7058c17035aa7408e9babaa76c8ac7541b4444b720d.png

Why the from-scratch CNN is not an automatic win#

546 labelled 128×128 scans are a small conv-net dataset. Data augmentation here is only a flip; there is no multi-site sample. ImageNet filters, even from natural photos, still transfer enough edge and blob detectors to improve ranking quality (AUC) and recall. The next chapter asks whether those nets are looking at the lesion.

Navigation: ← Previous: Classical Models | Next: Interpretation →

Contents