Scores and calibration over time. Updated after each recalibration.
AUC (Area Under the ROC Curve) — Measures how well the model ranks patients (discrimination): can it separate higher-risk from lower-risk? 0.5 = random, 1.0 = perfect. Your model's AUC (~0.91) is unchanged by calibration; calibration does not change ranking.
Brier score — Measures how accurate the probabilities are (calibration): when the model says "30% risk," do about 30% of those patients have the event? Lower is better (0 = perfect). Calibration improves this.
Original vs calibrated — "Original" = the base Random Forest (pure data-driven). "Calibrated" = same model after literature-informed Platt scaling so that predicted probabilities better match observed frequencies.
Articles used / Lit. quality score — Number of PROBAST-compliant articles from the literature database used to inform calibration, and an aggregate quality score. More/better literature can slightly adjust the calibration curve.
95% CI (confidence interval) — A range (low to high) that we are 95% confident contains the true Brier improvement. From a bootstrap test (2,000 resamples of the test set). If the whole interval is above 0, the improvement is consistent with a real gain.
p-value — Probability that we'd see an improvement this large (or larger) if the calibrated model were no better than the original. Small p (e.g. < 0.05 or < 0.001) means the calibrated model is statistically significantly better.
The original model is overconfident: it tends to predict probabilities that are too high (e.g. it says 70% when the true rate is closer to 30–40%). That's why its curve on the calibration plot sits below the "Perfect calibration" line. The model's ranking (AUC) is strong; it's the scale of the probabilities that's off. Literature-informed calibration (Platt scaling) corrects that scale so the green "Literature-Calibrated" curve sits much closer to the diagonal, and the Brier score drops. So the "bad look" of the original is expected and is exactly what calibration is meant to fix.
AUC and Brier for each recalibration run. Use this to see that AUC stays stable and Brier improves with calibration as you add literature and recalibrate.
X-axis: Predicted probability. Y-axis: Observed frequency. The dashed diagonal = perfect calibration (the goal; no model can be "better" than this). The blue line (original) is the raw data-driven model with no calibration step—it sits below the diagonal because it's overconfident. The green line (calibrated) is the same model after Platt scaling; it's meant to sit close to the diagonal, i.e. near perfect, not above it.
No calibration plot yet. Run Recalibrate from Literature Admin.
One row per recalibration. Lets you track how AUC, Brier, and article count change over time.