Summary and guide for calzone

We provide a summary of the calibration metrics provides by calzone, including the pros and cons of each metrics. For a more detailed explanation of each metrics and how to calculate them using calzone, please refer to the specific notebook.

display(HTML(styled_df.to_html(escape=False)))

Metrics	Description	Pros	Cons	Meaning
Expected calibration error (ECE)	Using binned reliability diagram (equal-width or equal-count binning), sum of absolute difference, weighted by bin count.	• Intuitive • Easy to calculate	• Depend on binning • Depend on class-by-class/top-class	Average deviation from true probability
Maximum calibration error (MCE)	Using binned reliability diagram (equal-width or equal-count binning), Maximum absolute difference.	• Intuitive • Easy to calculate	• Depend on binning • Depend on class-by-class/top-class	Maximum deviation from true probability
Hosmer-Lemeshow test	Using binned reliability diagram (equal-width or equal-count binning), Chi-squared based test using expected and observed.	• Intuitive • Statistical meaning	• Depend on binning • Low power • Wrong coverage	Test of calibration
Spiegelhalter's z test	Decomposition of brier score. Normal distributed	• Doesn't rely on binning • Statistical meaning	• Doesn't detect prevalence shift	Test of calibration
Cox's analysis	Logistic regression of the logits	• Doesn't rely on binning • Hints at miscalibration type	• Failed to capture some miscalibration	A logit fit to the calibration curve
Integrated calibration index (ICI)	Similar to ECE, using smooth fit (usually losse) instead of binning to get the calibration curve	• Doesn't rely on binning • Capture all kind of miscalibration	• Depend on the choice of curve fitting • Depend on fitting parameters	Average deviation from true probability

Guide to calzone and calibration metrics

calzone aims to access whether a model achieves moderate calibration, meaning whether \(\mathbb{P}(D=1|\hat{P}=p)=p\) for all \(p\in[0,1]\).

To accurately assess the calibration of a machine learning model, it’s important to use a dataset that’s both comprehensive and representative of the intended population. Calibration metrics such as reliability diagrams or expected calibration error aren’t meaningful if the dataset doesn’t reflect the real-world data the model is meant to operate on. For example, if certain prediction ranges or subgroups are underrepresented, the model might appear well-calibrated overall but actually perform poorly in those regions. In other words, without sufficient coverage of the prediction space, especially across relevant clinical or demographic groups, calibration results can be misleading. Ensuring good coverage helps make sure that the evaluation actually reflects how the model will behave in practice.

calzone takes in a csv dataset which contains the probability of each class and the true label. Most metrics in calzone only work with binary classification and which transforms the problem into 1-vs-rest when calcualte the metrics. Therefore, you need to specify the class-of-interest when using the metrics. The only exception is the Top-class Expected calibration error (\(ECE_{top}\)) and Top-class Maximum calibration error (\(MCE_{top}\)) metrics which only measure the calibration of the class with highest predicted probability hence works for multi-class problems. See the corresponding documentation for more details.

We recommend using reliability diagrams to visualize calibration. If you notice that the model consistently over- or underestimates probabilities for a certain class, it’s worth checking whether that’s just due to a prevalence shift. Prevalence shift happens when the proportion of positive and negative cases in your evaluation data is different from what the model was trained on. This can cause the predicted probabilities to be systematically off, even if the model itself hasn’t changed. You can check for this by applying a prevalence adjustment and then re-plotting the reliability diagrams. After the adjustment, take another look at the calibration metrics to see if the issue persists. This helps separate problems caused by distribution shift from those that are due to poor calibration.

For a general sense of average probability deviation, we recommend using the Cox and Loess integrated calibration index (ICI) as they don’t depend on binning. Alternatively, ECE can be used to measure the same but the result will depend on the binning scheme you used. If the probabilities distribution is highly skewed toward 0 and 1, use equal-count binning for ECE.

Please refer to the notebooks for detailed descriptions of each metric.