{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Summary and guide for calzone\n", "\n", "We provide a summary of the calibration metrics provides by calzone, including the pros and cons of each metrics. For a more detailed explanation of each metrics and how to calculate them using calzone, please refer to the specific notebook." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "nbsphinx": "hidden" }, "outputs": [], "source": [ "import pandas as pd\n", "from IPython.display import display, HTML\n", "data = {\n", " 'Metrics': ['Expected calibration error
(ECE)', 'Maximum calibration error
(MCE)', 'Hosmer-Lemeshow test', \"Spiegelhalter's z test\", \"Cox's analysis\", 'Integrated calibration index
(ICI)'],\n", " 'Description': [\n", " '
Using binned reliability diagram
(equal-width or equal-count binning),
sum of absolute difference, weighted by bin count.
',\n", " '
Using binned reliability diagram
(equal-width or equal-count binning),
Maximum absolute difference.
',\n", " '
Using binned reliability diagram
(equal-width or equal-count binning),
Chi-squared based test using expected and observed.
',\n", " '
Decomposition of brier score.
Normal distributed
',\n", " '
Logistic regression of the logits

',\n", " '
Similar to ECE, using smooth fit (usually losse)
instead of binning to get
the calibration curve
'\n", " ],\n", " 'Pros': [\n", " '
• Intuitive
• Easy to calculate
',\n", " '
• Intuitive
• Easy to calculate
',\n", " '
• Intuitive
• Statistical meaning
',\n", " '
• Doesn\\'t rely on binning
• Statistical meaning
',\n", " '
• Doesn\\'t rely on binning
• Hints at miscalibration type
',\n", " '
• Doesn\\'t rely on binning
• Capture all kind of miscalibration
'\n", " ],\n", " 'Cons': [\n", " '
• Depend on binning
• Depend on class-by-class/top-class
',\n", " '
• Depend on binning
• Depend on class-by-class/top-class
',\n", " '
• Depend on binning
• Low power
• Wrong coverage
',\n", " '
• Doesn\\'t detect prevalence shift
',\n", " '
• Failed to capture some miscalibration
',\n", " '
• Depend on the choice of curve fitting
• Depend on fitting parameters
'\n", " ],\n", " 'Meaning': [\n", " '
Average deviation from
true probability
',\n", " '
Maximum deviation from
true probability
',\n", " '
Test of
calibration
',\n", " '
Test of
calibration
',\n", " '
A logit fit to the
calibration curve
',\n", " '
Average deviation from
true probability
'\n", " ]\n", "}\n", "df = pd.DataFrame(data)\n", "\n", "# Apply custom styling\n", "styled_df = df.style.set_properties(**{'text-align': 'left', 'white-space': 'pre-wrap'})\n", "styled_df = styled_df.set_table_styles([dict(selector='th', props=[('text-align', 'center')])])\n", "\n", "styled_df = styled_df.hide(axis=\"index\")\n", "\n", "# Display the styled dataframe\n", "# ### Export PNG format of the table\n", "# import dataframe_image as dfi\n", "\n", "# dfi.export(styled_df,\"mytable.png\",table_conversion = 'matplotlib',dpi=300)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
MetricsDescriptionProsConsMeaning
Expected calibration error
(ECE)
Using binned reliability diagram
(equal-width or equal-count binning),
sum of absolute difference, weighted by bin count.
• Intuitive
• Easy to calculate
• Depend on binning
• Depend on class-by-class/top-class
Average deviation from
true probability
Maximum calibration error
(MCE)
Using binned reliability diagram
(equal-width or equal-count binning),
Maximum absolute difference.
• Intuitive
• Easy to calculate
• Depend on binning
• Depend on class-by-class/top-class
Maximum deviation from
true probability
Hosmer-Lemeshow test
Using binned reliability diagram
(equal-width or equal-count binning),
Chi-squared based test using expected and observed.
• Intuitive
• Statistical meaning
• Depend on binning
• Low power
• Wrong coverage
Test of
calibration
Spiegelhalter's z test
Decomposition of brier score.
Normal distributed
• Doesn't rely on binning
• Statistical meaning
• Doesn't detect prevalence shift
Test of
calibration
Cox's analysis
Logistic regression of the logits

• Doesn't rely on binning
• Hints at miscalibration type
• Failed to capture some miscalibration
A logit fit to the
calibration curve
Integrated calibration index
(ICI)
Similar to ECE, using smooth fit (usually losse)
instead of binning to get
the calibration curve
• Doesn't rely on binning
• Capture all kind of miscalibration
• Depend on the choice of curve fitting
• Depend on fitting parameters
Average deviation from
true probability
\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "display(HTML(styled_df.to_html(escape=False)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Guide to calzone and calibration metrics\n", "\n", "calzone aims to access whether a model achieves moderate calibration, meaning whether $\\mathbb{P}(D=1|\\hat{P}=p)=p$ for all $p\\in[0,1]$.\n", "\n", "To accurately assess the calibration of a machine learning model, it's important to use a dataset that's both comprehensive and representative of the intended population. Calibration metrics such as reliability diagrams or expected calibration error aren't meaningful if the dataset doesn't reflect the real-world data the model is meant to operate on. For example, if certain prediction ranges or subgroups are underrepresented, the model might appear well-calibrated overall but actually perform poorly in those regions. In other words, without sufficient coverage of the prediction space, especially across relevant clinical or demographic groups, calibration results can be misleading. Ensuring good coverage helps make sure that the evaluation actually reflects how the model will behave in practice.\n", "\n", "calzone takes in a csv dataset which contains the probability of each class and the true label. Most metrics in calzone only work with binary classification and which transforms the problem into 1-vs-rest when calcualte the metrics. Therefore, you need to specify the class-of-interest when using the metrics. The only exception is the Top-class Expected calibration error ($ECE_{top}$) and Top-class Maximum calibration error ($MCE_{top}$) metrics which only measure the calibration of the class with highest predicted probability hence works for multi-class problems. See the corresponding documentation for more details.\n", "\n", "\n", "We recommend using reliability diagrams to visualize calibration. If you notice that the model consistently over- or underestimates probabilities for a certain class, it’s worth checking whether that’s just due to a prevalence shift. Prevalence shift happens when the proportion of positive and negative cases in your evaluation data is different from what the model was trained on. This can cause the predicted probabilities to be systematically off, even if the model itself hasn’t changed. You can check for this by applying a prevalence adjustment and then re-plotting the reliability diagrams. After the adjustment, take another look at the calibration metrics to see if the issue persists. This helps separate problems caused by distribution shift from those that are due to poor calibration.\n", "\n", "For a general sense of average probability deviation, we recommend using the Cox and Loess integrated calibration index (ICI) as they don't depend on binning. Alternatively, ECE can be used to measure the same but the result will depend on the binning scheme you used. If the probabilities distribution is highly skewed toward 0 and 1, use equal-count binning for ECE. \n", "\n", "Please refer to the notebooks for detailed descriptions of each metric." ] } ], "metadata": { "kernelspec": { "display_name": "general", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.5" } }, "nbformat": 4, "nbformat_minor": 2 }