{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary and guide for calzone\n",
    "\n",
    "We provide a summary of the calibration metrics provides by calzone, including the pros and cons of each metrics. For a more detailed explanation of each metrics and how to calculate them using calzone, please refer to the specific notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "nbsphinx": "hidden"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/kwoklung.fan/anaconda3/envs/uq/lib/python3.12/site-packages/dataframe_image/converter/matplotlib_table.py:147: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead.\n",
      "  if not thead and not tbody:\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from IPython.display import display, HTML\n",
    "data = {\n",
    "    'Metrics': ['Expected calibration error<br>(ECE)', 'Maximum calibration error<br>(MCE)', 'Hosmer-Lemeshow test', \"Spiegelhalter's z test\", \"Cox's analysis\", 'Integrated calibration index<br> (ICI)'],\n",
    "    'Description': [\n",
    "        '<div>Using binned reliability diagram<br>(equal-width or equal-count binning),<br>sum of absolute difference, weighted by bin count.</div>',\n",
    "        '<div>Using binned reliability diagram<br>(equal-width or equal-count binning),<br>Maximum absolute difference.</div>',\n",
    "        '<div>Using binned reliability diagram<br>(equal-width or equal-count binning),<br>Chi-squared based test using expected and observed.</div>',\n",
    "        '<div>Decomposition of brier score.<br>Normal distributed<br> </div>',\n",
    "        '<div>Logistic regression of the logits<br> <br> </div>',\n",
    "        '<div>Similar to ECE, using smooth fit (usually losse)<br>instead of binning to get<br>the calibration curve</div>'\n",
    "    ],\n",
    "    'Pros': [\n",
    "        '<div>• Intuitive<br>• Easy to calculate</div>',\n",
    "        '<div>• Intuitive<br>• Easy to calculate</div>',\n",
    "        '<div>• Intuitive<br>• Statistical meaning</div>',\n",
    "        '<div>• Doesn\\'t rely on binning<br>• Statistical meaning</div>',\n",
    "        '<div>• Doesn\\'t rely on binning<br>• Hints at miscalibration type</div>',\n",
    "        '<div>• Doesn\\'t rely on binning<br>• Capture all kind of miscalibration</div>'\n",
    "    ],\n",
    "    'Cons': [\n",
    "        '<div>• Depend on binning <br>• Depend on class-by-class/top-class</div>',\n",
    "        '<div>• Depend on binning <br>• Depend on class-by-class/top-class</div>',\n",
    "        '<div>• Depend on binning <br>• Low power<br>• Wrong coverage</div>',\n",
    "        '<div>• Doesn\\'t detect prevalence shift</div>',\n",
    "        '<div>• Failed to capture some miscalibration</div>',\n",
    "        '<div>• Depend on the choice of curve fitting<br>• Depend on fitting parameters</div>'\n",
    "    ],\n",
    "    'Meaning': [\n",
    "        '<div>Average deviation from<br>true probability</div>',\n",
    "        '<div>Maximum deviation from<br>true probability</div>',\n",
    "        '<div>Test of<br>calibration</div>',\n",
    "        '<div>Test of<br>calibration</div>',\n",
    "        '<div>A logit fit to the<br>calibration curve</div>',\n",
    "        '<div>Average deviation from<br>true probability</div>'\n",
    "    ]\n",
    "}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Apply custom styling\n",
    "styled_df = df.style.set_properties(**{'text-align': 'left', 'white-space': 'pre-wrap'})\n",
    "styled_df = styled_df.set_table_styles([dict(selector='th', props=[('text-align', 'center')])])\n",
    "\n",
    "styled_df = styled_df.hide(axis=\"index\")\n",
    "\n",
    "# Display the styled dataframe\n",
    "#display(HTML(styled_df.to_html(escape=False)))\n",
    "import dataframe_image as dfi\n",
    "\n",
    "dfi.export(styled_df,\"mytable.png\",table_conversion = 'matplotlib',dpi=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](mytable.png \"Title\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Guide to calzone and calibration metrics\n",
    "\n",
    "calzone aims to access whether a model achieves moderate calibration, meaning whether $\\mathbb{P}(\\hat{Y}=Y|\\hat{P}=p)=p$ for all $p\\in[0,1]$.\n",
    "\n",
    "To accurately assess the calibration of machine learning models, it is essential to have a comprehensive and reprensative dataset with sufficient coverage of the prediction space. The calibration metrics is not meaningful if the dataset is not representative of true intended population.\n",
    "\n",
    "calzone takes in a csv dataset which contains the probability of each class and the true label. Most metrics in calzone only work with binary classification and which transforms the problem into 1-vs-rest when calcualte the metrics. Therefore, you need to specify the class-of-interest when using the metrics. The only exception is the Top-class Expected calibration error ($ECE_{top}$) and Top-class Maximum calibration error ($MCE_{top}$) metrics which only measure the calibration of the class with highest predicted probability hence works for multi-class problems. See the corresponding documentation for more details.\n",
    "\n",
    "\n",
    "We recommend visualizing calibration using reliability diagrams. If you observe general over- or under-estimation of probabilities for a given class, consider applying a prevalence adjustment to determine if it's solely due to prevalence shift. After prevalence adjustment, plot the reliability diagrams again and examine the results of calibration metrics.\n",
    "\n",
    "For a general sense of average probability deviation, we recommend using the Cox and Loess integrated calibration index (ICI) as they don't depend on binning. Alternativly, ECE can be used to measure the same but the result will depend on the binning scheme you used. If the probabilities distribution is highly skewed toward 0 and 1, use equal-count binning for ECE. \n",
    "\n",
    "Please refer to the notebooks for detailed descriptions of each metric."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "uq",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}