Welcome to the documentation for calzone

calzone is a Python package for calculation of various calibration metrics. This work is credited to Kwok Lung (Jason) Fan and Qian Cao.

The calzone package provides a suite of tools for assessing and improving the calibration of machine learning models, particularly for binary classification tasks. It offers calibration metrics and visualization tools for displaying reliability diagrams. Please read the summary and guide section in this documentation first before using the package.

Key features of calzone include:

• Calculation of calibration metrics (ECE, MCE, Hosmer-Lemeshow test, spiegelhalter z test, etc.)

• Visualization functions for reliability diagrams

• Bootstrapping capabilities for confidence interval estimation

• Subgroup analysis for calibration metrics

• Command line interface scripts for batch processing

• Multi-class extension by using 1-vs-rest or top-class only

To accurately assess the calibration of machine learning models, it is essential to have a comprehensive and reprensative dataset with sufficient coverage of the prediction space. The calibration metrics is not meaningful if the dataset is not representative of true intended population.

We hope you find calzone useful in your machine learning projects!

Contents:

• Welcome to the documentation for calzone
• Quick Start
  • Installation
  • Command line interface
  • Using calzone in python
• Summary and guide for calzone
  • Guide to calzone and calibration metrics
• Reliability diagram
  • References
• Exepected Calibration Error(ECE) and Maximum Calibration Error (MCE)
  • Theoretical Background
  • Pros of ECE and MCE
  • Cons of ECE and MCE
  • Calculating ECE and MCE with calzone
  • ECE and MCE as function of bin size
  • Reference
• Hosmer-Lemeshow test (HL test)
  • Theoretical Background
  • Pros of HL test
  • Cons of HL Test
  • Calculating HL test statistics and p-value with calzone
  • P-value of HL test
  • Reference
• COX calibration analysis
  • Theoretical Background
  • Pros of Cox calibration analysis
  • Cons of Cox calibration analysis
  • Calculating Cox slope and intercept with calzone
  • P-value of COX slope and intecept test
  • References
• Integrated Calibration Index (ICI)
  • Theoretical Background
  • Pros of ICI
  • Cons of ICI
  • Calculating LOESS ICI and COX ICI using calzone
  • Visualization of the fitted curve
  • Reference
• Spiegelhalter’s Z-test
  • Theoretical background
  • Pros of Spiegelhalter’s Z test
  • Cons of Spiegelhalter’s Z test
  • Calculating the Spieegelhalter Z score and p-value using calzone
  • P-value of Spiegelhalter’s z test
  • Reference
• Prevalence adjustment
  • Preform prevalence adjustment in calzone
  • Prevalence adjustment and constant shift in logit of class-of-interest
  • References
• Subgroup analysis
• Multiclass extension
• Running the GUI
• Validating metrics with external packages
  • Reliability diagram
  • Expected calibration error and Z test
  • HL test
  • reference
• calzone
  • calzone package