.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples\5_compare\plot_0_compare_classification.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code or to run this example in your browser via Binder .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_5_compare_plot_0_compare_classification.py: Model Comparison: Classification ======================================================== .. GENERATED FROM PYTHON SOURCE LINES 8-9 Experiment initialization and data preparation .. GENERATED FROM PYTHON SOURCE LINES 9-19 .. code-block:: default from piml import Experiment from piml.models import TreeClassifier from piml.models import FIGSClassifier from piml.models import XGB2Classifier exp = Experiment() exp.data_loader(data="TaiwanCredit", silent=True) exp.data_summary(feature_exclude=["LIMIT_BAL", "SEX", "EDUCATION", "MARRIAGE", "AGE"], silent=True) exp.data_prepare(target="FlagDefault", task_type="classification", silent=True) .. GENERATED FROM PYTHON SOURCE LINES 20-21 Train Model .. GENERATED FROM PYTHON SOURCE LINES 21-25 .. code-block:: default exp.model_train(TreeClassifier(), name="Tree") exp.model_train(FIGSClassifier(), name="FIGS") exp.model_train(XGB2Classifier(), name="XGB2") .. GENERATED FROM PYTHON SOURCE LINES 26-27 Accuracy comparison with ACC .. GENERATED FROM PYTHON SOURCE LINES 27-29 .. code-block:: default exp.model_compare(models=["Tree", "FIGS", "XGB2"], show="accuracy_plot", metric="ACC", figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_001.png :alt: Predictive Performance :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 30-31 Accuracy comparison with AUC .. GENERATED FROM PYTHON SOURCE LINES 31-33 .. code-block:: default exp.model_compare(models=["Tree", "FIGS", "XGB2"], show="accuracy_plot", metric="AUC", figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_002.png :alt: Predictive Performance :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_002.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 34-35 Accuracy comparison with F1 .. GENERATED FROM PYTHON SOURCE LINES 35-37 .. code-block:: default exp.model_compare(models=["Tree", "FIGS", "XGB2"], show="accuracy_plot", metric="F1", figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_003.png :alt: Predictive Performance :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_003.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 38-39 Overfit comparison with ACC metric .. GENERATED FROM PYTHON SOURCE LINES 39-42 .. code-block:: default exp.model_compare(models=["Tree", "FIGS", "XGB2"], show="overfit", slice_method="histogram", slice_feature="PAY_1", bins=10, metric="ACC", original_scale=True, figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_004.png :alt: Overfit Comparison :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_004.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 43-44 Overfit comparison with AUC metric .. GENERATED FROM PYTHON SOURCE LINES 44-48 .. code-block:: default exp.model_compare(models=["Tree", "FIGS", "XGB2"], show="overfit", slice_method="histogram", slice_feature="PAY_1", metric="AUC", original_scale=True, figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_005.png :alt: Overfit Comparison :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_005.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 49-50 Reliability bandwidth comparison .. GENERATED FROM PYTHON SOURCE LINES 50-51 .. code-block:: default exp.model_compare(models=["Tree", "FIGS", "XGB2"], show="reliability_bandwidth", figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_006.png :alt: Uncertainty Comparison :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_006.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 52-53 Reliability diagram comparison .. GENERATED FROM PYTHON SOURCE LINES 53-55 .. code-block:: default exp.model_compare(models=["Tree", "FIGS", "XGB2"], show="reliability_perf", bins=10, figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_007.png :alt: Reliability Diagram: Testing Data :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_007.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 56-57 Robustness comparison with default settings .. GENERATED FROM PYTHON SOURCE LINES 57-59 .. code-block:: default exp.model_compare(models=["Tree", "FIGS", "XGB2"], show="robustness_perf", figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_008.png :alt: Model Performance: Perturb on All Features :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_008.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 60-61 Robustness comparison on worst alpha-percent samples .. GENERATED FROM PYTHON SOURCE LINES 61-63 .. code-block:: default exp.model_compare(models=["Tree", "FIGS", "XGB2"], show="robustness_perf_worst", alpha=0.3, figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_009.png :alt: 30%-Worst Sample Performance :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_009.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 64-65 Resilience comparison with worst-sample scenario .. GENERATED FROM PYTHON SOURCE LINES 65-68 .. code-block:: default exp.model_compare(models=["Tree", "FIGS", "XGB2"], show="resilience_perf", resilience_method="worst-sample", immu_feature=None, metric="AUC", figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_010.png :alt: Resilience Test :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_010.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 69-70 Resilience distance comparison with worst-sample scenario .. GENERATED FROM PYTHON SOURCE LINES 70-72 .. code-block:: default exp.model_compare(models=["Tree", "FIGS", "XGB2"], show="resilience_distance", resilience_method="worst-sample", metric="AUC", alpha=0.3, figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_011.png :alt: Distribution Shift: 30%-Worst vs Remaining :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_classification_011.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-timing **Total running time of the script:** ( 1 minutes 21.717 seconds) **Estimated memory usage:** 120 MB .. _sphx_glr_download_auto_examples_5_compare_plot_0_compare_classification.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: binder-badge .. image:: images/binder_badge_logo.svg :target: https://mybinder.org/v2/gh/selfexplainml/piml-toolbox/main?urlpath=lab/tree/./docs/_build/html/notebooks/auto_examples/5_compare/plot_0_compare_classification.ipynb :alt: Launch binder :width: 150 px .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_0_compare_classification.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_0_compare_classification.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_