.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples\4_testing\plot_5_resilience_cls.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code or to run this example in your browser via Binder .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_4_testing_plot_5_resilience_cls.py: Resilience: Classification ===================================== .. GENERATED FROM PYTHON SOURCE LINES 8-9 Experiment initialization and data preparation .. GENERATED FROM PYTHON SOURCE LINES 9-17 .. code-block:: default from piml import Experiment from piml.models import XGB2Classifier exp = Experiment() exp.data_loader(data="TaiwanCredit", silent=True) exp.data_summary(feature_exclude=["LIMIT_BAL", "SEX", "EDUCATION", "MARRIAGE", "AGE"], silent=True) exp.data_prepare(target="FlagDefault", task_type="classification", silent=True) .. GENERATED FROM PYTHON SOURCE LINES 18-19 Train Model .. GENERATED FROM PYTHON SOURCE LINES 19-20 .. code-block:: default exp.model_train(model=XGB2Classifier(max_depth=2, n_estimators=100), name="XGB2") .. GENERATED FROM PYTHON SOURCE LINES 21-22 Resilience performance against worst sample scenario .. GENERATED FROM PYTHON SOURCE LINES 22-23 .. code-block:: default exp.model_diagnose(model="XGB2", show="resilience_perf", resilience_method="worst-sample", figsize=(5, 4)) .. image-sg:: /auto_examples/4_testing/images/sphx_glr_plot_5_resilience_cls_001.png :alt: Resilience Test :srcset: /auto_examples/4_testing/images/sphx_glr_plot_5_resilience_cls_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 24-25 Resilience performance against hard sample scenario .. GENERATED FROM PYTHON SOURCE LINES 25-26 .. code-block:: default exp.model_diagnose(model="XGB2", show="resilience_perf", resilience_method="hard-sample", figsize=(5, 4)) .. image-sg:: /auto_examples/4_testing/images/sphx_glr_plot_5_resilience_cls_002.png :alt: Resilience Test :srcset: /auto_examples/4_testing/images/sphx_glr_plot_5_resilience_cls_002.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 27-28 Resilience performance against outer sample scenario .. GENERATED FROM PYTHON SOURCE LINES 28-29 .. code-block:: default exp.model_diagnose(model="XGB2", show="resilience_perf", resilience_method="outer-sample", figsize=(5, 4)) .. image-sg:: /auto_examples/4_testing/images/sphx_glr_plot_5_resilience_cls_003.png :alt: Resilience Test :srcset: /auto_examples/4_testing/images/sphx_glr_plot_5_resilience_cls_003.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 30-31 Resilience performance against worst cluster scenario .. GENERATED FROM PYTHON SOURCE LINES 31-32 .. code-block:: default exp.model_diagnose(model="XGB2", show="resilience_perf", resilience_method="worst-cluster", figsize=(5, 4)) .. image-sg:: /auto_examples/4_testing/images/sphx_glr_plot_5_resilience_cls_004.png :alt: Resilience Test :srcset: /auto_examples/4_testing/images/sphx_glr_plot_5_resilience_cls_004.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 33-34 Marginal distributional distance between full sample and worst sample .. GENERATED FROM PYTHON SOURCE LINES 34-36 .. code-block:: default exp.model_diagnose(model="XGB2", show="resilience_distance", resilience_method="worst-sample", distance_metric="PSI", alpha=0.3, figsize=(5, 4)) .. image-sg:: /auto_examples/4_testing/images/sphx_glr_plot_5_resilience_cls_005.png :alt: Distribution Shift: 30%-Worst vs Remaining :srcset: /auto_examples/4_testing/images/sphx_glr_plot_5_resilience_cls_005.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 37-38 Marginal distributional distance between full sample and worst sample with not Non-immutable features .. GENERATED FROM PYTHON SOURCE LINES 38-40 .. code-block:: default exp.model_diagnose(model="XGB2", show="resilience_distance", resilience_method="worst-sample", distance_metric="PSI", immu_feature="PAY_1", alpha=0.3, figsize=(5, 4)) .. image-sg:: /auto_examples/4_testing/images/sphx_glr_plot_5_resilience_cls_006.png :alt: Distribution Shift: 30%-Worst vs Remaining :srcset: /auto_examples/4_testing/images/sphx_glr_plot_5_resilience_cls_006.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 41-42 Marginal distributional distance between full sample and worst sample with worst-cluster scenario .. GENERATED FROM PYTHON SOURCE LINES 42-44 .. code-block:: default exp.model_diagnose(model="XGB2", show="resilience_distance", resilience_method="worst-cluster", distance_metric="WD1", n_clusters=10, figsize=(5, 4)) .. image-sg:: /auto_examples/4_testing/images/sphx_glr_plot_5_resilience_cls_007.png :alt: Distribution Shift: 10-Cluster-Worst vs Remaining :srcset: /auto_examples/4_testing/images/sphx_glr_plot_5_resilience_cls_007.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 45-46 Marginal histogram plot for full sample and worst sample .. GENERATED FROM PYTHON SOURCE LINES 46-48 .. code-block:: default exp.model_diagnose(model="XGB2", show="resilience_shift_histogram", resilience_method="worst-sample", show_feature="PAY_1", original_scale=True, figsize=(5, 4)) .. image-sg:: /auto_examples/4_testing/images/sphx_glr_plot_5_resilience_cls_008.png :alt: Distribution Shift: 10%-Worst vs Remaining :srcset: /auto_examples/4_testing/images/sphx_glr_plot_5_resilience_cls_008.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 49-50 Marginal density plot for full sample and worst sample .. GENERATED FROM PYTHON SOURCE LINES 50-52 .. code-block:: default exp.model_diagnose(model="XGB2", show="resilience_shift_density", resilience_method="worst-sample", show_feature="PAY_1", original_scale=True, figsize=(5, 4)) .. image-sg:: /auto_examples/4_testing/images/sphx_glr_plot_5_resilience_cls_009.png :alt: Distribution Shift: 10%-Worst vs Remaining :srcset: /auto_examples/4_testing/images/sphx_glr_plot_5_resilience_cls_009.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-timing **Total running time of the script:** ( 1 minutes 41.200 seconds) **Estimated memory usage:** 56 MB .. _sphx_glr_download_auto_examples_4_testing_plot_5_resilience_cls.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: binder-badge .. image:: images/binder_badge_logo.svg :target: https://mybinder.org/v2/gh/selfexplainml/piml-toolbox/main?urlpath=lab/tree/./docs/_build/html/notebooks/auto_examples/4_testing/plot_5_resilience_cls.ipynb :alt: Launch binder :width: 150 px .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_5_resilience_cls.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_5_resilience_cls.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_