.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples\4_testing\plot_7_segmented_cls.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_4_testing_plot_7_segmented_cls.py>`
        to download the full example code or to run this example in your browser via Binder

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_4_testing_plot_7_segmented_cls.py:


Segmented Diagnose (Classification)
========================================================

.. GENERATED FROM PYTHON SOURCE LINES 8-9

Experiment initialization and data preparation

.. GENERATED FROM PYTHON SOURCE LINES 9-17

.. code-block:: default

    from piml import Experiment
    from piml.models import XGB2Classifier

    exp = Experiment()
    exp.data_loader("SimuCredit", silent=True)
    exp.data_summary(feature_exclude=["Race", "Gender"], silent=True)
    exp.data_prepare(target="Approved", task_type="classification", silent=True)








.. GENERATED FROM PYTHON SOURCE LINES 18-19

Train Model

.. GENERATED FROM PYTHON SOURCE LINES 19-21

.. code-block:: default

    exp.model_train(XGB2Classifier(), name="XGB2")








.. GENERATED FROM PYTHON SOURCE LINES 22-23

Summary of all segments (top 10 with the worst performance)

.. GENERATED FROM PYTHON SOURCE LINES 23-26

.. code-block:: default

    result = exp.segmented_diagnose(model="XGB2", show="segment_table", segment_method="auto", return_data=True)
    result.data.head(10)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    <pandas.io.formats.style.Styler object at 0x0000020332A6B820>


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>Segment ID</th>
          <th>Feature</th>
          <th>Segment</th>
          <th>Size</th>
          <th>ACC</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0</td>
          <td>Balance</td>
          <td>[0.1831, 0.2088)</td>
          <td>63</td>
          <td>0.539683</td>
        </tr>
        <tr>
          <th>1</th>
          <td>1</td>
          <td>Mortgage</td>
          <td>[0.0447, 0.064)</td>
          <td>487</td>
          <td>0.597536</td>
        </tr>
        <tr>
          <th>2</th>
          <td>2</td>
          <td>Mortgage</td>
          <td>[0.072, 0.0761)</td>
          <td>87</td>
          <td>0.597701</td>
        </tr>
        <tr>
          <th>3</th>
          <td>3</td>
          <td>Balance</td>
          <td>[-inf, 0.011)</td>
          <td>575</td>
          <td>0.603478</td>
        </tr>
        <tr>
          <th>4</th>
          <td>4</td>
          <td>Mortgage</td>
          <td>[-inf, 0.0234)</td>
          <td>176</td>
          <td>0.619318</td>
        </tr>
        <tr>
          <th>5</th>
          <td>5</td>
          <td>Mortgage</td>
          <td>[0.0404, 0.0447)</td>
          <td>101</td>
          <td>0.623762</td>
        </tr>
        <tr>
          <th>6</th>
          <td>6</td>
          <td>Utilization</td>
          <td>[0.2221, 0.309)</td>
          <td>538</td>
          <td>0.624535</td>
        </tr>
        <tr>
          <th>7</th>
          <td>7</td>
          <td>Mortgage</td>
          <td>[0.2859, 0.316)</td>
          <td>56</td>
          <td>0.625000</td>
        </tr>
        <tr>
          <th>8</th>
          <td>8</td>
          <td>Mortgage</td>
          <td>[0.0234, 0.0404)</td>
          <td>384</td>
          <td>0.635417</td>
        </tr>
        <tr>
          <th>9</th>
          <td>9</td>
          <td>Utilization</td>
          <td>[-inf, 0.2221)</td>
          <td>999</td>
          <td>0.644645</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 27-28

Summary of all segments of a given feature (top 10 with the worst performance)

.. GENERATED FROM PYTHON SOURCE LINES 28-32

.. code-block:: default

    result = exp.segmented_diagnose(model="XGB2", show="segment_table", segment_method="auto",
                                    segment_feature="Balance", return_data=True)
    result.data





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    <pandas.io.formats.style.Styler object at 0x00000203172F50D0>


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>Segment ID</th>
          <th>Feature</th>
          <th>Segment</th>
          <th>Size</th>
          <th>ACC</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0</td>
          <td>Balance</td>
          <td>[0.1831, 0.2088)</td>
          <td>63</td>
          <td>0.539683</td>
        </tr>
        <tr>
          <th>1</th>
          <td>1</td>
          <td>Balance</td>
          <td>[-inf, 0.011)</td>
          <td>575</td>
          <td>0.603478</td>
        </tr>
        <tr>
          <th>2</th>
          <td>2</td>
          <td>Balance</td>
          <td>[0.011, 0.0184)</td>
          <td>446</td>
          <td>0.650224</td>
        </tr>
        <tr>
          <th>3</th>
          <td>3</td>
          <td>Balance</td>
          <td>[0.0184, 0.0211)</td>
          <td>158</td>
          <td>0.658228</td>
        </tr>
        <tr>
          <th>4</th>
          <td>4</td>
          <td>Balance</td>
          <td>[0.0211, 0.0469)</td>
          <td>1119</td>
          <td>0.675603</td>
        </tr>
        <tr>
          <th>5</th>
          <td>5</td>
          <td>Balance</td>
          <td>[0.2804, inf]</td>
          <td>45</td>
          <td>0.688889</td>
        </tr>
        <tr>
          <th>6</th>
          <td>6</td>
          <td>Balance</td>
          <td>[0.0469, 0.1831)</td>
          <td>1506</td>
          <td>0.724436</td>
        </tr>
        <tr>
          <th>7</th>
          <td>7</td>
          <td>Balance</td>
          <td>[0.2088, 0.2804)</td>
          <td>88</td>
          <td>0.795455</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 33-34

Accuracy talbe of the samples in that segment

.. GENERATED FROM PYTHON SOURCE LINES 34-37

.. code-block:: default

    exp.segmented_diagnose(model="XGB2", show="accuracy_table", 
                           segment_id=0, segment_method="auto", segment_feature="Balance")





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

               ACC      AUC       F1 LogLoss   Brier
                                                
    Train   0.7563   0.8512   0.7622  0.4899  0.1604
    Test    0.5397   0.5777   0.5538  0.7566  0.2760
    Gap    -0.2166  -0.2735  -0.2084  0.2667  0.1156




.. GENERATED FROM PYTHON SOURCE LINES 38-39

Residual analysis of the samples in that segment

.. GENERATED FROM PYTHON SOURCE LINES 39-43

.. code-block:: default

    exp.segmented_diagnose(model="XGB2", show="accuracy_residual", 
                           segment_id=0, segment_method="auto", segment_feature="Balance",
                           show_feature="Mortgage", figsize=(5, 4))




.. image-sg:: /auto_examples/4_testing/images/sphx_glr_plot_7_segmented_cls_001.png
   :alt: Residual Plot
   :srcset: /auto_examples/4_testing/images/sphx_glr_plot_7_segmented_cls_001.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 44-45

Weakspot analysis of the samples in that segment

.. GENERATED FROM PYTHON SOURCE LINES 45-49

.. code-block:: default

    exp.segmented_diagnose(model="XGB2", show="weakspot", 
                           segment_id=0, segment_method="auto", segment_feature="Balance",
                           slice_features=["Mortgage"], metric="AUC", figsize=(5, 4))




.. image-sg:: /auto_examples/4_testing/images/sphx_glr_plot_7_segmented_cls_002.png
   :alt: Weak Regions
   :srcset: /auto_examples/4_testing/images/sphx_glr_plot_7_segmented_cls_002.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 50-51

Distributional distance comparison between the specificed segment and the remaining (feature-by-feature)

.. GENERATED FROM PYTHON SOURCE LINES 51-55

.. code-block:: default

    res = exp.segmented_diagnose(model="XGB2", show="distribution_shift",
                                 segment_id=0, segment_method="auto", segment_feature="Balance",
                                 figsize=(5, 4), return_data=True)




.. image-sg:: /auto_examples/4_testing/images/sphx_glr_plot_7_segmented_cls_003.png
   :alt: Data distance(In segment vs. out of segment)
   :srcset: /auto_examples/4_testing/images/sphx_glr_plot_7_segmented_cls_003.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 56-57

Distributional distance comparison between the specificed segment and the remaining (density of one selected feature)

.. GENERATED FROM PYTHON SOURCE LINES 57-60

.. code-block:: default

    res = exp.segmented_diagnose(model="XGB2", show="distribution_shift",
                                 segment_id=0, segment_method="auto", segment_feature="Balance",
                                 show_feature="Mortgage", figsize=(5, 4), return_data=True)



.. image-sg:: /auto_examples/4_testing/images/sphx_glr_plot_7_segmented_cls_004.png
   :alt: Distribution plot
   :srcset: /auto_examples/4_testing/images/sphx_glr_plot_7_segmented_cls_004.png
   :class: sphx-glr-single-img






.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 0 minutes  51.680 seconds)

**Estimated memory usage:**  41 MB


.. _sphx_glr_download_auto_examples_4_testing_plot_7_segmented_cls.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example


    .. container:: binder-badge

      .. image:: images/binder_badge_logo.svg
        :target: https://mybinder.org/v2/gh/selfexplainml/piml-toolbox/main?urlpath=lab/tree/./docs/_build/html/notebooks/auto_examples/4_testing/plot_7_segmented_cls.ipynb
        :alt: Launch binder
        :width: 150 px



    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_7_segmented_cls.py <plot_7_segmented_cls.py>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_7_segmented_cls.ipynb <plot_7_segmented_cls.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_