.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples\4_testing\plot_1_weakspot_cls.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_4_testing_plot_1_weakspot_cls.py>`
        to download the full example code or to run this example in your browser via Binder

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_4_testing_plot_1_weakspot_cls.py:


WeakSpot: Classification    
=====================================

.. GENERATED FROM PYTHON SOURCE LINES 8-9

Experiment initialization and data preparation

.. GENERATED FROM PYTHON SOURCE LINES 9-17

.. code-block:: default

    from piml import Experiment
    from piml.models import XGB2Classifier

    exp = Experiment()
    exp.data_loader(data="TaiwanCredit", silent=True)
    exp.data_summary(feature_exclude=["LIMIT_BAL", "SEX", "EDUCATION", "MARRIAGE", "AGE"], silent=True)
    exp.data_prepare(target="FlagDefault", task_type="classification", silent=True)








.. GENERATED FROM PYTHON SOURCE LINES 18-19

Train Model

.. GENERATED FROM PYTHON SOURCE LINES 19-21

.. code-block:: default

    exp.model_train(XGB2Classifier(), name="XGB2")








.. GENERATED FROM PYTHON SOURCE LINES 22-23

Histogram-based weakspot for a single feature

.. GENERATED FROM PYTHON SOURCE LINES 23-28

.. code-block:: default

    results = exp.model_diagnose(model="XGB2", show="weakspot", slice_method="histogram", 
                                 slice_features=["PAY_1"], threshold=1.1, min_samples=100,
                                 return_data=True, figsize=(5, 4))
    results.data




.. image-sg:: /auto_examples/4_testing/images/sphx_glr_plot_1_weakspot_cls_001.png
   :alt: Weak Regions
   :srcset: /auto_examples/4_testing/images/sphx_glr_plot_1_weakspot_cls_001.png
   :class: sphx-glr-single-img



.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>[PAY_1</th>
          <th>PAY_1)</th>
          <th>#Test</th>
          <th>#Train</th>
          <th>test_ACC</th>
          <th>train_ACC</th>
          <th>Gap</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0.2</td>
          <td>0.8</td>
          <td>1350</td>
          <td>5440</td>
          <td>0.7052</td>
          <td>0.6888</td>
          <td>0.0164</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 29-30

Histogram-based weakspot for two features

.. GENERATED FROM PYTHON SOURCE LINES 30-35

.. code-block:: default

    results = exp.model_diagnose(model="XGB2", show="weakspot", slice_method="histogram", 
                                 slice_features=["PAY_1", "PAY_2"], threshold=1.1, min_samples=100,
                                 return_data=True, figsize=(5, 4))
    results.data




.. image-sg:: /auto_examples/4_testing/images/sphx_glr_plot_1_weakspot_cls_002.png
   :alt: Weak Regions
   :srcset: /auto_examples/4_testing/images/sphx_glr_plot_1_weakspot_cls_002.png
   :class: sphx-glr-single-img



.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>[PAY_1</th>
          <th>PAY_1)</th>
          <th>[PAY_2</th>
          <th>PAY_2)</th>
          <th>#Test</th>
          <th>#Train</th>
          <th>test_ACC</th>
          <th>train_ACC</th>
          <th>Gap</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0.4</td>
          <td>0.5</td>
          <td>0.3750</td>
          <td>0.5625</td>
          <td>69</td>
          <td>244</td>
          <td>0.8551</td>
          <td>0.7336</td>
          <td>0.1215</td>
        </tr>
        <tr>
          <th>1</th>
          <td>0.3</td>
          <td>0.4</td>
          <td>0.0000</td>
          <td>0.2500</td>
          <td>191</td>
          <td>800</td>
          <td>0.6963</td>
          <td>0.6675</td>
          <td>0.0288</td>
        </tr>
        <tr>
          <th>2</th>
          <td>0.2</td>
          <td>0.3</td>
          <td>0.1111</td>
          <td>0.2222</td>
          <td>268</td>
          <td>956</td>
          <td>0.7276</td>
          <td>0.7207</td>
          <td>0.0069</td>
        </tr>
        <tr>
          <th>3</th>
          <td>0.3</td>
          <td>0.4</td>
          <td>0.3750</td>
          <td>0.6250</td>
          <td>322</td>
          <td>1354</td>
          <td>0.7205</td>
          <td>0.7164</td>
          <td>0.0041</td>
        </tr>
        <tr>
          <th>4</th>
          <td>0.2</td>
          <td>0.3</td>
          <td>0.3333</td>
          <td>0.5556</td>
          <td>351</td>
          <td>1430</td>
          <td>0.6154</td>
          <td>0.6119</td>
          <td>0.0035</td>
        </tr>
        <tr>
          <th>5</th>
          <td>0.0</td>
          <td>0.1</td>
          <td>0.3750</td>
          <td>0.6250</td>
          <td>77</td>
          <td>357</td>
          <td>0.6234</td>
          <td>0.6303</td>
          <td>-0.0069</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 36-37

Histogram-based weakspot for a single feature on test set

.. GENERATED FROM PYTHON SOURCE LINES 37-41

.. code-block:: default

    results = exp.model_diagnose(model="XGB2", show="weakspot", slice_method="histogram", 
                                 slice_features=["PAY_1"], threshold=1.1, min_samples=100,
                                 use_test=True, return_data=True, figsize=(5, 4))
    results.data



.. image-sg:: /auto_examples/4_testing/images/sphx_glr_plot_1_weakspot_cls_003.png
   :alt: Weak Regions
   :srcset: /auto_examples/4_testing/images/sphx_glr_plot_1_weakspot_cls_003.png
   :class: sphx-glr-single-img



.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>[PAY_1</th>
          <th>PAY_1)</th>
          <th>#Test</th>
          <th>#Train</th>
          <th>test_ACC</th>
          <th>train_ACC</th>
          <th>Gap</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0.2222</td>
          <td>0.4444</td>
          <td>1265</td>
          <td>5090</td>
          <td>0.6964</td>
          <td>0.6876</td>
          <td>0.0088</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 42-43

Histogram-based weakspot for a single feature using AUC metric

.. GENERATED FROM PYTHON SOURCE LINES 43-48

.. code-block:: default

    results = exp.model_diagnose(model="XGB2", show="weakspot", slice_method="histogram", 
                                 slice_features=["PAY_1"], threshold=1.1, min_samples=100,
                                 metric="AUC", return_data=True, figsize=(5, 4))
    results.data




.. image-sg:: /auto_examples/4_testing/images/sphx_glr_plot_1_weakspot_cls_004.png
   :alt: Weak Regions
   :srcset: /auto_examples/4_testing/images/sphx_glr_plot_1_weakspot_cls_004.png
   :class: sphx-glr-single-img



.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>[PAY_1</th>
          <th>PAY_1)</th>
          <th>#Test</th>
          <th>#Train</th>
          <th>test_AUC</th>
          <th>train_AUC</th>
          <th>Gap</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0.2</td>
          <td>0.3</td>
          <td>752</td>
          <td>2936</td>
          <td>0.6478</td>
          <td>0.7071</td>
          <td>-0.0593</td>
        </tr>
        <tr>
          <th>1</th>
          <td>0.1</td>
          <td>0.2</td>
          <td>3521</td>
          <td>13975</td>
          <td>0.6350</td>
          <td>0.6970</td>
          <td>-0.0620</td>
        </tr>
        <tr>
          <th>2</th>
          <td>0.3</td>
          <td>0.4</td>
          <td>513</td>
          <td>2154</td>
          <td>0.5481</td>
          <td>0.6325</td>
          <td>-0.0844</td>
        </tr>
        <tr>
          <th>3</th>
          <td>0.5</td>
          <td>0.6</td>
          <td>14</td>
          <td>62</td>
          <td>0.5250</td>
          <td>0.7179</td>
          <td>-0.1929</td>
        </tr>
        <tr>
          <th>4</th>
          <td>0.6</td>
          <td>0.7</td>
          <td>1</td>
          <td>25</td>
          <td>NaN</td>
          <td>0.6314</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>5</th>
          <td>0.7</td>
          <td>0.8</td>
          <td>1</td>
          <td>10</td>
          <td>NaN</td>
          <td>0.4400</td>
          <td>NaN</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 49-50

Tree-based weakspot for a single feature using ACC metric

.. GENERATED FROM PYTHON SOURCE LINES 50-55

.. code-block:: default

    results = exp.model_diagnose(model="XGB2", show="weakspot", slice_method="tree", 
                                 slice_features=["PAY_1"], threshold=1.1, min_samples=100,
                                 metric="ACC", return_data=True, figsize=(5, 4))
    results.data




.. image-sg:: /auto_examples/4_testing/images/sphx_glr_plot_1_weakspot_cls_005.png
   :alt: Weak Regions
   :srcset: /auto_examples/4_testing/images/sphx_glr_plot_1_weakspot_cls_005.png
   :class: sphx-glr-single-img



.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>[PAY_1</th>
          <th>PAY_1)</th>
          <th>#Test</th>
          <th>#Train</th>
          <th>test_ACC</th>
          <th>train_ACC</th>
          <th>Gap</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0.2778</td>
          <td>1.0000</td>
          <td>602</td>
          <td>2528</td>
          <td>0.7276</td>
          <td>0.6994</td>
          <td>0.0282</td>
        </tr>
        <tr>
          <th>1</th>
          <td>0.1667</td>
          <td>0.2778</td>
          <td>752</td>
          <td>2936</td>
          <td>0.6862</td>
          <td>0.6798</td>
          <td>0.0063</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 0 minutes  44.136 seconds)

**Estimated memory usage:**  43 MB


.. _sphx_glr_download_auto_examples_4_testing_plot_1_weakspot_cls.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example


    .. container:: binder-badge

      .. image:: images/binder_badge_logo.svg
        :target: https://mybinder.org/v2/gh/selfexplainml/piml-toolbox/main?urlpath=lab/tree/./docs/_build/html/notebooks/auto_examples/4_testing/plot_1_weakspot_cls.ipynb
        :alt: Launch binder
        :width: 150 px



    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_1_weakspot_cls.py <plot_1_weakspot_cls.py>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_1_weakspot_cls.ipynb <plot_1_weakspot_cls.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_