.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples\4_testing\plot_2_overfit_cls.py"
.. LINE NUMBERS ARE GIVEN BELOW.
.. only:: html
.. note::
:class: sphx-glr-download-link-note
:ref:`Go to the end <sphx_glr_download_auto_examples_4_testing_plot_2_overfit_cls.py>`
to download the full example code or to run this example in your browser via Binder
.. rst-class:: sphx-glr-example-title
.. _sphx_glr_auto_examples_4_testing_plot_2_overfit_cls.py:
Overfit: Classification
=====================================
.. GENERATED FROM PYTHON SOURCE LINES 8-9
Experiment initialization and data preparation
.. GENERATED FROM PYTHON SOURCE LINES 9-12
.. code-block:: default
from piml import Experiment
from piml.models import XGB2Classifier
.. GENERATED FROM PYTHON SOURCE LINES 13-14
Load data and train models.
.. GENERATED FROM PYTHON SOURCE LINES 14-19
.. code-block:: default
exp = Experiment()
exp.data_loader(data="TaiwanCredit", silent=True)
exp.data_summary(feature_exclude=["LIMIT_BAL", "SEX", "EDUCATION", "MARRIAGE", "AGE"], silent=True)
exp.data_prepare(target="FlagDefault", task_type="classification", silent=True)
.. GENERATED FROM PYTHON SOURCE LINES 20-21
Train Model
.. GENERATED FROM PYTHON SOURCE LINES 21-23
.. code-block:: default
exp.model_train(XGB2Classifier(), name="XGB2")
.. GENERATED FROM PYTHON SOURCE LINES 24-25
Histogram-based overfit test for a single feature
.. GENERATED FROM PYTHON SOURCE LINES 25-29
.. code-block:: default
results = exp.model_diagnose(model="XGB2", show="overfit", slice_method="histogram",
slice_features=["BILL_AMT1"], threshold=1.05, min_samples=20,
original_scale=True, return_data=True, figsize=(5, 4))
results.data
.. image-sg:: /auto_examples/4_testing/images/sphx_glr_plot_2_overfit_cls_001.png
:alt: Overfit Regions
:srcset: /auto_examples/4_testing/images/sphx_glr_plot_2_overfit_cls_001.png
:class: sphx-glr-single-img
.. raw:: html
<div class="output_subarea output_html rendered_html output_result">
<div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>[BILL_AMT1</th>
<th>BILL_AMT1)</th>
<th>#Test</th>
<th>#Train</th>
<th>test_ACC</th>
<th>train_ACC</th>
<th>Gap</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>0.2879</td>
<td>0.3781</td>
<td>41</td>
<td>131</td>
<td>0.7561</td>
<td>0.8473</td>
<td>-0.0912</td>
</tr>
</tbody>
</table>
</div>
</div>
<br />
<br />
.. GENERATED FROM PYTHON SOURCE LINES 30-31
Histogram-based overfit test for two features
.. GENERATED FROM PYTHON SOURCE LINES 31-36
.. code-block:: default
results = exp.model_diagnose(model="XGB2", show="overfit", slice_method="histogram",
slice_features=["PAY_1", "BILL_AMT1"], threshold=1.05, min_samples=20,
original_scale=True, return_data=True, figsize=(5, 4))
results.data
.. image-sg:: /auto_examples/4_testing/images/sphx_glr_plot_2_overfit_cls_002.png
:alt: Overfit Regions
:srcset: /auto_examples/4_testing/images/sphx_glr_plot_2_overfit_cls_002.png
:class: sphx-glr-single-img
.. raw:: html
<div class="output_subarea output_html rendered_html output_result">
<div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>[PAY_1</th>
<th>PAY_1)</th>
<th>[BILL_AMT1</th>
<th>BILL_AMT1)</th>
<th>#Test</th>
<th>#Train</th>
<th>test_ACC</th>
<th>train_ACC</th>
<th>Gap</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>0.2222</td>
<td>0.3333</td>
<td>0.7167</td>
<td>0.8037</td>
<td>30</td>
<td>117</td>
<td>0.6333</td>
<td>0.7094</td>
<td>-0.0761</td>
</tr>
<tr>
<th>1</th>
<td>0.3333</td>
<td>0.4444</td>
<td>0.7311</td>
<td>0.7953</td>
<td>40</td>
<td>160</td>
<td>0.6000</td>
<td>0.7000</td>
<td>-0.1000</td>
</tr>
<tr>
<th>2</th>
<td>0.3333</td>
<td>0.4444</td>
<td>0.9237</td>
<td>0.9558</td>
<td>69</td>
<td>338</td>
<td>0.5797</td>
<td>0.6834</td>
<td>-0.1037</td>
</tr>
<tr>
<th>3</th>
<td>0.2222</td>
<td>0.3333</td>
<td>0.2814</td>
<td>0.3685</td>
<td>24</td>
<td>75</td>
<td>0.7083</td>
<td>0.8133</td>
<td>-0.1050</td>
</tr>
<tr>
<th>4</th>
<td>0.0000</td>
<td>0.1111</td>
<td>0.6213</td>
<td>0.6715</td>
<td>20</td>
<td>73</td>
<td>0.6500</td>
<td>0.7808</td>
<td>-0.1308</td>
</tr>
</tbody>
</table>
</div>
</div>
<br />
<br />
.. GENERATED FROM PYTHON SOURCE LINES 37-38
Histogram-based overfit test for a single feature on test set
.. GENERATED FROM PYTHON SOURCE LINES 38-42
.. code-block:: default
results = exp.model_diagnose(model="XGB2", show="overfit", slice_method="histogram",
slice_features=["BILL_AMT1"], threshold=1.05, min_samples=20,
use_test=True, original_scale=True, return_data=True, figsize=(5, 4))
results.data
.. image-sg:: /auto_examples/4_testing/images/sphx_glr_plot_2_overfit_cls_003.png
:alt: Overfit Regions
:srcset: /auto_examples/4_testing/images/sphx_glr_plot_2_overfit_cls_003.png
:class: sphx-glr-single-img
.. raw:: html
<div class="output_subarea output_html rendered_html output_result">
<div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>[BILL_AMT1</th>
<th>BILL_AMT1)</th>
<th>#Test</th>
<th>#Train</th>
<th>test_ACC</th>
<th>train_ACC</th>
<th>Gap</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>0.2879</td>
<td>0.3781</td>
<td>41</td>
<td>131</td>
<td>0.7561</td>
<td>0.8473</td>
<td>-0.0912</td>
</tr>
</tbody>
</table>
</div>
</div>
<br />
<br />
.. rst-class:: sphx-glr-timing
**Total running time of the script:** ( 0 minutes 35.038 seconds)
**Estimated memory usage:** 40 MB
.. _sphx_glr_download_auto_examples_4_testing_plot_2_overfit_cls.py:
.. only:: html
.. container:: sphx-glr-footer sphx-glr-footer-example
.. container:: binder-badge
.. image:: images/binder_badge_logo.svg
:target: https://mybinder.org/v2/gh/selfexplainml/piml-toolbox/main?urlpath=lab/tree/./docs/_build/html/notebooks/auto_examples/4_testing/plot_2_overfit_cls.ipynb
:alt: Launch binder
:width: 150 px
.. container:: sphx-glr-download sphx-glr-download-python
:download:`Download Python source code: plot_2_overfit_cls.py <plot_2_overfit_cls.py>`
.. container:: sphx-glr-download sphx-glr-download-jupyter
:download:`Download Jupyter notebook: plot_2_overfit_cls.ipynb <plot_2_overfit_cls.ipynb>`
.. only:: html
.. rst-class:: sphx-glr-signature
`Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_