.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples\0_data\plot_4_data_quality.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_0_data_plot_4_data_quality.py>`
        to download the full example code or to run this example in your browser via Binder

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_0_data_plot_4_data_quality.py:


Data Quality Check
=====================================
Data quality analysis result using the BikeSharing dataset as example

.. GENERATED FROM PYTHON SOURCE LINES 9-10

Experiment initialization and data preparation

.. GENERATED FROM PYTHON SOURCE LINES 10-19

.. code-block:: default

    from piml import Experiment
    from piml.data.outlier_detection import (PCA, CBLOF, IsolationForest, KMeansTree, 
                                             OneClassSVM, KNN, HBOS, ECOD)

    exp = Experiment()
    exp.data_loader(data="BikeSharing", silent=True)
    exp.data_summary(feature_exclude=["yr", "mnth", "temp"], silent=True)
    exp.data_prepare(target="cnt", task_type="regression", silent=True)








.. GENERATED FROM PYTHON SOURCE LINES 20-21

Data integrity check for each column 

.. GENERATED FROM PYTHON SOURCE LINES 21-24

.. code-block:: default

    res = exp.data_quality(show='integrity_single_column_check', return_data=True)
    res.data





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    <pandas.io.formats.style.Styler object at 0x0000022054BE01C0>


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <style type="text/css">
    #T_70230 th.col_heading {
      text-align: center;
    }
    #T_70230 th.col_heading.level0 {
      font-size: 1.0em;
    }
    #T_70230 th {
      border-style: solid;
      border-width: 1px;
    }
    #T_70230 td {
      border-style: solid;
      border-width: 1px;
    }
    </style>
    <table id="T_70230">
      <thead>
        <tr>
          <th class="blank level0" >&nbsp;</th>
          <th id="T_70230_level0_col0" class="col_heading level0 col0" >Is Single Value</th>
          <th id="T_70230_level0_col1" class="col_heading level0 col1" >Null Ratio</th>
          <th id="T_70230_level0_col2" class="col_heading level0 col2" colspan="2">Mixed Data Types</th>
          <th id="T_70230_level0_col4" class="col_heading level0 col4" colspan="2">Long String</th>
          <th id="T_70230_level0_col6" class="col_heading level0 col6" colspan="2">Special Characters</th>
          <th id="T_70230_level0_col8" class="col_heading level0 col8" colspan="2">New Categories</th>
        </tr>
        <tr>
          <th class="blank level1" >&nbsp;</th>
          <th id="T_70230_level1_col0" class="col_heading level1 col0" ></th>
          <th id="T_70230_level1_col1" class="col_heading level1 col1" ></th>
          <th id="T_70230_level1_col2" class="col_heading level1 col2" >Mixed</th>
          <th id="T_70230_level1_col3" class="col_heading level1 col3" >Categorical : Numerical</th>
          <th id="T_70230_level1_col4" class="col_heading level1 col4" >Num</th>
          <th id="T_70230_level1_col5" class="col_heading level1 col5" >Index</th>
          <th id="T_70230_level1_col6" class="col_heading level1 col6" >Ratio</th>
          <th id="T_70230_level1_col7" class="col_heading level1 col7" >Example Samples</th>
          <th id="T_70230_level1_col8" class="col_heading level1 col8" >Num</th>
          <th id="T_70230_level1_col9" class="col_heading level1 col9" >Example Samples</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th id="T_70230_level0_row0" class="row_heading level0 row0" >season</th>
          <td id="T_70230_row0_col0" class="data row0 col0" >False</td>
          <td id="T_70230_row0_col1" class="data row0 col1" >0.000000</td>
          <td id="T_70230_row0_col2" class="data row0 col2" >False</td>
          <td id="T_70230_row0_col3" class="data row0 col3" >-</td>
          <td id="T_70230_row0_col4" class="data row0 col4" >0</td>
          <td id="T_70230_row0_col5" class="data row0 col5" >[]</td>
          <td id="T_70230_row0_col6" class="data row0 col6" >0.000000</td>
          <td id="T_70230_row0_col7" class="data row0 col7" >[]</td>
          <td id="T_70230_row0_col8" class="data row0 col8" >0</td>
          <td id="T_70230_row0_col9" class="data row0 col9" >[]</td>
        </tr>
        <tr>
          <th id="T_70230_level0_row1" class="row_heading level0 row1" >hr</th>
          <td id="T_70230_row1_col0" class="data row1 col0" >False</td>
          <td id="T_70230_row1_col1" class="data row1 col1" >0.000000</td>
          <td id="T_70230_row1_col2" class="data row1 col2" >False</td>
          <td id="T_70230_row1_col3" class="data row1 col3" >-</td>
          <td id="T_70230_row1_col4" class="data row1 col4" >0</td>
          <td id="T_70230_row1_col5" class="data row1 col5" >[]</td>
          <td id="T_70230_row1_col6" class="data row1 col6" >0.000000</td>
          <td id="T_70230_row1_col7" class="data row1 col7" >[]</td>
          <td id="T_70230_row1_col8" class="data row1 col8" >-</td>
          <td id="T_70230_row1_col9" class="data row1 col9" >-</td>
        </tr>
        <tr>
          <th id="T_70230_level0_row2" class="row_heading level0 row2" >holiday</th>
          <td id="T_70230_row2_col0" class="data row2 col0" >False</td>
          <td id="T_70230_row2_col1" class="data row2 col1" >0.000000</td>
          <td id="T_70230_row2_col2" class="data row2 col2" >False</td>
          <td id="T_70230_row2_col3" class="data row2 col3" >-</td>
          <td id="T_70230_row2_col4" class="data row2 col4" >0</td>
          <td id="T_70230_row2_col5" class="data row2 col5" >[]</td>
          <td id="T_70230_row2_col6" class="data row2 col6" >0.000000</td>
          <td id="T_70230_row2_col7" class="data row2 col7" >[]</td>
          <td id="T_70230_row2_col8" class="data row2 col8" >0</td>
          <td id="T_70230_row2_col9" class="data row2 col9" >[]</td>
        </tr>
        <tr>
          <th id="T_70230_level0_row3" class="row_heading level0 row3" >weekday</th>
          <td id="T_70230_row3_col0" class="data row3 col0" >False</td>
          <td id="T_70230_row3_col1" class="data row3 col1" >0.000000</td>
          <td id="T_70230_row3_col2" class="data row3 col2" >False</td>
          <td id="T_70230_row3_col3" class="data row3 col3" >-</td>
          <td id="T_70230_row3_col4" class="data row3 col4" >0</td>
          <td id="T_70230_row3_col5" class="data row3 col5" >[]</td>
          <td id="T_70230_row3_col6" class="data row3 col6" >0.000000</td>
          <td id="T_70230_row3_col7" class="data row3 col7" >[]</td>
          <td id="T_70230_row3_col8" class="data row3 col8" >-</td>
          <td id="T_70230_row3_col9" class="data row3 col9" >-</td>
        </tr>
        <tr>
          <th id="T_70230_level0_row4" class="row_heading level0 row4" >workingday</th>
          <td id="T_70230_row4_col0" class="data row4 col0" >False</td>
          <td id="T_70230_row4_col1" class="data row4 col1" >0.000000</td>
          <td id="T_70230_row4_col2" class="data row4 col2" >False</td>
          <td id="T_70230_row4_col3" class="data row4 col3" >-</td>
          <td id="T_70230_row4_col4" class="data row4 col4" >0</td>
          <td id="T_70230_row4_col5" class="data row4 col5" >[]</td>
          <td id="T_70230_row4_col6" class="data row4 col6" >0.000000</td>
          <td id="T_70230_row4_col7" class="data row4 col7" >[]</td>
          <td id="T_70230_row4_col8" class="data row4 col8" >0</td>
          <td id="T_70230_row4_col9" class="data row4 col9" >[]</td>
        </tr>
        <tr>
          <th id="T_70230_level0_row5" class="row_heading level0 row5" >weathersit</th>
          <td id="T_70230_row5_col0" class="data row5 col0" >False</td>
          <td id="T_70230_row5_col1" class="data row5 col1" >0.000000</td>
          <td id="T_70230_row5_col2" class="data row5 col2" >False</td>
          <td id="T_70230_row5_col3" class="data row5 col3" >-</td>
          <td id="T_70230_row5_col4" class="data row5 col4" >0</td>
          <td id="T_70230_row5_col5" class="data row5 col5" >[]</td>
          <td id="T_70230_row5_col6" class="data row5 col6" >0.000000</td>
          <td id="T_70230_row5_col7" class="data row5 col7" >[]</td>
          <td id="T_70230_row5_col8" class="data row5 col8" >0</td>
          <td id="T_70230_row5_col9" class="data row5 col9" >[]</td>
        </tr>
        <tr>
          <th id="T_70230_level0_row6" class="row_heading level0 row6" >atemp</th>
          <td id="T_70230_row6_col0" class="data row6 col0" >False</td>
          <td id="T_70230_row6_col1" class="data row6 col1" >0.000000</td>
          <td id="T_70230_row6_col2" class="data row6 col2" >False</td>
          <td id="T_70230_row6_col3" class="data row6 col3" >-</td>
          <td id="T_70230_row6_col4" class="data row6 col4" >0</td>
          <td id="T_70230_row6_col5" class="data row6 col5" >[]</td>
          <td id="T_70230_row6_col6" class="data row6 col6" >0.000000</td>
          <td id="T_70230_row6_col7" class="data row6 col7" >[]</td>
          <td id="T_70230_row6_col8" class="data row6 col8" >-</td>
          <td id="T_70230_row6_col9" class="data row6 col9" >-</td>
        </tr>
        <tr>
          <th id="T_70230_level0_row7" class="row_heading level0 row7" >hum</th>
          <td id="T_70230_row7_col0" class="data row7 col0" >False</td>
          <td id="T_70230_row7_col1" class="data row7 col1" >0.000000</td>
          <td id="T_70230_row7_col2" class="data row7 col2" >False</td>
          <td id="T_70230_row7_col3" class="data row7 col3" >-</td>
          <td id="T_70230_row7_col4" class="data row7 col4" >0</td>
          <td id="T_70230_row7_col5" class="data row7 col5" >[]</td>
          <td id="T_70230_row7_col6" class="data row7 col6" >0.000000</td>
          <td id="T_70230_row7_col7" class="data row7 col7" >[]</td>
          <td id="T_70230_row7_col8" class="data row7 col8" >-</td>
          <td id="T_70230_row7_col9" class="data row7 col9" >-</td>
        </tr>
        <tr>
          <th id="T_70230_level0_row8" class="row_heading level0 row8" >windspeed</th>
          <td id="T_70230_row8_col0" class="data row8 col0" >False</td>
          <td id="T_70230_row8_col1" class="data row8 col1" >0.000000</td>
          <td id="T_70230_row8_col2" class="data row8 col2" >False</td>
          <td id="T_70230_row8_col3" class="data row8 col3" >-</td>
          <td id="T_70230_row8_col4" class="data row8 col4" >0</td>
          <td id="T_70230_row8_col5" class="data row8 col5" >[]</td>
          <td id="T_70230_row8_col6" class="data row8 col6" >0.000000</td>
          <td id="T_70230_row8_col7" class="data row8 col7" >[]</td>
          <td id="T_70230_row8_col8" class="data row8 col8" >-</td>
          <td id="T_70230_row8_col9" class="data row8 col9" >-</td>
        </tr>
        <tr>
          <th id="T_70230_level0_row9" class="row_heading level0 row9" >cnt</th>
          <td id="T_70230_row9_col0" class="data row9 col0" >False</td>
          <td id="T_70230_row9_col1" class="data row9 col1" >0.000000</td>
          <td id="T_70230_row9_col2" class="data row9 col2" >False</td>
          <td id="T_70230_row9_col3" class="data row9 col3" >-</td>
          <td id="T_70230_row9_col4" class="data row9 col4" >0</td>
          <td id="T_70230_row9_col5" class="data row9 col5" >[]</td>
          <td id="T_70230_row9_col6" class="data row9 col6" >0.000000</td>
          <td id="T_70230_row9_col7" class="data row9 col7" >[]</td>
          <td id="T_70230_row9_col8" class="data row9 col8" >-</td>
          <td id="T_70230_row9_col9" class="data row9 col9" >-</td>
        </tr>
      </tbody>
    </table>

    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 25-26

Data integrity check for duplicated samples 

.. GENERATED FROM PYTHON SOURCE LINES 26-29

.. code-block:: default

    res = exp.data_quality(show='integrity_duplicated_samples', return_data=True)
    res.data





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

                    Leakage  season   hr  holiday  weekday  workingday  \
    [1507, 9867]      False     1.0  4.0      0.0      2.0         1.0   
    [9822, 17336]     False     1.0  5.0      0.0      0.0         0.0   
    [13559, 13727]     True     3.0  4.0      0.0      2.0         1.0   
    [5598, 14639]     False     3.0  4.0      0.0      5.0         1.0   
    [7958, 8126]      False     4.0  6.0      0.0      6.0         0.0   

                    weathersit   atemp   hum  windspeed   cnt  
    [1507, 9867]           1.0  0.2727  0.64     0.0000   2.0  
    [9822, 17336]          2.0  0.2273  0.48     0.2985   2.0  
    [13559, 13727]         1.0  0.6061  0.83     0.0896   6.0  
    [5598, 14639]          1.0  0.5606  0.88     0.0000   8.0  
    [7958, 8126]           1.0  0.2576  0.65     0.1045  11.0  


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>Leakage</th>
          <th>season</th>
          <th>hr</th>
          <th>holiday</th>
          <th>weekday</th>
          <th>workingday</th>
          <th>weathersit</th>
          <th>atemp</th>
          <th>hum</th>
          <th>windspeed</th>
          <th>cnt</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>[1507, 9867]</th>
          <td>False</td>
          <td>1.0</td>
          <td>4.0</td>
          <td>0.0</td>
          <td>2.0</td>
          <td>1.0</td>
          <td>1.0</td>
          <td>0.2727</td>
          <td>0.64</td>
          <td>0.0000</td>
          <td>2.0</td>
        </tr>
        <tr>
          <th>[9822, 17336]</th>
          <td>False</td>
          <td>1.0</td>
          <td>5.0</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>0.0</td>
          <td>2.0</td>
          <td>0.2273</td>
          <td>0.48</td>
          <td>0.2985</td>
          <td>2.0</td>
        </tr>
        <tr>
          <th>[13559, 13727]</th>
          <td>True</td>
          <td>3.0</td>
          <td>4.0</td>
          <td>0.0</td>
          <td>2.0</td>
          <td>1.0</td>
          <td>1.0</td>
          <td>0.6061</td>
          <td>0.83</td>
          <td>0.0896</td>
          <td>6.0</td>
        </tr>
        <tr>
          <th>[5598, 14639]</th>
          <td>False</td>
          <td>3.0</td>
          <td>4.0</td>
          <td>0.0</td>
          <td>5.0</td>
          <td>1.0</td>
          <td>1.0</td>
          <td>0.5606</td>
          <td>0.88</td>
          <td>0.0000</td>
          <td>8.0</td>
        </tr>
        <tr>
          <th>[7958, 8126]</th>
          <td>False</td>
          <td>4.0</td>
          <td>6.0</td>
          <td>0.0</td>
          <td>6.0</td>
          <td>0.0</td>
          <td>1.0</td>
          <td>0.2576</td>
          <td>0.65</td>
          <td>0.1045</td>
          <td>11.0</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 30-31

Data integrity check for correlated features 

.. GENERATED FROM PYTHON SOURCE LINES 31-34

.. code-block:: default

    res = exp.data_quality(show='integrity_highly_correlated_features', return_data=True)
    res.data





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    <pandas.io.formats.style.Styler object at 0x000002205340DDC0>


.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <style type="text/css">
    #T_e1a85_row0_col0, #T_e1a85_row1_col1, #T_e1a85_row2_col2, #T_e1a85_row3_col3, #T_e1a85_row4_col4, #T_e1a85_row5_col5, #T_e1a85_row6_col6, #T_e1a85_row7_col7, #T_e1a85_row8_col8, #T_e1a85_row9_col9 {
      background-color: #d73c5b;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row0_col1, #T_e1a85_row6_col7 {
      background-color: #91afd4;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row0_col2, #T_e1a85_row0_col4, #T_e1a85_row0_col5, #T_e1a85_row1_col2, #T_e1a85_row1_col4, #T_e1a85_row2_col0, #T_e1a85_row2_col5, #T_e1a85_row3_col5, #T_e1a85_row4_col0, #T_e1a85_row4_col5, #T_e1a85_row5_col0, #T_e1a85_row5_col2, #T_e1a85_row5_col4, #T_e1a85_row7_col1, #T_e1a85_row7_col3, #T_e1a85_row7_col6, #T_e1a85_row7_col8, #T_e1a85_row7_col9, #T_e1a85_row8_col2, #T_e1a85_row9_col7 {
      background-color: #4479bb;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row0_col3, #T_e1a85_row8_col3 {
      background-color: #5383bf;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row0_col6 {
      background-color: #e38b9d;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row0_col7 {
      background-color: #c8d5e5;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row0_col8 {
      background-color: #bdcde2;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row0_col9 {
      background-color: #e2e7ee;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row1_col0, #T_e1a85_row3_col0, #T_e1a85_row7_col2, #T_e1a85_row8_col4, #T_e1a85_row8_col6 {
      background-color: #477bbb;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row1_col3 {
      background-color: #4f80be;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row1_col5, #T_e1a85_row8_col7 {
      background-color: #5484c0;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row1_col6 {
      background-color: #81a3ce;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row1_col7 {
      background-color: #5887c1;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row1_col8 {
      background-color: #b8c9e0;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row1_col9 {
      background-color: #ebbfc8;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row2_col1, #T_e1a85_row3_col1 {
      background-color: #8eadd3;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row2_col3 {
      background-color: #7299c9;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row2_col4, #T_e1a85_row4_col2 {
      background-color: #628ec4;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row2_col6, #T_e1a85_row8_col5 {
      background-color: #5f8cc3;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row2_col7 {
      background-color: #a2bad9;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row2_col8 {
      background-color: #92b0d4;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row2_col9, #T_e1a85_row3_col9, #T_e1a85_row4_col9, #T_e1a85_row5_col8 {
      background-color: #a7bedb;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row3_col2, #T_e1a85_row4_col6 {
      background-color: #6791c6;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row3_col4, #T_e1a85_row3_col6, #T_e1a85_row5_col3 {
      background-color: #5081bf;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row3_col7, #T_e1a85_row3_col8, #T_e1a85_row4_col8 {
      background-color: #95b1d5;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row4_col1 {
      background-color: #90aed3;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row4_col3 {
      background-color: #5b89c2;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row4_col7 {
      background-color: #a3bbd9;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row5_col1 {
      background-color: #9eb8d8;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row5_col6 {
      background-color: #779dcb;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row5_col7 {
      background-color: #efd7dc;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row5_col9 {
      background-color: #c5d3e4;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row6_col0 {
      background-color: #e48fa0;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row6_col1 {
      background-color: #b4c7df;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row6_col2, #T_e1a85_row9_col2, #T_e1a85_row9_col4 {
      background-color: #4d7fbe;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row6_col3 {
      background-color: #4b7ebd;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row6_col4 {
      background-color: #5585c0;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row6_col5 {
      background-color: #6892c6;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row6_col8 {
      background-color: #88a8d0;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row6_col9 {
      background-color: #eed6db;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row7_col0 {
      background-color: #799ecc;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row7_col4 {
      background-color: #487cbc;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row7_col5 {
      background-color: #d4dde9;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row8_col0 {
      background-color: #7a9fcc;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row8_col1, #T_e1a85_row9_col8 {
      background-color: #b5c8df;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row8_col9 {
      background-color: #c0cfe3;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row9_col0 {
      background-color: #9db7d7;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row9_col1 {
      background-color: #ecc6ce;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row9_col3 {
      background-color: #5a88c2;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row9_col5 {
      background-color: #769ccb;
      color: #f1f1f1;
      width: 15em;
      text-align: center;
    }
    #T_e1a85_row9_col6 {
      background-color: #e0e6ed;
      color: #000000;
      width: 15em;
      text-align: center;
    }
    </style>
    <table id="T_e1a85">
      <thead>
        <tr>
          <th class="blank level0" >&nbsp;</th>
          <th id="T_e1a85_level0_col0" class="col_heading level0 col0" >season</th>
          <th id="T_e1a85_level0_col1" class="col_heading level0 col1" >hr</th>
          <th id="T_e1a85_level0_col2" class="col_heading level0 col2" >holiday</th>
          <th id="T_e1a85_level0_col3" class="col_heading level0 col3" >weekday</th>
          <th id="T_e1a85_level0_col4" class="col_heading level0 col4" >workingday</th>
          <th id="T_e1a85_level0_col5" class="col_heading level0 col5" >weathersit</th>
          <th id="T_e1a85_level0_col6" class="col_heading level0 col6" >atemp</th>
          <th id="T_e1a85_level0_col7" class="col_heading level0 col7" >hum</th>
          <th id="T_e1a85_level0_col8" class="col_heading level0 col8" >windspeed</th>
          <th id="T_e1a85_level0_col9" class="col_heading level0 col9" >cnt</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th id="T_e1a85_level0_row0" class="row_heading level0 row0" >season</th>
          <td id="T_e1a85_row0_col0" class="data row0 col0" >1.00</td>
          <td id="T_e1a85_row0_col1" class="data row0 col1" >0.01</td>
          <td id="T_e1a85_row0_col2" class="data row0 col2" >0.00</td>
          <td id="T_e1a85_row0_col3" class="data row0 col3" >0.01</td>
          <td id="T_e1a85_row0_col4" class="data row0 col4" >0.00</td>
          <td id="T_e1a85_row0_col5" class="data row0 col5" >0.00</td>
          <td id="T_e1a85_row0_col6" class="data row0 col6" >0.77</td>
          <td id="T_e1a85_row0_col7" class="data row0 col7" >0.16</td>
          <td id="T_e1a85_row0_col8" class="data row0 col8" >0.16</td>
          <td id="T_e1a85_row0_col9" class="data row0 col9" >0.26</td>
        </tr>
        <tr>
          <th id="T_e1a85_level0_row1" class="row_heading level0 row1" >hr</th>
          <td id="T_e1a85_row1_col0" class="data row1 col0" >0.01</td>
          <td id="T_e1a85_row1_col1" class="data row1 col1" >1.00</td>
          <td id="T_e1a85_row1_col2" class="data row1 col2" >0.00</td>
          <td id="T_e1a85_row1_col3" class="data row1 col3" >-0.00</td>
          <td id="T_e1a85_row1_col4" class="data row1 col4" >0.00</td>
          <td id="T_e1a85_row1_col5" class="data row1 col5" >0.05</td>
          <td id="T_e1a85_row1_col6" class="data row1 col6" >0.13</td>
          <td id="T_e1a85_row1_col7" class="data row1 col7" >-0.28</td>
          <td id="T_e1a85_row1_col8" class="data row1 col8" >0.14</td>
          <td id="T_e1a85_row1_col9" class="data row1 col9" >0.51</td>
        </tr>
        <tr>
          <th id="T_e1a85_level0_row2" class="row_heading level0 row2" >holiday</th>
          <td id="T_e1a85_row2_col0" class="data row2 col0" >0.00</td>
          <td id="T_e1a85_row2_col1" class="data row2 col1" >0.00</td>
          <td id="T_e1a85_row2_col2" class="data row2 col2" >1.00</td>
          <td id="T_e1a85_row2_col3" class="data row2 col3" >0.10</td>
          <td id="T_e1a85_row2_col4" class="data row2 col4" >0.09</td>
          <td id="T_e1a85_row2_col5" class="data row2 col5" >0.00</td>
          <td id="T_e1a85_row2_col6" class="data row2 col6" >0.03</td>
          <td id="T_e1a85_row2_col7" class="data row2 col7" >0.01</td>
          <td id="T_e1a85_row2_col8" class="data row2 col8" >0.00</td>
          <td id="T_e1a85_row2_col9" class="data row2 col9" >0.03</td>
        </tr>
        <tr>
          <th id="T_e1a85_level0_row3" class="row_heading level0 row3" >weekday</th>
          <td id="T_e1a85_row3_col0" class="data row3 col0" >0.01</td>
          <td id="T_e1a85_row3_col1" class="data row3 col1" >-0.00</td>
          <td id="T_e1a85_row3_col2" class="data row3 col2" >0.10</td>
          <td id="T_e1a85_row3_col3" class="data row3 col3" >1.00</td>
          <td id="T_e1a85_row3_col4" class="data row3 col4" >0.04</td>
          <td id="T_e1a85_row3_col5" class="data row3 col5" >0.00</td>
          <td id="T_e1a85_row3_col6" class="data row3 col6" >-0.01</td>
          <td id="T_e1a85_row3_col7" class="data row3 col7" >-0.04</td>
          <td id="T_e1a85_row3_col8" class="data row3 col8" >0.01</td>
          <td id="T_e1a85_row3_col9" class="data row3 col9" >0.03</td>
        </tr>
        <tr>
          <th id="T_e1a85_level0_row4" class="row_heading level0 row4" >workingday</th>
          <td id="T_e1a85_row4_col0" class="data row4 col0" >0.00</td>
          <td id="T_e1a85_row4_col1" class="data row4 col1" >0.00</td>
          <td id="T_e1a85_row4_col2" class="data row4 col2" >0.09</td>
          <td id="T_e1a85_row4_col3" class="data row4 col3" >0.04</td>
          <td id="T_e1a85_row4_col4" class="data row4 col4" >1.00</td>
          <td id="T_e1a85_row4_col5" class="data row4 col5" >0.00</td>
          <td id="T_e1a85_row4_col6" class="data row4 col6" >0.05</td>
          <td id="T_e1a85_row4_col7" class="data row4 col7" >0.02</td>
          <td id="T_e1a85_row4_col8" class="data row4 col8" >0.01</td>
          <td id="T_e1a85_row4_col9" class="data row4 col9" >0.03</td>
        </tr>
        <tr>
          <th id="T_e1a85_level0_row5" class="row_heading level0 row5" >weathersit</th>
          <td id="T_e1a85_row5_col0" class="data row5 col0" >0.00</td>
          <td id="T_e1a85_row5_col1" class="data row5 col1" >0.05</td>
          <td id="T_e1a85_row5_col2" class="data row5 col2" >0.00</td>
          <td id="T_e1a85_row5_col3" class="data row5 col3" >0.00</td>
          <td id="T_e1a85_row5_col4" class="data row5 col4" >0.00</td>
          <td id="T_e1a85_row5_col5" class="data row5 col5" >1.00</td>
          <td id="T_e1a85_row5_col6" class="data row5 col6" >0.11</td>
          <td id="T_e1a85_row5_col7" class="data row5 col7" >0.42</td>
          <td id="T_e1a85_row5_col8" class="data row5 col8" >0.08</td>
          <td id="T_e1a85_row5_col9" class="data row5 col9" >0.15</td>
        </tr>
        <tr>
          <th id="T_e1a85_level0_row6" class="row_heading level0 row6" >atemp</th>
          <td id="T_e1a85_row6_col0" class="data row6 col0" >0.77</td>
          <td id="T_e1a85_row6_col1" class="data row6 col1" >0.13</td>
          <td id="T_e1a85_row6_col2" class="data row6 col2" >0.03</td>
          <td id="T_e1a85_row6_col3" class="data row6 col3" >-0.01</td>
          <td id="T_e1a85_row6_col4" class="data row6 col4" >0.05</td>
          <td id="T_e1a85_row6_col5" class="data row6 col5" >0.11</td>
          <td id="T_e1a85_row6_col6" class="data row6 col6" >1.00</td>
          <td id="T_e1a85_row6_col7" class="data row6 col7" >-0.05</td>
          <td id="T_e1a85_row6_col8" class="data row6 col8" >-0.04</td>
          <td id="T_e1a85_row6_col9" class="data row6 col9" >0.42</td>
        </tr>
        <tr>
          <th id="T_e1a85_level0_row7" class="row_heading level0 row7" >hum</th>
          <td id="T_e1a85_row7_col0" class="data row7 col0" >0.16</td>
          <td id="T_e1a85_row7_col1" class="data row7 col1" >-0.28</td>
          <td id="T_e1a85_row7_col2" class="data row7 col2" >0.01</td>
          <td id="T_e1a85_row7_col3" class="data row7 col3" >-0.04</td>
          <td id="T_e1a85_row7_col4" class="data row7 col4" >0.02</td>
          <td id="T_e1a85_row7_col5" class="data row7 col5" >0.42</td>
          <td id="T_e1a85_row7_col6" class="data row7 col6" >-0.05</td>
          <td id="T_e1a85_row7_col7" class="data row7 col7" >1.00</td>
          <td id="T_e1a85_row7_col8" class="data row7 col8" >-0.29</td>
          <td id="T_e1a85_row7_col9" class="data row7 col9" >-0.36</td>
        </tr>
        <tr>
          <th id="T_e1a85_level0_row8" class="row_heading level0 row8" >windspeed</th>
          <td id="T_e1a85_row8_col0" class="data row8 col0" >0.16</td>
          <td id="T_e1a85_row8_col1" class="data row8 col1" >0.14</td>
          <td id="T_e1a85_row8_col2" class="data row8 col2" >0.00</td>
          <td id="T_e1a85_row8_col3" class="data row8 col3" >0.01</td>
          <td id="T_e1a85_row8_col4" class="data row8 col4" >0.01</td>
          <td id="T_e1a85_row8_col5" class="data row8 col5" >0.08</td>
          <td id="T_e1a85_row8_col6" class="data row8 col6" >-0.04</td>
          <td id="T_e1a85_row8_col7" class="data row8 col7" >-0.29</td>
          <td id="T_e1a85_row8_col8" class="data row8 col8" >1.00</td>
          <td id="T_e1a85_row8_col9" class="data row8 col9" >0.13</td>
        </tr>
        <tr>
          <th id="T_e1a85_level0_row9" class="row_heading level0 row9" >cnt</th>
          <td id="T_e1a85_row9_col0" class="data row9 col0" >0.26</td>
          <td id="T_e1a85_row9_col1" class="data row9 col1" >0.51</td>
          <td id="T_e1a85_row9_col2" class="data row9 col2" >0.03</td>
          <td id="T_e1a85_row9_col3" class="data row9 col3" >0.03</td>
          <td id="T_e1a85_row9_col4" class="data row9 col4" >0.03</td>
          <td id="T_e1a85_row9_col5" class="data row9 col5" >0.15</td>
          <td id="T_e1a85_row9_col6" class="data row9 col6" >0.42</td>
          <td id="T_e1a85_row9_col7" class="data row9 col7" >-0.36</td>
          <td id="T_e1a85_row9_col8" class="data row9 col8" >0.13</td>
          <td id="T_e1a85_row9_col9" class="data row9 col9" >1.00</td>
        </tr>
      </tbody>
    </table>

    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 35-36

Data quality check for score distribution plot 

.. GENERATED FROM PYTHON SOURCE LINES 36-38

.. code-block:: default

    exp.data_quality(method=PCA(), show='od_score_distribution', threshold=0.999, figsize=(5, 4))




.. image-sg:: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_001.png
   :alt: Score Distribution (PCA)
   :srcset: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_001.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 39-40

Data quality check for score distribution plot 

.. GENERATED FROM PYTHON SOURCE LINES 40-42

.. code-block:: default

    exp.data_quality(method=CBLOF(), show='od_score_distribution', threshold=0.999, figsize=(5, 4))




.. image-sg:: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_002.png
   :alt: Score Distribution (CBLOF)
   :srcset: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_002.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 43-44

Data quality check for score distribution plot

.. GENERATED FROM PYTHON SOURCE LINES 44-46

.. code-block:: default

    exp.data_quality(method=IsolationForest(), show='od_score_distribution', threshold=0.999, figsize=(5, 4))




.. image-sg:: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_003.png
   :alt: Score Distribution (IsolationForest)
   :srcset: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_003.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 47-48

Data quality check for score distribution plot

.. GENERATED FROM PYTHON SOURCE LINES 48-50

.. code-block:: default

    exp.data_quality(method=KMeansTree(), show='od_score_distribution', threshold=0.999, figsize=(5, 4))




.. image-sg:: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_004.png
   :alt: Score Distribution (KMeansTree)
   :srcset: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_004.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 51-52

Data quality check for score distribution plot 

.. GENERATED FROM PYTHON SOURCE LINES 52-54

.. code-block:: default

    exp.data_quality(method=KNN(), show='od_score_distribution', threshold=0.999, figsize=(5, 4))




.. image-sg:: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_005.png
   :alt: Score Distribution (KNN)
   :srcset: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_005.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 55-56

Data quality check for score distribution plot 

.. GENERATED FROM PYTHON SOURCE LINES 56-58

.. code-block:: default

    exp.data_quality(method=HBOS(), show='od_score_distribution', threshold=0.999, figsize=(5, 4))




.. image-sg:: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_006.png
   :alt: Score Distribution (HBOS)
   :srcset: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_006.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 59-60

Data quality check for score distribution plot 

.. GENERATED FROM PYTHON SOURCE LINES 60-62

.. code-block:: default

    exp.data_quality(method=ECOD(), show='od_score_distribution', threshold=0.999, figsize=(5, 4))




.. image-sg:: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_007.png
   :alt: Score Distribution (ECOD)
   :srcset: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_007.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 63-64

Data quality check for score distribution plot 

.. GENERATED FROM PYTHON SOURCE LINES 64-66

.. code-block:: default

    exp.data_quality(method=OneClassSVM(), show='od_score_distribution', threshold=0.999, figsize=(5, 4))




.. image-sg:: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_008.png
   :alt: Score Distribution (OneClassSVM)
   :srcset: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_008.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 67-68

Data quality check for score distribution plot 

.. GENERATED FROM PYTHON SOURCE LINES 68-71

.. code-block:: default

    exp.data_quality(method=PCA(), show='od_marginal_outlier_distribution',
                     threshold=0.999, figsize=(5, 4))




.. image-sg:: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_009.png
   :alt: Outliers distribution (PCA)
   :srcset: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_009.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 72-73

Compare different outlier detection algorithms  

.. GENERATED FROM PYTHON SOURCE LINES 73-76

.. code-block:: default

    exp.data_quality(method=[PCA(), CBLOF()], show='od_tsne_comparison',
                     threshold=[0.999, 0.999], figsize=(5, 4))




.. image-sg:: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_010.png
   :alt: t-SNE projection
   :srcset: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_010.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 77-78

Select a method and threshold and apply the outlier removal (you can also specify train, test, or all data)

.. GENERATED FROM PYTHON SOURCE LINES 78-81

.. code-block:: default

    exp.data_quality(method=CBLOF(), show='od_score_distribution', dataset="train",
                     threshold=0.999, remove_outliers=True, figsize=(5, 4))




.. image-sg:: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_011.png
   :alt: Score Distribution (CBLOF)
   :srcset: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_011.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 82-83

Compare the train and test data energy distance.

.. GENERATED FROM PYTHON SOURCE LINES 83-85

.. code-block:: default

    exp.data_quality(show='drift_test_info')





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

       Train Size  Test Size  Energy Distance
    0       13889       3476         0.000488




.. GENERATED FROM PYTHON SOURCE LINES 86-87

Compare the train and test marginal data drift feature-by-feature. 

.. GENERATED FROM PYTHON SOURCE LINES 87-89

.. code-block:: default

    exp.data_quality(show='drift_test_distance', figsize=(5, 4))




.. image-sg:: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_012.png
   :alt: Population Stability Index (PSI) - Top 30
   :srcset: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_012.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 90-91

Compare the train and test marginal data drift of a given feature. 

.. GENERATED FROM PYTHON SOURCE LINES 91-93

.. code-block:: default

    exp.data_quality(show="drift_test_distance", distance_metric="PSI", psi_buckets='quantile',
                     show_feature="atemp", figsize=(5, 4))



.. image-sg:: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_013.png
   :alt: Distribution plot
   :srcset: /auto_examples/0_data/images/sphx_glr_plot_4_data_quality_013.png
   :class: sphx-glr-single-img






.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 2 minutes  3.914 seconds)

**Estimated memory usage:**  1326 MB


.. _sphx_glr_download_auto_examples_0_data_plot_4_data_quality.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example


    .. container:: binder-badge

      .. image:: images/binder_badge_logo.svg
        :target: https://mybinder.org/v2/gh/selfexplainml/piml-toolbox/main?urlpath=lab/tree/./docs/_build/html/notebooks/auto_examples/0_data/plot_4_data_quality.ipynb
        :alt: Launch binder
        :width: 150 px



    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_4_data_quality.py <plot_4_data_quality.py>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_4_data_quality.ipynb <plot_4_data_quality.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_