.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples\1_train\plot_1_hpo_random.py"
.. LINE NUMBERS ARE GIVEN BELOW.
.. only:: html
.. note::
:class: sphx-glr-download-link-note
:ref:`Go to the end <sphx_glr_download_auto_examples_1_train_plot_1_hpo_random.py>`
to download the full example code or to run this example in your browser via Binder
.. rst-class:: sphx-glr-example-title
.. _sphx_glr_auto_examples_1_train_plot_1_hpo_random.py:
HPO - Random Search
=========================================
.. GENERATED FROM PYTHON SOURCE LINES 7-8
Experiment initialization and data preparation
.. GENERATED FROM PYTHON SOURCE LINES 8-17
.. code-block:: default
import scipy
from piml import Experiment
from piml.models import GLMClassifier
exp = Experiment()
exp.data_loader("SimuCredit", silent=True)
exp.data_summary(feature_exclude=["Race", "Gender"], silent=True)
exp.data_prepare(target="Approved", task_type="classification", silent=True)
.. GENERATED FROM PYTHON SOURCE LINES 18-19
Train Model
.. GENERATED FROM PYTHON SOURCE LINES 19-21
.. code-block:: default
exp.model_train(model=GLMClassifier(), name="GLM")
.. GENERATED FROM PYTHON SOURCE LINES 22-23
Define hyperparameter search space for grid search
.. GENERATED FROM PYTHON SOURCE LINES 23-26
.. code-block:: default
parameters = {'l1_regularization': scipy.stats.uniform(0, 0.1),
'l2_regularization': scipy.stats.uniform(0, 0.1)}
.. GENERATED FROM PYTHON SOURCE LINES 27-28
Tune hyperparameters of registered models
.. GENERATED FROM PYTHON SOURCE LINES 28-32
.. code-block:: default
result = exp.model_tune("GLM", method="randomized", parameters=parameters, n_runs=100,
metric="AUC", test_ratio=0.2)
result.data
.. raw:: html
<div class="output_subarea output_html rendered_html output_result">
<div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>Rank(by AUC)</th>
<th>AUC</th>
<th>time</th>
</tr>
<tr>
<th>params</th>
<th></th>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<th>{'l1_regularization': 0.020239411195815772, 'l2_regularization': 0.002493911719174813}</th>
<td>1</td>
<td>0.729472</td>
<td>0.058955</td>
</tr>
<tr>
<th>{'l1_regularization': 0.0801649627572255, 'l2_regularization': 0.0009505206902733599}</th>
<td>2</td>
<td>0.729471</td>
<td>0.049783</td>
</tr>
<tr>
<th>{'l1_regularization': 0.07054458515086691, 'l2_regularization': 0.002443422808146689}</th>
<td>3</td>
<td>0.729470</td>
<td>0.070307</td>
</tr>
<tr>
<th>{'l1_regularization': 0.0200826243066503, 'l2_regularization': 0.0038925653368613645}</th>
<td>4</td>
<td>0.729469</td>
<td>0.054850</td>
</tr>
<tr>
<th>{'l1_regularization': 0.08763141285690618, 'l2_regularization': 0.0011317924903120004}</th>
<td>5</td>
<td>0.729469</td>
<td>0.038847</td>
</tr>
<tr>
<th>...</th>
<td>...</td>
<td>...</td>
<td>...</td>
</tr>
<tr>
<th>{'l1_regularization': 0.014018374389893852, 'l2_regularization': 0.08885523820070576}</th>
<td>96</td>
<td>0.729274</td>
<td>0.046059</td>
</tr>
<tr>
<th>{'l1_regularization': 0.06720040139301987, 'l2_regularization': 0.08585578494124796}</th>
<td>97</td>
<td>0.729273</td>
<td>0.041242</td>
</tr>
<tr>
<th>{'l1_regularization': 0.05105133900697683, 'l2_regularization': 0.0845254996471495}</th>
<td>97</td>
<td>0.729273</td>
<td>0.047853</td>
</tr>
<tr>
<th>{'l1_regularization': 0.05346229516211951, 'l2_regularization': 0.09880860499665617}</th>
<td>99</td>
<td>0.729267</td>
<td>0.046578</td>
</tr>
<tr>
<th>{'l1_regularization': 0.09599608264816151, 'l2_regularization': 0.09690348478707304}</th>
<td>100</td>
<td>0.729255</td>
<td>0.044768</td>
</tr>
</tbody>
</table>
<p>100 rows × 3 columns</p>
</div>
</div>
<br />
<br />
.. GENERATED FROM PYTHON SOURCE LINES 33-34
Refit model using a selected hyperparameter
.. GENERATED FROM PYTHON SOURCE LINES 34-37
.. code-block:: default
params = result.get_params_ranks(rank=1)
exp.model_train(GLMClassifier(**params), name="GLM-HPO-RandSearch")
.. GENERATED FROM PYTHON SOURCE LINES 38-39
Compare the default model and HPO refitted model
.. GENERATED FROM PYTHON SOURCE LINES 39-41
.. code-block:: default
exp.model_diagnose("GLM", show="accuracy_table")
.. rst-class:: sphx-glr-script-out
.. code-block:: none
ACC AUC F1 LogLoss Brier
Train 0.6722 0.7309 0.6965 0.6047 0.2088
Test 0.6690 0.7318 0.6976 0.6073 0.2095
Gap -0.0032 0.0009 0.0011 0.0026 0.0008
.. GENERATED FROM PYTHON SOURCE LINES 42-43
Compare the default model and HPO refitted model
.. GENERATED FROM PYTHON SOURCE LINES 43-44
.. code-block:: default
exp.model_diagnose("GLM-HPO-RandSearch", show="accuracy_table")
.. rst-class:: sphx-glr-script-out
.. code-block:: none
ACC AUC F1 LogLoss Brier
Train 0.6721 0.7309 0.6964 0.6047 0.2088
Test 0.6690 0.7318 0.6976 0.6073 0.2095
Gap -0.0031 0.0009 0.0012 0.0026 0.0008
.. rst-class:: sphx-glr-timing
**Total running time of the script:** ( 1 minutes 24.760 seconds)
**Estimated memory usage:** 15 MB
.. _sphx_glr_download_auto_examples_1_train_plot_1_hpo_random.py:
.. only:: html
.. container:: sphx-glr-footer sphx-glr-footer-example
.. container:: binder-badge
.. image:: images/binder_badge_logo.svg
:target: https://mybinder.org/v2/gh/selfexplainml/piml-toolbox/main?urlpath=lab/tree/./docs/_build/html/notebooks/auto_examples/1_train/plot_1_hpo_random.ipynb
:alt: Launch binder
:width: 150 px
.. container:: sphx-glr-download sphx-glr-download-python
:download:`Download Python source code: plot_1_hpo_random.py <plot_1_hpo_random.py>`
.. container:: sphx-glr-download sphx-glr-download-jupyter
:download:`Download Jupyter notebook: plot_1_hpo_random.ipynb <plot_1_hpo_random.ipynb>`
.. only:: html
.. rst-class:: sphx-glr-signature
`Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_