.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples\5_compare\plot_0_compare_regression.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code or to run this example in your browser via Binder .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_5_compare_plot_0_compare_regression.py: Model Comparison: Regression ================================================== .. GENERATED FROM PYTHON SOURCE LINES 8-9 Experiment initialization and data preparation .. GENERATED FROM PYTHON SOURCE LINES 9-18 .. code-block:: default from piml import Experiment from piml.models import GLMRegressor from xgboost import XGBRegressor exp = Experiment() exp.data_loader(data="BikeSharing", silent=True) exp.data_summary(feature_exclude=["yr", "mnth", "temp"], silent=True) exp.data_prepare(target="cnt", task_type="regression", silent=True) .. GENERATED FROM PYTHON SOURCE LINES 19-20 Train Model .. GENERATED FROM PYTHON SOURCE LINES 20-25 .. code-block:: default exp.model_train(GLMRegressor(), name="GLM") exp.model_train(XGBRegressor(max_depth=2), name="XGB2") exp.model_train(XGBRegressor(max_depth=7), name="XGB7") .. GENERATED FROM PYTHON SOURCE LINES 26-27 Accuracy comparison with MSE .. GENERATED FROM PYTHON SOURCE LINES 27-29 .. code-block:: default exp.model_compare(models=["GLM", "XGB2", "XGB7"], show="accuracy_plot", metric="MSE", figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_001.png :alt: Predictive Performance :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 30-31 Accuracy comparison with MAE .. GENERATED FROM PYTHON SOURCE LINES 31-33 .. code-block:: default exp.model_compare(models=["GLM", "XGB2", "XGB7"], show="accuracy_plot", metric="MAE", figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_002.png :alt: Predictive Performance :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_002.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 34-35 Accuracy comparison with R2 .. GENERATED FROM PYTHON SOURCE LINES 35-37 .. code-block:: default exp.model_compare(models=["GLM", "XGB2", "XGB7"], show="accuracy_plot", metric="R2", figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_003.png :alt: Predictive Performance :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_003.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 38-39 Overfit comparison with MSE metric .. GENERATED FROM PYTHON SOURCE LINES 39-43 .. code-block:: default exp.model_compare(models=["GLM", "XGB2", "XGB7"], show="overfit", slice_method="histogram", slice_feature="hr", bins=10, metric="MSE", original_scale=True, figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_004.png :alt: Overfit Comparison :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_004.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 44-45 Overfit comparison with MAE metric .. GENERATED FROM PYTHON SOURCE LINES 45-49 .. code-block:: default exp.model_compare(models=["GLM", "XGB2", "XGB7"], show="overfit", slice_method="histogram", slice_feature="hr", metric="MAE", original_scale=True, figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_005.png :alt: Overfit Comparison :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_005.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 50-51 Reliability coverage comparison .. GENERATED FROM PYTHON SOURCE LINES 51-53 .. code-block:: default exp.model_compare(models=["GLM", "XGB2", "XGB7"], show="reliability_coverage", alpha=0.1, figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_006.png :alt: Empirical Coverage :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_006.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 54-55 Reliability bandwidth comparison .. GENERATED FROM PYTHON SOURCE LINES 55-57 .. code-block:: default exp.model_compare(models=["GLM", "XGB2", "XGB7"], show="reliability_bandwidth", alpha=0.1, figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_007.png :alt: Bandwidth of Testing Data :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_007.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 58-59 Robustness comparison with default settings .. GENERATED FROM PYTHON SOURCE LINES 59-60 .. code-block:: default exp.model_compare(models=["GLM", "XGB2", "XGB7"], show="robustness_perf", figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_008.png :alt: Model Performance: Perturb on All Features :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_008.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 61-62 Robustness comparison on worst alpha-percent samples .. GENERATED FROM PYTHON SOURCE LINES 62-64 .. code-block:: default exp.model_compare(models=["GLM", "XGB2", "XGB7"], show="robustness_perf_worst", alpha=0.3, figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_009.png :alt: 30%-Worst Sample Performance :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_009.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 65-66 Resilience comparison with worst-sample scenario .. GENERATED FROM PYTHON SOURCE LINES 66-69 .. code-block:: default exp.model_compare(models=["GLM", "XGB2", "XGB7"], show="resilience_perf", resilience_method="worst-sample", immu_feature=None, metric="MAE", figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_010.png :alt: Resilience Test :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_010.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 70-71 Resilience distance comparison with worst-sample scenario .. GENERATED FROM PYTHON SOURCE LINES 71-73 .. code-block:: default exp.model_compare(models=["GLM", "XGB2", "XGB7"], show="resilience_distance", resilience_method="worst-sample", metric="MAE", alpha=0.3, figsize=(5, 4)) .. image-sg:: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_011.png :alt: Distribution Shift: 30%-Worst vs Remaining :srcset: /auto_examples/5_compare/images/sphx_glr_plot_0_compare_regression_011.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-timing **Total running time of the script:** ( 1 minutes 21.459 seconds) **Estimated memory usage:** 81 MB .. _sphx_glr_download_auto_examples_5_compare_plot_0_compare_regression.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: binder-badge .. image:: images/binder_badge_logo.svg :target: https://mybinder.org/v2/gh/selfexplainml/piml-toolbox/main?urlpath=lab/tree/./docs/_build/html/notebooks/auto_examples/5_compare/plot_0_compare_regression.ipynb :alt: Launch binder :width: 150 px .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_0_compare_regression.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_0_compare_regression.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_