Overfit: Regression

Experiment initialization and data preparation

from piml import Experiment
from piml.models import XGB2Regressor

exp = Experiment()
exp.data_loader(data="BikeSharing", silent=True)
exp.data_summary(feature_exclude=["yr", "mnth", "temp"], silent=True)
exp.data_prepare(target="cnt", task_type="regression", silent=True)

Train Model

exp.model_train(model=XGB2Regressor(), name="XGB2")

Histogram-based overfit test for a single feature

results = exp.model_diagnose(model="XGB2", show="overfit", slice_method="histogram",
                             slice_features=["hr"], threshold=1.05, min_samples=100,
                             return_data=True, figsize=(5, 4))
results.data

	[hr	hr)	#Test	#Train	test_MSE	train_MSE	Gap
0	0.3	0.4	445	1736	0.0219	0.0197	0.0022
1	0.6	0.7	445	1743	0.0103	0.0096	0.0008
2	0.8	0.9	285	1171	0.0081	0.0075	0.0006

Histogram-based overfit test for two features

results = exp.model_diagnose(model="XGB2", show="overfit", slice_method="histogram",
                             slice_features=["hr", "atemp"], threshold=1.05, min_samples=100,
                             return_data=True, figsize=(5, 4))
results.data

	[hr	hr)	[atemp	atemp)	#Test	#Train	test_MSE	train_MSE	Gap
0	0.7	0.8	0.5000	0.6636	103	366	0.0421	0.0350	0.0071
1	0.3	0.4	0.2318	0.4636	173	645	0.0250	0.0198	0.0052
2	0.6	0.7	0.1364	0.5606	240	882	0.0095	0.0078	0.0017
3	0.0	0.1	0.2546	0.3940	105	432	0.0020	0.0011	0.0009

Histogram-based overfit test for a single feature using MAE metric

results = exp.model_diagnose(model="XGB2", show="overfit", slice_method="histogram",
                             slice_features=["atemp"], threshold=1.05, min_samples=100,
                             metric="MAE", return_data=True, figsize=(5, 4))
results.data

	[atemp	atemp)	#Test	#Train	test_MAE	train_MAE	Gap
0	0.197	0.4924	1649	6580	0.0639	0.0589	0.005

Estimated memory usage: 31 MB