piml.scored_test.test_resilience_distance

piml.scored_test.test_resilience_distance(x, y, prediction, prediction_proba=None, feature_names=None, feature_types=None, target_name=None, train_idx=None, test_idx=None, task_type=None, random_state=None, resilience_method=None, immu_feature=None, metric=None, alpha=None, n_clusters=None, psi_buckets=None, distance_metric=None, figsize=None)

Compare data distance between samples in the worst region and the remaining region.

Parameters:
xndarray of shape (n_samples, n_features), default=None

The covariate data.

yndarray of shape (n_samples, ), default=None

The actual response.

predicitonndarray of shape (n_samples, ), default=None

The model prediction for regression tasks and predicted class for binary classification tasks.

prediciton_probandarray of shape (n_samples, ), default=None

The predicted probability of the positive class in binary classificaiton tasks. Not need for regression tasks.

task_type{‘regression’, ‘classification’}, default=None

The task type.

feature_nameslist, default=None

Feature names.

feature_types: list, default=None

Feature types, can be ‘numerical’ or ‘categorical’.

target_namestr, default=None

Target name.

train_idxarray-like of shape (n_samples_train,), default=None

If train_idx and test_idx are not None, it will be ignored.

test_idxarray-like of shape (n_samples_test,), default=None

If train_idx and test_idx are not None, it will be ignored.

random_stateint, default=None

Random seed for train / test split. If None, it will be 0.

resilience_method{‘worst-sample’, ‘hard-sample’, ‘outer-sample’, ‘worst-cluster’}, default=None
The method used for selecting worst samples.

If None, it will be ‘worst-sample’.

  • ‘worst-sample’: Select the worst samples according to the loss of each sample; the worst samples are related to models.

  • ‘hard-sample’: Use a deep XGB model to distinguish hard and easy samples; the worst samples are the same for different models.

  • ‘outer-sample’: Use the Euclidean distance of each sample to the mean of X as a surrogate of worstness; the worst samples are the same for different models.

  • ‘worst-cluster’: Fit a K-means using the X, and then select the worst performing cluste as the worst samples; the worst samples are related to models.

immu_featurestr, default=None

The name of immutable feature. If None, it will be an empty list.

alphafloat, default=None

This parameter has different meanings for different diagnostics. It determines the ratio for worst sampling, which can only be 0.1, 0.2, …, 0.9. If None, it will be 0.1.

n_clustersint, default=None

The number of clusters in K-means used when resilience_method = ‘worst-cluster’. If None, it will be 10.

metric{‘MSE’, ‘MAE’, ‘R2’, ‘ACC’, ‘AUC’, ‘F1’, ‘LogLoss’, ‘Brier’}, default=None

Performance metric.

  • For classification tasks: ‘ACC’, ‘AUC’, ‘F1’, ‘LogLoss’, ‘Brier’.

  • For regression tasks: ‘MSE’, ‘MAE’, ‘R2’.

distance_metric{‘PSI’, ‘WD1’, ‘KS’}, default=None

The distance metric used when show = ‘reliability_distance’ or ‘resilience_distance’. If None, it will be ‘PSI’.

  • ‘PSI’: Population stability index.

  • ‘WD1’: Wasserstein distance (1D).

  • ‘KS’: Kolmogorov-Smirnov test.

psi_buckets{‘uniform’, ‘quantile’}, default=None

Bucketing strategy for PSI metric. If None, it will be ‘uniform’.

figsizetuple, default=None

Figure size of the plot. If None, it will be (8, 6).