piml.scored_test.test_reliability_distance

piml.scored_test.test_reliability_distance(x, y, prediction, prediction_proba=None, feature_names=None, feature_types=None, target_name=None, train_idx=None, test_idx=None, task_type=None, random_state=None, alpha=None, threshold=None, distance_metric=None, figsize=None)

Compare data distance between reliable and unreliable samples.

Parameters:

xndarray of shape (n_samples, n_features), default=None

The covariate data.

yndarray of shape (n_samples, ), default=None

The actual response.

predicitonndarray of shape (n_samples, ), default=None

The model prediction for regression tasks and predicted class for binary classification tasks.

prediciton_probandarray of shape (n_samples, ), default=None

The predicted probability of the positive class in binary classificaiton tasks. Not need for regression tasks.

task_type{‘regression’, ‘classification’}, default=None

The task type.

feature_nameslist, default=None

Feature names.

feature_types: list, default=None

Feature types, can be ‘numerical’ or ‘categorical’.

target_namestr, default=None

Target name.

train_idxarray-like of shape (n_samples_train,), default=None

If train_idx and test_idx are not None, it will be ignored.

test_idxarray-like of shape (n_samples_test,), default=None

If train_idx and test_idx are not None, it will be ignored.

random_stateint, default=None

Random seed for train / test split. If None, it will be 0.

alphafloat, default=None

The error rate in Split Conformal Prediction (only for regression tasks). If None, it will be 0.1.

thresholdfloat, default=None

The threshold to determine whether data are inside or outside coverage.

If None, it will be 1.1.

distance_metric{‘PSI’, ‘WD1’, ‘KS’}, default=None

The distance metric. If None, it will be ‘PSI’.

• ‘PSI’: Population stability index.

• ‘WD1’: Wasserstein distance (1D).

• ‘KS’: Kolmogorov-Smirnov test.

figsizetuple, default=None

Figure size of the plot. If None, it will be (8, 6).