Appendix D — Supplementary Material for Chapter 3

Keywords

Artificial Intelligence, Trustworthy AI, Counterfactual Explanations, Algorithmic Recourse

D.1 Detailed Results: Synthetic Data

D.1.1 Line Charts

The evolution of the evaluation metrics over the course of the experiment is shown for different datasets in Figure D.1 to Figure D.4.

Figure D.1space Evolution of evaluation metrics over the course of the experiment. Data: Circles.
Figure D.2space Evolution of evaluation metrics over the course of the experiment. Data: Linearly Separable.
Figure D.3space Evolution of evaluation metrics over the course of the experiment. Data: Moons.
Figure D.4space Evolution of evaluation metrics over the course of the experiment. Data: Overlapping.

D.1.2 Error Bar Charts

The evaluation metrics at the end of the experiment are shown for different datasets in Figure D.5 to Figure D.8.

Figure D.5space Evaluation metrics at the end of the experiment. Data: Circles.
Figure D.6space Evaluation metrics at the end of the experiment. Data: Linearly Separable.
Figure D.7space Evaluation metrics at the end of the experiment. Data: Moons.
Figure D.8space Evaluation metrics at the end of the experiment. Data: Overlapping.

D.1.3 Statistical Significance

Table D.1 presents the tests for statistical significance of the estimated MMD metrics.

Table D.1space Tests for statistical significance of the estimated MMD metrics. We have highlighted p-values smaller than the significance level \(\alpha=0.05\) in bold. Data: Synthetic.
Metric Data Generator Model p-value
MMD Circles DICE Deep Ensemble 0.988
MMD Circles DICE Linear 1.0
MMD Circles DICE MLP 0.99
MMD Circles Generic (γ=0.5) Deep Ensemble 0.996
MMD Circles Generic (γ=0.5) Linear 0.996
MMD Circles Generic (γ=0.5) MLP 0.99
MMD Circles Greedy Deep Ensemble 0.992
MMD Circles Greedy Linear 1.0
MMD Circles Greedy MLP 0.994
MMD Circles Latent Deep Ensemble 0.9975
MMD Circles Latent Linear 0.9925
MMD Circles Latent MLP 1.0
MMD Linearly Separable DICE Deep Ensemble 0.0
MMD Linearly Separable DICE Linear 0.0
MMD Linearly Separable DICE MLP 0.0
MMD Linearly Separable Generic (γ=0.5) Deep Ensemble 0.0
MMD Linearly Separable Generic (γ=0.5) Linear 0.0
MMD Linearly Separable Generic (γ=0.5) MLP 0.0
MMD Linearly Separable Greedy Deep Ensemble 0.0
MMD Linearly Separable Greedy Linear 0.0
MMD Linearly Separable Greedy MLP 0.0
MMD Linearly Separable Latent Deep Ensemble 0.748
MMD Linearly Separable Latent Linear 0.768
MMD Linearly Separable Latent MLP 0.69
MMD Moons DICE Deep Ensemble 0.0
MMD Moons DICE Linear 0.0
MMD Moons DICE MLP 0.0
MMD Moons Generic (γ=0.5) Deep Ensemble 0.0
MMD Moons Generic (γ=0.5) Linear 0.0
MMD Moons Generic (γ=0.5) MLP 0.0
MMD Moons Greedy Deep Ensemble 0.0
MMD Moons Greedy Linear 0.0
MMD Moons Greedy MLP 0.0
MMD Moons Latent Deep Ensemble 0.0
MMD Moons Latent Linear 0.0
MMD Moons Latent MLP 0.0
MMD Overlapping DICE Deep Ensemble 0.0
MMD Overlapping DICE Linear 0.0
MMD Overlapping DICE MLP 0.0
MMD Overlapping Generic (γ=0.5) Deep Ensemble 0.0
MMD Overlapping Generic (γ=0.5) Linear 0.0
MMD Overlapping Generic (γ=0.5) MLP 0.0
MMD Overlapping Greedy Deep Ensemble 0.0
MMD Overlapping Greedy Linear 0.0
MMD Overlapping Greedy MLP 0.0
MMD Overlapping Latent Deep Ensemble 0.0
MMD Overlapping Latent Linear 0.0
MMD Overlapping Latent MLP 0.0
PP MMD Circles DICE Deep Ensemble 0.996
PP MMD Circles DICE Linear 0.796
PP MMD Circles DICE MLP 0.9975
PP MMD Circles Generic (γ=0.5) Deep Ensemble 1.0
PP MMD Circles Generic (γ=0.5) Linear 0.996
PP MMD Circles Generic (γ=0.5) MLP 0.992
PP MMD Circles Greedy Deep Ensemble 1.0
PP MMD Circles Greedy Linear 0.0
PP MMD Circles Greedy MLP 0.996
PP MMD Circles Latent Deep Ensemble 0.9975
PP MMD Circles Latent Linear 0.0
PP MMD Circles Latent MLP 0.994
PP MMD Linearly Separable DICE Deep Ensemble 0.9525
PP MMD Linearly Separable DICE Linear 0.0
PP MMD Linearly Separable DICE MLP 0.964
PP MMD Linearly Separable Generic (γ=0.5) Deep Ensemble 0.958
PP MMD Linearly Separable Generic (γ=0.5) Linear 0.0
PP MMD Linearly Separable Generic (γ=0.5) MLP 0.944
PP MMD Linearly Separable Greedy Deep Ensemble 0.716
PP MMD Linearly Separable Greedy Linear 0.0
PP MMD Linearly Separable Greedy MLP 0.684
PP MMD Linearly Separable Latent Deep Ensemble 0.856
PP MMD Linearly Separable Latent Linear 0.46
PP MMD Linearly Separable Latent MLP 0.852
PP MMD Moons DICE Deep Ensemble 0.865
PP MMD Moons DICE Linear 0.0
PP MMD Moons DICE MLP 0.87
PP MMD Moons Generic (γ=0.5) Deep Ensemble 0.678
PP MMD Moons Generic (γ=0.5) Linear 0.0
PP MMD Moons Generic (γ=0.5) MLP 0.84
PP MMD Moons Greedy Deep Ensemble 0.388
PP MMD Moons Greedy Linear 0.0
PP MMD Moons Greedy MLP 0.346
PP MMD Moons Latent Deep Ensemble 0.902
PP MMD Moons Latent Linear 0.004
PP MMD Moons Latent MLP 0.91
PP MMD Overlapping DICE Deep Ensemble 0.0
PP MMD Overlapping DICE Linear 0.0
PP MMD Overlapping DICE MLP 0.002
PP MMD Overlapping Generic (γ=0.5) Deep Ensemble 0.004
PP MMD Overlapping Generic (γ=0.5) Linear 0.0
PP MMD Overlapping Generic (γ=0.5) MLP 0.002
PP MMD Overlapping Greedy Deep Ensemble 0.002
PP MMD Overlapping Greedy Linear 0.0
PP MMD Overlapping Greedy MLP 0.004
PP MMD Overlapping Latent Deep Ensemble 0.034
PP MMD Overlapping Latent Linear 0.012
PP MMD Overlapping Latent MLP 0.034
PP MMD (grid) Circles DICE Deep Ensemble 0.762
PP MMD (grid) Circles DICE Linear 0.814
PP MMD (grid) Circles DICE MLP 0.7375
PP MMD (grid) Circles Generic (γ=0.5) Deep Ensemble 0.89
PP MMD (grid) Circles Generic (γ=0.5) Linear 0.994
PP MMD (grid) Circles Generic (γ=0.5) MLP 0.688
PP MMD (grid) Circles Greedy Deep Ensemble 0.568
PP MMD (grid) Circles Greedy Linear 0.0
PP MMD (grid) Circles Greedy MLP 0.776
PP MMD (grid) Circles Latent Deep Ensemble 1.0
PP MMD (grid) Circles Latent Linear 0.0
PP MMD (grid) Circles Latent MLP 0.996
PP MMD (grid) Linearly Separable DICE Deep Ensemble 0.0
PP MMD (grid) Linearly Separable DICE Linear 0.0
PP MMD (grid) Linearly Separable DICE MLP 0.0
PP MMD (grid) Linearly Separable Generic (γ=0.5) Deep Ensemble 0.0
PP MMD (grid) Linearly Separable Generic (γ=0.5) Linear 0.0
PP MMD (grid) Linearly Separable Generic (γ=0.5) MLP 0.0
PP MMD (grid) Linearly Separable Greedy Deep Ensemble 0.0
PP MMD (grid) Linearly Separable Greedy Linear 0.0
PP MMD (grid) Linearly Separable Greedy MLP 0.0
PP MMD (grid) Linearly Separable Latent Deep Ensemble 0.0
PP MMD (grid) Linearly Separable Latent Linear 0.0
PP MMD (grid) Linearly Separable Latent MLP 0.0
PP MMD (grid) Moons DICE Deep Ensemble 0.1225
PP MMD (grid) Moons DICE Linear 0.0
PP MMD (grid) Moons DICE MLP 0.01
PP MMD (grid) Moons Generic (γ=0.5) Deep Ensemble 0.016
PP MMD (grid) Moons Generic (γ=0.5) Linear 0.0
PP MMD (grid) Moons Generic (γ=0.5) MLP 0.02
PP MMD (grid) Moons Greedy Deep Ensemble 0.006
PP MMD (grid) Moons Greedy Linear 0.0
PP MMD (grid) Moons Greedy MLP 0.0
PP MMD (grid) Moons Latent Deep Ensemble 0.114
PP MMD (grid) Moons Latent Linear 0.004
PP MMD (grid) Moons Latent MLP 0.174
PP MMD (grid) Overlapping DICE Deep Ensemble 0.002
PP MMD (grid) Overlapping DICE Linear 0.0
PP MMD (grid) Overlapping DICE MLP 0.0
PP MMD (grid) Overlapping Generic (γ=0.5) Deep Ensemble 0.0
PP MMD (grid) Overlapping Generic (γ=0.5) Linear 0.0
PP MMD (grid) Overlapping Generic (γ=0.5) MLP 0.0
PP MMD (grid) Overlapping Greedy Deep Ensemble 0.0
PP MMD (grid) Overlapping Greedy Linear 0.0
PP MMD (grid) Overlapping Greedy MLP 0.002
PP MMD (grid) Overlapping Latent Deep Ensemble 0.208
PP MMD (grid) Overlapping Latent Linear 0.02
PP MMD (grid) Overlapping Latent MLP 0.342

D.2 Detailed Results: Real-World Data

D.2.1 Line Charts

The evolution of the evaluation metrics over the course of the experiment is shown for different datasets in Figure D.9 to Figure D.11.

Figure D.9space Evolution of evaluation metrics over the course of the experiment. Data: California Housing.
Figure D.10space Evolution of evaluation metrics over the course of the experiment. Data: Credit Default.
Figure D.11space Evolution of evaluation metrics over the course of the experiment. Data: GMSC.

D.2.2 Error Bar Charts

The evaluation metrics at the end of the experiment are shown for different datasets in Figure D.12 to Figure D.14.

Figure D.12space Evaluation metrics at the end of the experiment. Data: California Housing.
Figure D.13space Evaluation metrics at the end of the experiment. Data: Credit Default.
Figure D.14space Evaluation metrics at the end of the experiment. Data: GMSC.

D.2.3 Statistical Significance

Table D.2 presents the tests for statistical significance of the estimated MMD metrics.

Table D.2space Tests for statistical significance of the estimated MMD metrics. We have highlighted p-values smaller than the significance level \(\alpha=0.05\) in bold. Data: Real-World.
Metric Data Generator Model p-value
MMD Cal Housing DICE Deep Ensemble 0.0
MMD Cal Housing DICE Linear 0.0
MMD Cal Housing DICE MLP 0.0
MMD Cal Housing Generic (γ=0.5) Deep Ensemble 0.0
MMD Cal Housing Generic (γ=0.5) Linear 0.0
MMD Cal Housing Generic (γ=0.5) MLP 0.0
MMD Cal Housing Greedy Deep Ensemble 0.0
MMD Cal Housing Greedy Linear 0.0
MMD Cal Housing Greedy MLP 0.0
MMD Cal Housing Latent Deep Ensemble 0.0
MMD Cal Housing Latent Linear 0.0
MMD Cal Housing Latent MLP 0.0
MMD Credit Default DICE Deep Ensemble 1.0
MMD Credit Default DICE Linear 1.0
MMD Credit Default DICE MLP 1.0
MMD Credit Default Generic (γ=0.5) Deep Ensemble 1.0
MMD Credit Default Generic (γ=0.5) Linear 1.0
MMD Credit Default Generic (γ=0.5) MLP 1.0
MMD Credit Default Greedy Deep Ensemble 1.0
MMD Credit Default Greedy Linear 1.0
MMD Credit Default Greedy MLP 1.0
MMD Credit Default Latent Deep Ensemble 0.0
MMD Credit Default Latent Linear 1.0
MMD Credit Default Latent MLP 0.0
MMD GMSC DICE Deep Ensemble 0.082
MMD GMSC DICE Linear 0.51
MMD GMSC DICE MLP 0.338
MMD GMSC Generic (γ=0.5) Deep Ensemble 0.306
MMD GMSC Generic (γ=0.5) Linear 0.278
MMD GMSC Generic (γ=0.5) MLP 0.128
MMD GMSC Greedy Deep Ensemble 0.032
MMD GMSC Greedy Linear 0.006
MMD GMSC Greedy MLP 0.0
MMD GMSC Latent Deep Ensemble 0.0
MMD GMSC Latent Linear 0.0
MMD GMSC Latent MLP 0.0
PP MMD Cal Housing DICE Deep Ensemble 0.0
PP MMD Cal Housing DICE Linear 0.0
PP MMD Cal Housing DICE MLP 0.0
PP MMD Cal Housing Generic (γ=0.5) Deep Ensemble 0.0
PP MMD Cal Housing Generic (γ=0.5) Linear 0.0
PP MMD Cal Housing Generic (γ=0.5) MLP 0.0
PP MMD Cal Housing Greedy Deep Ensemble 0.0
PP MMD Cal Housing Greedy Linear 0.0
PP MMD Cal Housing Greedy MLP 0.0
PP MMD Cal Housing Latent Deep Ensemble 0.0
PP MMD Cal Housing Latent Linear 0.0
PP MMD Cal Housing Latent MLP 0.0
PP MMD Credit Default DICE Deep Ensemble 0.0
PP MMD Credit Default DICE Linear 0.0
PP MMD Credit Default DICE MLP 0.0
PP MMD Credit Default Generic (γ=0.5) Deep Ensemble 0.0
PP MMD Credit Default Generic (γ=0.5) Linear 0.0
PP MMD Credit Default Generic (γ=0.5) MLP 0.0
PP MMD Credit Default Greedy Deep Ensemble 0.0
PP MMD Credit Default Greedy Linear 0.044
PP MMD Credit Default Greedy MLP 0.0
PP MMD Credit Default Latent Deep Ensemble 0.0
PP MMD Credit Default Latent Linear 0.436
PP MMD Credit Default Latent MLP 0.0
PP MMD GMSC DICE Deep Ensemble 0.032
PP MMD GMSC DICE Linear 0.0
PP MMD GMSC DICE MLP 0.0
PP MMD GMSC Generic (γ=0.5) Deep Ensemble 0.018
PP MMD GMSC Generic (γ=0.5) Linear 0.0
PP MMD GMSC Generic (γ=0.5) MLP 0.0
PP MMD GMSC Greedy Deep Ensemble 0.02
PP MMD GMSC Greedy Linear 0.0
PP MMD GMSC Greedy MLP 0.0
PP MMD GMSC Latent Deep Ensemble 0.008
PP MMD GMSC Latent Linear 0.0
PP MMD GMSC Latent MLP 0.0
PP MMD (grid) Cal Housing DICE Deep Ensemble 0.0
PP MMD (grid) Cal Housing DICE Linear 0.0
PP MMD (grid) Cal Housing DICE MLP 0.0
PP MMD (grid) Cal Housing Generic (γ=0.5) Deep Ensemble 0.0
PP MMD (grid) Cal Housing Generic (γ=0.5) Linear 0.0
PP MMD (grid) Cal Housing Generic (γ=0.5) MLP 0.004
PP MMD (grid) Cal Housing Greedy Deep Ensemble 0.0
PP MMD (grid) Cal Housing Greedy Linear 0.0
PP MMD (grid) Cal Housing Greedy MLP 0.0
PP MMD (grid) Cal Housing Latent Deep Ensemble 0.006
PP MMD (grid) Cal Housing Latent Linear 0.01
PP MMD (grid) Cal Housing Latent MLP 0.026
PP MMD (grid) Credit Default DICE Deep Ensemble 0.0
PP MMD (grid) Credit Default DICE Linear 0.0
PP MMD (grid) Credit Default DICE MLP 0.0
PP MMD (grid) Credit Default Generic (γ=0.5) Deep Ensemble 0.0
PP MMD (grid) Credit Default Generic (γ=0.5) Linear 0.0
PP MMD (grid) Credit Default Generic (γ=0.5) MLP 0.0
PP MMD (grid) Credit Default Greedy Deep Ensemble 0.164
PP MMD (grid) Credit Default Greedy Linear 0.0
PP MMD (grid) Credit Default Greedy MLP 0.0
PP MMD (grid) Credit Default Latent Deep Ensemble 0.0
PP MMD (grid) Credit Default Latent Linear 0.044
PP MMD (grid) Credit Default Latent MLP 0.0
PP MMD (grid) GMSC DICE Deep Ensemble 0.0
PP MMD (grid) GMSC DICE Linear 0.0
PP MMD (grid) GMSC DICE MLP 0.004
PP MMD (grid) GMSC Generic (γ=0.5) Deep Ensemble 0.002
PP MMD (grid) GMSC Generic (γ=0.5) Linear 0.0
PP MMD (grid) GMSC Generic (γ=0.5) MLP 0.0
PP MMD (grid) GMSC Greedy Deep Ensemble 0.0
PP MMD (grid) GMSC Greedy Linear 0.0
PP MMD (grid) GMSC Greedy MLP 0.0
PP MMD (grid) GMSC Latent Deep Ensemble 0.0
PP MMD (grid) GMSC Latent Linear 0.0
PP MMD (grid) GMSC Latent MLP 0.03

D.3 Detailed Results: Mitigation

D.3.1 Line Charts

The evolution of the evaluation metrics over the course of the experiment is shown for different datasets in Figure D.15 to Figure D.21.

Figure D.15space Evolution of evaluation metrics over the course of the experiment. Data: California Housing.
Figure D.16space Evolution of evaluation metrics over the course of the experiment. Data: Circles.
Figure D.17space Evolution of evaluation metrics over the course of the experiment. Data: Credit Default.
Figure D.18space Evolution of evaluation metrics over the course of the experiment. Data: GMSC.
Figure D.19space Evolution of evaluation metrics over the course of the experiment. Data: Linearly Separable.
Figure D.20space Evolution of evaluation metrics over the course of the experiment. Data: Moons.
Figure D.21space Evolution of evaluation metrics over the course of the experiment. Data: Overlapping.

D.3.2 Error Bar Charts

The evaluation metrics at the end of the experiment are shown for different datasets in Figure D.22 to Figure D.28.

Figure D.22space Evaluation metrics at the end of the experiment. Data: California Housing.
Figure D.23space Evaluation metrics at the end of the experiment. Data: Circles.
Figure D.24space Evaluation metrics at the end of the experiment. Data: Credit Default.
Figure D.25space Evaluation metrics at the end of the experiment. Data: GMSC.
Figure D.26space Evaluation metrics at the end of the experiment. Data: Linearly Separable.
Figure D.27space Evaluation metrics at the end of the experiment. Data: Moons.
Figure D.28space Evaluation metrics at the end of the experiment. Data: Overlapping.

D.3.3 Statistical Significance

Table D.3 presents the tests for statistical significance of the estimated MMD metrics.

Table D.3space Tests for statistical significance of the estimated MMD metrics using mitigation strategies. We have highlighted p-values smaller than the significance level \(\alpha=0.05\) in bold. Data: Synthetic.
Metric Data Generator Model p-value
MMD Circles ClapROAR Deep Ensemble 0.984
MMD Circles ClapROAR Linear 1.0
MMD Circles ClapROAR MLP 0.992
MMD Circles Generic (γ=0.5) Deep Ensemble 0.99
MMD Circles Generic (γ=0.5) Linear 1.0
MMD Circles Generic (γ=0.5) MLP 0.994
MMD Circles Generic (γ=0.9) Deep Ensemble 0.996
MMD Circles Generic (γ=0.9) Linear 1.0
MMD Circles Generic (γ=0.9) MLP 0.992
MMD Circles Gravitational Deep Ensemble 0.998
MMD Circles Gravitational Linear 1.0
MMD Circles Gravitational MLP 0.998
MMD Circles Latent Deep Ensemble 1.0
MMD Circles Latent Linear 1.0
MMD Circles Latent MLP 1.0
MMD Linearly Separable ClapROAR Deep Ensemble 0.0
MMD Linearly Separable ClapROAR Linear 0.0
MMD Linearly Separable ClapROAR MLP 0.0
MMD Linearly Separable Generic (γ=0.5) Deep Ensemble 0.0
MMD Linearly Separable Generic (γ=0.5) Linear 0.0
MMD Linearly Separable Generic (γ=0.5) MLP 0.0
MMD Linearly Separable Generic (γ=0.9) Deep Ensemble 0.0
MMD Linearly Separable Generic (γ=0.9) Linear 0.0
MMD Linearly Separable Generic (γ=0.9) MLP 0.0
MMD Linearly Separable Gravitational Deep Ensemble 0.05
MMD Linearly Separable Gravitational Linear 0.092
MMD Linearly Separable Gravitational MLP 0.078
MMD Linearly Separable Latent Deep Ensemble 0.724
MMD Linearly Separable Latent Linear 0.75
MMD Linearly Separable Latent MLP 0.742
MMD Moons ClapROAR Deep Ensemble 0.0
MMD Moons ClapROAR Linear 0.0
MMD Moons ClapROAR MLP 0.0
MMD Moons Generic (γ=0.5) Deep Ensemble 0.0
MMD Moons Generic (γ=0.5) Linear 0.0
MMD Moons Generic (γ=0.5) MLP 0.0
MMD Moons Generic (γ=0.9) Deep Ensemble 0.0
MMD Moons Generic (γ=0.9) Linear 0.0
MMD Moons Generic (γ=0.9) MLP 0.0
MMD Moons Gravitational Deep Ensemble 0.0
MMD Moons Gravitational Linear 0.0
MMD Moons Gravitational MLP 0.0
MMD Moons Latent Deep Ensemble 0.0
MMD Moons Latent Linear 0.0
MMD Moons Latent MLP 0.0
MMD Overlapping ClapROAR Deep Ensemble 0.0
MMD Overlapping ClapROAR Linear 0.0
MMD Overlapping ClapROAR MLP 0.0
MMD Overlapping Generic (γ=0.5) Deep Ensemble 0.0
MMD Overlapping Generic (γ=0.5) Linear 0.0
MMD Overlapping Generic (γ=0.5) MLP 0.0
MMD Overlapping Generic (γ=0.9) Deep Ensemble 0.0
MMD Overlapping Generic (γ=0.9) Linear 0.0
MMD Overlapping Generic (γ=0.9) MLP 0.0
MMD Overlapping Gravitational Deep Ensemble 0.0
MMD Overlapping Gravitational Linear 0.0
MMD Overlapping Gravitational MLP 0.0
MMD Overlapping Latent Deep Ensemble 0.0
MMD Overlapping Latent Linear 0.0
MMD Overlapping Latent MLP 0.0
PP MMD Circles ClapROAR Deep Ensemble 0.998
PP MMD Circles ClapROAR Linear 0.996
PP MMD Circles ClapROAR MLP 0.998
PP MMD Circles Generic (γ=0.5) Deep Ensemble 0.998
PP MMD Circles Generic (γ=0.5) Linear 0.8
PP MMD Circles Generic (γ=0.5) MLP 1.0
PP MMD Circles Generic (γ=0.9) Deep Ensemble 0.998
PP MMD Circles Generic (γ=0.9) Linear 0.996
PP MMD Circles Generic (γ=0.9) MLP 1.0
PP MMD Circles Gravitational Deep Ensemble 0.978
PP MMD Circles Gravitational Linear 0.0
PP MMD Circles Gravitational MLP 0.986
PP MMD Circles Latent Deep Ensemble 1.0
PP MMD Circles Latent Linear 0.0
PP MMD Circles Latent MLP 0.998
PP MMD Linearly Separable ClapROAR Deep Ensemble 0.962
PP MMD Linearly Separable ClapROAR Linear 0.916
PP MMD Linearly Separable ClapROAR MLP 0.958
PP MMD Linearly Separable Generic (γ=0.5) Deep Ensemble 0.922
PP MMD Linearly Separable Generic (γ=0.5) Linear 0.0
PP MMD Linearly Separable Generic (γ=0.5) MLP 0.916
PP MMD Linearly Separable Generic (γ=0.9) Deep Ensemble 0.968
PP MMD Linearly Separable Generic (γ=0.9) Linear 0.376
PP MMD Linearly Separable Generic (γ=0.9) MLP 0.968
PP MMD Linearly Separable Gravitational Deep Ensemble 0.976
PP MMD Linearly Separable Gravitational Linear 0.904
PP MMD Linearly Separable Gravitational MLP 0.982
PP MMD Linearly Separable Latent Deep Ensemble 0.862
PP MMD Linearly Separable Latent Linear 0.428
PP MMD Linearly Separable Latent MLP 0.83
PP MMD Moons ClapROAR Deep Ensemble 0.966
PP MMD Moons ClapROAR Linear 0.462
PP MMD Moons ClapROAR MLP 0.956
PP MMD Moons Generic (γ=0.5) Deep Ensemble 0.822
PP MMD Moons Generic (γ=0.5) Linear 0.0
PP MMD Moons Generic (γ=0.5) MLP 0.812
PP MMD Moons Generic (γ=0.9) Deep Ensemble 0.818
PP MMD Moons Generic (γ=0.9) Linear 0.086
PP MMD Moons Generic (γ=0.9) MLP 0.87
PP MMD Moons Gravitational Deep Ensemble 0.9775
PP MMD Moons Gravitational Linear 0.446
PP MMD Moons Gravitational MLP 0.984
PP MMD Moons Latent Deep Ensemble 0.922
PP MMD Moons Latent Linear 0.008
PP MMD Moons Latent MLP 0.94
PP MMD Overlapping ClapROAR Deep Ensemble 0.46
PP MMD Overlapping ClapROAR Linear 0.178
PP MMD Overlapping ClapROAR MLP 0.486
PP MMD Overlapping Generic (γ=0.5) Deep Ensemble 0.0
PP MMD Overlapping Generic (γ=0.5) Linear 0.0
PP MMD Overlapping Generic (γ=0.5) MLP 0.004
PP MMD Overlapping Generic (γ=0.9) Deep Ensemble 0.122
PP MMD Overlapping Generic (γ=0.9) Linear 0.066
PP MMD Overlapping Generic (γ=0.9) MLP 0.13
PP MMD Overlapping Gravitational Deep Ensemble 0.514
PP MMD Overlapping Gravitational Linear 0.156
PP MMD Overlapping Gravitational MLP 0.564
PP MMD Overlapping Latent Deep Ensemble 0.048
PP MMD Overlapping Latent Linear 0.006
PP MMD Overlapping Latent MLP 0.046
PP MMD (grid) Circles ClapROAR Deep Ensemble 0.984
PP MMD (grid) Circles ClapROAR Linear 0.996
PP MMD (grid) Circles ClapROAR MLP 0.99
PP MMD (grid) Circles Generic (γ=0.5) Deep Ensemble 0.886
PP MMD (grid) Circles Generic (γ=0.5) Linear 0.814
PP MMD (grid) Circles Generic (γ=0.5) MLP 0.814
PP MMD (grid) Circles Generic (γ=0.9) Deep Ensemble 0.84
PP MMD (grid) Circles Generic (γ=0.9) Linear 0.988
PP MMD (grid) Circles Generic (γ=0.9) MLP 0.932
PP MMD (grid) Circles Gravitational Deep Ensemble 0.55
PP MMD (grid) Circles Gravitational Linear 0.0
PP MMD (grid) Circles Gravitational MLP 0.406
PP MMD (grid) Circles Latent Deep Ensemble 0.996
PP MMD (grid) Circles Latent Linear 0.0
PP MMD (grid) Circles Latent MLP 0.99
PP MMD (grid) Linearly Separable ClapROAR Deep Ensemble 0.0
PP MMD (grid) Linearly Separable ClapROAR Linear 0.006
PP MMD (grid) Linearly Separable ClapROAR MLP 0.0
PP MMD (grid) Linearly Separable Generic (γ=0.5) Deep Ensemble 0.0
PP MMD (grid) Linearly Separable Generic (γ=0.5) Linear 0.0
PP MMD (grid) Linearly Separable Generic (γ=0.5) MLP 0.0
PP MMD (grid) Linearly Separable Generic (γ=0.9) Deep Ensemble 0.0
PP MMD (grid) Linearly Separable Generic (γ=0.9) Linear 0.0
PP MMD (grid) Linearly Separable Generic (γ=0.9) MLP 0.0
PP MMD (grid) Linearly Separable Gravitational Deep Ensemble 0.408
PP MMD (grid) Linearly Separable Gravitational Linear 0.342
PP MMD (grid) Linearly Separable Gravitational MLP 0.668
PP MMD (grid) Linearly Separable Latent Deep Ensemble 0.0
PP MMD (grid) Linearly Separable Latent Linear 0.0
PP MMD (grid) Linearly Separable Latent MLP 0.0
PP MMD (grid) Moons ClapROAR Deep Ensemble 0.0
PP MMD (grid) Moons ClapROAR Linear 0.458
PP MMD (grid) Moons ClapROAR MLP 0.004
PP MMD (grid) Moons Generic (γ=0.5) Deep Ensemble 0.0
PP MMD (grid) Moons Generic (γ=0.5) Linear 0.0
PP MMD (grid) Moons Generic (γ=0.5) MLP 0.016
PP MMD (grid) Moons Generic (γ=0.9) Deep Ensemble 0.006
PP MMD (grid) Moons Generic (γ=0.9) Linear 0.09
PP MMD (grid) Moons Generic (γ=0.9) MLP 0.03
PP MMD (grid) Moons Gravitational Deep Ensemble 0.4
PP MMD (grid) Moons Gravitational Linear 0.456
PP MMD (grid) Moons Gravitational MLP 0.426
PP MMD (grid) Moons Latent Deep Ensemble 0.344
PP MMD (grid) Moons Latent Linear 0.008
PP MMD (grid) Moons Latent MLP 0.114
PP MMD (grid) Overlapping ClapROAR Deep Ensemble 0.4075
PP MMD (grid) Overlapping ClapROAR Linear 0.256
PP MMD (grid) Overlapping ClapROAR MLP 0.298
PP MMD (grid) Overlapping Generic (γ=0.5) Deep Ensemble 0.002
PP MMD (grid) Overlapping Generic (γ=0.5) Linear 0.0
PP MMD (grid) Overlapping Generic (γ=0.5) MLP 0.0
PP MMD (grid) Overlapping Generic (γ=0.9) Deep Ensemble 0.154
PP MMD (grid) Overlapping Generic (γ=0.9) Linear 0.104
PP MMD (grid) Overlapping Generic (γ=0.9) MLP 0.116
PP MMD (grid) Overlapping Gravitational Deep Ensemble 0.356
PP MMD (grid) Overlapping Gravitational Linear 0.27
PP MMD (grid) Overlapping Gravitational MLP 0.344
PP MMD (grid) Overlapping Latent Deep Ensemble 0.324
PP MMD (grid) Overlapping Latent Linear 0.01
PP MMD (grid) Overlapping Latent MLP 0.204

Table D.4 presents the tests for statistical significance of the estimated MMD metrics.

Table D.4space Tests for statistical significance of the estimated MMD metrics using mitigation strategies. We have highlighted p-values smaller than the significance level \(\alpha=0.05\) in bold. Data: Real-World.
Metric Data Generator Model p-value
MMD Cal Housing ClapROAR Deep Ensemble 0.0
MMD Cal Housing ClapROAR Linear 0.0
MMD Cal Housing ClapROAR MLP 0.0
MMD Cal Housing Generic (γ=0.5) Deep Ensemble 0.0
MMD Cal Housing Generic (γ=0.5) Linear 0.0
MMD Cal Housing Generic (γ=0.5) MLP 0.0
MMD Cal Housing Generic (γ=0.9) Deep Ensemble 0.0
MMD Cal Housing Generic (γ=0.9) Linear 0.0
MMD Cal Housing Generic (γ=0.9) MLP 0.0
MMD Cal Housing Gravitational Deep Ensemble 0.0
MMD Cal Housing Gravitational Linear 0.0
MMD Cal Housing Gravitational MLP 0.0
MMD Cal Housing Latent Deep Ensemble 0.0
MMD Cal Housing Latent Linear 0.0
MMD Cal Housing Latent MLP 0.0
MMD Credit Default ClapROAR Deep Ensemble 1.0
MMD Credit Default ClapROAR Linear 1.0
MMD Credit Default ClapROAR MLP 1.0
MMD Credit Default Generic (γ=0.5) Deep Ensemble 1.0
MMD Credit Default Generic (γ=0.5) Linear 1.0
MMD Credit Default Generic (γ=0.5) MLP 1.0
MMD Credit Default Generic (γ=0.9) Deep Ensemble 1.0
MMD Credit Default Generic (γ=0.9) Linear 1.0
MMD Credit Default Generic (γ=0.9) MLP 1.0
MMD Credit Default Gravitational Deep Ensemble 0.0
MMD Credit Default Gravitational Linear 0.0
MMD Credit Default Gravitational MLP 0.0
MMD Credit Default Latent Deep Ensemble 0.0
MMD Credit Default Latent Linear 0.8
MMD Credit Default Latent MLP 0.0
MMD GMSC ClapROAR Deep Ensemble 0.15
MMD GMSC ClapROAR Linear 0.0
MMD GMSC ClapROAR MLP 0.214
MMD GMSC Generic (γ=0.5) Deep Ensemble 0.938
MMD GMSC Generic (γ=0.5) Linear 0.856
MMD GMSC Generic (γ=0.5) MLP 0.932
MMD GMSC Generic (γ=0.9) Deep Ensemble 0.758
MMD GMSC Generic (γ=0.9) Linear 0.004
MMD GMSC Generic (γ=0.9) MLP 0.93
MMD GMSC Gravitational Deep Ensemble 0.0
MMD GMSC Gravitational Linear 0.0
MMD GMSC Gravitational MLP 0.0
MMD GMSC Latent Deep Ensemble 0.0
MMD GMSC Latent Linear 0.0
MMD GMSC Latent MLP 0.0
PP MMD Cal Housing ClapROAR Deep Ensemble 0.0
PP MMD Cal Housing ClapROAR Linear 0.0
PP MMD Cal Housing ClapROAR MLP 0.0
PP MMD Cal Housing Generic (γ=0.5) Deep Ensemble 0.0
PP MMD Cal Housing Generic (γ=0.5) Linear 0.0
PP MMD Cal Housing Generic (γ=0.5) MLP 0.0
PP MMD Cal Housing Generic (γ=0.9) Deep Ensemble 0.0
PP MMD Cal Housing Generic (γ=0.9) Linear 0.0
PP MMD Cal Housing Generic (γ=0.9) MLP 0.0
PP MMD Cal Housing Gravitational Deep Ensemble 0.0
PP MMD Cal Housing Gravitational Linear 0.0
PP MMD Cal Housing Gravitational MLP 0.0
PP MMD Cal Housing Latent Deep Ensemble 0.0
PP MMD Cal Housing Latent Linear 0.0
PP MMD Cal Housing Latent MLP 0.0
PP MMD Credit Default ClapROAR Deep Ensemble 0.0
PP MMD Credit Default ClapROAR Linear 0.0
PP MMD Credit Default ClapROAR MLP 0.0
PP MMD Credit Default Generic (γ=0.5) Deep Ensemble 0.0
PP MMD Credit Default Generic (γ=0.5) Linear 0.0
PP MMD Credit Default Generic (γ=0.5) MLP 0.0
PP MMD Credit Default Generic (γ=0.9) Deep Ensemble 0.0
PP MMD Credit Default Generic (γ=0.9) Linear 0.0
PP MMD Credit Default Generic (γ=0.9) MLP 0.0
PP MMD Credit Default Gravitational Deep Ensemble 0.0
PP MMD Credit Default Gravitational Linear 0.0
PP MMD Credit Default Gravitational MLP 0.0
PP MMD Credit Default Latent Deep Ensemble 0.0
PP MMD Credit Default Latent Linear 0.0
PP MMD Credit Default Latent MLP 0.0
PP MMD GMSC ClapROAR Deep Ensemble 0.0
PP MMD GMSC ClapROAR Linear 0.0
PP MMD GMSC ClapROAR MLP 0.0
PP MMD GMSC Generic (γ=0.5) Deep Ensemble 0.0
PP MMD GMSC Generic (γ=0.5) Linear 0.0
PP MMD GMSC Generic (γ=0.5) MLP 0.0
PP MMD GMSC Generic (γ=0.9) Deep Ensemble 0.0
PP MMD GMSC Generic (γ=0.9) Linear 0.0
PP MMD GMSC Generic (γ=0.9) MLP 0.0
PP MMD GMSC Gravitational Deep Ensemble 0.0
PP MMD GMSC Gravitational Linear 0.0
PP MMD GMSC Gravitational MLP 0.0
PP MMD GMSC Latent Deep Ensemble 0.0
PP MMD GMSC Latent Linear 0.0
PP MMD GMSC Latent MLP 0.0
PP MMD (grid) Cal Housing ClapROAR Deep Ensemble 0.044
PP MMD (grid) Cal Housing ClapROAR Linear 0.004
PP MMD (grid) Cal Housing ClapROAR MLP 0.012
PP MMD (grid) Cal Housing Generic (γ=0.5) Deep Ensemble 0.0
PP MMD (grid) Cal Housing Generic (γ=0.5) Linear 0.0
PP MMD (grid) Cal Housing Generic (γ=0.5) MLP 0.0
PP MMD (grid) Cal Housing Generic (γ=0.9) Deep Ensemble 0.002
PP MMD (grid) Cal Housing Generic (γ=0.9) Linear 0.0
PP MMD (grid) Cal Housing Generic (γ=0.9) MLP 0.0
PP MMD (grid) Cal Housing Gravitational Deep Ensemble 0.0
PP MMD (grid) Cal Housing Gravitational Linear 0.014
PP MMD (grid) Cal Housing Gravitational MLP 0.0625
PP MMD (grid) Cal Housing Latent Deep Ensemble 0.0
PP MMD (grid) Cal Housing Latent Linear 0.002
PP MMD (grid) Cal Housing Latent MLP 0.0
PP MMD (grid) Credit Default ClapROAR Deep Ensemble 0.0
PP MMD (grid) Credit Default ClapROAR Linear 0.0
PP MMD (grid) Credit Default ClapROAR MLP 0.0
PP MMD (grid) Credit Default Generic (γ=0.5) Deep Ensemble 0.0
PP MMD (grid) Credit Default Generic (γ=0.5) Linear 0.0
PP MMD (grid) Credit Default Generic (γ=0.5) MLP 0.0
PP MMD (grid) Credit Default Generic (γ=0.9) Deep Ensemble 0.0
PP MMD (grid) Credit Default Generic (γ=0.9) Linear 0.0
PP MMD (grid) Credit Default Generic (γ=0.9) MLP 0.0
PP MMD (grid) Credit Default Gravitational Deep Ensemble 0.0
PP MMD (grid) Credit Default Gravitational Linear 0.0
PP MMD (grid) Credit Default Gravitational MLP 0.0
PP MMD (grid) Credit Default Latent Deep Ensemble 0.0
PP MMD (grid) Credit Default Latent Linear 0.078
PP MMD (grid) Credit Default Latent MLP 0.0
PP MMD (grid) GMSC ClapROAR Deep Ensemble 0.0
PP MMD (grid) GMSC ClapROAR Linear 0.0
PP MMD (grid) GMSC ClapROAR MLP 0.0
PP MMD (grid) GMSC Generic (γ=0.5) Deep Ensemble 0.0
PP MMD (grid) GMSC Generic (γ=0.5) Linear 0.0
PP MMD (grid) GMSC Generic (γ=0.5) MLP 0.0
PP MMD (grid) GMSC Generic (γ=0.9) Deep Ensemble 0.0
PP MMD (grid) GMSC Generic (γ=0.9) Linear 0.0
PP MMD (grid) GMSC Generic (γ=0.9) MLP 0.0
PP MMD (grid) GMSC Gravitational Deep Ensemble 0.0
PP MMD (grid) GMSC Gravitational Linear 0.0
PP MMD (grid) GMSC Gravitational MLP 0.0
PP MMD (grid) GMSC Latent Deep Ensemble 0.0
PP MMD (grid) GMSC Latent Linear 0.0
PP MMD (grid) GMSC Latent MLP 0.0