5  Mitigation Strategies

Code 
projectdir = splitpath(pwd()) |>
    ss -> joinpath(ss[1:findall([s == "endogenous-macrodynamics-in-algorithmic-recourse" for s in ss])[1]]...) 
cd(projectdir)
Code 
models = [
    :LogisticRegression, 
    :FluxModel, 
    :FluxEnsemble,
]
opt = Flux.Descent(0.01) 
generators = Dict(
    :Generic=>GenericGenerator(opt = opt, decision_threshold=0.5),
    :Latent=>REVISEGenerator(opt = opt),
    :Generic_conservative=>GenericGenerator(opt = opt, decision_threshold=0.9),
    :Gravitational=>GravitationalGenerator(opt = opt),
    :ClapROAR=>ClapROARGenerator(opt = opt)
)

5.1 Mitigation Strategies

5.1.1 Running the Experiment

5.1.1.1 Synthetic

Code 
max_obs = 1000
catalogue = load_synthetic(max_obs)
choices = [
    :linearly_separable, 
    :overlapping, 
    :circles, 
    :moons,
]
data_sets = filter(p -> p[1] in choices, catalogue)
Code 
experiments = set_up_experiments(data_sets,models,generators)
Code 
n_evals = 5
n_rounds = 50
evaluate_every = Int(round(n_rounds/n_evals))
n_folds = 5
T = 100
using Serialization
results = run_experiments(
    experiments;
    save_path=output_path,evaluate_every=evaluate_every,n_rounds=n_rounds, n_folds=n_folds, T=T
)
Serialization.serialize(joinpath(output_path,"results_synthetic.jls"),results)

5.1.1.2 Real World

Code 
models = [
    :LogisticRegression, 
    :FluxModel, 
    :FluxEnsemble,
]
opt = Flux.Descent(0.01) 
generators = Dict(
    :Generic=>GenericGenerator(opt = opt, decision_threshold=0.5),
    :Latent=>REVISEGenerator(opt = opt),
    :Generic_conservative=>GenericGenerator(opt = opt, decision_threshold=0.9),
    :Gravitational=>GravitationalGenerator(opt = opt),
    :ClapROAR=>ClapROARGenerator(opt = opt)
)
Code 
max_obs = 2500
data_path = data_dir("real_world")
data_sets = load_real_world(max_obs)
choices = [
    :cal_housing, 
    :credit_default, 
    :gmsc, 
]
data_sets = filter(p -> p[1] in choices, data_sets)
Code 
using CounterfactualExplanations.DataPreprocessing: unpack
bs = 500
function data_loader(data::CounterfactualData)
    X, y = unpack(data)
    data = Flux.DataLoader((X,y),batchsize=bs)
    return data
end
model_params = (batch_norm=false,n_hidden=64,n_layers=3,dropout=true,p_dropout=0.1)
Code 
experiments = set_up_experiments(
    data_sets,models,generators; 
    pre_train_models=100, model_params=model_params, 
    data_loader=data_loader
)
Code 
n_evals = 5
n_rounds = 50
evaluate_every = Int(round(n_rounds/n_evals))
n_folds = 5
n_samples = 10000
T = 100
generative_model_params = (epochs=250, latent_dim=8)
results = run_experiments(
    experiments;
    save_path=output_path,evaluate_every=evaluate_every,n_rounds=n_rounds, n_folds=n_folds, T=T, n_samples=n_samples,
    generative_model_params=generative_model_params
)
Serialization.serialize(joinpath(output_path,"results_real_world.jls"),results)

5.1.2 Results

Code 
data_names = [
    "California Housing",
    "Circles",
    "Credit Default",
    "GMSC",
    "Linearly Separable", 
    "Moons", 
    "Overlapping"
]
results = merge(EMAR.load_synthetic_mitigation(), EMAR.load_real_world_mitigation())

5.1.2.1 Animated GIFs

Code 
using Images
line_charts = Dict()
errorbar_charts = Dict()
for (data_name, res) in results
    plt = plot_res(res)
    Images.save(joinpath(www_path, "line_chart_$(data_name).png"), plt)
    Images.save(joinpath(www_paper_path, "line_chart_$(data_name).png"), plt)
    line_charts[data_name] = plt
    plt = plot_res(res,maximum(res.output.n))
    Images.save(joinpath(www_path, "errorbar_chart_$(data_name).png"), plt)
    Images.save(joinpath(www_paper_path, "errorbar_chart_$(data_name).png"), plt)
    errorbar_charts[data_name] = plt
end

5.1.2.2 Line Charts

The evolution of the evaluation metrics over the course of the experiment is shown for different datasets in Figure 5.1 to Figure 5.7.

Figure 5.1: Evolution of evaluation metrics over the course of the experiment. Data: California Housing.
Figure 5.2: Evolution of evaluation metrics over the course of the experiment. Data: Circles.
Figure 5.3: Evolution of evaluation metrics over the course of the experiment. Data: Credit Default.
Figure 5.4: Evolution of evaluation metrics over the course of the experiment. Data: GMSC.
Figure 5.5: Evolution of evaluation metrics over the course of the experiment. Data: Linearly Separable.
Figure 5.6: Evolution of evaluation metrics over the course of the experiment. Data: Moons.
Figure 5.7: Evolution of evaluation metrics over the course of the experiment. Data: Overlapping.

5.1.2.3 Error Bar Charts

The evaluation metrics at the end of the experiment are shown for different datasets in Figure 5.8 to Figure 5.14.

Figure 5.8: Evaluation metrics at the end of the experiment. Data: California Housing.
Figure 5.9: Evaluation metrics at the end of the experiment. Data: Circles.
Figure 5.10: Evaluation metrics at the end of the experiment. Data: Credit Default.
Figure 5.11: Evaluation metrics at the end of the experiment. Data: GMSC.
Figure 5.12: Evaluation metrics at the end of the experiment. Data: Linearly Separable.
Figure 5.13: Evaluation metrics at the end of the experiment. Data: Moons.
Figure 5.14: Evaluation metrics at the end of the experiment. Data: Overlapping.

5.1.2.4 Bootstrap

5.1.2.4.1 Synthetic
Code 
n_bootstrap = 100
df = run_bootstrap(results, n_bootstrap; filename=joinpath(output_path,"bootstrap_synthetic.csv"))

Table 5.1 presents the tests for statistical significance of the estimated MMD metrics.

Code 
df = EMAR.aggregate_bs_mitigation_synthetic()
CSV.write(joinpath(www_path, "bootstrap_synthetic.csv"), df)
CSV.write(joinpath(www_paper_path, "bootstrap_synthetic.csv"), df)
EMAR.tabulate_bs(df, Val(:html))
Table 5.1: Tests for statistical significance of the estimated MMD metrics using mitigation strategies. We have highlighted p-values smaller than the significance level \(\alpha=0.05\) in bold. Data: Synthetic.
Metric Data Generator Model p-value
MMD Circles ClapROAR Deep Ensemble 0.984
MMD Circles ClapROAR Linear 1.0
MMD Circles ClapROAR MLP 0.992
MMD Circles Generic (γ=0.5) Deep Ensemble 0.99
MMD Circles Generic (γ=0.5) Linear 1.0
MMD Circles Generic (γ=0.5) MLP 0.994
MMD Circles Generic (γ=0.9) Deep Ensemble 0.996
MMD Circles Generic (γ=0.9) Linear 1.0
MMD Circles Generic (γ=0.9) MLP 0.992
MMD Circles Gravitational Deep Ensemble 0.998
MMD Circles Gravitational Linear 1.0
MMD Circles Gravitational MLP 0.998
MMD Circles Latent Deep Ensemble 1.0
MMD Circles Latent Linear 1.0
MMD Circles Latent MLP 1.0
MMD Linearly Separable ClapROAR Deep Ensemble 0.0
MMD Linearly Separable ClapROAR Linear 0.0
MMD Linearly Separable ClapROAR MLP 0.0
MMD Linearly Separable Generic (γ=0.5) Deep Ensemble 0.0
MMD Linearly Separable Generic (γ=0.5) Linear 0.0
MMD Linearly Separable Generic (γ=0.5) MLP 0.0
MMD Linearly Separable Generic (γ=0.9) Deep Ensemble 0.0
MMD Linearly Separable Generic (γ=0.9) Linear 0.0
MMD Linearly Separable Generic (γ=0.9) MLP 0.0
MMD Linearly Separable Gravitational Deep Ensemble 0.05
MMD Linearly Separable Gravitational Linear 0.092
MMD Linearly Separable Gravitational MLP 0.078
MMD Linearly Separable Latent Deep Ensemble 0.724
MMD Linearly Separable Latent Linear 0.75
MMD Linearly Separable Latent MLP 0.742
MMD Moons ClapROAR Deep Ensemble 0.0
MMD Moons ClapROAR Linear 0.0
MMD Moons ClapROAR MLP 0.0
MMD Moons Generic (γ=0.5) Deep Ensemble 0.0
MMD Moons Generic (γ=0.5) Linear 0.0
MMD Moons Generic (γ=0.5) MLP 0.0
MMD Moons Generic (γ=0.9) Deep Ensemble 0.0
MMD Moons Generic (γ=0.9) Linear 0.0
MMD Moons Generic (γ=0.9) MLP 0.0
MMD Moons Gravitational Deep Ensemble 0.0
MMD Moons Gravitational Linear 0.0
MMD Moons Gravitational MLP 0.0
MMD Moons Latent Deep Ensemble 0.0
MMD Moons Latent Linear 0.0
MMD Moons Latent MLP 0.0
MMD Overlapping ClapROAR Deep Ensemble 0.0
MMD Overlapping ClapROAR Linear 0.0
MMD Overlapping ClapROAR MLP 0.0
MMD Overlapping Generic (γ=0.5) Deep Ensemble 0.0
MMD Overlapping Generic (γ=0.5) Linear 0.0
MMD Overlapping Generic (γ=0.5) MLP 0.0
MMD Overlapping Generic (γ=0.9) Deep Ensemble 0.0
MMD Overlapping Generic (γ=0.9) Linear 0.0
MMD Overlapping Generic (γ=0.9) MLP 0.0
MMD Overlapping Gravitational Deep Ensemble 0.0
MMD Overlapping Gravitational Linear 0.0
MMD Overlapping Gravitational MLP 0.0
MMD Overlapping Latent Deep Ensemble 0.0
MMD Overlapping Latent Linear 0.0
MMD Overlapping Latent MLP 0.0
PP MMD Circles ClapROAR Deep Ensemble 0.998
PP MMD Circles ClapROAR Linear 0.996
PP MMD Circles ClapROAR MLP 0.998
PP MMD Circles Generic (γ=0.5) Deep Ensemble 0.998
PP MMD Circles Generic (γ=0.5) Linear 0.8
PP MMD Circles Generic (γ=0.5) MLP 1.0
PP MMD Circles Generic (γ=0.9) Deep Ensemble 0.998
PP MMD Circles Generic (γ=0.9) Linear 0.996
PP MMD Circles Generic (γ=0.9) MLP 1.0
PP MMD Circles Gravitational Deep Ensemble 0.978
PP MMD Circles Gravitational Linear 0.0
PP MMD Circles Gravitational MLP 0.986
PP MMD Circles Latent Deep Ensemble 1.0
PP MMD Circles Latent Linear 0.0
PP MMD Circles Latent MLP 0.998
PP MMD Linearly Separable ClapROAR Deep Ensemble 0.962
PP MMD Linearly Separable ClapROAR Linear 0.916
PP MMD Linearly Separable ClapROAR MLP 0.958
PP MMD Linearly Separable Generic (γ=0.5) Deep Ensemble 0.922
PP MMD Linearly Separable Generic (γ=0.5) Linear 0.0
PP MMD Linearly Separable Generic (γ=0.5) MLP 0.916
PP MMD Linearly Separable Generic (γ=0.9) Deep Ensemble 0.968
PP MMD Linearly Separable Generic (γ=0.9) Linear 0.376
PP MMD Linearly Separable Generic (γ=0.9) MLP 0.968
PP MMD Linearly Separable Gravitational Deep Ensemble 0.976
PP MMD Linearly Separable Gravitational Linear 0.904
PP MMD Linearly Separable Gravitational MLP 0.982
PP MMD Linearly Separable Latent Deep Ensemble 0.862
PP MMD Linearly Separable Latent Linear 0.428
PP MMD Linearly Separable Latent MLP 0.83
PP MMD Moons ClapROAR Deep Ensemble 0.966
PP MMD Moons ClapROAR Linear 0.462
PP MMD Moons ClapROAR MLP 0.956
PP MMD Moons Generic (γ=0.5) Deep Ensemble 0.822
PP MMD Moons Generic (γ=0.5) Linear 0.0
PP MMD Moons Generic (γ=0.5) MLP 0.812
PP MMD Moons Generic (γ=0.9) Deep Ensemble 0.818
PP MMD Moons Generic (γ=0.9) Linear 0.086
PP MMD Moons Generic (γ=0.9) MLP 0.87
PP MMD Moons Gravitational Deep Ensemble 0.9775
PP MMD Moons Gravitational Linear 0.446
PP MMD Moons Gravitational MLP 0.984
PP MMD Moons Latent Deep Ensemble 0.922
PP MMD Moons Latent Linear 0.008
PP MMD Moons Latent MLP 0.94
PP MMD Overlapping ClapROAR Deep Ensemble 0.46
PP MMD Overlapping ClapROAR Linear 0.178
PP MMD Overlapping ClapROAR MLP 0.486
PP MMD Overlapping Generic (γ=0.5) Deep Ensemble 0.0
PP MMD Overlapping Generic (γ=0.5) Linear 0.0
PP MMD Overlapping Generic (γ=0.5) MLP 0.004
PP MMD Overlapping Generic (γ=0.9) Deep Ensemble 0.122
PP MMD Overlapping Generic (γ=0.9) Linear 0.066
PP MMD Overlapping Generic (γ=0.9) MLP 0.13
PP MMD Overlapping Gravitational Deep Ensemble 0.514
PP MMD Overlapping Gravitational Linear 0.156
PP MMD Overlapping Gravitational MLP 0.564
PP MMD Overlapping Latent Deep Ensemble 0.048
PP MMD Overlapping Latent Linear 0.006
PP MMD Overlapping Latent MLP 0.046
PP MMD (grid) Circles ClapROAR Deep Ensemble 0.984
PP MMD (grid) Circles ClapROAR Linear 0.996
PP MMD (grid) Circles ClapROAR MLP 0.99
PP MMD (grid) Circles Generic (γ=0.5) Deep Ensemble 0.886
PP MMD (grid) Circles Generic (γ=0.5) Linear 0.814
PP MMD (grid) Circles Generic (γ=0.5) MLP 0.814
PP MMD (grid) Circles Generic (γ=0.9) Deep Ensemble 0.84
PP MMD (grid) Circles Generic (γ=0.9) Linear 0.988
PP MMD (grid) Circles Generic (γ=0.9) MLP 0.932
PP MMD (grid) Circles Gravitational Deep Ensemble 0.55
PP MMD (grid) Circles Gravitational Linear 0.0
PP MMD (grid) Circles Gravitational MLP 0.406
PP MMD (grid) Circles Latent Deep Ensemble 0.996
PP MMD (grid) Circles Latent Linear 0.0
PP MMD (grid) Circles Latent MLP 0.99
PP MMD (grid) Linearly Separable ClapROAR Deep Ensemble 0.0
PP MMD (grid) Linearly Separable ClapROAR Linear 0.006
PP MMD (grid) Linearly Separable ClapROAR MLP 0.0
PP MMD (grid) Linearly Separable Generic (γ=0.5) Deep Ensemble 0.0
PP MMD (grid) Linearly Separable Generic (γ=0.5) Linear 0.0
PP MMD (grid) Linearly Separable Generic (γ=0.5) MLP 0.0
PP MMD (grid) Linearly Separable Generic (γ=0.9) Deep Ensemble 0.0
PP MMD (grid) Linearly Separable Generic (γ=0.9) Linear 0.0
PP MMD (grid) Linearly Separable Generic (γ=0.9) MLP 0.0
PP MMD (grid) Linearly Separable Gravitational Deep Ensemble 0.408
PP MMD (grid) Linearly Separable Gravitational Linear 0.342
PP MMD (grid) Linearly Separable Gravitational MLP 0.668
PP MMD (grid) Linearly Separable Latent Deep Ensemble 0.0
PP MMD (grid) Linearly Separable Latent Linear 0.0
PP MMD (grid) Linearly Separable Latent MLP 0.0
PP MMD (grid) Moons ClapROAR Deep Ensemble 0.0
PP MMD (grid) Moons ClapROAR Linear 0.458
PP MMD (grid) Moons ClapROAR MLP 0.004
PP MMD (grid) Moons Generic (γ=0.5) Deep Ensemble 0.0
PP MMD (grid) Moons Generic (γ=0.5) Linear 0.0
PP MMD (grid) Moons Generic (γ=0.5) MLP 0.016
PP MMD (grid) Moons Generic (γ=0.9) Deep Ensemble 0.006
PP MMD (grid) Moons Generic (γ=0.9) Linear 0.09
PP MMD (grid) Moons Generic (γ=0.9) MLP 0.03
PP MMD (grid) Moons Gravitational Deep Ensemble 0.4
PP MMD (grid) Moons Gravitational Linear 0.456
PP MMD (grid) Moons Gravitational MLP 0.426
PP MMD (grid) Moons Latent Deep Ensemble 0.344
PP MMD (grid) Moons Latent Linear 0.008
PP MMD (grid) Moons Latent MLP 0.114
PP MMD (grid) Overlapping ClapROAR Deep Ensemble 0.4075
PP MMD (grid) Overlapping ClapROAR Linear 0.256
PP MMD (grid) Overlapping ClapROAR MLP 0.298
PP MMD (grid) Overlapping Generic (γ=0.5) Deep Ensemble 0.002
PP MMD (grid) Overlapping Generic (γ=0.5) Linear 0.0
PP MMD (grid) Overlapping Generic (γ=0.5) MLP 0.0
PP MMD (grid) Overlapping Generic (γ=0.9) Deep Ensemble 0.154
PP MMD (grid) Overlapping Generic (γ=0.9) Linear 0.104
PP MMD (grid) Overlapping Generic (γ=0.9) MLP 0.116
PP MMD (grid) Overlapping Gravitational Deep Ensemble 0.356
PP MMD (grid) Overlapping Gravitational Linear 0.27
PP MMD (grid) Overlapping Gravitational MLP 0.344
PP MMD (grid) Overlapping Latent Deep Ensemble 0.324
PP MMD (grid) Overlapping Latent Linear 0.01
PP MMD (grid) Overlapping Latent MLP 0.204
5.1.2.4.2 Real-World

Table 5.2 presents the tests for statistical significance of the estimated MMD metrics.

Code 
df = EMAR.aggregate_bs_mitigation_real_world()
CSV.write(joinpath(www_path, "bootstrap_real_world.csv"), df)
CSV.write(joinpath(www_paper_path, "bootstrap_real_world.csv"), df)
EMAR.tabulate_bs(df, Val(:html))
Table 5.2: Tests for statistical significance of the estimated MMD metrics using mitigation strategies. We have highlighted p-values smaller than the significance level \(\alpha=0.05\) in bold. Data: Real-World.
Metric Data Generator Model p-value
MMD Cal Housing ClapROAR Deep Ensemble 0.0
MMD Cal Housing ClapROAR Linear 0.0
MMD Cal Housing ClapROAR MLP 0.0
MMD Cal Housing Generic (γ=0.5) Deep Ensemble 0.0
MMD Cal Housing Generic (γ=0.5) Linear 0.0
MMD Cal Housing Generic (γ=0.5) MLP 0.0
MMD Cal Housing Generic (γ=0.9) Deep Ensemble 0.0
MMD Cal Housing Generic (γ=0.9) Linear 0.0
MMD Cal Housing Generic (γ=0.9) MLP 0.0
MMD Cal Housing Gravitational Deep Ensemble 0.0
MMD Cal Housing Gravitational Linear 0.0
MMD Cal Housing Gravitational MLP 0.0
MMD Cal Housing Latent Deep Ensemble 0.0
MMD Cal Housing Latent Linear 0.0
MMD Cal Housing Latent MLP 0.0
MMD Credit Default ClapROAR Deep Ensemble 1.0
MMD Credit Default ClapROAR Linear 1.0
MMD Credit Default ClapROAR MLP 1.0
MMD Credit Default Generic (γ=0.5) Deep Ensemble 1.0
MMD Credit Default Generic (γ=0.5) Linear 1.0
MMD Credit Default Generic (γ=0.5) MLP 1.0
MMD Credit Default Generic (γ=0.9) Deep Ensemble 1.0
MMD Credit Default Generic (γ=0.9) Linear 1.0
MMD Credit Default Generic (γ=0.9) MLP 1.0
MMD Credit Default Gravitational Deep Ensemble 0.0
MMD Credit Default Gravitational Linear 0.0
MMD Credit Default Gravitational MLP 0.0
MMD Credit Default Latent Deep Ensemble 0.0
MMD Credit Default Latent Linear 0.8
MMD Credit Default Latent MLP 0.0
MMD GMSC ClapROAR Deep Ensemble 0.15
MMD GMSC ClapROAR Linear 0.0
MMD GMSC ClapROAR MLP 0.214
MMD GMSC Generic (γ=0.5) Deep Ensemble 0.938
MMD GMSC Generic (γ=0.5) Linear 0.856
MMD GMSC Generic (γ=0.5) MLP 0.932
MMD GMSC Generic (γ=0.9) Deep Ensemble 0.758
MMD GMSC Generic (γ=0.9) Linear 0.004
MMD GMSC Generic (γ=0.9) MLP 0.93
MMD GMSC Gravitational Deep Ensemble 0.0
MMD GMSC Gravitational Linear 0.0
MMD GMSC Gravitational MLP 0.0
MMD GMSC Latent Deep Ensemble 0.0
MMD GMSC Latent Linear 0.0
MMD GMSC Latent MLP 0.0
PP MMD Cal Housing ClapROAR Deep Ensemble 0.0
PP MMD Cal Housing ClapROAR Linear 0.0
PP MMD Cal Housing ClapROAR MLP 0.0
PP MMD Cal Housing Generic (γ=0.5) Deep Ensemble 0.0
PP MMD Cal Housing Generic (γ=0.5) Linear 0.0
PP MMD Cal Housing Generic (γ=0.5) MLP 0.0
PP MMD Cal Housing Generic (γ=0.9) Deep Ensemble 0.0
PP MMD Cal Housing Generic (γ=0.9) Linear 0.0
PP MMD Cal Housing Generic (γ=0.9) MLP 0.0
PP MMD Cal Housing Gravitational Deep Ensemble 0.0
PP MMD Cal Housing Gravitational Linear 0.0
PP MMD Cal Housing Gravitational MLP 0.0
PP MMD Cal Housing Latent Deep Ensemble 0.0
PP MMD Cal Housing Latent Linear 0.0
PP MMD Cal Housing Latent MLP 0.0
PP MMD Credit Default ClapROAR Deep Ensemble 0.0
PP MMD Credit Default ClapROAR Linear 0.0
PP MMD Credit Default ClapROAR MLP 0.0
PP MMD Credit Default Generic (γ=0.5) Deep Ensemble 0.0
PP MMD Credit Default Generic (γ=0.5) Linear 0.0
PP MMD Credit Default Generic (γ=0.5) MLP 0.0
PP MMD Credit Default Generic (γ=0.9) Deep Ensemble 0.0
PP MMD Credit Default Generic (γ=0.9) Linear 0.0
PP MMD Credit Default Generic (γ=0.9) MLP 0.0
PP MMD Credit Default Gravitational Deep Ensemble 0.0
PP MMD Credit Default Gravitational Linear 0.0
PP MMD Credit Default Gravitational MLP 0.0
PP MMD Credit Default Latent Deep Ensemble 0.0
PP MMD Credit Default Latent Linear 0.0
PP MMD Credit Default Latent MLP 0.0
PP MMD GMSC ClapROAR Deep Ensemble 0.0
PP MMD GMSC ClapROAR Linear 0.0
PP MMD GMSC ClapROAR MLP 0.0
PP MMD GMSC Generic (γ=0.5) Deep Ensemble 0.0
PP MMD GMSC Generic (γ=0.5) Linear 0.0
PP MMD GMSC Generic (γ=0.5) MLP 0.0
PP MMD GMSC Generic (γ=0.9) Deep Ensemble 0.0
PP MMD GMSC Generic (γ=0.9) Linear 0.0
PP MMD GMSC Generic (γ=0.9) MLP 0.0
PP MMD GMSC Gravitational Deep Ensemble 0.0
PP MMD GMSC Gravitational Linear 0.0
PP MMD GMSC Gravitational MLP 0.0
PP MMD GMSC Latent Deep Ensemble 0.0
PP MMD GMSC Latent Linear 0.0
PP MMD GMSC Latent MLP 0.0
PP MMD (grid) Cal Housing ClapROAR Deep Ensemble 0.044
PP MMD (grid) Cal Housing ClapROAR Linear 0.004
PP MMD (grid) Cal Housing ClapROAR MLP 0.012
PP MMD (grid) Cal Housing Generic (γ=0.5) Deep Ensemble 0.0
PP MMD (grid) Cal Housing Generic (γ=0.5) Linear 0.0
PP MMD (grid) Cal Housing Generic (γ=0.5) MLP 0.0
PP MMD (grid) Cal Housing Generic (γ=0.9) Deep Ensemble 0.002
PP MMD (grid) Cal Housing Generic (γ=0.9) Linear 0.0
PP MMD (grid) Cal Housing Generic (γ=0.9) MLP 0.0
PP MMD (grid) Cal Housing Gravitational Deep Ensemble 0.0
PP MMD (grid) Cal Housing Gravitational Linear 0.014
PP MMD (grid) Cal Housing Gravitational MLP 0.0625
PP MMD (grid) Cal Housing Latent Deep Ensemble 0.0
PP MMD (grid) Cal Housing Latent Linear 0.002
PP MMD (grid) Cal Housing Latent MLP 0.0
PP MMD (grid) Credit Default ClapROAR Deep Ensemble 0.0
PP MMD (grid) Credit Default ClapROAR Linear 0.0
PP MMD (grid) Credit Default ClapROAR MLP 0.0
PP MMD (grid) Credit Default Generic (γ=0.5) Deep Ensemble 0.0
PP MMD (grid) Credit Default Generic (γ=0.5) Linear 0.0
PP MMD (grid) Credit Default Generic (γ=0.5) MLP 0.0
PP MMD (grid) Credit Default Generic (γ=0.9) Deep Ensemble 0.0
PP MMD (grid) Credit Default Generic (γ=0.9) Linear 0.0
PP MMD (grid) Credit Default Generic (γ=0.9) MLP 0.0
PP MMD (grid) Credit Default Gravitational Deep Ensemble 0.0
PP MMD (grid) Credit Default Gravitational Linear 0.0
PP MMD (grid) Credit Default Gravitational MLP 0.0
PP MMD (grid) Credit Default Latent Deep Ensemble 0.0
PP MMD (grid) Credit Default Latent Linear 0.078
PP MMD (grid) Credit Default Latent MLP 0.0
PP MMD (grid) GMSC ClapROAR Deep Ensemble 0.0
PP MMD (grid) GMSC ClapROAR Linear 0.0
PP MMD (grid) GMSC ClapROAR MLP 0.0
PP MMD (grid) GMSC Generic (γ=0.5) Deep Ensemble 0.0
PP MMD (grid) GMSC Generic (γ=0.5) Linear 0.0
PP MMD (grid) GMSC Generic (γ=0.5) MLP 0.0
PP MMD (grid) GMSC Generic (γ=0.9) Deep Ensemble 0.0
PP MMD (grid) GMSC Generic (γ=0.9) Linear 0.0
PP MMD (grid) GMSC Generic (γ=0.9) MLP 0.0
PP MMD (grid) GMSC Gravitational Deep Ensemble 0.0
PP MMD (grid) GMSC Gravitational Linear 0.0
PP MMD (grid) GMSC Gravitational MLP 0.0
PP MMD (grid) GMSC Latent Deep Ensemble 0.0
PP MMD (grid) GMSC Latent Linear 0.0
PP MMD (grid) GMSC Latent MLP 0.0

5.1.2.5 Chart in paper (Figure 6)

Figure 5.15 shows the chart that went into the paper.

Code 
Images.load(joinpath(www_artifact_path,"paper_synthetic_results.png"))
Figure 5.15: Chart in paper

5.1.2.6 Chart in paper (Figure 8)

Code 
results = EMAR.load_real_world_mitigation()

Figure 5.16 shows the chart that went into the paper.

Code 
Images.load(joinpath(www_artifact_path,"paper_real_world_results.png"))
Figure 5.16: Chart in paper