Endogenous Macrodynamics in Algorithmic Recourse

IEEE Conference on Secure and Trustworthy Machine Learning

Patrick Altmeyer

Delft University of Technology

Giovan Angela

Aleksander Buszydlik

Karol Dobiczek

Arie van Deursen

Cynthia C. S. Liem

May 20, 2025

Quick Intro

Currently 2nd year of PhD in Trustworthy Artificial Intelligence at Delft University of Technology.
Working on Counterfactual Explanations and Probabilistic Machine Learning with applications in Finance.
Previously, educational background in Economics and Finance and two years at the Bank of England.
Enthusiastic about free open-source software, in particular Julia and Quarto.

Motivation and Contributions

In a nutshell …

[…] we run experiments that simulate the application of recourse in practice using various state-of-the-art counterfactual generators and find that all of them induce substantial domain and model shifts.

Counterfactual Explanation (CE) explain how inputs into a model need to change for it to produce different outputs.
Counterfactual Explanations that involve realistic and actionable changes can be used for the purpose of Algorithmic Recourse (AR) to help individuals who face adverse outcomes.

🎯 Key Contributions

We find that the induced shifts are substantial enough to likely impede the applicability of Algorithmic Recourse in some situations.
Fortunately, we find various strategies to mitigate these concerns.
Our simulation framework for studying recourse dynamics is fast and open-sourced.

Proof-of-Concept

Example 1 (Consumer Credit)

Suppose Figure 1 relates to an automated decision-making system used by a retail bank to evaluate credit applicants.
Creditworthiness decreases in the South-East direction.
Outcome: bank supplies credit to more borrowers (orange), but these borrowers are riskier on average, which represents a cost to the retail bank.

Example 2 (Student Admission)

Suppose Figure 1 relates to an automated decision-making system used by a university in its student admission process.
Likelihood of students completing their degree decreases in the South-East direction.
Outcome: more students are admitted to university (orange), but they are more likely to fail their degree; the university suspends its efforts to offer AR, which represents a cost to future applicants.

Gradient-Based Recourse Revisited

Algorithmic Recourse

Even though […] interpretability is of great importance and should be pursued, explanations can, in principle, be offered without opening the “black box”. (Wachter, Mittelstadt, and Russell 2017)

Framework

. . .

Objective originally proposed by Wachter, Mittelstadt, and Russell (2017) is as follows

\[ \min_{x^\prime \in \mathcal{X}} \text{cost}(x^\prime) \ \ \ \mbox{s. t.} \ \ \ M(x^\prime) = y^* \qquad(1)\]

Typically this is approximated through regularization:

\[ x^\prime = \arg \min_{x^\prime} \text{yloss}(M(x^\prime),y^*) + \lambda \text{cost}(x^\prime) \qquad(2)\]

Intuition

. . .

Figure 2: Counterfactuals for Give Me Some Credit dataset (Kaggle 2011).

From individual recourse …

We include the following generators in our simulation experiments below: REVISE (Joshi et al. 2019), CLUE (Antorán et al. 2020), DiCE (Mothilal, Sharma, and Tan 2020) and a Greedy approach that relies on probabilistic models (Schut et al. 2021).

All of them can be described by the following generalized form of Equation 2:

\[ \begin{aligned} \mathbf{s}^\prime &= \arg \min_{\mathbf{s}^\prime \in \mathcal{S}} \left\{ {\text{yloss}(M(f(\mathbf{s}^\prime)),y^*)}+ \lambda {\text{cost}(f(\mathbf{s}^\prime)) } \right\} \end{aligned} \qquad(3)\]

Here \(\mathbf{s}^\prime=\left\{s_k^\prime\right\}_K\) is a \(K\)-dimensional array of counterfactual states and \(f: \mathcal{S} \mapsto \mathcal{X}\) maps from the counterfactual state space to the feature space.

Figure 3: Feature space (left) and counterfactual state space (right).

… towards collective recourse

All of the different approaches introduced above tackle the problem of Algorithmic Recourse from the perspective of one single individual.

We propose to extend Equation 3 as follows:

\[ \begin{aligned} \mathbf{s}^\prime &= \arg \min_{\mathbf{s}^\prime \in \mathcal{S}} \{ {\text{yloss}(M(f(\mathbf{s}^\prime)),y^*)} \\ &+ \lambda_1 {\text{cost}(f(\mathbf{s}^\prime))} + \lambda_2 {\text{extcost}(f(\mathbf{s}^\prime))} \} \end{aligned} \qquad(4)\]

Here \(\text{cost}(f(\mathbf{s}^\prime))\) denotes the proxy for private costs faced by the individual; the newly introduced term \(\text{extcost}(f(\mathbf{s}^\prime))\) is meant to capture external costs generated by changes to \(\mathbf{s}^\prime\).

🎓 Negative Externalities

The underlying concept of private and external costs is borrowed from Economics and well-established in that field: when the decisions or actions by some individual market participant generate external costs, then the market is said to suffer from negative externalities and is considered inefficient (Pindyck and Rubinfeld 2014).

Modeling Endogenous Macrodynamics in Algorithmic Recourse

Research Questions

Principal Concerns

RQ 1 (Endogenous Shifts) Does the repeated implementation of recourse provided by state-of-the-art generators lead to shifts in the domain and model?

RQ 2 (Costs) If so, are these dynamics substantial enough to be considered costly to stakeholders involved in real-world automated decision-making processes?

RQ 3 (Heterogeneity) Do different counterfactual generators yield significantly different outcomes in this context? Furthermore, is there any heterogeneity concerning the chosen classifier and dataset?

RQ 4 (Drivers) What are the drivers of endogenous dynamics in Algorithmic Recourse?

Secondary Concerns

RQ 5 (Mitigation Strategies) What are potential mitigation strategies with respect to endogenous macrodynamics in AR?

Domain Shifts

Maximum Mean Discrepancy (MMD): a measure of the distance between the kernel mean embeddings of two samples; in our context, large values indicate that a domain shift indeed seems to have occurred.

Model Shifts

Perturbations: following Upadhyay, Joshi, and Lakkaraju (2021) we define \(\Delta=||\theta_{t+1}-\theta_{t}||^2\), that is the euclidean distance between the vectors of parameters before and after retraining the model \(M\).
Predicted Probability MMD (PP MMD): instead of applying MMD to features directly, we apply it to the predicted probabilities assigned to a set of samples by the model \(M\).
Disagreement Coefficient: this metric was introduced in Hanneke (2007) and estimates \(p(M(x) \neq M^\prime(x))\), that is the probability that two classifiers disagree on the predicted outcome for a randomly chosen sample.
Decisiveness: we define the metric simply as \({\frac{1}{N}}\sum_{i=0}^N(\sigma(M(x)) - 0.5)^2\) where \(M(x)\) are predicted logits from a binary classifier and \(\sigma\) denotes the sigmoid function; it quantifies the likelihood that a model assigns a high probability to its classification of any given sample.
Performance: we compute the classifier’s F-score on a test sample that we leave untouched throughout the experiment.

Classifiers

Simple linear classifier—Logistic Regression.
Multilayer perceptron—MLP.
Deep Ensemble composed of five MLPs following Lakshminarayanan, Pritzel, and Blundell (2017).

Generative Models

Different specifications of a plain-vanilla Variational Autoencoder (VAE)

Table 1: Model parameters.

Neural network architectures and training parameters.
Model	Data	Hidden Dim.	Latent Dim.	Hidden Layers	Batch	Dropout	Epochs
MLP	Synthetic	32	-	1	-	-	100
MLP	Real-World	64	-	2	500	0.1	100
VAE	Synthetic	32	2	1	-	-	100
VAE	Real-World	32	8	1	-	-	250

Synthetic Data

Four synthetic binary classification datasets consisting of 1000 samples each: Overlapping, Linearly Separable, Circles and Moons (Figure 4).

Figure 4: Synthetic classification datasets used in our experiments. Samples from the negative class (\(y=0\)) are marked in orange while samples of the positive class (\(y=1\)) are marked in blue.

Real-World Data

Three real-world datasets from the Finance and Economics domain: all tabular and can be used for binary classification.

The Give Me Some Credit dataset: predict whether a borrower is likely to experience financial difficulties in the next two years (Kaggle 2011).
The UCI defaultCredit dataset (Yeh and Lien 2009): a benchmark dataset that can be used to train binary classifiers to predict the whether credit card clients default on their payment.
The California Housing dataset Pace and Barry (1997): continuous outcome variable binarized as \(\tilde{y}=\mathbb{I}_{y>\text{median}(Y)}\) indicating if the median house price of a given district is above the median of all districts.

Principal Concerns

Wachter (Wachter, Mittelstadt, and Russell 2017)
REVISE (Joshi et al. 2019)
CLUE (Antorán et al. 2020)
DiCE (Mothilal, Sharma, and Tan 2020)
Greedy (Schut et al. 2021)

Secondary Concerns

More Conservative Decision Thresholds
Classifier Preserving ROAR (ClaPROAR):

\[ \begin{aligned} \text{extcost}(f(\mathbf{s}^\prime)) = l(M(f(\mathbf{s}^\prime)),y^\prime) \end{aligned} \qquad(5)\]

Gravitational Counterfactual Explanations:

\[ \begin{aligned} \text{extcost}(f(\mathbf{s}^\prime)) = \text{dist}(f(\mathbf{s}^\prime),\bar{x}^*) \end{aligned} \qquad(6)\]

Principal Findings

Secondary Findings

Summary of Findings

Principal Concerns

Firstly, endogenous dynamics do emerge in our experiments (Proposition 1) and we find them substantial enough to be considered costly (Proposition 2)
Secondly, the choice of the counterfactual generator matters, with Latent Space search generally having a dampening effect (Proposition 3).
The observed dynamics, therefore, seem to be driven by a discrepancy between counterfactual outcomes that minimize costs to individuals and outcomes that comply with the data-generating process (Proposition 4).

Secondary Concerns

Our findings indicate that all three proposed mitigation strategies are at least at par with LS generators (Proposition 5).

Discussion

Key Takeaways 🔑

Our findings indicate that state-of-the-art approaches to Algorithmic Recourse induce substantial domain and model shifts.

We would argue that the expected external costs of individual recourse should be shared by all stakeholders.

A straightforward way to achieve this is to penalize external costs in the counterfactual search objective function (Equation 4).

Various simple strategies based on this notion can be effectively used to mitigate shifts.

Limitations

Private vs. External Costs

We fall short of providing any definitive answers as to how to trade off private vs. external costs.
Proposed strategies are a good starting point, but they are ad-hoc.

. . .

Experimental Setup

Experimental design is a vast over-simplification of potential real-world scenarios.

. . .

Causal Modelling

Have focused on popular counterfactual generators that do not incorporate any causal knowledge.
Perturbations therefore may involve changes to variables that affect the outcome predicted by the black-box model, but not the true, causal outcome.
Future work would likely benefit from including recent approaches to AR that incorporate causal knowledge such Karimi, Schölkopf, and Valera (2021).

. . .

Classifiers

We have limited our analysis to differentiable linear and non-linear classifiers; empirical evidence suggests that other models such as boosted decision trees outperform DL on tabular data (Borisov et al. (2022), Grinsztajn, Oyallon, and Varoquaux (2022)).

Q & A ❓

Final Things 🏁

More Resources 📚

Read on …

Granular results for all of our experiments can be found in this online companion: https://www.patalt.org/endogenous-macrodynamics-in-algorithmic-recourse/.
Blog post introducing Counterfactual Explanations: [TDS, homepage].

… or get busy 🖥️

CounterfactualExplanations.jl (Altmeyer 2022) provides an extensible, fast and language-agnostic implementation in Julia.
We have built a framework that extends the functionality from static benchmarks to simulation experiments: AlgorithmicRecourseDynamics.jl.
The Github repository containing all the code used to produce the results in this paper can be found here.

Image Sources

Copyright for stock images belongs to TU Delft.
All other images, graphics or animations were created by us.

References

Altmeyer, Patrick. 2022. “CounterfactualExplanations.jl - a Julia Package for Counterfactual Explanations and Algorithmic Recourse.” https://github.com/pat-alt/CounterfactualExplanations.jl.

Antorán, Javier, Umang Bhatt, Tameem Adel, Adrian Weller, and José Miguel Hernández-Lobato. 2020. “Getting a Clue: A Method for Explaining Uncertainty Estimates.” https://arxiv.org/abs/2006.06848.

Borisov, Vadim, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. 2022. “Deep Neural Networks and Tabular Data: A Survey.” IEEE Transactions on Neural Networks and Learning Systems.

Grinsztajn, Léo, Edouard Oyallon, and Gaël Varoquaux. 2022. “Why Do Tree-Based Models Still Outperform Deep Learning on Tabular Data?” https://arxiv.org/abs/2207.08815.

Hanneke, Steve. 2007. “A Bound on the Label Complexity of Agnostic Active Learning.” In Proceedings of the 24th International Conference on Machine Learning, 353–60. https://doi.org/10.1145/1273496.1273541.

Joshi, Shalmali, Oluwasanmi Koyejo, Warut Vijitbenjaronk, Been Kim, and Joydeep Ghosh. 2019. “Towards Realistic Individual Recourse and Actionable Explanations in Black-Box Decision Making Systems.” https://arxiv.org/abs/1907.09615.

Kaggle. 2011. “Give Me Some Credit, Improve on the State of the Art in Credit Scoring by Predicting the Probability That Somebody Will Experience Financial Distress in the Next Two Years.” https://www.kaggle.com/c/GiveMeSomeCredit; Kaggle. https://www.kaggle.com/c/GiveMeSomeCredit.

Karimi, Amir-Hossein, Bernhard Schölkopf, and Isabel Valera. 2021. “Algorithmic Recourse: From Counterfactual Explanations to Interventions.” In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 353–62. FAccT ’21. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3442188.3445899.

Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. 2017. “Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles.” In Proceedings of the 31st International Conference on Neural Information Processing Systems, 6405–16. NIPS’17. Red Hook, NY, USA: Curran Associates Inc.

Mothilal, Ramaravind K, Amit Sharma, and Chenhao Tan. 2020. “Explaining Machine Learning Classifiers Through Diverse Counterfactual Explanations.” In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 607–17. https://doi.org/10.1145/3351095.3372850.

Pace, R Kelley, and Ronald Barry. 1997. “Sparse Spatial Autoregressions.” Statistics & Probability Letters 33 (3): 291–97. https://doi.org/10.1016/s0167-7152(96)00140-x.

Pedregosa, Fabian, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, et al. 2011. “Scikit-Learn: Machine Learning in Python.” The Journal of Machine Learning Research 12: 2825–30.

Pindyck, Robert S, and Daniel L Rubinfeld. 2014. Microeconomics. Pearson Education.

Schut, Lisa, Oscar Key, Rory McGrath, Luca Costabello, Bogdan Sacaleanu, Yarin Gal, et al. 2021. “Generating Interpretable Counterfactual Explanations By Implicit Minimisation of Epistemic and Aleatoric Uncertainties.” In International Conference on Artificial Intelligence and Statistics, 1756–64. PMLR.

Upadhyay, Sohini, Shalmali Joshi, and Himabindu Lakkaraju. 2021. “Towards Robust and Reliable Algorithmic Recourse.” Advances in Neural Information Processing Systems 34: 16926–37.

Wachter, Sandra, Brent Mittelstadt, and Chris Russell. 2017. “Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR.” Harv. JL & Tech. 31: 841. https://doi.org/10.2139/ssrn.3063289.

Yeh, I-Cheng, and Che-hui Lien. 2009. “The Comparisons of Data Mining Techniques for the Predictive Accuracy of Probability of Default of Credit Card Clients.” Expert Systems with Applications 36 (2): 2473–80. https://doi.org/10.1016/j.eswa.2007.12.020.