Bayesian Deep Learning

Background

  • Jospin et al. (2020) provide a detailed and hands-on introduction to Bayesian Deep Learning.
  • Murphy (2022) is a text book that treats machine learning from a probabilistic perspective. It includes sections dedicated to deep learning.

Interpretability

  • Ish-Horowicz et al. (2019) proposes an entropy-based measure for interpreting Bayesian Neural Networks. For a summary see here.

Uncertainty quantification and applications

  • Gal and Ghahramani (2016) demonstrate that a dropout neural network is equivalent to approximate inference in Bayesian modelling of deep Gaussian processes. This makes it straight-forward to quantify uncertainty in deep learning through simple Monte-Carlo methods.
  • Gal, Islam, and Ghahramani (2017) propose a way towards deep active Bayesian learning that plays with the ideas of aleatoric and epsitemic uncertainty: a structured approach to human-in-the-loop deep learning that can work with small data sets.
    • Kirsch, Van Amersfoort, and Gal (2019) extend these ideas.

Computational efficiency

  • Quantum computing likely to make probabilistic modelling more computationally efficient. Kehoe et al. (2021) propose a Bayesian approach to DL using quantum processors that promises to be more robust than conventional DNNs.
  • Using simple concentration inequalities Maxim Panov proposes a measure for total uncertainty of Deep Neural Networks (no numerical methods needed) – missing a paper references here.
  • Compare explainability in Bayesian setting (e.g. RATE (Ish-Horowicz et al. 2019)) to surrogate (and counterfactual) explainers? (ING models)
  • Link to AFR track on quantum ML.
  • Link to uncertainty quantification for Deep Vector Autoregression (agusti2021deep?).

References

Gal, Yarin, and Zoubin Ghahramani. 2016. “Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning.” In International Conference on Machine Learning, 1050–59. PMLR.
Gal, Yarin, Riashat Islam, and Zoubin Ghahramani. 2017. “Deep Bayesian Active Learning with Image Data.” In International Conference on Machine Learning, 1183–92. PMLR.
Ish-Horowicz, Jonathan, Dana Udwin, Seth Flaxman, Sarah Filippi, and Lorin Crawford. 2019. “Interpreting Deep Neural Networks Through Variable Importance.” arXiv Preprint arXiv:1901.09839.
Jospin, Laurent Valentin, Wray Buntine, Farid Boussaid, Hamid Laga, and Mohammed Bennamoun. 2020. “Hands-on Bayesian Neural Networks–a Tutorial for Deep Learning Users.” arXiv Preprint arXiv:2007.06823.
Kehoe, Aidan, Peter Wittek, Yanbo Xue, and Alejandro Pozas-Kerstjens. 2021. “Defence Against Adversarial Attacks Using Classical and Quantum-Enhanced Boltzmann Machines.” Machine Learning: Science and Technology.
Kirsch, Andreas, Joost Van Amersfoort, and Yarin Gal. 2019. “Batchbald: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning.” Advances in Neural Information Processing Systems 32: 7026–37.
Murphy, Kevin P. 2022. Probabilistic Machine Learning: An Introduction. MIT Press.