Effortless Bayesian Deep Learning through Laplace Redux

JuliaCon 2022

Patrick Altmeyer

Overview

  • The Case for Bayesian Deep Learning
  • Laplace Redux in Julia 📦
    • From Bayesian Logistic Regression …
    • … to Bayesian Neural Networks.
  • Goals and Ambitions 🎯

The Case for Bayesian Deep Learning

Bayesian Model Averaging

Don’t put all your 🥚 in one 🧺.

  • In Deep Learning we typically maximise highly non-convex functions full of local optima and saddle points.
  • There may be many \(\hat\theta_1, ..., \hat\theta_m\) that are slightly different, but yield similar performance.

[…] parameters correspond to a diverse variety of compelling explanations for the data. (Wilson 2020)

\(\theta\) is a random variable. Shouldn’t we treat it that way?

\[ p(y|x,\mathcal{D}) = \int p(y|x,\theta)p(\theta|\mathcal{D})d\theta \tag{1}\]

Intractable!

In practice we typically rely on a plugin approximation (Murphy 2022).

\[ p(y|x,\mathcal{D}) = \int p(y|x,\theta)p(\theta|\mathcal{D})d\theta \approx p(y|x,\hat\theta) \tag{2}\]

Yes, “plugin” is literal … can we do better?

Enter: Bayesian Deep Learning 🔮

Yes, we can!

MCMC (see Turing)

Variational Inference (Blundell et al. 2015)

Monte Carlo Dropout (Gal and Ghahramani 2016)

Deep Ensembles (Lakshminarayanan et al. 2017)

Laplace Redux (Immer et al. (2020),Daxberger et al. (2021))

Figure 1: Pierre-Simon Laplace as chancellor of the Senate under the First French Empire. Source: Wikipedia