Before we start:

— Largely based on the UAI 2024 Tutorial with the same name.

— The 2024 tutorial was co-created with Prof. Alexander Terenin (Cornell).

Resources: https://vab.im/probai2025/

Part I: background, motivation, technical fundamentals

1. Model-based decision-making
2. Probabilistic modeling and uncertainty
3. Digression: Gaussian processes, the "gold standard"
4. Geometric learning: working with structured data

Part II: geometric probabilistic models

1. Example problem: Setup
2. Basic types of probabilistic models
3. Example problem: Evaluation
4. Emerging applications
5. Software
6. Summary

References

Neural network baseline

Deep ensembles:

• Train $K$ neural networks $f_1, \ldots, f_K$ with different initializations.
• Prediction: average $f(x) = \frac{1}{K} \sum_{k=1}^K f_k(x)$.
• Uncertainty: std deviation $\sigma(x) = (\frac{1}{K} \sum_{k=1}^K (f_k(x) - f(x))^2)^{1/2}$.

Open Notebook 1 [Google Colab]
at https://vab.im/probai2025/.

Plot the uncertainty of the ensemble.

Neural network ensemble

Bayesian neural network using MCMC

Gaussian process: RBF kernel

Gaussian process: polynomial kernel

Gaussian process: RBF kernel

Gaussian process: polynomial kernel

Ensemble

Bayesian neural network

Vague idea: regulate uncertainty using symmetries

"Gold standard" for learning with uncertainty in low dimension.

Formally, GPs — random functions with jointly Gaussian marginals.

Bayesian learning: prior GP + data = posterior GP

Prior GPs are determined by kernels (covariance functions)
— have to be positive semi-definite,
— not all kernels define "good" GPs.

RBF kernel (Gaussian, Heat, Diffusion), part of a wider Matérn kernel family: $$ k_{\kappa, \sigma^2}(x,x') = \sigma^2 \exp\del{-\frac{\norm{x-x'}^2}{2\kappa^2}} $$

Why is it standard?

Intuitively, the simplest possible kernel which is:

• stationary: $k(x + s, x' + s) = k(x, x')$;
• smooth: $k$ is infinitely differentiable.

Regression and classification: $$ \htmlClass{fragment}{ \min_{f\in\c{F}} \sum_{i=1}^N L(y_i, f(x_i)) } $$ where

• $(x_i,y_i)$: training data
• $L$: loss function, such as square loss $\norm{y_i - f(x_i)}^2$
• $\c{F}$: some set of functions $f : X \to Y$

Spaces $X$ and/or $Y$ carry geometric structure

Probabilistic regression and classification: $$ \htmlClass{fragment}{ y_i \~ p_{y\given f}(\.\given f(x_i)) } $$ where

• $(x_i,y_i)$: training data
• $p_{y\given f}$: likelihood, such as Gaussian $y_i \~\f{N}(f(x_i), \sigma^2)$
• Bayesian approach: place a prior $p_f$ over functions $f : X \to Y$

Spaces $X$ and/or $Y$ carry geometric structure

Graphs

Spaces of Graphs

Meshes

Manifolds

And many more!

Consider a dataset of images of circles of different locations and sizes:

Two ways of representing points on a sphere:

• Points in $\R^3$: $(x,y,z) \in \R^3$ with $x^2 + y^2 + z^2 = 1$
• Angles: $(\theta,\phi)$ representing latitude and longitude

Different numbers: same point on sphere

Predictions should depend on our data, not on how we store it numerically

Graphs can be represented as adjacency matrices in many ways.

Good models should be invariant to this ambiguity.

Original dataset

Shifted dataset

We want training to be equivariant to shifts:
Training on a shifted dataset should result in a shifted model.

True for stationary GPs, usually not true for neural networks.

For graphs, we want training to be equivariant to permutations that preserve the graph structure.

Design principle: models which respect the space's symmetries

• Symmetry-based extrapolation
• Symmetry-based uncertainty

..unless data indicates otherwise?

Just as relevant in classical Euclidean setting

Geometric learning: right technical language for studying this

Graph neural network: predicted antibiotic activity in halicin, prev. studied for diabetes

Shown in mice to have broad-spectrum antibiotic activity

Figure and results: Stokes et al. (Cell, 2020)

Graph neural networks: used to improve complementary product recommendation

Improved performance in production compared to Amazon's prior approach

Figure and results: Hao et al. (CIKM 2020)

Graph neural networks: Google Maps traffic ETA prediction

Improved accuracy in production compared to prior approach

Figure and results: Derrow-Pinion et al. (CIKM 2021)

Part I: background, motivation, technical fundamentals

1. Model-based decision-making
2. Probabilistic modeling and uncertainty
3. Digression: Gaussian processes, the "gold standard"
4. Geometric learning: working with structured data

Part II: geometric probabilistic models

1. Example problem: Setup
2. Basic types of probabilistic models
3. Example problem: Evaluation
4. Emerging applications
5. Software
6. Summary

References

Dataset details: Borovitskiy et al. (AISTATS 2021)

Base setting: $f: X \to Y$, where both $X$ and $Y$ are spaces of graphs.

• Let $\m{A}$ be a $d \x d$ weighted adjacency matrix.
• Let $\m{X}$ be the $d \x m$ matrix of node features:
  — $\v{x}_{i} = (\m{X}_{i 1}, \ldots, \m{X}_{i m})$ is the feature vector of node $i$.

A graph convolutional layer computes a new $d \x m'$ matrix of node features:

where $\m{W}$ are trainable weights, $d_i$ are degrees and $\c{N}(i)$ are neighbor sets.

Set $\v{x}_i$ to be, e.g., either of

• constant: $\m{x}_i = (1, \ldots, 1)$;
• index $i$ zero-one encoded: $\m{x}_i = (0, \ldots, 1, \ldots, 0)$;
• masked targets: $\m{x}_i = y_i$ if known, $\m{x}_i = 0$ otherwise.

Resulting model $f_{\theta}$ maps $\m{X}$ to $\v{y}$.

Take some loss $L(\cdot, \cdot)$ and train by solving

where $d$ is the shortest path distance on a given graph $G$.

Graph nodes: not well-defined unless $G$ can be isometrically embedded into a Hilbert space.

Schoenberg (Trans. Am. Math. Soc. 1938).

Not well-defined on the square graph.

Another characterization of $k_{\kappa, \sigma^2}$ in terms of the Laplacian $\Delta$:

$k_{\kappa, \sigma^2} \propto \sigma^2 \exp(-\frac{\kappa^2}{2} \Delta)$ .

On graphs: use graph Laplacian. $k_{\kappa, \sigma^2}$ is known as heat/diffusion kernel.

Claim. $k_{\kappa, \sigma^2}$ is positive definite, "stationary" and "smooth".

$k_{\kappa, \sigma^2}$ can be made explicit:

where $\lambda_n, f_n$ are eigenvalues and eigenvectors of the graph Laplacian $\Delta$.

Deep ensemble

Deep ensembles:

• Train $K$ neural networks $f_1, \ldots, f_K$ with different initializations.
• Prediction: average $f(x) = \frac{1}{K} \sum_{k=1}^K f_k(x)$.
• Uncertainty: std deviation $\sigma(x) = (\frac{1}{K} \sum_{k=1}^K (f_k(x) - f(x))^2)^{1/2}$.

Open Notebook 2 [Google Colab]
at https://vab.im/probai2025/.

Compute RMSE and NLL of the ensemble.

Bayesian graph CNN

Gaussian process

This example: easy-to-use and visualize benchmark for probabilistc regression on geometric data

Geometric Gaussian processes:

• Reliable off-the-shelf probabilistic models
• Solid baselines to benchmark other models against

https://geometric-kernels.github.io/

Joint postures: $\mathbb{T}^d$

Rotations: $\mathbb{S}_3$, $\operatorname{SO}(3)$

Manipulability: $\operatorname{SPD(3)}$

Jaquier et al. (CoRL 2019, CoRL 2021), figures by the authors

Prediction

Uncertainty

Matérn Gaussian processes on meshes (independently proposed)

Coveney et al. (TBE 2019, PTRSA 2020), figures by the authors

Based on ensembles of geodesic CNNs

Neural Concept, video from neuralconcept.com

Neural networks beyond the graph setting: various small libraries

Geometric machine learning: increasingly popular

• Waves: kernels on structured data, geometric deep learning

Geometric probabilistic models: emerging research area

• Geometric probabilistic > geometric + probabilistic
• Geometric Gaussian processes: solid off-the-shelf baselines