• Motivation: Model-based decision-making
• Basic types of probabilistic models
• Probabilistic models on geometric domains
• Motivating geometric Gaussian processes: A benchmark

Geometric Gaussian processes

• Theory
• Various geometric settings
• Implementation

Summary

Probabilistic models: input ⟶ distribution (prediction + uncertainty)

Allows computing queries like probability of improvement: $\!\alpha(x) \!=\! \mathbb{P} \! \del{f(x) \!>\!\! f^*}$

They can drive selection of the next point to evaluate [Bayesian optimization]

Gaussian processes (GPs) — gold standard.
In practice: General-purpose Gaussian processes (RBF and Matérn kernels).

• Inductive bias:
  • Stationarity: training on a shifted data produces a shifted model.
  • Smoothness: continuity and differentiability.
  
  
• Scope:
• Work very well in low dimensions (e.g. tabular data).
• Struggle in high dimensions (e.g. image / text data).
  Though deep ensembles and Bayesian neural networks struggle too!

Geometric GPs: extend scope to geometric settings.

Dataset details: Borovitskiy et al. (AISTATS 2021)

Figure: Uncertainty estimates of a geometric Gaussian process.     

This benchmark: geometric GP is best, GNN ensemble fails.

Gaussian processes — random functions with jointly Gaussian marginals.

Bayesian learning: prior GP + data = posterior GP

Prior GPs are determined by kernels (covariance functions)
— have to be positive semi-definite (PSD),
— not all kernels define "good" GPs.

Matérn kernels:

$$ \htmlData{class=fragment fade-out,fragment-index=6}{ \footnotesize \mathclap{ k_{\nu, \kappa, \sigma^2}(x,x') = \sigma^2 \frac{2^{1-\nu}}{\Gamma(\nu)} \del{\sqrt{2\nu} \frac{\abs{x-x'}}{\kappa}}^\nu K_\nu \del{\sqrt{2\nu} \frac{\abs{x-x'}}{\kappa}} } } \htmlData{class=fragment d-print-none,fragment-index=6}{ \footnotesize \mathclap{ k_{\infty, \kappa, \sigma^2}(x,x') = \sigma^2 \exp\del{-\frac{\abs{x-x'}^2}{2\kappa^2}} } } $$

$\nu\to\infty$: RBF kernel (Gaussian, heat, diffusion)

$\nu = 1/2$

$\nu = 3/2$

$\nu = 5/2$

$\nu = \infty$

Graph nodes: not PSD for some $\kappa$ unless can be isometrically embedded into a Hilbert space.

Schoenberg (Trans. Am. Math. Soc. 1938).

Manifolds: not PSD for some $\kappa$ (in some cases—for all) unless the manifold is isometric to $\mathbb{R}^d$.

Feragen et al. (CVPR 2015) and Da Costa et al. (SIMODS 2025).

Not PSD on the square graph.

Not PSD on the circle manifold.

$$ \htmlData{class=fragment,fragment-index=0}{ \ubr{\del{\frac{2\nu}{\kappa^2} - \Delta}^{\frac{\nu}{2}+\frac{d}{4}} f = \c{W}}{\t{Matérn}} } $$

SPDE turns into a stochastic linear system: solution has kernel $$ \htmlData{fragment-index=2,class=fragment}{ k_{\nu, \kappa, \sigma^2}(i, j) = \frac{\sigma^2}{C_{\nu, \kappa}} \sum_{n=0}^{\abs{V}-1} \Phi_{\nu, \kappa}(\lambda_n) \v{f_n}(i)\v{f_n}(j) } $$

$\lambda_n, \v{f_n}$: eigenvalues and eigenvectors of the Laplacian matrix

$$ \htmlData{fragment-index=4,class=fragment}{ \Phi_{\nu, \kappa}(\lambda) = \begin{cases} \htmlData{fragment-index=5,class=fragment}{ \del{\frac{2\nu}{\kappa^2} - \lambda}^{-\nu} } & \htmlData{fragment-index=5,class=fragment}{ \nu\lt\infty : \t{Matérn} } \\ \htmlData{fragment-index=6,class=fragment}{ e^{-\frac{\kappa^2}{2} \lambda} } & \htmlData{fragment-index=6,class=fragment}{ \nu = \infty : \t{heat (sq. exp., RBF)} } \end{cases} } $$

Compact manifolds: $$ \htmlData{class=fragment}{ \begin{aligned} k_{\nu, \kappa, \sigma^2}(x,x') &= \frac{\sigma^2}{C_{\nu, \kappa}} \sum_{n=0}^\infty \Phi_{\nu, \kappa}(\lambda_n) f_n(x) f_n(x') \\ &= \frac{\sigma^2}{C_{\nu, \kappa}} \sum_{l=0}^\infty \Phi_{\nu, \kappa}(\lambda_l) \underbrace{\sum_{s=1}^{d_s} f_{l, s}(x) f_{l, s}(x')}_{G_l(x, x')} \end{aligned} } $$

Infinite summation where $\lambda_n, f_n$ are Laplace–Beltrami eigenpairs

Addition theorems: more efficient $G_l(x, x')$-based expressions

Non-compact manifolds:

• No summation: instead, integral approximated via Monte Carlo

Spaces of graphs:

• Geometry: make a set of graphs to be a node set of a larger graph
• Then, heat and Matérn kernels are automatically defined
• Larger graph is huge: need clever computational techniques

https://geometric-kernels.github.io/

Supported spaces:

Supported backends: PyTorch, JAX, TensorFlow, NumPy.

Geometric machine learning: increasingly popular

• Waves: kernels on structured data, geometric deep learning

Geometric probabilistic models: emerging research area

• Geometric probabilistic > geometric + probabilistic
• Geometric Gaussian processes: strong off-the-shelf baselines