# COMP760: Geometry and Generative Models

# Co-Instructors: Joey Bose and Prakash Panangaden

# Course Overview

In recent years Deep Generative Models have seen remarkable success over a variety of data domains such as images, text, and audio to name a few. However, the predominant approach in many of these models (e.g. GANS, VAE, Normalizing Flows) is to treat data as fixed-dimensional continuous vectors in some Euclidean space, despite significant evidence to the contrary (e.g. 3D molecules). This course places a direct emphasis on learning generative models for complex geometries described via manifolds, such as spheres, tori, hyperbolic spaces, implicit surfaces, and homogeneous spaces. The purpose of this seminar course is to understand the key design principles that underpin the new wave of geometry-aware generative models that treat the rich geometric structure in data as a first-class citizen. This seminar course will also serve to develop extensions to these approaches at the leading edge of research and as a result, a major component of the course will focus on class participation through presenting papers and a thematically-relevant course project.

- Location: Mila Auditorium 1
- Time: Fridays 1-4pm
- Office Hours: TBD
- Capacity: 25-30
**(Registration will need instructor approval. Please email us)**

# Prerequisites

This course is designed to bring students to the current frontier of knowledge on geometric generative models so that ideally, their course projects can make a novel contribution that can either be algorithmic, theoretical, or empirical. A previous background in machine learning is strongly recommended. Linear algebra, basic multivariate calculus, basics of working with probability, and programming skills are required. No background in geometry or generative models is needed for the course but any such knowledge may aid in a deeper engagement with the course material. However, it is strongly recommended that this course is not the first—or even second—Machine Learning course, and if there are any specific doubts please contact the instructor for special permission or equivalency of prerequisites.

- Math 223 (Linear Algebra 2) or equivalent
- COMP 551 (Applied Machine Learning) or equivalent
- Math 323 (Probability) or equivalent
- Math 222 (Calculus 3) or equivalent

# Tentative Course Structure

The first third of the course will consist of lectures covering necessary background material in geometric deep learning and generative models, after which every week will focus on student presentations on a pre-selected topic, using a couple of papers as reference. Each student must prepare one presentation individually or as a pair for the semester which is independent of their course project. The final two weeks will be reserved for course project presentations. In-class discussion will center around:

- Understanding the strengths and weaknesses of these methods.
- Understanding the relationships between these methods, and with previous approaches.
- Extensions or applications of these methods.
- Experiments that might better illuminate their properties.

# Grading

- 20% In-class paper presentations
- 30% Student project presentation
- 50% Student project report

# Reading List:

## Textbooks

There is no required textbook for this course but the “Geometric Deep Learning proto book” is a good reference for much of the technical foundations covered in this course.

**Reading List is subject to change**

## Prerequisite readings:

- Automatic Differentiation and Variational Inference
- Normalizing Flows for Probabilistic Modeling and Inference

## Week 1: Geometry Primer Part I (Prakash Lectures):

- Lecture Notes
- Topology: open and closed sets, continuous functions, convergence, metrics, compactness.
- Manifolds: smoothness, charts, tangent vectors, tensors, tangent bundle.

## Week 2: Geometry Primer Part II (Prakash Lectures):

- Riemannian metrics
- Affine connections
- Lie groups
- Symmetric spaces.

## Week 3: Deep Generative Models Primer Part I (Joey Lectures)

Deep generative models learn to transform unstructured noise to highly structured data like natural images. While these models may come in various forms but they can be broadly classified as either likelihood-based or implicit models. The former model class is already quite rich with popular modeling families such as VAE’s, and Normalizing Flows and will be the starting point for this week’s topics.

- Auto-Encoding Variational Bayes
- Variational Inference with Normalizing Flows
- Density Estimation with RealNVP
- Neural Ordinary Differential Equations

## Week 4: Deep Generative Models Primer Part II (Joey Lectures)

In recent years there has been a resurgence of old ideas but repackaged for a modern time. Specifically, score matching and diffusion models have existed in some form or other prior to their recent renaissance but the main technical novelties lie—beyond new techniques that allow for fast training, inference, and impressive sampling quality—in their connection to Stochastic Differential Equations, VAE’s and Continuous Normalizing Flows. This week will focus on exploring these connections in detail and as we will see many new ideas are old ideas in disguise.

- Generative Adversarial Networks
- Generative Modeling by Estimating Gradients of the Data Distribution
- Denoising diffusion probabilistic models
- Score-Based Generative Modeling through Stochastic Differential Equations
- Score Matching Blog Post

## Week 5: Distributions on Manifolds

Defining probability distributions on manifolds requires specific care as many familiar notions in Euclidean geometry become incompatible for general manifolds. For example, the classical Gaussian distribution has at least three different instantiations in the manifold setting, e.g. Restricted, Wrapped, and the Riemannian normal distribution. We will focus on covering mainly the Riemannian normal distribution as well as the Von Mises Fisher distribution which is typically used in Spherical geometry.

- Reparameterizing Distributions on Lie Groups
- Directional data analysis under the general projected normal distribution
- Dispersion on a Sphere
- Intrinsic Statistics on Riemannian Manifolds: Basic Tools for Geometric Measurements
- Riemannian Gaussian Distributions on the Space of Symmetric Positive Definite Matrices

## Week 6: Spherical Geometry

The first wave of geometry-aware deep generative models focused largely on spherical geometry—i.e. Riemannian manifolds with positive curvature—as it is perhaps most accessible after Euclidean geometry. As a result, there are a few prominent work that exploit this structure to define hyperspherical latent spaces, or more complex distributions on the Sphere using Normalizing Flows or even GANs.

- Hyperspherical Variational Auto-Encoders
- Normalizing Flows on Tori and Spheres
- The Power Spherical distribution
- Sphere Generative Adversarial Network Based on Geometric Moment Matching

## Week 7: Hyperbolic Geometry

Hyperbolic spaces—i.e. manifolds with constant negative curvature—have become an increasingly useful geometry in the modern machine learning toolkit. From modelling social networks, trees, biological networks, to hierarchical diffusion processes hyperbolic spaces have found tremendous practical advantages over Euclidean counterparts. Naturally, extending generative models to hyperbolic space requires taking into account manifold specific operations like the exponential and logarithmic maps, parallel transport all of which are key design decisions when constructing neural architectures that operate on these spaces.

- Continuous Hierarchical Representations with Poincaré Variational Auto-Encoders
- A Wrapped Normal Distribution on Hyperbolic Space for Gradient-Based Learning
- Latent Variable Modelling with Hyperbolic Normalizing Flows

## Week 8: Product Manifolds

Disentanglement is perhaps one of the key goals of unsupervised learning. For generative models, this effectively means isolating the main generative factors that give rise to observed data. Modeling such generative factors as latent variables in VAE like the setup is the dominant paradigm to infuse geometric priors about the data into practical modeling inductive biases. This week we will turn our attention to the various types of geometric structure that one can attach to a latent space such as group structure and product manifolds.

- Towards a definition of disentangled representations
- Learning mixed-curvature representations in product spaces
- Mixed-curvature Variational Autoencoders

## Week 9: Normalizing Flows on Riemannian Manifolds

How can we define flexible probability distributions on general Riemannian manifolds? Turns out one natural way to do so is to define an easy-to-sample prior distribution and a time-evolving vector field that transports this density to the desired target. This effectively generalizes the continuous normalizing flow approach previously seen in week 3 to Riemannian manifolds and this week we will cover 3 papers that were published concurrently on this very topic. Extending the CNF’s to manifolds typically requires backpropping through an ODE solver which is computationally expensive. In this week we will cover a series of normalizing flows that sidestep this expensive computation by using various methodological innovations such as using convex potentials from Riemannian optimal transport to neural implementation of Moser’s trick which led to MoserFlow (NeurIPS 2021 outstanding paper).

- Riemannian Continuous Normalizing Flows
- Neural Manifold Ordinary Differential Equations
- Neural Ordinary Differential Equations on Manifolds
- Moser Flow: Divergence-based Generative Modeling on Manifolds
- Matching Normalizing Flows and Probability Paths on Manifolds
- The Riemannian Geometry of Deep Generative Models
- Riemannian Convex Potential Maps

## Week 10: Equivariant Generative Models

Much of observed data is a result of physical processes which have symmetries. These symmetries manifest themselves as equivariances and invariances to certain transformation groups, e.g. translation, rotation, scaling, etc …, and imbuing generative models with these structural inductive biases is a core design principle. In this week we will cover many types of equivariant generative models and their application to physics, molecular dynamics, and many more practical domains.

- Equivariant Flows: exact likelihood generative learning for symmetric densities
- Equivariant Manifold Flows
- E(n) Equivariant Normalizing Flows
- Equivariant Discrete Normalizing Flows
- Implicit Riemannian Concave Potential Maps
- Sampling SU(N) with gauge equivariant flows

## Week 11: Geometric Score and Diffusion Models

Score and Diffusion models are the current state of the art generative models for both likelihood estimation as well sample quality. Given their performance for images it is a natural question on whether these models can be adapted to more complex geometries. A recent influx of papers definitively answer this question with compelling samples ranging from spherical, hyperbolic and toroidal geometry but also practical use cases in molecular simulations.

- Equivariant Diffusion for Molecule Generation in 3D
- Riemannian Score-Based Generative Modeling
- Riemannian Diffusion Models