# COMP760: Geometry and Generative Models

# Co-Instructors: Joey Bose and Prakash Panangaden

# Course Overview

In recent years Deep Generative Models have seen remarkable success over a variety of data domains such as images, text, and audio to name a few. However, the predominant approach in many of these models (e.g. GANS, VAE, Normalizing Flows) is to treat data as fixed-dimensional continuous vectors in some Euclidean space, despite significant evidence to the contrary (e.g. 3D molecules). This course places a direct emphasis on learning generative models for complex geometries described via manifolds, such as spheres, tori, hyperbolic spaces, implicit surfaces, and homogeneous spaces. The purpose of this seminar course is to understand the key design principles that underpin the new wave of geometry-aware generative models that treat the rich geometric structure in data as a first-class citizen. This seminar course will also serve to develop extensions to these approaches at the leading edge of research and as a result, a major component of the course will focus on class participation through presenting papers and a thematically-relevant course project.

- Location: Mila Auditorium 1
- Time: Fridays 1pm - 4pm
- Office Hours: By email correspondance.
- Anonymous Feedback Form

# Prerequisites

This course is designed to bring students to the current frontier of knowledge on geometric generative models so that ideally, their course projects can make a novel contribution that can either be algorithmic, theoretical, or empirical. A previous background in machine learning is strongly recommended. Linear algebra, basic multivariate calculus, basics of working with probability, and programming skills are required. No background in geometry or generative models is needed for the course but any such knowledge may aid in a deeper engagement with the course material. However, it is strongly recommended that this course is not the first—or even second—Machine Learning course, and if there are any specific doubts please contact the instructor for special permission or equivalency of prerequisites.

- Math 223 (Linear Algebra 2) or equivalent
- COMP 551 (Applied Machine Learning) or equivalent
- Math 323 (Probability) or equivalent
- Math 222 (Calculus 3) or equivalent

# Course Structure

The first third of the course will consist of lectures covering necessary background material in geometric deep learning and generative models, after which every week will focus on student presentations on a pre-selected topic, using a couple of papers as reference. Each student must prepare one presentation individually or as a pair for the semester which is independent of their course project. The final two weeks will be reserved for course project presentations. In-class discussion will center around:

- Understanding the strengths and weaknesses of these methods.
- Understanding the relationships between these methods, and with previous approaches.
- Extensions or applications of these methods.
- Experiments that might better illuminate their properties.

# Grading

There will be an ungraded but mandatory project extended abstract/project proposal submission on Week 6 of the course. The course project, including project proposal, must be formatted in Latex using the NeurIPS 2022 (Preprint option) template. Latex Template

- Weekly Paper Reviews (1-page) on any
**Core**paper from that week. (Ungraded) - 20% In-class paper presentations
- 30% Student project presentation
- 50% Student project report

# Course Project

The course project can be done individually, in pairs, or in rare cases teams of 3. The project will be evaluated wholistically, on a number of criterias—not unsimilar to an actual review of a conference paper. The grading will take into account the number of people on a project. Some of these include: novelty of the idea, how well you present them in the report, how clearly you position your work relative to existing literature, how illuminating your experiments are, and well-supported your conclusions are.

Each course project will require a 2 page project proposal which can be thought of as an extended abstract. The actual project report will need to be 4-8 pages in length in the format of a NeurIPS paper in Preprint mode Latex template. The report should include an abstract, introduction, coverage of related work, a minimum viable project with some demonstration either theoretical or empirical of the ideas soundness, and if needed some future looking thoughts on how to proceed if given more time.

The Due Date for the Course Project is December 16th, 2022 at midnight.

# Reading List:

## Textbooks

There is no required textbook for this course but the “Geometric Deep Learning proto book” is a good reference for much of the technical foundations covered in this course. Another fantastic resource is the “Differential Geometry for Generative Modeling” textbook.

## Prerequisite readings/material:

- Self-Assessment Quiz
- Automatic Differentiation and Variational Inference
- Normalizing Flows for Probabilistic Modeling and Inference

## Sept 2nd - Week 1: Geometry Primer Part I (Prakash Lectures):

- Lecture Notes
- Week 1 Administrative Slides
- Topology: open and closed sets, continuous functions, convergence, metrics, compactness.
- Manifolds I
- Smoothness
- Charts

## Sept 9th - Week 2: Geometry Primer Part II (Prakash Lectures):

- Lecture Notes
- Manifolds II
- Tangent Vectors
- Tensors
- Tangent Bundle
- Metrics
- Affine connections
- Lie groups

## Sept 16th - Week 3: Deep Generative Models Primer Part I (Joey Lectures)

Deep generative models learn to transform unstructured noise to highly structured data like natural images. While these models may come in various forms but they can be broadly classified as either likelihood-based or implicit models. The former model class is already quite rich with popular modeling families such as VAE’s, and Normalizing Flows and will be the starting point for this week’s topics.

- Lecture Notes
- Week 3 Slides
- Auto-Encoding Variational Bayes
- Variational Inference with Normalizing Flows
- Density Estimation with RealNVP
- Neural Ordinary Differential Equations

## Sept 23rd - Week 4: Deep Generative Models Primer Part II (Joey Lectures)

In recent years there has been a resurgence of old ideas but repackaged for a modern time. Specifically, score matching and diffusion models have existed in some form or other prior to their recent renaissance but the main technical novelties lie—beyond new techniques that allow for fast training, inference, and impressive sampling quality—in their connection to Stochastic Differential Equations, VAE’s and Continuous Normalizing Flows. This week will focus on exploring these connections in detail and as we will see many new ideas are old ideas in disguise.

- Lecture Notes
- Week 4 Slides
- Generative Adversarial Networks
- Generative Modeling by Estimating Gradients of the Data Distribution
- Denoising diffusion probabilistic models
- Score-Based Generative Modeling through Stochastic Differential Equations
- Score Matching Blog Post

## Sept 30th - Week 5: Distributions on Manifolds

Defining probability distributions on manifolds requires specific care as many familiar notions in Euclidean geometry become incompatible for general manifolds. For example, the classical Gaussian distribution has at least three different instantiations in the manifold setting, e.g. Restricted, Wrapped, and the Riemannian normal distribution. We will focus on covering mainly the Riemannian normal distribution as well as the Von Mises Fisher distribution which is typically used in Spherical geometry.

### Core Readings

- Lecture Notes
- Week 5 Slides
- Reparameterizing Distributions on Lie Groups Presentation Slides
- Intrinsic Statistics on Riemannian Manifolds: Basic Tools for Geometric Measurements Presentation Slides
- Dispersion on a Sphere

### Extra Readings

- Directional data analysis under the general projected normal distribution
- Riemannian Gaussian Distributions on the Space of Symmetric Positive Definite Matrices

## Oct 7th - Week 6: Spherical Geometry

**Project Proposals Due**

The first wave of geometry-aware deep generative models focused largely on spherical geometry—i.e. Riemannian manifolds with positive curvature—as it is perhaps most accessible after Euclidean geometry. As a result, there are a few prominent work that exploit this structure to define hyperspherical latent spaces, or more complex distributions on the Sphere using Normalizing Flows or even GANs.

### Core Readings

- Lecture Notes
- Week 6 Slides
- Hyperspherical Variational Auto-Encoders Presentation Slides
- Normalizing Flows on Tori and Spheres Presentation Slides
- The Power Spherical distribution Presentation Slides

### Extra Readings

- Variational Autoencoders with Riemannian Brownian Motion Priors
- Sphere Generative Adversarial Network Based on Geometric Moment Matching

## Oct 14th - Week 7: Hyperbolic Geometry

Hyperbolic spaces—i.e. manifolds with constant negative curvature—have become an increasingly useful geometry in the modern machine learning toolkit. From modelling social networks, trees, biological networks, to hierarchical diffusion processes hyperbolic spaces have found tremendous practical advantages over Euclidean counterparts. Naturally, extending generative models to hyperbolic space requires taking into account manifold specific operations like the exponential and logarithmic maps, parallel transport all of which are key design decisions when constructing neural architectures that operate on these spaces.

### Core Readings

- Lecture Notes
- Week 7 Slides
- Google Collab Wrapped Normal Tutorial
- Continuous Hierarchical Representations with Poincaré Variational Auto-Encoders Presentation Slides
- A Wrapped Normal Distribution on Hyperbolic Space for Gradient-Based Learning Presentation Slides
- Latent Variable Modelling with Hyperbolic Normalizing Flows Presentation Slides

## Oct 21st - Week 8: Product and Latent Manifolds

Disentanglement is perhaps one of the key goals of unsupervised learning. For generative models, this effectively means isolating the main generative factors that give rise to observed data. Modeling such generative factors as latent variables in VAE like the setup is the dominant paradigm to infuse geometric priors about the data into practical modeling inductive biases. This week we will turn our attention to the various types of geometric structure that one can attach to a latent space such as group structure and product manifolds.

### Core Readings

- Lecture Notes: Read Chapter 1-6, and 8-10
- Week 8 Slides
- Towards a definition of disentangled representations Presentation Slides
- Learning mixed-curvature representations in product spaces Presentation Slides
- Mixed-curvature Variational Autoencoders

### Extra Readings

- Latent Space Oddity: on the Curvature of Deep Generative Models
- Metrics for deep generative models Presentation Slides
- Variational Autoencoders with Riemannian Brownian Motion Priors
- Pulling back information geometry

## Oct 28th - Week 9: Normalizing Flows on Riemannian Manifolds

How can we define flexible probability distributions on general Riemannian manifolds? Turns out one natural way to do so is to define an easy-to-sample prior distribution and a time-evolving vector field that transports this density to the desired target. This effectively generalizes the continuous normalizing flow approach previously seen in week 3 to Riemannian manifolds and this week we will cover 3 papers that were published concurrently on this very topic. Extending the CNF’s to manifolds typically requires backpropping through an ODE solver which is computationally expensive. In this week we will cover a series of normalizing flows that sidestep this expensive computation by using various methodological innovations such as using convex potentials from Riemannian optimal transport to neural implementation of Moser’s trick which led to MoserFlow (NeurIPS 2021 outstanding paper).

### Core Readings

- Week 9 Slides
- Riemannian Continuous Normalizing Flows
- Moser Flow: Divergence-based Generative Modeling on Manifolds Presentation Slides
- Matching Normalizing Flows and Probability Paths on Manifolds Presentation Slides

### Extra Readings

- Neural Manifold Ordinary Differential Equations
- Neural Ordinary Differential Equations on Manifolds
- The Riemannian Geometry of Deep Generative Models
- Riemannian Convex Potential Maps

## Nov 4th - Week 10: Equivariant Generative Models

Much of observed data is a result of physical processes which have symmetries. These symmetries manifest themselves as equivariances and invariances to certain transformation groups, e.g. translation, rotation, scaling, etc …, and imbuing generative models with these structural inductive biases is a core design principle. In this week we will cover many types of equivariant generative models and their application to physics, molecular dynamics, and many more practical domains.

### Core Readings

- Week 10 Slides
- Equivariant Flows: exact likelihood generative learning for symmetric densities Presentation Slides
- Equivariant Manifold Flows Presentation Slides
- Group Equivariant Generative Adversarial Networks Presentation Slides
- E(n) Equivariant Normalizing Flows Presentation Slides

### Extra Readings

- Equivariant Finite Normalizing Flows
- Implicit Riemannian Concave Potential Maps
- Sampling SU(N) with gauge equivariant flows

## Nov 11th - Week 11: Geometric Score and Diffusion Models

Score and Diffusion models are the current state of the art generative models for both likelihood estimation as well sample quality. Given their performance for images it is a natural question on whether these models can be adapted to more complex geometries. A recent influx of papers definitively answer this question with compelling samples ranging from spherical, hyperbolic and toroidal geometry but also practical use cases in molecular simulations.

### Core Readings

- Week 11: Presentation and Course Project Guidelines
- A short note on the Feynmann-Kac Representation
- Equivariant Diffusion for Molecule Generation in 3D Presentation Slides
- Riemannian Score-Based Generative Modeling Presentation Slides
- Riemannian Diffusion Models Presentation Slides
- GeoDiff: A Geometric Diffusion Model For Molecular Conformation Generation Presentation Slides

## Nov 18th - Week 12: Student Presentations Part I

## Nov 25th - Week 13: Student Presentations Part II

**Last class of the semester. Projects reports are due on Dec 16th**