COMP760: Geometry and Generative Models

Home Page

Co-Instructors: Joey Bose and Prakash Panangaden

Course Overview

In recent years Deep Generative Models have seen remarkable success over a variety of data domains such as images, text, and audio to name a few. However, the predominant approach in many of these models (e.g. GANS, VAE, Normalizing Flows) is to treat data as fixed-dimensional continuous vectors in some Euclidean space, despite significant evidence to the contrary (e.g. 3D molecules). This course places a direct emphasis on learning generative models for complex geometries described via manifolds, such as spheres, tori, hyperbolic spaces, implicit surfaces, and homogeneous spaces. The purpose of this seminar course is to understand the key design principles that underpin the new wave of geometry-aware generative models that treat the rich geometric structure in data as a first-class citizen. This seminar course will also serve to develop extensions to these approaches at the leading edge of research and as a result, a major component of the course will focus on class participation through presenting papers and a thematically-relevant course project.

  • Location: Mila Auditorium 1
  • Time: Fridays 1pm - 4pm
  • Office Hours: By email correspondance.
  • Anonymous Feedback Form

Prerequisites

This course is designed to bring students to the current frontier of knowledge on geometric generative models so that ideally, their course projects can make a novel contribution that can either be algorithmic, theoretical, or empirical. A previous background in machine learning is strongly recommended. Linear algebra, basic multivariate calculus, basics of working with probability, and programming skills are required. No background in geometry or generative models is needed for the course but any such knowledge may aid in a deeper engagement with the course material. However, it is strongly recommended that this course is not the first—or even second—Machine Learning course, and if there are any specific doubts please contact the instructor for special permission or equivalency of prerequisites.

  • Math 223 (Linear Algebra 2) or equivalent
  • COMP 551 (Applied Machine Learning) or equivalent
  • Math 323 (Probability) or equivalent
  • Math 222 (Calculus 3) or equivalent

Course Structure

The first third of the course will consist of lectures covering necessary background material in geometric deep learning and generative models, after which every week will focus on student presentations on a pre-selected topic, using a couple of papers as reference. Each student must prepare one presentation individually or as a pair for the semester which is independent of their course project. The final two weeks will be reserved for course project presentations. In-class discussion will center around:

  • Understanding the strengths and weaknesses of these methods.
  • Understanding the relationships between these methods, and with previous approaches.
  • Extensions or applications of these methods.
  • Experiments that might better illuminate their properties.

Grading

There will be an ungraded but mandatory project extended abstract/project proposal submission on Week 6 of the course. The course project, including project proposal, must be formatted in Latex using the NeurIPS 2022 (Preprint option) template. Latex Template

  • Weekly Paper Reviews (1-page) on any Core paper from that week. (Ungraded)
  • 20% In-class paper presentations
  • 30% Student project presentation
  • 50% Student project report

Course Project

The course project can be done individually, in pairs, or in rare cases teams of 3. The project will be evaluated wholistically, on a number of criterias—not unsimilar to an actual review of a conference paper. The grading will take into account the number of people on a project. Some of these include: novelty of the idea, how well you present them in the report, how clearly you position your work relative to existing literature, how illuminating your experiments are, and well-supported your conclusions are.

Each course project will require a 2 page project proposal which can be thought of as an extended abstract. The actual project report will need to be 4-8 pages in length in the format of a NeurIPS paper in Preprint mode Latex template. The report should include an abstract, introduction, coverage of related work, a minimum viable project with some demonstration either theoretical or empirical of the ideas soundness, and if needed some future looking thoughts on how to proceed if given more time.

The Due Date for the Course Project is December 16th, 2022 at midnight.

Reading List:

Textbooks

There is no required textbook for this course but the “Geometric Deep Learning proto book” is a good reference for much of the technical foundations covered in this course. Another fantastic resource is the “Differential Geometry for Generative Modeling” textbook.

Prerequisite readings/material:

Sept 2nd - Week 1: Geometry Primer Part I (Prakash Lectures):

Sept 9th - Week 2: Geometry Primer Part II (Prakash Lectures):

  • Lecture Notes
  • Manifolds II
  • Tangent Vectors
  • Tensors
  • Tangent Bundle
  • Metrics
  • Affine connections
  • Lie groups

Sept 16th - Week 3: Deep Generative Models Primer Part I (Joey Lectures)

Deep generative models learn to transform unstructured noise to highly structured data like natural images. While these models may come in various forms but they can be broadly classified as either likelihood-based or implicit models. The former model class is already quite rich with popular modeling families such as VAE’s, and Normalizing Flows and will be the starting point for this week’s topics.

Sept 23rd - Week 4: Deep Generative Models Primer Part II (Joey Lectures)

In recent years there has been a resurgence of old ideas but repackaged for a modern time. Specifically, score matching and diffusion models have existed in some form or other prior to their recent renaissance but the main technical novelties lie—beyond new techniques that allow for fast training, inference, and impressive sampling quality—in their connection to Stochastic Differential Equations, VAE’s and Continuous Normalizing Flows. This week will focus on exploring these connections in detail and as we will see many new ideas are old ideas in disguise.

Sept 30th - Week 5: Distributions on Manifolds

Defining probability distributions on manifolds requires specific care as many familiar notions in Euclidean geometry become incompatible for general manifolds. For example, the classical Gaussian distribution has at least three different instantiations in the manifold setting, e.g. Restricted, Wrapped, and the Riemannian normal distribution. We will focus on covering mainly the Riemannian normal distribution as well as the Von Mises Fisher distribution which is typically used in Spherical geometry.

Core Readings

Extra Readings

Oct 7th - Week 6: Spherical Geometry

Project Proposals Due

The first wave of geometry-aware deep generative models focused largely on spherical geometry—i.e. Riemannian manifolds with positive curvature—as it is perhaps most accessible after Euclidean geometry. As a result, there are a few prominent work that exploit this structure to define hyperspherical latent spaces, or more complex distributions on the Sphere using Normalizing Flows or even GANs.

Core Readings

Extra Readings

Oct 14th - Week 7: Hyperbolic Geometry

Hyperbolic spaces—i.e. manifolds with constant negative curvature—have become an increasingly useful geometry in the modern machine learning toolkit. From modelling social networks, trees, biological networks, to hierarchical diffusion processes hyperbolic spaces have found tremendous practical advantages over Euclidean counterparts. Naturally, extending generative models to hyperbolic space requires taking into account manifold specific operations like the exponential and logarithmic maps, parallel transport all of which are key design decisions when constructing neural architectures that operate on these spaces.

Core Readings

Oct 21st - Week 8: Product and Latent Manifolds

Disentanglement is perhaps one of the key goals of unsupervised learning. For generative models, this effectively means isolating the main generative factors that give rise to observed data. Modeling such generative factors as latent variables in VAE like the setup is the dominant paradigm to infuse geometric priors about the data into practical modeling inductive biases. This week we will turn our attention to the various types of geometric structure that one can attach to a latent space such as group structure and product manifolds.

Core Readings

Extra Readings

Oct 28th - Week 9: Normalizing Flows on Riemannian Manifolds

How can we define flexible probability distributions on general Riemannian manifolds? Turns out one natural way to do so is to define an easy-to-sample prior distribution and a time-evolving vector field that transports this density to the desired target. This effectively generalizes the continuous normalizing flow approach previously seen in week 3 to Riemannian manifolds and this week we will cover 3 papers that were published concurrently on this very topic. Extending the CNF’s to manifolds typically requires backpropping through an ODE solver which is computationally expensive. In this week we will cover a series of normalizing flows that sidestep this expensive computation by using various methodological innovations such as using convex potentials from Riemannian optimal transport to neural implementation of Moser’s trick which led to MoserFlow (NeurIPS 2021 outstanding paper).

Core Readings

Extra Readings

Nov 4th - Week 10: Equivariant Generative Models

Much of observed data is a result of physical processes which have symmetries. These symmetries manifest themselves as equivariances and invariances to certain transformation groups, e.g. translation, rotation, scaling, etc …, and imbuing generative models with these structural inductive biases is a core design principle. In this week we will cover many types of equivariant generative models and their application to physics, molecular dynamics, and many more practical domains.

Core Readings

Extra Readings

Nov 11th - Week 11: Geometric Score and Diffusion Models

Score and Diffusion models are the current state of the art generative models for both likelihood estimation as well sample quality. Given their performance for images it is a natural question on whether these models can be adapted to more complex geometries. A recent influx of papers definitively answer this question with compelling samples ranging from spherical, hyperbolic and toroidal geometry but also practical use cases in molecular simulations.

Core Readings

Nov 18th - Week 12: Student Presentations Part I

Nov 25th - Week 13: Student Presentations Part II

Last class of the semester. Projects reports are due on Dec 16th