Simon Fraser University
Engaging the World

Course Outlines

Fall 2024 - CMPT 409 D100

Special Topics in Theoretical Computing Science (3)

Optimization for Machine Learning

Class Number: 6311

Delivery Method: In Person

Overview

  • Course Times + Location:

    Sep 4 – Oct 11, 2024: Tue, 1:30–2:20 p.m.
    Burnaby

    Oct 16 – Dec 3, 2024: Tue, 1:30–2:20 p.m.
    Burnaby

    Sep 4 – Dec 3, 2024: Thu, 12:30–2:20 p.m.
    Burnaby

  • Prerequisites:

    CMPT 307 with a minimum grade of C-.

Description

CALENDAR DESCRIPTION:

Current topics in theoretical computing science depending on faculty and student interest.

COURSE DETAILS:

Numerical optimization plays a key role in designing better algorithms for data science and artificial intelligence. This course (Optimization for Machine Learning) introduces the foundational concepts of convex and non-convex optimization with applications to machine learning. It will give the students experience in 1. Proving theoretical guarantees for optimization algorithms, 2. Analyzing machine learning (ML) problems from an optimization perspective and 3. Developing and analyzing new optimization methods for ML applications.

* Note that this course is cross-listed as a special topics course in both Theory and Artificial Intelligence *

Topics

  • Basics: Subdifferentials, Optimality conditions, Conjugates, Lipschitz continuity, Convexity
  • Machine Learning Basics: Linear/Logistic Regression
  • (Non)-Convex minimization 1: (Projected/Proximal) Gradient Descent, Nesterov/Polyak momentum
  • (Non)-Convex minimization 2: Mirror Descent, Newton/Quasi-Newton/Gauss-Newton method
  • (Non)-Convex minimization 3: Stochastic gradient descent (SGD), Variance reduction techniques
  • (Non)-Convex minimization 4: Adaptivity for SGD, Coordinate Descent
  • Applications to training ML models (logistic regression, kernel machines, neural networks)
  • Online optimization 1: Regret minimization, Online to Batch, Follow the (regularized) leader
  • Online optimization 2: Optimistic Gradient Descent, Adaptive gradient methods (AdaGrad, Adam)
  • Applications to Imitation learning, Reinforcement learning
  • Min-Max optimization 1: Primal-dual methods, (Stochastic) Gradient Descent-Ascent, Proximal point
  • Min-Max optimization 2: (Stochastic) Extragradient, Acceleration, Variance reduction
  • Applications to GANs, Robust optimization, Multi-agent RL

Grading

NOTES:

There will be a couple of assignments with the major evaluation components being a final project. The details will be discussed in the first week of classes.

REQUIREMENTS:

Students taking the course are expected to have a solid understanding of probability (e.g. CMPT 210, STAT 270, STAT 271), linear algebra (e.g. MATH 240), and some basic knowledge of multivariable calculus (e.g. MATH 251). It is encouraged (though not necessary) for students to have taken (or be simultaneously taking) an introductory course on machine learning (e.g. CMPT 410/726). 

Materials

MATERIALS + SUPPLIES:

Reference Books

  • Convex Optimization, Boyd and Vandenberghe, 2004, 9780521833783
  • Numerical Optimization, Nocedal and Wright, 2006, 9780387303031
  • First-order Methods in Optimization, Beck, 2017, 9781611974980
  • Convex Optimization: Algorithms and Complexity, Bubeck, 2014, 9781601988607
  • Lectures on Convex Optimization, Nesterov, 2018, 9783319915777

REQUIRED READING NOTES:

Your personalized Course Material list, including digital and physical textbooks, are available through the SFU Bookstore website by simply entering your Computing ID at: shop.sfu.ca/course-materials/my-personalized-course-materials.

Registrar Notes:

ACADEMIC INTEGRITY: YOUR WORK, YOUR SUCCESS

SFU’s Academic Integrity website http://www.sfu.ca/students/academicintegrity.html is filled with information on what is meant by academic dishonesty, where you can find resources to help with your studies and the consequences of cheating. Check out the site for more information and videos that help explain the issues in plain English.

Each student is responsible for his or her conduct as it affects the university community. Academic dishonesty, in whatever form, is ultimately destructive of the values of the university. Furthermore, it is unfair and discouraging to the majority of students who pursue their studies honestly. Scholarly integrity is required of all members of the university. http://www.sfu.ca/policies/gazette/student/s10-01.html

RELIGIOUS ACCOMMODATION

Students with a faith background who may need accommodations during the term are encouraged to assess their needs as soon as possible and review the Multifaith religious accommodations website. The page outlines ways they begin working toward an accommodation and ensure solutions can be reached in a timely fashion.