Simon Fraser University
Engaging the World

Course Outlines

Fall 2025 - CMPT 409 D200

Special Topics in Theoretical Computing Science (3)

Optimization for Machine Learning

Class Number: 5518

Delivery Method: In Person

Overview

  • Course Times + Location:

    Sep 3 – Dec 2, 2025: Tue, 11:30 a.m.–1:20 p.m.
    Burnaby

    Sep 3 – Dec 2, 2025: Thu, 11:30 a.m.–12:20 p.m.
    Burnaby

  • Prerequisites:

    CMPT 307 with a minimum grade of C-.

Description

CALENDAR DESCRIPTION:

Current topics in theoretical computing science depending on faculty and student interest.

COURSE DETAILS:

Numerical optimization plays a key role in designing better algorithms for data science and artificial intelligence. This course (Optimization for Machine Learning) introduces the foundational concepts of convex and non-convex optimization with applications to machine learning. It will give the students experience in 1. Proving theoretical guarantees for optimization algorithms, 2. Analyzing machine learning (ML) problems from an optimization perspective and 3. Developing and analyzing new optimization methods for ML applications.

* Note that this course is cross-listed as a special topics course in both Theory and Artificial Intelligence *

Topics

  • Basics: Subdifferentials, Optimality conditions, Conjugates, Lipschitz continuity, Convexity
  • Machine Learning Basics: Linear/Logistic Regression
  • (Non)-Convex minimization 1: (Projected/Proximal) Gradient Descent, Nesterov/Polyak momentum
  • (Non)-Convex minimization 2: Mirror Descent, Newton/Quasi-Newton/Gauss-Newton method
  • (Non)-Convex minimization 3: Stochastic gradient descent (SGD), Variance reduction techniques
  • (Non)-Convex minimization 4: Adaptivity for SGD, Coordinate Descent
  • Applications to training ML models (logistic regression, kernel machines, neural networks)
  • Online optimization 1: Regret minimization, Online to Batch, Follow the (regularized) leader
  • Online optimization 2: Optimistic Gradient Descent, Adaptive gradient methods (AdaGrad, Adam)
  • Applications to Imitation learning, Reinforcement learning
  • Min-Max optimization 1: Primal-dual methods, (Stochastic) Gradient Descent-Ascent, Proximal point
  • Min-Max optimization 2: (Stochastic) Extragradient, Acceleration, Variance reduction
  • Applications to GANs, Robust optimization, Multi-agent RL

Grading

NOTES:

There will be a couple of assignments with the major evaluation components being a final project. The details will be discussed in the first week of classes.

REQUIREMENTS:

Students taking the course are expected to have a solid understanding of probability (e.g. CMPT 210, STAT 270, STAT 271), linear algebra (e.g. MATH 240), and some basic knowledge of multivariable calculus (e.g. MATH 251). It is encouraged (though not necessary) for students to have taken (or be simultaneously taking) an introductory course on machine learning (e.g. CMPT 410/726).

Materials

MATERIALS + SUPPLIES:

Reference Books

  • Convex Optimization, Boyd and Vandenberghe, 2004, 9780521833783
  • Numerical Optimization, Nocedal and Wright, 2006, 9780387303031
  • First-order Methods in Optimization, Beck, 2017, 9781611974980
  • Convex Optimization: Algorithms and Complexity, Bubeck, 2014, 9781601988607
  • Lectures on Convex Optimization, Nesterov, 2018, 9783319915777

REQUIRED READING NOTES:

Your personalized Course Material list, including digital and physical textbooks, are available through the SFU Bookstore website by simply entering your Computing ID at: shop.sfu.ca/course-materials/my-personalized-course-materials.

Department Undergraduate Notes:

The following are default policies in the School of Computing Science. Please check your course syllabus whether the instructor has chosen a different policy for your class, otherwise the following policies apply.
 
  • Students must attain an overall passing grade on the weighted average of exams in the course in order to get a C- or higher.
  • All student requests for accommodations for their religious practices must be made in writing by the end of the first week of classes, or no later than one week after a student adds a course. After considering a request, an instructor may provide a concession or may decline to do so. Students requiring accommodations as a result of a disability can contact the Centre for Accessible Learning (caladmin@sfu.ca).

Registrar Notes:

ACADEMIC INTEGRITY: YOUR WORK, YOUR SUCCESS

At SFU, you are expected to act honestly and responsibly in all your academic work. Cheating, plagiarism, or any other form of academic dishonesty harms your own learning, undermines the efforts of your classmates who pursue their studies honestly, and goes against the core values of the university.

To learn more about the academic disciplinary process and relevant academic supports, visit: 


RELIGIOUS ACCOMMODATION

Students with a faith background who may need accommodations during the term are encouraged to assess their needs as soon as possible and review the Multifaith religious accommodations website. The page outlines ways they begin working toward an accommodation and ensure solutions can be reached in a timely fashion.