Summer 2023 - CMPT 419 D100

Special Topics in Artificial Intelligence (3)

Class Number: 4011

Delivery Method: In Person

Overview

  • Course Times + Location:

    May 8 – Aug 4, 2023: Mon, 10:30 a.m.–12:20 p.m.
    Burnaby

    May 8 – Aug 4, 2023: Wed, 10:30–11:20 a.m.
    Burnaby

Description

CALENDAR DESCRIPTION:

Current topics in artificial intelligence depending on faculty and student interest.

COURSE DETAILS:

Advanced Methods in Natural Language Processing

Deep learning for NLP has been successful at improving results on many NLP tasks by relying on scaling up models, and as a consequence the models that scale better are the ones that are studied and used both in research and in industry. This course will be focused on current advanced methods that are considered state-of-the-art in NLP. These approaches are built on the Transformer model which gets rid of difficult-to-train recurrent models in favor of positional encoding, multi-headed self-attention and various techniques to allow training very deep feed-forward layers. This course will focus on the various aspects of training large models on natural language, fine-tuning them to specific NLP tasks and then using inference and model selection to run these models. 

The course will focus on the following topics, with an emphasis on efficient approaches that can be run without an expensive compute budget:
  1. Transformer models in NLP
  2. Data Collection and Preprocessing
  3. Model design choices
  4. Pre-training (PLMs - pre-trained language models)
  5. Fine-tuning PLMs
  6. Inference
  7. Model Selection

We will be reading papers on these topics and students will be presenting the papers and leading the discussion in class in collaboration with the instructor. The overall goal of the course will be for each student to produce a project that focuses on improving efficiency in one or more of these aspects of advanced methods in NLP.

COURSE-LEVEL EDUCATIONAL GOALS:

At the end of this course, the student should:

  1. Understand advanced NLP models such as Transformers.
  2. Be able to write code to implement such models.
  3. Fine-tune an existing pre-trained language model for a particular NLP task or tasks.
  4. Fine-tune an existing pre-trained language model with frozen parameters.
  5. Write the inference code necessary to solve an NLP task.
  6. Understand hyperparameter search.
  7. Be able to modify the basic model architecture to improve the state-of-the-art in at least one of the topics covered in this course.

Grading

  • Programming assignments 30%
  • In class exercises and presentations 30%
  • Project proposal 10%
  • Poster presentation 15%
  • Final project paper 15%

REQUIREMENTS:

  • CMPT 413
  • Pytorch
  • Intellectual curiosity

Materials

MATERIALS + SUPPLIES:

Papers to read will be provided on the Syllabus page of the course website.

REQUIRED READING NOTES:

Your personalized Course Material list, including digital and physical textbooks, are available through the SFU Bookstore website by simply entering your Computing ID at: shop.sfu.ca/course-materials/my-personalized-course-materials.

Registrar Notes:

ACADEMIC INTEGRITY: YOUR WORK, YOUR SUCCESS

SFU’s Academic Integrity website http://www.sfu.ca/students/academicintegrity.html is filled with information on what is meant by academic dishonesty, where you can find resources to help with your studies and the consequences of cheating. Check out the site for more information and videos that help explain the issues in plain English.

Each student is responsible for his or her conduct as it affects the university community. Academic dishonesty, in whatever form, is ultimately destructive of the values of the university. Furthermore, it is unfair and discouraging to the majority of students who pursue their studies honestly. Scholarly integrity is required of all members of the university. http://www.sfu.ca/policies/gazette/student/s10-01.html

RELIGIOUS ACCOMMODATION

Students with a faith background who may need accommodations during the semester are encouraged to assess their needs as soon as possible and review the Multifaith religious accommodations website. The page outlines ways they begin working toward an accommodation and ensure solutions can be reached in a timely fashion.