Overview

Description

CALENDAR DESCRIPTION:

NLP plays an essential role in many applications, allowing people to communicate with computers through dialog systems, communicate with each other through machine translation systems, and find and process information. The course covers advanced topics in NLP, introducing the state-of-the-art methods for computational understanding, analysis, and generation of natural language text.

COURSE DETAILS:

In recent years, advances in deep learning models for NLP has transformed the ability of computers to converse with human using language, giving us multi-lingual, multi-models that are capable of answering questions, composing message, translating and summarizing documents.   The development of large language models (LLMs) are built on top of neural models such as transformers, that allows for the scaling up of models and training with large amounts of data.  In this course, we will focus on current state-of-the-art methods in NLP including how to do parameter efficient fine-tuning, techniques for scaling models to long sequences, etc.  We will also go beyond transformers to learn alternative architectures such as state-space models. 

Topics

1. Transformer models in NLP
2. Model design choices
3. Pre-training and post-training for LLMs
4. Advanced inference techniques for LLMs
5. LLM agents 
6. Multimodal models

We will be reading papers on these topics and students will be presenting the papers and leading the discussion in class in collaboration with the instructor. The overall goal of the course will be for each student to produce a project building on state-of-the-art models in NLP.

COURSE-LEVEL EDUCATIONAL GOALS:

At the end of this course, the student should:

1. Understand advanced NLP models such as Transformers and how they are pretrained and fine-tuned in modern LLMs.
2. Be able to write code to implement such models and how to modify code to adapt these models for new tasks
3. Be able to fine-tune an existing pre-trained language model for particular NLP tasks 
4. Be able to read and understand recent NLP research papers, and clearly present and explain key developments in these papers
5. Be able to propose a new idea building on top of recent NLP work, and conduct experiments to test the validity of the idea
6. Be able to write a clear report summarizing their work

Grading

Materials

MATERIALS + SUPPLIES:

Reference Books

• Neural Network Methods for Natural Language Processing, Yoav Goldberg, Morgan and Claypool, 2017. 
• Natural Language Processing, Jacob Eisenstein, The MIT Press, 2018. 

REQUIRED READING:

RECOMMENDED READING:

REQUIRED READING NOTES: