Overview

Description

CALENDAR DESCRIPTION:

NLP plays an essential role in many applications, allowing people to communicate with computers through dialog systems, communicate with each other through machine translation systems, and find and process information. The course covers advanced topics in NLP, introducing the state-of-the-art methods for computational understanding, analysis, and generation of natural language text.

COURSE DETAILS:

In recent years, advances in deep learning models for NLP has transformed the ability of computers to converse with human using language, giving us multi-lingual, multi-models that are capable of answering questions, composing message, translating and summarizing documents.   The development of large language models (LLMs) are built on top of neural models such as transformers, that allows for the scaling up of models and training with large amounts of data.  In this course, we will focus on current state-of-the-art methods in NLP including how to do parameter efficient fine-tuning, techniques for scaling models to long sequences, etc.  We will also go beyond transformers to learn alternative architectures such as state-space models. 

Topics

1. Transformer models in NLP
2. Model design choices
3. Pre-training and fine-tuning LMs
4. Advanced inference techniques 
5. LLM agents 
6. Multimodal models

We will be reading papers on these topics and students will be presenting the papers and leading the discussion in class in collaboration with the instructor. The overall goal of the course will be for each student to produce a project building on state-of-the-art models in NLP.

COURSE-LEVEL EDUCATIONAL GOALS:

At the end of this course, the student should:

1. Understand advanced NLP models such as Transformers.
2. Be able to write code to implement such models.
3. Fine-tune an existing pre-trained language model for a particular NLP task or tasks.
4. Fine-tune an existing pre-trained language model with frozen parameters.
5. Write the inference code necessary to solve an NLP task.
6. Understand hyperparameter search.
7. Be able to modify the basic model architecture to improve the state-of-the-art in at least one of the topics covered in this course.

Grading

Materials

MATERIALS + SUPPLIES:

Reference Books

• Speech and Language Processing (3rd ed), Dan Jurafsky and James H. Martin, 2019, https://web.stanford.edu/~jurafsky/slp3/
• Neural Network Methods for Natural Language Processing, Yoav Goldberg, Morgan and Claypool, 2017, http://www.morganclaypool.com/doi/10.2200/S00762ED1V01Y201703HLT037
• Natural Language Processing, Jacob Eisenstein, The MIT Press, 2018, https://github.com/jacobeisenstein/gt-nlp-class/blob/master/notes/eisenstein-nlp-notes.pdf

RECOMMENDED READING:

REQUIRED READING NOTES: