Date Lecture Topics
8/30 Lecture 1.1:
Course introduction
[ slides | video ]

Multimodal core challenges
Course syllabus
9/1 Lecture 1.2:
Multimodal applications and datasets
[ slides | video ]

Research tasks and datasets
Team projects
9/6 Lecture 2.1:
Basic concepts: neural networks
[ slides | video ]

Gradient and optimization
Loss functions and neural networks
9/8 Lecture 2.2:
Unimodal representations
[ slides | video ]

Dimensions of heterogeneity
Visual representations
9/13 Lecture 3.1:
Unimodal representations
[ slides | video ]

Language representations
Signals, graphs and other modalities
9/15 Lecture 3.2:
Multimodal representations
[ slides | video ]

Cross-modal interactions
Multimodal fusion
9/20 Lecture 4.1:
Multimodal representations
[ slides | video ]

Coordinated representations
Multimodal fission
9/22 Lecture 4.2:
Multimodal alignment and grounding
[ slides | video ]

Explicit alignment
Multimodal grounding
9/27 Lecture 5.1: Project Hours (live working sessions instead of lectures)
9/29 Lecture 5.2:
Aligned representations
[ slides | video ]

Self-attention transformer models
Masking and self-supervised learning
10/4 Lecture 6.1: Multimodal aligned representations
[ slides | video ]

Multimodal transformers
Video and graph representations
10/6 Lecture 6.2:
Multimodal Reasoning
[ slides | video ]

Structured and hierarchical models
Memory models
10/11 Lecture 7.1:
Multimodal Reasoning
[ slides | video ]

Reinforcement learning
Discrete structure learning
10/13 Lecture 7.2:
Multimodal Reasoning
[ slides | video ]

Logical and causal inference
External knowledge
10/18 Lecture 8.1:
Fall Break Break – No lectures
10/20 Lecture 8.2:
Fall Break – No lectures
10/25 Lecture 9.1:
Generation
[ slides | video ]

Translation, summarization, creation
{"Generative models"=>"VAEs"}
10/27 Lecture 9.2:
Generation
[ slides | video ]

GANs and diffusion models
Model evaluation and ethics
11/1 Lecture 10.1:
Project Hours
[ slides | video ]

11/3 Lecture 10.2:
Project Hours
[ slides | video ]

11/8 Lecture 11.1:
Transference
[ slides | video ]

Modality transfer
Multimodal co-learning
11/10 Lecture 11.2:
Quantification
[ slides | video ]

Heterogeneity and interactions
Biases and fairness
11/15 Lecture 12.1:
Project Hours
[ slides | video ]

Project Hours
11/17 Lecture 12.2:
New research directions
[ slides | video ]

Recent approaches in multimodal ML
11/22 Lecture 13.1:
Thanksgiving Week – No Class –
[ slides | video ]

11/24 Lecture 13.2: Thanksgiving Week – No Class –
[ slides | video ]

11/30 Lecture 14.1: Language, Vision, and Actionst
[ slides | video ]

Motion and navigation
Robots and embodied AI
12/2 Lecture 14.2:
Multimodal Applications
[ slides | video ]

Healthcare and affective computing
Artificial social intelligence
12/6 Lecture 15.1: Final project assignment (live working sessions instead of lectures)
12/8 Lecture 15.2: Final project assignment (live working sessions instead of lectures)