Foundations of Graphical Models
Fall 2018
Columbia University

Course information

• Description and Syllabus
• Instructor: David M. Blei (office hours]
• Teaching Assistant: Keyon Vafa (Office hours: Tue 2-4PM in DSI)
• Meeting: Tue/Thu 8:40AM-9:55AM, Havermeyer 209
• Piazza site
• Final project guidelines
• LaTeX template

Assignments

• Homework 0
• Homework 1; datasets
• Homework 2; datasets
• Project milestone

Topics and readings

Below are the topics of the class and some readings about each. (Some readings are not yet set.) The readings from Blei (2018) will be made available to the class.

The readings are at different levels: some are basic and some are advanced. We chose them to provide interesting and fundamental material about the topics; the lectures will not necessarily cover or follow all of this material.

1. Introduction
   • "Build, compute, critique, repeat: Data analysis with latent variable models" (Blei, 2014)
   • Slides
2. The ingredients of probabilistic models
   • "The ingredients of probabilistic models" (Blei, 2018; Chapter 2)
   • "Model-based machine learning" (Bishop, 2013)
   • "Some issues in the foundations of statistics" (Freedman, 1994)
3. Bayesian mixture models and the Gibbs sampler
   • "Bayesian mixture models (and an introduction to Gibbs sampling)" (Blei, 2018; Chapter 3)
   • "Identifying Bayesian mixture models" (Betancourt, 2018)
   • "Probabilistic inference using Markov chain Monte Carlo methods" (Sections 1-4) (Neal, 1993)
4. Mixed-membership models, topic models, and variational inference
   • "Mixed-membership models (and an introduction to variational inference)" (Blei, 2018; Chapter 4)
   • "Probabilistic topic models" (Blei, 2012)
   • "Applications of topic models" (Boyd-Graber et al., 2017)
   • "Variational inference: A review for statisticians" (Blei et al., 2017)
5. Matrix factorization and efficient MAP inference
   • "Matrix factorization models" (Blei, 2018; Chapter 5)
   • "Matrix factorization techniques for recommender systems" (Koren et al., 2009)
   • "Learning the parts of objects by non-negative matrix factorization" (Lee and Seung, 1999)
   • "Scalable Recommendation with Hierarchical Poisson Factorization" (Gopalan et al., 2015)
   • "Tensor decompositions and applications" (Kolda and Bader, 2009)
6. Deep generative models and black box variational inference
   • "Stochastic backpropagation and approximate inference in deep generative models" (Rezende et al., 2014)
   • "Autoencoding variational Bayes" (Kingma and Welling, 2013)
   • "Deep exponential families" (Ranganath et al., 2015)
   • "Black box variational inference" (Ranganath et al., 2014)
   • "Mean field theory for sigmoid belief networks" (Saul et al., 1996)
   • "Representation learning: A review and new perspectives" (Bengio et al., 2013)
7. Exponential families, conjugate priors, and generalized linear models
   • "The exponential family" (Blei, 2018; Chapter 7)
   • "The exponential family" (Bishop, 2006; Section 2.4)
   • "An outline of generalized linear models" (McCullagh and Nelder, 1989; Chapter 2)
   • "Conjugate priors for exponential families" (Diaconis and Ylvisaker, 1979)
   • "Exponential families in theory and practice" (Efron, 2018)
8. Hierarchical models, robust models, and empirical Bayes
   • "Multi-level structures" (Gelman and Hill, 2007; Chapter 11)
   • "Multi-level linear models: The basics" (Gelman and Hill, 2007; Chapter 12)
   • "Bayes, oracle Bayes, and empirical Bayes" (Efron, 2017)
9. The theory of graphical models
   • "Conditional independence and factorization" (Jordan, 2003; Chapter 2)
   • "The elimination algorithm" (Jordan, 2003; Chapter 3)
   • "Probability propagation and factor graphs" (Jordan, 2003; Chapter 4)
10. Advanced topics in variational inference
    • "Graphical models, exponential families, and variational inference" (Waingwright and Jordan, 2008)
    • "Covariance, robustness, and variational Bayes" (Broderick et al., 2018)
11. Model checking and model diagnosis
    • "Bayesianly justifiable and relevant frequency calculations for the applied statistician" (Rubin, 1984)
    • "Posterior predictive assessment of model fitness via realized discrepancies" (Gelman et al., 1996)
    • "Philosophy and the practice of Bayesian statistics" (Gelman and Shalizi, 2013)
12. An introduction to causality
    • "Statistics and causal inference" (Holland, 1986)
    • "Causal inference in statistics: An overview" (Pearl, 2009)